A front and back end combined hidden danger analysis system and method thereof

Abstract

A front-end and back-end combined hidden danger analysis system and method thereof, the system comprising: a front-end camera module, a video target recognition module, an image definition recognition module, an image similarity recognition module, a front-end slice compression module, a back-end image analysis module, a back-end hidden danger analysis result module and a front-end loudspeaker module, the method comprising: one: image acquisition; two: hidden danger target recognition; three: image definition recognition; four: image similarity recognition; five: video slice compression; six: image analysis; seven: hidden danger analysis; eight: hidden danger broadcast reminder; the video frame is preliminarily detected through the front-end video target recognition module, only the frame containing the hidden danger target is reserved for subsequent processing; combined with the image definition recognition module and the image similarity recognition module, further eliminate the fuzzy and repeated frames, generate high-quality video segments, effectively reduce the image data amount required to be processed by the back-end multi-modal large model, thereby greatly reduce the consumption of token.

Description

Technical Field

[0001] This invention relates to the field of data processing, specifically to a front-end and back-end combined hidden danger analysis system and method. Background Technology

[0002] With increasingly stringent safety requirements in industrial sites, public safety, and construction, real-time identification and early warning of safety hazards have become crucial for ensuring personnel safety and preventing accidents. Currently, mobile video capture devices such as law enforcement recorders are widely used in on-site inspections and safety checks, recording the situation to provide evidence for post-incident investigation. However, existing technologies primarily rely on manual video playback for hazard identification, which is inefficient and makes real-time early warning difficult.

[0003] In recent years, with the development of artificial intelligence technology, especially the breakthroughs of multimodal large models (such as Qwen-VL and Qwen2-VL) in image understanding and visual question answering, new possibilities have been provided for the automated analysis of video content. By inputting video data into large models for semantic understanding, intelligent identification and judgment of security risks can be achieved. However, directly uploading complete video streams to servers for analysis has the following technical drawbacks: Excessive resource consumption: The processing of input data by multimodal large models depends on the number of tokens. Long-duration, high-frame-rate video streams will consume a lot of computing resources, leading to a sharp increase in processing costs and making it difficult to scale up in practical engineering.

[0004] High bandwidth requirements: Real-time uploading of high-definition video streams places high demands on network bandwidth, especially in mobile network environments, which can easily cause delays and stuttering, affecting real-time performance.

[0005] Invalid data redundancy: There are a lot of duplicate, blurry or irrelevant frames in the video. If all of them are uploaded without filtering, it will not only waste resources, but also interfere with the model's judgment efficiency. Summary of the Invention

[0006] The purpose of this invention is to provide a front-end and back-end integrated hidden danger analysis system and method, which avoids the excessive use of token resources by the full video in subsequent multimodal large models, thereby solving the existing technical defects and unmet technical requirements.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a front-end and back-end integrated hazard analysis system, including a law enforcement recorder, wherein the law enforcement recorder is further equipped with: The front-end camera module is used to capture images in front of it, generate video frame data based on the images, and then send the video frame data to the video target recognition module, the image sharpness recognition module, and the image similarity recognition module. The video target recognition module receives video frame data sent by the front-end camera module, detects the main target objects for hazard analysis in the video frame data (such as fire cabinets / fire extinguishers / electric boxes / wires / corridors / stairs / warning signs, etc.), and sends them to the front-end slicing and compression module. The image sharpness recognition module receives video frame data sent by the front-end camera module, obtains its sharpness gradient based on the video frame data, obtains the sharpness of the video frame based on its sharpness gradient, and sends the sharpness of the video frame to the front-end slicing and compression module. The image similarity recognition module receives video frame data sent by the front-end camera module, performs feature extraction and similarity calculation on the current video frame and the previous video frame, and then sends the similarity result to the front-end slice compression module. The front-end slicing and compression module receives the main target object for hidden danger analysis from the video frame data sent by the video target recognition module, the clarity of the video frame sent by the image clarity recognition module, and the similarity results sent by the image similarity recognition module. Based on the above information, it performs intelligent slicing and encoding strategy control to obtain video segments with high clarity and no hidden danger targets. The video segments are then compressed and sent to the back-end image analysis module. The backend image analysis module receives compressed video segments sent by the frontend slicing and compression module, processes the compressed video segments to obtain structured analysis results containing safety hazard risk judgment conclusions and confidence scores, and sends the structured analysis results to the backend hazard analysis result module. The backend hazard analysis result module receives the structured analysis results sent by the backend image analysis module, and matches the structured analysis results with the preset safety hazard processing rule library, that is, hazard analysis and processing, to obtain the corresponding processing results, and sends the processing results to the frontend speaker module, while storing the structured analysis results. The front-end speaker module receives the processing results sent by the back-end hidden danger analysis result module, converts the processing results into voice broadcast content, and broadcasts the voice broadcast content.

[0008] A front-end and back-end integrated hazard analysis method includes the following steps: 1. Image Acquisition 1.1) Acquire images of the area in front of it using a front-end camera module; 1.2) Generate video frame data based on the image and send the video frame data to the video target recognition module; II. Hazard Target Identification 2.1) The video target recognition module receives video frame data sent from the front-end camera module; 2.2) The main target objects for hazard analysis are detected in the video frame data through the video target recognition module; 2.3) Send the main target objects for hidden danger analysis in the video frame data to the front-end slice compression module; III. Image Sharpness Recognition 3.1) Receive video frame data sent from the front-end camera module through the image sharpness recognition module; 3.2) Obtain the sharpness gradient from the video frame data, and then obtain the sharpness of the video frame based on the sharpness gradient; 3.3) Send the resolution of the video frames to the front-end slicing and compression module; IV. Image Similarity Recognition 4.1) Receive video frame data sent from the front-end camera module through the image similarity recognition module; 4.2) Perform feature extraction and similarity calculation between the current video frame and the previous video frame; 4.3) Then send the similarity results to the front-end slice compression module; In this application, feature extraction employs techniques commonly found in the prior art.

[0009] V. Video Segment Compression 5.1) The front-end slicing and compression module receives the main target objects for hidden danger analysis from the video frame data sent by the video target recognition module, the clarity of the video frames sent by the image clarity recognition module, and the similarity results sent by the image similarity recognition module. 5.2) Based on the information received in step 5.1), perform intelligent slicing and encoding strategy control to obtain high-definition video segments without any hidden targets, and then compress the video segments. 5.3) Send the compressed video clips to the backend image analysis module; VI. Image Analysis 6.1) The backend image analysis module receives the compressed video segments sent by the frontend slicing and compression module; 6.2) The compressed video segments are processed to obtain structured analysis results containing security risk assessment conclusions and confidence scores; 6.3) Send the structured analysis results to the backend hidden danger analysis results module; VII. Hazard Analysis 7.1) Receive structured analysis results sent from the backend image analysis module through the backend hidden danger analysis result module; 7.2) And in conjunction with the preset safety hazard handling rule base, the structured analysis results are matched to obtain the corresponding processing results; 7.3) The processing results are sent to the front-end speaker module, and the structured analysis results are stored in the database for archiving and management. VIII. Hazard Reporting and Reminder 8.1) Receive the processing results sent by the backend hidden danger analysis result module through the front-end speaker module; 8.2) The processing results are converted into voice broadcast content and broadcast to provide real-time reminders to on-site personnel, forming a closed-loop feedback mechanism.

[0010] Preferably, the specific content of the main target object for hazard analysis in the detected video frame data in step 2.2) is as follows: The video target recognition module (i.e., the computing power of the law enforcement recorder) uses the YOLO lightweight neural network detector to detect targets of interest in the video frames. The targets of interest are the main objects of hazard analysis (such as fire cabinets / fire extinguishers / electrical boxes / wires / corridors / stairs / warning signs, etc.).

[0011] In this application, "YOLO lightweight neural network detector" refers to a single-stage object detection neural network model based on the YOLO (You Only LookOnce) architecture, designed to be lightweight by reducing the number of network layers, employing depthwise separable convolutions, and reducing the number of channels, making it suitable for mobile / embedded devices. In other words, "lightweight YOLO" / "lightweight neural network detector" refers to a YOLO-like single-stage object detection model with ≤10M parameters that can perform real-time inference (≥15FPS) on CPU / embedded devices, including but not limited to Tiny. YOLO, YOLOv5 n、YOLOv8 n, MobileYOLO, etc.

[0012] Furthermore, the categories of interest are based on preset data information.

[0013] Preferably, the specific content of obtaining the video frame clarity in step 3.2) is as follows: 3.3.1) Use the Sobel operator to calculate the gradient in the horizontal and vertical directions for each point in the video frame; 3.3.2) The horizontal and vertical gradients are squared, summed, and then the square root is taken to form the sharpness gradient; 3.3.3) Calculate the average of all sharpness gradients of the video frame, and use the result as the sharpness of the video frame.

[0014] Preferably, the specific content of the similarity calculation in step 4.2) is as follows: The image similarity recognition module uses the absolute difference method (SAD) to calculate the difference between the current video frame and the previous video frame pixel by pixel. The differences are accumulated and averaged, then divided by the pixel quantization range (e.g., 255) to obtain the difference coefficient of the video frame. Subtracting the difference coefficient of the video frame from 1 gives the similarity value. When the similarity value is high, the video frame image changes little; when the similarity value is low, the video frame image changes.

[0015] In this application, the quantization range of the pixel is a preset data information, and "absolute difference method (SAD)" can be understood as the sum of absolute differences.

[0016] Preferably, the specific content of obtaining the compressed video segment in step 5.2) is as follows: 5.2.1) When a video frame is detected to contain a potential hazard, a video segment is generated; (video frames containing potential hazard targets are spliced ​​together, splicing process is described in 5.2.2 and 5.2.3). 5.2.2) When the similarity of video frames is higher than a preset threshold, the video frames are discarded. 5.2.3) Within a fixed period, the current video frame is compared with the previously retained video frame in terms of clarity. The video frame with the higher clarity value is retained (and supplemented). When a video frame without any potential target appears, the video segment generation process ends. 5.2.4) Use an H.265 video encoder to compress the generated video clips to obtain compressed video clips. In this application, steps 5.2.1)-5.2.3) are intelligent slicing, which is an encoding strategy control. Here, "H.265 video encoder" refers to High Efficiency Video Coding (HEVC), also known as H.265.

[0017] Preferably, the specific content of obtaining the structured analysis results in step 6.2) is as follows: 6.2.1) Utilize the image understanding and visual question answering capabilities of a multimodal large model to perform high-level semantic analysis on video segments and construct potential hazard analysis prompts; 6.2.2) The video segmentation and detection results are used as inputs, and the outputs are structured analysis results including the judgment conclusions on safety hazards and confidence scores.

[0018] The specific content of the hidden danger handling in step 7.2) is as follows: 7.2.1) Rule matching: Based on the type of hazard in the safety hazard risk assessment conclusion (such as "fire extinguisher is blocked", "fire cabinet door cannot be opened" and "exposed wires"), query the corresponding rule entries in the rule base. If no matching rule entry is found, the default strategy (record only) will be used for processing. 7.2.2) Compare the confidence score in the analysis results of the successful rule matching in step 7.2.1) with the confidence threshold set in the rule. If the confidence score is greater than or equal to the confidence threshold, it is determined to be a valid hidden danger and the processing result is obtained. If the confidence score is less than the confidence threshold, it is processed according to the default strategy (record only).

[0019] In this application, the confidence threshold is a preset content, and the risk assessment conclusion of the safety hazard includes the hazard type.

[0020] Preferably, the specific content of obtaining the voice broadcast content in step 8.2) is as follows: The processing results are converted into voice broadcast content by using local TTS speech synthesis technology through the front-end speaker module.

[0021] In this application, "TTS speech synthesis technology" refers to a TTS speech synthesis system (Text-to-Speech), which is an artificial intelligence term, and "speech prompt structured hidden danger data" refers to the processing result.

[0022] Compared with the prior art, the beneficial effects of the present invention are: 1. Significantly reduce token consumption for large models: The front-end video target recognition module performs preliminary detection on video frames, retaining only frames containing potential targets for subsequent processing; combined with the image sharpness recognition module and the image similarity recognition module, blurry and duplicate frames are further removed to generate high-quality video clips, effectively reducing the amount of image data that the back-end multimodal large model needs to process, thereby significantly reducing token consumption.

[0023] 2. Improve the accuracy of hazard identification: Since the front-end slicing and compression module only retains clear, significantly changed video clips containing hazard targets, the back-end large model can focus on high-quality input, avoid redundant information interference, improve the semantic understanding and judgment accuracy of hazard targets, and generate more reliable structured analysis results.

[0024] 3. Reduced network transmission pressure: The front-end video segment compression module uses H.265 encoding to compress the selected video segments, significantly reducing data size and dependence on network bandwidth. This makes it suitable for real-time upload requirements in mobile network environments and ensures stable system operation in complex environments. 4. Real-time closed-loop feedback: The back-end hazard analysis result module, combined with the rule base, generates processing instructions and sends them to the front-end speaker module. Through TTS voice synthesis technology, real-time on-site reminders are achieved, forming a closed-loop mechanism of "identification-analysis-feedback" to improve on-site response speed and safety management efficiency.

[0025] 5. Supports data archiving and backtracking: Backend analysis results can be stored in the database for subsequent statistical analysis, accountability, and model optimization. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the information interaction logic between the modules in this invention; In the diagram: Front-end camera module 1, video target recognition module 2, image clarity recognition module 3, image similarity recognition module 4, front-end slicing and compression module 5, back-end image analysis module 6, and back-end hidden danger analysis result module 7. Detailed Implementation

[0027] The following will refer to the appendices in the embodiments of the present invention. Figure 1 The technical solutions in the embodiments of the present invention are clearly and completely described herein. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figure 1 Embodiments of the present invention: Example: like Figure 1 As shown: A front-end and back-end integrated hazard analysis system includes a law enforcement recorder, which is further equipped with: The front-end camera module 1 is used to capture images in front of it, generate video frame data based on the images, and then send the video frame data to the video target recognition module 2, the image clarity recognition module 3 and the image similarity recognition module 4. The video target recognition module 2 receives video frame data sent by the front-end camera module 1, detects the main target object for hidden danger analysis in the video frame data, and sends it to the front-end slicing and compression module 5. Image sharpness recognition module 3 receives video frame data sent by front-end camera module 1, obtains its sharpness gradient based on the video frame data, obtains the sharpness of the video frame based on its sharpness gradient, and sends the sharpness of the video frame to front-end slicing and compression module 5. Image similarity recognition module 4 receives video frame data sent by front-end camera module 1, calculates the similarity between the current video frame and the previous video frame, and then sends the similarity result to front-end slice compression module 5. The front-end slicing and compression module 5 receives the main target object for hidden danger analysis from the video frame data sent by the video target recognition module 2, the clarity of the video frame sent by the image clarity recognition module 3, and the similarity results sent by the image similarity recognition module 4. Based on the above information, it performs intelligent slicing and encoding strategy control to obtain a video segment with high definition and no hidden danger targets. Then, it compresses the video segment and sends the compressed video segment to the back-end image analysis module 6. The backend image analysis module 6 receives the compressed video segments sent by the frontend slicing compression module 5, processes the compressed video segments to obtain a structured analysis result containing a safety hazard risk judgment conclusion and a confidence score, and sends the structured analysis result to the backend hazard analysis result module 7. The backend hidden danger analysis result module 7 receives the structured analysis result sent by the backend image analysis module 6, and performs matching processing on the structured analysis result in combination with the preset safety hidden danger processing rule library to obtain the corresponding processing result. The processing result is then sent to the frontend speaker module 8, and the structured analysis result is stored at the same time. The front-end speaker module 8 receives the processing results sent by the back-end hidden danger analysis result module 7, converts the processing results into voice broadcast content, and broadcasts the voice broadcast content.

[0029] In this embodiment, the system described above is combined with a law enforcement recorder.

[0030] A front-end and back-end integrated hazard analysis method includes the following steps: 1. Image Acquisition 1.1) Acquire images of the area in front of it via the front-end camera module 1; 1.2) Generate video frame data based on the image and send the video frame data to the video target recognition module 2; III. Hazard Target Identification 2.1) The video target recognition module 2 receives video frame data sent from the front-end camera module 1; 2.2) The main target objects for hazard analysis in the video frame data are detected by the video target recognition module 2; The video target recognition module 2 uses the YOLO lightweight neural network detector to detect targets of interest in the video frames. These targets of interest are the main objects of subsequent hazard analysis.

[0031] 2.3) Send the main target objects for hidden danger analysis in the video frame data to the front-end slice compression module 5; III. Image Sharpness Recognition 3.1) The image sharpness recognition module 3 receives video frame data sent from the front-end camera module 1; 3.2) Obtain the sharpness gradient from the video frame data, and then obtain the sharpness of the video frame based on the sharpness gradient; 3.2.1) Use the Sobel operator to calculate the gradient in the horizontal and vertical directions for each point in the video frame; 3.2.2) The horizontal and vertical gradients are squared, summed, and then the square root is taken to form the sharpness gradient; 3.2.3) Calculate the average of all sharpness gradients of the video frame, and use the result as the sharpness of the video frame.

[0032] 3.3) Send the resolution of the video frames to the front-end slicing and compression module 5; IV. Image Similarity Recognition 4.1) The image similarity recognition module 4 receives video frame data sent from the front-end camera module 1; 4.2) Calculate the similarity between the current video frame and the previous video frame; The specific content of similarity calculation is as follows: The image similarity recognition module 4 uses the absolute difference method (SAD) to calculate the difference between the current video frame and the previous video frame pixel by pixel, accumulates the difference values ​​and then calculates the average, and then divides it by the quantization range of the pixels, 255, to obtain the difference coefficient of the video frame. Subtract the difference coefficient of the video frame from 1 to obtain the similarity value.

[0033] 4.3) Then send the similarity results to the front-end slice compression module 5; V. Video Segment Compression 5.1) The front-end slicing and compression module 5 receives the main target objects for hidden danger analysis from the video frame data sent by the video target recognition module 2, the clarity of the video frame sent by the image clarity recognition module 3, and the similarity results sent by the image similarity recognition module 4. 5.2) Based on the information received in step 5.1), perform intelligent slicing and encoding strategy control to obtain high-definition video segments without any hidden targets, and then compress the video segments. 5.2.1) When a video frame is detected to contain a potential hazard, a video segment is generated. 5.2.2) When the similarity of video frames is higher than a preset threshold, the video frames are discarded. 5.2.3) Within a fixed period, the current video frame is compared with the previously retained video frame in terms of clarity. The video frame with the higher clarity value is retained. When a video frame without any potential target appears, the video segment ends. 5.2.4) Use an H.265 video encoder to compress the video clips to obtain the compressed video clips.

[0034] 5.3) Send the compressed video clip to the backend image analysis module 6; VI. Image Analysis 6.1) The backend image analysis module 6 receives the compressed video clips sent from the frontend slicing and compression module 5; 6.2) The compressed video segments are processed to obtain structured analysis results containing security risk assessment conclusions and confidence scores; 6.2.1) Utilize the image understanding and visual question answering capabilities of a multimodal large model to perform high-level semantic analysis on video segments and construct potential hazard analysis prompts; 6.2.2) The video segmentation and detection results are used as inputs, and the outputs are structured analysis results including the judgment conclusions on safety hazards and confidence scores.

[0035] 6.3) Send the structured analysis results to the backend hidden danger analysis results module 7; VII. Hazard Analysis 7.1) Receive structured analysis results sent from the backend image analysis module 6 through the backend hidden danger analysis result module 7; 7.2) The structured analysis results are matched with the pre-set safety hazard handling rule base to obtain the processing results; 7.2.1) Rule matching: Based on the hazard type in the safety hazard risk assessment conclusion, query the corresponding rule entry in the rule base. If no matching rule entry exists, only the structured analysis result is recorded. 7.2.2) Compare the confidence score in the analysis results of the successful matching of the rules in 7.2.1) with the confidence threshold set in the rules. If the confidence score is greater than or equal to the confidence threshold, it is determined to be a valid hidden danger and the processing result is obtained. If the confidence score is less than the confidence threshold, only the structured analysis result is recorded.

[0036] 7.3) Send the processing results to the front-end speaker module 8, and store the structured analysis results at the same time; that is, notify the front-end to process and record them in the hidden event database of the central service.

[0037] VIII. Hazard Reporting and Reminder 8.1) Receive the processing results sent by the back-end hidden danger analysis result module 7 through the front-end speaker module 8; 8.2) Convert the processing results into voice broadcast content and broadcast the voice broadcast content.

[0038] That is, the processing results are converted into voice broadcast content by the local TTS speech synthesis technology through the front-end speaker module 8.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0040] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A front and back end integration vulnerability analysis system, characterized by, Including a law enforcement recorder, which is also equipped with: The front-end camera module (1) is used to capture images in front of it, generate video frame data based on the images, and then send the video frame data to the video target recognition module (2), the image clarity recognition module (3) and the image similarity recognition module (4). The video target recognition module (2) receives video frame data sent by the front-end camera module (1), detects the main target object for hidden danger analysis in the video frame data, and sends it to the front-end slice compression module (5). Image sharpness recognition module (3) receives video frame data sent by front-end camera module (1), obtains its sharpness gradient according to the video frame data, obtains the sharpness of the video frame according to its sharpness gradient, and sends the sharpness of the video frame to front-end slice compression module (5). Image similarity recognition module (4) receives video frame data sent by front-end camera module (1), performs feature extraction and similarity calculation on the current video frame and the previous video frame, and then sends the similarity result to front-end slice compression module (5). The front-end slicing compression module (5) receives the main target object of the hidden danger analysis in the video frame data sent by the video target recognition module (2), the clarity of the video frame sent by the image clarity recognition module (3), and the similarity result sent by the image similarity recognition module (4). Based on the above information, it performs intelligent slicing and encoding strategy control to obtain a video segment with high definition and no hidden danger target appearing in the video frame. Then, it compresses the video segment and sends the compressed video segment to the back-end image analysis module (6). The back-end image analysis module (6) receives the compressed video segment sent by the front-end slice compression module (5), processes the compressed video segment, obtains the structured analysis result containing the safety hazard risk judgment conclusion and confidence score, and sends the structured analysis result to the back-end hazard analysis result module (7). The back-end hidden danger analysis result module (7) receives the structured analysis result sent by the back-end image analysis module (6), and matches the structured analysis result with the preset safety hazard processing rule library to obtain the corresponding processing result. The processing result is sent to the front-end speaker module (8), and the corresponding structured analysis result is stored at the same time. The front-end speaker module (8) receives the processing results sent by the back-end hidden danger analysis result module (7), converts the processing results into voice broadcast content, and broadcasts the voice broadcast content.

2. A front-end and back-end combined hidden danger analysis method, characterized by comprising the following steps:

1. Image Acquisition 1.1) Acquire images of the front of the device through the front-end camera module (1); 1.2) and generate video frame data based on the image, and send the video frame data to the video target recognition module (2). II. Hazard Target Identification 2.1) The video frame data sent by the front-end camera module (1) is received through the video target recognition module (2); 2.2) The main target objects for hidden danger analysis in the video frame data are detected by the video target recognition module (2); 2.3) Send the main target objects for hidden danger analysis in the video frame data to the front-end slice compression module (5); III. Image Sharpness Recognition 3.1) The image clarity recognition module (3) receives video frame data sent by the front-end camera module (1); 3.2) Obtain the sharpness gradient from the video frame data, and then obtain the sharpness of the video frame based on the sharpness gradient; 3.3) Send the resolution of the video frames to the front-end slicing and compression module (5); IV. Image Similarity Recognition 4.1) The image similarity recognition module (4) receives video frame data sent by the front-end camera module (1); 4.2) Calculate the similarity between the current video frame and the previous video frame; 4.3) Then send the similarity results to the front-end slice compression module (5); V. Video Segment Compression 5.1) The main target object of the hidden danger analysis in the video frame data sent by the video target recognition module (2) is received by the front-end slicing compression module (5), the clarity of the video frame sent by the image clarity recognition module (3), and the similarity result sent by the image similarity recognition module (4); 5.2) Based on the information received in step 5.1), perform intelligent slicing and encoding strategy control to obtain high-definition video segments without any hidden targets, and then compress the video segments. 5.3) Send the compressed video clip to the backend image analysis module (6); VI. Image Analysis 6.1) The compressed video clips sent by the front-end slicing and compression module (5) are received through the back-end image analysis module (6); 6.2) The compressed video segments are processed to obtain structured analysis results containing security risk assessment conclusions and confidence scores; 6.3) Send the structured analysis results to the backend hidden danger analysis results module (7); VII. Hazard Analysis 7.1) Receive the structured analysis results sent by the backend image analysis module (6) through the backend hidden danger analysis result module (7); 7.2) The structured analysis results are matched with the pre-set safety hazard handling rule base to obtain the processing results; 7.3) The processing results are sent to the front-end speaker module (8), and the structured analysis results are stored at the same time; VIII. Hazard Reporting and Reminder 8.1) Receive the processing results sent by the back-end hidden danger analysis result module (7) through the front-end speaker module (8), 8.2) Convert the processing results into voice broadcast content and broadcast the voice broadcast content.

3. The method of claim 2, wherein, The specific content of the main target object for hazard analysis in the detected video frame data in step 2.2) is as follows: The video target recognition module (2) uses the YOLO lightweight neural network detector on it to detect the target of interest in the video frame, which is the main object of subsequent hazard analysis.

4. The method of claim 3, wherein, The specific details of obtaining the video frame sharpness in step 3.2) are as follows: 3.2.1) Use the Sobel operator to calculate the gradient in the horizontal and vertical directions for each point in the video frame; 3.2.2) The horizontal and vertical gradients are squared, summed, and then the square root is taken to form the sharpness gradient; 3.2.3) Calculate the average of all sharpness gradients of the video frame, and use the result as the sharpness of the video frame.

5. The method according to claim 2, 3 or 4, wherein, The specific content of the similarity calculation in step 4.2) is as follows: The image similarity recognition module (4) uses the absolute difference method to calculate the difference between the current video frame and the previous video frame pixel by pixel, accumulates the difference values ​​and then calculates the average, and then divides it by the quantization range of the pixels to obtain the difference coefficient of the video frame. The similarity value is obtained by subtracting the difference coefficient of the video frame from 1.

6. The front-end and back-end combined hidden danger analysis method according to claim 5, characterized in that, The specific content of obtaining the compressed video segment in step 5.2) is as follows: 5.2.1) When a video frame is detected to contain a potential hazard, a video segment is generated. 5.2.2) When the similarity of video frames is higher than a preset threshold, the video frames are discarded. 5.2.3) Within a fixed period, the current video frame is compared with the previously retained video frame in terms of clarity. The video frame with the higher clarity value is retained. When a video frame without any potential target appears, the video segment generation process ends. 5.2.4) Use an H.265 video encoder to compress the generated video clips to obtain compressed video clips.

7. The front-end and back-end combined hidden danger analysis method according to claim 6, characterized in that, The specific content of obtaining the structured analysis results in step 6.2) is as follows: 6.2.1) Utilize the image understanding and visual question answering capabilities of a multimodal large model to perform high-level semantic analysis on video segments and construct potential hazard analysis prompts; 6.2.2) The video segmentation and detection results are used as inputs, and the outputs are structured analysis results including the judgment conclusions on safety hazards and confidence scores.

8. The front-end and back-end combined hidden danger analysis method according to claim 2, characterized in that, The specific content of the hidden danger handling in step 7.2) is as follows: 7.2.1) Rule matching: Based on the hazard type in the safety hazard risk assessment conclusion, query the corresponding rule entry in the rule base. If no matching rule entry is found, the default strategy will be applied. 7.2.2) Compare the confidence score in the analysis results of the successful rule matching in step 7.2.1) with the confidence threshold set in the rule. If the confidence score is greater than or equal to the confidence threshold, it is determined to be a valid hidden danger and the processing result is obtained. If the confidence score is less than the confidence threshold, it is processed according to the default strategy.

9. A front-end and back-end combined hidden danger analysis method according to claim 2, 3, 4, 6, 7 or 8, characterized in that, The specific content of obtaining the voice broadcast content in step 8.2) is as follows: The processing results are converted into voice broadcast content by using local TTS speech synthesis technology through the front-end speaker module (8).

Images