High-altitude operation safety peccancy detection method based on YOLO and multi-modal model

Abstract

The invention discloses a high-altitude operation safety peccancy detection method based on YOLO and a multi-modal model, and belongs to the field of computer vision and artificial intelligence, and the method comprises the steps: carrying out the extraction and preprocessing of structured space information based on YOLO; invalid target filtering based on MLLM semantic verification is carried out; personnel safety equipment detection based on MLLM space perception and semantic understanding; visual output and structured output are fused; continuously optimizing the YOLO model based on MLLM feedback; and result visualization and redrawing are carried out. According to the method, the problems of low detection precision, insufficient semantic understanding and lack of adaptive ability in the prior art are solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence, specifically relating to a method for detecting safety violations in high-altitude operations based on YOLO and multimodal models. Background Technology

[0002] Currently, construction site safety monitoring mainly relies on manual inspections and traditional video analysis technologies. While manual inspections are highly accurate, they are costly and cannot achieve continuous 24-hour monitoring. Traditional image processing-based algorithms (such as background subtraction, color detection, and contour extraction) are greatly affected by changes in lighting, occlusion, and interference from multiple targets, resulting in limited effectiveness in practical applications.

[0003] In recent years, deep learning-based object detection algorithms (such as YOLO, SSD, and Faster R-CNN) have been widely used in video analysis tasks. However, when faced with complex behavioral scenarios, these algorithms can only output object bounding boxes and lack the ability to understand the semantics of the behavior. For example, when detecting scenarios such as "not wearing a seatbelt" or "not wearing a helmet," the compliance of the behavior cannot be determined solely by the object detection results.

[0004] In addition, existing intelligent video analysis systems still have the following problems: (1) insufficient model adaptability, requiring separate training of models for different scenarios; (2) lack of structured output of detection results, making it impossible to generate standardized reports. Therefore, there is an urgent need for a new type of intelligent video analysis algorithm that can maintain detection accuracy while possessing semantic understanding, scene adaptation, and high precision capabilities. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a high-altitude operation safety violation detection method based on YOLO and a multimodal model, which solves the problems of low detection accuracy, insufficient semantic understanding, and lack of adaptive capabilities in existing technologies.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: a method for detecting safety violations in high-altitude operations based on YOLO and multimodal models, comprising the following steps: S1: Use the YOLO model to perform human detection on the input power high-altitude operation video frames, output the bounding box coordinates, unique personnel ID and detection confidence of each detected object, filter low confidence detection results according to the preset confidence threshold, and retain the high confidence personnel detection boxes and corresponding information; S2: Input the high-confidence person detection boxes and corresponding video frames into the multimodal big language model. Guide the multimodal big language model to make a semantic validity judgment on each person detection box through predefined prompt words, retain the relevant information of the valid person detection boxes, and store the invalid target data as YOLO model optimization samples. S3: Based on the relevant information in the valid personnel detection box, determine whether the personnel are in a high-altitude working state, whether they are wearing safety helmets properly, and whether they are wearing safety belts and safety ropes by using predefined prompts, and output the detection results in structured JSON format; S4: Establish a mapping relationship between the bounding box coordinates of the detected object, the personnel ID information and the detection results, and generate a list of violations containing personnel ID, violation type and location information based on whether the personnel are in a high-altitude work state, according to the corresponding violation judgment rules. S5: Collect stored invalid target data, build a dynamic difficult example sample library, and when the number of samples in the sample library reaches a preset threshold, start online adaptive fine-tuning of the YOLO model. After the fine-tuning is completed, realize hot update of the model. S6: Overlay the list of violations with the original video frames, dynamically adjust the bounding box color according to the violation status, add text labels containing personnel ID and violation details, and output a fully labeled image or video stream to achieve high-altitude operation safety violation detection based on YOLO and multimodal models.

[0007] Furthermore, the YOLO model is YOLOv11, the bounding box coordinates are represented in the format [x_min, y_min, x_max, y_max], and the unique personnel ID is automatically generated by the YOLO model and increments from 1.

[0008] Furthermore, the video frames are obtained by disassembling the camera video stream using OpenCV or FFmpeg.

[0009] Furthermore, the predefined prompts include instructions to focus only on areas in the image containing YOLO detection box information, as well as feature description instructions for safety helmets, safety belts, and safety ropes.

[0010] Furthermore, the specific rules for determining violations are as follows: For workers working at heights, check whether safety helmets, safety belts, and safety ropes are all worn; if any is missing, it is marked as a violation. For ground personnel, only helmets are checked, while seat belts and safety ropes are ignored.

[0011] Furthermore, the dynamic difficult sample library is managed using a first-in-first-out or time-weighted strategy; during fine-tuning, difficult samples are assigned a higher weight coefficient than normal samples; and the total loss of the YOLO model is calculated based on this weight coefficient during fine-tuning.

[0012] The beneficial effects of this invention are: (1) Significantly improved accuracy and robustness: Through the “dynamic attention guidance” and “two-way verification closed loop” mechanism, the system integrates the positioning accuracy of YOLO and the semantic depth of MLLM. When facing occlusion, small targets and complex backgrounds, it shows stability that surpasses that of a single model or a simple cascaded model, and the false detection rate and false negative rate are significantly reduced.

[0013] (2) True scene adaptation capability: Thanks to the "context-aware prompt word generation" technology, the system can automatically adjust the analysis strategy according to the visual context. It can achieve accurate violation judgment without retraining the model for different scenarios, and has extremely strong generalization ability.

[0014] (3) System Evolvability: Semantic feedback data (such as false detection samples) generated during bidirectional interaction can be automatically collected and used as high-quality training data for subsequent continuous fine-tuning of the YOLO model, enabling the entire system to have the ability to self-optimize and continuously evolve. Attached Figure Description

[0015] Figure 1 This is a flowchart of the high-altitude operation safety violation detection method based on YOLO and multimodal models of the present invention. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0017] like Figure 1 As shown, a method for detecting safety violations in high-altitude operations based on YOLO and multimodal models includes the following steps: S1: Use the YOLO model to perform human detection on the input power high-altitude operation video frames, output the bounding box coordinates, unique personnel ID and detection confidence of each detected object, filter low confidence detection results according to the preset confidence threshold, and retain the high confidence personnel detection boxes and corresponding information; In this embodiment, a YOLOv11 model fine-tuned using massive amounts of construction site scene data is used to perform efficient and accurate human detection on the input video frames. The bounding box coordinates, unique ID, and detection confidence score for each person are output, and preliminary filtering is performed based on the confidence score threshold. High-altitude images are processed by a retrained YOLO detection model to obtain images containing person detection boxes, with each person detection box containing a different ID automatically generated by YOLO starting from 1.

[0018] YOLO also saves the bounding box coordinates (usually represented as [x_min, y_min, x_max, y_max]) and a unique person ID for each detected person. Additionally, YOLO records the confidence score for each box to filter out low-quality detections. For example, if three people are detected in a frame, the YOLO output might be: ID: 1, coordinates: [100, 200, 150, 300], confidence score: 0.9; ID: 2, coordinates: [300, 150, 350, 250], confidence score: 0.8; ID: 3, coordinates: [500, 100, 550, 200], confidence score: 0.4 (below the threshold, filtered out).

[0019] This invention uses YOLO for human detection first, which restricts the input of MLLM to the bounding box of the person, effectively reducing background noise and allowing the model to focus on semantic analysis of the human body region.

[0020] The YOLO detection phase sets a confidence threshold (e.g., 0.5) to filter out low-confidence false detections (such as misidentifying objects as people). This ensures that only high-confidence personnel bounding boxes are sent to the MLLM for further analysis, preventing the MLLM from performing meaningless equipment detections on non-person targets and reducing error propagation. This two-stage process essentially constitutes a dual verification: YOLO handles spatial localization, and MLLM handles semantic verification, jointly improving system robustness.

[0021] S2: Input the high-confidence person detection boxes and corresponding video frames into the multimodal large language model. Guide the multimodal large language model (MLLM) to make a semantic validity judgment on each person detection box through predefined prompt words, retain the relevant information of the valid person detection boxes, and store the invalid target data as YOLO model optimization samples. In this embodiment, targets are filtered in MLLM using predefined prompts based on the people frame information detected by YOLO.

[0022] The system first performs a first round of front-end filtering based on visual features using YOLO's confidence threshold to exclude obviously unreliable detection boxes.

[0023] For candidate targets filtered by the front end, MLLM performs semantic validity checks. MLLM determines whether each person detection bounding box provided by YOLO contains a "valid person". For example, it identifies mannequins in the background, oddly shaped instruments, or blurry humanoid outlines as non-human targets.

[0024] In its output structured JSON data, MLLM explicitly identifies the validity conclusion for each ID, for example: "whether "Is a valid person": "no" Upon receiving this result, the subsequent violation analysis module will skip the analysis of all security equipment marked as invalid targets. These targets will not be included in the violation judgment logic, but will be retained as samples for continuous optimization of YOLO.

[0025] Traditional algorithms exhibit significant differences in detection capabilities across different scenarios, requiring retraining for each scenario, and the accuracy of the detection model decreases when crossing scenarios. This invention introduces MLLM visual understanding capabilities, resulting in strong generalization ability and consistent detection performance across different scenarios.

[0026] Existing methods can only determine the object category based on the detection box, making it difficult to understand the semantics of behavior. This invention utilizes a multimodal large language model combined with prompt engineering technology to guide the model to perform semantic reasoning based on visual input, achieving complex behavior recognition such as "whether the safety helmet is worn correctly" and "whether the safety belt is worn when working at height".

[0027] S3: Based on the relevant information in the valid personnel detection box, determine whether the personnel are in a high-altitude working state, whether they are wearing safety helmets properly, and whether they are wearing safety belts and safety ropes by using predefined prompts, and output the detection results in structured JSON format; The predefined prompts include instructions to focus only on areas in the image containing YOLO detection bounding boxes, as well as instructions for describing the features of safety helmets, safety belts, and safety ropes.

[0028] In this embodiment, for the valid personnel identified in S2, predefined prompt words are used to perform scene and equipment detection analysis of valid targets in MLLM.

[0029] First, the visual analysis is strictly limited to the area within the bounding box of the person by the cue words, "only focusing on the area in the image with YOLO detection box information".

[0030] Then, based on the visual information within the frame, it is determined whether the person is "working at height".

[0031] Then, based on the situation judgment result, different equipment inspection logics are executed to guide the model to output violation information. The detection targets include: whether high-altitude work is being carried out, whether a safety helmet is being worn, whether a safety belt is being worn, and whether a safety rope is being worn. MLLM outputs structured JSON data, where each object contains a "Person ID" field, which corresponds one-to-one with the ID assigned by YOLO. For example: [ {"Personnel ID": "1", "Is this a valid person": "yes", "Are you at high altitude": "yes", "Are you wearing a helmet": "yes", "Are you wearing a safety belt": "no", "Are you wearing a safety rope": "no"}, {"Personnel ID": "2","Is it a valid person": "no"} ] This invention optimizes the prompts. First, the prompts focus the large model's attention on the personnel bounding boxes detected by YOLO, avoiding excessive attention to the background and improving the accuracy of personnel and safety equipment detection. Second, a second judgment is made on the personnel bounding boxes detected by YOLO, preventing falsely detected targets from entering equipment analysis and avoiding equipment analysis of non-personnel targets. Finally, the equipment information to be detected is described in detail, allowing the large model to better learn equipment features and enhancing detection accuracy.

[0032] Existing systems output simple information, which is not convenient for subsequent security analysis. The detection results output by this invention are in structured JSON format, including violation type, location information, personnel number, and text description. It can directly generate a security report with graphics and text, realizing the integration of detection, traceability, and management.

[0033] S4: Establish a mapping relationship between the bounding box coordinates of the detected object, the personnel ID information and the detection results, and generate a list of violations containing personnel ID, violation type and location information based on whether the personnel are in a high-altitude work state, according to the corresponding violation judgment rules. In this embodiment, the violation analysis module receives the JSON output of MLLM and performs matching based on the original ID and coordinate information provided by YOLO.

[0034] The module maintains a mapping table that associates each ID with its coordinates and violation status.

[0035] The parsing logic applies different violation rules depending on whether the person is at a height (determined by MLLM): (1) For workers working at height ("yes" for "whether at height"), check whether the safety helmet, safety belt and safety rope are all worn. If any is missing, it is marked as a violation.

[0036] (2) For ground personnel ("whether they are at high altitude" is "no"), only check whether they are wearing a helmet, and ignore the safety belt and safety rope.

[0037] After matching is completed, the system generates a final list of violations, including the person's ID and the type of violation. At the same time, it saves images containing the IDs and coordinate frame information of invalid persons in the system.

[0038] S5: Collect stored invalid target data, build a dynamic difficult example sample library, and when the number of samples in the sample library reaches a preset threshold, start online adaptive fine-tuning of the YOLO model. After the fine-tuning is completed, realize hot update of the model. From the output of step S4, the system captures in real time all invalid targets determined by MLLM as "no" for "Is this a valid person?". For each invalid target, the system automatically creates a hard negative sample data packet. This data packet contains: Original image frame: The complete scene image when a false detection occurs.

[0039] Error detection box coordinates: The initial, incorrect bounding box generated by YOLO [x_min, y_min, x_max, y_max].

[0040] Real Label: Since MLLM has determined that it is not a human, the data packet is automatically labeled "background" or "non-human".

[0041] The system maintains a dynamic hard example sample library to store the hard negative samples generated in the above steps. This sample library is managed using a first-in, first-out (FIFO) or time-weighted strategy to ensure that it always contains the most recent and representative false positive samples, thus allowing the YOLO model to adapt to the latest changes in the scenario.

[0042] The fine-tuning process is not continuous, but is controlled by triggering conditions to balance performance improvement and computational overhead. It is automatically triggered when the number of samples in the difficult example sample library accumulates to a predetermined threshold (2000).

[0043] To prevent the model from forgetting existing knowledge and to enhance the correction of new false detection patterns, we introduced a dynamic weighting mechanism based on sample type during fine-tuning. When constructing the incremental training set, the system assigns a weight coefficient to each training sample: negative samples drawn from the hard example sample library are given a higher weight W_hard (e.g., W_hard = 3.0). Normal samples (positive samples and ordinary negative samples) drawn from the original training data are given a standard weight W_normal = 1.0.

[0044] During fine-tuning, the standard YOLO loss function (which includes bounding box loss, object loss, and classification loss) remains unchanged, but when calculating the total loss, the loss for each sample is multiplied by its corresponding weight coefficient. Therefore, the contribution of hard negative samples to the total loss is amplified, thereby guiding the model's learning focus to "identify and correct such errors".

[0045] After fine-tuning, the system automatically generates a new version of the YOLO model weight file.

[0046] Then, the system performs a hot update of the model: without interrupting the monitoring service, the old version of the YOLO model running online is smoothly replaced with the newly trained weight file.

[0047] S6: Overlay the list of violations with the original video frames, dynamically adjust the bounding box color according to the violation status, add text labels containing personnel ID and violation details, and output a fully labeled image or video stream to achieve high-altitude operation safety violation detection based on YOLO and multimodal models.

[0048] Boundary box drawing: Based on the coordinates provided by YOLO, a boundary box is drawn for each person on the original image. The color of the box is dynamically adjusted according to the violation status (e.g., red indicates violation, green indicates compliance).

[0049] Add labels: Add text labels near each bounding box to display the person's ID and violation details (e.g., "ID:1, not wearing a seatbelt").

[0050] The final output is a fully labeled image or video stream, which is easy for security managers to view and trace intuitively.

[0051] This invention's algorithm can be applied to various high-altitude work scenarios, such as construction sites and power construction, enabling intelligent identification, semantic understanding, and real-time alarms for worker violations. By integrating an MLLM inference module into the video surveillance system, this algorithm possesses high scene adaptability, capable of identifying and detecting high-altitude operations in different scenarios. This invention supports centralized cloud deployment and boasts advantages such as high precision, high real-time performance, and strong scalability.

[0052] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.

Claims

1. A high-altitude operation safety violation detection method based on YOLO and a multi-modal model, characterized in that, Comprise the following steps: S1: using YOLO model to the input power aerial work video frame carries out human detection, exports each detection object's boundary frame coordinate, unique personnel ID and detection confidence, according to the pre-set confidence threshold filters low confidence detection result, retains high confidence personnel detection frame and corresponding information; S2: high confidence personnel detection frame and corresponding video frame input multimodal large language model, through the pre-defined prompt word guide multimodal large language model to each personnel detection frame carries out semantic level effectiveness judgment, retains effective personnel detection frame related information, stores invalid target data as YOLO model optimization sample simultaneously; S3: according to effective personnel detection frame related information, through the pre-defined prompt word judges whether personnel is in aerial work state, whether standard safety helmet is worn, whether safety belt and safety rope are worn, exports the detection result of structured JSON format; S4: based on the boundary frame coordinate of detection object, personnel ID information and detection result establishes mapping relationship, according to whether personnel is in aerial work state uses corresponding violation judgment rule, generates the violation list containing personnel ID, violation type, location information; S5: collect invalid target data, construct dynamic difficult example sample library, when the sample quantity in sample library reaches the pre-set threshold, starts YOLO model online adaptive fine-tuning, realizes model hot update after fine-tuning is completed; S6: the violation list and original video frame are superimposed, and the boundary frame color is dynamically adjusted according to the violation state, and the text label containing personnel ID and violation details is added, and the annotated complete image or video stream is output, realizing the aerial work safety violation detection based on YOLO and multimodal model.

2. The YOLO and multi-modal model-based high-altitude operation safety violation detection method according to claim 1, characterized in that, The YOLO model is YOLOv11, the boundary frame coordinate is represented in [x_min, y_min, x_max, y_max] format, and the unique personnel ID is automatically generated by the YOLO model and starts from 1.

3. The YOLO and multi-modal model-based aerial work safety violation detection method according to claim 1, characterized in that, The video frame is obtained by opencv or ffmpeg to get the camera video stream and then disassembled.

4. The YOLO and multi-modal model-based high-altitude operation safety violation detection method according to claim 1, characterized in that, The pre-defined prompt word includes the instruction of only paying attention to the area with YOLO detection frame information in the picture and the feature description instruction of safety helmet, safety belt and safety rope.

5. The YOLO and multi-modal model-based safety violation detection method for aerial work according to claim 1, characterized in that, The violation judgment rule is as follows: For aerial work personnel, check whether safety helmet, safety belt and safety rope are all worn, and mark as violation if any is missing; For ground personnel, only check whether safety helmet is worn, and ignore safety belt and safety rope.

6. The YOLO and multi-modal model-based high-altitude operation safety violation detection method according to claim 1, characterized in that, The dynamic difficult example sample library uses advanced first-in first-out or time weight strategy management;In the fine-tuning process, the difficult example sample is given a higher weight coefficient than the normal sample;The total loss of YOLO model is calculated based on the weight coefficient during fine-tuning.

Images