Unsafe behavior detection method applied to complex assembly scene
By improving the YOLOv7 model to detect unsafe behaviors in complex assembly scenarios, the problem of inaccurate judgment by traditional methods is solved, and real-time and accurate safety behavior monitoring and alarms are achieved, which is suitable for lightweight embedded devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF REMOTE SENSING EQUIP
- Filing Date
- 2025-12-16
- Publication Date
- 2026-05-01
AI Technical Summary
Traditional unsafe behavior control methods are inaccurate and incomplete in complex assembly scenarios, and cannot monitor the safety status of the human-machine environment in real time, which may lead to missed or false judgments.
An improved YOLOv7 model is used, and by optimizing the model structure and loss function and training with a dataset, it is used to detect and alert on unsafe behaviors in complex assembly scenarios and is deployed in lightweight embedded devices.
It enables real-time and accurate detection in complex assembly scenarios, distinguishes between safe and unsafe behaviors, and provides alarm prompts to reduce safety accidents and save labor costs.
Smart Images
Figure CN121963074A_ABST
Abstract
Description
A method for detecting unsafe behaviors in complex assembly scenarios Technical Field
[0001] This invention belongs to the technical field of unsafe behavior detection, specifically relating to a method for detecting unsafe behavior in complex assembly scenarios. Background Technology
[0002] Traditional manufacturing workshops rely primarily on human experience for managing unsafe worker behavior, supplemented by safety reminders. They mainly use methods such as camera monitoring, regular safety officer inspections, on-site safety warning signs, safety doors, and light curtains. However, these methods are prone to omissions and misjudgments and cannot monitor the safety status of the human-machine environment in real time. Summary of the Invention
[0003] The purpose of this invention is to provide a method for detecting unsafe behaviors in complex assembly scenarios, so as to solve the technical problem that traditional unsafe behavior control judgments are inaccurate and incomplete.
[0004] To achieve the above objectives, the present invention adopts the following technical solution:
[0005] A method for detecting unsafe behaviors in complex assembly scenarios includes: identifying unsafe factors in hoisting and pyrotechnic scenarios, and collecting hoisting and pyrotechnic datasets in complex assembly environments; optimizing the YOLOv7 model structure and loss function to obtain an improved YOLOv7 model; training the improved YOLOv7 model using the dataset to obtain a detection model; and deploying the detection model to detect and alert on unsafe behaviors.
[0006] An unsafe behavior detection system for complex assembly scenarios includes: a data module for identifying unsafe factors in hoisting and pyrotechnic scenarios and collecting hoisting and pyrotechnic datasets in complex assembly environments; a model module for optimizing the YOLOv7 model structure and loss function to obtain an improved YOLOv7 model; a training module for training the improved YOLOv7 model using the dataset to obtain a detection model; and a detection module for deploying the detection model to detect and alert on unsafe behaviors.
[0007] A computer-readable storage medium storing a computer program configured to execute the above-described method for detecting unsafe behaviors in complex assembly scenarios at runtime.
[0008] An electronic device includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the above-described method for detecting unsafe behavior in complex assembly scenarios through the computer program.
[0009] In this invention, unsafe factors in hoisting and pyrotechnic scenarios are identified, and hoisting and pyrotechnic datasets are collected in complex assembly environments. The YOLOv7 model structure and loss function are optimized to obtain an improved YOLOv7 model. The improved YOLOv7 model is trained using the dataset to obtain a detection model. This detection model is then deployed to detect and alert on unsafe behaviors. Based on the improved YOLOv7 model, it can accurately detect and distinguish between safe and unsafe behaviors in real time in complex industrial assembly scenarios such as pyrotechnics and hoisting, and issue alerts for unsafe behaviors. This solves the technical problems of inaccurate and incomplete judgment in traditional unsafe behavior control. Furthermore, it is highly portable and easy to deploy in lightweight embedded environments, reducing safety accidents while saving manpower costs. Attached Figure Description
[0010] Figure 1 is a flowchart illustrating an unsafe behavior detection method applied to a complex assembly scenario according to an embodiment of the present invention.
[0011] Figure 2 is a schematic diagram of the workflow of an unsafe behavior detection method applied to a complex assembly scenario in an embodiment of the present invention;
[0012] Figure 3 is a schematic diagram of the control information flow of an unsafe behavior detection method applied to a complex assembly scenario in an embodiment of the present invention.
[0013] Figure 4 is a schematic diagram of the local convolutional network structure based on bottleneck in an embodiment of the present invention;
[0014] Figure 5 is a schematic diagram of an unsafe behavior detection system applied to a complex assembly scenario according to an embodiment of the present invention. Detailed Implementation
[0015] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description and claims. It should be noted that the drawings are all in a very simplified form and are not to a precise scale, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.
[0016] It should be noted that, in order to clearly illustrate the content of this invention, several embodiments are provided to further explain different implementations of the invention. These embodiments are enumerated rather than exhaustive. Furthermore, for the sake of brevity, content mentioned in the preceding embodiments is often omitted in the following embodiments. Therefore, content not mentioned in the later embodiments can be referred to in the preceding embodiments.
[0017] Example 1
[0018] A method for detecting unsafe behaviors in complex assembly scenarios, as shown in Figure 1, includes:
[0019] S102, identify unsafe factors in hoisting and pyrotechnic scenarios, and collect hoisting and pyrotechnic datasets in complex assembly environments;
[0020] S104, optimize the YOLOv7 model structure and loss function to obtain an improved YOLOv7 model;
[0021] S106, The improved YOLOv7 model is trained using the dataset to obtain the detection model;
[0022] S108, Deploy the detection model to detect and alert on unsafe behaviors.
[0023] As an optional implementation method, unsafe factors in hoisting and pyrotechnic scenarios are identified, including: unsafe factors in hoisting scenarios include: obstacles in the work area, distractions, improper clothing, workers under the hoisted object, overcrowding in the work area, unprotected hoisting hooks, and failure to wear safety helmets; unsafe factors in pyrotechnic scenarios include: cluttered workbench, pyrotechnic workpieces not placed in designated positions, improper clothing, failure to wear work wristbands, failure to perform static electricity removal before work, unauthorized use of mobile phones during work, and failure to wear safety helmets.
[0024] As an optional implementation method, a dataset of hoisting and pyrotechnic operations in complex assembly environments is collected, including: collecting on-site video recordings of pyrotechnic and hoisting processes in different workshops, and constructing an unsafe behavior dataset by extracting frames from the collected videos.
[0025] As an optional implementation, an unsafe behavior dataset is constructed, including: reducing and labeling the dataset, and using images without detected targets as negative samples.
[0026] The methods for constructing and annotating the dataset are not limited to: unifying the format of the filtered images into JPEG; using the LabelImg annotation tool to load the images, draw bounding boxes, and class labels, mapping each annotation file to an image, and exporting it as a YOLO annotation format XML annotation file with the following format:
[0027] <class_id><x_center><y_center> <width> <height>After labeling, the dataset is randomly divided into training, test, and validation sets in an 8:1:1 ratio using a dataset splitting script.
[0028] As an optional implementation, the YOLOv7 model structure is optimized by: replacing the ordinary convolutional structure of the YOLOv7 model with a bottleneck structure; adding a large-scale feature detection layer on the feature fusion layer; and introducing an attention mechanism for optimization.
[0029] The traditional YOLOv7 CBS module is modified by replacing the ordinary convolutional structure with a bottleneck structure. The bottleneck structure adds 1×1 convolutional layers before and after the original convolutional layers. First, the 1×1 convolutional layers are used to reduce the dimensionality, then the regular convolutional operation is performed, and finally, a 1×1 convolutional layer is used to increase the dimensionality of the image to restore its original number of channels.
[0030] As an optional implementation, the ordinary convolutional structure of the YOLOv7 model is replaced with a bottleneck structure, including: modifying the common file to add the bottleneck structure to the basic components; modifying the models / yolov7.py file to add the bottleneck structure in the parse_model function; and modifying the yolov7.yaml file to add the bottleneck structure.
[0031] As an optional implementation, a large-scale feature detection layer is added to the feature fusion layer, including: modifying the models folder to add a large-scale feature detection layer before the three-layer feature detection layer; updating the input and output channels to change the scale to 160×160, 80×80, 40×40, and 20×20; and adjusting the forward propagation logic by adding a feature fusion layer in the forward function.
[0032] As an optional implementation, an attention mechanism is introduced for optimization, including performing a Squeeze operation, an Excitation operation, and a recalibration operation in sequence.
[0033] As an optional implementation, optimizing the YOLOv7 model loss function includes adding an aspect ratio penalty term to the YOLOv7 model loss function.
[0034] It is not limited to converting trained models into ONNX format, deploying them on NVIDIA core processing boards (Jetson OrinNX) embedded information processing devices, and accelerating inference speed through TensorRT to detect and alert on unsafe behaviors in real time.
[0035] Specifically, the unsafe behavior detection method described above for complex assembly scenarios is based on an improved YOLOv7 model, and its format is converted to make it applicable to lightweight embedded devices, facilitating its application in industrial scenarios.
[0036] The system workflow and data flow of the unsafe behavior detection method applied to complex assembly scenarios are shown in Figures 2 and 3, respectively. The hardware of the entire system consists of imaging equipment, information processing equipment, audio prompting equipment, a host computer system, and a higher-level system. The imaging equipment outputs images into the information processing equipment, which integrates detection algorithms. The information processing equipment sends the detected images and detection results to the host computer for display and controls the audio prompts based on the detection results. The host computer can automatically report the detection results, such as time and screenshots, to the higher-level system and allows for manual correction of the results. The entire implementation process mainly includes the following steps:
[0037] Step 1: Identify the unsafe factors in hoisting and pyrotechnic scenarios.
[0038] By collecting relevant accident data and reviewing company safety management regulations, common unsafe factors in hoisting and pyrotechnic scenarios were summarized, and the annotation categories for the dataset were determined. The hoisting scenario included the following seven unsafe factors: obstacles in the work area, distraction, improper attire, workers under the hoisted object, overcrowding in the work area, unprotected hoisting hooks, and failure to wear safety helmets. The pyrotechnic scenario included the following seven unsafe factors: a cluttered workbench, workpieces not placed in designated locations, improper attire, failure to wear work wristbands, failure to perform pre-work static electricity removal, unauthorized use of mobile phones during work, and failure to wear safety helmets. A total of 14 unsafe behaviors were identified and used as the annotation content for the dataset.
[0039] Step 2: Collect hoisting and pyrotechnic data sets in actual complex assembly scenarios.
[0040] Step 2.1, Data Source.
[0041] A dataset of unsafe behaviors in complex scenarios was built, including two scenarios: pyrotechnics and hoisting. All data was collected offline, with on-site video recordings of pyrotechnics and hoisting processes in different workshops. Frames were extracted from the video data every second to construct a preliminary dataset of unsafe behaviors. Data with poor image quality was then filtered out manually, removing completely blurry or occluded images to prevent the model from learning too many useless features as detection criteria. Moderately blurry data was not filtered to reflect realistic usage scenarios and image quality, thus increasing the model's generalization ability. Using this method, a dataset of 8700 images was obtained, of which approximately 3% of the images contained no detected target and were included as negative samples.
[0042] Step 2.2, Data Processing.
[0043] The filtered images were standardized to JPEG format; the LabelImg annotation tool was used to load the images, draw bounding boxes, and add category labels. Each annotation file was mapped one-to-one with the image, and exported as an XML annotation file in YOLO annotation format.<class_id><x_center><y_center> <width> <height>After labeling, the dataset is randomly divided into training, test, and validation sets in an 8:1:1 ratio using a dataset splitting script.
[0044] Step 3: Using the YOLOv7 model, replace half of its ordinary convolutional structure with a bottleneck structure.
[0045] The traditional YOLOv7 CBS module is modified by replacing half of the ordinary convolutional structures with bottleneck structures. The bottleneck structure is shown in Figure 4(a), and the fused network structure is shown in Figure 4(b). This module adds 1×1 convolutional layers before and after the original feature extraction convolutional layers. The first 1×1 convolutional layer reduces the input dimension, and the second 1×1 convolutional layer restores the output dimension, performing a dimensionality increase operation. This structure effectively reduces the number of computational parameters. Assuming both input and output are required to be 512 dimensions, the ordinary convolutional structure directly uses 3×3 convolutional kernels for convolution, resulting in 512×3×3×512=2359296 kernel parameters. Using the bottleneck structure, two 1×1 convolutional layers are used to reduce the image dimension to 128 dimensions for convolution, resulting in a kernel parameter count of [missing value].
[0046] 512×1×1×128+128×3×3×128+128×1×1×512=278528, the number of parameters is significantly reduced.
[0047] The process for replacing the traditional CBS module is as follows:
[0048] Step 3.1: Modify the common file to add the bottleneck module to the basic components;
[0049] Step 3.2: Modify the models / yolov7.py file and add a bottleneck structure to the parse_model function;
[0050] Step 3.3: Modify the yolov7.yaml file and add the bottleneck structure.
[0051] Step 4: Add a large-scale feature detection layer on top of the existing feature fusion layer in YOLOv7.
[0052] To improve the detection and recognition capability of small targets, a larger-scale feature detection layer was added, giving the network a smaller receptive field and making it more sensitive to small and occluded targets. The original network model consisted of three feature detection layers with scales of 80×80, 40×40, and 20×20. A larger-scale feature detection layer with a scale of 160×160 was added before the first detection layer. The specific steps are as follows:
[0053] Step 4.1: Modify the "models" folder to add a new convolutional layer, i.e., a large-scale feature detection layer, before the three-layer feature detection layer;
[0054] Step 4.2: Update the input and output channels so that their scale changes from the original [80×80, 40×40, 20×20] to [160×160, 80×80, 40×40, 20×20];
[0055] Step 4.3: Adjust the forward propagation logic and add a feature fusion layer to the forward function.
[0056] Step 5: Introduce an attention mechanism (SENet) to optimize the YOLOv7 model.
[0057] In the original YOLOv7 architecture, an SE attention mechanism is introduced for optimization, focusing the network's attention on the target to be identified, thereby improving the network's recognition ability. Specifically, this includes the following three operations:
[0058] Squeeze operation: For feature map X∈R b×c×h×w Where b is the batch size, c is the number of channels, and h and w are the height and width, respectively, the channel description is obtained through global average pooling:
[0059]
[0060] The result z is a c-dimensional vector representing the global information for each channel.
[0061] Excitation operation: Calculates the weights for each channel using two fully connected layers:
[0062] s=Rxcitation(z)=σ(fc2(ReLU(fc1(z) (2)
[0063] Here, fc1 and fc2 are two fully connected layers, and σ is the Sigmoid function.
[0064] Recalibration: Finally, the original feature map is recalibrated to obtain the weighted feature map.
[0065] Step 6: Optimize the loss function of the YOLOv7 model by adding an aspect ratio penalty term to obtain an improved YOLOv7 model as the detection model.
[0066] An aspect ratio penalty term is added to the loss function to constrain the detection results by minimizing the difference in aspect ratio between the actual and predicted target boxes. The loss function formula is as follows:
[0067]
[0068] The fractional term is the smallest closed shape C formed by the true bounding box and the predicted bounding box, and is the proportion of the area of each of the two bounding boxes to the total area of C.
[0069] Step 7: Deploy the trained model to the embedded information processing device (NX board) and use TensorRT to speed up inference and perform real-time detection and alarm prompts for unsafe behaviors.
[0070] To enable industrial applications, the model is converted to ONNX format on the NVIDIA Jetson Orin NX core processing board, the information processing device where the algorithm is to be deployed. The inference speed is accelerated by the hardware's built-in TensorRT library. Finally, based on the detection results, an alarm is issued for unsafe behaviors.
[0071] This technology detects unsafe behaviors in complex assembly scenarios, especially hoisting and pyrotechnic scenarios. It distinguishes between safe and unsafe behaviors in real time and provides voice alarms for unsafe behaviors, thereby achieving the goal of industrial safety production. Its advantages lie in the fact that the detection can ensure effective real-time supervision, and the model's operating environment is lightweight enough to be easily deployed in industrial scenarios to ensure safe production and achieve the goal of intelligent supervision.
[0072] Example 2
[0073] An unsafe behavior detection system for complex assembly scenarios, as shown in Figure 5, includes:
[0074] Data module 502 is used to identify unsafe factors in hoisting and pyrotechnic scenarios and to collect hoisting and pyrotechnic datasets in complex assembly environments.
[0075] Model module 504 is used to optimize the YOLOv7 model structure and loss function to obtain an improved YOLOv7 model;
[0076] Training module 506 is used to train the improved YOLOv7 model using the dataset to obtain the detection model;
[0077] The detection module 508 is used to deploy the detection model to detect and alert on unsafe behaviors.
[0078] Optionally, the data module 502 identifies unsafe factors in hoisting and pyrotechnic scenarios, including: unsafe factors in hoisting scenarios include: obstacles in the work area, distractions, improper clothing, workers under the hoisted object, overcrowding in the work area, unprotected hoisting hooks, and failure to wear safety helmets; unsafe factors in pyrotechnic scenarios include: a cluttered workbench, pyrotechnic workpieces not placed in designated positions, improper clothing, failure to wear work wristbands, failure to perform static electricity removal before work, unauthorized use of mobile phones during work, and failure to wear safety helmets.
[0079] Optionally, the aforementioned data module 502 collects hoisting and pyrotechnic datasets in complex assembly environments, including: collecting on-site video recordings of pyrotechnic and hoisting processes in different workshops, and constructing an unsafe behavior dataset by extracting frames from the collected videos.
[0080] Optionally, the data module 502 constructs an unsafe behavior dataset, including: simplifying and labeling the dataset, and using images without detected targets as negative samples.
[0081] Optionally, the above model module 504 optimizes the YOLOv7 model structure, including: replacing the ordinary convolutional structure of the YOLOv7 model with a bottleneck structure; adding a large-scale feature detection layer on the feature fusion layer; and introducing an attention mechanism for optimization.
[0082] Optionally, the above model module 504 replaces the ordinary convolutional structure of the YOLOv7 model with a bottleneck structure, including: modifying the common file to add the bottleneck structure to the basic components; modifying the models / yolov7.py file to add the bottleneck structure in the parse_model function; and modifying the yolov7.yaml file to add the bottleneck structure.
[0083] Optionally, the above model module 504 adds a large-scale feature detection layer on the feature fusion layer, including: modifying the models folder to add a large-scale feature detection layer before the three-layer feature detection layer; updating the input and output channels to change the scale to 160×160, 80×80, 40×40, and 20×20; and adjusting the forward propagation logic by adding a feature fusion layer in the forward function.
[0084] Optionally, the above model module 504 introduces an attention mechanism for optimization, including performing a Squeeze operation, an Excitation operation, and a recalibration operation in sequence.
[0085] Optionally, the above model module 504 optimizes the YOLOv7 model loss function by adding an aspect ratio penalty term to the YOLOv7 model loss function.
[0086] In this embodiment, unsafe factors in hoisting and pyrotechnic scenarios are identified, and hoisting and pyrotechnic datasets are collected in complex assembly environments. The YOLOv7 model structure and loss function are optimized to obtain an improved YOLOv7 model. The improved YOLOv7 model is trained using the dataset to obtain a detection model. The detection model is deployed to detect and alert on unsafe behaviors. Based on the improved YOLOv7 model, it is possible to accurately detect and distinguish between safe and unsafe behaviors in real time in complex industrial assembly scenarios such as pyrotechnics and hoisting, and to issue alerts for unsafe behaviors. This solves the technical problems of inaccurate and incomplete judgment in traditional unsafe behavior control. At the same time, it has strong portability and is easy to deploy in lightweight embedded environments, reducing safety accidents while saving manpower costs.
[0087] Example 3
[0088] In another aspect, the present invention provides an electronic device for implementing the above-described method for detecting unsafe behaviors in complex assembly scenarios. This electronic device is not limited to a terminal device or server within a system. The electronic device includes, but is not limited to, a memory and a processor. The memory stores a computer program, and the processor is configured to execute the steps of any of the above method embodiments via the computer program.
[0089] Example 4
[0090] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional embodiments of the unsafe behavior detection method applied to complex assembly scenarios. The computer program is configured to execute the steps in any of the above-described method embodiments during runtime.< / height> < / width> < / height> < / width>
Claims
1. A method for detecting unsafe behaviors in complex assembly scenarios, characterized in that, include: Identify unsafe factors in hoisting and pyrotechnic scenarios, and collect hoisting and pyrotechnic datasets in complex assembly environments. The YOLOv7 model structure and loss function are optimized to obtain an improved YOLOv7 model; the improved YOLOv7 model is trained using a dataset to obtain a detection model; the detection model is deployed to detect and alert on unsafe behaviors.
2. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 1, characterized in that, Unsafe factors identified in hoisting and pyrotechnic scenarios include: Hoisting scenarios include: obstacles in the work area, distractions, improper attire, workers under the hoisted object, overcrowding in the work area, unprotected hoisting hooks, and failure to wear safety helmets; Pyrotechnic scenarios include: a cluttered workbench, pyrotechnic workpieces not placed in designated locations, improper attire, failure to wear work wristbands, failure to perform pre-work static electricity removal, unauthorized use of mobile phones during work, and failure to wear safety helmets.
3. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 1, characterized in that, Collect data sets on hoisting and pyrotechnic operations in complex assembly environments, including: collecting on-site video recordings of pyrotechnic and hoisting processes in different workshops, and constructing an unsafe behavior dataset by extracting frames from the collected videos.
4. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 3, characterized in that, Construct an unsafe behavior dataset, including: simplifying and labeling the dataset, and using images without detected targets as negative samples.
5. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 1, characterized in that, The YOLOv7 model structure was optimized, including replacing the ordinary convolutional structure with a bottleneck structure; adding a large-scale feature detection layer on top of the feature fusion layer; and introducing an attention mechanism for optimization.
6. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 5, characterized in that, Replace the ordinary convolutional structure of the YOLOv7 model with the bottleneck structure, including: modifying the common file to add the bottleneck structure to the basic components; modifying the models / yolov7.py file to add the bottleneck structure in the parse_model function; and modifying the yolov7.yaml file to add the bottleneck structure.
7. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 5, characterized in that, Add a large-scale feature detection layer to the feature fusion layer, including: modifying the models folder to add a large-scale feature detection layer before the three-layer feature detection layer; updating the input and output channels to change the scale to 160×160, 80×80, 40×40, and 20×20; and adjusting the forward propagation logic by adding a feature fusion layer to the forward function.
8. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 5, characterized in that, An attention mechanism is introduced for optimization, including performing Squeeze, Excitation, and recalibration operations sequentially.
9. The method for detecting unsafe behaviors in complex assembly scenarios as described in claim 1, characterized in that, Optimize the YOLOv7 model loss function, including adding an aspect ratio penalty term to the YOLOv7 model loss function.
10. An unsafe behavior detection system applied in complex assembly scenarios, characterized in that, include: The data module is used to identify unsafe factors in hoisting and pyrotechnic scenarios, and to collect hoisting and pyrotechnic datasets in complex assembly environments. The model module is used to optimize the YOLOv7 model structure and loss function to obtain an improved YOLOv7 model. The training module is used to train the improved YOLOv7 model using the dataset to obtain the detection model; the detection module is used to deploy the detection model to detect and alert on unsafe behaviors.