SOP behavior detection method based on YOLO target detection

Abstract

The invention provides an SOP behavior detection method, system and device based on YOLO target detection and a medium, and the method comprises the steps: S1, obtaining a monitoring video of a specified SOP point location, carrying out the frame extraction processing of the monitoring video to generate a picture, and carrying out the motion disassembly and behavior motion labeling of the picture according to a corresponding SOP behavior specification; s2, a Backbone network, a Head module and a loss function of the YOLOV5 network model are optimized; and S3, inputting the picture into the optimized YOLOV5 network model to carry out SOP behavior detection, and judging whether an operation behavior is correct or not. By adopting the YOLOV5 model, the problems of low efficiency and hysteresis of manual monitoring of SOP behavior actions are effectively solved, the labor cost is reduced, the operation behavior actions can be effectively standardized, and the product yield is improved; and meanwhile, the YOLOV5 model is improved, so that a tiny target can be accurately detected, and the universality is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior detection, and in particular to a SOP behavior detection method based on YOLO target detection. Background Art

[0002] In the context of pursuing automated, unmanned, and integrated equipment production, some production operation processes in some industries still cannot be completely handed over to machines. For example, the installation of mobile phone cables, car interiors and seats, and the installation of drive assemblies such as engines, gearboxes, and motors, this type of assembly operation still relies mainly on manual operation. The main reason is that this type of operation behavior is very different, requires high coordination, and requires more flexible actions. In addition, the space utilization rate of this type of operation behavior is generally low, that is, the specified operation process is completed in a smaller operation space; using machines to complete such operations requires a certain amount of action space, and the operation of the machine is more mechanical. It takes time to change the model in a short time to complete other operation processes. Therefore, for the assembly operation, most automation manufacturers still use manual completion. However, the uncertainty during manual operation is an important factor that affects product quality and restricts production efficiency. Therefore, how to efficiently monitor and manage manual operation behavior has become the main thinking direction of many automation manufacturers.

[0003] At present, most automation manufacturers generally employ employees to monitor manual operation behaviors through video surveillance and site inspections. Video surveillance refers to the deployment of surveillance cameras on the production line where employees operate. The cameras transmit the images to the monitoring center. The staff of the monitoring center manually discover abnormal operation processes in the images by watching the videos captured by the cameras and record and respond to them. Site inspections refer to the periodic in-depth visits of inspection personnel designated by the manufacturer to the operation site to check whether the operators are operating in violation of regulations.

[0004] However, since assembly lines often include multiple locations, video surveillance usually involves a one-to-many situation, that is, one inspection employee observes operating employees at multiple locations. Manual monitoring consumes a lot of human resources and is prone to fatigue and laziness due to boring work, thus missing out on abnormal behavior incidents. In addition, reports are often not timely, making it difficult to discover and correct problems in a timely manner. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a SOP behavior detection method, system, device and medium based on YOLO target detection to solve the technical problems in the prior art that manual monitoring consumes a lot of human resources, is prone to fatigue and slackness due to boring work, thereby missing abnormal behavior events, and reporting is often not timely, which is difficult to discover and correct in time.

[0006] The present invention provides a SOP behavior detection method based on YOLO target detection, comprising:

[0007] S1. Obtain the surveillance video of the designated SOP point, extract frames from the surveillance video to generate pictures, and perform action decomposition and action labeling on the pictures according to the corresponding SOP behavior regulations;

[0008] S2. Optimize the Backbone network, Head module and loss function of the YOLOV5 network model;

[0009] S3. Input the image into the optimized YOLOV5 network model to perform SOP behavior detection and determine whether the operation behavior is correct.

[0010] Optionally, obtaining the surveillance video of the designated SOP point at least includes:

[0011] Obtain surveillance videos of SOP points covering different operators, different time periods, and different operating environments.

[0012] Optionally, the step of performing action decomposition and action labeling on the image according to the corresponding SOP behavior regulations includes:

[0013] According to the SOP behavior regulations, the actions in the picture are disassembled. For actions that cannot be accurately distinguished in the picture, the result is used as the target. Otherwise, the action is used as the target, and the target is marked with a detection frame to form a marker.

[0014] Optionally, the Backbone network includes:

[0015] A CBAM module is introduced into the Backbone network, and the size of the feature map input into the CBAM module is limited to 20×20.

[0016] Optionally, the Head module includes:

[0017] There are four Head modules, corresponding to resolutions of 160×160, 80×80, 40×40 and 20×20 respectively.

[0018] Optionally, the loss function is optimized, including:

[0019]

[0020] Among them, the regression loss function L box Using CIOULoss, the confidence loss function L obj And the classification loss function L cls FocalLoss is used, and the definition of FocalLoss is:

[0021] FL(P t )=-α t (1-p t ) γ log(p t )

[0022] Among them, α t is the balance parameter and γ is the focusing parameter.

[0023] Optionally, the image is input into the optimized YOLOV5 network model to perform SOP behavior detection, and determine whether the operation behavior is correct, including:

[0024] S301, create a corresponding execution flag and action execution sequence vector table according to the operation steps specified in the SOP behavior;

[0025] S302, inputting the picture into the optimized YOLOV5 network model in chronological order to obtain the action target in the picture;

[0026] S303, copying the flag bit corresponding to the action target, and recording the sequence value of the action target into a sequence vector table;

[0027] S304, judging the flag bit to determine whether there is an unexecuted action in the flag bit, if yes, it is determined as a behavior detection error, otherwise, further judging the sequence value to determine whether the sequence value conforms to the action execution sequence vector table, if yes, it is determined as a behavior detection error, otherwise, it is determined that the behavior detection is correct.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] By adopting the YOLOV5 model, the problems of inefficiency and lag in manual monitoring of SOP behavior actions are effectively solved. While reducing labor costs, it can also effectively standardize operational behaviors and improve product yield. At the same time, the YOLOV5 model has been improved to enable accurate detection of small targets, enhancing its versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0033] Figure 2 This is a schematic diagram of the improved YOLOV5 model structure of the present invention;

[0034] Figure 3 This is a schematic diagram of the CBAM module structure in the present invention;

[0035] Figure 4 This is a schematic diagram for identification and comparison in the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other implementation cases obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The functional units with the same labels in the examples of the present invention have the same and similar structures and functions.

[0037] See also Figure 1 , the present invention provides a SOP behavior detection method based on YOLO target detection, comprising:

[0038] S1. Obtain the surveillance video of the designated SOP point, extract frames from the surveillance video to generate pictures, and perform action decomposition and action labeling on the pictures according to the corresponding SOP behavior regulations;

[0039] S2. Optimize the Backbone network, Head module and loss function of the YOLOV5 network model;

[0040] S3. Input the image into the optimized YOLOV5 network model to perform SOP behavior detection and determine whether the operation behavior is correct.

[0041] In this embodiment, S1, obtain the surveillance video of the specified SOP point, extract the frame of the surveillance video to generate an image, and perform action decomposition and action labeling on the image according to the corresponding SOP behavior regulations

[0042] First, obtain surveillance videos of designated SOP points. The surveillance video content covers different operators and different time periods, which can effectively reflect the operational differences between different people and the changes in the operating environment in different time periods.

[0043] Secondly, the surveillance video is extracted and converted into images, and then the actions are disassembled according to the SOP behavioral standards. The disassembly of actions follows two principles: first, for actions that are obviously different from other operations, the action is used as the target; second, for similar actions that cannot be accurately distinguished, the action result is used as the target; use the detection frame to manually mark the data to form a marker.

[0044] Then, in order to solve the problem of some operation actions with high similarity and easy to be misdetected, oversampling is used to deal with it. When training the target detection model, images with similar actions are copied, so that the number of times the targets with similar actions appear in the training set data increases, thereby improving the distinction between similar action targets, which is conducive to the model to better learn different action targets.

[0045] S2. Optimize the Backbone network, Head module and loss function of the YOLOV5 network model.

[0046] The YOLOV5s model is improved. The original YOLOV5 model consists of three parts: Backbone network for feature extraction, Neck for feature fusion and three prediction heads. After the image is input into the model, it passes through the three parts in sequence. The improvement of the present invention is to add a convolutional block attention module (CBAM) to the Backbone network; and expand the original three prediction heads to four; finally, the classification loss function BCELoss is adjusted to FocalLoss;

[0047] Among them, see Figure 2 , the improved YOLOV5, after the picture is input into the model, it passes through the three parts of Backbone, Neck and Head in sequence; the Backbone consists of CBS module + C3 module + CBAM module + SPPF module, and the CBS module is a simple feature extraction tool, which consists of Convolution (Convolution, Conv) + Batch Normalization (Batch Normalization, BN) + SiLU activation function.

[0048] The C3 module follows the residual design principle, and the input features Figure 1 The branch first passes through Conv1, then passes through Bottleneck to obtain sub-features Figure 1 , the other branch gets sub-features after Conv2 Figure 2 , and finally the sub-feature Figure 1 and 2 After splicing, the final output is input into Conv3; the SPPF module splices the four results after CBS and three consecutive maximum pooling (Maxpool), and then extracts features from CBS.

[0049] See also Figure 3 , CBAM module, in order to better obtain feature information, especially to distinguish the feature information of similar action targets, the present invention adds a lightweight attention CBAM module (Convolutional Block Attention Module) in the Backbone stage, aiming to emphasize the meaningful features on the channel and spatial axes through the channel attention module and the spatial attention module, while suppressing redundant features; the feature map size of the input CBAM module is 20×20, which is 32 times smaller than the full image 640×640, so it has no effect on the reasoning time of the model.

[0050] See also Figure 2 ,The Neck part follows the original model design and adopts a bidirectional feature pyramid structure (PAN-FPN, PAN-Feature pyramid network). ,The overall part contains multiple CBS and C3 modules, which convey strong semantic features (High-Level features) from the Backbone part, i.e. the FPN layer from top down, while the Neck part conveys strong ,positioning features (Low-Level features) from bottom up, and then aggregates features from different backbone layers to different ,detection layers.

[0051] For the Head module, when facing a tiny target, the features of such targets usually contain only a few pixels. As the network depth increases, the feature map of the small target is further reduced, making it extremely difficult for the model to extract features of the tiny target. The present invention immediately outputs an additional predicted Head, Small Object Detection Head (SODH), after the third upsample (Up sample) in the Neck stage, which aims to detect feature maps with higher resolution (160×160), which can better extract features from small and tiny targets. That is, the total resolution included (160×160, 80×80, 40×40 and 20×20)

[0052] Furthermore, the loss function is optimized.

[0053] The loss function is defined as:

[0054]

[0055] Among them, the regression loss function L box Using CIOULoss, the confidence loss function L objThe classification loss function originally used BCE Loss, but in the behavior action target detection task of this scheme, the requirements for the accuracy and confidence value of the coordinate frame are not as high as the category accuracy, that is, there must be clear classification requirements for different actions, so higher requirements are put forward for the classification loss function. Therefore, the focal loss function FocalLoss is used. The definition of FocalLoss is:

[0056] FL(P t )=-α t (1-p t ) γ log(p t )

[0057] Among them, α t is a balance parameter, and γ is a focus parameter. When the value of γ is larger, the loss value of samples that are easy to distinguish will be smaller, and the model's attention can be focused on samples that are difficult to distinguish. When the value of γ becomes larger, the range of samples that obtain small loss values ​​will be expanded. Therefore, Focal Loss can focus the model's learning on difficult-to-classify samples by reducing the weight of easy-to-classify samples.

[0058] S3. Input the image into the optimized YOLOV5 network model to perform SOP behavior detection and determine whether the operation behavior is correct.

[0059] S301, create a corresponding execution flag and action execution sequence vector table according to the operation steps specified in the SOP behavior;

[0060] S302, inputting the picture into the optimized YOLOV5 network model in chronological order to obtain the action target in the picture;

[0061] S303, copying the flag bit corresponding to the action target, and recording the sequence value of the action target into a sequence vector table;

[0062] S304. The flag is judged to determine whether there is an unexecuted action. If so, it is determined as a behavior detection error and a prompt is given. Otherwise, the sequence value is further judged to determine whether the sequence value conforms to the action execution sequence vector table. If so, it is determined as a behavior detection error and a prompt is given. Otherwise, the behavior detection is determined to be correct. After the workpiece leaves the operation area, the number of workpieces and the number of NG workpieces are recorded, and all steps are reset.

[0063] Through the above improvements, this scheme finally obtains a target detection model with small model parameters, high accuracy and fast inference time. Combined with Table 1, the specific data comparison is as follows; From the data in the table, it can be found that adding the fourth prediction Head has the greatest contribution to the model improvement. Compared with the Baseline, mAP@.5 and mAP@[.5:.95] are increased by 5.10% and 5.44% respectively; When the CBAM module is added alone, the effect of the model becomes worse, which is related to the fact that the target detection data collected by the frame extraction contains more small targets. If these small targets are not extracted by features at the beginning of the model, the CBAM module will not play a good role; Finally, after adding both modules to the Baseline model, SODH can effectively transfer the model's feature attention to the target, especially the focus on small targets, so that the CBAM module can play a greater role, and finally mAP@.5 and mAP@[.5:.95] are increased by 10.2% and 18.5%.

[0064]

[0065] See also Figure 4 Compared with the Baseline model, the improved model has better recognition of smaller hand motion targets; from the picture in b, we can see that the improved model can effectively detect the target of the hand picking up the workpiece and the target of the hand pressing the release button. In the picture in a, the Baseline model did not detect the hand motion target.

[0066] The present invention effectively solves the problems of inefficiency and lag in manual monitoring of SOP behavior actions by adopting the YOLOV5 model, which reduces labor costs and can effectively standardize operating behaviors and improve product yield. At the same time, the YOLOV5 model is improved to accurately detect small targets and enhance its versatility.

[0067] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0068] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A SOP behavior detection method based on YOLO target detection, characterized in that: include: S1. Obtain the surveillance video of the designated SOP point, extract frames from the surveillance video to generate pictures, and perform action decomposition and action labeling on the pictures according to the corresponding SOP behavior regulations; S2. Optimize the Backbone network, Head module and loss function of the YOLOV5 network model; S3. Input the image into the optimized YOLOV5 network model to perform SOP behavior detection and determine whether the operation behavior is correct.

2. The SOP behavior detection method based on YOLO target detection as claimed in claim 1, characterized in that, The acquisition of the surveillance video of the designated SOP point at least includes: Obtain surveillance videos of SOP points covering different operators, different time periods, and different operating environments.

3. The SOP behavior detection method based on YOLO target detection as claimed in claim 1, characterized in that, The action decomposition and action labeling of the image according to the corresponding SOP behavior regulations include: According to the SOP behavior regulations, the actions in the picture are disassembled. For actions that cannot be accurately distinguished in the picture, the result is used as the target. Otherwise, the action is used as the target, and the target is marked with a detection frame to form a marker.

4. The SOP behavior detection method based on YOLO target detection as claimed in claim 3, characterized in that, The Backbone network includes: A CBAM module is introduced into the Backbone network, and the size of the feature map input into the CBAM module is limited to 20×20.

5. the SOP behavior detection method based on YOLO target detection as claimed in claim 1, is characterized in that, The Head module includes: There are four Head modules, corresponding to resolutions of 160×160, 80×80, 40×40 and 20×20 respectively.

6. the SOP behavior detection method based on YOLO target detection as claimed in claim 1, is characterized in that, The loss function is optimized, including: Among them, the regression loss function L box Using CIOULoss, the confidence loss function L obj And the classification loss function L cls FocalLoss is used, and the definition of FocalLoss is: FL(P t )=-a t (1-p t ) γ log(p t ) Among them, α t is the balance parameter and γ is the focusing parameter.

7. The SOP behavior detection method based on YOLO target detection as claimed in claim 3, characterized in that, The image is input into the optimized YOLOV5 network model to perform SOP behavior detection and determine whether the operation behavior is correct, including: S301, create a corresponding execution flag and action execution sequence vector table according to the operation steps specified in the SOP behavior; S302, inputting the picture into the optimized YOLOV5 network model in chronological order to obtain the action target in the picture; S303, copying the flag bit corresponding to the action target, and recording the sequence value of the action target into a sequence vector table; S304, judging the flag bit to determine whether there is an unexecuted action in the flag bit, if yes, it is determined as a behavior detection error, otherwise, further judging the sequence value to determine whether the sequence value conforms to the action execution sequence vector table, if yes, it is determined as a behavior detection error, otherwise, it is determined that the behavior detection is correct.

Images