Finger key point detection method for factory operation safety system

Abstract

The invention relates to a finger key point detection method for a factory operation safety system. The method comprises the following steps: acquiring input data; performing target detection and key point detection on the input data through a target detection network; the target detection network comprises a network backbone structure, a neck structure and a detection structure; the detection structure comprises a target detection head and a key point detection head; the key point detection head comprises a first convolution layer, a convolution and self-attention fusion module ACmix and a second convolution layer which are connected in sequence. The method has the advantages that the target detection network is optimized through the convolution and self-attention fusion module ACmix, the advantages of the convolutional neural network in the aspect of capturing image local features are reserved, and the perception ability of the model to global image semantics is remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a finger key point detection method for a factory operation safety system. Background Art

[0002] In the field of computer vision, YOLO (You Only Look Once) is a fast and accurate target detection algorithm that has been developed into multiple versions, each of which is improved and optimized on the basis of the previous version. YOLO performs well in both accuracy and real-time performance of target detection. At the same time, due to the relative simplicity of its model structure and the small number of parameters, it is easy to deploy on a variety of different hardware platforms. These advantages make YOLO widely used in various downstream tasks of computer vision, mainly including target classification, target detection, image segmentation, and key point detection. However, in some specific industrial scenarios, such as intelligent systems designed to ensure property and personnel safety, finger key point detection is an important module. In order to ensure high accuracy, the system requires further improvement of the robustness of the YOLO key point detection model.

[0003] Although the original YOLO key point detection model has certain performance, it still has some limitations and room for improvement. The following aspects are the main reasons for its limited performance: 1. Model structure limitations: The YOLO model was originally designed mainly for object detection, and its key point detection capability is relatively weak, especially when there are complex spatial relationships between key points. YOLO's key point detection is achieved by predicting the relative positions of key points within the detected object box, which may not be sufficient to capture the fine spatial layout and dynamic changes between fingers.

[0004] 2. Limitations of the dataset: Finger key point detection requires high-quality, diverse annotated data to train the model. If the training data is insufficient or inaccurately annotated, it will be difficult for the model to learn effective feature representations, resulting in poor detection performance.

[0005] 3. Target pose changes: In practical applications, the position and pose of fingers may change significantly. The YOLO model may not perform well in handling such complex geometric changes, so it is necessary to improve the robustness of the model through a stronger feature extraction network or attention mechanism. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and to propose a finger key point detection method for a factory operation safety system.

[0007] In a first aspect, a finger key point detection method for a factory operation safety system is provided, comprising: Step 1: Get input data; Step 2: Performing target detection and key point detection on the input data through a target detection network; The target detection network includes a network backbone structure, a neck structure and a detection structure; the detection structure includes a target detection head and a key point detection head; the key point detection head includes a first convolutional layer, a convolution and self-attention fusion module ACmix and a second convolutional layer connected in sequence.

[0008] Preferably, in step 2, the operation process of the convolution and self-attention fusion module ACmix includes: Get the input feature map, use the convolution kernel to perform linear mapping on the input feature map, and generate a weight matrix; According to the weight matrix, a convolution operation is used to extract local features of the input feature map to obtain a first feature map; A self-attention mechanism is used to capture the long-distance dependency between different positions in the feature map, and a second feature map is obtained according to the long-distance dependency; The first feature map and the second feature map are weightedly summed by using learnable hyperparameters α and β to obtain an output feature map; the dimension of the output feature map is consistent with the dimension of the input feature map.

[0009] Preferably, in step 2, the target detection network performs feature enhancement processing in the training phase, and the feature enhancement processing includes: The RT-DETR model is trained using the palm target detection dataset to obtain the deviations dx and dy between the target box predicted by RT-DETR and the real target box in the x-axis and y-axis directions of the image; Gaussian mixture model GMM is used to model dx and dy, and sampling is performed to obtain supplementary eigenvalues; The supplementary feature value is added to the original training set to obtain an extended training set, and the target detection network is trained according to the extended training set. Preferably, step 1 further includes: performing data enhancement processing on the input data, and the data enhancement processing includes: mixing the actual application scene image with the finger key point data set.

[0010] In a second aspect, a finger key point detection device for a factory operation safety system is provided, which is used to perform any finger key point detection method described in the first aspect, including: An acquisition module, used to obtain input data; A detection module, used for performing target detection and key point detection on the input data through a target detection network; The target detection network includes a network backbone structure, a neck structure and a detection structure; the detection structure includes a target detection head and a key point detection head; the key point detection head includes a first convolutional layer, a convolution and self-attention fusion module ACmix and a second convolutional layer connected in sequence.

[0011] In a third aspect, a vision-based factory error prevention system is provided, comprising: an image input unit, a detection unit, a calculation unit, and a judgment unit; Wherein, the image input unit is used to obtain an operation image; The detection unit is used to execute any finger key point detection method described in the first aspect to detect the positions of the target and the finger key points in the operation image; The calculation unit calculates the distance between the target and the finger key point according to their positions in the operation image, and determines the target position closest to the finger key point and its state; The judging unit judges whether the executed operation is accurate according to the target position closest to the finger key point and its state, and issues an alarm if it is inaccurate.

[0012] According to a fourth aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0013] In a fifth aspect, an electronic device is provided, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement any method as described in the first aspect.

[0014] The beneficial effects of the present invention are: 1. The present invention optimizes the target detection network through the convolution and self-attention fusion module ACmix, which not only retains the advantages of the convolutional neural network in capturing local features of the image, but also significantly enhances the model's perception of the global image semantics. In the present invention, the convolution and self-attention fusion module ACmix can effectively characterize the precise position information of the key points of the fingers and deeply understand the relationship between them.

[0015] 2. The present invention performs feature enhancement processing on the target detection network during the training phase, uses the RT-DETR target detector for targeted training of the palm, and obtains the deviation between the predicted frame of RT-DETR and the real frame. In the key point detection model training phase, according to the deviation distribution in the x-axis and y-axis directions, GMM is used to generate additional offsets of the real target frame through dense sampling, thereby improving the target detection performance and reducing missed detection or false detection, thereby avoiding affecting the subsequent key point detection.

[0016] 3. The present invention creates a finger key point dataset in an industrial scenario, combines it with the open source MHP dataset for data enhancement, and improves the generalization ability of the target detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the overall framework of a key point detection model provided by an embodiment of the present invention; Figure 2 A schematic diagram of specific implementation details of the ACmix module provided in an embodiment of the present invention; Figure 3 A flow chart of a factory error prevention system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0019] Embodiment 1: To solve the problems of the prior art, such as Figure 1 As shown, Embodiment 1 of the present application provides a finger key point detection method for a factory operation safety system, comprising: Step 1: Get input data.

[0020] Step 2: Performing target detection and key point detection on the input data through a target detection network; The target detection network includes a network backbone structure, a neck structure and a detection structure; the detection structure includes a target detection head and a key point detection head; the key point detection head includes a first convolutional layer, a convolution and self-attention fusion module ACmix and a second convolutional layer connected in sequence.

[0021] Specifically, the target detection network of this application uses the open source YOLOv9 backbone network as the network backbone structure, and the neck structure adopts the PAN (Path Aggregation Network) feature fusion module, which is an effective multi-scale feature fusion technology. It can transfer information between features at different levels, thereby enhancing the model's detection ability for targets of different sizes. It is widely used as a general neck module of the YOLO series. The key point detection head uses the open source YOLOv9 framework detection head, which contains a convolutional layer for regressing the target box position.

[0022] Since the key point detection head structure in the original YOLO model is relatively simple and contains only three convolutional layers, in order to improve the accuracy of key point detection, this application uses the ACmix module to replace the middle convolutional layer, such as Figure 1 As shown in the figure. The ACmix module cleverly combines the convolution mechanism with the multi-head self-attention mechanism. This innovative design enables the model to efficiently encode the spatial position of each finger joint and their relationship. In this way, the model can accurately capture the subtle changes of fingers when performing complex movements, and further realize the in-depth capture and fusion of local details and global visual representations in the feature map. This fusion mechanism greatly enhances the accuracy and robustness of the model's detection of finger key points.

[0023] Specifically, Figure 2 As shown in the figure, the operation process of the convolution and self-attention fusion module ACmix is ​​designed to be divided into multiple stages to give full play to the advantages of convolution and self-attention mechanisms. The following is a detailed description of each stage: In stage A, the input feature map is obtained by first using three The convolution kernel performs linear mapping on the input feature map to generate weight matrices for query, key, and value. These weight matrices provide the necessary input for the subsequent self-attention mechanism, allowing the model to focus on the key areas in the feature map.

[0024] In the subsequent stages B and C, the module further processes the feature map by applying convolution and self-attention mechanisms in parallel. Stage B focuses on using convolution operations to extract local features and enhance the model's ability to capture detailed information. Stage C uses the self-attention mechanism to capture long-distance dependencies between different positions in the feature map, allowing the model to better understand the spatial position relationship between finger joints. The combination of these two stages enables the model to capture global contextual information while maintaining local details.

[0025] Finally, by introducing learnable hyperparameters α and β, the feature maps of stages B and C are weighted and summed, which effectively integrates the information of both and obtains the output feature map, whose dimension is consistent with that of the input feature map.

[0026] Embodiment 2: Based on Example 1, Example 2 of the present application provides a more specific finger key point detection method for a factory operation safety system, including: Step 1: Get input data.

[0027] In step 1, the input data is subjected to data enhancement processing, and the data enhancement processing includes: not only using pictures of actual application scenarios for model training, but also mixing with a certain proportion of open source finger key point data for subsequent training and verification. For example, the data collected in the industrial scene is mixed with a part of the screened open source Multiview Handpose dataset to enhance the detection ability of the model from the data level. The screening method is to remove pictures of similar scenes or similar gestures. The dataset for model training and verification contains a total of 2091 pictures, involving 2115 hand instances.

[0028] Step 2: Performing target detection and key point detection on the input data through a target detection network; The target detection network includes a network backbone structure, a neck structure and a detection structure; the detection structure includes a target detection head and a key point detection head; the key point detection head includes a first convolutional layer, a convolution and self-attention fusion module ACmix and a second convolutional layer connected in sequence.

[0029] The target detection network of the embodiment of the present application uses the open source YOLOv9 backbone network as the network backbone structure. It should be noted that the YOLO network structure can flexibly select an appropriate scaling scale according to specific hardware resource conditions, especially the video memory of the graphics card. The scale of the network can be changed by scaling. The larger the network parameters, the more video memory is occupied during training, but the better the effect.

[0030] In addition, the YOLOv9 network architecture is not fixed, it has high flexibility and can be easily replaced by other versions of the YOLO series, including but not limited to YOLOv5, YOLOv7 and YOLOv8. It can also be replaced with other backbone networks according to some specific requirements. It has been verified that the application of the present invention on the three backbone networks of EfficientNet, Resnet and HRnet can improve the model performance.

[0031] It should be noted that the key point detection method based on YOLO follows a top-down strategy, that is, first detect the target and determine the target frame, and then accurately regress the finger key points according to the position of the target frame. Therefore, the accuracy of target detection is crucial, which is directly related to the final effect of finger key point detection. In order to enhance the detection performance, this application performs feature enhancement processing on the target detection network during the training stage.

[0032] Feature enhancement processing includes: first, using the open source palm target detection dataset to conduct targeted training on the Real-time Detection Transformer (RT-DETR) model, which is difficult to train but has excellent performance. Through this training process, the deviations dx and dy between the target box predicted by RT-DETR and the real target box in the x-axis and y-axis directions of the image are obtained, and the distribution of these deviations roughly conforms to the characteristics of Gaussian distribution. In order to further optimize these deviation data, this design uses a Gaussian mixture model (GMM) to model dx and dy, and implements sampling operations. This process generates a series of supplementary feature values, which are incorporated into the real data during the training process as an extension of the original training set. Such feature enhancement methods effectively improve the target detection capability of the model, thereby achieving more accurate detection of finger key points.

[0033] The experimental results of applying the innovative method in this design to YOLOv5, YOLOv8, and YOLOv9 are shown in the following table. The evaluation index in Table 1 is mAP (mean average precision), which is a common performance index in object detection and key point detection tasks, and is calculated by precision, recall, and intersection over union (IoU). and Measures the model's ability to detect hand bounding boxes. and Used to measure the model's ability to detect finger key points. It refers to the accuracy of the model detection results when the IoU threshold is 0.5. It refers to the average accuracy of the model's accurate detection results when the IoU threshold is 0.5 to 0.95. After using the GMM-based feature enhancement method and ACmix module in this design, the performance of the YOLO model has been improved.

[0034] Table 1 Performance of this application on YOLOv5, YOLOv8, and YOLOv9

[0035] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0036] Embodiment 3: Based on Examples 1 and 2, Example 3 of the present application provides a vision-based factory error prevention system. In the actual production environment of the factory, the correct execution of the workflow is the key link to ensure the safe and stable operation of the factory, which is directly related to personnel safety and production efficiency. With the rapid development of artificial intelligence (AI) and computer vision technology, vision-based error prevention systems provide new solutions to improve operational accuracy and safety. This system effectively prevents the occurrence of human error accidents by analyzing the behavior of operators in real time.

[0037] Specifically, the vision-based factory error prevention system includes: an image input unit, a detection unit, a calculation unit and a judgment unit; Wherein, the image input unit is used to obtain the operation image. For example, Figure 3 As shown, the image input unit may be a wearable device (such as AR glasses), which may acquire an operation image in real time, for example, acquiring any video frame from an operation video as an operation image.

[0038] The detection unit is used to perform the finger key point detection method to detect the position of the target and the finger key point in the operation image. This function is crucial to determine the correctness of the personnel operation. In addition, the system also integrates a variety of algorithms, including detection algorithms for multiple target states such as factory switches and personnel, OCR algorithms for text or digital content recognition on electrical cabinets, and QR code recognition algorithms.

[0039] The calculation unit calculates the distance between the target and the finger key point according to the positions of the two in the operation image, and determines the target position closest to the finger key point and its state.

[0040] The judgment unit judges whether the executed operation is correct according to the position of the target closest to the finger key point and its state. Once it is found that the target pointed by the finger does not match the target specified in the workflow, or the state of the target does not meet the operation requirements, the system will immediately issue a warning to effectively prevent the occurrence of misoperation.

[0041] Embodiment 4: On the basis of embodiments 1 and 2, embodiment 4 of the present application provides a finger key point detection device for a factory operation safety system, comprising: An acquisition module, used to obtain input data; A detection module, used for performing target detection and key point detection on the input data through a target detection network; The target detection network includes a network backbone structure, a neck structure and a detection structure; the detection structure includes a target detection head and a key point detection head; the key point detection head includes a first convolutional layer, a convolution and self-attention fusion module ACmix and a second convolutional layer connected in sequence.

[0042] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A finger key point detection method for a factory operation safety system, characterized in that: include: Step 1: Get input data; Step 2: Performing target detection and key point detection on the input data through a target detection network; The target detection network includes a network backbone structure, a neck structure and a detection structure; the detection structure includes a target detection head and a key point detection head; the key point detection head includes a first convolutional layer, a convolution and self-attention fusion module ACmix and a second convolutional layer connected in sequence.

2. The finger key point detection method for factory operation safety system according to claim 1, characterized in that: In step 2, the operation process of the convolution and self-attention fusion module ACmix includes: Get the input feature map, use the convolution kernel to perform linear mapping on the input feature map, and generate a weight matrix; According to the weight matrix, a convolution operation is used to extract local features of the input feature map to obtain a first feature map; A self-attention mechanism is used to capture the long-distance dependency between different positions in the feature map, and a second feature map is obtained according to the long-distance dependency; The first feature map and the second feature map are weightedly summed by using learnable hyperparameters α and β to obtain an output feature map; the dimension of the output feature map is consistent with the dimension of the input feature map.

3. The finger key point detection method for factory operation safety system according to claim 2, characterized in that: In step 2, the target detection network performs feature enhancement processing in the training phase, and the feature enhancement processing includes: The RT-DETR model is trained using the palm target detection dataset to obtain the deviations dx and dy between the target box predicted by RT-DETR and the real target box in the x-axis and y-axis directions of the image; Gaussian mixture model GMM is used to model dx and dy, and sampling is performed to obtain supplementary eigenvalues; The supplementary feature value is added to the original training set to obtain an extended training set, and the target detection network is trained according to the extended training set.

4. The finger key point detection method for factory operation safety system according to claim 3, characterized in that: Step 1 also includes: performing data enhancement processing on the input data, and the data enhancement processing includes: mixing the actual application scene picture with the finger key point data set.

5. The finger key point detection method for factory operation safety system according to claim 4, characterized in that: In step 2, the neck structure is a feature fusion module PAN.

6. A finger key point detection device for a factory operation safety system, characterized in that: The method for detecting finger key points according to any one of claims 1 to 5 comprises: An acquisition module is used to obtain input data; A detection module, used for performing target detection and key point detection on the input data through a target detection network; The target detection network includes a network backbone structure, a neck structure and a detection structure; the detection structure includes a target detection head and a key point detection head; the key point detection head includes a first convolutional layer, a convolution and self-attention fusion module ACmix and a second convolutional layer connected in sequence.

7. A vision-based factory error prevention system, characterized in that: include: Image input unit, detection unit, calculation unit and judgment unit; Wherein, the image input unit is used to obtain an operation image; The detection unit is used to execute the finger key point detection method according to any one of claims 1 to 5, and detect the positions of the target and the finger key points in the operation image; The calculation unit calculates the distance between the target and the finger key point according to their positions in the operation image, and determines the target position closest to the finger key point and its state; The judging unit judges whether the executed operation is accurate according to the target position closest to the finger key point and its state, and issues an alarm if it is inaccurate.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes the finger key point detection method according to any one of claims 1 to 5.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the finger key point detection method as described in any one of claims 1 to 5.

Images