A mechanical arm target object detection method based on an improved YOLOv5 algorithm

Abstract

A mechanical arm target object detection method based on an improved YOLOv5 algorithm belongs to the technical field of image recognition, S1, a data set is acquired and data preprocessing is performed; S2, an existing YOLOv5 network is improved; S21, a space-coordinate attention mechanism SCAA is built, accurate positioning of target objects and local feature information is realized, and the space attention mechanism SA and the coordinate attention mechanism CA are combined inspired by the convolution attention mechanism CBAM; S22, a multi-scale feature fusion network, the Neck structure of YOLOv5 adopts a combination of FPN+PANeT, and the YOLOv5 algorithm model network is improved; S23, anchor frame parameters are optimized, YOLOv5 adopts self-adaptive anchor frames to automatically learn training data, and the target frame gt of the data set is automatically calculated by using the Kmeans clustering algorithm; S3, a network model is trained. The present application can solve the problem of inaccurate positioning of target objects and local features, and the positioning of target objects is more accurate and the robustness is better without reducing the recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, specifically relating to a method for detecting target objects grasped by a robotic arm based on an improved YOLOv5 algorithm. Background Technology

[0002] The YOLOv5 object detection algorithm boasts fast detection speed and high accuracy, demonstrating good performance on open-source datasets. However, its performance needs improvement in accurately identifying and locating custom dataset samples and local features. For the overall input image, the detected target object occupies a small portion of the image, with the rest appearing as background. After multiple convolutions, this background information accumulates iteratively, leading to information redundancy and interfering with target object recognition and localization. Furthermore, in the pose estimation algorithm, this project requires using the YOLOv5 network to locate labeled local features and then using the positional information of each feature for subsequent pose estimation. Therefore, accurate localization is crucial for this experiment. To obtain accurate localization information, this invention first combines spatial attention (SA) and coordinate attention (CA) mechanisms to propose a spatial-coordinate attention mechanism (SCAA). Then, an SCAA module is added to the YOLOv5 backbone network, and a BiFPN module (bidirectional feature pyramid network) is introduced into the Neck detection layer (object detection layer). The anchor box parameters are optimized according to the needs of the custom dataset, and a YOLOv5-ABN object detection algorithm based on YOLOv5 is proposed (a new algorithm that integrates spatial attention and coordinate attention mechanisms and adds a BiFPN module). The improved YOLOv5-ABN algorithm is trained on the custom dataset and the Cornell dataset. Ablation experiments are performed on the improved module, and its performance is compared and analyzed with existing object detection algorithms. Summary of the Invention

[0003] To address the inaccurate localization of target objects and local features in YOLOv5, this invention proposes a method for detecting target objects in robotic arm grasping based on an improved YOLOv5 algorithm, comprising the following steps:

[0004] S1. Obtaining the dataset and performing data preprocessing:

[0005] Download the Cornell dataset from the internet, use the common annotation software labelimg to annotate the obtained data images, and after annotation, convert the PASCAL format file to YOLO format file to complete the dataset creation;

[0006] S2. Improve the existing YOLOv5 network;

[0007] S21. Construct a spatial-coordinate attention mechanism SCAA to achieve accurate localization of target objects and local feature information. Inspired by the convolutional attention mechanism CBAM, combine the spatial attention mechanism SA and the coordinate attention mechanism CA.

[0008] S22. Multi-scale feature fusion network: YOLOv5's Neck structure adopts a combination of FPN and PANET. Based on the superiority of BiFPN network feature fusion method and drawing on the characteristics of BiFPN structure, the YOLOv5 algorithm model network is improved.

[0009] S23. Optimize anchor box parameters. YOLOv5 uses adaptive anchor boxes to automatically learn from the training data and automatically calculates the target box gt of the dataset by using the Kmeans clustering algorithm.

[0010] S3. Train the network model.

[0011] The beneficial effects of this invention are as follows: This invention proposes a target object detection method for robotic arm grasping based on an improved YOLOv5 algorithm. This method combines spatial attention (SA) and coordinate attention (CA) mechanisms to propose a spatial-coordinate attention mechanism (SCAA). Then, an SCAA module is added to the YOLOv5 backbone network, a BiFPN module is introduced into the Neck detection layer, and anchor box parameters are optimized according to the needs of a custom dataset, resulting in a YOLOv5-ABN target detection algorithm. This invention uses the improved YOLOv5-ABN algorithm to train on a custom dataset and the Cornell dataset, conducts ablation experiments on the improved module, and compares its performance with existing target detection algorithms, demonstrating the effectiveness of the proposed method and further improving the accuracy of object recognition. Attached Figure Description

[0012] Figure 1 This is a flowchart of the SCAA network structure of the present invention;

[0013] Figure 2 This is a diagram of the improved feature fusion network structure of the present invention;

[0014] Figure 3 This is a partial image of a single object from the Cornell dataset of this invention;

[0015] Figure 4 These are partial images of the multi-target grasping objects created in this invention.

[0016] Figure 5 This is a graph of the loss function of the model in this invention;

[0017] Figure 6 This is the PR curve before the improvement of this invention;

[0018] Figure 7 This is the improved PR curve diagram of the present invention;

[0019] Figure 8 An image showing the detection results of the target object in this invention;

[0020] Figure 9 This is a statistical chart of the confusion matrix of the present invention. Detailed Implementation

[0021] A method for detecting target objects in robotic arm grasping based on an improved YOLOv5 algorithm includes the following steps:

[0022] S1. Dataset Acquisition and Data Preprocessing

[0023] Download the Cornell dataset from the internet and use the common annotation software labelimg to annotate the obtained data images. After annotation, convert the PASCAL format files to YOLO format files to complete the dataset creation.

[0024] S2. Improve the existing YOLOv5 network.

[0025] (1) The construction of the spatial-coordinate attention mechanism (SCAA) is inspired by the convolutional attention mechanism (CBAM) to achieve accurate localization of target objects and local feature information. The spatial attention mechanism (SA) and the coordinate attention mechanism (CA) are combined.

[0026] (2) Multi-scale feature fusion network: YOLOv5’s Neck structure adopts a combination of FPN and PANET. Based on the advantages of BiFPN network feature fusion, this project will draw on the characteristics of BiFPN structure to improve the YOLOv5 algorithm model network.

[0027] FPN (Feature Pyramid Network), PANET (Path Aggregation Network)

[0028] (3) Anchor box parameter optimization: YOLOv5 uses AutoLearningBoundingBoxAnchors to automatically learn from the training data and automatically calculates the ground truth of the dataset using the Kmeans clustering algorithm in order to obtain suitable anchor boxes. Therefore, it is necessary to optimize the anchor box parameters.

[0029] S3, Training the network model

[0030] The processor used in this experiment is:

[0031] The system used an Intel(R) Core(TM) i5-10200H CPU at 2.40GHz, 16GB of RAM, an NVIDIA GeForce RTX 2080 Ti GPU, and Windows 10 64-bit operating system. The experimental software used was PyCharm, and the entire experiment was based on the deep learning framework PyTorch 1.9.0. The experimental environment was Python 3.8, and the GPU acceleration software was CUDA 11.1 and cuDNN 8.1.1. The depth camera used in the experiment was an Intel D435i stereo camera.

[0032] S21. Design of a Spatial-Coordinate Attention Mechanism

[0033] For the overall input image, the detected target object occupies a very small proportion of the overall image, with the rest of the image information appearing as background. After multiple convolutions, the background information accumulates iteratively, resulting in information redundancy and interfering with target object recognition and localization. Furthermore, in the pose estimation algorithm, this project requires using the YOLOv5 algorithm network to locate the labeled local feature information and utilize the positional information of each feature for subsequent pose estimation. Therefore, accurate localization is crucial for this experiment.

[0034] To achieve accurate localization of target objects and local feature information, this project, inspired by the Convolutional Attention Mechanism (CBAM), combines the Spatial Attention Mechanism (SA) and the Coordinate Attention Mechanism (CA) to propose a Spatial-Coordinate Attention Mechanism (SCAA), with the structure as follows: Figure 1 As shown, firstly, after the feature information passes through the spatial attention mechanism, the maximum and average values ​​of the two channels are extracted and weighted and fused to output feature information with spatial location information. Then, the features with spatial location information are aggregated and weighted along the horizontal and vertical directions respectively through the coordinate attention mechanism to form two independent feature maps. Finally, the features with orientation encoding information are stored. This dual attention to feature information in both the spatial and coordinate dimensions will make the localization more accurate.

[0035] like Figure 1 As shown in the diagram, the SCAA network structure mainly combines spatial-coordinate attention mechanisms, which can more accurately identify the detected objects.

[0036] The specific working steps of the SCAA module are as follows: Assuming the input feature information is N, the input features are compressed at the channel dimension level. Then, max pooling and average pooling operations are performed on the compressed features, that is, the maximum value and average value are extracted in the two channels respectively. Finally, the feature maps of the two channels are fused and passed through a 7x7 convolutional layer to obtain the weight coefficients Ns. The output feature Nf is the product of the input features N and Ns, as shown in formulas (1) and (2):

[0037]

[0038] N f =NXN s (2)

[0039] The entire spatial range has a height of H and a width of w. Within the vertical range of (H, 1), the coordinate information in the vertical direction is encoded and stored using a pooling kernel. Within the horizontal range of (1, w), the coordinate information in the horizontal direction is encoded and stored using a pooling kernel, as shown in formula (3).

[0040]

[0041] Features with precise encoded information are concatenated, and then subjected to a 1x1 convolution transform function F1 and a non-linear activation function. The intermediate feature f of the encoded spatial information is obtained, as shown in formula (4):

[0042]

[0043] Then, attention weights mh and mw are generated using 1x1 convolutions Fh and Fw, as shown in Equation (5):

[0044]

[0045] Finally, the output of the attention module is calculated using formula (6):

[0046]

[0047] In formula (1), f represents a 7x7 convolution, and N s Represents the weighting coefficient; Represents average pooling; Represents max pooling; Represents a non-linear activation function;

[0048] N in formula (2) f This represents the output characteristic; N s Represents the weight coefficient; N represents the input feature;

[0049] In formula (3), c represents the number of channels; i represents the i-th row and j represents the j-th column; h represents the height of the feature map; and w represents the width of the feature map. Used to calculate the value vector corresponding to each head h in the attention mechanism, it belongs to the weight matrix; It is used to calculate the value vector corresponding to each head w in the attention mechanism, and is also part of the weight matrix;

[0050] In formula (4), f represents the characteristic; F1 represents the linear transformation; D h The number of channels in the feature map; D w R represents the dimension of each element in the input feature vector; CIτX(H+W) The whole represents a matrix (H represents the height of the feature map; W represents the width of the feature map); Represents a non-linear activation function;

[0051] In formula (5) F h F represents the height feature vector at a specific location (h1, h2) in the input feature map; w This represents the feature representation on a specific channel w in the input feature map; m h The attention weighting coefficient h represents the input feature map m; m w The attention weighting coefficients w represent the input feature map m; f h f represents the feature representation at a specific location (h, w); w F represents the feature representation at a specific location (h, w); h and F w These represent linear operations performed on the input feature map; Represents a non-linear activation function;

[0052] To highlight the target's feature information and more accurately identify and locate the calibrated feature information, this paper adds a SCAA module after the C3 module of the backbone network. After obtaining the target's deep feature information through the C3 module, the SCAA module first obtains the spatial dimension weights, and then encodes the feature information with the large spatial weights in the horizontal and vertical directions, thus better identifying and locating the target model features. By adding the SCAA module to the backbone network of the YOLOv5 algorithm, the backbone network is changed from 10 layers to 13 layers. The new network structure is shown in Table 1, where the improved modules are indicated in bold.

[0053] Table 1 Improved Backbone Network Structure (Note: By adding the SCAA module to the backbone network of the YOLOv5 algorithm, the Backbone network was changed from 10 layers to 13 layers)

[0054] Module Number Filters Arguments Output Focus 1 32 [3,32,3] 320x320 Conv 1 64 [32,64,3,2] 160x160 C3 3 64 [64,64,1] 160x160 SCAA 3 64 [64,64] 160x160 Conv 1 128 [64,128,3,2] 80x80 C3 9 128 [128,128,3] 80x80 SCAA 3 128 [128,128] 80x80 Conv 1 256 [128,256,3,2] 40x40 C3 9 256 [256,256,3] 40x40 SCAA 3 256 [256,256] 40x40 Conv 1 512 [256,512,3,2] 20x20 SPP 1 512 [512,512,[5,9,13]] 20x20 C3 3 512 [512,512,1,False] 20x20

[0055] Step 2) Design of multi-scale feature fusion network

[0056] YOLOv5's Neck structure uses a combination of FPN and PANET. Based on the advantages of BiFPN's feature fusion method, this project will draw on the characteristics of the BiFPN structure to improve the YOLOv5 algorithm model network. The improved model structure is as follows: Figure 2 As shown, the low-level feature information is directly passed to the high-level features through skip connections. The improvement aims to fuse high-level semantic feature information with low-level feature information. The more low-level feature information is fused, the more accurate the model's target localization will be, the smaller the bounding box regression loss will be, and the better the overall detection performance will be.

[0057] like Figure 2 As shown, the improved feature fusion network integrates high-level semantic features with low-level features. The more low-level features are integrated, the more accurate the model's target localization becomes, the smaller the bounding box regression loss becomes, and the better the overall detection performance becomes.

[0058] The working principle of the novel feature fusion network will be explained below. The feature fusion method is shown in formula (7):

[0059] feature = [f1; f2; f3] (7)

[0060] Where f1, f2, and f3 represent three features of the bidirectional feature fusion network, [;] indicates pairwise concatenation of two features according to the channel dimension, and feature represents the new feature generated after concatenation of features according to the channel dimension. Pi represents the multi-scale fusion feature extracted by the feedforward deep neural network.

[0061] In the original fusion network, N3, N4, and N5 represent multi-scale feature detection of small, medium, and large targets, respectively. Feature N3 does not utilize the low-level feature N2, but instead utilizes the upsampled feature F3. Considering the need to detect local small features of the sample model, in order to improve the accuracy of local feature detection, this paper considers making full use of the low-level feature P2. Feature N2 is obtained by fusing and concatenating features P2 and F3. The CSP operator in the YOLOv5 structure is denoted as function F, as shown in formula (8):

[0062] N2 = F[P2; F3] (8)

[0063] N3, N4, and N5 are multi-scale fusion features generated through a novel bidirectional feature fusion network. Taking feature N3 generation as an example, feature P2 is downsampled to generate feature P3, and features P4 and P5 are concatenated and fused along the channel to obtain feature F4. Feature F4 is upsampled and concatenated with feature P3 along the channel dimension to generate feature F3. Multi-scale fusion feature N3 is generated by concatenating features N2, feature F3, and feature P3 along the channel dimension. Downsample refers to downsampling by a factor of two through convolution. The fusion method of feature N3 is shown in formula (9).

[0064] N3=F[Downsample(N2);F3;P3] (9)

[0065] Similarly, the fusion methods for features N4 and N5 are shown in formulas (10) and (11):

[0066] N4=F[Downsample(N3);F4;P4] (10)

[0067] N5=F[Downsample(N4);P5] (11)

[0068] This design directly integrates low-level feature information with high-level semantic information and feeds it into the loss function. It includes both low-level localization information and high-level semantic information, which helps to detect local small features and has superior performance. The improved novel bidirectional feature fusion network is used to detect custom model datasets. Compared with the YOLOv5 algorithm, the improved algorithm has significantly improved the accuracy of sample model detection and localization.

[0069] S23, Anchor Frame Parameter Optimization

[0070] YOLOv5 uses AutoLearningBoundingBoxAnchors to automatically learn from the training data. It automatically calculates the ground truth of the dataset using the Kmeans clustering algorithm to obtain suitable anchor boxes. Therefore, the anchor box parameters need to be optimized. The optimized anchor box sizes are shown in Table 2.

[0071] Table 2 Anchor Box Scale Assignment Table (Note: Appropriate anchor boxes are obtained by automatically calculating the target boxes of the dataset using the K-means clustering algorithm)

[0072]

[0073] like Figure 3 The image shown is a partial image of a single object from the Cornell dataset, representing a single object being grasped from the Cornell dataset.

[0074] like Figure 4 The image shown is a partial image of a self-made multi-target grasping object; the multi-target grasping object.

[0075] like Figure 5 The loss function curve of the model shown is shown in the figure. The loss change curve of the improved Yolov5 recognition model is shown in the figure: the horizontal axis corresponds to the epoch-number of training, and the vertical axis corresponds to the loss value. The figure above shows the change of training set bounding box loss, confidence loss and class loss value. It can be seen that through multiple iterations of training, the error of the model gradually decreases and the curve gradually converges.

[0076] like Figure 6 The PR curve before improvement is shown. PR is easy to understand, referring to precision and recall respectively. P represents the proportion of predicted boxes that are ground truth (gt) boxes, and R represents the proportion of all gt boxes that are predicted. It is speculated that the PR value here is obtained under given confidence and IOU thresholds; the map value of this graph is 92.4.

[0077] like Figure 7 The improved PR curve shown has a map value of 96.6, which is a significant improvement.

[0078] like Figure 8 The target detection results shown demonstrate the effectiveness of the trained model. It can be seen that the model can basically identify each target object with corresponding accuracy.

[0079] like Figure 9 The confusion matrix shown in the figure represents the overall confidence statistics for the 24 categories of objects trained. It can be seen that the recognition accuracy of most objects is very high, while a very few objects have relatively low confidence.

[0080] Table 3 Ablation Experiment (Note: The YOLOv5 basic algorithm was improved, and different modules were added to compare the accuracy of three important indicators (recall, precision, and map value). Finally, the experiment proved that the best results were achieved when all three improvements were integrated into YOLOv5.)

[0081]

[0082] Table 4 Performance Comparison of Mainstream Detection Algorithms (Note: This table mainly compares the improved algorithm with other mainstream algorithms. FPS represents the detection speed, and Precision represents the accuracy. It can be seen that the improved algorithm has a slightly lower detection speed, but a significant improvement in accuracy.)

[0083]

[0084] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A mechanical arm target object detection method based on improved YOLOv5 algorithm, characterized in that, Comprise the following steps: S1, obtain data set and carry out data pretreatment: Download the Cornell data set from the network, use the general annotation software labelimg annotation tool to annotate the obtained data picture, after annotation, convert the PASCAL format file into the YOLO format file, complete the data set production; S2, improve the existing YOLOv5 network; S21, build space-coordinate attention mechanism SCAA, realize accurate positioning of target object and local feature information, inspired by convolution attention mechanism CBAM, combine spatial attention mechanism SA and coordinate attention mechanism CA; The working steps of the SCAA module are as follows: assuming that the input feature information is N, the input feature is compressed at the channel dimension, then the compressed feature is subjected to maximum pooling and average pooling operations, that is, the maximum value and the average value are extracted at two channels respectively, finally the feature maps of the two channels are fused, and a 7x7 convolution layer is passed to obtain weight coefficients N s , and the output feature N f is the product of the input feature N and N s , as shown in formulas (1) and (2): (1) (2) The entire space range is high as H, wide as w, in the vertical range of (H, 1), the vertical direction coordinate information is encoded and saved by using the pooling kernel, in the horizontal range of (1, w), the horizontal direction coordinate information is encoded and saved by using the pooling kernel, as shown in formula (3): (3) The features with the precise encoded information are concatenated and then passed through a 1x1 convolutional transformation function F1 and a non-linear activation function to obtain the intermediate features f with the encoded spatial information, as shown in equation (4): (4) Then use 1x1 convolution and Generate attention weights and As shown in equation (5): (5) Finally, the output of the attention module is calculated by formula (6): (6) After the C3 module of the backbone network, the SCAA module is added, after obtaining the deep feature information of the target through the C3 module, the SCAA module is used again, the spatial dimension weight is obtained first, then the feature information with large spatial weight is encoded in the horizontal and vertical directions, which is better for identifying and positioning the target model, the SCAA module is added in the backbone network of YOLOv5 algorithm, and the Backbone network is changed from 10 layers to 13 layers; S22, multi-scale feature fusion network, the Neck structure of YOLOv5 adopts the combination of FPN+PANeT, according to the superiority of BiFPN network feature fusion mode, and referring to the characteristics of BiFPN structure, the YOLOv5 algorithm model network is improved; In step S22, a multi-scale feature fusion network is designed: the Neck structure of YOLOv5 adopts the combination of FPN+PANeT, according to the superiority of BiFPN network feature fusion mode, the YOLOv5 algorithm model network is improved, and the low-level feature information is directly transmitted to the high-level feature through the jump connection, and the high-level semantic feature information is fused with the low-level information feature; The working mode of the new feature fusion network is as follows: the feature fusion mode is as shown in formula (7): (7) Wherein, f1, f2, f3 represent three features of the bidirectional feature fusion network, [;] represents two-by-two splicing of two features according to the channel dimension number, feature represents a new feature generated by splicing the features according to the channel dimension, and Pi represents the multi-scale fusion feature extracted by the feedforward deep neural network; In the original fusion network, N3, N4 and N5 represent multi-scale feature detection of small targets, medium targets and large targets respectively, feature N3 does not use low-level feature N2, but uses feature F3 after upsampling, feature N2 is obtained by fusing and splicing feature P2 and feature F3, and the CSP operator in YOLOv5 structure is denoted as function F, as shown in formula (8): (8) N3, N4, N5 are multi-scale fusion features generated by a new bidirectional feature fusion network, feature P2 generates feature P3 through down sampling, feature P4 and feature P5 are fused by channel splicing to obtain feature F4, feature F4 is generated by up sampling and splicing with feature P3 in channel dimension to generate feature F3, and multi-scale fusion feature N3 is generated by splicing feature N2, feature F3 and feature P3 in channel dimension, Downsample means that the feature is down sampled by two times through convolution, and the fusion mode of feature N3 is shown in formula (9): (9) Similarly, the fusion modes of features N4 and N5 are shown in formulas (10) and (11): (10) (11)； S23, optimize anchor box parameters, YOLOv5 uses adaptive anchor boxes to automatically learn training data, and automatically calculates the target box gt of the data set by using the Kmeans clustering algorithm; S3, train the network model.

2. The mechanical arm target object detection method based on the improved YOLOv5 algorithm according to claim 1, wherein In the step S21, first, the feature information is fused by weighting the maximum value and the average value of two channels after the spatial attention mechanism, and the feature information with spatial position information is output; then the feature with spatial position information is aggregated and weighted along the horizontal direction and the vertical direction respectively through the coordinate attention mechanism, two independent feature maps are formed, and finally the feature with direction encoding information is stored.

3. The mechanical arm target object detection method based on the improved YOLOv5 algorithm according to claim 1, wherein, In the step S23, the anchor box parameters are optimized, YOLOv5 uses adaptive anchor boxes to automatically learn the training data, and automatically calculates the target box gt of the data set by using the Kmeans clustering algorithm to obtain suitable anchor boxes.

Images