Improved multi-person pose estimation correction algorithm for yolov5

Abstract

The application provides a multi-person pose estimation correction algorithm for improving YOLOv5.The method is characterized in that: a skip attention module is integrated into a backbone network of YOLOv5, a weighted bidirectional feature pyramid is used in a neck network, the skip attention module and a Transformer encoder are jointly used, a detection head is added in a detection part, the network is more sensitive to micro targets, finally, key point object information is obtained by using the network for prediction and the key point object information is trimmed, and the technical problem of pose prediction of small scale targets in a high-density crowd is solved.The application improves the feature extraction capability of the network for an interesting area, focuses on useful target objects, strengthens the feature fusion capability of the network, makes the network pay more attention to global information, increases the detection head for small target detection, makes the network pay more attention to small scale targets, finally, the pose information is corrected combined with key point information, and finally, the AP50 of the prediction result is improved by 2.2% and the AP75 is improved by 3.3% compared with YOLOv5 on a COCO dataset.The application is suitable for being applied as a pose prediction precision method for small scale targets in a high-density crowd.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to an improved YOLOv5 multi-person pose estimation correction algorithm. Background Technology

[0002] Currently, human pose estimation, as a key area of ​​computer vision, is widely used in human-computer interaction, sensors, and commercial applications. The purpose of 2D human pose estimation is to detect key points of human body parts and connect them in an orderly manner. Traditional algorithms are based on graph structure models for human pose prediction, but their prediction accuracy is low and their speed is slow. Therefore, deep learning-based object detection algorithms are applied to human pose prediction. These are currently mainly divided into two-stage object detection algorithms (such as Mask-RCNN, Fast-RCNN, etc.) and single-stage object detection algorithms (such as YOLO, etc.).

[0003] Two-stage object detection algorithms require extracting candidate regions from the input image, adjusting each candidate region to a fixed size, and then inputting it into a convolutional neural network. The results are then fed into a classifier for category determination. Girshick et al. proposed using Fast-RCNN for human pose prediction. This method can accurately predict single, unoccluded figures, but it suffers from issues such as chaotic prediction of key points and neglecting the prediction of occluded figures when the figure density is high or key parts are occluded, resulting in reduced prediction accuracy.

[0004] Single-stage object detection algorithms directly generate the target's category probability and location coordinates, obtaining the final detection result in a single detection. Cao et al. introduced a method to encode the orientation of the limbs by associating body parts to form a complete human pose, called Part Association Field (PAF). This method learns keypoint locations and partial affinity fields through heatmaps, achieving bottom-up parsing of human pose. Newell et al. performed single-stage multi-person pose estimation using heatmaps by predicting association embedding labels and keypoints, grouping keypoints with similar label values ​​into a single pose. Nieet et al. introduced a new representation called Structured Pose Representation (SPR), setting a root keypoint for each person and encoding the relative positions of the remaining keypoints through displacement mapping. Kreisset et al. extended the concept of PAF, proposing a Part Intensity Field (PIF) for locating body parts, and combining PIF and PAF to form a composite field. By incorporating scale-dependent Laplacian L1 loss, their method outperformed previous methods on low-resolution images. Cheng et al. used HigherHRNet to address the scale variation problem in single-stage human pose estimation. This network uses a high-resolution feature pyramid during training and multi-resolution heatmap aggregation during inference.

[0005] The above methods have improved the accuracy of human pose estimation from different perspectives, but there is still a lack of effective methods for predicting small-scale targets in high-density crowds. Summary of the Invention

[0006] To improve the pose prediction accuracy of small-scale targets in high-density crowds, this invention proposes an improved YOLOv5 multi-person pose estimation correction algorithm. This method integrates a jump attention module into the YOLOv5 backbone network, then utilizes a weighted bidirectional feature pyramid in the neck network, combining the jump attention module with a Transformer encoder. A detection head is added to the detection part, making the network more sensitive to small targets. Finally, the network predicts keypoint object information and refines it to obtain the final multi-person pose estimation result, thus solving the technical problem of pose prediction for small-scale targets in high-density crowds.

[0007] The solution adopted by this invention to solve the technical problem is:

[0008] First, a skip attention module is integrated into the backbone network of YOLOv5 to help the network find regions of interest in images. Second, in the neck network, a weighted bidirectional feature pyramid is used to improve the network's ability to fuse features between feature maps of different scales, and the skip attention module is used in conjunction with the Transformer encoder to enable the network to acquire global information and rich contextual information. Third, a detection head is added to the detection part to make the network more sensitive to small targets. Finally, the keypoint object information predicted by the network is used to correct the pose object information to obtain the final multi-person pose estimation result.

[0009] Furthermore, a jump attention mechanism (JAM) is introduced into the backbone network to improve the network model's sensitivity to important feature information while reducing attention to useless features.

[0010] Furthermore, in the neck network, a weighted bidirectional feature pyramid network (BiFPN) is used to replace the path-aggregation network (PANet). Adding an extra weight to each input during feature fusion improves the network's ability to learn features at different scales. Simultaneously, a jump attention mechanism (JAM) is used in conjunction with the Transformer encoder to capture global and contextual information, enhancing the network model's feature fusion capability.

[0011] Furthermore, in the head part, a detection head generated from low-level and high-resolution features is added to improve the network model's tolerance for small-scale targets.

[0012] Finally, the human pose object and each key point object are predicted, and the pose object information is corrected using the obtained key point object information to improve the network's prediction accuracy of human pose.

[0013] The positive effects are as follows: First, by adding JAM to the backbone network of YOLOv5, this invention improves the network's ability to extract features from regions of interest, resists obfuscated information, and focuses on useful targets. Second, by replacing PANet in the neck network with BiFPN, the network's feature fusion ability is enhanced. Simultaneously, the combination of Transformer and JAM makes the network more focused on global information. Furthermore, a high-resolution small target detection head is added, making the network more focused on small-scale targets. Finally, the pose information is corrected by combining keypoint information to obtain the final prediction result. In summary, the method described in this application can achieve higher recognition accuracy for small-scale targets. Compared with YOLOv5, this invention improves AP50 by 2.2% and AP75 by 3.3% on the COCO dataset, verifying the accuracy and robustness of this invention. It is suitable for application as a method for improving the pose prediction accuracy of small-scale targets in high-density crowds. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the architecture of the present invention;

[0015] Figure 2 This invention provides an improved YOLOv5 network structure diagram;

[0016] Figure 3 This is a schematic diagram of the skip attention mechanism of the present invention;

[0017] Figure 4 This is a schematic diagram of the Transformer encoding structure;

[0018] Figure 5 Diagrams showing the network structures of PANet and BiFPN;

[0019] Figure 6 To visualize and compare experimental results using the method described in this invention on the COCO dataset;

[0020] Figure 7 To construct qualitative results of human pose estimation in crowded scenes using the method described in this invention;

[0021] Figure 8 To construct a model using the method described in this invention to predict qualitative results of pose objects on the COCO dataset;

[0022] Figure 9 To construct a model using the method described in this invention to predict the qualitative results of keypoint objects on the COCO dataset. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0024] In this invention, all embodiments, implementation methods, and features can be combined with each other without contradiction or conflict. In this invention, conventional equipment, devices, and components can be commercially available or self-made according to the disclosure of this invention. In this invention, to highlight the key points, some conventional operations and equipment, devices, and components are omitted or only briefly described.

[0025] First, a jump attention module is incorporated into the backbone network of YOLOv5 to help the network find regions of interest in images;

[0026] A jump attention mechanism (JAM) is introduced into the backbone network to improve the network model's sensitivity to important feature information while reducing attention to useless features.

[0027] See the attached instruction manual. Figure 2 The image is input into the YOLO detector to initially obtain key point information and pose information. However, since there are many detected anchor boxes, the NMS algorithm is used to filter the anchor boxes to obtain candidate pose objects and key point objects. Then, the candidate pose objects are corrected using the candidate key point objects to obtain the final human pose estimation result.

[0028] See the attached instruction manual. Figure 3 Attention mechanisms are general modules for enhancing image features, enabling network models to focus more on important features while downplaying secondary features. Traditional attention mechanisms process image features at both the channel and spatial levels, but neglect the direct relationship between input and output features. Therefore, to maximize the utilization of channel information and spatial location information in the feature map, this invention utilizes a skip attention mechanism as a method for the network to find regions of interest in an image, such as... Figure 3As shown in (a), this method consists of two parts: Jump Channel Attention Mechanism (JCAM) and Jump Spatial Attention Mechanism (JSAM), as described in Figures 1-2. Figure 3 As shown in (b) and (c), the Jump Channel Attention (JCAM) mechanism primarily focuses on the content being detected, while the Jump Spatial Attention (JSAM) mechanism primarily focuses on the location of the target being detected. For a given feature map, attention maps are derived sequentially along both channel and spatial dimensions. These attention maps are then multiplied with the input feature map for adaptive feature refinement.

[0029] The feature map C input to the Jump Attention Mechanism (JAM) module first passes through the Jump Channel Attention Mechanism (JCAM) module, which primarily focuses on the channel level of the features. Each feature channel is generated by different convolutional kernels, thus contributing differently to the network model. However, the network model still acquires feature information from each channel equally, preventing channels containing important feature information from standing out. Therefore, this application uses Jump Channel Attention Mechanism (JCAM) to redistribute weights based on the amount of feature information contained in each feature channel, enabling the network to selectively focus on important features while weakening secondary features.

[0030] The Jump Channel Attention (JCAM) mechanism operates as follows: Average pooling and max pooling are applied to the feature map C to generate two distinct contextual feature descriptors, which are then used to aggregate the feature mapping information. These descriptors are then fed into a multilayer perceptron consisting of two fully connected layers, where the channel attention weights are redistributed to obtain the intermediate features C. max With C avg The two intermediate features are summed, and then non-linearly activated using the Sigmoid function before being fused with the feature map C to obtain the channel interest feature map C. m The calculation process is as follows:

[0031] C max =MLP(AvgPool(C)) (1)

[0032] C avg =MLP(MaxPool(C)) (2)

[0033] C m =σ(C max +C avg )·C (3)

[0034] Where MaxPool represents the max pooling operation on the feature map C, and AvgPool represents the average pooling operation on the feature map. MLP stands for Multilayer Perceptron, and σ is the Sigmoid function.

[0035] After considering the channel level, the channel interest feature map C m The information is fed into the Jump Spatial Attention Mechanism (JSAM) module, which uses JSAM to learn the importance of information at different spatial locations, thereby enhancing the ability to represent information.

[0036] The specific operation of the Skip Spatial Attention (JSAM) mechanism is as follows: After performing max pooling and average pooling operations on the values ​​of different channels at the same point in the same spatial plane, the two feature maps are concatenated to obtain an information-rich feature descriptor. A 7×7 convolution kernel is used for convolution, and the values ​​are normalized using the Sigmoid function to obtain an interest feature map. Skip connections are then used to connect the interest feature map with the channel interest feature map C. m By merging the two, a direct connection is established, and spatial interest feature map C is obtained. k The calculation process is shown in equation (4):

[0037] C k =σ(f 7×7 [AvgPool(C m MaxPool(C) m )])·C m (4)

[0038] Secondly, in the neck network, a weighted bidirectional feature pyramid is used to improve the network's ability to fuse features between feature maps of different scales, and a skip attention module and a Transformer encoder are used together to enable the network to acquire global information and rich contextual information.

[0039] In the neck network, a weighted bidirectional feature pyramid network (BiFPN) replaces the path-aggregation network (PANet). Adding an extra weight to each input during feature fusion improves the network's ability to learn features at different scales. Simultaneously, JAM and a Transformer encoder are used together to capture global and contextual information, enhancing the network model's feature fusion capabilities.

[0040] The Transformer encoder in the YOLOv5 backbone network has a relatively weak ability to acquire global information. Therefore, this application replaces some convolutional blocks and CSP bottleneck blocks with Transformer encoder blocks, enabling the network to capture global information and rich contextual information. This supplements the missing global information caused by convolution and improves the network's ability to detect small-scale targets. (See appendix to the specification.) Figure 4 The Transformer encoder mainly consists of an input module, an encoding module, and an output module. This module first quantizes the relative positional relationships between features through positional encoding (PE) operations. It then uses a sine function to calculate features at even positions and a cosine function to calculate features at odd positions, obtaining the input X for the encoding module. A .

[0041] The encoding module consists of multiple encoding layers, each containing a multi-head attention (MHA) and a feed-forward network (FFN), with connections between sub-layers via residuals. The MHA in this module helps the current node acquire contextual semantic information, focusing on different regions within the global receptive field, enabling the network to focus on the target region and obtain effective features. Considering that this process might not be well-fitting for complex situations, a feed-forward neural network is used to improve the model's fitting ability. The attention mechanism, as the core of this module, is based on a given query matrix Q∈R. n×d and key-value pair matrix K, V∈R n×d The calculation is as follows:

[0042]

[0043] Multi-head attention (MHA) utilizes n learnable linear transformations to project queries, keys, and values ​​into different subspaces. After performing n operations in parallel, the outputs of the n attentions are concatenated with the linear transformation matrix W. o The final output is obtained by multiplying the results. The calculation formula is as follows:

[0044] MHA(Q,K,V)=Concat(head1;...;head n W o (6)

[0045]

[0046] Among them, W i Q W i K Wi V The matrices corresponding to Q, K, and V projected onto the i-th subspace, respectively; W o To compute the matrix of linear transformation of the head, head i For different spatial heads, n is the number of multi-head attention heads, d k This represents the result of focusing on features from n different spaces, and is usually set to d / n.

[0047] Residual connections are used to superimpose the initial results with the outputs of the encoding module sublayers, followed by hierarchical normalization. To prevent overfitting due to increased network depth, Dropout layers are used to process the encoding module, which helps the network converge better and obtain the final output. The specific operation is as follows:

[0048] X B =LayerNorm(Attention(X) A )+X A (8)

[0049] FFN(x)=max(0,xW1+b1)W2+b2 (9)

[0050] X C =LayerNorm(FFN(X) B )+X B (10)

[0051] Where FFN stands for forward feedback computation, LayerNorm is the layered normalization operation, and X... B X is the result obtained after the first stratified normalization. C The result is obtained after the second hierarchical normalization, where W1, W2, b1, and b2 represent the weights and biases in the forward feedback, respectively.

[0052] In this application, the weighted bidirectional feature pyramid network (BiFPN) is used instead of PANet, which is used as the neck network of YOLOv5. Although PANet can transmit high-level semantic information through downsampling, the contributions of feature information at different scales in the feature fusion process are different. Therefore, this application uses the weighted bidirectional feature pyramid network (BiFPN) instead of PANet, and the network structure is as follows. Figure 5 As shown, a fast normalized fusion method is used to weight and fuse high-level features rich in semantics and low-level features with high-resolution information in order to learn the importance of different input features. The calculation of the fast normalized fusion method is as follows:

[0053]

[0054] Where O represents the result of fast normalization fusion, and wi To learn the weight parameters, the parameters range from [0,1] and are used to distinguish the importance of different features during feature fusion. i The features to be fused are ε, which is a minimum value used to avoid errors caused by numerical instability.

[0055] The output of the weighted bidirectional feature pyramid network includes both coarse-grained and fine-grained features. During training, the network can autonomously assign corresponding weights to dissimilar fused features, thereby improving the network's representation ability and accuracy.

[0056] Furthermore, the detection part is improved by adding a detection head generated from low-level and high-resolution features, which enhances the network model's tolerance for small-scale targets and makes the network more sensitive to tiny targets.

[0057] YOLOv5's detection mechanism primarily relies on detection heads. The main function of these heads is to detect the location and category of objects in feature maps. Different scale detection heads detect different image features. YOLOv5 typically uses three detection heads to detect images at three different scales: 20×20, 40×40, and 80×80. The corresponding receptive fields are 32×32, 16×16, and 8×8.

[0058] However, in real-world scenarios, the proportion of the target person in an image is affected by factors such as camera distance and crowd density. When the camera position is far from the target or the crowd is densely packed, the number of pixels occupied by the target in the image will decrease. When the target exceeds the receptive field, the detection results will be interfered with.

[0059] Therefore, this application adds a 160×160 detector head to the original three detector heads, reducing the receptive field to 4×4, enabling the network to acquire output grids of various scales. Smaller-scale grids have larger receptive fields for detecting larger targets, while larger-scale grids have smaller receptive fields for detecting smaller targets. By predicting images at these different scales, keypoint objects focusing on the person's keypoints and pose objects focusing on the person as a whole can be obtained.

[0060] Since there are too many prediction results for the target, the predicted pose object O is analyzed using the NMS method based on the target pose IOU threshold τ1 and the target keypoint IOU threshold τ2. P and key object O K Filter the samples to obtain candidate pose objects O. P′ and candidate key point objects O K′ The calculation formula is as follows:

[0061] OP′ =NMS(O P ,τ1) (12)

[0062] O K′ =NMS(O K ,τ2) (13)

[0063] Finally, the human pose object and each key point object are predicted. The obtained key point object information is used to correct the pose object information, thereby improving the network's prediction accuracy of human pose and obtaining the final multi-person pose estimation result.

[0064] The Amend algorithm, specifically designed for keypoint objects, is used to detect individual keypoints with strong local features. Common keypoints in human pose estimation include elbows and knees. Traditional human pose estimation connects detected keypoints in a bottom-up manner. However, keypoint objects do not carry information about the person or pose; they exist in a subspace of a pose object. Pose objects are better suited for keypoints with weak local features because they allow the network to learn spatial relationships between a set of keypoints.

[0065] Therefore, this application proposes a modified algorithm, Amend, which utilizes candidate keypoint objects O. K′ Correct candidate pose object O P′ To improve the accuracy of human pose estimation, the algorithm steps are as follows:

[0066] (1) Initialize the pose object and pose confidence to 0.

[0067] (2) Traverse candidate pose objects O P′ Obtain the confidence score of candidate pose objects.

[0068] (3) Traverse the key points of each pose object and obtain the coordinates.

[0069] (4) Save the pose objects whose pose confidence is greater than the confidence tolerance to P′.

[0070] (5) When P′ and candidate key point object O K′ If neither is empty, the two will be merged.

[0071] (6) Traverse candidate key point objects O K′ This allows us to obtain the location of the maximum probability and the confidence level of key point objects.

[0072] (7) Calculate the Euclidean distance between key points in the pose object and key point objects, and obtain the minimum distance index.

[0073] (8) If the minimum distance is less than the distance tolerance and the confidence of the pose object is less than the confidence of the key point object, then replace the coordinates and confidence of the pose object with the coordinates and confidence of the key point.

[0074] (9) Connect the final determined key point coordinates to obtain the corrected multi-person pose estimation results.

[0075] Experiments were conducted using the method described in this application. The experimental process and results are as follows:

[0076] 1. Dataset and evaluation metrics:

[0077] The experiment used the COCO dataset, a large dataset for multi-person pose estimation. The training set contains 118,000 images, the validation set contains 5,000 images, and the test set consists of 20,000 images.

[0078] The evaluation metrics use different average precision (AP) to measure the model's detection capability. 50 and AP 75 These represent the AP values ​​when the IOU is 0.5 and 0.75, respectively. L This represents the AP value for bounding boxes with a pixel area greater than 96×96. The calculation formula is as follows:

[0079]

[0080]

[0081]

[0082] Where TP represents the number of correct results in the detector output, FP represents the number of incorrect results in the detector output, and FN represents the number of undetected true values. AP is the area enclosed by the PR curve and the coordinate axes, and the size of the area is positively correlated with the model's performance.

[0083] 2. Experimental Setup

[0084] The input images were resized to 640×640 pixels, with a batch size of 8, and training was performed for 300 epochs. The Adam optimizer was used with an initial learning rate of 0.0001. Data augmentation was performed during training using techniques such as mosaic manipulation, HSV color space perturbation, horizontal flipping, translation, and scaling. The loss L for a single image was calculated as follows:

[0085] L=N b (aL obj +bL box +cL cls +dL kps (17)

[0086] Among them, L obj For confidence loss, L box For bounding box regression loss, L cls For classification loss, L kps The loss is the critical point loss, and the total loss L is the weighted sum of the losses of each part, calculated by batch size N. b Scaling is applied. a, b, c, and d all represent the weight percentage of the corresponding loss function in the total loss function.

[0087] 3. Experimental results on the COCO dataset

[0088] To verify the performance of the method described in this application, qualitative and quantitative experiments were conducted to compare it with human pose estimation methods in recent years. The results are shown in Table 1 and 2. Figure 6 As shown.

[0089] As shown in Table 1, the quantitative experimental results indicate that although the accuracy (AP) of the proposed method on the COCO dataset is lower than that of the HigherHRNet-W48 method, the AP... 50 AP 75 AP L In all metrics, it is superior to other methods; compared to the YOLOPose method, it performs better in AP. 50 and AP 75 The figures are 0.8% and 3.3% higher, respectively.

[0090] As can be seen from the experimental data and prediction results, the method described in this application effectively extracts human features from images by using a jump attention mechanism; promotes multi-scale feature fusion by using a BiFPN network that combines JAM and Transformer encoders; the small target detection head makes the network pay more attention to small-scale targets; and the modified algorithm improves the prediction accuracy of human pose.

[0091] Table 1: Experimental comparison results on the COCO dataset

[0092]

[0093] See the attached instruction manual. Figure 6 , Figure 6 A set of visual experimental comparison results are presented on the COCO dataset, among which Figure 6 (ac) represent the original image, the prediction results of the HigherHRNet method, and the FCpose method, respectively. Figure 6(Ours) represents the prediction results of this paper. Thicker (red) borders indicate prediction errors or incomplete predictions. As shown in the figures, in the first row of images, the figures on the left and in the middle are occluded, causing the loss of key features. Compared with HigherHRnet and FCpose, the method described in this application achieves more accurate human pose estimation. In the second row of images, the figures are relatively concentrated in crowded scenes, with similar local areas and many small objects. The method described in this application can better achieve multi-person pose estimation. In the third row of images, the figures' clothing colors are the same, indicating local occlusion. The method described in this application can effectively eliminate interference and has high robustness. In the fourth row of images, the overlap rate of the figures in the middle is high. Compared with other methods, the method described in this application achieves higher accuracy in human pose estimation.

[0094] See the attached instruction manual. Figure 7-9 To verify that the method described in this application can achieve robust human pose estimation for target figures in crowded scenes, a prediction process in crowded scenes is presented. Figure 7 This is a qualitative result of the method model described in this application performing human pose estimation on the COCO dataset. Figure 8 , Figure 9 This describes the human pose estimation process of the method described in this application on the COCO dataset. Experimental results show that even with a large number of detected targets, the method described in this application can still enable the network to focus on small-scale human targets, verifying the high robustness of the model.

[0095] 4. Ablation test

[0096] To verify the effectiveness of each module, this application adds each module to the original model for verification. The experimental results are shown in Table 2.

[0097] Table 2: Ablation Experiment Results

[0098]

[0099] Add JAM, AP to the neck of the YOLOv5 backbone network 50 The value increased by 0.4%, AP 75 The value increased by 0.8%, indicating that skip attention can make the network pay more attention to the detection target; adding a Transformer to the neck network part, AP 50 The value increased by 1.4%, AP 75 The value increased by 3.1%, indicating that the Transformer encoder enables the network to acquire global and contextual information; replacing the path aggregation network in the neck network with BiFPN, AP 50 The value increased by 0.6%, AP 75The value increased by 1%, indicating that BiFPN can enhance the feature fusion capability of the network; adding a detection head to the detection head part, AP 50 The value increased by 1.4%, AP 75 The value increased by 1.9%, indicating that the added detection head is more sensitive to the detection of small-scale targets; by adding the Amend algorithm to the original model, AP 50 The value increased by 1.6%, AP 75 The value increased by 2.8%, indicating that correcting pose information with keypoint information can effectively improve the accuracy of human pose estimation; under the combined effect of these multiple modules, AP 50 The value increased by 2.2%, AP 75 The value increased by 3.3%, and the overall network performance reached its optimal level, verifying the effectiveness of the collaboration between the various modules.

[0100] In summary, the improved YOLOv5 multi-person pose estimation correction algorithm proposed in this application first adds JAM to the YOLOv5 backbone network to improve the network's feature extraction ability for regions of interest, resist obfuscation, and focus on useful target objects. Second, it replaces PANet in the neck network with BiFPN to enhance the network's feature fusion ability, and combines Transformer and JAM to make the network pay more attention to global information. Third, it adds a high-resolution small target detection head to make the network pay more attention to small-scale targets. Finally, it combines keypoint information to correct the pose information to obtain the final prediction result. Experiments on the COCO dataset show that, compared with other methods, the method described in this application achieves higher accuracy, and ablation experiments verify the effectiveness of each module presented in the method described in this application.

[0101] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An improved YOLOv5 multi-person pose estimation correction algorithm, characterized in that: In the backbone network of YOLOv5, a jump attention module is incorporated to help the network find regions of interest in images; A skip attention mechanism is introduced into the backbone network to improve the network model's sensitivity to important feature information while weakening its attention to useless features; The skip attention mechanism is divided into two parts: skip channel attention mechanism and skip space attention mechanism. The skip channel attention mechanism mainly focuses on the content to be detected, while the skip space attention mechanism mainly focuses on the location of the target to be detected. For a given feature map, an attention map is derived sequentially along the two independent dimensions of channel and space. Then, the attention map is multiplied with the input feature map to perform adaptive feature refinement. The feature map C input to the skip attention mechanism module first passes through the skip channel attention mechanism module, which mainly focuses on the channel level of the features. Each feature channel is generated by different convolutional kernels, so their contributions to the network model are different. However, the network model still acquires the feature information of each path equally, so the channel containing important feature information cannot stand out. Therefore, this application uses the skip channel attention mechanism to redistribute the weights according to the amount of feature information contained in the feature channel, so that the network can focus on important features in a targeted manner, while weakening secondary features. The specific operation of the skip channel attention mechanism is as follows: Average pooling and max pooling are performed on the feature map C to generate two different context feature descriptors, thereby aggregating the feature mapping information; these descriptors are then input into a multilayer perceptron consisting of two fully connected layers to redistribute the channel attention weights, resulting in intermediate features C. max With C avg The two intermediate features are summed, and then non-linearly activated using the Sigmoid function before being fused with the feature map C to obtain the channel interest feature map C. m The calculation process is as follows: ； Where MaxPool represents the max pooling operation on the feature map C, and AvgPool represents the average pooling operation on the feature map; MLP stands for Multilayer Perceptron. For the Sigmoid function; After considering the channel level, the channel interest feature map C m The information is input into the jump spatial attention mechanism module, which uses the jump spatial attention mechanism to learn the importance of information at different spatial locations, thereby enhancing the ability to represent information. The specific operation of the skip spatial attention mechanism is as follows: After performing max pooling and average pooling operations on the values ​​of different channels at the same point in the same spatial plane, the two feature maps are concatenated to obtain an information-rich feature descriptor. A 7×7 convolution kernel is used for convolution, and the values ​​are normalized using the Sigmoid function to obtain an interest feature map. A skip connection is then used to connect the interest feature map with the channel interest feature map C. m By merging the two, a direct connection is established, and spatial interest feature map C is obtained. k The calculation process is shown in equation (4): ； Secondly, in the neck network, a weighted bidirectional feature pyramid is used to improve the network's ability to fuse features between feature maps of different scales, and a skip attention module and a Transformer encoder are used together to enable the network to acquire global information and rich contextual information. The neck network utilizes a weighted bidirectional feature pyramid to improve the network's ability to fuse features between feature maps of different scales, and combines a skip attention mechanism with a Transformer encoder to enable the network to acquire global information and rich contextual information. In the neck network part, a weighted bidirectional feature pyramid network is used to replace the path aggregation network; an additional weight is added to each input during feature fusion, which is beneficial for the network to learn features at different scales; at the same time, a jump attention mechanism and a Transformer encoder are used together to capture global and contextual information and improve the feature fusion capability of the network model. The Transformer encoder in YOLOv5 addresses the network's weakness in acquiring global information. By replacing some convolutional blocks and CSP bottleneck blocks with a Transformer encoder block, the network can capture global information and rich contextual information, thus supplementing the missing global information caused by convolution and improving its ability to detect small-scale targets. The Transformer encoder mainly consists of an input module, an encoding module, and an output module. The input module first quantifies the relative positional relationships between features through positional encoding, using a sine function to calculate features at even positions and a cosine function to calculate features at odd positions, thus obtaining the input X to the encoding module. A ; The encoding module consists of multiple encoding layers, each sub-layer containing a multi-head attention and a feedforward neural network, with the sub-layers connected via residuals. The multi-head attention within this module helps the current node obtain contextual semantic information, focusing on different regions within the global receptive field, enabling the network to focus on the target region and acquire effective features. Considering that this process may not be well-fitting for complex situations, a feedforward neural network is used to improve the model's fitting ability. The attention mechanism, as the core of this module, is based on a given query matrix. and key-value pair matrix The calculation is as follows: ； Multi-head attention utilizes n learnable linear transformations to project queries, keys, and values ​​into different subspaces. After performing n operations in parallel, the outputs of the n attention points are concatenated with the linear transformation matrix W. 0 The final output is obtained by multiplying the results; the calculation formula is as follows: ； ； in, , , W corresponds to the matrices projected onto the i-th subspace from Q, K, and V, respectively; W 0 To compute the matrix of linear transformation of the head, head i For different spatial heads, n is the number of multi-head attention heads, d k This represents the result of focusing on features from n different spaces, and is usually set to d / n; Residual connections are used to superimpose the initial results with the outputs of the encoding module sublayers, followed by hierarchical normalization. This prevents overfitting due to increased network depth. Dropout layers are used to process the encoding module, which helps the network converge better and obtain the final output. The specific operations are as follows: ； Where FFN stands for forward feedback computation, LayerNorm is the layered normalization operation, and X... B X is the result obtained after the first stratified normalization. C The result is obtained after the second hierarchical normalization, where W1, W2, b1, and b2 represent the weights and biases in the forward feedback, respectively. Specifically, the weighted bidirectional feature pyramid network replaces PANet, and a fast normalized fusion method is used to weight and fuse high-level features rich in semantics and low-level features with high-resolution information in order to learn the importance of different input features. The calculation of the fast normalized fusion method is as follows: ； Where O represents the result of fast normalization fusion, and W i The weight parameters are learned, with values ​​ranging from [0,1], and are used to distinguish the importance of different features during feature fusion; I i Features to be integrated It is a minimum value used to avoid errors caused by numerical instability; The output of the weighted bidirectional feature pyramid network includes both coarse-grained and fine-grained features. During training, the network can autonomously assign corresponding weights to dissimilar fused features, thereby improving the network's representation ability and accuracy. Furthermore, an additional detection head is added to the detection section to make the network more sensitive to small targets; The detection part is improved by adding a detection head generated from low-level and high-resolution features, which improves the network model's tolerance for small-scale targets and makes the network more sensitive to tiny targets. By adding a 160×160 detector to the original three detectors, the receptive field is reduced to 4×4, allowing the network to acquire output grids of various scales. The smaller grids have larger receptive fields and are used to detect larger targets, while the larger grids have smaller receptive fields and are used to detect smaller targets. By predicting images of these different scales, keypoint objects with keypoints of the person as the main body and pose objects with the person as the main body can be obtained. Because there are too many prediction results for the target, the target pose IOU threshold is used. IOU threshold with target key points The NMS method is used to predict the pose object O. p and key object O K Filter the data to obtain candidate pose objects. and candidate key point objects The calculation formula is as follows: ； Finally, the pose object information is corrected using the key point object information obtained from the network prediction to obtain the final multi-person pose estimation result; Predict human pose objects and each key point object, use the obtained key point object information to correct the pose object information, improve the network's prediction accuracy of human pose, and obtain the final multi-person pose estimation result. The correction algorithm specifically uses keypoint objects to detect individual keypoints with strong local features; common keypoints in human pose estimation include the elbow and knee joints; and it utilizes candidate keypoint objects. Correct candidate pose objects To improve the accuracy of human pose estimation, the algorithm steps are as follows: 1) Initialize the pose object and pose confidence to 0; 2) Traverse candidate pose objects Obtain the confidence score of the candidate pose object; 3) Iterate through the key points of each pose object and obtain their coordinates; 4) Save pose objects whose pose confidence is greater than the confidence tolerance to [the appropriate location]. middle; 5) When and candidate key point objects If neither is empty, merge the two. 6) Traverse candidate key point objects Get the location of the maximum probability and the confidence level of key point objects; 7) Calculate the Euclidean distance between keypoints in the pose object and keypoint objects, and obtain the minimum distance index; 8) If the minimum distance is less than the distance tolerance, and the confidence level of the pose object is less than the confidence level of the key point object, then replace the coordinates and confidence level of the pose object with the coordinates and confidence level of the key point. 9) Connect the final determined key point coordinates to obtain the corrected multi-person pose estimation results.

2. The improved YOLOv5 multi-person pose estimation correction algorithm as described in claim 1, characterized in that: The image is input into the YOLO detector to initially obtain key point information and pose information. The NMS algorithm is used to filter anchor boxes to obtain candidate pose objects and key point objects. The candidate pose objects are then corrected using the candidate key point objects to obtain the final human pose estimation result.

Images