Improved YOLOv7 human body key point detection method

Abstract

The invention discloses an improved YOLOv7 human body key point detection method. According to the method, manual labeling is carried out through LabelImage to obtain abundant training samples; replacing an original MP module in the YOLOv7 with an L-MP module to improve feature extraction, and adding a LiftDown Pool branch and residual connection; an EDH-Conv module is newly added between a Backbone module and a Head module of the YOLOv7 network, and the output of the EDH-Conv module is used as spatial weight distribution to correct an activated region, so that the detection capability of the network on a small target is further improved; an improved RSwin Transformer encoder is added to improve the detection effect of small targets and overlapped targets in a complex scene; a loss function is changed into an improved key point detection loss based on L2 loss by referring to Wise-IoU LOSS, and a dynamic focusing mechanism used for adjusting a key point error weight is introduced to accelerate convergence of the model. According to the method, the human body key point detection and identification capability in a complex scene is enhanced, and the method has higher robustness, accuracy and efficiency for human body key point detection, identification and counting tasks in sports.

Description

Technical Field

[0001] The present invention belongs to the field of human key point detection and recognition, and specifically relates to a human key point detection method based on improved YOLOv7. Background Art

[0002] Human key point detection and recognition is an important research direction in the field of computer vision. In the field of sports, it is an important part of the analysis and evaluation of movements. With the increasing popularity of physical education and the continuous improvement of competitive level, it is becoming more and more important to judge the normativeness of movements and count standard movements. In this context, human key point detection and recognition tasks play an important role in judging the normativeness of movements. Therefore, using computer vision technology to detect human key points and then realize the recognition of normative movements has become a research hotspot in the fields of physical education and sports competition.

[0003] With the standardization and competitive development of sports, traditional analysis methods often rely on manual observation and recording. This method has differences, high labor costs and personnel training expenses, and staff may have problems such as standard judgment of movements and counting errors, which affect the fairness and standardization of sports. In recent years, as deep learning technology has been widely used in the field of human key point detection and recognition, the accuracy and efficiency of human key point detection and recognition have been greatly improved, and the labor costs and subjective influence of traditional analysis methods have been reduced. However, in the actual application process, in the natural environment, due to problems such as different light intensity, shooting angles and distances, YOLOv7 has some unsatisfactory aspects in human key point detection and recognition.

[0004] For target detection, Zeng et al. (Zeng LM, Hou J, Chen ZR, et al. Lightweight traffic sign detection network based on weak semantic segmentation [J]. Computer Engineering, 2022, 48 (9): 269-276, 285.) proposed a weak semantic segmentation method to improve the shortcomings of slow target detection and low recognition accuracy when high-resolution input is used. They designed a lightweight improved network based on YOLOv4, using the improved MobileNetv3-Lite as the basic semantic feature extraction network, and using the combination of FPN and PAN to achieve dual feature fusion. In addition, a dynamic enhancement attachment based on weak semantic segmentation was designed for the detection head, without increasing the number of floating-point operations of the model too much. In the actual application process training, the comprehensive performance in target detection has a great advantage. For feature upsampling, Zhao et al. (Zhao J, Snoek CG M. Liftpool: Bidirectional convnet pooling [J]. arXiv preprint arXiv: 2104.00996, 2021.) proposed a lift pool for the bidirectional pooling layer, which includes a down-pool and a lift pool, and can perform down-pooling and up-pooling at the same time. It improves the accuracy and robustness of image classification, generates more refined up-sampled feature maps, and improves the ability to cope with fine-grained image classification, super-resolution challenges, or tasks that require high detail preservation. For visual transformers, Liu et al. (Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 10012-10022.) proposed an improved Swin Transformer, which uses a layered transformer and shifted window calculations to solve the difference between language and vision. The shift-and-window scheme limits attention calculations to non-overlapping local windows while allowing cross-window connections, achieving higher efficiency. It also proves that the layered design and shift-window approach are beneficial to the full MLP architecture.Regarding the loss function, Tong et al. (Tong Z, Chen Y, Xu Z, et al. Wise-IoU: bounding box regression loss with dynamic focusing mechanism [J]. arXiv preprint arXiv: 2301.10051, 2023.) proposed a dynamic non-monotonic FM loss method based on IoU, and used a gradient gain distribution strategy to reduce the competitiveness of high-quality anchor boxes, reduce the harmful gradients generated by low-quality examples, and improve the overall performance of the detector.

[0005] In summary, although YOLOv7 has made some progress in the detection and recognition of key points on the human body, it still needs to be improved to better meet the needs of action normative judgment and action counting in the field of sports. This will bring more standardized and intelligent operation and management to the fields of physical education and sports competition, and promote the development and improvement of physical education and sports competition. Summary of the invention

[0006] The present invention aims to overcome the deficiencies in the above-mentioned prior art and proposes an improved YOLOv7 human key point detection method to improve the recognition efficiency, robustness and accuracy of sports detection and recognition counting in complex scenes.

[0007] In order to achieve the above content of the invention, the present invention provides an improved YOLOv7 human key point detection method, comprising the following steps:

[0008] (1) Collect sit-up image samples;

[0009] (2) Preprocess and label the image;

[0010] (3) Training set production based on data augmentation;

[0011] (4) Replace the original MP module in YOLOv7 with the L-MP module to improve feature extraction;

[0012] (5) Add an EDH-Conv module between the Backbone module and the Head module of the YOLOv7 network;

[0013] (6) Add RSwin Transformer encoder to the YOLOv7 network to optimize feature extraction;

[0014] (7) Train the deep network based on the optimized loss function and realize key point detection.

[0015] Furthermore, the collecting of image samples of middle school students doing sit-ups refers to collecting motion image samples in different environments, angles and distances, so that the image samples include pictures with different light intensities and different human body shooting angles; the preprocessing and labeling of the images refers to adjusting the size of the input images to 608×608 through the OpenCV library, and using OpenPose to label human key points; the production of a training set based on data augmentation refers to using Mosaic data augmentation, color perturbation and brightness adjustment methods to produce a data set for training; the key point detection based on loss function optimization refers to adaptively modifying the loss function with reference to Wise-IoU LOSS, adjusting it to an L2 loss suitable for key point detection, introducing a dynamic non-monotonic focusing mechanism for error weight distribution of key points, and adding geometric constraints to ensure a reasonable positional relationship between key points.

[0016] Furthermore, the replacement of the original MP module in YOLOv7 with the L-MP module to improve feature extraction refers to replacing the original MP module of YOLOv7 with an improved L-MP module, and the improved L-MP module refers to a new module obtained by adding a branch including a LiftDown Pool module on the basis of the MP module, outputting the outputs of the LiftDown Pool module and the Maxpool module to the subsequent module through a skip connection method, and then aggregating the results of the three branches through a cat method, and finally modifying the convolution method of the third branch; the newly added branch including the LiftDown Pool module refers to a branch formed by connecting the LiftDown Pool module and the 1×1 convolution module in series; the outputting the outputs of the LiftDown Pool module and the Maxpool module to the subsequent module through a skip connection method refers to inputting the result of the LiftDown Pool module into the cat C module through a skip connection method, and the cat C module refers to the module obtained by connecting the LiftDown Pool module and the 1×1 convolution module of the first branch in series; at the same time, the result of the Maxpool module is input into the cat B module through a skip connection method, and the cat The B module refers to the Maxpool module of the second branch and the 1×1 convolution module of the second branch, which are connected in series with the four 1×1 convolution modules in the third branch and then aggregated in cat mode; the jump connection method allows the input features to be directly passed to the subsequent layers, ensuring that the low-level features extracted early are retained, effectively avoiding gradient disappearance and feature attenuation; the results of the three branches are aggregated in cat mode, which means that the cat C module containing the results of the first branch and the cat B module containing the results of the second branch and the third branch are aggregated and output in series in cat mode; by combining LiftDownPool and super downsampling technology, the complementary downsampling feature map obtained can retain more key information in the downsampling process and avoid the loss of key features; the convolution method of modifying the third branch refers to modifying the original 1×1 convolution module and a 3×3 convolution module in the third branch to four 1×1 convolution modules in series, reducing the number of parameters in each layer, thereby speeding up the calculation speed.

[0017] Furthermore, the newly added EDH-Conv module between the Backbone module and the Head module of the YOLOv7 network refers to the newly added EDH-Conv module between the Backbone module and the Head module of the YOLOv7 network; the EDH-Conv module refers to the output of the two CBS modules in the YOLOv7 network that are connected to the backbone network and the output of the ELAN module in the backbone network as the three inputs of the EDH-Conv module, and then the three inputs are successively distributed to three different parallel branches, and then the features of the three branches are Figure 1 The three branches are connected in series to form a total feature map, and the feature map is subjected to a 1×1 convolution to obtain the final feature map output; the three branches refer to that the first branch includes a 3×3 convolution with a step size of 2 and a 3×3 transposed convolution with a step size of 2; the second branch consists of a 1×1 convolution, a 3×3 convolution and a 3×3 convolution with a dilation rate of 3 to expand the receptive field and obtain richer contextual information; the third branch includes a 1×1 convolution and a 3×3 convolution with a dilation rate of 1; the feature maps obtained by the above three branches are connected in series and aggregated through a Concat and a 1×1 convolution to obtain the final feature map output; the EDH-Conv module improves the detection ability of the YOLOv7 network for small targets, enhances the modeling of detail information at different levels, and improves the expression ability of high-level semantic features.

[0018] Furthermore, the said adding RSwin Transformer encoder to the YOLOv7 network to optimize feature extraction refers to adding a new RSwin Transformer module between the third and fourth CBS modules of YOLOv7; the said RSwin Transformer module refers to using the output of the CBS module in the YOLOv7 network as the input of the RSwin Transformer module, and then passing through the Convolutional Token Embedding module, the input features are generated into tokens and then distributed to four parallel stage modules, the output feature maps of the four parallel stage modules are Concat to obtain the total feature map, and then upsampled by the deconvolution Deconv module to extract more accurate features; the said stage module refers to a module composed of a conv layer and an RSwin Transformer Block module, the conv layer reduces the number of tokens at the beginning of the stage and doubles the channel dimension, and then inputs the RSwin Transformer Block module; the said RSwin Transformer Block module refers to the Swin Tranformer The multi-layer perceptron MLP in Block is replaced by the residual-based Res-MLP, which includes two layers of fully connected networks and directly passes the input to the output through residual connections to reduce the gradient vanishing problem and improve the training efficiency of the model; the feature maps of the above four parallel stage modules are aggregated and concat to obtain a complete feature map; the RSwin Transformer module improves the multi-scale feature extraction capability of the YOLOv7 network, improves the modeling of long-distance dependencies, and improves the detection of small and overlapping targets in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of an embodiment of the present invention;

[0020] Figure 2 The L-MP module in the improved YOLOv7 of the embodiment of the present invention;

[0021] Figure 3 This is a structural diagram of the EDH-Conv module of an embodiment of the present invention;

[0022] Figure 4 It is a structural diagram of the RSWin Transformer encoder in the improved YOLOv7 of an embodiment of the present invention;

[0023] Figure 5 It is a structural diagram of RSwin Tramsformer Block according to an embodiment of the present invention;

[0024] Figure 6 It is a structural diagram of the Res_MLP module of an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical route of the present invention is further described in detail below through specific embodiments and drawings.

[0026] The present invention is an embodiment of an improved YOLOv7 human key point detection method, comprising the following steps:

[0027] 1. Collect sit-up image samples

[0028] Collect sit-up image samples in different environments, angles, and distances to ensure that the image samples include pictures with different brightness, complex backgrounds, clothing occlusions, multi-angle rotations and tilts, and different postures.

[0029] 2. Preprocess and label the image

[0030] The OpenCV library is used to adjust the size of the input image to a uniform size of 608×608. OpenPose is used to automatically extract the key joints of the human body during sit-ups (such as head, torso, knees, arms, etc.) to generate a key point annotation training set for the sit-up action.

[0031] In the image annotation stage, a combination of automatic extraction and manual correction annotation is adopted. OpenPose is used for preliminary annotation, and inaccurate annotations are manually corrected through LabelImage to improve the accuracy of annotation.

[0032] 3. Training set creation based on data augmentation

[0033] The dataset for training is created using Mosaic data augmentation, color perturbation, and brightness adjustment methods.

[0034] Among them, Mosaic data enhancement refers to synthesizing multiple images into one image by random scaling, cropping, arranging and splicing, randomly changing the RGB channel values ​​in the image and adjusting the brightness to simulate changes under different lighting conditions, thereby generating new image data, and adding occlusions to simulate action occlusion problems in actual scenes.

[0035] 4. Replace the original MP module of YOLOv7 with the improved L-MP module to improve feature extraction

[0036] Replace the original MP module of YOLOv7 with an improved L-MP module to improve feature extraction capabilities: In order to solve the problem of information loss in the original YOLOv7 network during downsampling, this patent proposes to replace the MP module in YOLOv7 with an improved L-MP module to improve the effectiveness of feature extraction.

[0037] In the downsampling module of YOLOv7, the original MP module consists of two branches, which are mainly used to achieve downsampling. The first branch first undergoes maximum pooling (Maxpool), which is used to downsample features and extract important information, and then adjusts the number of channels through a 1×1 convolution. The second branch first changes the number of channels through a 1×1 convolution, and then passes through a 3×3 convolution block with a step size of 2, which is also used for downsampling. Finally, the results of the two branches are combined by addition to form a feature map after super downsampling.

[0038] The traditional MP module has several significant shortcomings: first, the single maximum pooling method easily leads to the loss of small target feature information, especially when dealing with smaller targets; second, the design cannot effectively retain low-level features, which in turn affects the effect of subsequent feature fusion.

[0039] Therefore, a new L-MP module is introduced in the downsampling module of the YOLOv7 network. On the basis of the original structure, the L-MP module adds a new branch containing the LiftDown Pool module, and passes the output of LiftDownPool and Maxpool to the subsequent network layers through skip connections. Finally, the results of the three branches are summarized through the cat operation and output to the subsequent layers, such as Figure 2 shown.

[0040] Regarding LiftDown Pool, taking a one-dimensional signal as an example, LiftDown Pool is used for a given signal x = [x1, x2, x3, ..., x n ], where x n ∈R, decompose it, and get the downsampled approximate signal s and difference signal d according to formula (1).

[0041] s,d=F(x) (1)

[0042] in (here represents the function composition operator), including three main functions: segmentation (downsampling), prediction, and update.

[0043] Split function Divide the signal x into two groups with even and odd indices, that is, x e and x o , specifically: x e=[x2,x4,...,x 2k ] and x o =[x1,x3,...,x 2k+1 ].

[0044] Prediction function According to formula (2), using the even index signal x e To predict the odd index signal x o .

[0045] d = x o -P(x e ) (2)

[0046] Updated Features By adding the difference signal d to the even signal x e , and generate a smoothed signal s according to formula (3).

[0047] s=x e +U(d) (3)

[0048] This implementation combines the upsampling and prediction update processes, which can effectively retain the key features of the sit-up action in the downsampling stage, ensuring that the low-level features extracted early are retained, thereby avoiding the problems of gradient vanishing and feature attenuation, significantly improving the accuracy of key point detection, reducing the amount of calculation and accelerating the reasoning process. In terms of visualization, LiftDown Pool performs better than MaxPool on feature maps. LiftDown Pool not only retains richer structural information, but also effectively reduces the alias effect. In contrast, MaxPool often loses key information when processing details, especially in the extraction of small target features.

[0049] The cat C module contains the LiftDown Pool module, which is used to downsample feature maps, decompose different frequency information of the input feature map, and generate sub-band features with higher resolution, followed by a 1×1 convolution module, which is used to compress or convert channels to reduce feature dimensions and optimize computational efficiency; the cat C module directly passes the results of the LiftDownPool module to subsequent layers through skip connections to ensure that the low-level information of the feature map is retained and smoothly transmitted; finally, the outputs of the LiftDown Pool module and the 1×1 convolution module are fused through the cat operation, and the fused features are passed to subsequent modules. The cat C module fuses the features of different branches, integrates details and global feature information, and can enrich the expressiveness of the feature map.

[0050] Retain low-level feature information: Through the skip connection mechanism, the cat C module can avoid the attenuation of feature information in deep networks and ensure that low-level features are still influential in subsequent layers. The cat C module is a feature fusion structure that integrates LiftDownPool and 1×1 convolution. By integrating different features, retaining low-level information and optimizing computational efficiency, it improves the network's expressiveness in detecting small targets and complex scenes.

[0051] The cat B module is a feature fusion module, which is designed to integrate feature maps of different branches to enhance the sensitivity to posture changes and the ability to retain key features. The second branch in the cat B module contains the Maxpool module, which is responsible for downsampling the feature map and extracting more abstract posture features. A 1×1 convolution module is connected after the Maxpool branch to compress the number of channels, reduce the computational burden and optimize the feature dimension. The cat B module also contains a third branch, in which four 1×1 convolution modules are connected in series to further optimize and simplify the feature expression. The output of the Maxpool branch and the third branch are fused through the cat operation to form the output feature map of the cat B module. In the image feature extraction of sit-ups, the angle changes of the shoulders, waist and legs are the key to judging the accuracy of the action. The cat B module captures the posture information of a larger structure by fusing the features of the Maxpool branch, and combines the convolution features in the third branch to effectively fuse feature information of different scales, thereby improving the recognition accuracy of the posture. The 1×1 convolution module reduces the number of feature channels, so that the cat B module reduces the amount of calculation when fusing information, thereby improving the calculation efficiency. For real-time detection of sit-ups, cat The efficiency optimization of the B module enables the network to process image frames in a shorter time and improve the response speed of detection.

[0052] Using the cat C module together with the cat B module and finally fusing the output can achieve enhanced feature diversity, comprehensive integration of details and global information, and higher detection accuracy and stability. The cat C module focuses on the extraction of detail features and the retention of low-level features, while the cat B module focuses on the fusion of abstract features and multi-scale information. Fusion of the outputs of these two modules can capture the complete features of details and global structures in sit-up images at the same time. For example, in the detection of sit-up actions, the combined use of cat C and cat B can simultaneously obtain subtle movement changes in key parts such as shoulders and waists as well as the overall posture structure, thereby being more accurate in identifying different posture states such as standing up and lying down. When using cat C or catB modules alone, although both have skip connection structures, their feature types are different, and their individual effects will be limited in information transmission. After combining the two, the feature fusion structure of cat C and cat B can avoid the loss of key features in the downsampling process and retain the effective information of the input image to a greater extent, thereby enhancing the feature transfer efficiency of the network; the joint use of cat C and cat B reduces the overall computational burden through the optimization of 1×1 convolution. Cat C retains low-level features, and cat B fuses information and reduces redundancy; the combined use of the two makes it possible to process image data more efficiently while improving accuracy, thereby meeting real-time requirements.

[0053] The L-MP module specifically includes the following operations:

[0054] 1. Input feature map representation

[0055] Let the input feature map be F input , which contains the feature data of key parts such as the head, torso, and legs in the sit-up image.

[0056] 2. Branch definition in L-MP module

[0057] The first branch (LiftDown Pool branch): contains the LiftDown Pool module and the 1×1 convolution module in series. The LiftDown Pool operation can preserve the key details in the sit-up image, such as the slight displacement between the head and shoulders. Let the LiftDown Pool operation be LiftDownPool(·), and the 1×1 convolution operation be Conv 1×1 (·), then the output of the first branch is calculated according to formula (4).

[0058] F branch1 =Conv 1×1 (LiftDownPool(F input )) (4)

[0059] The second branch (Maxpool branch): contains the Maxpool module and the 1×1 convolution module in series. The Maxpool branch is used to extract the action structure features of a relatively large range, such as the movement trajectory of the trunk and legs during sit-ups. Assuming the Maxpool operation is MaxPool(·), the output of the second branch is calculated according to formula (5).

[0060] F branch2 =Conv 1×1 (MaxPool(F input )) (5)

[0061] The third branch (four-layer 1×1 convolution module): contains four 1×1 convolution layers in series, which is used to further compress and optimize the channels of the input feature map, reduce the computational burden, and is suitable for real-time detection of sit-up action feature extraction. The combined operation of the four 1×1 convolution layers is Then the output of the third branch is calculated according to formula (6).

[0062]

[0063] 3. Skip Connections

[0064] cat C module: output F of LiftDown Pool branch branch1 The input is sent to the cat C module through a skip connection to enhance the retention of detail features and ensure that the position information of key parts (such as head and shoulders) is not lost in subsequent layers. The output of the cat C module is calculated according to formula (7).

[0065]

[0066] cat B module: output F of Maxpool branch branch2 and the output F of the third branch branch3 The connection is made through the cat operation, which integrates the large-scale features of the action (such as the movement of the torso and legs). The output of the cat B module is calculated according to formula (8).

[0067]

[0068] 4. Final summary of the three-branch results

[0069] The outputs of the cat C module and the cat B module are finally summarized through the cat operation to generate the fused feature map F of the L-MP module L-MP , this feature map contains the details and overall action information of the sit-up process, expressed as formula (9).

[0070]

[0071] In summary, formula P keypoints =g(F L-MP ) expresses the final feature map F fused by the L-MP module L-MP The role in the subsequent key point detection module makes the final output key point P keypoints It can achieve both high accuracy and high efficiency in real-time sit-up detection.

[0072] 5. Add EDH-Conv module to YOLOv7 network to enhance the detection ability of small targets

[0073] In order to overcome the problem that the original YOLOv7 network will gradually converge its focus range during data flow and focus the field of view on the real detection area, resulting in the target human body in the sit-up counting task may occupy a relatively small area in the image, and the network is difficult to accurately identify and locate the human body posture from the image, resulting in false detection and missed detection. As the data flows in the network, the deep feature map may not be accurately mapped to the human body posture area in the original image, resulting in inaccurate posture estimation, this patent proposes to add an EDH-Conv module between the Backbone module and the Head module of YOLOv7, with a structure as follows Figure 3 As shown in the figure, the output of this module is used as the spatial weight distribution learned by the model to correct the features, thereby improving the detection ability of the model. While expanding the receptive field of the module, multi-scale context is combined to improve the detection ability of small targets.

[0074] Specifically, the EDH-Conv module consists of three parallel branches, Concat, and a 1×1 convolution. The three parallel branches are a first branch consisting of a 3×3 convolution with a stride of 2 and a 3×3 transposed convolution with a stride of 2, a second branch consisting of a 1×1 convolution, a 3×3 convolution, and a 3×3 convolution with a dilation rate of 3, and a third branch consisting of a 1×1 convolution and a 3×3 convolution with a dilation rate of 1. The first branch uses a convolution with a stride of 2 as an encoder and a transposed convolution with a stride of 2 as a decoder to verify whether the semantic segmentation is effective. The second branch adds a 3×3 standard convolution with a dilation rate of 3 to obtain basic features, concatenates the features of the third branch, and finally performs feature fusion and channel dimensionality reduction through a 1×1 standard convolution, so as to obtain a larger receptive field through a dilated convolution with different dilation rates, and then fuses the features of each branch to obtain accurate context information and increase the detection capability of small targets.

[0075] The calculation of the output feature map of the EDH-Conv module is as follows:

[0076] Assume that the input of the EDH-Conv module is I, and after passing through the first parallel branch, O1 is obtained. Assume that the operation function of the 3×3 convolution with a step size of 2 in the first branch is Conv1, and the operation function of the 3×3 transposed convolution with a step size of 2 is TConv1, then O1 is calculated according to formula (10).

[0077] O1=TConv1(Conv1(I)) (10)

[0078] The input I of the EDH-Conv module passes through the second branch to obtain O2. Suppose the operation function of the 1×1 convolution in the second branch is Conv2, the operation function of the 3×3 convolution is Conv3, and the operation function of the 3×3 convolution with a dilation rate of 3 is DConv3, then O2 is calculated according to formula (11).

[0079] O2=Conv2(I)+Conv3(I)+Dconv3(I) (11)

[0080] The input I of the EDH-Conv module passes through the third branch to obtain O3. Assume that the operation function of the 1×1 convolution in the third branch is Conv4, and the operation function of the 3×3 convolution with a dilation rate of 1 is DConv1, then O3 is calculated according to formula (12).

[0081] O3=Conv4(I)+DConv1(I) (12)

[0082] Finally, the outputs O1, O2, and O3 of the above three branches are used as inputs through a Concat and a 1×1 convolution to finally obtain O EDH , let the concatenation operation function be COncat, and the 1×1 convolution operation function be Conv5, then O EDH Calculate according to formula (13).

[0083] O EDH =Conv5(Concat(O1,o2,o3)) (13)

[0084] This completes the calculation of the output feature map of the EDH-Conv module.

[0085] Since the datasets provided by human posture estimation tasks such as sit-ups lack semantic labels, only the annotation information can be used to generate semantic masks, which are called weak semantic masks. The specific method is to input the original image, process the image through the semantic segmentation module, and generate an output matrix. For each pixel point on the output matrix, if the coordinates of the point fall within the pre-annotated coordinate box, the value of the point is set to 1, and if it is not in the box, it is set to 0. In order to enhance the clarity of the boundary, the pixel value falling on the boundary of the annotation box is set to 255. The generated pixel value map is mapped to a size that matches the Head module of the YOLOv7 network to obtain the final weak semantic mask. In training, the generated weak semantic mask is used as a supervisory signal to train the network. In the ED-Conv module, the output spatial attention weight reflects the degree of attention of the model to different regions, that is, the spatial attention weight.

[0086] With this approach, the model can be effectively trained to recognize and count human actions even in the absence of fine annotations. The weak semantic mask provides an indirect supervision signal, allowing the network to learn spatial features related to actions, thereby improving the accuracy of action recognition. In addition, the introduction of spatial attention weights enables the network to adaptively focus on action-related areas in the image, further improving the network's ability to detect small objects.

[0087] 6. Add RSwin Transformer encoder to YOLOv7 network to optimize feature extraction

[0088] This patent proposes to add RSwin Transformer encoder to the YOLOv7 network to optimize feature extraction. Specifically, a new RSwin Transformer module is added between the third and fourth CBS modules of the original YOLOv7 network to optimize multi-scale feature extraction and improve the modeling of long-distance dependencies, thereby improving the detection of small and overlapping targets in complex scenes. At the same time, the local perception advantage of the original YOLOv7 network is retained, and the accuracy and robustness of target detection are improved; wherein, RSwin Transformer consists of Convolutional Token Embeeding, Conv, RSwin-Transformer-Block, Concat layer, and Deconv module.

[0089] A new RSwinTransformer module is added between the third and fourth CBS modules in the backbone network of the original YOLOv7 network. Specifically, Figure 1As shown in the figure, the backbone network uses an RGB image of size ××3 as the input object, inputs it into the Convolutional Token Embedding layer for feature extraction and embedding mapping (the convolution kernel size is 7×7 and the step length is 4), and generates a preliminary Token embedding feature map; then it is input into four parallel stage structures for feature extraction, that is, multiple windows are input into four parallel stages at the same time, and feature extraction operations are performed in parallel. Each stage processes the input features independently, and through the operation of Conv→RSwin Transformer Block, it can better extract multi-scale feature maps. Among them, each stage first performs a convolution operation based on a convolution kernel size of 3×3 and a step length of 8 on the input features to reduce the number of tokens and double the channel dimension, and then further extracts local and global feature representations through RSwin Transformer Block. After completing the feature extraction of each stage, the outputs of the four stages are spliced ​​through the Concat layer to form a comprehensive feature representation, thereby improving the efficiency and accuracy of feature extraction without increasing too much computation. After the above Conv→RSwin Transformer Block→Concat operation, the size of the feature map is / 32× / 32×8, which is used to enhance the network's ability to detect small and overlapping targets in complex scenes. After the feature extraction is completed, the Deconv module is used to upsample the feature map three times. Each upsampling doubles the length and width of the feature map, and the number of channels is halved, retaining high-resolution information. The size of the feature map is / 32× / 32×64, thereby optimizing multi-scale feature extraction and improving the detection effect of small and overlapping targets in complex scenes.

[0090] The Convolutional Token Embedding layer is to convert the token map x of the input image into i-1 After convolution mapping to the new token map f(x i-1 ), the convolution kernel size is s×s, and the step size is so. Therefore, we can adjust the convolution parameters to flexibly control the dimension and number of token features, while reducing the length of the token sequence and increasing the feature dimension of the token, thereby achieving spatial downsampling and rich expression capabilities. The height H and width W of the new token map are calculated according to formula (14) and formula (15):

[0091]

[0092]

[0093] Among them, Hi-1 is the height of the old token map, w i-1 is the width of the old token map, p is the padding size, s is the convolution kernel size, and o is the overlap size.

[0094] Among them, RSwin-Transformer-Block is composed of multiple window multi-head self-attention (W-MSA) layers and moving window multi-head self-attention (SW-MSA) layers, residual multi-layer perceptron (Res-MLP) layers and layer normalization (LN) and other computing units through residual connections. The residual multi-layer perceptron (Res-MLP) layer replaces the original MLP module with the Res_MLP module, such as Figure 2 and Figure 3 As shown in the figure, the residual structure is realized by adding an input tensor as the identity branch between the input tensor and the first linear transformation, and adding an X+identity layer between the second Dropout layer and the output tensor for residual connection. On the basis of retaining the two linear transformations, activation functions, normalization and Dropout layers in the original MLP module, the Res_MLP module can directly pass the original information of the input tensor to the output, improving the feature expression ability and training stability, and alleviating the gradient disappearance problem.

[0095] Among them, the Deconv deconvolution module inserts zero-value pixels in the input feature map and convolves it to enlarge the feature map size to multiple times of the original size, so that the network can model on a larger receptive field to extract more local feature information. Specifically, the module contains six layers of convolution operations, namely: the first layer uses a 3×3 convolution kernel to reduce the number of channels to 256 and upsample by one, the second layer uses a 4×4 convolution kernel to keep the number of channels at 256 and continue upsampling; the third and fourth layers use 3×3 and 4×4 convolution kernels respectively to reduce the number of channels to 128 and further expand the feature map; the fifth and sixth layers of convolution use 3×3 and 4×4 convolution kernels to reduce the number of channels to 64 and complete the final upsampling. After three upsamplings, the length and width of the feature map increase to eight times the original, and finally output a high-resolution feature map with a size of ( / 32× / 32×64).

[0096] 7. Train the deep network based on the optimized loss function and implement key point detection

[0097] The loss function is changed from Wise-IoU LOSS to an improved key point detection loss based on L2 loss, and the dynamic weight is based on the degree of abnormality β i Calculate and introduce geometric constraints to accelerate convergence and improve detection accuracy;

[0098] Specifically, it is calculated according to formula (16):

[0099] L loss =L keypoint +γ·L geo (16)

[0100] Among them, the key point detection loss is calculated according to formula (17) and formula (18):

[0101]

[0102]

[0103] Dynamic weight r i Based on the degree of abnormality β i Calculate according to formula (19):

[0104]

[0105] The degree of abnormality is the ratio of the prediction error to the mean error, calculated according to formula (20):

[0106]

[0107] Geometric constraint loss L geo Calculate according to formula (21):

[0108]

[0109] Among them, p i and p j is the actual key point position, and is the predicted key point position, i and j are the joint point pairs.

[0110] A dynamic focusing mechanism is introduced to adjust the error weights of key points. The optimization process aims to minimize the interference of high-error and low-error key points on the model, thereby improving the overall detection accuracy and accelerating the convergence of the model.

[0111] Specifically, it is calculated according to formula (22):

[0112]

[0113] Among them, r i The dynamic weight β of each key point i It is the abnormality degree of the key point, which is used to dynamically adjust the optimization priority of each key point.

[0114] Compared with the prior art, the present invention has the following beneficial effects:

[0115] (1) In the present invention, the MP module in the original YOLOv7 network is replaced by an improved L-MP module, which has significant advantages over the traditional MP module. By introducing the LiftDown Pool branch, the retention of detail features and multi-scale information is enhanced, and the detail loss in the downsampling process is improved; the residual connection method is used to ensure the smooth transmission of the gradient and avoid the gradient disappearance, thereby improving the training stability and detection accuracy of the model; the 3×3 convolution is replaced by multiple 1×1 convolutions, which reduces the computational complexity, improves the computational efficiency while ensuring accuracy, and is more suitable for application scenarios with high real-time performance.

[0116] (2) Based on the original YOLOv7 network, the present invention adds an EDH-Conv module, namely, an Encoder-Decoder-Hole Convolutional module, between the Backbone module and the Head module of the YOLOv7 network. The EDH-Conv module consists of three different parallel branches, a Concat and a 1×1 convolution in series. The three parallel branches refer to that the first branch includes a 3×3 convolution with a step size of 2 and a 3×3 transposed convolution with a step size of 2, the second branch includes a 1×1 convolution, a 3×3 convolution and a 3×3 convolution with a dilation rate of 3, and the third branch includes a 1×1 convolution and a 3×3 convolution with a dilation rate of 1. The EDH-Conv module improves the detection capability of the YOLOv7 network for small targets, enhances the modeling of detail information at different levels, and improves the expression capability of high-level semantic features.

[0117] (3) The present invention adds an RSwin Transformer encoder to the original YOLOv7 network to optimize the feature extraction module, adds the RSwin Transformer encoder between the third and fourth CBS modules of the YOLOv7 network, and retains the original local perception mechanism. The self-attention mechanism of the RSwin Transformer encoder makes up for the shortcomings of the YOLOv7 network in insufficient capture of small targets and overlapping targets during feature extraction, improves the modeling ability of long-distance dependencies, thereby optimizing multi-scale feature extraction and improving the detection effect of small targets and overlapping targets in complex scenes.

[0118] (4) Based on the original YOLOv7 network, the present invention improves the loss function to a key point detection loss based on L2 loss with reference to Wise-IoU LOSS, and introduces a dynamic non-monotonic focusing mechanism and geometric constraints, which is not only suitable for key point detection tasks in posture analysis, but also can accurately adjust the weight distribution of high and low error key points, enhance the model's optimization of medium error key points, and reduce the negative impact of low-quality key points. At the same time, geometric constraints ensure the reasonable positional relationship between key points, making the detection results more consistent with the human body structure, thereby improving the detection accuracy, convergence speed and robustness of the model, and further improving the overall performance of the posture analysis system.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation methods of the present invention can still be modified or replaced without departing from the spirit and scope of the present invention. Any modification or replacement should be included in the scope of protection of the claims of the present invention.

Claims

1. An improved YOLOv7 human key point detection method, characterized in that An improved YOLOv7 network structure is constructed, the original MP module is replaced by an improved L-MP module, and an EDH-Conv module is added between the Backbone module and the Head module of YOLOv7, and the improved RSwin Transformer encoder is introduced between the third and fourth CBS modules of YOLOv7 to enhance the accuracy of human key point detection, including the following steps: (1) collecting image samples of middle school students doing sit-ups; (2) preprocessing and labeling the images; (3) preparing a training set based on data augmentation; (4) replacing the original MP module of YOLOv7 with the improved L-MP module to improve feature extraction; (5) adding an EDH-Conv module between the Backbone module and the Head module of the YOLOv7 network; (6) An improved RSwin Transformer encoder is introduced between the third and fourth CBS modules of YOLOv7; (7) A deep network is trained based on an optimized loss function, and key point detection is realized; the image samples of middle school students doing sit-ups are collected, which refers to collecting motion image samples under different environments, angles and distances, so that the image samples contain pictures with different illumination intensities and different human shooting angles; the image preprocessing and labeling refers to adjusting the size of the input image to 608×608 through the OpenCV library, and using OpenPose to label the key points of the human body; the training set preparation based on data augmentation refers to preparing a data set for training using Mosaic data augmentation, color perturbation and brightness adjustment methods; the key point detection based on loss function optimization refers to adaptively modifying the loss function with reference to Wise-IoU, adjusting it to an L2 loss suitable for key point detection, introducing a dynamic non-monotonic focusing mechanism for error weight allocation of key points, and adding geometric constraints to ensure a reasonable positional relationship between key points.

2. An improved YOLOv7 human key point detection method according to claim 1, characterized in that: The improved L-MP module replaces the original MP module of YOLOv7 to improve feature extraction, which means replacing the original MP module of YOLOv7 with the improved L-MP module. The improved L-MP module refers to a new module obtained by adding a branch including the LiftDownPool module on the basis of the MP module, outputting the outputs of the LiftDown Pool module and the Maxpool module to the subsequent module through a skip connection method, and then aggregating the results of the three branches through a cat method, and finally modifying the convolution method of the third branch; the newly added branch including the LiftDown Pool module refers to a branch formed by connecting the LiftDown Pool module and the 1×1 convolution module in series; the output of the LiftDown Pool module and the Maxpool module to the subsequent module through a skip connection method refers to inputting the result of the LiftDown Pool module into the cat C module through a skip connection method, and the cat C module refers to the module obtained by connecting the LiftDown Pool module and the 1×1 convolution module of the first branch in series; at the same time, the result of the Maxpool module is input into the cat B module through a skip connection method, and the cat The B module refers to the Maxpool module of the second branch and the 1×1 convolution module of the second branch, which are connected in series with the four 1×1 convolution modules in the third branch and then aggregated in cat mode; the jump connection method allows the input features to be directly passed to the subsequent layers, ensuring that the low-level features extracted early are retained, effectively avoiding gradient disappearance and feature attenuation; the results of the three branches are aggregated in cat mode, which means that the cat C module containing the results of the first branch and the cat B module containing the results of the second branch and the third branch are aggregated and output in series in cat mode; by combining LiftDownPool and super downsampling technology, the complementary downsampling feature map obtained can retain more key information in the downsampling process and avoid the loss of key features; the convolution method of modifying the third branch refers to modifying the original 1×1 convolution module and a 3×3 convolution module in the third branch to four 1×1 convolution modules in series, reducing the number of parameters in each layer, thereby speeding up the calculation speed.

3. An improved YOLOv7 human key point detection method according to claim 1, characterized in that: The added EDH-Conv module, i.e., the Encoder-Decoder-Hole Convolutional module, refers to adding a new EDH-Conv module between the Backbone module and the Head module of the YOLOv7 network; the EDH-Conv module refers to using the outputs of the two CBS modules in the YOLOv7 network that are connected to the backbone network and the output of the ELAN module in the backbone network as the three inputs of the EDH-Conv module, and then the three inputs are successively distributed to three different parallel branches, and then the feature maps of the three branches are serially aggregated into a total feature map, and the feature map is subjected to a 1×1 convolution to obtain the final feature map output; the three branches refer to that the first branch includes The first branch consists of a 3×3 convolution with a stride of 2 and a 3×3 transposed convolution with a stride of 2; the second branch consists of a 1×1 convolution, a 3×3 convolution and a 3×3 convolution with a dilation rate of 3 to expand the receptive field and obtain richer contextual information; the third branch includes a 1×1 convolution and a 3×3 convolution with a dilation rate of 1; the feature maps obtained by the above three branches are serially aggregated through a Concat and a 1×1 convolution to obtain the final feature map output; the EDH-Conv module improves the YOLOv7 network's ability to detect small targets, enhances the modeling of detail information at different levels, and improves the expression ability of high-level semantic features.

4. An improved YOLOv7 human key point detection method according to claim 1, characterized in that: The improved RSwin Transformer encoder refers to adding a RSwin Transformer module between the third and fourth CBS modules of YOLOv7; the RSwin Transformer module refers to using the output of the CBS module in the YOLOv7 network as the input of the RSwin Transformer module, and then passing through the Convolutional Token Embedding module, the input features are generated into tokens and then distributed to four parallel stage modules, the output feature maps of the four parallel stage modules are concat to obtain the total feature map, and then the deconvolution Deconv module is upsampled and the Head module is adjusted for regression to extract more accurate features; the stage module refers to a module composed of a LiftDownPool layer and an RSwin Transformer Block module, the LiftDownPool layer extracts the low-frequency and high-frequency information of the input feature map at the beginning of the stage, and then inputs it into the RSwin Transformer Block module; the RSwin Transformer Block module refers to the SwinTranformer The multi-layer perceptron MLP in Block is replaced by the residual-based Res-MLP, which includes two layers of fully connected networks and directly passes the input to the output through residual connections to reduce the gradient vanishing problem and improve the training efficiency of the model; the feature maps of the above four parallel stage modules are aggregated and concat to obtain a complete feature map; the RSwin Transformer module improves the multi-scale feature extraction capability of the TOLOv7 network, improves the modeling of long-distance dependencies, and improves the detection of small and overlapping targets in complex scenes.

Images