An Improved YOLOv7 Human Keypoint Detection Method

Abstract

This invention discloses an improved YOLOv7 human keypoint detection method. This method obtains rich training samples through manual annotation using LabelImages; it replaces the original MP module in YOLOv7 with an L-MP module to improve feature extraction, and adds a LiftDown Pool branch and residual connections; it adds an EDH-Conv module between the Backbone and Head modules of the YOLOv7 network, using its output as a spatial weight distribution to correct activation regions, further improving the network's ability to detect small targets; it adds an improved RSwin Transformer encoder to improve the detection performance of small and overlapping targets in complex scenes; it changes the loss function from Wise-IoU LOSS to an improved keypoint detection loss based on L2 loss, and introduces a dynamic focusing mechanism to adjust keypoint error weights, accelerating model convergence. This method enhances the ability to detect and recognize human keypoints in complex scenes, and has stronger robustness, accuracy, and efficiency for human keypoint detection, recognition, and counting tasks in sports.

Description

Technical Field

[0001] This invention belongs to the field of human key point detection and recognition, specifically, it relates to an improved human key point detection method based on YOLOv7. Background Technology

[0002] Human keypoint detection and recognition is an important research direction in the field of computer vision. In sports, it is a crucial aspect of motion analysis and evaluation. With the increasing popularity of physical education and the continuous improvement of competitive levels, judging the standardization of movements and counting standardized actions has become increasingly important. Against this backdrop, human keypoint detection and recognition plays a vital role in judging the standardization of movements. Therefore, utilizing computer vision technology for human keypoint detection, and subsequently achieving the recognition of standardized movements, has become a research hotspot in the fields of physical education and competitive sports.

[0003] With the standardization and competitiveness of sports, traditional analysis methods often rely on manual observation and recording. This approach suffers from inconsistencies, high labor costs and training expenses, and potential errors in judging movement accuracy and counting by staff, thus affecting the fairness and standardization of sports. In recent years, the widespread application of deep learning technology in the field of human keypoint detection and recognition has greatly improved the accuracy and efficiency of human keypoint detection and recognition, and reduced the labor costs and subjective influence of traditional analysis methods. However, in practical applications under natural environments, due to differences in lighting intensity, shooting angle, and distance, YOLOv7 still has shortcomings in human keypoint detection and recognition.

[0004] For object detection, ZENG et al. (ZENG LM, HOU J, CHEN ZR, et al. Lightweight traffic sign detection network based on weak semantic segmentation[J]. Computer Engineering, 2022, 48(9): 269-276, 285.) proposed a method based on weak semantic segmentation to improve the slow speed and low accuracy of object detection at high resolution input. They designed a lightweight improved network based on YOLOv4, using the improved MobileNetv3-Lite as the basic semantic feature extraction network, and using a combination of FPN and PAN to achieve dual feature fusion. They also designed a dynamic enhancement attachment based on weak semantic segmentation for the detection head without increasing the floating-point operations of the model. In the training process of practical application, it has a significant advantage in comprehensive performance in object detection. For feature upsampling, Zhao et al. (Zhao J, Snoek CG M. Liftpool: Bidirectional convnet pooling[J]. arXiv preprint arXiv:2104.00996,2021.) proposed a bidirectional pooling layer with lifting, including downpooling and lifting, which can perform downpooling and uppooling simultaneously. This improves the accuracy and robustness of image classification, generates more refined upsampled feature maps, and improves the ability to handle fine-grained image classification, super-resolution challenges, or tasks with high detail preservation requirements. Regarding visual transformers, Liu et al. (Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2021:10012-10022.) proposed an improved SwinTransformer, which uses a hierarchical transformer and shifted windows to solve the language-to-vision difference. The shift-window approach restricts attention computation to non-overlapping local windows while allowing cross-window connections, achieving higher efficiency. Furthermore, it is demonstrated that the hierarchical design and shift-window method are beneficial for full MLP architectures.Regarding the loss function, Tong et al. (Tong Z, Chen Y, Xu Z, et al. Wise-IoU: boundingbox regression loss with dynamic focusing mechanism[J]. arXiv preprint arXiv:2301.10051,2023.) proposed a loss method based on IoU dynamic non-monotonic FM, and reduced the competitiveness of high-quality anchor boxes by using a gradient gain allocation strategy, thereby reducing the generation of harmful gradients from low-quality examples and improving the overall performance of the detector.

[0005] In conclusion, while YOLOv7 has made some progress in human keypoint detection and recognition technology, further improvements are needed to better meet the demands of motion standardization judgment and motion counting in the sports field. This will bring more standardized and intelligent operation and management to sports education and competitive sports, promoting their development and improvement. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of the existing technology and proposes an improved YOLOv7 human key point detection method to improve the recognition efficiency, robustness and accuracy of sports detection and recognition counting in complex scenarios.

[0007] To achieve the above-mentioned invention, this invention provides an improved YOLOv7 human keypoint detection method, comprising the following steps:

[0008] (1) Collect sit-up image samples;

[0009] (2) Preprocess and label the image;

[0010] (3) Training set creation based on data augmentation;

[0011] (4) Replace the original MP module in YOLOv7 with the L-MP module to improve feature extraction;

[0012] (5) Add an EDH-Conv module between the Backbone module and the Head module in the YOLOv7 network;

[0013] (6) Add RSwin Transformer encoder to YOLOv7 network to optimize feature extraction;

[0014] (7) Train a deep network based on an optimized loss function and implement key point detection.

[0015] Furthermore, the collection of image samples of middle school students doing sit-ups refers to collecting motion image samples under different environments, angles, and distances, so that the image samples include pictures with different lighting intensities and different human body shooting angles; the image preprocessing and annotation refers to adjusting the size of the input image to a uniform 608×608 using the OpenCV library and using OpenPose to annotate human body key points; the creation of the training set based on data augmentation refers to creating a training dataset using Mosaic data augmentation, color perturbation, and brightness adjustment methods; the key point detection based on loss function optimization refers to adaptively modifying the loss function with reference to Wise-IoU LOSS, adjusting it to L2 loss suitable for key point detection, introducing a dynamic non-monotonic focusing mechanism for error weight allocation of key points, and adding geometric constraints to ensure reasonable positional relationships between key points.

[0016] Furthermore, the improvement of feature extraction by replacing the original MP module in YOLOv7 with an L-MP module refers to replacing the original MP module in YOLOv7 with an improved L-MP module. The improved L-MP module is a new module obtained by adding a branch containing a LiftDown Pool module on top of the MP module, outputting the outputs of the LiftDown Pool module and the Maxpool module to subsequent modules via skip connections, summarizing the results of the three branches using a cat method, and finally modifying the convolution method of the third branch. The added branch containing the LiftDown Pool module is a branch formed by concatenating the LiftDown Pool module and a 1×1 convolution module. The outputs of the LiftDown Pool module and the Maxpool module being output to subsequent modules via skip connections means that the result of the LiftDown Pool module is input to the cat C module via a skip connection. The cat C module is the module obtained by concatenating the LiftDown Pool module and the 1×1 convolution module of the first branch. Simultaneously, the result of the Maxpool module is input to the cat B module via a skip connection. Module B refers to the concatenation of the Maxpool module of the second branch and the 1×1 convolutional module of the second branch, followed by the concatenation of four 1×1 convolutional modules in the third branch, and then summed using a cat method. The skip connection method allows input features to be directly passed to subsequent layers, ensuring that early extracted low-level features are preserved, effectively avoiding gradient vanishing and feature decay. The summarization of the three branches using a cat method refers to concatenating the cat C module (containing the results of the first branch) with the cat B module (containing the results of the second and third branches) using a cat method and then summing the output. By combining LiftDownPool and super-downsampling techniques, the resulting complementary downsampled feature map can retain more key information during downsampling, avoiding the loss of key features. The modification of the convolution method in the third branch refers to changing the original concatenation of one 1×1 convolutional module and one 3×3 convolutional module in the third branch to four concatenated 1×1 convolutional modules, reducing the number of parameters in each layer and thus accelerating the computation speed.

[0017] Furthermore, the addition of an EDH-Conv module between the Backbone and Head modules of the YOLOv7 network refers to adding an EDH-Conv module between the Backbone and Head modules of the YOLOv7 network. The EDH-Conv module uses the outputs of two CBS modules connected to the backbone network and the output of the ELAN module in the backbone network as its three inputs. These three inputs are then sequentially distributed to three different parallel branches, and the characteristics of the three branches are then analyzed. Figure 1 The features obtained from the three branches are concatenated and summarized into a total feature map, which is then processed by a 1×1 convolution to obtain the final feature map output. The three branches are as follows: the first branch consists of a 3×3 convolution with a stride of 2 and a transposed 3×3 convolution with a stride of 2; the second branch consists of a 1×1 convolution, a 3×3 convolution, and a 3×3 convolution with a dilation of 3 to expand the receptive field and obtain richer contextual information; the third branch consists of a 1×1 convolution and a 3×3 convolution with a dilation of 1. The feature maps obtained from the above three branches are concatenated and summarized, processed by a Concat and a 1×1 convolution to obtain the final feature map output. The EDH-Conv module improves the YOLOv7 network's ability to detect small targets, enhances the modeling of detailed information at different levels, and improves the expressive power of high-level semantic features.

[0018] Furthermore, the addition of an RSwin Transformer encoder to optimize feature extraction in the YOLOv7 network refers to adding an RSwin Transformer module between the third and fourth CBS modules of YOLOv7. The RSwin Transformer module takes the output of the CBS module in the YOLOv7 network as its input, then passes through a Convolutional Token Embedding module to generate tokens, which are then distributed to four parallel stage modules. The output feature maps of the four parallel stage modules are concatted to obtain a total feature map, which is then upsampled through a deconvolution (Deconv) module to extract more accurate features. Each stage module consists of a conv layer and an RSwin Transformer Block module. The conv layer reduces the number of tokens and doubles the channel dimension at the beginning of the stage before inputting into the RSwin Transformer Block module. The RSwin Transformer Block module refers to the RSwin Transformer... The Multilayer Perceptron (MLP) in the Block is replaced with a Res-MLP based on residuals. This Res-MLP consists of two fully connected layers and passes the input directly to the output through residual connections to reduce the gradient vanishing problem and improve the training efficiency of the model. The feature maps of the above four parallel stage modules are combined and concat to obtain a complete feature map. The RSwin Transformer module improves the multi-scale feature extraction capability of the YOLOv7 network, improves the modeling of long-distance dependencies, and enhances the detection performance of small and overlapping targets in complex scenes. Attached Figure Description

[0019] Figure 1 This is a flowchart of an embodiment of the present invention;

[0020] Figure 2 This refers to the L-MP module in the improved YOLOv7 of this invention.

[0021] Figure 3 This is a structural diagram of the EDH-Conv module according to an embodiment of the present invention;

[0022] Figure 4 This is a structural diagram of the RSWin Transformer encoder in the improved YOLOv7 according to an embodiment of the present invention;

[0023] Figure 5 This is a structural diagram of the RSwin Tramsformer Block according to an embodiment of the present invention;

[0024] Figure 6 This is a structural diagram of the Res_MLP module in an embodiment of the present invention. Detailed Implementation

[0025] The technical approach of the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0026] This invention is an embodiment of an improved YOLOv7 human keypoint detection method, comprising the following steps:

[0027] 1. Collect sit-up image samples

[0028] Collect sit-up image samples from different environments, angles, and distances to ensure that the image samples include pictures with different brightness, complex backgrounds, clothing obscuring the image, multiple angles of rotation and tilt, and different postures.

[0029] 2. Image preprocessing and annotation.

[0030] The input image size was adjusted to 608×608 using the OpenCV library. OpenPose was used to automatically extract key joints of the human body (such as head, torso, knees, arms, etc.) during sit-ups, generating a key point annotation training set for sit-up movements.

[0031] The image annotation stage employs a combination of automatic extraction and manual annotation correction. OpenPose is used for initial annotation, and LabelImage is used for manual correction of inaccurate annotations to improve the accuracy of annotation.

[0032] 3. Training set creation based on data augmentation

[0033] The training dataset was created using Mosaic data augmentation, color perturbation, and brightness adjustment techniques.

[0034] Mosaic data augmentation refers to combining multiple images into a single image by randomly scaling, cropping, arranging, and stitching them together. It simulates changes under different lighting conditions by randomly changing the RGB channel values ​​in the image and adjusting the brightness, thereby generating new image data. It also simulates motion occlusion problems in real-world scenes by adding occluders.

[0035] 4. Replace the original MP module in YOLOv7 with the improved L-MP module to enhance feature extraction.

[0036] To improve feature extraction capabilities, this patent proposes replacing the original MP module in YOLOv7 with an improved L-MP module to address the information loss issue during downsampling in the original YOLOv7 network.

[0037] In the downsampling module of YOLOv7, the original MP module consists of two branches, which are mainly used to implement downsampling. The first branch first goes through max pooling, which is used to downsample features and extract important information. Then, it adjusts the number of channels through a 1×1 convolution. The second branch first changes the number of channels through a 1×1 convolution, and then goes through a 3×3 convolution block with a stride of 2, which is also used for downsampling. Finally, the results of the two branches are combined by addition to form a feature map that has been super-downsampled.

[0038] Traditional max pooling modules have several significant drawbacks: First, the single max pooling method is prone to losing feature information of small targets, especially when processing smaller targets; second, this design cannot effectively preserve low-level features, thus affecting the effect of subsequent feature fusion.

[0039] Therefore, a new L-MP module was introduced into the downsampling module of the YOLOv7 network. The L-MP module adds a new branch containing a LiftDown Pool module to the original structure, and passes the outputs of LiftDownPool and Maxpool to subsequent network layers via skip connections. Finally, the results from the three branches are summarized using a cat operation and output to subsequent layers, such as... Figure 2 As shown.

[0040] Regarding LiftDown Pooling, taking a one-dimensional signal as an example, LiftDown Pooling is used for a given signal x = [x1, x2, x3, ..., x...]. n ], where x n ∈R, decompose it, and obtain the approximate downsampled signal s and the difference signal d according to formula (1).

[0041] s,d=F(x) (1)

[0042] in (here The function composition operator (representing the combination of functions) includes three main functions: segmentation (downsampling), prediction, and update.

[0043] Segmentation function Divide the signal x into two groups: even-indexed and odd-indexed, i.e., x e and x o Specifically: x e=[x2,x4,...,x 2k ] and x o =[x1,x3,...,x 2k+1 ].

[0044] Predictive function Using formula (2) with even index signal x e To predict odd-indexed signals x o .

[0045] d = x o -P(x e (2)

[0046] Update features By adding the difference signal d to the even signal x e A smooth signal s is generated according to formula (3).

[0047] s = x e +U(d) (3)

[0048] This implementation combines upsampling and prediction update processes, effectively preserving key features in sit-up movements during the downsampling stage. It ensures that low-level features extracted early are retained, thus avoiding gradient vanishing and feature decay problems, significantly improving the accuracy of keypoint detection, reducing computation, and accelerating the inference process. In terms of visualization, LiftDown Pool outperforms MaxPool on feature maps. LiftDown Pool not only preserves richer structural information but also effectively reduces aliasing effects. In contrast, MaxPool often loses key information when processing details, especially in the extraction of small target features.

[0049] The `cat C` module includes a LiftDown Pool module, which downsamples the feature map, decomposes the different frequency information of the input feature map, and generates more resolution sub-band features. This is followed by a 1×1 convolutional module to compress or transform the channels, reducing feature dimensionality and optimizing computational efficiency. The `cat C` module directly passes the results of the LiftDown Pool module to subsequent layers through skip connections, ensuring that low-level information of the feature map is preserved and successfully passed. Finally, the outputs of the LiftDown Pool module and the 1×1 convolutional module are fused using the `cat` operation, and this fused feature is passed to subsequent modules. The `cat C` module fuses features from different branches, integrating detailed and global feature information, thus enriching the expressiveness of the feature map.

[0050] Preserving low-level feature information: Through the skip connection mechanism, the cat C module can avoid feature information decay in deep networks and ensure that low-level features still have influence in subsequent layers. The cat C module is a feature fusion structure that combines LiftDownPool and 1×1 convolution. By integrating different features, preserving low-level information and optimizing computational efficiency, it improves the network's performance in detecting small targets and complex scenes.

[0051] The cat B module is a feature fusion module designed to integrate feature maps from different branches to enhance sensitivity to pose changes and the retention of key features. The second branch of the cat B module includes a Maxpool module, responsible for downsampling the feature map and extracting more abstract pose features. A 1×1 convolutional module is connected after the Maxpool branch to compress the number of channels, reducing computational burden and optimizing feature dimensions. The cat B module also includes a third branch, with four 1×1 convolutional modules cascaded for further optimization and simplification of feature representation. The outputs of the Maxpool branch and the third branch are fused using the cat operation to form the output feature map of the cat B module. In sit-up image feature extraction, the angle changes of the shoulders, waist, and legs are crucial for judging the accuracy of the movement. The cat B module captures pose information of larger structures by fusing features from the Maxpool branch and combines them with the convolutional features in the third branch, effectively fusing feature information of different scales and improving the accuracy of pose recognition. The 1×1 convolutional module reduces the number of feature channels, thus reducing the computational load during information fusion and improving computational efficiency. For real-time detection of sit-ups, cat... The efficiency optimization of module B enables the network to process image frames in a shorter time, improving the detection response speed.

[0052] Using the cat C and cat B modules together and fusing them in the final output can achieve enhanced feature diversity, comprehensive integration of detailed and global information, and higher detection accuracy and stability. The cat C module focuses on extracting detailed features and preserving low-level features, while the cat B module focuses on fusing abstract features with multi-scale information. Fusing the outputs of these two modules can simultaneously capture the complete features of details and global structure in sit-up images. For example, in the detection of sit-up movements, using cat C and cat B together can simultaneously obtain subtle changes in the movements of key parts such as the shoulders and waist, as well as the overall posture structure, thus making it more accurate in recognizing different posture states such as getting up and lying down. When using cat C or cat B modules alone, although they both have skip connection structures, their feature types are different, and their individual use will be limited in terms of information transmission. When combined, the feature fusion structure of cat C and cat B can avoid losing key features during downsampling and retain more effective information of the input image, thereby enhancing the feature transfer efficiency of the network. The joint use of cat C and cat B reduces the overall computational burden through the optimization of 1×1 convolution. cat C retains low-level features, while cat B fuses information and reduces redundancy. The combined use of the two enables more efficient processing of image data while improving accuracy, thus meeting real-time requirements.

[0053] The L-MP module specifically includes the following operations:

[0054] 1. Input feature map representation

[0055] Let the input feature map be F input It includes feature data of key parts such as the head, torso, and legs in sit-up images.

[0056] 2. Branch definitions in L-MP modules

[0057] The first branch (LiftDown Pool branch) contains a LiftDown Pool module and a 1×1 convolution module connected in series. The LiftDown Pool operation can preserve key details in sit-up images, such as subtle displacements between the head and shoulders. Let the LiftDown Pool operation be LiftDownPool(·), and the 1×1 convolution operation be Conv. 1×1 If (·), then the output of the first branch is calculated according to formula (4).

[0058] F branch1 =Conv 1×1 (LiftDownPool(F input (4)

[0059] The second branch (Maxpool branch): contains the Maxpool module and the 1×1 convolution module connected in series. The Maxpool branch is used to extract relatively large range of motion structure features, such as the torso and leg movement trajectories during sit-ups. If the Maxpool operation is MaxPool(·), then the output of the second branch is calculated according to formula (5).

[0060] F branch2 =Conv 1×1 (MaxPool(F input (5)

[0061] The third branch (four-layer 1×1 convolutional module): contains the cascaded operation of four 1×1 convolutional layers, used to further compress and optimize the channels of the input feature map, reducing computational burden. It is suitable for feature extraction in real-time detection of sit-up movements. Let the combined operation of the four 1×1 convolutional layers be... The output of the third branch is calculated according to formula (6).

[0062]

[0063] 3. Jump connection

[0064] cat C module: This will output the LiftDown Pool branch F... branch1 By skipping connections to the cat C module, the preservation of detailed features is enhanced, ensuring that the positional information of key parts (such as the head and shoulders) is not lost in subsequent layers. The output of the cat C module is calculated according to formula (7).

[0065]

[0066] The `cat B` module: This will output the `F` branch of the Maxpool branch. branch2 and the output F of the third branch branch3 The cat operation is used to connect and integrate the large-scale features of the motion (such as the movement of the torso and legs). The output of the cat B module is calculated according to formula (8).

[0067]

[0068] 4. Final summary of the three-branch results

[0069] The outputs of the cat C and cat B modules are finally summarized using the cat operation to generate the fused feature map F of the L-MP module. L-MP This feature map contains details and overall motion information during the sit-up process, and is represented by formula (9).

[0070]

[0071] In summary, formula P keypoints =g(F L-MP The final feature map F, fused by the L-MP module, represents the result. L-MP Its role in the subsequent keypoint detection module is to ensure that the final output keypoint P keypoints It can achieve both high accuracy and high efficiency in real-time sit-up detection.

[0072] 5. Add the EDH-Conv module to the YOLOv7 network to enhance the detection capability for small targets.

[0073] To overcome the problem of false positives and false negatives in sit-up counting tasks caused by the original YOLOv7 network gradually converging its focus on the actual detection area during data flow, which makes it difficult for the network to accurately identify and locate the human posture in the image due to the relatively small area occupied by the target human body, this patent proposes to add an EDH-Conv module between the Backbone and Head modules of YOLOv7, as shown in the structure below. This addresses the issue that as data flows through the network, deep feature maps may not accurately map to the human posture region in the original image, leading to inaccurate posture estimation. Figure 3 As shown, the output of this module is used as the spatial weight distribution learned by the model to correct features, thereby improving the model's detection capability. While expanding the receptive field of the module, it is combined with multi-scale context to improve the detection capability of small targets.

[0074] Specifically, the EDH-Conv module consists of three parallel branches, a concatenation layer, and a 1×1 convolution. The three parallel branches are: a first branch consisting of a 3×3 convolution with a stride of 2 and a transposed 3×3 convolution with a stride of 2; a second branch consisting of a 1×1 convolution, a 3×3 convolution, and a 3×3 convolution with a dilation rate of 3; and a third branch consisting of a 1×1 convolution and a 3×3 convolution with a dilation rate of 1. The first branch uses a 2-stride convolution as the encoder and a 2-stride transposed convolution as the decoder to verify the effectiveness of semantic segmentation. The second branch adds a standard 3×3 convolution with a dilation rate of 3 to obtain basic features. The features from the third branch are concatenated, and finally, a standard 1×1 convolution is used for feature fusion and channel dimensionality reduction. This achieves a larger receptive field through dilated convolutions with different dilation rates. Furthermore, the features from each branch are fused to obtain accurate contextual information and increase the detection capability for small targets.

[0075] The EDH-Conv module output feature map is calculated as follows:

[0076] Let the input of the EDH-Conv module be I, and after passing through the first parallel branch, we get O1. Let the operation function of the 3×3 convolution with a stride of 2 in the first branch be Conv1, and the operation function of the 3×3 transpose convolution with a stride of 2 be TConv1. Then O1 is calculated according to formula (10).

[0077] O1=TConv1(Conv1(I)) (10)

[0078] The input I of the EDH-Conv module is obtained as O2 after passing through the second branch. Let the operation function of 1×1 convolution in the second branch be Conv2, the operation function of 3×3 convolution be Conv3, and the operation function of 3×3 convolution with an inflation rate of 3 be DConv3. Then O2 is calculated according to formula (11).

[0079] O2=Conv2(I)+Conv3(I)+Dconv3(I) (11)

[0080] The input I of the EDH-Conv module is obtained as O3 after passing through the third branch. Let the operation function of 1×1 convolution in the third branch be Conv4, and the operation function of 3×3 convolution with an expansion rate of 1 be DConv1. Then O3 is calculated according to formula (12).

[0081] O3=Conv4(I)+DConv1(I) (12)

[0082] Finally, the outputs O1, O2, and O3 from the above three branches are used as input, processed by a Concat and a 1×1 convolution, to obtain O. EDH Let the concatenation operation function be COncat, and the 1×1 convolution operation function be Conv5, then O EDH Calculate according to formula (13).

[0083] O EDH =Conv5(Concat(O1,o2,o3)) (13)

[0084] This completes the calculation of the output feature map of the EDH-Conv module.

[0085] Because datasets for human pose estimation tasks such as sit-ups lack semantic labels, semantic masks can only be generated using labeled information; these are called weak semantic masks. Specifically, the original image is input, processed by a semantic segmentation module, and an output matrix is ​​generated. For each pixel in the output matrix, if its coordinates fall within a pre-labeled bounding box, its value is set to 1; otherwise, it is set to 0. To enhance boundary clarity, pixels falling on the bounding box boundaries are set to 255. The generated pixel value map is mapped to a size matching the YOLOv7 network's Head module to obtain the final weak semantic mask. During training, the generated weak semantic mask is used as a supervisory signal to train the network. In the ED-Conv module, the output spatial attention weights reflect the model's attention to different regions; these are the spatial attention weights.

[0086] This method enables effective training of models to recognize and count human actions, even in the absence of fine annotations. Weak semantic masks provide an indirect supervisory signal, allowing the network to learn action-related spatial features, thereby improving the accuracy of action recognition. Furthermore, the introduction of spatial attention weights allows the network to adaptively focus on action-related regions in the image, further enhancing its ability to detect small targets.

[0087] 6. Add the RSwin Transformer encoder to the YOLOv7 network to optimize feature extraction.

[0088] This patent proposes adding an RSwin Transformer encoder to the YOLOv7 network to optimize feature extraction. Specifically, an RSwin Transformer module is added between the third and fourth CBS modules of the original YOLOv7 network to optimize multi-scale feature extraction and improve the modeling of long-distance dependencies, thereby improving the detection performance of small and overlapping targets in complex scenes. At the same time, it retains the local perception advantage of the original YOLOv7 network, improving the accuracy and robustness of target detection. The RSwin Transformer consists of Convolutional Token Embeeding, Conv, RSwin-Transformer-Block, Concat layer, and Deconv module.

[0089] A new RSwinTransformer module is added between the third and fourth CBS modules in the original YOLOv7 network backbone. Specifically, as follows: Figure 1As shown, the backbone network uses a ××3 RGB image as input, which is fed into the Convolutional Token Embedding layer for feature extraction and embedding mapping (7×7 kernel size, stride 4) to generate a preliminary token embedding feature map. Then, it is input into a four-parallel stage structure for feature extraction, where multiple windows are simultaneously input into four parallel stages, performing feature extraction operations in parallel. Each stage processes the input features independently, using a Conv→RSwin Transformer Block operation to better extract multi-scale feature maps. Each stage first performs a convolution operation on the input features with a 3×3 kernel size and an 8-stride to reduce the number of tokens and double the channel dimension, then further extracts local and global feature representations through the RSwin Transformer Block. After completing their respective feature extractions, the outputs of the four stages are concatenated through a Concat layer to form a comprehensive feature representation, thereby improving the efficiency and accuracy of feature extraction without increasing computational load excessively. After the Conv→RSwin Transformer Block→Concat operation described above, the resulting feature map size is / 32× / 32×8, which enhances the network's ability to detect small and overlapping targets in complex scenes. After feature extraction, the Deconv module performs three upsampling operations on the feature map. Each upsampling doubles the width and height of the feature map while halving the number of channels, preserving high-resolution information. The resulting feature map size is / 32× / 32×64, thus optimizing multi-scale feature extraction and improving the detection performance of small and overlapping targets in complex scenes.

[0090] The Convolutional Token Embedding layer refers to embedding the token map of the input image into a single layer. i-1 After convolution, it is mapped to a new token map f(x). i-1 The convolution kernel size is s×s, and the stride is so. Therefore, we can adjust the convolution parameters to flexibly control the dimension and number of token features. While reducing the length of the token sequence, we can increase the feature dimension of the token, thereby achieving spatial downsampling and rich expressive power. The height H and width W of the new token map are calculated according to formulas (14) and (15):

[0091]

[0092]

[0093] Among them, Hi-1 It's the old token map that's higher, w i-1 is the width of the old token map, p is the padding size, s is the kernel size, and o is the overlap size.

[0094] The RSwin-Transformer-Block consists of multiple window multi-head self-attention (W-MSA) layers, moving window multi-head self-attention (SW-MSA) layers, residual multilayer perceptron (Res-MLP) layers, and layer normalization (LN) computational units connected by residuals. The Residual Multilayer Perceptron (Res-MLP) layer replaces the original MLP module with a Res_MLP module, such as... Figure 2 and Figure 3 As shown, the Res_MLP module achieves a residual structure by adding an input tensor as an identity branch between the input tensor and the first linear transformation, and by adding an X+identity layer between the second dropout layer and the output tensor for residual connection. While retaining the two linear transformations, activation functions, normalization, and dropout layers of the original MLP module, the Res_MLP module can directly pass the original information of the input tensor to the output, improving feature representation ability and training stability, and alleviating the gradient vanishing problem.

[0095] The Deconv deconvolution module enlarges the feature map size several times over by inserting zero-value pixels into the input feature map and then convolving it. This allows the network to model over a larger receptive field, extracting more local feature information. Specifically, the module contains six convolutional layers: the first layer uses a 3×3 kernel to reduce the number of channels to 256 and upsamples by half; the second layer uses a 4×4 kernel to maintain the number of channels at 256 and continues upsampling; the third and fourth layers use 3×3 and 4×4 kernels respectively to reduce the number of channels to 128 and further enlarge the feature map; the fifth and sixth layers use 3×3 and 4×4 kernels to reduce the number of channels to 64, completing the final upsampling. After three upsampling operations, the feature map's dimensions increase to eight times its original size, resulting in a high-resolution output feature map of size (32×64).

[0096] 7. Train a deep network based on an optimized loss function and implement keypoint detection.

[0097] The loss function is changed from Wise-IoU Loss to an improved keypoint detection loss based on L2 loss, with dynamic weights based on the anomaly level β. i The calculation is performed, and geometric constraints are introduced to accelerate convergence and improve detection accuracy;

[0098] Specifically, calculate according to formula (16):

[0099] L loss =L keypoint +γ·L geo (16)

[0100] The key point detection loss is calculated according to formulas (17) and (18):

[0101]

[0102]

[0103] Dynamic weight r i Based on the degree of anomaly β i Calculate according to formula (19):

[0104]

[0105] The degree of anomaly is the ratio of the prediction error to the mean error, calculated according to formula (20):

[0106]

[0107] Geometric constraint loss L geo Calculate according to formula (21):

[0108]

[0109] Where, p i and p j For the actual key point location, and Here, i and j represent the predicted keypoint locations, and i and j are keypoint pairs.

[0110] A dynamic focusing mechanism is introduced to adjust the error weights of key points. The optimization process aims to minimize the interference of high-error and low-error key points on the model, thereby improving the overall detection accuracy and accelerating the convergence of the model.

[0111] Specifically, calculate according to formula (22):

[0112]

[0113] Where, r i Dynamic weight β for each keypoint i The degree of anomaly at key points is used to dynamically adjust the optimization priority of each key point.

[0114] Compared with the prior art, the present invention has the following beneficial effects:

[0115] (1) In this invention, the MP module in the original YOLOv7 network is replaced with an improved L-MP module, which has significant advantages over the traditional MP module. By introducing the LiftDown Pool branch, the preservation of detailed features and multi-scale information is enhanced, and the loss of details during downsampling is improved. The residual connection method is used to ensure the smooth transmission of gradients and avoid gradient vanishing, thereby improving the training stability and detection accuracy of the model. The 3×3 convolution is replaced with multiple 1×1 convolutions, which reduces the computational complexity and improves the computational efficiency while ensuring accuracy, making it more suitable for real-time application scenarios.

[0116] (2) Based on the original YOLOv7 network, this invention adds an EDH-Conv module, namely the Encoder-Decoder-Hole Convolutional module, between the Backbone and Head modules of the YOLOv7 network. The EDH-Conv module consists of three different parallel branches, a Concat, and a 1×1 convolution in series. The three parallel branches are: the first branch consists of a 3×3 convolution with a stride of 2 and a 3×3 transposed convolution with a stride of 2; the second branch consists of a 1×1 convolution, a 3×3 convolution, and a 3×3 convolution with a dilation rate of 3; and the third branch consists of a 1×1 convolution and a 3×3 convolution with a dilation rate of 1. The EDH-Conv module improves the YOLOv7 network's ability to detect small targets, enhances the modeling of detailed information at different levels, and improves the ability to express high-level semantic features.

[0117] (3) In this invention, the RSwin Transformer encoder is added to the original YOLOv7 network to optimize the feature extraction module. The RSwin Transformer encoder is added between the third and fourth CBS modules of the YOLOv7 network, while retaining the original local perception mechanism. The self-attention mechanism of the RSwin Transformer encoder makes up for the shortcomings of the YOLOv7 network in capturing small and overlapping targets during feature extraction, improves the ability to model long-distance dependencies, thereby optimizing multi-scale feature extraction and improving the detection effect of small and overlapping targets in complex scenes.

[0118] (4) Based on the original YOLOv7 network, this invention improves the loss function to a keypoint detection loss based on L2 loss, referencing Wise-IoU LOSS, and introduces a dynamic non-monotonic focusing mechanism and geometric constraints. This not only applies to keypoint detection tasks in pose analysis but also precisely adjusts the weight allocation of high and low error keypoints, enhancing the model's optimization for medium error keypoints and reducing the negative impact of low-quality keypoints. Simultaneously, geometric constraints ensure reasonable positional relationships between keypoints, making the detection results more consistent with human anatomy, thereby improving the model's detection accuracy, convergence speed, and robustness, further enhancing the overall performance of the pose analysis system.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or substitutions can still be made to the specific implementation methods of the present invention without departing from the spirit and scope of the present invention. All such modifications or substitutions should be covered within the scope of protection of the claims of the present invention.

Claims

1. An improved YOLOv7 human keypoint detection method, characterized in that... An improved YOLOv7 network structure was constructed by replacing the original MP module with an improved L-MP module, adding an EDH-Conv module between the Backbone and Head modules of YOLOv7, and introducing an improved RSwin Transformer encoder between the third and fourth CBS modules of YOLOv7 to enhance the accuracy of human keypoint detection. The steps included: (1) collecting image samples of middle school students doing sit-ups; (2) preprocessing and labeling the images; (3) creating a training set based on data augmentation; (4) replacing the original MP module of YOLOv7 with an improved L-MP module to improve feature extraction; (5) adding an EDH-Conv module between the Backbone and Head modules of the YOLOv7 network; and (6) introducing an improved RSwin encoder between the third and fourth CBS modules of YOLOv7. Transformer encoder; (7) Training a deep network based on an optimized loss function and implementing key point detection; The collection of image samples of middle school students doing sit-ups refers to collecting motion image samples under different environments, angles and distances, so that the image samples contain pictures with different light intensities and different human shooting angles; The preprocessing and annotation of the images refers to adjusting the size of the input image to 608×608 using the OpenCV library and using OpenPose to annotate human key points; The production of the training set based on data augmentation refers to using Mosaic data augmentation, color perturbation and brightness adjustment methods to produce a dataset for training; The training of a deep network based on an optimized loss function and implementing key point detection refers to adapting the loss function to Wise-IoU and adjusting it to L2 loss suitable for key point detection, introducing a dynamic non-monotonic focusing mechanism for error weight allocation of key points, and adding geometric constraints to ensure reasonable positional relationships between key points.

2. The improved YOLOv7 human key point detection method according to claim 1, characterized in that, The improvement of feature extraction by replacing the original MP module of YOLOv7 with an improved L-MP module refers to replacing the original MP module of YOLOv7 with an improved L-MP module. The improved L-MP module is based on the MP module, adding a branch containing a LiftDown Pool module, and then using a skip connection to output the outputs of the LiftDown Pool and Maxpool modules to subsequent modules. The results of the three branches are then summarized using a cat method, and finally, the convolution method of the third branch is modified to obtain the new module. The added branch containing the LiftDown Pool module is a branch formed by concatenating the LiftDown Pool module and a 1×1 convolution module. The skip connection to output the outputs of the LiftDown Pool and Maxpool modules to subsequent modules means that the result of the LiftDown Pool module is input to the cat C module using a skip connection. The cat C module is the module obtained by concatenating the LiftDown Pool module and the 1×1 convolution module from the first branch. Simultaneously, the result of the Maxpool module is input to the cat B module using a skip connection. Module B refers to the concatenation of the Maxpool module of the second branch and the 1×1 convolutional module of the second branch, followed by the concatenation of four 1×1 convolutional modules in the third branch, and then summed using a cat method. The skip connection method allows input features to be directly passed to subsequent layers, ensuring that early extracted low-level features are preserved, effectively avoiding gradient vanishing and feature decay. The summarization of the three branches using a cat method refers to concatenating the cat C module (containing the results of the first branch) with the cat B module (containing the results of the second and third branches) using a cat method and then summing the output. By combining LiftDownPool and super-downsampling techniques, the resulting complementary downsampled feature map can retain more key information during downsampling, avoiding the loss of key features. The modification of the convolution method in the third branch refers to changing the original concatenation of one 1×1 convolutional module and one 3×3 convolutional module in the third branch to four concatenated 1×1 convolutional modules, reducing the number of parameters in each layer and thus accelerating the computation speed.

3. The improved YOLOv7 human key point detection method according to claim 1, characterized in that, The addition of the EDH-Conv module, or Encoder-Decoder-Hole Convolutional module, refers to adding an EDH-Conv module between the Backbone and Head modules of the YOLOv7 network. The EDH-Conv module takes the outputs of two CBS modules connected to the backbone network and the output of the ELAN module in the backbone network as its three inputs. These three inputs are then sequentially assigned to three different parallel branches. The feature maps from the three branches are concatenated to form a total feature map, which is then subjected to a 1×1 convolution to obtain the final feature map output. The three branches consist of: a first branch comprising a 3×3 convolution with a stride of 2 and a transposed 3×3 convolution with a stride of 2; and a second branch consisting of a 1×1 convolution, a 3×3 convolution, and a 3×3 convolution with a dilation of 3, to expand the receptive field and obtain richer contextual information. The third branch includes a 1×1 convolution and a 3×3 convolution with an inflation rate of 1. The feature maps obtained from the above three branches are concatenated and summarized through a Concat and a 1×1 convolution to obtain the final feature map output. The EDH-Conv module improves the YOLOv7 network's ability to detect small targets, enhances the modeling of detailed information at different levels, and improves the expressive power of high-level semantic features.

4. An improved YOLOv7 human key point detection method according to claim 1, characterized in that, The improved RSwin Transformer encoder refers to adding an RSwin Transformer module between the third and fourth CBS modules of YOLOv7. The RSwin Transformer module takes the output of the CBS module in the YOLOv7 network as its input, then passes through a Convolutional Token Embedding module to generate tokens from the input features, which are then distributed to four parallel stage modules. The output feature maps of the four parallel stage modules are concatted to obtain a total feature map, which is then upsampled by a deconvolution module and adjusted by a head module to extract more accurate features. Each stage module consists of a LiftDownPool layer and an RSwin Transformer Block module. The LiftDownPool layer extracts low-frequency and high-frequency information from the input feature map at the beginning of the stage before inputting it into the RSwin Transformer Block module. The RSwin Transformer Block module replaces the Multilayer Perceptron (MLP) in the RSwin Transformer Block with a residual-based Res-MLP. It includes two fully connected layers and passes the input directly to the output through residual connections to reduce the gradient vanishing problem and improve the training efficiency of the model; the feature maps of the above four parallel stage modules are combined and concat to obtain a complete feature map; the RSwin Transformer module improves the multi-scale feature extraction capability of the TOLOv7 network, improves the modeling of long-distance dependencies, and enhances the detection effect of small and overlapping targets in complex scenes.

Images