A high-resolution human key point detection method and system

Abstract

The application discloses a kind of high-resolution human key point detection method and system, method includes as follows:S1, selecting dataset;S2, the data set is preprocessed;S3, construct pyramid type high-resolution network, include shuffle module and feature fusion module;Wherein, shuffle module is composed of feature reservation branch and feature extraction branch, feature extraction branch includes PSA module and convolution layer;Feature fusion module first from high-resolution branch is fused to low-resolution branch, then from low-resolution branch high-resolution is fused, finally the result of twice fusion is added, as output;S4, using the data set after S2 pre-processing is trained to the human key point network of S3 construction;S5, the network that shows optimal performance in training is selected to carry out human key point prediction, after introducing super-resolution head heat map post-processing method, obtains key point position.The application can extract rich multi-scale spatial information, capture important features in cross-dimension channel attention vector.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a high-resolution human key point detection method and system. BACKGROUND

[0002] Human key point detection technology is widely used in sports, virtual reality, medical rehabilitation and motion posture correction fields. In recent years, the heat map based method has achieved remarkable results in human key point detection. Specifically, this method generates a heat map for each key point, and each heat map contains a two-dimensional Gaussian distribution centered on the true joint position, which can suppress false positives and smooth the training process. Therefore, the heat map based method has stronger generalization ability and is easier to optimize. However, due to the need to discretize continuous coordinate values into small scale heat maps, and the resolution limit of heat maps, there is a non-negligible quantization error. Due to the existence of quantization error, many heat map based methods perform poorly when directly decoding. Most heat map based methods rely heavily on further post-processing to alleviate the quantization error, but this post-processing is poorly optimized.

[0003] Studies have shown that high-resolution networks have great advantages in human key point detection. It connects the high-resolution to low-resolution convolutional stream in parallel; repeatedly exchanges information between layers, and the obtained features are more semantically rich and spatially accurate. At the same time, attention mechanism can well capture human key point information, and it is widely used in computer vision fields such as image classification, object detection, instance segmentation, semantic segmentation, scene parsing and action localization. There are two kinds of attention mechanisms: channel attention and spatial attention. Studies have shown that using channel attention, spatial attention or both can significantly improve performance. The most commonly used channel attention method is Squeeze-and-Excitation Networks (SENet), which can significantly improve performance at a relatively low cost. However, the disadvantage of SE-Net is that it ignores the importance of spatial information.

[0004] Therefore, some researchers have proposed Bottleneck Attention Module (BAM) and Convolutional Block Attention Module (CBAM) to effectively combine spatial attention and channel attention to enrich the attention map. However, there are still two important and challenging problems. One is how to effectively capture and utilize the spatial information of different scale feature maps to enrich the feature space. The second is that channel or spatial attention can only effectively capture local information, but cannot establish long-term channel dependency.

[0005] Correspondingly, other methods are also proposed in the industry to solve the above two problems, such as a method based on multi-scale feature representation and cross-channel information interaction (PyConv, Res2Net, HS-ResNet) is currently proposed. However, the above methods all bring high network complexity, and the network has a large calculation burden. SUMMARY

[0006] In view of the deficiencies of the existing human key point detection method in processing spatial information, establishing long-distance channel dependence, and the quantization error problem existing in the heat map post-processing, the present application proposes a high-resolution human key point detection method and system based on a pyramid attention (PyramidSqueeze Attention, PSA) module. The present application can extract rich multi-scale spatial information, and capture important features across dimensions in the channel attention vector.

[0007] The technical scheme of the present application is as follows:

[0008] A high-resolution human key point detection method, specifically comprising the following steps:

[0009] S1, selecting a data set containing multiple human images in natural scenes, and recording the coordinates of the human key points in each image.

[0010] S2, preprocessing the data set of step S1 before feature extraction.

[0011] S3, based on the HRNet network, a pyramid high-resolution network (Pyramid Squeeze Attention-High Resolution Network, PSA-HRNet) is constructed by combining a Shuffle-PSA module: the pyramid high-resolution network includes two main modules: a shuffle module and a feature fusion module. The shuffle module is composed of a feature reservation branch and a feature extraction branch, the feature reservation branch is a residual connection that reserves the original features, and the feature extraction branch includes a PSA module and a standard convolution layer, which can enable the network to learn diverse feature representations. The fusion process of the feature fusion module is first from the high-resolution branch to the low-resolution branch, then from the low-resolution branch to the high-resolution, and finally the results of the two fusion processes are added as the output of the feature fusion module.

[0012] S4, using the human key point data set preprocessed in step S2 to train the human key point network constructed in step S3.

[0013] S5, after the training in step S4 is completed, the network with the best performance in the training is selected to perform human key point prediction, and a super-resolution head (SR Head) heatmap post-processing method is introduced in the prediction to obtain low-error key point positions.

[0014] As a preferred solution, in step S1, each human body image is a three-channel color image, and the total number of key points of the human body is 17.

[0015] As a preferred solution, in step S1, there is only one human body object in each human body image.

[0016] As a preferred solution, in step S1, the 17 key points in the human body are left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0017] As a preferred solution, in step S2, the specific method of data preprocessing is to scale the width and height of the pedestrian detection frame in the training data to the corresponding scale at the target original ratio, and then perform corresponding padding. The pedestrians in the training data images are cropped using a fixed aspect ratio of 4:3, and then scaled to a fixed size of 256x192.

[0018] As a preferred solution, all the cropped images obtained above are randomly horizontally flipped with a probability of 0.5, randomly scaled with a scaling ratio between 0.65 and 1.35, and randomly rotated within a rotation angle range of -45° to +45°.

[0019] As a preferred solution, in step S3, the feature extraction branch includes a 1x1 convolution, a PSA module, a 1x1 convolution, and a channel shuffle connected in sequence.

[0020] As a preferred solution, in step S3, batch normalization (BN) and ReLU activation function operations are added after each convolution; and a batch normalization BN operation is added after the PSA module.

[0021] As a preferred solution, in step S3, the specific formula for the feature fusion module to fuse from the high-resolution branch to the low-resolution branch is:

[0022]

[0023] In the formula, n is the total number of branches; A i is the output result of the i-th branch before feature fusion; H iis the fusion result of the i-th branch when fusing from high resolution to low resolution; the F(a, b, H) function represents performing resolution conversion on the output result H of the a branch to realize fusion with the b branch, and a is a high resolution branch adjacent to b.

[0024] As a preferred solution, in step S3, the specific formula for the lightweight feature fusion module to fuse from the low resolution branch to the high resolution branch is:

[0025]

[0026] In the formula, n is the total number of branches; A i is the output result of the i-th branch before feature fusion; L i is the fusion result of the i-th branch when fusing from low resolution to high resolution; the F(a', b', L) function represents performing resolution conversion on the output result L of the a' branch to realize fusion with the b' branch, and a' is a low resolution branch adjacent to b'.

[0027] As a preferred solution, in step S3, the number of base channels of the Stage2, Stage3, and Stage4 branches of the lightweight human key point detection is set in an increasing manner, and the number of channels of the same branch is different in different stages.

[0028] Let the network learn different human key point information at different scales to improve the generalization ability of the network, design a multi-scale convolutional neural network structure, so that the network can learn different human key point information at different scales.

[0029] As a preferred solution, in step S4, the loss function used to calculate the key points is as follows:

[0030]

[0031] In the formula, H j (p) is the prediction result, indicating the probability of the key point j at p, is the real generated heat map, and whether the key point j is visible is indicated by v j , v j =0 when the key point j is invisible, and v j =1 when the key point j is visible. The convolutional neural network is trained using the above loss function.

[0032] As a preferred solution, in step S5, a light heat map post-processing method SR Head is introduced, which encodes the key points into key point embeddings through an encoder, and then decodes and splices, effectively reducing the quantization error. After realizing human key point prediction by using PSA-HRNet, a more accurate key point heat map can be obtained through the heat map decoding processing of SR Head. The SRPSA-HRNet network constructed by combining PSA-HRNet and SR Head not only significantly enhances the generalization ability of the network and reduces the quantization error.

[0033] In the present application, the PSA module improves the network performance by enhancing the multi-scale representation ability and long-distance channel dependence. At the same time, the present application introduces a light heat map post-processing method Super-Resolution Head, which first encodes various key points into key point embeddings through a key point encoder, then decodes them through a separate large-core convolution, calculates multiple heat maps about the data, and finally splices all the key point heat maps together through pixel shuffling Shuffle to obtain a high-resolution heat map of the data, thereby effectively reducing the quantization error.

[0034] The present application introduces a PSA module in the Shuffle Block module to replace the traditional 3x3 convolution block, thereby constructing a Shuffle-PSA (Shuffle-Pyramid Squeeze Attention) module, so that the network can learn diverse feature representations. On this basis, the Shuffle-PSA module is integrated into the HRNet high-resolution network to replace the redundant Basic Block module, thereby constructing a more efficient human key point detection network, namely a pyramid high-resolution network (Pyramid Squeeze Attention-High Resolution Network, PSA-HRNet).

[0035] After the heat map decoding processing of SR Head, a more accurate key point heat map can be obtained. The Super Resolution Pyramid Squeeze Attention-High Resolution Network (SRPSA-HRNet) constructed by combining PSA-HRNet and SR Head not only significantly enhances the generalization ability of the network and reduces the quantization error, but also exhibits flexible adaptability in diverse application scenarios, thereby promoting the continuous progress of key point detection technology.

[0036] The application also discloses a high-resolution human key point detection system for executing the method.

[0037] The data set selection module selects a data set, wherein the data set contains human body images and records coordinates of human body key points in the images.

[0038] The preprocessing module pre-processes the selected data set.

[0039] The network construction module constructs a pyramid type high-resolution network, which comprises a mixing module and a feature fusion module; wherein the mixing module is composed of a feature reservation branch and a feature extraction branch, and the feature extraction branch comprises a PSA module and a convolution layer; the feature fusion module firstly fuses from a high-resolution branch to a low-resolution branch, and then fuses from the low-resolution branch to the high-resolution branch, and finally adds the results of the two fusions as an output.

[0040] The network training module trains the human key point network constructed by the network construction module by using the pre-processed data set.

[0041] The detection module selects the network with the best performance in the training to predict human key points, and introduces a super-resolution head heat map post-processing method to obtain the key point positions.

[0042] Compared with the prior art, the application has the following characteristics:

[0043] The application collects data containing human body images and performs preprocessing operations to enhance the generalization ability of the network and improve the accuracy of the network.

[0044] The application proposes a PSA-HRNet network, which can extract rich multi-scale spatial information and capture important features across dimensions in channel attention, enhance multi-scale representation ability and long-distance channel dependence, enable the network to learn diverse feature representations, effectively capture frequently changing information relationships in the input feature map, and improve network performance.

[0045] The application introduces a lightweight heat map post-processing method SR Head, encodes various key points into key point embeddings through a key point encoder, decodes them through a single large-core convolution, calculates multiple heat maps about the data, and finally splices all the key point heat maps together through pixel shuffle to obtain the HR heat map of the data, effectively reducing the quantization error.

[0046] The application enables the same branch to have different channel numbers at different stages in the human key point detection network, which can enable the network to extract more valuable features as the network gets deeper, and the feature expression ability is stronger. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 A flowchart of a human key point detection method according to a preferred embodiment of the present application.

[0049] Figure 2 A schematic diagram of a predefined human key point template.

[0050] Figure 3 A network structure diagram of a human key point detection method according to a preferred embodiment of the present application.

[0051] Figure 4 A structure diagram of a pyramid attention module PSA.

[0052] Figure 5 A structure diagram of a pyramid shuffle module Shuffle-PSA.

[0053] Figure 6 A network diagram of a heat map post-processing method SR Head.

[0054] Figure 7 A block diagram of a human key point detection system according to a preferred embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] As Figure 1 described, the present embodiment provides a high-resolution human key point detection method based on a PSA module, and builds a PSA-HRNet network, which can extract rich multi-scale spatial information without significantly increasing the algorithm complexity, and capture important features across dimensions in channel attention. By enhancing the multi-scale representation ability and long-distance channel dependency, the network performance is improved. Specifically, the following steps are included:

[0057] S1, obtaining a human key point detection dataset;

[0058] This embodiment uses the COCO dataset, which contains more than 200,000 pictures and 250,000 human bodies marked with 17 key points, wherein the COCO train2017 training set contains 57K pictures and 150K human body images, and the COCO va12017 verification set contains 5000 pictures. The training set is used to train the network, and the verification set is used to evaluate the performance of the trained network.

[0059] In this embodiment, the number of human key points is 17, as shown in Table 1. Figure 2 Table 1 is a human key point structure table.

[0060] Table 1 Human key point structure

[0061] Number Meaning Number Meaning Number Meaning 1 Nose 2 Left eye 3 Right eye 4 Left ear 5 Right ear 6 Left shoulder 7 Right shoulder 8 Left elbow 9 Right elbow 10 Left wrist 11 Right wrist 12 Left hip 13 Right hip 14 Left knee 15 Right knee 16 Left ankle 17 Right ankle

[0062] S2, data preprocessing;

[0063] The obtained original data is preprocessed. First, for the COCO2017 dataset, the image resolution used in network training is 256x192, so the image needs to be processed accordingly. This process is not simply stretching the picture to 256x192, but keeping the original scale of the target, scaling to the appropriate size, and then filling as needed. The pedestrians in the dataset image are cropped using a fixed aspect ratio of 4:3, and the cropped image is scaled to a fixed size of 256x192. In addition, some enhancement measures are also implemented on the data, including random rotation, random scaling, random horizontal flip, and Half Body (there is a certain probability of cutting the target, only keeping the key points of the upper body or lower body), for example, the cropped image is randomly horizontally flipped with a probability of 0.5, randomly scaled by a scaling ratio of 0.65 to 1.35, and randomly rotated within a rotation angle range of-45°~+45°.

[0064] S3, network building;

[0065] HRNet has strong functions in semantic segmentation, human key point detection, target detection, picture classification, etc., and the present application aims to tap its potential in human key point detection. In this embodiment, improvements are made in the high-resolution network based on the PSA module. On the basis of the HRNet network, this embodiment constructs a Shuffle-PSA module to build a more powerful human key point detection network PSA-HRNet, as shown in Figure 3The network first down-samples using a convolution operation, then goes through a Layer 1 layer, followed by three transition layers and three feature extraction stages. After each transition layer, a new scale branch is added, thus constructing a multi-branch network structure. Each feature extraction stage continuously extracts feature information with the help of this multi-branch parallel structure, and fuses different resolution features from different branches to form a feature fusion module to obtain richer feature representations. Finally, the network takes the extracted features of the highest resolution branch as output.

[0066] The first layer (Layer 1) is four stacked Shuffle-PSA modules, which are a fusion of Shuffle modules and PSA modules. The structure of each module is as follows: the input information first passes through a Channel Split module, which divides the channels of the input image into two branches, a feature reservation branch and a feature extraction branch. The feature reservation branch is a residual connection that preserves the original features, and the feature extraction branch first passes through a 1x1 convolution kernel with a stride of 1, a BN (Batch Normalization) layer, and a ReLU activation function, then passes through a PSA module, followed by a BN (Batch Normalization) layer, and finally passes through a 1x1 convolution kernel with a stride of 1, a BN layer, and a ReLU activation function. The PSA module can effectively capture the frequently changing information relationships in the input feature map, preventing the lack of diversity in information transmission caused by fixed shuffle rules. The PSA module is composed of SEWeight, Softmax, SPC, etc. Among them, the SEWeight module is a variant of the SE module, used to extract channel attention weights. It obtains the channel information of the input feature map through global average pooling, and then generates the attention weight of each channel through a fully connected layer and an activation function (such as ReLU and Sigmoid). The Softmax function is used to convert the original attention weight output by the SEWeight module into normalized weights that can be summed. Softmax ensures that the attention weights of all channels add up to 1, so that the contributions of different channels can be weighted. The SPC module is the core part of the PSA module, responsible for multi-scale feature extraction. The input feature map is first divided into S groups in the channel dimension, each group having C / S channels. Then, each group of feature maps is processed by different scale convolution kernels to capture spatial information of different scales. The processed multi-scale feature maps are again concatenated in the channel dimension to form a feature map that integrates multi-scale information. Its detailed structure is shown in the accompanying drawings Figure 4 .

[0067] Finally, the residual connection and feature extraction part are spliced together, and then the channel shuffle module is used for channel mixing to form a Shuffle-PSA module, which realizes the feature fusion between groups. The detailed structure is shown in the following figure: Figure 5 .

[0068] Extraction stages 2, 3, and 4 are the operations of multiple branches after Transition 1, Transition 2, and Transition 3. Each branch needs to pass through four superimposed Shuffle-PSA modules, then fuse information of different scales. The output on each scale branch is fused by the outputs of all branches.

[0069] S4, train the convolutional neural network using the loss function. The loss function used to calculate the key points is as follows:

[0070]

[0071] where J is the human key point, H j (p) is the prediction result, representing the probability of key point j at p, is the real generated heat map, and v j represents whether key point j is visible, v j = 0 when key point j is not visible, and v j = 1 when key point j is visible.

[0072] S5, select the network with the best performance in training to predict human key points, and introduce a lightweight heat map post-processing method SR Head. Through the key point encoder, various key points are encoded into key point embeddings, and then decoded through a single large kernel convolution to calculate multiple heat maps about the data. Finally, all key point heat maps are spliced together through pixel shuffle to obtain the HR heat map of the data, effectively reducing the quantization error and alleviating the problem of low resolution features being difficult to supervise. The detailed structure is shown in the following figure: Figure 6 In this embodiment, the SRPSA-HRNet network is constructed by combining PSA-HRNet and SR Head, which significantly enhances the generalization ability of the network and reduces the quantization error.

[0073] In summary, the embodiment proposes a high-performance human key point detection method, which is based on an HRNet network and builds a human key point detection model SPSA-HRNet that can effectively establish long-distance channel dependency. A new module Shuffle-PSA is added in the HRNet extraction module, which can maintain a high accuracy without significantly increasing the parameter amount. The PSA-HRNet human key point detection model proposed in the embodiment enhances the multi-scale representation capability and long-distance channel dependency, so that the network can learn diversified features. At the same time, a lightweight heat map prediction method SR Head is introduced, which effectively reduces the quantization error. The SPSA-HRNet model constructed by combining PSA-HRNet and SR Head not only significantly enhances the generalization ability of the network and reduces the quantization error, but also shows flexible adaptability in diversified scenarios, and promotes the continuous progress of key point detection.

[0074] As shown in Figure 7 The embodiment discloses a high-resolution human key point detection system for executing the above method, which comprises the following modules:

[0075] A data set selection module is configured to select a data set containing human images and record the coordinates of human key points in the images.

[0076] A preprocessing module is configured to preprocess the selected data set.

[0077] A network construction module is configured to construct a pyramid-type high-resolution network, which comprises two modules: a shuffle module and a feature fusion module. The shuffle module is composed of a feature reservation branch and a feature extraction branch, and the feature extraction branch comprises a PSA module and two standard convolution layers. The feature fusion module first fuses from a high-resolution branch to a low-resolution branch, and then fuses from a low-resolution branch to a high-resolution branch. Finally, the results of the two fusions are added together as the output.

[0078] A network training module is configured to train the human key point network constructed by the network construction module using the preprocessed data set.

[0079] A detection module is configured to select the network with the best performance in the training to predict human key points, and introduce a super-resolution head heat map post-processing method to obtain the key point positions.

[0080] Other contents of the embodiment can refer to the above method embodiment.

[0081] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the described embodiments. Various changes, modifications, replacements, and variations of the embodiments including components can be made by those skilled in the art without departing from the principles and spirit of the present application, and still fall within the scope of the present application.

Claims

1. A high-resolution human keypoint detection method, characterized by Comprising the following steps: S1, selecting a data set, the data set containing human body images, and recording the coordinates of the human body key points in the images; S2, preprocessing the data set of step S1; S3, constructing a pyramid type high resolution network, comprising two modules: a shuffle module and a feature fusion module; wherein the shuffle module is composed of a feature reservation branch and a feature extraction branch, and the feature extraction branch comprises a PSA module and a convolution layer; the feature fusion module first fuses from the high resolution branch to the low resolution branch, then fuses the high resolution from the low resolution branch, and finally adds the results of the two fusions as the output; In this step, the specific formula for the feature fusion module to fuse from the high resolution branch to the low resolution branch is: wherein n is the total number of branches; A i is the output result of the i-th branch before feature fusion; H i is the fusion result of the i-th branch when fusing from high resolution to low resolution; F(a, b, H) is a function representing resolution conversion of the output result H of the a-th branch to achieve fusion with the b-th branch, a is a high resolution branch adjacent to b; In this step, the specific formula for the feature fusion module to fuse from the low resolution branch to the high resolution branch is: wherein L i is the fusion result of the i-th branch when fusing from low resolution to high resolution; F(a', b', L) is a function representing the resolution conversion of the output result L of the a' branch to achieve fusion with the b' branch, and a' is a low resolution branch adjacent to b'. S4, using the preprocessed data set of step S2 to train the human key point network constructed in step S3; S5, selecting the network with the best performance in the training to predict the human key points, and introducing a super resolution head heat map post-processing method to obtain the key point positions.

2. The high-resolution human keypoint detection method of claim 1, wherein: In step S2, the preprocessing is as follows: the width and height of the pedestrian detection frame in the data set are scaled to the target original scale to a set scale, then the corresponding padding is performed, the pedestrians in the data set images are cropped using a fixed aspect ratio of 4:3, and after cropping, the pedestrians are scaled to a fixed size of 256x192. The cropped images are randomly horizontally flipped with a probability of 0.5, randomly scaled with a scaling ratio of 0.65 to 1.35, and randomly rotated within a rotation angle range of -45°~ +45°.

3. The high-resolution human keypoint detection method of claim 2, wherein: In step S3, the feature extraction branch comprises a 1x1 convolution, a PSA module, a 1x1 convolution and channel shuffling connected in sequence.

4. The high-resolution human keypoint detection method of claim 1, wherein: In step S3, a BN layer and a ReLU activation function are added after the 1x1 convolution; a BN layer is added after the PSA module.

5. The high-resolution human keypoint detection method of claim 4, wherein: In step S5, the human key point network is trained using a loss function, and the loss function is as follows:

6. The high-resolution human keypoint detection method of any one of claims 1-5, wherein: Comprising the following modules: wherein, J is a human keypoint, H i (p) is a prediction result representing a probability of the keypoint j being at p, is a real generated heat map, v j denotes whether the keypoint j is visible or not.

7. A high-resolution human keypoint detection system configured to perform the method of any one of claims 1-6, wherein, A data set selection module: selecting a data set, the data set containing human body images, and recording the coordinates of the human body key points in the images; A preprocessing module: preprocessing the selected data set; A network construction module: constructing a pyramid type high resolution network, comprising two modules: a shuffle module and a feature fusion module; wherein the shuffle module is composed of a feature reservation branch and a feature extraction branch, and the feature extraction branch comprises a PSA module and a convolution layer; the feature fusion module first fuses from the high resolution branch to the low resolution branch, then fuses the high resolution from the low resolution branch, and finally adds the results of the two fusions as the output; A network training module: using the preprocessed data set to train the human key point network constructed by the network construction module; A detection module: selecting the network with the best performance in the training to predict the human key points, and introducing a super resolution head heat map post-processing method to obtain the key point positions. ​

Images