Human instance-aware multi-person pose estimation method

Abstract

The present invention discloses a multi-person pose estimation method based on explicit self-attention supervision of Gaussian functions, comprising the following steps: Step 1, selecting a suitable deep learning network as a feature extractor; Step 2, constructing a parallel convolution and self-attention module; Step 3, constructing a key point position prediction network; Step 4, performing explicit supervision of the self-attention branch in the parallel module based on a Gaussian function; Step 5, inputting a training image, applying data augmentation technology to expand the training samples, and outputting the model using a labeled supervision; Step 6, backpropagating the loss until the training converges; Step 7, in the testing phase, inputting the test image into the final multi-person pose estimation model to perform pose estimation of multiple human instances. This application solves the technical problems of low efficiency and poor interpretability of multi-person pose estimation. Through the method of this application, the feature learning process of key point positioning and key point matching can be unified, thereby accelerating the speed of multi-person pose estimation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and pattern recognition, and specifically relates to a multi-person posture estimation model based on human instance perception, which can be applied to many fields such as virtual reality, intelligent security, and human-computer interaction. Background Art

[0002] Human pose estimation is a research task in computer vision. It aims to use computers to analyze and interpret human figures in images or videos, thereby inferring the coordinates of their joints. Multi-person pose estimation targets images with multiple human instances and typically involves two steps: keypoint localization and keypoint matching.

[0003] Currently, there are two main approaches to pose estimation in multi-person scenarios: top-down and bottom-up. Top-down approaches first detect people and then identify joints. Bottom-up approaches first identify joints and then the body. Both strategies have their advantages and disadvantages. Top-down approaches rely heavily on detector performance and cannot address occlusions. Furthermore, independent detectors increase the model's inference time and computational burden, significantly impacting the algorithm's real-time performance. Bottom-up approaches, on the other hand, offer faster inference speed and do not require additional detectors. However, they require complex joint grouping algorithms and other post-processing modules. The model cannot directly learn features related to human instances and typically requires the introduction of skeletal constraints or other image semantic information to facilitate joint matching. The difficulty of multi-person pose estimation lies in resolving occlusions between people and preventing keypoints from being misassigned to other people. Previous approaches, to simultaneously learn keypoint location and matching information, incorporate structured human skeleton information into the model or use vector fields to describe the relationships between joints. These methods rely on manually designed annotations to supervise model training, significantly increasing the algorithm's implementation complexity and failing to fully exploit the human kinematic information in the data. Summary of the Invention

[0004] In order to solve the above problems in the prior art, the present invention proposes an efficient multi-person pose estimation model, which utilizes the advantage of convolutional neural networks' sensitivity to position information to enhance the positioning of key points, utilizes the global receptive field of self-attention to enhance human instance perception, and simultaneously learns key point positioning and key point grouping.

[0005] The method comprises the following steps:

[0006] Step 1: Select a suitable deep learning network as a feature extractor;

[0007] Step 2: Construct a parallel convolution and self-attention module to encode the local information and global connections required for key point positioning;

[0008] Step 3: construct a key point position prediction network to output a key point position heat map;

[0009] Step 4: Perform explicit supervision on the self-attention branch in the parallel module based on the Gaussian function. Use the Gaussian function to weight the attention distribution map near each key point to obtain the attention distribution map corresponding to each human instance.

[0010] Step 5: Input the training image and use data augmentation techniques to expand the training samples. The output keypoint location heatmap is compared with the annotated ground truth heatmap using a mean square error loss. The output human instance attention distribution map is compared with the annotated human instance mask using a mean square error loss. The weighted sum of these two losses yields the final loss function.

[0011] Step 6: Back-propagate the loss until the training converges to obtain the final multi-person pose estimation network model;

[0012] Step 7: During the testing phase, the test image is input into the final multi-person pose estimation model to perform pose estimation on multiple human instances.

[0013] Preferably, step 1 requires selecting a suitable deep feature extraction network to ensure that the key point features can be stably obtained after the image input passes through the feature extractor. The selection of a deep feature extraction network is crucial because different network architectures and parameter settings will directly affect the effect of feature extraction. A good deep feature extraction network considers the following two aspects. The first is the network structure. Selecting a suitable network structure is the key to ensuring that the feature extractor can effectively extract key point features in the image. Some classic network structures, such as ResNet, Hourglass and ViT, often achieve good results in image processing tasks. The second is parameter setting. The parameter setting of the deep feature extraction network also needs to be adjusted according to the specific task. This includes selecting the appropriate convolution kernel size, number of convolution layers, pooling strategy, etc. Reasonable parameter settings can improve the perception and expression capabilities of the feature extractor, thereby better capturing key point features.

[0014] Preferably, in step 2, because the underlying features lack the local features and global features required for key point positioning, after the underlying features are obtained by the feature extractor, the feature map will be sent to the parallel convolution and attention module for further feature extraction. In order to reduce the computational burden, the feature map of the input convolution branch is kept at the input resolution unchanged, while the feature map of the input self-attention branch is downsampled k times. In the convolution branch, the original single convolution layer is replaced by the hourglass module. After the first convolution layer, a polarized self-attention mechanism is introduced to enhance the modeling capability of the convolution branch.

[0015] Preferably, in step 3, the feature map is fed into a keypoint prediction network after multiple stages of computation using parallel convolution and self-attention modules, which outputs a keypoint heatmap of K channels. Each channel's heatmap corresponds to a keypoint of the human body, and the heatmap contains zero to several keypoint position responses, thus completing the keypoint localization for multi-person pose estimation.

[0016] Preferably, in step 4, an explicit supervised self-attention group matching algorithm based on Gaussian distribution is proposed. The present invention defines a new key point matching method, which uses the attention scores in the self-attention layer to match the key points in the same human instance. The first step is to calculate the attention distribution map corresponding to the key points and define the attention score matrix of the self-attention layer as The existing key point coordinates (x, y), take out all the attention distribution maps within the range R centered on the key point {A i,j |(ix) 2 +(jy) 2 ≤R 2}, calculate the Gaussian weight α corresponding to the coordinate (i, j) i,j , the attention distribution map corresponding to this key point is In the second step, all key point attention distribution maps of the same human instance are added together and the mean square error loss is performed with the human instance mask in the annotation.

[0017] Preferably, in step 5, the input training data is augmented to enhance the generalization performance of the model. By augmenting the training data, diversity and variability can be introduced, thereby improving the model's adaptability to a variety of different samples. The main data augmentation methods used in the present invention include random rotation and random flipping, random scaling, random cropping, etc.

[0018] The present invention addresses the technical issues of low efficiency and poor interpretability in multi-person pose estimation. This method unifies the feature learning processes for keypoint location and keypoint matching, accelerating multi-person pose estimation. The proposed method offers high detection accuracy and a wide range of applications, ensuring accurate keypoint detection and matching while minimizing the number of network parameters and computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the multi-person posture estimation method based on human instance perception provided by the present invention.

[0020] Figure 2 This is the network model framework constructed by the present invention.

[0021] Figure 3 Schematic diagram of self-attention reasoning constructed for this invention.

[0022] Figure 4 Schematic diagram of the reasoning of the method of the present invention. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0024] The present invention discloses a multi-person posture estimation method based on human instance perception. Figure 1 As shown, the following steps are included:

[0025] Step 1: Select an appropriate deep feature extraction network. Before keypoint location and matching, the image must first be encoded into corresponding low-level features through a feature extractor. The feature extractor consists of a convolutional network or a Transformer-based network, such as ResNet-50 or Swin-Transformer. After the image passes through the feature extractor, a set of feature maps is generated. Assuming the downsampling rate of the feature extractor is r, the resolution of this set of feature maps is calculated to be H × W × C, where H and W are the image input resolutions of the model. Before the low-level feature maps are fed into the next stage, a 1×1 convolution is performed to adjust the number of feature map channels.

[0026] Step 2: Construct a parallel convolution and self-attention module to encode the local information and global connections required for key point positioning. The present invention has made a unique design for each branch of the module. First, the convolution branch, the resolution of the feature map of the input convolution branch remains unchanged. The hourglass module is introduced in the convolution branch, and after the first convolution layer, the polarization self-attention mechanism is introduced to enhance the convolution branch's perception of spatial information and channel information. After the above adjustments, the local feature capture ability of the parallel module is greatly enhanced, and the key point position can be located more efficiently. The polarization self-attention is defined as follows:

[0027]

[0028]

[0029]

[0030] Step 3: Construct a keypoint location prediction network and output a keypoint location heatmap. This is then compared with the annotated ground-truth heatmap using a mean squared error loss. The prediction network consists of a transposed convolution and a channel-wise convolution. The transposed convolution upsamples the feature map to a resolution of 160×160, and then the channel-wise convolution predicts the probability of a keypoint for each pixel in the feature map. This yields K keypoint location heatmaps, where K represents the number of keypoints in the human instance.

[0031] In step 4, the self-attention branch in the parallel module is explicitly supervised based on the Gaussian function. The attention distribution map corresponding to the key points of the human body is taken out from the attention branch, and the mean square error loss is performed with the human instance mask in the annotation.

[0032] After obtaining the positions of all key points in step 3, key points within the same human instance need to be grouped together and connected into human poses. This paper proposes an explicitly supervised self-attention group matching algorithm based on Gaussian distribution. This grouping algorithm uses a human instance mask to supervise the attention distribution map corresponding to each human instance, strengthening the self-attention mechanism's perception of the human instance. A Gaussian function is used to weight the attention distribution map near each key point, reducing key point matching errors caused by key point positioning deviations.

[0033] Now assume that there are N stages in total, and each stage contains several parallel convolution and self-attention modules with the same hyperparameters. Keep the self-attention matrix of the last layer, and now assume that the position of the kth key point is now (x i ,y i ,v i ), where x i ,y i Indicates the coordinate position of the key point, v i Represents the visibility of key points. First, the attention matrix is ​​transformed into a size of h×w×h×w, where h and w are the spatial dimensions of the input attention branch feature map. Then the true value of the key point coordinates is downsampled to the same magnification as the attention matrix to obtain the key point coordinates corresponding to the scale of the attention matrix. Considering that the predicted key point positions are generally near the true value of the key point, the present invention uses a Gaussian function to calculate the weights of all positions within a range of R from the true value of the key point coordinates. The Gaussian function formula is defined as follows:

[0034]

[0035] After obtaining the weights, take out the attention matrices corresponding to all positions with a radius of R around the true value of the key point, add them together according to the obtained weights, and obtain the attention distribution map corresponding to the key point. After obtaining the attention distribution maps of all key points of the same person in the same calculation method, add them together and take the average to obtain the attention distribution map of the human instance. Because the attention matrix has been normalized, in order to facilitate the subsequent loss with the true value mask of the human instance, the attention distribution map is divided by the maximum value in the attention matrix to obtain an attention distribution map with a value between (0, 1). After obtaining the predicted attention distribution map of the human instance, it is also necessary to calculate the true value mask of the human instance corresponding to the instance. The specific method is to take out the instance mask corresponding to the human instance from the annotation, and all positions within the mask range are set to 1, and the rest are set to 0.

[0036] Step 5: Data augmentation is performed on the input training image to enhance the model's generalization capabilities. After the image is input, K keypoint location heatmaps are obtained. The mean squared error (MSE) between the predicted attention distribution map and the human instance mask is calculated to obtain the keypoint matching loss.

[0037] The present invention adopts the following data augmentation methods:

[0038] (1) Randomly scale the image with a scaling factor between 0.75 and 1.5.

[0039] (2) Randomly flip the image, and the probability of flipping each image is 0.5.

[0040] (3) Randomly rotate the image with a rotation angle between -30° and 30°.

[0041] (4) Randomly translate the image, with the horizontal or vertical translation distance ranging from -40 to 40.

[0042] In step 6, the key point positioning loss and the key point matching loss are weighted and summed to obtain the overall error loss of the model. The network parameters are learned using back propagation technology so that the model can obtain the ability to locate and match key points.

[0043] Step 7, inference phase, the model uses the Hungarian algorithm to group key points based on the human instance mask. Assume that all key points are identified in the image Where i is the keypoint index, k is the human instance index in the image, K is the number of keypoint types, and M is the total number of human instances in the image. The attention distribution map corresponding to the keypoint locations is then obtained from the attention layer in the final stage. Keypoints are matched one by one, and only after all keypoints of the same type are matched will the next set of keypoints be matched. Each set of keypoints contains multiple keypoint coordinates corresponding to different human instances.

[0044] Key point grouping depends on the correlation between candidate key points and grouped human frames, where {S j |j∈[1,M]} represents the grouped human frame, and p represents the key point to be grouped. The calculated correlation score can be used to determine the degree of match between the key point to be matched and the human frame, with the highest correlation score being the optimal match. Based on the above process, the Hungarian algorithm can be used to connect all key points to be matched with the human frame, completing the key point grouping. The correlation degree definition formula is as follows:

[0045]

[0046] The present invention proposes a method for multi-person posture estimation based on human instance perception and explicit attention supervision. The method utilizes parallel convolution and self-attention modules to learn the global connection between local texture features and key points required for key point positioning, and efficiently locates the key point positions. In order to match key points to human instances, the present invention designs a key point grouping algorithm based on explicit supervised attention of Gaussian functions. By explicitly supervising the self-attention scores, the key point associations in the same human instance are strengthened, and the key point associations in different human instances are weakened, thereby achieving end-to-end multi-person posture estimation. By explicitly supervising self-attention and introducing Gaussian functions for correction, the model has the ability to simultaneously learn key point positioning and key point matching. From the perspective of feature learning, the present invention integrates the information that needs to be learned for multi-person posture estimation into feature learning, which greatly enhances the interpretability of the model.

[0047] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for multi-person pose estimation based on human instance perception, characterized in that: The method comprises the following steps: Step 1: Select a suitable deep learning network as a feature extractor; Step 2: Construct a parallel convolution and self-attention module to encode the local information and global connections required for key point positioning; Step 3: construct a key point position prediction network to output a key point position heat map; Step 4: Perform explicit supervision on the self-attention branch in the parallel module based on the Gaussian function. Use the Gaussian function to weight the attention distribution map near each key point to obtain the attention distribution map corresponding to each human instance. Step 5: Input the training image and use data augmentation technology to expand the training samples; perform mean square error loss on the output key point position heat map and the labeled true value heat map, and perform mean square error loss on the output human instance attention distribution map and the labeled human instance mask. The weighted sum of the two losses is used to obtain the final loss function; Step 6: Back-propagate the loss until the training converges to obtain the final multi-person pose estimation network model; Step 7: During the testing phase, the test image is input into the final multi-person pose estimation model to perform pose estimation on multiple human instances.

2. The method for multi-person pose estimation based on human instance perception according to claim 1, characterized in that: In step 2, in order to encode the local context information around the key points and the long-range connections between the key points, a parallel convolution and self-attention module is constructed. This module combines two different calculation methods, convolution and self-attention mechanisms, into the same module in a parallel manner, giving full play to the advantages of different operators.

3. The method for multi-person pose estimation based on human instance perception according to claim 1, characterized in that: In step 3, in order to locate the key points more accurately, the key point position heat map calculated by transposed convolution and channel convolution is used.

4. The method for multi-person pose estimation based on human instance perception according to claim 1, wherein: In step 4, a key point grouping method based on explicit attention supervision of a Gaussian function is used. This method uses a human instance mask to supervise the attention distribution map corresponding to each human instance, strengthens the self-attention mechanism's perception of the human instance, and uses a Gaussian function to weight the attention distribution map near each key point, thereby reducing the key point matching error caused by the deviation of the key point positioning.

5. The method for multi-person pose estimation based on human instance perception according to claim 1, characterized in that: In step 5, data augmentation is performed on the input training image to increase the generalization ability of the model.

Images