Human pose estimation method based on attention and adversarial network, medium and equipment

Abstract

The application provides a human pose estimation method based on attention and adversarial network, a medium and equipment; wherein the method is: inputting an image to be predicted into a data enhancement module based on an adversarial network for data enhancement processing to obtain a data enhanced image; inputting the data enhanced image into a human pose estimation network based on multi-scale spatial attention to obtain an output heat map; converting the output heat map into spatial coordinates of each key point to generate a human pose, and then obtaining a human pose estimation result. The method can obtain spatial attention from different scale features, simultaneously perform multi-scale fusion on the attention, combine global and local attention features, finally generate more fine spatial attention, and can solve the prediction problem of occluded images and complex background images.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and more specifically, to a method, medium, and device for human pose estimation based on attention and adversarial networks. Background Technology

[0002] Human pose estimation, as an important task in the field of computer vision, has received widespread attention in recent years. Human pose estimation is a technique for locating key points of the human body in an image, such as the head, wrists, and ankles. It forms the basis for downstream tasks such as motion analysis and has wide applications in medicine, sports, daily life, and gaming. The workflow of human pose estimation involves extracting key features from an image containing a human body, and then obtaining the relative position coordinates of each key point of the human body in the image from these features.

[0003] Traditional human pose estimation methods use manually extracted features such as histograms of orientation gradients, color features, and shape features, and model these features using simple SVM classifiers or graph structure models, achieving good results on simple datasets. However, these methods face some unavoidable problems. First, manually extracted features are only superficial image features, while high-level image features such as semantic features of the human body cannot be extracted. Second, the quality of manually extracted features is highly dependent on the experience of professional researchers, and different manual feature extraction methods are required for different scenarios, resulting in low efficiency. In addition, images contain factors such as lighting interference, shooting angles, and varying human body scales, in which case manually extracted features often fail to achieve satisfactory results.

[0004] With the development of deep learning and its research and application in computer vision, convolutional neural network-based methods have become the mainstream approach for human pose estimation. For single-person human pose estimation methods, based on different representations of the learning objective, they can be divided into two categories: coordinate regression-based methods and heatmap regression-based methods. Among them, heatmap regression-based pose estimation is the more mainstream approach. In heatmap regression-based methods, each keypoint coordinate is represented as a heatmap of the same size as the original image; the heatmap is generated by a Gaussian kernel function centered on the keypoint coordinates, with each point in the heatmap ranging from 0 to 1. Points closer to the keypoint have larger values ​​in the heatmap, and points farther away have smaller values. This method can retain richer information in the image; therefore, most mainstream human pose estimation methods are currently based on heatmap regression, such as HRNet and Stacked Hourglass. However, current mainstream human pose estimation methods all have certain problems: First, when faced with occluded images or images with complex environmental backgrounds, these methods cannot make accurate predictions due to the limitations of their own network structures. Secondly, current mainstream human pose estimation methods suffer from overfitting, exhibiting high accuracy on the training set but poor accuracy on the test set.

[0005] In summary, there is still room for improvement in current deep learning-based methods for human pose estimation. Summary of the Invention

[0006] To overcome the shortcomings and deficiencies of the existing technology, the present invention aims to provide a human pose estimation method, medium and device based on attention and adversarial networks. The method can obtain spatial attention from features at different scales, and simultaneously perform multi-scale fusion of attention, combining global and local attention features to finally generate more refined spatial attention, which can solve the prediction problem of occluded images and complex background images.

[0007] To achieve the above objectives, the present invention provides a human pose estimation method based on attention and adversarial networks, comprising the following steps:

[0008] S1. Input the image to be predicted into the adversarial network-based data augmentation module for data augmentation processing to obtain a data-augmented image;

[0009] S2. Input the data-enhanced image into a human pose estimation network based on multi-scale spatial attention to obtain the output heatmap;

[0010] S3. Convert the output heatmap into the spatial coordinates of each key point to generate human pose and thus obtain the human pose estimation result.

[0011] In S2, the human pose estimation network includes one or more multi-scale network modules and multi-scale spatial attention modules; the number of multi-scale spatial attention modules is the same as the number of multi-scale network modules, and there is a one-to-one correspondence between the multi-scale network modules and the multi-scale spatial attention modules; wherein, the multi-scale spatial attention module includes a multi-scale spatial attention feature extractor and a multi-scale spatial attention fusion module.

[0012] The input to the multi-scale spatial attention module is the multi-scale features obtained by the multi-scale network module. After the multi-scale features pass through the spatial attention feature extractor of the corresponding scale, the spatial attention of the corresponding scale is obtained. Then, it is input into the multi-scale spatial attention fusion module to perform multi-scale fusion of the spatial attention to generate the final high-resolution target spatial attention. Then, the target spatial attention is combined with the input highest-resolution multi-scale features to obtain the spatial attention-weighted features, and then the final output heatmap is generated.

[0013] Preferably, in step S1, the data enhancement process of the data enhancement module includes the following steps:

[0014] X1. Perform Cutout data augmentation on the input image with random keypoints as the center to obtain a preliminary data-augmented image;

[0015] X2. The input image and keypoint coordinates are transformed by a spatial transformation network. The spatial transformation network includes a ResNet50 network, pooling layers, and fully connected layers. After the input image passes through the ResNet50 network and pooling layers, intermediate features are obtained. These intermediate features are flattened and keypoint coordinates (x, y) are added. Then, the image is passed through a fully connected layer to obtain the image transformation parameters (θ, r, Δx, Δy). Here, θ represents the rotation angle, r represents the aspect ratio of the rectangle, Δx represents the displacement of the x-coordinate of the rectangle's center point, and Δy represents the displacement of the y-coordinate of the rectangle's center point.

[0016] X3. For the cropping rectangle of the data augmented image obtained by X1, perform an affine transformation using the graphic transformation parameters (θ, r, Δx, Δy) obtained by X2 to obtain the transformed rectangle; crop the input image based on the transformed rectangle to obtain the data augmented image.

[0017] Preferably, in the multi-scale spatial attention module, the multi-scale spatial attention feature extractor consists of a 3x3 convolutional layer one, a 1x1 convolutional layer two, and three 1x1 convolutional + Sigmoid normalization modules.

[0018] Preferably, in the multi-scale spatial attention module, the multi-scale spatial attention fusion module performs multi-scale fusion of spatial attention, which means: after upsampling each scale of spatial attention, it has the same feature dimension as the highest resolution spatial attention; then all spatial attention are merged and connected, and after passing through a 1x1 convolutional layer, a batch normalization layer, and a ReLU normalization layer, a high-resolution target spatial attention that combines global and local information is obtained.

[0019] Preferably, the data augmentation module and the human pose estimation network refer to the data augmentation module and the human pose estimation network obtained after training the initial data augmentation module and the human pose estimation network.

[0020] Preferably, training the initial data augmentation module and the human pose estimation network includes the following steps:

[0021] Y1, Randomly initialized data augmentation module and human pose estimation network;

[0022] Y2. Alternately train the data augmentation module and the human pose estimation network: In one gradient update, perform one forward propagation, then fix the parameters of the data augmentation module and backpropagate to update the parameters of the human pose estimation network; after the update, perform another forward propagation, fix the parameters of the human pose estimation network, and then backpropagate to update the parameters of the data augmentation module; iterate the training rounds until both the data augmentation module and the human pose estimation network converge.

[0023] Preferably, in Y2, when the data augmentation module and the human pose estimation network are trained alternately, the loss function of the human pose estimation network is Loss predictor for:

[0024] Loss predictor =MSE(output,gt)+MSE(output) A ,gt)

[0025] Wherein, output represents the output heatmap obtained after directly inputting the image into the human pose estimation network; A This indicates the output heatmap obtained after the image is first input into the data augmentation module for data augmentation processing, and then the data-augmented image is input into the human pose estimation network; gt represents the true heatmap generated based on the labeled coordinates; MSE represents the mean squared error loss function;

[0026] Loss function of data augmentation module generator for:

[0027] Loss generator =-Losspredictor .

[0028] Preferably, in step S3, converting the output heatmap into the spatial coordinates of each key point means: obtaining the coordinates of the point with the largest value in the output heatmap, converting the coordinates of the largest point into the spatial coordinates of the original image, and using them as the spatial coordinates of the key point.

[0029] A storage medium storing a computer program that, when executed by a processor, causes the processor to perform the aforementioned human pose estimation method based on attention and adversarial networks.

[0030] A computing device includes a processor and a memory for storing processor-executable programs, wherein when the processor executes the programs stored in the memory, it implements the above-described human pose estimation method based on attention and adversarial networks.

[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0032] 1. This invention proposes a human pose estimation network based on multi-scale spatial attention. It solves the prediction problem of occluded images and complex background images through a multi-scale spatial attention mechanism, enhances feature extraction of the human foreground region, and eliminates background interference. Unlike conventional spatial attention mechanisms, the multi-scale spatial attention mechanism proposed in this invention combines the multi-scale characteristics of the human pose estimation network structure to obtain spatial attention from features of different scales in the network. At the same time, it performs multi-scale fusion of attention, combining global and local attention features to finally generate more refined spatial attention.

[0033] 2. This invention proposes a data augmentation module based on adversarial networks. By using adversarial networks to generate data augmented samples that selectively discard human appearance features, it significantly reduces model overfitting. Traditional data augmentation methods such as Cutout and Random Crop discard images to reduce overfitting; however, these methods randomly discard images. Research indicates that the main factor causing overfitting in human pose estimation models is the appearance features of the image, specifically the foreground region of the human body. The data augmentation module based on adversarial networks proposed in this invention utilizes adversarial learning to generate data augmented images, selectively discarding foreground regions of the human body, resulting in a more significant effect on mitigating model overfitting.

[0034] 3. Using the human posture estimation method proposed in this invention to predict human posture helps improve the machine's recognition accuracy of human posture and is of great significance for improving the effect of downstream applications such as human motion recognition and real-time human positioning. Attached Figure Description

[0035] Figure 1 This is a flowchart of the human pose estimation method based on attention and adversarial networks of the present invention;

[0036] Figure 2 This is a schematic diagram of the data enhancement module based on adversarial networks of the present invention;

[0037] Figure 3 This is a structural block diagram of the spatial transformation network of the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of the human pose estimation network based on multi-scale spatial attention in this invention;

[0039] Figure 5 This is a structural block diagram of the multi-scale spatial attention module of the present invention;

[0040] Figure 6 This is a block diagram of the multi-scale spatial attention feature extractor of the present invention;

[0041] Figure 7 This is a structural block diagram of the multi-scale spatial attention fusion module of the present invention;

[0042] Figure 8 This is a schematic diagram illustrating the processing of the heatmap output during the training of the data augmentation module and the human pose estimation network of this invention. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0044] Example 1

[0045] This embodiment presents a human pose estimation method based on attention and adversarial networks, such as... Figure 1 As shown, it includes the following steps:

[0046] S1. Input the image to be predicted into the adversarial network-based data augmentation module for data augmentation processing to obtain the data-augmented image.

[0047] like Figure 2 As shown, the data augmentation process of the data augmentation module includes the following steps:

[0048] X1. Perform a Cutout data augmentation operation on the input image centered on random key points. The Cutout data augmentation operation used in this invention centers on random labeled key points and specifies the length and width of the cropped part, such as 64*64. After Cutout, a preliminary data augmented image is obtained.

[0049] X2. The input image and keypoint coordinates are transformed by a spatial transformation network; the spatial transformation network includes a ResNet50 network, pooling layers, and fully connected layers, such as... Figure 3 As shown, the input image is processed through a ResNet50 network and pooling layers to obtain intermediate features. These intermediate features are then flattened and keypoint coordinates (x, y) are added. After passing through a fully connected layer, the image transformation parameters (θ, r, Δx, Δy) are obtained. Here, θ represents the rotation angle, r represents the aspect ratio of the rectangle, Δx represents the displacement of the x-coordinate of the rectangle's center point, and Δy represents the displacement of the y-coordinate of the rectangle's center point.

[0050] X3. For the cropping rectangle of the data-enhanced image obtained by X1, perform an affine transformation using the graphic transformation parameters (θ,r,Δx,Δy) obtained by X2 to obtain the transformed rectangle; crop the input image based on the transformed rectangle to obtain the data-enhanced image.

[0051] S2. Input the data-augmented image into a human pose estimation network based on multi-scale spatial attention to obtain the output heatmap.

[0052] This invention constructs a corresponding multi-scale spatial attention module based on the human pose estimation network structure. Currently, mainstream human pose estimation networks all contain one or more multi-scale network modules to extract human feature information at different scales. Multi-scale features refer to feature maps with different resolutions, i.e., feature maps with different aspect ratios, such as 256x256 and 64x64; the larger the aspect ratio of the feature map, the higher the resolution and the smaller the scale of the feature. These network modules contain features at multiple scales and perform feature fusion between different scales during feature extraction to simultaneously obtain both global and local human features.

[0053] like Figure 4 As shown, the human pose estimation network of the present invention includes one or more multi-scale network modules and multi-scale spatial attention modules; the number of multi-scale spatial attention modules is the same as the number of multi-scale network modules, and there is a one-to-one correspondence between the multi-scale network modules and the multi-scale spatial attention modules.

[0054] The multi-scale spatial attention module includes a multi-scale spatial attention feature extractor and a multi-scale spatial attention fusion module, such as... Figure 5 As shown, the multi-scale spatial attention feature extractor consists of a 3x3 convolutional layer, a 1x1 convolutional layer, and three 1x1 convolutional + sigmoid normalization modules, as follows. Figure 6As shown. This multi-scale spatial attention feature extractor performs multiple convolutions on the input multi-scale features to extract rich spatial information and finally transforms them into spatial attention at the corresponding scale. The multi-scale spatial attention fusion module performs multi-scale fusion of spatial attention, which means: after upsampling each scale of spatial attention, it has the same feature dimension as the highest resolution spatial attention; then, all spatial attention are merged and connected, and passed through a 1x1 convolutional layer, a batch normalization layer, and a ReLU normalization layer, as shown. Figure 7 As shown, a high-resolution target spatial attention that combines global and local information is obtained.

[0055] The input to the multi-scale spatial attention module is the multi-scale features obtained by the multi-scale network module. After the multi-scale features pass through the spatial attention feature extractor of the corresponding scale, the spatial attention of the corresponding scale is obtained. Then, it is input into the multi-scale spatial attention fusion module to perform multi-scale fusion of the spatial attention and generate the final high-resolution target spatial attention. Then, the target spatial attention is combined with the highest resolution multi-scale features of the input (dot multiplication) to obtain the spatial attention weighted features, and then the final output heatmap is generated.

[0056] S3. Convert the output heatmap into the spatial coordinates of each key point to generate human pose and thus obtain the human pose estimation result.

[0057] Converting the output heatmap into the spatial coordinates of each key point means: finding the coordinates of the key point with the largest value in the output heatmap, converting the coordinates of the largest point into the spatial coordinates of the original image, and using them as the spatial coordinates of the key point.

[0058] The data augmentation module and the human pose estimation network refer to the data augmentation module and the human pose estimation network obtained after training the initial data augmentation module and the human pose estimation network. In this invention, an adversarial training method is used to alternately train the data augmentation module and the human pose estimation network, so that the data augmentation module generates as many data augmentation samples as possible by cropping the foreground portion of the image, thereby enhancing the data augmentation effect; at the same time, the robustness of the human pose estimation network to data augmentation samples is enhanced, reducing the overfitting of the model.

[0059] This invention employs an inverse loss function for training. Specifically, training the initial data augmentation module and the human pose estimation network involves the following steps:

[0060] Y1, Randomly initialized data augmentation module and human pose estimation network;

[0061] Y2. Alternately train the data augmentation module and the human pose estimation network: In one gradient update, perform one forward propagation, then fix the parameters of the data augmentation module and backpropagate to update the parameters of the human pose estimation network; after the update, perform another forward propagation, fix the parameters of the human pose estimation network, and then backpropagate to update the parameters of the data augmentation module; iterate the training rounds until both the data augmentation module and the human pose estimation network converge.

[0062] Preferably, in Y2, when the data augmentation module and the human pose estimation network are trained alternately, the loss function of the human pose estimation network is Loss predictor for:

[0063] Loss predictor =MSE(output,gt)+MSE(output) A ,gt)

[0064] Wherein, output represents the output heatmap obtained after directly inputting the image into the human pose estimation network; A This indicates the output heatmap obtained after first inputting the image into the data augmentation module for data augmentation processing, and then inputting the data-augmented image into the human pose estimation network, such as... Figure 8 As shown; gt represents the actual heat map generated based on the labeled coordinates; MSE represents the mean squared error loss function;

[0065] Loss function of data augmentation module generator for:

[0066] Loss generator =-Loss predictor .

[0067] Example 2

[0068] This embodiment provides a storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the human pose estimation method based on attention and adversarial networks as described in Embodiment 1.

[0069] Example 3

[0070] This embodiment provides a computing device, including a processor and a memory for storing processor-executable programs. The processor, when executing the program stored in the memory, implements the human pose estimation method based on attention and adversarial networks as described in Embodiment 1.

[0071] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A human pose estimation method based on attention and adversarial networks, characterized in that: Includes the following steps: S1. Input the image to be predicted into the adversarial network-based data augmentation module for data augmentation processing to obtain a data-augmented image; S2. Input the data-enhanced image into a human pose estimation network based on multi-scale spatial attention to obtain the output heatmap; S3. Convert the output heatmap into the spatial coordinates of each key point to generate human pose and thus obtain the human pose estimation result. In S2, the human pose estimation network includes one or more multi-scale network modules and multi-scale spatial attention modules; the number of multi-scale spatial attention modules is the same as the number of multi-scale network modules, and there is a one-to-one correspondence between the multi-scale network modules and the multi-scale spatial attention modules; wherein, the multi-scale spatial attention module includes a multi-scale spatial attention feature extractor and a multi-scale spatial attention fusion module. The input to the multi-scale spatial attention module is the multi-scale features obtained by the multi-scale network module; after the multi-scale features are processed by the spatial attention feature extractor of the corresponding scale, the spatial attention of the corresponding scale is obtained; then it is input to the multi-scale spatial attention fusion module to perform multi-scale fusion of the spatial attention to generate the final high-resolution target spatial attention; then the target spatial attention is combined with the input highest resolution multi-scale features to obtain the spatial attention weighted features, and then the final output heatmap is generated. The data augmentation module and human pose estimation network refer to the data augmentation module and human pose estimation network obtained after training the initial data augmentation module and human pose estimation network. The training of the initial data augmentation module and the human pose estimation network includes the following steps: Y1, Randomly initialized data augmentation module and human pose estimation network; Y2. Alternately train the data augmentation module and the human pose estimation network: In one gradient update, perform one forward propagation, then fix the parameters of the data augmentation module and backpropagate to update the parameters of the human pose estimation network; after the update, perform another forward propagation, fix the parameters of the human pose estimation network, and then backpropagate to update the parameters of the data augmentation module; iterate the training rounds until both the data augmentation module and the human pose estimation network converge. In Y2, when the data augmentation module and the human pose estimation network are trained alternately, the loss function of the human pose estimation network is... for: ； in, This represents the output heatmap obtained after directly inputting an image into a human pose estimation network; This indicates that the image is first input into the data augmentation module for data augmentation processing, and then the data-augmented image is input into the human pose estimation network to obtain the output heatmap; This represents the actual heat map generated based on the labeled coordinates; This represents the mean squared error loss function; Loss function of data augmentation module for: 。 2. The human pose estimation method based on attention and adversarial networks according to claim 1, characterized in that: In step S1, the data enhancement process of the data enhancement module includes the following steps: X1. Perform Cutout data augmentation on the input image with random keypoints as the center to obtain a preliminary data-augmented image; X2. The input image and keypoint coordinates are transformed by a spatial transformation network. The spatial transformation network includes a ResNet50 network, pooling layers, and fully connected layers. After the input image passes through the ResNet50 network and pooling layers, intermediate features are obtained. These intermediate features are flattened and then added to the keypoint coordinates. The graphics transformation parameters are then obtained through a fully connected layer. ,r, , ;in This indicates the rotation angle, and r represents the aspect ratio of the rectangle. This represents the displacement of the x-coordinate of the center point of the rectangle. This represents the displacement of the ordinate of the center point of the rectangle; X3. For the cropping rectangle of the data-enhanced image obtained in X1, use the graphic transformation parameters obtained in X2. ,r, , Perform an affine transformation to obtain the transformed rectangular frame; The input image is cropped based on the transformed rectangle to obtain the data-enhanced image.

3. The human pose estimation method based on attention and adversarial networks according to claim 1, characterized in that: In the multi-scale spatial attention module, the multi-scale spatial attention feature extractor consists of a 3x3 convolutional layer one, a 1x1 convolutional layer two, and three 1x1 convolutional + Sigmoid normalization modules.

4. The human pose estimation method based on attention and adversarial networks according to claim 1, characterized in that: In the multi-scale spatial attention module, the multi-scale spatial attention fusion module performs multi-scale fusion of spatial attention, which means that after upsampling each scale of spatial attention, it has the same feature dimension as the highest resolution spatial attention. Then, all spatial attentions are merged and connected, and after passing through a 1x1 convolutional layer, a batch normalization layer, and a ReLU normalization layer, a high-resolution target spatial attention that combines global and local information is obtained.

5. The human pose estimation method based on attention and adversarial networks according to claim 1, characterized in that: In step S3, converting the output heatmap into the spatial coordinates of each key point means: finding the coordinates of the point with the largest value in the output heatmap, converting the coordinates of the largest point into the spatial coordinates of the original image, and using them as the spatial coordinates of the key point.

6. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to perform the human pose estimation method based on attention and adversarial networks as described in any one of claims 1-5.

7. A computing device, comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the human pose estimation method based on attention and adversarial networks as described in any one of claims 1-5.

Images