A human posture estimation method, a human posture estimation device and a smart device

Abstract

The application discloses a human posture estimation method and device, an intelligent device and a computer readable storage medium. The method comprises the following steps: inputting a to-be-processed human image into a trained posture estimation model, the model comprising a feature pyramid network and a lightweight backbone network, the backbone network being used for extracting features of multiple levels in the to-be-processed human image, and the feature pyramid network being used for fusing features of different levels extracted by the backbone network; obtaining a first estimation result according to the deepest layer feature output by the backbone network, the result being used for indicating whether each human key point is visible in the to-be-processed human image; and obtaining a second estimation result according to a heat map and an offset map output by the feature pyramid network, the result being used for indicating the coordinates of each human key point in the to-be-processed human image. Through the application scheme, fast and high-accuracy human posture estimation can be realized on an embedded intelligent device.

Description

Technical Field

[0001] This application belongs to the field of human-computer interaction technology, and in particular relates to a human pose estimation method, a human pose estimation device, an intelligent device, and a computer-readable storage medium. Background Technology

[0002] Human pose estimation is an important research topic in the field of human-computer interaction. It is usually a prerequisite task for human behavior recognition, human behavior analysis and human-computer interaction, with the aim of accurately locating the position of key points of the human body in an image.

[0003] Currently, most pose estimation models are large, with many parameters and high floating-point computation requirements. They can only achieve near real-time performance on high-performance Graphics Processing Units (GPUs), and cannot be applied to embedded intelligent devices such as robots. Therefore, how to achieve fast and accurate human pose estimation on embedded intelligent devices has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a human pose estimation method, a human pose estimation device, an intelligent device, and a computer-readable storage medium, which can realize fast and highly accurate human pose estimation on embedded intelligent devices.

[0005] In a first aspect, this application provides a human pose estimation method, including:

[0006] The human image to be processed is input into a trained pose estimation model, which includes a Feature Pyramid Network (FPN) and a lightweight backbone network. The backbone network is used to extract features from multiple levels in the human image to be processed, and the Feature Pyramid Network is used to fuse the features extracted from different levels by the backbone network.

[0007] The first estimation result is obtained based on the deepest features output by the backbone network. The first estimation result is used to indicate whether each human key point is visible in the human image to be processed.

[0008] The second estimation result is obtained based on the heatmap and offset map output by the feature pyramid network. The second estimation result is used to indicate the coordinates of each human key point in the human image to be processed. The heatmap is used to predict the pixel-level position of each human key point, and the offset map is used to predict the local offset of the pixel-level position of each human key point at the sub-pixel level.

[0009] Secondly, this application provides a human posture estimation device, comprising:

[0010] The input module is used to input the human image to be processed into the trained pose estimation model. The pose estimation model includes a feature pyramid network and a lightweight backbone network. The backbone network is used to extract features from multiple levels in the human image to be processed, and the feature pyramid network is used to fuse the features extracted from different levels by the backbone network.

[0011] The first determining module is used to obtain a first estimation result based on the deepest features output by the backbone network. The first estimation result is used to indicate whether each human key point is visible in the human image to be processed.

[0012] The second determining module is used to obtain a second estimation result based on the heat map and offset map output by the feature pyramid network. The second estimation result is used to indicate the coordinates of each human key point in the human image to be processed. The heat map is used to predict the pixel-level position of each human key point, and the offset map is used to predict the local offset of the pixel-level position of each human key point at the sub-pixel level.

[0013] Thirdly, this application provides an intelligent device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method as described in the first aspect.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect.

[0015] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method as described in the first aspect.

[0016] The advantages of this application compared to existing technologies are as follows: This application estimates various human keypoints in the human image to be processed using a trained pose estimation model. It can be understood that once the positions of the human keypoints are determined, the human pose is also determined. Specifically, the pose estimation model mainly consists of two components: a lightweight backbone network and a feature pyramid network. The backbone network is used to extract features from multiple levels in the human image to be processed. Because the backbone network is lightweight, the feature extraction speed is improved, thereby achieving rapid recognition of human pose. The feature pyramid network is used to fuse the features extracted from different levels by the backbone network, so that the heatmap and offset map output by the feature pyramid network can take into account both deep and shallow features, thereby achieving highly accurate recognition of human pose.

[0017] It is understandable that the beneficial effects of aspects two through five can be found in the relevant descriptions in aspect one, and will not be repeated here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the implementation process of the human pose estimation method provided in the embodiments of this application;

[0020] Figure 2 This is an example diagram showing the location and numbering of key points on the human body provided in the embodiments of this application;

[0021] Figure 3 This is a data flow example diagram of the attitude estimation model provided in the embodiments of this application;

[0022] Figure 4 This is another data flow example diagram of the attitude estimation model provided in the embodiments of this application;

[0023] Figure 5 This is a structural block diagram of the human pose estimation device provided in the embodiments of this application;

[0024] Figure 6 This is a schematic diagram of the structure of the smart device provided in the embodiments of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] To illustrate the technical solution proposed in this application, specific embodiments are described below.

[0027] The human pose estimation method proposed in the embodiments of this application is described below. This human pose estimation method can be applied to embedded intelligent devices such as robots, as well as non-embedded intelligent devices such as smartphones. The embodiments of this application do not limit the specific type of intelligent device.

[0028] Please see Figure 1The implementation process of this human pose estimation method is detailed below:

[0029] Step 101: Input the human image to be processed into the trained pose estimation model. The pose estimation model includes a lightweight backbone network and a feature pyramid network.

[0030] The smart device in this embodiment may integrate a camera; alternatively, the smart device may establish a communication connection with a camera or other devices integrated with a camera, without limitation herein. When the smart device integrates a camera, it can capture images through the camera and, upon determining that the image contains human information, use the image as a human image to be processed. When the smart device establishes a communication connection with a camera or other devices integrated with a camera, it can receive images sent by the camera or other device through the communication connection and, upon determining that the image contains human information, use the image as a human image to be processed.

[0031] The lightweight backbone network in this embodiment can be obtained by adjusting an existing lightweight network. The existing lightweight network can be MobileNet or ShuffleNet, etc., and the version is not limited, such as MobileNetV3. The adjustment to this existing lightweight network can be: removing the last few layers of the lightweight network, retaining only the convolutional layers used for feature extraction. It can be understood that the earlier convolutional layers extract shallow features, and the later convolutional layers extract deep features; that is, the backbone network can extract features from multiple levels in the human image to be processed.

[0032] The feature pyramid network in this embodiment can fuse features from different levels extracted by the backbone network. Its operation can be understood as combining shallow and deep features from the backbone network with upsampling. This embodiment does not modify the network framework of the feature pyramid network; only the number of channels needs to be determined according to specific requirements. It is understood that richer feature information can be obtained through the feature pyramid network.

[0033] Step 102: Obtain the first estimation result based on the deepest features output by the backbone network. The first estimation result is used to indicate whether each human key point is visible in the human image to be processed.

[0034] Human body keypoints include the eyes, nose, ears, and various joints. Specifically, the pose estimation model uses the COCO dataset during training. Based on this dataset, the locations of each human body keypoint are as follows: Figure 2As shown, the keypoints used in this dataset are numbered and named as follows: {0-nose, 1-Leye, 2-Reye, 3-Lear, 4-Rear, 5-Lshoulder, 6-Rshoulder, 7-Lelbow, 8-Relbow, 9-Lwrist, 10-Rwrist, 11-Lhip, 12-Rhip, 13-Lknee, 14-Rknee, 15-Lankle, 16-Rankle}. That is, in this embodiment, the number of keypoints considered is 17.

[0035] Considering that in real-world applications, parts of the human body may be obscured by foreground objects and therefore, human pose estimation tasks need to predict whether key points on the human body are visible. Based on this, in this embodiment, a first estimation result can be obtained from the features output by the last convolutional layer of the backbone network (i.e., the deepest layer features). This first estimation result specifically represents the visibility flag of each key point on the human body, denoted by the parameter v. As an example, a visibility flag of "1" indicates that the corresponding key point is not visible; a visibility flag of "2" indicates that the corresponding key point is visible. In other words, the first estimation result indicates whether each key point on the human body is visible in the image of the human body to be processed.

[0036] Step 103: Obtain the second estimation result based on the heat map and offset map output by the feature pyramid network. The second estimation result is used to indicate the coordinates of each human key point in the human image to be processed.

[0037] After fusing shallow and deep features from the backbone network and performing layer-by-layer upsampling, the feature pyramid network ultimately outputs heatmaps and offset maps. The heatmaps predict the location information of each keypoint in the human image being processed, specifically the pixel-level location of each keypoint. The offset maps predict the local offset from each output feature pixel to the precise sub-pixel location of each keypoint, that is, the sub-pixel local offset of the pixel-level location of each human keypoint.

[0038] When the backbone network downsamples features, quantization errors can occur. For example, the positions of x=0 and x=2 in the 192*192 features output by a convolutional layer of the backbone network will both be mapped to x=0 in the 48*48 features output by the next convolutional layer. Furthermore, the heatmap output by the feature pyramid network predicts pixel-level positions. Therefore, determining the position of each human keypoint in the human image to be processed solely based on the heatmap is likely to result in errors. Based on this, the pose estimation model in this embodiment combines the heatmap and sub-pixel-level offset information (i.e., offset map) to determine the precise position of each human keypoint in the human image to be processed.

[0039] Taking a 224×224×3 human body image as an example, please refer to [link / reference]. Figure 3 , Figure 3 A data flow example of the pose estimation model is given. The first estimation result is 17×1 data, which is the visible marker v corresponding to 17 human body key points; the second estimation result is 17×2 data, which is the (x,y) coordinates of the 17 human body key points in the human body image to be processed.

[0040] In some embodiments, when the output of the pose estimation model has a large error, the location of the problem can be located using heatmaps and offset maps. It can be understood that problems with large errors in the output can be simply classified into the following situations: (1) too many / too few human keypoints; (2) unreasonable positions of human keypoints; (3) failure to identify human keypoints (i.e., no human keypoints in the output). Corresponding to the above three situations, the heatmap may be: (1) the heatmap is significantly missing / too many pixels; (2) similar pixels exist in the background or other locations of the heatmap, or the highlighted areas are diffused, leading to incorrect highlighting and thus incorrect human keypoint positions; (3) there are no highlights in the heatmap, which is most likely caused by network training failure (non-convergence). After locating the problematic keypoint area based on the heatmap, the problematic pixel can be accurately located using an offset map.

[0041] In some embodiments, when the backbone network undergoes multiple convolutional processes, some shallow features may be lost. Although the feature pyramid network fuses deep and shallow features to output heatmaps and offset maps, it still cannot guarantee that the heatmaps and offset maps will not lose shallow features. Based on this, the pose estimation model can further compensate the heatmaps and offset maps with the shallow features output by the backbone network. The process of determining the second estimation result is as follows: after fusing (concat) the heatmap, offset map, and shallow features output by the backbone network, the second estimation result is obtained based on the fused features.

[0042] exist Figure 3Based on this, we optimize and obtain Figure 4 Please see. Figure 4 , Figure 4 An example of the data flow for the attitude estimation model is given when shallow features are used to compensate for the heatmap and offset map.

[0043] In some embodiments, the lightweight backbone network has a shallow structure, with some convolutional layers having small receptive fields. To expand the receptive field, improve the backbone network's sensitivity to features, and help the pose estimation model achieve better performance, this embodiment introduces an attention mechanism for the convolutional layers of the backbone network. The focus is on convolutional layers with small receptive fields, which are often applied in the shallow layers of the lightweight backbone network. This attention mechanism can also be introduced into deeper convolutional layers of the backbone network, but its effect on improving model accuracy is limited, and it may even lead to a decrease in model accuracy.

[0044] It's understandable that any convolutional layer with a small receptive field can have its receptive field expanded using this attention mechanism. In practice, this simply involves adding a call to the function implementing the attention mechanism at the corresponding code location. The implementation process of this attention mechanism for each convolutional layer is as follows:

[0045] A1, calculate the global features of the original input features of the convolutional layer on each feature channel.

[0046] Let X be the original input feature of the convolutional layer. Then, the pose estimation model can first calculate the transformation feature of the original input feature X in each feature channel using the following equation (1):

[0047] u c =k c *X (1)

[0048] Where, k c This represents the filter kernel corresponding to feature channel c; u c This represents the transformation feature of the original input feature X on feature channel c. It is understood that the filter kernels corresponding to different feature channels can be the same or different; this is not limited here.

[0049] After obtaining the transformed features of the original input feature X in each feature channel, a global average pooling operation is performed on any transformed feature to encode the entire spatial feature of the corresponding feature channel into a channel-level global feature. The process of this global average pooling operation can be represented by the following equation (2):

[0050]

[0051] Among them, z cThe global feature of feature channel c is represented; H and W represent the dimensions of the transformed feature, where H represents the height and W represents the width; i represents the x-coordinate and j represents the y-coordinate; u c (i,j) is used to represent the point with coordinates (i,j) in the transformed feature. At this point, the global feature for each feature channel can be obtained.

[0052] A2, calculate the recalibration coefficients for each feature channel based on all global features.

[0053] To obtain the relationship between different feature channels and learn the nonlinear relationship between them, a gating mechanism can be used to calculate the recalibration coefficients of each feature channel. These recalibration coefficients can be understood as weights, used to express the importance of the corresponding feature channel. The calculation process of these recalibration coefficients is shown in equation (3) below:

[0054] s=σ(W2δ(W1z)) (3)

[0055] Where z represents the global features of all feature channels; s represents the set of recalibration coefficients corresponding to each feature channel; δ represents the ReLU activation function; and σ represents the sigmoid activation function. and All represent 1×1 fully connected convolution operations, where r represents the dimensionality reduction coefficient and C represents the total number of feature channels in the current convolutional layer. As shown in equation (3), in this step, firstly, the global features of all feature channels are processed by W1; then, the result of the first convolution is processed by δ; next, the result of the first activation is processed by W2; finally, the result of the second convolution is processed by σ, thus obtaining the recalibration coefficients corresponding to each feature channel. In the above process, the first convolution W1 is used to reduce dimensionality, the second convolution W2 is used to restore dimensionality, and the first activation δ and the second activation σ are used to learn the nonlinear relationships between different feature channels. The above process reduces the complexity of the pose estimation model and improves its generalization ability.

[0056] A3, based on each recalibration coefficient, recalibrates the original input features on each feature channel to obtain new input features for the convolutional layer.

[0057] By performing channel-wise multiplication, the original input features can be recalibrated along the channel dimension, as shown in equation (4) below:

[0058]

[0059] Among them, s c The recalibration coefficients of the characteristic channel c are obtained through equation (3) above; u cThe transformation characteristics of the feature channel c are obtained through the above equation (1); This represents the features of the recalibrated feature channel c. By reassembling the features of each recalibrated feature channel according to the channel order, the new input features of the convolutional layer can be obtained. It can be understood that the subsequent inputs to this convolutional layer are no longer the original input features, but rather the new input features. In other words, the subsequent operations of this convolutional layer will be performed using these new input features.

[0060] The attention mechanism proposed in this application can be viewed as learning the weight coefficients of each feature channel, thereby making the pose estimation model more discriminative of the features of each feature channel. Through this attention mechanism, the backbone network can focus on the features of feature channels with more information and suppress the features of unimportant feature channels, thus compensating for the lack of receptive field in lightweight backbone networks.

[0061] In some embodiments, to further improve the accuracy of the pose estimation model, the activation function used by the backbone network can be optimized. As an example only, the activation function used by the backbone network (i.e., the optimized activation function) can be expressed as follows (5):

[0062]

[0063] Where x represents the input of the activation function used by the backbone network, and Activation(x) represents the output of the activation function used by the backbone network. The form of the activation function used by the backbone network is obtained by modifying the swish activation function, which can improve accuracy and ensure that the processing in the embedded environment does not increase the computation time. The above equation (5) also involves the ReLU6 activation function, which can be represented by the following equation (6):

[0064] Relu6(y)=min(6,max(0,y)) (6)

[0065] Here, y represents the input of the Relu6 activation function, and Relu6(y) represents the output of the Relu6 activation function. It can be understood that if the Relu6 activation function were not used in equation (5), but instead a Relu activation function, the output range of the Relu activation function would be (0, +∞), which would lead to precision loss when using low-precision float16 calculations on mobile devices. Therefore, equation (5) uses the Relu6 activation function to limit the maximum output and avoid precision loss.

[0066] In some embodiments, during the training phase of the pose estimation model, the bone loss function can be used to calculate the loss of the heatmap and offset map. The bone loss function can be expressed as shown in the following equation (7):

[0067]

[0068] Where P represents the predicted human keypoint locations (i.e., the coordinates of the human keypoints in the sample images predicted by the pose estimation model during training); G represents the true human keypoint locations (i.e., the true coordinates of the human keypoints in the sample images, which can be obtained from the label data of the sample images); S represents a pre-defined set of human keypoint pairs; i,j∈S indicates that i and j are two different human keypoints corresponding to a human keypoint pair in the set. As an example only, the human keypoint pairs in the set are specifically composed of two highly correlated human keypoints. For example... Figure 2 As shown, the left eye (number 1) and the left ear (number 3) are in Figure 2 The two are directly connected and are highly correlated human keypoints, which can form a human keypoint pair; and so on. As an example only, the set of human keypoint pairs may include the following human keypoint pairs: {0-1, 0-2, 1-3, 2-4, 5-6, 5-7, 5-11, 6-8, 6-12, 7-9, 8-10, 11-12, 11-13, 12-14, 13-15, 14-16}.

[0069] It's understandable that the distance between two highly correlated human keypoints tends to remain relatively stable, while the distance between two low-correlation human keypoints is likely to fluctuate dramatically. For example, a person's hand can be raised or lowered; therefore, the distance between the elbow keypoint and the hip keypoint may change drastically, while the distance between the elbow keypoint and the wrist keypoint does not change with the raising or lowering of the hand. Therefore, the elbow and wrist keypoints are highly correlated human keypoints, while the elbow and hip keypoints are low-correlation human keypoints. During training, the set of human keypoint pairs is primarily constructed based on highly correlated human keypoints.

[0070] The skeletal loss function proposed in this application constrains the distance between highly correlated human key points, which can prevent the pose estimation model from predicting unreasonable poses and help improve the convergence speed of the pose estimation model.

[0071] In some embodiments, during the training phase of the pose estimation model, there may be insufficient sample image data or highly similar data distributions, which can affect the generalization ability and robustness of the pose estimation model. Therefore, to reduce this situation, data augmentation processing of the sample images during the training phase can be considered. The specific process is as follows:

[0072] B1, obtain the original sample image.

[0073] As described earlier, the COCO dataset can be used to train the pose estimation model, which will not be repeated here. The label data for each sample image includes: the ID label of the human keypoint, the coordinate label of the human keypoint, and the visibility marker label of the human keypoint.

[0074] B2 performs data augmentation on the original sample image to obtain the processed sample image.

[0075] As examples only, data augmentation processes include: cutout, random flipping, random rotation, random scaling, random translation, random jittering, and pixel transformation, etc., which are not limited here. It should be noted that due to the special nature of human pose estimation, for random flipping, vertical flipping cannot be used on the target (as this would cause the human body to be upside down), only mirror flipping (i.e., horizontal flipping) can be used on the target.

[0076] B3. Based on data augmentation processing, update the label data of the original sample image to obtain the label data of the processed sample image.

[0077] As an example only, after cropping any sample image, if the randomly cropped area includes information about human key points, the visibility flag corresponding to the cropped human key points can be updated from "visible" (i.e., v=2) to "invisible" (i.e., v=1).

[0078] As an example, after mirroring any sample image, the labels of the corresponding human keypoints on the left and right sides can be changed. For instance, after mirroring a target in a sample image, the labels of the keypoints on the left side need to be updated to those on the right side; for example, 1-Leye becomes 2-Reye, and 5-Lshoulder becomes 6-Rshoulder. Similarly, the labels of the keypoints on the right side need to be updated to those on the left side. This will not be elaborated further here.

[0079] It is understandable that for each type of data augmentation processing that affects the position and visibility of targets in the sample image, the corresponding update operation of the label data can be performed, which will not be elaborated here.

[0080] B4. The pose estimation model is trained based on the processed sample images and their label data.

[0081] That is, the sample images used during training are no longer the original sample images, but the processed sample images; correspondingly, when calculating the loss of the pose estimation model, the labels used are no longer the label data of the original sample images, but the label data of the processed sample images.

[0082] The above process enriches the training samples of the pose estimation model, thereby improving its generalization ability and robustness.

[0083] As can be seen from the above, the embodiments of this application use a trained pose estimation model to estimate various human key points in the human image to be processed. It can be understood that once the positions of the human key points are determined, the human pose is also determined. Specifically, the pose estimation model mainly consists of two components: a lightweight backbone network and a feature pyramid network. The backbone network is used to extract features from multiple levels in the human image to be processed. Because the backbone network is lightweight, the feature extraction speed is improved, thereby achieving rapid recognition of human pose. The feature pyramid network is used to fuse the features extracted from different levels by the backbone network, so that the heatmap and offset map output by the feature pyramid network can take into account both deep and shallow features, thereby achieving highly accurate recognition of human pose.

[0084] Corresponding to the human pose estimation method provided above, this application also provides a human pose estimation device. For example... Figure 5 As shown, the human posture estimation device 5 includes:

[0085] The input module 501 is used to input the human image to be processed into the trained pose estimation model. The pose estimation model includes a feature pyramid network and a lightweight backbone network. The backbone network is used to extract features at multiple levels in the human image to be processed, and the feature pyramid network is used to fuse the features at different levels extracted by the backbone network.

[0086] The first determining module 502 is used to obtain a first estimation result based on the deepest features output by the backbone network. The first estimation result is used to indicate whether each human key point is visible in the human image to be processed.

[0087] The second determining module 503 is used to obtain a second estimation result based on the heat map and offset map output by the feature pyramid network. The second estimation result is used to indicate the coordinates of each human key point in the human image to be processed. The heat map is used to predict the pixel-level position of each human key point, and the offset map is used to predict the local offset of the pixel-level position of each human key point at the sub-pixel level.

[0088] In some embodiments, the human pose estimation device 500 further includes: an attention mechanism module, used to perform attention mechanism operations on a specified convolutional layer; the attention mechanism module includes:

[0089] The first computing unit is used to calculate the global features of the original input features of the convolutional layer in each feature channel;

[0090] The second calculation unit is used to calculate the recalibration coefficients of each feature channel based on all global features;

[0091] The recalibration unit is used to recalibrate the original input features on each feature channel according to each recalibration coefficient, so as to obtain new input features for the convolutional layer.

[0092] In some embodiments, the first computing unit includes:

[0093] The first computational subunit is used to multiply the filter kernel corresponding to each feature channel with the original input feature to obtain the transformed feature of the feature channel.

[0094] The second computational subunit is used to perform global average pooling on the transformed features of the feature channels to obtain the global features of the original input features in the feature channels.

[0095] In some embodiments, the second computation unit is specifically used to sequentially perform a first convolution process, a first activation process, a second convolution process, and a second activation process on all global features to obtain the recalibration coefficients of each feature channel. The first convolution process is used to reduce the dimensionality, the second convolution process is used to restore the dimensionality, and the first and second activation processes are used to learn the nonlinear relationship between different feature channels.

[0096] In some embodiments, the activation function used by the backbone network is:

[0097]

[0098] Here, x represents the input to the activation function, and Activation(x) represents the output of the activation function.

[0099] In some embodiments, the loss function used by the pose estimation model during training for the heatmap and offset map is:

[0100]

[0101] Where P is the human keypoint location predicted by the pose estimation model; G is the actual human keypoint location; S is a preset set of human keypoint pairs; i,j∈S is used to indicate that i and j are two different human keypoints corresponding to a human keypoint pair in the set of human keypoint pairs.

[0102] In some embodiments, the human pose estimation device 500 further includes a training module; the training module includes:

[0103] The acquisition unit is used to acquire the original sample image;

[0104] The data augmentation processing unit is used to perform data augmentation processing on the original sample image to obtain the processed sample image;

[0105] The label data update unit is used to update the label data of the original sample image according to the data augmentation process, so as to obtain the label data of the processed sample image.

[0106] The training unit is used to train the pose estimation model based on the processed sample images and their label data.

[0107] As can be seen from the above, the embodiments of this application use a trained pose estimation model to estimate various human key points in the human image to be processed. It can be understood that once the positions of the human key points are determined, the human pose is also determined. Specifically, the pose estimation model mainly consists of two components: a lightweight backbone network and a feature pyramid network. The backbone network is used to extract features from multiple levels in the human image to be processed. Because the backbone network is lightweight, the feature extraction speed is improved, thereby achieving rapid recognition of human pose. The feature pyramid network is used to fuse the features extracted from different levels by the backbone network, so that the heatmap and offset map output by the feature pyramid network can take into account both deep and shallow features, thereby achieving highly accurate recognition of human pose.

[0108] Corresponding to the human pose estimation method provided above, this application also provides an intelligent device. Please refer to... Figure 6 The smart device 6 in this embodiment includes: a memory 601, and one or more processors 602. Figure 6Only one is shown in the image, along with a computer program stored in memory 601 and executable on the processor. Memory 601 stores software programs and units. The processor 602 executes various functional applications and data processing by running the software programs and units stored in memory 601 to obtain resources corresponding to preset events. Specifically, the processor 602 performs the following steps when running the computer program stored in memory 601:

[0109] The human image to be processed is input into a trained pose estimation model, which includes a feature pyramid network and a lightweight backbone network. The backbone network is used to extract features from multiple levels in the human image to be processed, and the feature pyramid network is used to fuse the features extracted from different levels by the backbone network.

[0110] The first estimation result is obtained based on the deepest features output by the backbone network. The first estimation result is used to indicate whether each human key point is visible in the human image to be processed.

[0111] The second estimation result is obtained based on the heatmap and offset map output by the feature pyramid network. The second estimation result is used to indicate the coordinates of each human key point in the human image to be processed. The heatmap is used to predict the pixel-level position of each human key point, and the offset map is used to predict the local offset of the pixel-level position of each human key point at the sub-pixel level.

[0112] Assuming the first possible implementation, in the second possible implementation provided as a basis, an attention mechanism is introduced into the specified convolutional layers of the backbone network; for each specified convolutional layer, the process of the attention mechanism includes:

[0113] Calculate the global features of the original input features of the convolutional layer in each feature channel;

[0114] Calculate the recalibration coefficients for each feature channel based on all global features;

[0115] Based on each recalibration coefficient, the original input features are recalibrated on each feature channel to obtain new input features for the convolutional layer.

[0116] In a third possible implementation provided based on the second possible implementation, global features of the original input features in each feature channel are calculated based on the original input features of the convolutional layer, including:

[0117] For each feature channel, the filter kernel corresponding to the feature channel is multiplied by the original input feature to obtain the transformed feature of the feature channel;

[0118] Global average pooling is performed on the transformed features of the feature channels to obtain the global features of the original input features in the feature channels.

[0119] In a fourth possible implementation provided based on the second possible implementation, the recalibration coefficients for each feature channel are calculated based on all global features, including:

[0120] All global features are sequentially processed by first convolution, first activation, second convolution, and second activation to obtain the recalibration coefficients of each feature channel. The first convolution is used to reduce dimensionality, the second convolution is used to restore dimensionality, and the first and second activations are used to learn the nonlinear relationships between different feature channels.

[0121] In a fifth possible implementation based on the first possible implementation, the activation function used by the backbone network is:

[0122]

[0123] Here, x represents the input to the activation function, and Activation(x) represents the output of the activation function.

[0124] In a sixth possible implementation provided based on the first possible implementation, the loss function used by the attitude estimation model during training for the heatmap and offset map is:

[0125]

[0126] Where P is the human keypoint location predicted by the pose estimation model; G is the actual human keypoint location; S is a preset set of human keypoint pairs; i,j∈S is used to indicate that i and j are two different human keypoints corresponding to a human keypoint pair in the set of human keypoint pairs.

[0127] In a seventh possible implementation based on the first possible implementation, the processor 602 further performs the following steps when running a computer program stored in the memory 601:

[0128] Obtain the original sample image;

[0129] The original sample image is subjected to data augmentation processing to obtain the processed sample image;

[0130] Based on the data augmentation process, the label data of the original sample image is updated to obtain the label data of the processed sample image;

[0131] The pose estimation model is trained based on the processed sample images and their label data.

[0132] It should be understood that, in the embodiments of this application, the processor 602 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0133] Memory 601 may include read-only memory and random access memory, and provides instructions and data to processor 602. Some or all of memory 601 may also include non-volatile random access memory. For example, memory 601 may also store device category information.

[0134] As can be seen from the above, the embodiments of this application use a trained pose estimation model to estimate various human key points in the human image to be processed. It can be understood that once the positions of the human key points are determined, the human pose is also determined. Specifically, the pose estimation model mainly consists of two components: a lightweight backbone network and a feature pyramid network. The backbone network is used to extract features from multiple levels in the human image to be processed. Because the backbone network is lightweight, the feature extraction speed is improved, thereby achieving rapid recognition of human pose. The feature pyramid network is used to fuse the features extracted from different levels by the backbone network, so that the heatmap and offset map output by the feature pyramid network can take into account both deep and shallow features, thereby achieving highly accurate recognition of human pose.

[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of functional units and modules is only used as an example. In practical applications, functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0136] In the embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of apparatus or units, and may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the embodiments of this application can also be implemented by a computer program instructing associated hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of each method embodiment. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer-readable storage devices, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0141] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for estimating human pose, characterized in that, include: The human image to be processed is input into a trained pose estimation model, which includes a feature pyramid network and a lightweight backbone network. The backbone network is used to extract features at multiple levels from the human image to be processed, and the feature pyramid network is used to fuse the features at different levels extracted by the backbone network. A first estimation result is obtained based on the deepest features output by the backbone network. The first estimation result is used to indicate whether each human key point is visible in the human image to be processed, specifically the visibility flag of each human key point. The second estimation result is obtained based on the heatmap and offset map output by the feature pyramid network, including: fusing the heatmap, the offset map and the shallow features output by the backbone network, and obtaining the second estimation result based on the fused features. The second estimation result is used to indicate the coordinates of each human key point in the human image to be processed. The heatmap is used to predict the pixel-level position of each human key point, and the offset map is used to compensate for the quantization error caused by the downsampling of the backbone network, so as to predict the local offset of the pixel-level position of each human key point at the sub-pixel level. The lightweight backbone network retains only convolutional layers used for feature extraction, and a specific attention mechanism is introduced into the designated convolutional layers of the backbone network; for each designated convolutional layer, the attention mechanism includes the following steps: Calculate the global features of the original input features of the specified convolutional layer in each feature channel; Based on all the global features, calculate the recalibration coefficients for each of the feature channels; Based on each of the recalibration coefficients, the original input features are recalibrated on each of the feature channels to obtain new input features for the specified convolutional layer.

2. The human pose estimation method as described in claim 1, characterized in that, The calculation of the global features of the original input features of the convolutional layer in each feature channel includes: For each feature channel, the filter kernel corresponding to the feature channel is multiplied by the original input feature to obtain the transformed feature of the feature channel; A global average pooling operation is performed on the transformed features of the feature channels to obtain the global features of the original input features in the feature channels.

3. The human pose estimation method as described in claim 1, characterized in that, The step of calculating the recalibration coefficients for each feature channel based on all the global features includes: All global features are sequentially subjected to a first convolution, a first activation, a second convolution, and a second activation to obtain recalibration coefficients for each feature channel. The first convolution is used to reduce dimensionality, the second convolution is used to restore dimensionality, and the first and second activations are used to learn the nonlinear relationships between different feature channels.

4. The human pose estimation method as described in claim 1, characterized in that, The activation function used in the backbone network is: Where x represents the input of the activation function, and Activation(x) represents the output of the activation function.

5. The human pose estimation method as described in claim 1, characterized in that, The loss function used by the attitude estimation model during training for the heatmap and the offset map is: Wherein, P is the human keypoint position predicted by the posture estimation model; G is the actual human keypoint position; S is a preset set of human keypoint pairs; i,j∈S is used to indicate that i and j are two different human keypoints corresponding to a human keypoint pair in the set of human keypoint pairs.

6. The human pose estimation method as described in claim 1, characterized in that, The training process of the pose estimation model includes: Obtain the original sample image; The original sample image is subjected to data augmentation processing to obtain the processed sample image; Based on the data augmentation process, the label data of the original sample image is updated to obtain the label data of the processed sample image; The pose estimation model is trained based on the processed sample images and the label data of the processed sample images.

7. A human posture estimation device, characterized in that, include: The input module is used to input the human image to be processed into a trained pose estimation model. The pose estimation model includes a feature pyramid network and a lightweight backbone network. The backbone network is used to extract features at multiple levels from the human image to be processed. The feature pyramid network is used to fuse the features at different levels extracted by the backbone network. The lightweight backbone network only retains the convolutional layers used for feature extraction. The first determining module is used to obtain a first estimation result based on the deepest features output by the backbone network. The first estimation result is used to indicate whether each human key point is visible in the human image to be processed, specifically the visibility flag of each human key point. The second determining module is used to obtain a second estimation result based on the heatmap and offset map output by the feature pyramid network, including: fusing the heatmap, the offset map and the shallow features output by the backbone network, and obtaining a second estimation result based on the fused features. The second estimation result is used to indicate the coordinates of each human key point in the human image to be processed. The heatmap is used to predict the pixel-level position of each human key point, and the offset map is used to compensate for the quantization error caused by the downsampling of the backbone network, so as to predict the local offset of the pixel-level position of each human key point at the sub-pixel level. The attention mechanism module is used to perform attention mechanism operations on a specified convolutional layer; the attention mechanism module includes: The first computing unit is used to calculate the global features of the original input features of the specified convolutional layer in each feature channel; The second calculation unit is used to calculate the recalibration coefficients of each of the feature channels based on all the global features; The recalibration unit is used to recalibrate the original input features on each of the feature channels according to the recalibration coefficients, so as to obtain the new input features of the specified convolutional layer.

8. A smart device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images