A single-stage real-time multi-person pose estimation method

Abstract

The present invention relates to a single-stage real-time multi-person pose estimation method, comprising: inputting an image to be detected into a backbone network, outputting a multi-level feature map; selecting the final three-level features from the multi-level feature map, performing cross-scale feature fusion, and retaining the final two-level fused features; inputting the two-level fused features into a prediction head, respectively, to predict the foreground confidence of each grid point, the bounding box, key point visibility, and pose representation of the human body instance corresponding to each grid point; filtering the predicted data to determine the filtered grid points; and processing the bounding boxes and pose representations corresponding to the filtered grid points by a key point heat map decoding module to obtain one-dimensional heat maps in the horizontal and vertical directions, which are used to represent the probability of coordinate categories in each direction. Compared with the existing technology, the present invention combines coordinate classification technology with a single-stage pose estimation framework, capable of achieving real-time multi-person pose estimation that balances speed and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a single-stage real-time multi-person posture estimation method. Background Art

[0002] Multi-person pose estimation is an important problem in computer vision. The goal is to detect multiple people in an image or video and estimate the locations of key points of each person, such as the head, hands, and feet. Multi-person pose estimation has many applications, such as augmented reality, motion analysis, and human-computer interaction.

[0003] Current mainstream real-time multi-person pose estimation methods can be divided into two-stage top-down methods and single-stage methods. The top-down method first uses a detector to generate bounding boxes for each person instance, and then performs pose estimation separately within each bounding box. Its advantage is high accuracy, but the inference time increases linearly with the number of people in the image.

[0004] Single-stage methods predict at one time whether each position in the image is a human instance or background, as well as the pose of the human instance. The inference time does not change with the increase in the number of people. For example, the YOLO-Pose algorithm evenly sets grid points on the image. At each grid point, the model predicts the confidence that the grid point position belongs to the foreground (human body), as well as the corresponding bounding box of the human body and the position of each key point. The positioning of the human key points is obtained through regression. That is, the model directly outputs the coordinate difference between each grid point and each key point, and uses weighted square loss to supervise the learning of the coordinate difference. However, this regression method has low accuracy.

[0005] In summary, the existing two-stage top-down method has a linear increase in inference time as the number of people in the image increases, making it difficult to achieve real-time inference requirements when there are a large number of people. Although the single-stage method can achieve real-time speed, its prediction accuracy is difficult to meet the requirements because it uses a simple regression method to predict the coordinates of key points. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a single-stage real-time multi-person pose estimation method, which combines coordinate classification technology with a single-stage pose estimation framework to achieve real-time multi-person pose estimation that takes into account both speed and accuracy.

[0007] The purpose of the present invention can be achieved by the following technical solutions: A single-stage real-time multi-person posture estimation method comprises the following steps:

[0008] S1. Input the image to be detected into the backbone network and output a multi-level feature map;

[0009] S2, select the final three-level features from the multi-level feature map, perform cross-scale feature fusion, and retain the final two-level fusion features;

[0010] S3. Input the two-level fusion features into the prediction head respectively to predict the foreground confidence of each grid point, the bounding box, key point visibility and posture representation of the human instance corresponding to each grid point;

[0011] S4, screening the predicted data to determine the screened grid points;

[0012] S5. Based on the bounding box and posture representation corresponding to the filtered grid points, the key point heat map decoding module processes the bounding box and posture representation to obtain a one-dimensional heat map in the horizontal and vertical directions, which is used to represent the probability of the coordinate category in each direction.

[0013] Furthermore, the prediction head in step S3 includes two types of convolutional networks: a normal convolutional network and a group convolutional network, wherein the normal convolutional network is used to predict the foreground confidence of each grid point;

[0014] The group of convolutional networks is used to predict the bounding box, key point visibility and posture representation of the human instance corresponding to each grid point.

[0015] Furthermore, the specific process of step S4 is as follows:

[0016] S41, using a preset threshold, filtering out grid points whose foreground confidence values are less than the preset threshold;

[0017] S42. Combine the bounding boxes corresponding to the grid points and use a non-maximum suppression algorithm to determine the filtered grid points.

[0018] Furthermore, the working process of the key point heat map decoding module in step S5 is as follows:

[0019] S51, performing expansion processing on the bounding box of each grid point;

[0020] S52, then divide the expanded external frame into B in the horizontal and vertical directions respectively. x , B y fragments, each fragment is a class;

[0021] S53, converting the coordinates of each class into a position encoding vector through sinusoidal position encoding, and performing linear transformation through a fully connected layer as the representation vector of the class;

[0022] S54. For the posture representation of each grid point, first generate the representation vectors of K key points through the fully connected layer and matrix transformation, perform inner product operation on each key point vector with the horizontal and vertical category representation matrices respectively, and after a Softmax normalization operation, obtain the probabilities of each category in the horizontal and vertical directions, that is, generate two one-dimensional heat maps.

[0023] Furthermore, the step S53 specifically includes the following steps:

[0024] S531, calculating the coordinate value of each category;

[0025] S532, generating a sinusoidal position code for each coordinate category, with a length of C;

[0026] S533, use the fully connected layer to perform linear transformation on various position codes. The length of the transformed vector is still C. At this time, the sizes of the coordinate category representation matrices in the horizontal and vertical directions are [B x ,C] and [B y ,C].

[0027] Furthermore, in step S531, the coordinates of the i-th category in the horizontal direction are:

[0028]

[0029] The coordinates of the i-th category in the vertical direction are:

[0030]

[0031] Among them, x l 、x r are the horizontal coordinates of the left and right edges of the expanded external frame, y t 、y b They are the vertical coordinates of the upper and lower edges of the expanded external frame.

[0032] Furthermore, in step S532, the position encoding vector of the i-th category in the horizontal direction is:

[0033]

[0034] Where t represents the temperature coefficient, sin and cos are the sine and cosine functions respectively. The general formula for the above position coding is:

[0035]

[0036] Among them, c represents the subscript of the position coding vector, and its value range is [0, C-1]. The position coding generation method of each category in the vertical direction is consistent with that in the horizontal direction.

[0037] Furthermore, the step S54 specifically includes the following steps:

[0038] S541, after the posture representation vector passes through a fully connected layer and matrix deformation operation, its matrix shape is [K, C], where K is the number of key points;

[0039] S542. Using the matrix obtained by processing with the self-attention module to enhance the structural consistency of the key points of the human body instance, the output of the self-attention module is a key point representation with a shape of [K, C].

[0040] S543, perform matrix multiplication on the key point representation and the transposed horizontal and vertical coordinate category representation matrices to obtain the shapes of [K, B x ] and [K,B y ] activation matrix;

[0041] S544. For each key point, record its activation vectors in the horizontal and vertical directions as follows:

[0042]

[0043]

[0044] Use the softmax function to normalize the activation vector to obtain the probability vector:

[0045]

[0046]

[0047] These two probability vectors are the one-dimensional heat maps of the key point in the horizontal and vertical directions;

[0048] S545. Calculate the key point coordinates by weighted summation.

[0049] Furthermore, the key point coordinates in step S545 are specifically:

[0050]

[0051]

[0052] in, are the horizontal and vertical coordinates of the key points respectively.

[0053] Furthermore, the key point heat map decoding module uses a loss function based on maximum likelihood estimation to supervise the learning of the one-dimensional heat map. The loss function based on maximum likelihood estimation is specifically:

[0054]

[0055] Among them, μ x is the horizontal coordinate value of the manually annotated key point, s represents the size of the human instance, and σ is the variance of the manual annotation error distribution predicted by the model.

[0056] Compared with the prior art, the present invention has the following advantages:

[0057] Based on the single-stage posture estimation framework, the present invention introduces coordinate classification to locate key points to replace the direct regression coordinate method. For each grid point evenly set on the image, the probability that the grid point position belongs to the foreground (human body), the corresponding bounding box and posture representation of the human body are simultaneously generated. A key point heat map decoding module is designed to dynamically adjust the one-dimensional heat map range of the horizontal and vertical axes according to the bounding box of each instance, divide the coordinates into instance areas, and use heat maps instead of regression to locate key points, which can greatly improve computational efficiency and at the same time improve positioning accuracy to a level close to that of the two-stage method.

[0058] The present invention builds a model based on the backbone network of the single-stage posture estimation framework and uses a two-level feature map, which can appropriately reduce the resolution and increase the receptive field, thereby ensuring the detection range and further saving computational overhead.

[0059] In order to supervise the learning of one-dimensional heat maps, this paper proposes a new loss function based on maximum likelihood estimation to learn the uncertainty of coordinate errors (i.e., the variance of the coordinate error distribution) and enhance the adaptability to different sample difficulties. When the key points are very difficult, the model will predict a larger variance, reducing the learning difficulty; when the key points are relatively simple, the model will predict a smaller variance, helping to make the one-dimensional heat map positioning more accurate.

[0060] In the process of calculating the coordinate classification heat map, the present invention uses the position encoding of each category coordinate value to dynamically generate the representation vector of the category, which can enhance the consistency between the representation vector of each category and the semantics of the category and improve the accuracy of coordinate category probability prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the method flow of the present invention;

[0062] Figure 2 Schematic diagram of the application framework of the embodiment. DETAILED DESCRIPTION

[0063] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] Example

[0065] like Figure 1 As shown, a single-stage real-time multi-person pose estimation method includes the following steps:

[0066] S1. Input the image to be detected into the backbone network and output a multi-level feature map;

[0067] S2, select the final three-level features from the multi-level feature map, perform cross-scale feature fusion, and retain the final two-level fusion features;

[0068] S3. Input the two-level fusion features into the prediction head respectively to predict the foreground confidence of each grid point, the bounding box, key point visibility and posture representation of the human instance corresponding to each grid point;

[0069] S4, screening the predicted data to determine the screened grid points;

[0070] S5. Based on the bounding box and posture representation corresponding to the filtered grid points, the key point heat map decoding module processes the bounding box and posture representation to obtain a one-dimensional heat map in the horizontal and vertical directions, which is used to represent the probability of the coordinate category in each direction.

[0071] This embodiment applies the above technical solution to build Figure 2 In the model architecture shown in the figure, after the image to be detected is input into the backbone network, a multi-level feature map is obtained. The last three-level features are taken for cross-scale feature fusion, and then the last two-level features P4 and P5 are retained. These two feature maps are respectively sent to the prediction head. Each prediction head contains two convolutional networks: a normal convolutional network is used to predict the foreground confidence of each grid point, and a group convolutional network is used to predict the bounding box, key point visibility and posture representation of the human instance corresponding to each grid point.

[0072] The grid points then need to be screened: first, a preset threshold is used to filter out grid points whose foreground confidence values do not meet the requirements; the filtered grid points are then subjected to non-maximum suppression based on the bounding box to determine the filtered grid points, and the bounding box and posture representation are used to decode the key point coordinates for the filtered grid points.

[0073] The bounding box of each grid point will be expanded by 1.25 times, and then the expanded bounding box will be divided into B in the horizontal and vertical directions respectively. x ,B y Each segment represents a class. The coordinates of each class are converted into a position encoding vector using sinusoidal position encoding. After a linear transformation through a fully connected layer, this vector serves as the representation vector for that class. The pose representation of each grid point first passes through a fully connected layer and a matrix transformation to generate a representation vector for K key points (K is the number of key points). Each key point vector is inner-producted with the horizontal and vertical class representation matrices to obtain the horizontal and vertical probabilities for each class, i.e., two one-dimensional heat maps.

[0074] In this example, based on a single-stage object detection framework, the backbone network uses CSPDarknet. The last three layers of feature maps are retained, and cross-scale feature fusion is performed using a hybrid encoder. After fusion, the last two layers of feature maps, P4 and P5, are retained. P4 is downsampled to 16, and P5 to 32. These two levels of features correspond to grid arrays with intervals of 16 and 32 pixels, respectively, on the original image.

[0075] The prediction head at each level contains two convolutional networks - a normal convolutional network is used to predict the foreground confidence of each grid point, and a group convolutional network predicts the bounding box, key point visibility and posture representation of the human instance corresponding to each grid point.

[0076] During training, this embodiment uses the candidate box assignment algorithm SimOTA to assign positive and negative samples. During inference, low-confidence grid points are filtered out based on a threshold, and then a non-maximum suppression algorithm is used to remove duplicate predictions. Finally, the key point heat map and coordinates are decoded.

[0077] The bounding box and pose representation are then decoded by the key point heat map module to generate one-dimensional heat maps in the horizontal and vertical directions, indicating the probability of the coordinate category in each direction. The detailed steps of key point heat map decoding include:

[0078] 1. The bounding box is expanded by 1.25 times. This is to ensure that the human body can be covered even when the bounding box prediction is inaccurate.

[0079] 2. Divide the expanded external frame into B in both horizontal and vertical directions x ,B y fragments, each fragment as a category;

[0080] 3. Calculate the coordinate values of each category. If the horizontal coordinates of the left and right edges of the expanded bounding box are x l ,x r , the vertical coordinates of the upper and lower edges are y t ,y b , then the coordinates of the i-th category in the horizontal direction are:

[0081]

[0082] The coordinates of the i-th category in the vertical direction are:

[0083]

[0084] 4. Generate a sinusoidal position code of length C for each coordinate category. The position code vector of the i-th category in the horizontal direction is:

[0085]

[0086] Where t represents the temperature coefficient, sin and cos are the sine function and cosine function respectively. The general formula of the above position coding is:

[0087]

[0088] Where c represents the subscript of the position encoding vector, and its value range is [0, C-1]. The position encoding generation method for each category in the vertical direction is the same as that in the horizontal direction;

[0089] 5. Use the fully connected layer to perform a linear transformation on the position codes. The length of the transformed vector is still C. At this time, the sizes of the horizontal and vertical coordinate category representation matrices are [B x ,C] and [B y ,C];

[0090] 6. After the pose representation vector passes through a fully connected layer and a matrix deformation operation, its matrix shape is [K, C], where K is the number of key points;

[0091] 7. Use the self-attention module to process the matrix obtained in step 6 to enhance the structural consistency of the key points of the human instance. The output of the self-attention module is a key point representation with the shape [K, C];

[0092] 8. Perform matrix multiplication on the key point representation and the transposed horizontal and vertical coordinate category representation matrices to obtain the shapes [K, B x ] and [K,B y ] activation matrix;

[0093] 9. For each key point, record its horizontal and vertical activation vectors as:

[0094]

[0095]

[0096] Normalize it using the softmax function to get the probability vector:

[0097]

[0098]

[0099] These two probability vectors are the one-dimensional heat maps of the key point in the horizontal and vertical directions;

[0100] 10. The final key point coordinates can be obtained by weighted summation:

[0101]

[0102] In addition, in order to supervise the learning of one-dimensional heat maps, this embodiment also proposes a new loss function based on maximum likelihood estimation:

[0103]

[0104] Among them, μ x is the horizontal coordinate value of the manually annotated keypoint, s represents the size of the human instance, and σ is the variance of the manual annotation error distribution predicted by the model. When the keypoint is difficult, the model predicts a larger variance, reducing the learning difficulty; when the keypoint is simple, the model predicts a smaller variance, which helps the one-dimensional heat map locate more accurately. This mechanism enhances the model's adaptability to samples of varying difficulty.

[0105] To verify the effectiveness of this solution, this embodiment also conducted a comparative test on the human pose estimation dataset COCO. From the test results, the small / medium / large models designed by this solution can achieve an accuracy of 67.6% / 71.5% / 73.3% respectively, which is an average 4% improvement in accuracy compared to YOLO-Pose. The inference speed of the three models on the GPU (graphics processing unit) is 236 / 172 / 141 frames / second respectively, meeting the requirements of real-time inference. In other words, compared with the single-stage pose estimation model YOLO-Pose based on regression methods, this solution introduces coordinate classification technology and locates key points through heat maps, which can improve the positioning accuracy to a level close to that of the two-stage method, while only increasing the amount of computation.

[0106] In summary, this proposal applies coordinate classification technology to single-stage pose estimation, replacing regression with heatmaps to locate keypoints. It also designs an instance-size-based coordinate classification heatmap decoding method to dynamically adjust the coordinate classification range and optimize training and inference efficiency. During the calculation of the coordinate classification heatmap, the positional encoding of each class's coordinate value is used to dynamically generate a representation vector for that class. Furthermore, a loss function based on maximum likelihood estimation is proposed to learn the uncertainty of coordinate errors and enhance adaptability to different samples. In practical applications, other single-stage object detection frameworks can be used as the base network and the proposed coordinate classification and loss function design can be applied. The number of coordinate classification categories can be adjusted to balance accuracy and speed. Different positional encoding methods can be used, such as learned positional encoding. The coordinate error distribution model in the loss function can be adjusted, such as using a Gaussian distribution instead of a Laplace distribution. Furthermore, detailed training and inference parameters of the model, such as image size, batch size, and learning rate, can be adjusted and optimized. The proposed coordinate classification and loss function design can also be applied to other keypoint prediction-based tasks, such as single-person pose estimation and hand keypoint detection, ensuring a balance between speed and accuracy.

Claims

1. A single-stage real-time multi-person pose estimation method, characterized in that: The following steps are involved: S1. Input the image to be detected into the backbone network and output a multi-level feature map; S2, select the final three-level features from the multi-level feature map, perform cross-scale feature fusion, and retain the final two-level fusion features; S3. Input the two-level fusion features into the prediction head respectively to predict the foreground confidence of each grid point, the bounding box, key point visibility and posture representation of the human instance corresponding to each grid point; S4, screening the predicted data to determine the screened grid points; S5. Based on the bounding box and posture representation corresponding to the filtered grid points, the key point heat map decoding module processes the bounding box and posture representation to obtain a one-dimensional heat map in the horizontal and vertical directions, which is used to represent the probability of the coordinate category in each direction; The specific process of step S4 is: S41, using a preset threshold, filtering out grid points whose foreground confidence values are less than the preset threshold; S42, combining the external frame corresponding to the grid point and using the non-maximum suppression algorithm to determine the filtered grid point; The working process of the key point heat map decoding module in step S5 is as follows: S51, performing expansion processing on the bounding box of each grid point; S52, then divide the expanded external frame into B in the horizontal and vertical directions respectively. x , B y fragments, each fragment is a class; S53, converting the coordinates of each class into a position encoding vector through sinusoidal position encoding, and performing linear transformation through a fully connected layer as the representation vector of the class; S54. For the posture representation of each grid point, first generate the representation vectors of K key points through the fully connected layer and matrix transformation, perform inner product operation on each key point vector with the horizontal and vertical category representation matrices respectively, and after a Softmax normalization operation, obtain the probabilities of each category in the horizontal and vertical directions, that is, generate two one-dimensional heat maps.

2. A single-stage real-time multi-person pose estimation method according to claim 1, characterized in that: The prediction head in step S3 includes two convolutional networks: a common convolutional network and a group convolutional network, wherein the common convolutional network is used to predict the foreground confidence of each grid point; The group of convolutional networks is used to predict the bounding box, key point visibility and posture representation of the human instance corresponding to each grid point.

3. A single-stage real-time multi-person posture estimation method according to claim 1, characterized in that: The step S53 specifically includes the following steps: S531, calculating the coordinate value of each category; S532, generating a sinusoidal position code for each coordinate category, with a length of C; S533, use the fully connected layer to perform linear transformation on various position codes. The length of the transformed vector is still C. At this time, the sizes of the coordinate category representation matrices in the horizontal and vertical directions are [B x ,C] and [B y ,C].

4. A single-stage real-time multi-person pose estimation method according to claim 3, characterized in that: In step S531, the coordinates of the i-th category in the horizontal direction are: The coordinates of the i-th category in the vertical direction are: Among them, x l 、x r are the horizontal coordinates of the left and right edges of the expanded external frame, y t 、y b They are the vertical coordinates of the upper and lower edges of the expanded external frame.

5. A single-stage real-time multi-person posture estimation method according to claim 4, characterized in that: In step S532, the position encoding vector of the i-th category in the horizontal direction is: Where t represents the temperature coefficient, sin and cos are the sine and cosine functions respectively. The general formula for the above position coding is: Among them, c represents the subscript of the position coding vector, and its value range is [0, C-1]. The position coding generation method of each category in the vertical direction is consistent with that in the horizontal direction.

6. A single-stage real-time multi-person pose estimation method according to claim 5, characterized in that: The step S54 specifically includes the following steps: S541, after the posture representation vector passes through a fully connected layer and matrix deformation operation, its matrix shape is [K, C], where K is the number of key points; S542. Using the matrix obtained by processing with the self-attention module to enhance the structural consistency of the key points of the human body instance, the output of the self-attention module is a key point representation with a shape of [K, C]. S543, perform matrix multiplication on the key point representation and the transposed horizontal and vertical coordinate category representation matrices to obtain the shapes of [K, B x ] and [K,B y ] activation matrix; S544. For each key point, record its activation vectors in the horizontal and vertical directions as follows: Use the softmax function to normalize the activation vector to obtain the probability vector: These two probability vectors are the one-dimensional heat maps of the key point in the horizontal and vertical directions; S545. Calculate the key point coordinates by weighted summation.

7. A single-stage real-time multi-person pose estimation method according to claim 6, characterized in that: The key point coordinates in step S545 are specifically: in, are the horizontal and vertical coordinates of the key points respectively.

8. A single-stage real-time multi-person pose estimation method according to claim 6, characterized in that: The key point heat map decoding module uses a loss function based on maximum likelihood estimation to supervise the learning of one-dimensional heat maps. The loss function based on maximum likelihood estimation is specifically: Among them, μ x is the horizontal coordinate value of the manually annotated key point, s represents the size of the human instance, and σ is the variance of the manual annotation error distribution predicted by the model.

Images