Human body posture estimation method, device, equipment and medium based on polar coordinate confidence representation

Abstract

A method, device, equipment and medium for human posture estimation based on polar coordinate confidence representation, relating to the technical field of posture recognition. The human posture estimation method comprises S1, obtaining an image to be processed, inputting it into a posture estimation backbone network, and obtaining a first feature map. S2, according to the first feature map, predicting the polar angle information of each key point relative to the root node through the polar angle confidence regression branch, and generating a polar angle confidence heat map. S3, according to the first feature map, predicting the polar axis length information from the root node to each key point through the polar axis length regression branch, and generating a polar axis length map. S4, according to the polar angle confidence heat map, obtaining a set of candidate root nodes by taking the maximum expectation operation. S5, according to the polar angle confidence heat map and the polar axis length map, obtaining a set of all candidate joint points by rectangular coordinate transformation. S6, according to the set of candidate root nodes and the set of candidate joint points, obtaining a human posture estimation map of the target image by multi-level clustering operation.

Description

Technical Field

[0001] The present invention relates to the technical field of posture recognition, and in particular to a method, device, equipment and medium for estimating human posture based on polar coordinate confidence representation. Background Art

[0002] Image human pose estimation is an important task in the field of computer vision. Its core goal is to accurately identify and locate the physiological pose of an individual in a monocular image or video sequence. This technology is of vital importance for understanding human behavior, analyzing movement patterns, and achieving efficient human-computer interaction.

[0003] Among the current 2D multi-person pose estimation methods, there are two main strategies: top-down and bottom-up. The top-down method first locates each person through a pedestrian detector, and then applies a single-person pose estimator to each person to locate key points. Although this method has high accuracy, its efficiency is limited by the performance of the pedestrian detector and is sensitive to occlusion. In contrast, the bottom-up method directly predicts the body joint positions of all individuals in the image, and then assigns these joint points to the corresponding human instances through a clustering algorithm. This method is usually better than the top-down method in terms of efficiency and memory overhead, but the complexity and accuracy of its clustering process have always been the key factors restricting its performance improvement.

[0004] In recent years, regression-based methods have provided a new approach for bottom-up pose estimation. These methods model human pose as the offset from the root node to each key point and achieve efficient training and reasoning through an end-to-end network framework.

[0005] However, these methods often require additional prediction heads to estimate the positions of person instances, which increases the computational complexity. In addition, existing regression methods still suffer from insufficient accuracy and poor robustness when dealing with key information such as polar angle and polar axis length.

[0006] In summary, although image human pose estimation technology has made significant progress, it still faces many challenges and requires continuous exploration and innovation to further improve its performance and practicality. Summary of the invention

[0007] The present invention provides a method, device, equipment and medium for estimating human posture based on polar coordinate confidence representation to improve at least one of the above technical problems.

[0008] In a first aspect, the present invention provides a method for estimating human posture based on polar coordinate confidence representation, which comprises steps S1 to S6.

[0009] S1. Obtain a target image to be processed, input the target image into a human posture estimation backbone network to extract features, and obtain a first feature map.

[0010] S2. According to the first feature map, the polar angle information of each key point relative to the root node is predicted through the polar angle confidence regression branch to generate a polar angle confidence heat map.

[0011] S3. According to the first feature graph, the polar length information from the root node to each key point is predicted by the polar length regression branch to generate a polar length graph, wherein the value in the polar length graph represents the radial distance from the root node to the key point.

[0012] S4. According to the polar angle confidence heat map, a set of candidate root nodes is obtained by taking a maximum expectation operation.

[0013] S5. According to the polar angle confidence heat map and the polar axis length map, a set of all candidate joint points is obtained through rectangular coordinate transformation.

[0014] S6. According to the candidate root node set and the candidate joint point set, a human body posture estimation graph of the target image is obtained through a multi-level clustering operation.

[0015] In an optional implementation, step S2 specifically includes steps S21 to S23.

[0016] S21. According to the first feature map, a two-layer convolutional neural network with a convolution kernel size of 3x3 is used to obtain the polar angle features of each channel.

[0017] S22. According to the first feature map, a two-layer convolutional neural network with a convolution kernel size of 3x3 is used to obtain the confidence distribution characteristics of the polar angle.

[0018] S23, embedding the confidence distribution feature into the polar angle feature of each channel, and then obtaining the polar angle confidence heat map through a convolutional neural network with a convolution kernel size of 1x1.

[0019] In an optional implementation, step S3 specifically includes step S31 to step S32.

[0020] S31. Divide the first feature graph into N sub-feature graphs.

[0021] S32: For each of the sub-feature graphs, a three-layer deformable convolution is used to regress the polar length of the corresponding key point to obtain the polar length graph, wherein the convolution kernel in the deformable convolution can adaptively adjust its own shape.

[0022] The formula for deformable convolution is:

[0023]

[0024] In the formula, is the output after convolution, is the center point of the convolution kernel, for Within the convolution kernel range offset, represents the set of real numbers, Represents the convolution weight coefficient of the corresponding point position, is the input matrix, is the learnable offset.

[0025] In an optional implementation, step S4 specifically includes: finding the maximum value point of each channel from the polar angle confidence heat map by using an argmax algorithm to obtain a multi-channel root node set.

[0026] In an optional implementation, step S5 is specifically: the candidate root node set, the polar length graph and the polar angle confidence heat map are calculated by rectangular coordinate transformation. The coordinates of the candidate joint points , get the set of candidate joint points.

[0027] The coordinate calculation model of the candidate root node is:

[0028]

[0029]

[0030] In the formula, For the Alternative joint points Axis coordinates, For the Alternative joint points Axis coordinates, For multi-channel root nodes Axis coordinates, For multi-channel root nodes Axis coordinates, is the prediction result of the polar length graph, is a collection of multi-channel root nodes, is the prediction result of the polar angle confidence heat map, is the radius of the mapping area.

[0031] In an optional implementation, step S6 specifically includes steps S61 to S66.

[0032] S61, dividing the candidate joint point set into five categories: head, left arm, right arm, left leg and right leg, wherein each category contains joint points of at least two subcategories.

[0033] S62. Cluster the candidate joint point sets of each major category respectively through the following steps. After obtaining the clustering results of the five major categories and the corresponding new root nodes, the five categories of body parts are regarded as five new joint points, so that the joint points belonging to the same human body are clustered to one root node, and the human body posture estimation map of the target image is obtained.

[0034] S63. First, sort each joint point in the classification according to the value of the polar angle confidence heat map at the corresponding position, and retain the top ten joint points.

[0035] S64, matching candidate joint points at different positions one by one. Since each joint point has a corresponding root node, the distances between the root nodes corresponding to the joint point groups after matching are calculated, and the group of joint points with the shortest distance is selected for retention.

[0036] S65. Repeat the previous two steps until there are no joint points that can be matched in pairs.

[0037] S66, after clustering the joint points at each position in the classification, take the coordinate mean of the root node group corresponding to each group of matched joint points as the new root node position of the group. If the value of the polar angle confidence heat map of the root node position is not higher than a threshold τ, the root node and the group of joint points will be discarded. The threshold τ is 0.05.

[0038] In an optional implementation, step S1 specifically includes steps S11 to S12.

[0039] S11, obtaining a target image to be processed.

[0040] S12: Input the target image into a human body posture estimation backbone network, predict the position information of each key point of the human body, and obtain the first feature map containing multiple channel extraction features.

[0041] In an optional embodiment, the training step further includes: predicting the position heat map of each key point through a heat map auxiliary prediction branch according to the first feature map. The heat map auxiliary prediction branch generates the key point heat map through a Gaussian kernel with adaptive variance.

[0042] In an optional embodiment, during training, the total loss function is:

[0043]

[0044] In the formula, is the total loss function, To weigh the weight parameter, set it to 0.01. is the loss function of the extreme angle confidence regression branch, is the loss function of the polar length regression branch, The loss function for the heat map-assisted prediction branch.

[0045] The loss function of the polar angle confidence regression branch is for:

[0046]

[0047] In the formula, express Loss function, Represents the weight graph, represents the element-wise multiplication operation, is the prediction result of the polar angle confidence heat map, is the ground truth of the polar angle confidence heatmap.

[0048] The loss function of the polar length regression branch is for:

[0049]

[0050] In the formula, is the number of key points, For the visibility of key points, Represents logarithm, To predict the polar length, is the true polar length, is the difference between the predicted polar angle and the true polar angle.

[0051] The heat map assists in predicting the loss function of the branch for:

[0052]

[0053] In the formula, for Norm, is the weight graph, is the element-wise product operation, The joint heat map for prediction, is the ground truth of the joint heatmap.

[0054] In a second aspect, the present invention provides a human posture estimation device based on polar coordinate confidence representation, which includes a human posture estimation module, a polar angle module, a polar axis length module, a root node module, a joint point module and a clustering module.

[0055] The human body posture estimation module is used to obtain a target image to be processed, input the target image into a human body posture estimation backbone network to extract features, and obtain a first feature map.

[0056] The polar angle module is used to predict the polar angle information of each key point relative to the root node through the polar angle confidence regression branch according to the first feature map, and generate a polar angle confidence heat map.

[0057] The polar length module is used to predict the polar length information from the root node to each key point through the polar length regression branch according to the first feature map, and generate a polar length map. The value in the polar length map represents the radial distance from the root node to the key point.

[0058] The root node module is used to obtain a set of candidate root nodes by taking the maximum expectation operation according to the polar angle confidence heat map.

[0059] The joint point module is used to obtain all candidate joint point sets through rectangular coordinate transformation according to the polar angle confidence heat map and the polar axis length map.

[0060] The clustering module is used to obtain a human body posture estimation graph of the target image through a multi-level clustering operation according to the candidate root node set and the candidate joint point set.

[0061] In a third aspect, the present invention provides a human body posture estimation device based on polar coordinate confidence representation, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a human body posture estimation method based on polar coordinate confidence representation as described in any paragraph of the first aspect.

[0062] In a fourth aspect, the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for estimating a human body posture based on polar coordinate confidence representation as described in any paragraph of the first aspect.

[0063] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0064] The present invention discloses a method for human posture estimation based on polar coordinate confidence characterization, which significantly improves the efficiency and accuracy of posture estimation through polar coordinate confidence characterization technology. The method integrates pedestrian detection and polar angle regression into a single branch, optimizes the calculation process, and reduces the calculation burden brought by the additional prediction head. At the same time, the innovative dual-branch network structure and the newly proposed median-IoU loss function not only simplify the optimization process, but also improve the accuracy of polar length regression.

[0065] In addition, the clustering strategy of multi-channel root nodes effectively reduces the coordinate mapping error and further improves the accuracy of pose estimation. Overall, the invention provides a more efficient and accurate solution for the field of human pose estimation while maintaining real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the specific implementation methods of the present invention. It should be understood that the following drawings only show certain specific implementation methods of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 It is a flowchart of the human body posture estimation method.

[0068] Figure 2 It is a network structure diagram of the human posture estimation method in the reasoning / application stage.

[0069] Figure 3 It is the network structure diagram of the human posture estimation method in the training phase.

[0070] Figure 4 It is the AL-IoU method map in the loss function of the polar length component.

[0071] Figure 5 It is the median-IoU method map in the loss function of the polar length component. DETAILED DESCRIPTION

[0072] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0073] Example 1, please refer to Figures 1 to 5The first embodiment of the present invention provides a method for estimating human posture based on polar coordinate confidence representation, which can be performed by a human posture estimation device based on polar coordinate confidence representation (hereinafter referred to as: human posture estimation device). In particular, it is performed by one or more processors in the human posture estimation device to implement steps S1 to S6.

[0074] It is understandable that the human body posture estimation device can be an electronic device with computing performance, such as a portable notebook computer, a desktop computer, a server, a smart phone or a tablet computer.

[0075] S1, obtaining a target image to be processed, inputting the target image into a human posture estimation backbone network to extract features, and obtaining a first feature map. Preferably, step S1 specifically includes steps S11 to S12.

[0076] S11, obtaining a target image to be processed.

[0077] S12: Input the target image into a human body posture estimation backbone network, predict the position information of each key point of the human body, and obtain the first feature map containing multiple channel extraction features.

[0078] The human body posture estimation backbone network is used to determine the positions of body parts of people in images or videos. It can be any existing human body posture estimation backbone network, and the present invention does not make specific limitations on this.

[0079] In this embodiment, the human pose estimation backbone network uses a convolutional backbone to extract common features of images. These feature maps capture visual information of different scales and levels in the image, and the features are used to generate polar angle confidence heat maps, polar axis length maps and auxiliary heat maps.

[0080] It should be noted that the input in the training phase is labeled training data, that is, the key points such as head, hands, and legs are labeled in the training phase, and then put into the model to train these key features, and the model learns these key features. Then, in the test phase / inference phase, put an original image in, and get the corresponding key points through the model. It can be understood as: in the test phase, the human posture estimation backbone network can identify and output multiple key points, and label these points with head, hands, legs, etc., and the subsequent steps are to assign these points to the corresponding human instances.

[0081] In this embodiment, the human posture estimation backbone network is provided with seventeen channels, which correspond to the seventeen joints of the human body in order, and the seventeen joints are divided into five categories. The five categories are: head, left arm, right arm, left leg and right leg. The head category includes five joints: eyes, nose and ears. The left arm category includes three joints: left wrist, left elbow and left shoulder. The right arm category includes three joints: right wrist, right elbow and right shoulder. The left leg category includes three joints: left hip, left knee and left ankle. The right leg category includes three joints: right hip, right knee and right ankle.

[0082] The human posture estimation backbone network recognizes organs through channels, and the five categories are the same. The first five are heads, and the next three are left arms. The distribution of key points is different. If the joint point is blocked, the detection result is 0.

[0083] S2. According to the first feature map, the polar angle information of each key point relative to the root node is predicted by the polar angle confidence regression branch to generate a polar angle confidence heat map. Preferably, step S2 specifically includes steps S21 to S23.

[0084] S21. According to the first feature map, a two-layer convolutional neural network with a convolution kernel size of 3x3 is used to obtain the polar angle features of each channel.

[0085] S22. According to the first feature map, a two-layer convolutional neural network with a convolution kernel size of 3x3 is used to obtain the confidence distribution characteristics of the polar angle.

[0086] S23, embedding the confidence distribution feature into the polar angle feature of each channel, and then obtaining the polar angle confidence heat map through a convolutional neural network with a convolution kernel size of 1x1.

[0087] In step S2, the feature map extracted from the convolutional backbone network is fed into the polar angle confidence regression branch. The purpose of this branch is to predict the polar angle information of each key point relative to the root node. In this branch, the network generates a polar angle confidence heat map, and the value of each position in the polar angle confidence heat map represents the polar angle of the position relative to the root node. The polar angle confidence heat map encodes information about the direction and confidence level of the pedestrian instance.

[0088] In polar coordinate confidence representation, the distribution of confidence is crucial prior knowledge. In order to use this prior knowledge to improve learning efficiency, in the polar angle confidence regression branch, an embodiment of the present invention designs a branch to discretely learn this distribution. Then, the confidence distribution features are embedded into the polar angle features of each channel obtained by the main network. Auxiliary distribution learning enables the network to quickly focus on the distribution near the target position, thereby reducing the learning time of the polar angle confidence regression for the confidence distribution. Afterwards, the output of the polar angle confidence regression is further aligned with the true value of the proposed polar coordinate confidence representation.

[0089] In the model training stage, in the traditional regression-based pose estimation method, focal loss or loss is usually used for training pedestrian detection. However, the real heatmaps of these pedestrian detections are constructed from Gaussian distributions, which is different from the proposed polar angle confidence heatmaps.

[0090] The distribution of the true polar angle confidence heat map is closer to the Laplace distribution. Therefore, based on this prior knowledge, the present embodiment adopts smooth loss is used to calculate the error between the predicted result of the polar angle confidence heat map and the true value.

[0091] The loss function of the polar angle confidence regression branch is for:

[0092]

[0093] In the formula, express Loss function, Represents the weight graph, represents the element-wise multiplication operation, is the prediction result of the polar angle confidence heat map, is the ground truth of the polar angle confidence heatmap.

[0094] The values ​​around the root node are all equal to 1, and the values ​​of the others are all equal to 0.1.

[0095] S3. According to the first feature graph, the polar length information from the root node to each key point is predicted by the polar length regression branch to generate a polar length graph. The value in the polar length graph represents the radial distance from the root node to the key point. Preferably, step S3 specifically includes steps S31 to S32.

[0096] S31. Divide the first feature graph into N sub-feature graphs.

[0097] S32: For each of the sub-feature graphs, a three-layer deformable convolution is used to regress the polar length of the corresponding key point to obtain the polar length graph, wherein the convolution kernel in the deformable convolution can adaptively adjust its own shape.

[0098] The formula for deformable convolution is:

[0099]

[0100] In the formula, is the output after convolution, is the center point of the convolution kernel, for Within the convolution kernel range offset, represents the set of real numbers, Represents the convolution weight coefficient of the corresponding point position, is the input matrix, is the learnable offset.

[0101] In step S3, the task of the polar length regression branch is to predict the polar length information from the root node to each key point. The network outputs a polar length map, in which the value represents the radial distance from the root node to the key point.

[0102] Due to the diversity of human body postures, it is more effective to regress the polar length of each key point separately than to estimate the polar length of all key points from a single branch. Therefore, in an embodiment of the present invention, a separation regression strategy is adopted to predict the polar length component in polar coordinates. In the polar length regression branch, the feature map output by the trunk is divided into N sub-feature maps. For each sub-feature map, a deformable convolution (DCN) is used to regress the polar length of the corresponding key point.

[0103] In the polar length regression of the embodiment of the present invention, the influence of the polar angle component needs to be considered. A simple method to introduce the polar angle component into the polar length regression is to transform the predicted polar length in the Polar-IoU loss into Replaced by the projected polar length ,get .

[0104] As shown below:

[0105]

[0106] However, instabilities prevent this loss from being the best choice for polar length regression. exist When the projected polar axis length is is approximately equal to 0, such as Figure 4As shown, a large loss is caused, which significantly affects the robustness of training.

[0107] In order to overcome the disadvantages of the above loss, an embodiment of the present invention proposes a median-IoU loss to ensure smooth training while considering the influence of the extreme angle.

[0108] In median-IoU loss, the triangle is first introduced The median, which can be considered and Here, the midpoint of a triangle is the line segment connecting a vertex to the midpoint of the opposite side. Depend on , The angle between them constitute.

[0109] Figure 5 For different Changes in the recommended median value.

[0110] The median of an edge is calculated by:

[0111]

[0112] Where: O = , O = , .

[0113] In this case, median-IoU can be expressed as:

[0114]

[0115] Finally, the loss function of median-IoU Expressed as the negative logarithm of median-IoU.

[0116] The loss function of the polar length regression branch is for:

[0117]

[0118] In the formula, is the number of key points, For the visibility of key points, Represents logarithm, To predict the polar length, is the true polar length, is the difference between the predicted polar angle and the true polar angle.

[0119] from Figure 5 It can be seen that The median change is very stable, without Different extreme values ​​ensure the stable convergence of the polar length regression branch.

[0120] also, It is robust to outliers in the predicted polar length. Due to the limitations of the logarithmic function, it is necessary to predict the polar length Preprocess it to ensure that it is greater than zero before inputting it into Polar-IoU loss or Length-IoU loss. In addition, right The value of is not sensitive because The median of the face value is always positive. Therefore, compared with the existing losses, the median of the face value is always positive. It is the best choice due to its greater robustness.

[0121] S4, according to the polar angle confidence heat map, by taking the maximum expectation operation, obtaining a candidate root node set. Preferably, step S4 specifically comprises: finding the maximum value point of each channel from the polar angle confidence heat map by using an argmax algorithm to obtain a multi-channel root node set.

[0122] After obtaining the polar angle confidence heat map and the polar axis length map, the embodiment of the present invention first performs a maximum expectation operation on the polar angle confidence heat map to obtain multiple candidate root nodes and corresponding polar angles. Preferably, the embodiment of the present invention uses the argmax algorithm to obtain a multi-channel root node set from the polar angle confidence heat map. In each channel, the embodiment of the present invention maintains 30 candidates. It is worth noting that the polar angle confidence heat map of each joint produces a unique root node set.

[0123] S5. Obtain all candidate joint point sets according to the polar angle confidence heat map and the polar axis length map by rectangular coordinate transformation. Preferably, step S5 is specifically: the candidate root node set, the polar axis length map and the polar angle confidence heat map are calculated by rectangular coordinate transformation. The coordinates of the candidate joint points , get the set of candidate joint points.

[0124] The coordinate calculation model of the candidate root node is:

[0125]

[0126]

[0127] In the formula, For the Alternative joint points Axis coordinates, For the Alternative joint points Axis coordinates, For multi-channel root nodes Axis coordinates, For multi-channel root nodes Axis coordinates, is the prediction result of the polar length graph, is a collection of multi-channel root nodes, is the prediction result of the polar angle confidence heat map, is the radius of the mapping area.

[0128] Specifically, the candidate root node set and the polar angle confidence heat map are combined with the polar axis length map to obtain the rectangular coordinates of all candidate joint points through rectangular coordinate transformation. A set of joint candidate sets (i.e., candidate joint point sets) corresponding to the candidate root node set is obtained.

[0129] S6. According to the candidate root node set and the candidate joint point set, a human body posture estimation graph of the target image is obtained through a multi-level clustering operation.

[0130] Specifically, it is difficult and time-consuming to cluster the candidate joint point set directly. Therefore, the embodiment of the present invention proposes a multi-level clustering strategy to reduce the complexity of joint candidate clustering, such as Figure 2 The multi-level clustering strategy obtains joint candidate points through multi-channel root nodes and clusters them, which reduces the error caused by coordinate mapping and improves the accuracy of the final posture.

[0131] On the basis of the above embodiment, in an optional embodiment of the present invention, step S6 specifically includes steps S61 to S66.

[0132] S61, dividing the candidate joint point set into five categories: head, left arm, right arm, left leg and right leg, wherein each category contains joint points of at least two subcategories.

[0133] S62. Cluster the candidate joint point sets of each major category respectively through the following steps. After obtaining the clustering results of the five major categories and the corresponding new root nodes, the five categories of body parts are regarded as five new joint points, so that the joint points belonging to the same human body are clustered to one root node, and the human body posture estimation map of the target image is obtained.

[0134] S63. First, sort each joint point in the classification according to the value of the polar angle confidence heat map at the corresponding position, and retain the top ten joint points.

[0135] S64, matching candidate joint points at different positions one by one. Since each joint point has a corresponding root node, the distances between the root nodes corresponding to the joint point groups after matching are calculated, and the group of joint points with the shortest distance is selected for retention.

[0136] S65. Repeat the previous two steps until there are no joint points that can be matched in pairs.

[0137] S66, after clustering the joint points at each position in the classification, take the coordinate mean of the root node group corresponding to each group of matched joint points as the new root node position of the group. If the value of the polar angle confidence heat map of the root node position is not higher than a threshold τ, the root node and the group of joint points will be discarded. The threshold τ is 0.05.

[0138] Different from the existing methods, the embodiment of the present invention extracts the root node from each joint channel, and then clusters these joint candidates according to the multi-channel root nodes, thereby reducing the influence of the root node prediction error.

[0139] Specifically, the human body is divided into five parts: head, left arm, right arm, left leg and right leg. The multi-level clustering strategy performs the following operations on the seed candidates of each part: first, select in descending order according to the value of the polar angle confidence heat map, and retain the top-ranked joint points. Next, cluster the candidates closest to the root node together. This process is iterated repeatedly until no candidate leaves the queue. If the value of the polar angle confidence heat map of the nearest root node position is not higher than a threshold τ, the root node will be discarded. The threshold τ is 0.05. At this point, five groups of body part candidates are obtained, and the new root node of each body part candidate is the one with the highest polar angle confidence among the cluster joint point root nodes. The clustering strategy of the embodiment of the present invention is characterized by simple implementation and taking into account the differences between each joint root node.

[0140] Take the clustering of left arm joints as an example: In the first step, the joints of the left wrist, left elbow, and left shoulder are sorted by the value of the polar angle confidence heat map at the corresponding position, and the top ten joints are retained. In the second step, the candidate joints at different positions are matched one by one. Since each joint has a corresponding root node, the distance between the root nodes corresponding to the joint group after matching is calculated, and the group of joints with the shortest distance is selected for retention. This process is iterated repeatedly until there are no joints that can be matched in pairs. After the joints at the three positions of the left arm are clustered, the coordinate mean of the root node group corresponding to each group of matched joints is taken as the new root node position of the group. If the value of the polar angle confidence heat map of the root node position is not higher than a threshold τ, the root node and the group of joints will be discarded, and the threshold τ is 0.05.

[0141] After obtaining the clustering results of the above five categories and the corresponding new root nodes, these five categories of body parts are regarded as five new joints. Repeat the clustering process of the first and second steps for these five joints, and the 17 joints belonging to the same human body can be clustered to one root node to obtain the human body posture estimation map of the target image.

[0142] A human posture estimation method based on polar coordinate confidence characterization of the present invention significantly improves the efficiency and accuracy of posture estimation through polar coordinate confidence characterization technology. The method eliminates redundant prediction heads by converting 2D offsets in a Cartesian coordinate system into polar angle and polar axis length components in a polar coordinate system. The method integrates pedestrian detection and polar angle regression into a single branch, optimizes the calculation process, and reduces the calculation burden caused by additional prediction heads. At the same time, the innovative dual-branch network structure and the newly proposed median-IoU loss function not only simplify the optimization process, but also improve the accuracy of polar axis length regression. In addition, the clustering strategy of multi-channel root nodes effectively reduces the coordinate mapping error and further improves the accuracy of posture estimation. Overall, the invention provides a more efficient and accurate solution for the field of human posture estimation while maintaining real-time performance.

[0143] Based on the above embodiment, in an optional embodiment of the present invention, the training step further includes: predicting the position heat map of each key point through a heat map auxiliary prediction branch according to the first feature map. The heat map auxiliary prediction branch generates the key point heat map through a Gaussian kernel with adaptive variance.

[0144] Specifically, the heat map auxiliary prediction branch is used to predict the position heat map of each key point. The output of this branch is a set of key point heat maps, each heat map corresponds to a body part. The heat map auxiliary prediction branch can provide an auxiliary training target during the training process, so that the features can retain more human body structure geometry information. This branch helps the network better learn the geometry of key points during training. This auxiliary branch is only used during training and is removed during inference.

[0145] The output of this branch is a heatmap corresponding to N keypoints, where a Gaussian kernel with adaptive variance is applied to generate the true keypoint heatmap.

[0146] During the training process, the heat map assists in predicting the loss function of the branch for:

[0147]

[0148] In the formula, for Norm, is the weight graph, is the element-wise product operation, The joint heat map for prediction, is the ground truth of the joint heatmap.

[0149] In the weight map, the position not in the body joint area is 0.1, and the other positions are .

[0150] To sum up, during training, the total loss function is:

[0151]

[0152] In the formula, is the total loss function, To weigh the weight parameter, set it to 0.01, is the loss function of the extreme angle confidence regression branch, is the loss function of the polar length regression branch, The loss function for the heat map-assisted prediction branch.

[0153] Embodiment 2: The first embodiment of the present invention provides a human posture estimation device based on polar coordinate confidence representation, which includes a human posture estimation module, a polar angle module, a polar axis length module, a root node module, a joint point module and a clustering module.

[0154] The human body posture estimation module is used to obtain a target image to be processed, input the target image into a human body posture estimation backbone network to extract features, and obtain a first feature map.

[0155] The polar angle module is used to predict the polar angle information of each key point relative to the root node through the polar angle confidence regression branch according to the first feature map, and generate a polar angle confidence heat map.

[0156] The polar length module is used to predict the polar length information from the root node to each key point through the polar length regression branch according to the first feature map, and generate a polar length map. The value in the polar length map represents the radial distance from the root node to the key point.

[0157] The root node module is used to obtain a set of candidate root nodes by taking the maximum expectation operation according to the polar angle confidence heat map.

[0158] The joint point module is used to obtain all candidate joint point sets through rectangular coordinate transformation according to the polar angle confidence heat map and the polar axis length map.

[0159] The clustering module is used to obtain a human body posture estimation graph of the target image through a multi-level clustering operation according to the candidate root node set and the candidate joint point set.

[0160] Embodiment 3: The first embodiment of the present invention provides a human body posture estimation device based on polar coordinate confidence representation, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a human body posture estimation method based on polar coordinate confidence representation as described in any paragraph of Embodiment 1.

[0161] Embodiment 4: Embodiment 1 of the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for estimating a human body posture based on polar coordinate confidence representation as described in any paragraph of Embodiment 1.

[0162] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0164] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0165] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0166] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0167] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0168] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for estimating human posture based on polar coordinate confidence representation, characterized in that: Include: Acquire a target image to be processed, input the target image into a human posture estimation backbone network to extract features, and acquire a first feature map; According to the first feature map, the polar angle information of each key point relative to the root node is predicted through the polar angle confidence regression branch to generate a polar angle confidence heat map; According to the first feature graph, predict the polar length information from the root node to each key point through the polar length regression branch to generate a polar length graph; wherein the value in the polar length graph represents the radial distance from the root node to the key point; According to the polar angle confidence heat map, a set of candidate root nodes is obtained by taking a maximum expectation operation; According to the polar angle confidence heat map and the polar axis length map, a set of all candidate joint points is obtained through rectangular coordinate transformation; According to the candidate root node set and the candidate joint point set, a human body posture estimation graph of the target image is obtained through a multi-level clustering operation; According to the first feature graph, predicting the polar length information from the root node to each key point through the polar length regression branch to generate a polar length graph specifically includes: Dividing the first feature map into N sub-feature maps; For each of the sub-feature maps, three layers of deformable convolution are used to regress the polar length of the corresponding key point to obtain the polar length map; wherein the convolution kernel in the deformable convolution can adaptively adjust its own shape; the formula of the deformable convolution is as follows: In the formula, is the output after convolution, is the center point of the convolution kernel, for Within the convolution kernel range offset, represents the set of real numbers, Represents the convolution weight coefficient of the corresponding point position, is the input matrix, is the learnable offset.

2. A method for estimating human posture based on polar coordinate confidence representation according to claim 1, characterized in that: According to the first feature map, the polar angle information of each key point relative to the root node is predicted through the polar angle confidence regression branch to generate a polar angle confidence heat map, which specifically includes: Obtain the polar angle features of each channel through a two-layer convolutional neural network with a convolution kernel size of 3x3 according to the first feature map; According to the first feature map, a two-layer convolutional neural network with a convolution kernel size of 3x3 is used to obtain the confidence distribution characteristics of the polar angle; The confidence distribution feature is embedded in the polar angle feature of each channel, and then a convolutional neural network with a convolution kernel size of 1x1 is passed to obtain the polar angle confidence heat map.

3. The method for human posture estimation based on polar coordinate confidence representation according to claim 1, characterized in that: According to the polar angle confidence heat map, a set of candidate root nodes is obtained by taking the maximum expectation operation, which specifically includes: Find the maximum value point of each channel from the polar angle confidence heat map through the argmax algorithm to obtain a multi-channel root node set; According to the polar angle confidence heat map and the polar axis length map, a set of all candidate joint points is obtained through rectangular coordinate transformation, specifically including: The candidate root node set, the polar length graph and the polar angle confidence heat map are calculated by rectangular coordinate transformation. The coordinates of the candidate joint points , obtain the candidate joint point set; among which, the coordinate calculation model of the candidate root node is: In the formula, For the Alternative joint points Axis coordinates, For the Alternative joint points Axis coordinates, For multi-channel root nodes Axis coordinates, For multi-channel root nodes Axis coordinates, is the prediction result of the polar length graph, is a collection of multi-channel root nodes, is the prediction result of the polar angle confidence heat map, is the radius of the mapping area.

4. A method for estimating human posture based on polar coordinate confidence representation according to any one of claims 1 to 3, characterized in that: According to the candidate root node set and the candidate joint point set, a human body posture estimation graph of the target image is obtained through a multi-level clustering operation, specifically including: Divide the candidate joint point set into five categories: head, left arm, right arm, left leg and right leg; each category contains at least two small categories of joint points; The candidate joint point sets of each major category are clustered respectively through the following steps. After obtaining the clustering results of five major categories and the corresponding new root nodes, the five categories of body parts are regarded as five new joint points, so that the joint points belonging to the same human body are clustered to one root node, and the human body posture estimation map of the target image is obtained; First, sort each joint point in the classification by the value of the polar angle confidence heat map at the corresponding position, and retain the top ten joint points; Match the candidate joint points at different positions one by one. Since each joint point has a corresponding root node, the distance between the root nodes corresponding to the matched joint point groups is calculated, and the joint point group with the shortest distance is selected for retention. Repeat the previous two steps until there are no joint points that can be matched in pairs; After clustering the joint points at each position in the classification, take the coordinate mean of the root node group corresponding to each group of matched joint points as the new root node position of the group; if the value of the polar angle confidence heat map of the root node position is not higher than a threshold τ, the root node and the group of joint points will be discarded; the threshold τ is 0.

05.

5. A method for estimating human posture based on polar coordinate confidence representation according to any one of claims 1 to 3, characterized in that: Obtaining a target image to be processed, inputting the target image into a human posture estimation backbone network to extract features, and obtaining a first feature map, specifically including: Obtaining a target image to be processed; The target image is input into a human body posture estimation backbone network to predict the position information of each key point of the human body, and the first feature map containing features extracted from multiple channels is obtained.

6. A method for estimating human posture based on polar coordinate confidence representation according to any one of claims 1 to 3, characterized in that: The training also includes: predicting the position heat map of each key point through the heat map auxiliary prediction branch according to the first feature map; wherein the heat map auxiliary prediction branch generates the key point heat map through the Gaussian kernel with adaptive variance; During training, the loss function of the polar angle confidence regression branch is for: In the formula, express Loss function, Represents the weight graph, represents the element-wise multiplication operation, is the prediction result of the polar angle confidence heat map, is the ground truth of the polar angle confidence heat map; During training, the loss function of the polar length regression branch is for: In the formula, is the number of key points, For the visibility of key points, Represents logarithm, To predict the polar length, is the true polar length, is the difference between the predicted polar angle and the true polar angle; During training, the heat map assists in predicting the loss function of the branch for: In the formula, for Norm, is the weight graph, is the element-wise product operation, The joint heat map for prediction, is the ground truth of the joint heat map; During training, the total loss function is: In the formula, The weight parameter is set to 0.

01.

7. A human body posture estimation device based on polar coordinate confidence representation, characterized in that: Include: A human posture estimation module is used to obtain a target image to be processed, input the target image into a human posture estimation backbone network to extract features, and obtain a first feature map; A polar angle module, used to predict the polar angle information of each key point relative to the root node through a polar angle confidence regression branch according to the first feature map, and generate a polar angle confidence heat map; A polar length module, configured to predict polar length information from a root node to each key point through a polar length regression branch according to the first feature graph, and generate a polar length graph; wherein the value in the polar length graph represents a radial distance from the root node to the key point; A root node module, used to obtain a set of candidate root nodes by taking a maximum expectation operation according to the polar angle confidence heat map; A joint point module, used for obtaining all candidate joint point sets through rectangular coordinate transformation according to the polar angle confidence heat map and the polar axis length map; A clustering module, used for obtaining a human body posture estimation graph of the target image through a multi-level clustering operation according to the candidate root node set and the candidate joint point set; The polar axis length module is specifically used for: Dividing the first feature map into N sub-feature maps; For each of the sub-feature maps, three layers of deformable convolution are used to regress the polar length of the corresponding key point to obtain the polar length map; wherein the convolution kernel in the deformable convolution can adaptively adjust its own shape; the formula of the deformable convolution is as follows: In the formula, is the output after convolution, is the center point of the convolution kernel, for Within the convolution kernel range offset, represents the set of real numbers, Represents the convolution weight coefficient of the corresponding point position, is the input matrix, is the learnable offset.

8. A human body posture estimation device based on polar coordinate confidence representation, characterized in that: It comprises a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a human body posture estimation method based on polar coordinate confidence representation as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for estimating human posture based on polar coordinate confidence representation as described in any one of claims 1 to 5.

Images