Skeleton keypoint non-maximum suppression method for multi-person full-body key pose estimation
By adopting the non-maximum suppression method based on the estimation of the whole-body key posture of the multi-person key point detection in the multi-person full-body key point detection, the aliasing problem in the detection of bone key point is solved. Improves detection accuracy and recall rate, and provides efficient algorithms for multi-person scenarios.
Patent Information
- Application Number
- CN202211711017.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-12-29
AI Technical Summary
The existing technology has the problem of aliasing of bone key points in the detection of whole-body key points in multiple people, especially during detection of whole-body key points. The non-maximum suppression algorithm is poor, resulting in serious detection interference when the distance between people is close. , low recall rate.
Using the non-maximum suppression method of bone key points based on multi-person whole-body key pose estimation, the distribution weighted distance and normalized weight are calculated by initializing the human mask and confidence, and the rapid merging algorithm is performed to calculate the distributed distance independently clustered. , improve the accuracy of non-maximum suppression.
It effectively reduces aliasing of bone key points, improves the accuracy and recall of whole-body key point detection, improves processing accuracy, and provides an efficient non-maximum suppression algorithm for multi-person scenarios.
Smart Images

Figure CN115984899B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method for suppressing non-maximum values of skeleton key points based on multi-person full-body key posture estimation. Background Art
[0002] Currently, in response to the needs of VR and AR motion capture scenarios, motion capture equipment and corresponding visual algorithms for the key points of the human body have been developed one after another. At present, the non-sensor method for full-body motion capture mainly adopts a two-stage method for cameras: pedestrian detection, extraction of detection frames, and then calculation of the coordinates of the skeleton key points using a posture estimation network.
[0003] However, this method will be used more and more as the number of people to be tested increases. Aliasing occurs in the key points of the human skeleton, especially for the key points of the whole body (136 key points, including face, hands, body, etc.). The aliasing is very serious. The current non-maximum suppression algorithm (NMS) for skeleton key points mainly performs non-maximum suppression on the pedestrian detection frame, and then extracts key points from the features in the frame. This method has serious defects and has the following disadvantages:
[0004] First, if the distance between people is relatively close, non-maximum suppression is performed on the pedestrian frame to obtain a frame that contains part of the human body, which interferes with the skeleton algorithm detection. Especially for the detection of key points of the whole body, the effect of some skeleton points is very poor;
[0005] Second, some pedestrian detection boxes are missed, and AR (recall rate) is lost for multi-person key point detection. Summary of the invention
[0006] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention discloses a skeleton key point non-maximum suppression method based on multi-person full-body key posture estimation, and the skeleton key point non-maximum suppression method comprises the following steps:
[0007] Step 1, initialize the human mask and human confidence (Human Score), wherein the human mask is represented as an nx1 dimensional vector, n represents the current number of people, the human mask takes values of True and False, all human masks are initially set to True, and the human confidence Human Score is the value corresponding to the pre-input skeleton key point prediction confidence array; then initialize the threshold gamma and the matching threshold (match_thresh) and merge threshold (merge_thresh), and keep looping when an element of the Human mask array is True, and skip the point without traversal if the mask value is False;
[0008] Step 2, select the index with the highest Human Score confidence as Pick_id, calculate the distribution weighted distance dist(id) of the index Pick_id from other unmatched ids (Human mask[id]=True), where, when each distribution weighted distance dist(id) is calculated, the number of key points that match the key point num_matches(id), and then traverse the ids of all unmatched skeletal key points. If the distribution weighted distance dist(id) is less than the threshold gamma or the number of key points that match the key point num_matches(id) is greater than the matching threshold: add the index id to the merge array (merge_arr[Pick_id]) and set the human mask of the id to the False value, that is, Human mask[id]=False;
[0009] Step 3, traverse each element in the selection array, where the selection array is used to store the ID of the currently selected detected person, and execute the fast merge algorithm fast_merge(pick_arr_id, merge_arr[pick_arr_id], pose_preds, pose_scores) to obtain the coordinates and confidence values of the skeleton key points after non-maximum suppression;
[0010] Step 4: If the maximum value of the confidence of the skeleton key points after non-maximum suppression is less than the merge threshold, skip this loop and add the obtained skeleton key point coordinates and confidence values after non-maximum suppression to the corresponding array, and return the output result at the same time.
[0011] Furthermore, the execution of the fast merge algorithm further includes:
[0012] Step 301, initialize the parameters, traverse the element merge point ID (merge_id) in the merge array merge_arr of all currently selected detection person IDs, traverse 136 skeleton key point elements (kpt), and calculate the Euclidean distance dist between the skeleton key point of the currently selected detection person ID and the element merge point ID. If the Euclidean distance dist is less than a preset distance threshold, set the mask of the skeleton key point to 1 for the corresponding bit of the merge point ID, otherwise set it to 0;
[0013] Step 302, calculate the value of the normalized weight norm_scores, merge the merge_id of all elements of the skeleton key point prediction coordinate array pose_preds[merge_id][kpt] according to the normalized weight norm_scores to obtain the final weighted merged point result, and then merge the merge_id of all elements of the skeleton key point prediction confidence array according to the normalized weight norm_scores to obtain the final weighted merged point result;
[0014] Step 303, adding the predicted point coordinates obtained by the calculation result of step 302 to the skeleton key point coordinate array after non-maximum suppression.
[0015] Furthermore, the distributed distances are divided into three clusters, wherein the three clusters are the face and the left and right hands of the human body.
[0016] Furthermore, define is a cluster set, the distribution distance can be calculated by the following formula:
[0017]
[0018] Among them, k is the key point (x, y) coordinate pair, N = 136 (the number of key points in the whole body), and the distributed distance is the mean distance, which calculates the distributed distance between id1 and id2 through weighted clusters.
[0019] Furthermore, the calculation formula for calculating the normalized weight is:
[0020]
[0021] Where Kx and Ky are the x and y coordinates of the merged key points, and Norm(id) is the normalized weight of the current merge id. and The coordinates of the skeleton key points of merge_id.
[0022] Furthermore, the step 1 further includes: inputting a human detection box array, a skeleton key point prediction coordinate array, and a skeleton key point prediction confidence array, and sorting the scores of the skeleton key point prediction confidence array in descending order.
[0023] The present invention also discloses an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the non-maximum suppression method of skeletal key points based on multi-person full-body key posture estimation by executing the executable instructions.
[0024] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the non-maximum suppression method of skeleton key points based on multi-person full-body key posture estimation is implemented.
[0025] Compared with the prior art, the present invention has the following beneficial effects: Therefore, in view of the aliasing problem existing in the above-mentioned multi-person whole body key point detection, the present invention designs a non-maximum suppression algorithm for skeleton key points for multi-person whole body key point detection. The algorithm cleverly uses the cluster center points between different body parts to perform non-global center merging on the key points detected by the adjacent boxes of the cluster center points. Then the final key point detection effect is output according to the merging weight. In addition, the algorithm proposed by the present invention calculates the human body detection score according to the key point confidence, and performs non-maximum suppression calculation according to the scores in descending order. The processing accuracy is effectively improved. For the whole body key point scene, it is an efficient and practical algorithm. At the same time, the present invention proposes a distributed distance (distribution distance metrics) DDM indicator. The face and left and right hands are clustered independently, and the distributed distance calculated by weighted clustering helps to improve the accuracy of the non-maximum suppression algorithm. Finally, the merged key point coordinate array and the skeleton key point confidence array are obtained by calculating the normalized weight, which is a new and innovative calculation method different from the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but the emphasis is placed on illustrating the principles of the embodiments. In the figures, the same reference numerals designate corresponding parts in different views.
[0027] Figure 1 It is a schematic diagram of the key point serial numbers and cluster centers of the human skeleton of the present invention.
[0028] Figure 2 It is a flow chart of the non-maximum suppression method of skeleton key points based on multi-person full-body key posture estimation of the present invention. DETAILED DESCRIPTION
[0029] The flowchart in this embodiment is as follows Figure 2 shown.
[0030] First, the algorithm inputs the human detection box array, the skeleton key point prediction coordinate array, and the skeleton key point prediction confidence array. After sorting the skeleton key point prediction confidence array in descending order, the merge[id] array is constructed for the descending ids (according to the distributed center distance method). Finally, all merge[id] are traversed to give the merged skeleton key point results.
[0031] The overall structure in this embodiment is described in the form of algorithmic pseudocode, and the algorithmic pseudocode is as follows:
[0032] Input: Array of human detection boxes bboxes(nx1), Array of predicted coordinates of skeletal key points pose_preds(nx136x2), Array of confidence scores of predicted skeletal key points pose_scores(nx136), where pose_scores = HumanScore, that is, each key point has a score, 136 key points, and each key point has a number of people.
[0033] Output: Array of coordinates of skeletal key points merge_poses(n1x136x2) after non-maximum suppression, Confidence score array merge_scores(n1x136)
[0034] 1. Initialize Human Mask (nx1 vector, n is the current number of people), Human Score (nx1 vector) sorted in descending order; Initialize the array of selected skeletal key points pick_arr = []; Array of merged skeletal key points merge_arr = [];
[0035] Human mask[0...n] = True; Initialize the thresholds gamma, match_thresh, and merge_thresh
[0036] 2. Keep looping while there are elements in the Human mask array that are True:
[0037] 3. Select the index with the highest confidence in Human Score as Pick_id,
[0038] 4. Calculate the distribution-weighted distance dist(id) of this index Pick_id from other un-matched ids (Human mask[id] = True)
[0039] 5. Calculate the number of key points num_matches(id) that match this key point when calculating each distribution-weighted distance dist(id)
[0040] 6. Traverse all un-matched ids, if dist(id) < gamma or num_matches(id) 》
[0041] match_thresh:
[0042] 7. Add the index id to merge_arr[Pick_id]
[0043] 8. Human mask[id] = False
[0044] 9. Traverse each element pick_arr_id in pick_arr:
[0045] 10. Execute the fast merge algorithm fast_merge(pick_arr_id, merge_arr[pick_arr_id], pose_preds, pose_scores) to obtain merge_pose and merge_score
[0046] 11. If max(merge_score[0...136]) < merge_thresh, skip this loop
[0047] 12. Add merge_pose and merge_score to the merge_poses and merge_scores arrays
[0048] 13. When the loop ends, return the output result
[0049] Among them, the pseudo-code implementation of the fast_merge(pick_arr_id, merge_arr[Pick_id], pose_preds, pose_scores) algorithm:
[0050] Input: The currently selected detected person ID: pick_arr_id; the merge array queue
[0051] merge_arr[pick_arr_id] (mx1, where m is the number of merge IDs to be merged);
[0052] The predicted coordinate array of skeletal key points pose_preds(nx136x2), the predicted confidence array of skeletal key points pose_scores(nx136)
[0053] Output: The merged key point coordinate array merge_pose(136x2), the skeletal key point confidence array merge_score(136x1)
[0054] 1. Initialize mask[merge_id_size]
[136] ; merge_id_size = the size of merge_arr; initialize dist_thresh
[0055] 2. Traverse all elements merge_id in merge_arr[pick_arr_id]:
[0056] 3. Traverse 136 skeletal key point elements kpt:
[0057] 4. Calculate the Euclidean distance dist between the key point kpt(pick_arr_id) and kpt(merge_id) of the current pick_arr_id
[0058] 5. If dist < dist_thresh: then mask[merge_id][kpt] = 1, otherwise 0
[0059] 6. Traverse all elements merge_id of merge_arr[pick_arr_id]:
[0060] 7. Traverse 136 key points kpt
[0061] 8. Calculate the normalized norm_scores = pose_scores[merge_id][kpt] *
[0062] mask[merge_id][kpt] / sum(pose_scores[merge_id][kpt] * mask[merge_id][kpt]); 9. Merge all merge_id of pose_preds[merge_id][kpt] according to the normalized weight norm_scores to obtain the final weighted merged point result (pose_x, pose_y)
[0063] 10. Merge all merge_id of pose_scores[merge_id][kpt] according to the normalized weight norm_scores to obtain the final weighted merged point confidence result pose_score
[0064] 11. Add (pose_x, pose_y) to merge_pose, and add pose_score to merge_score
[0065] 12. End the loop of kpt, and output the algorithm results merge_score and merge_pose
[0066] Annotations on related concepts involved in the algorithm in this embodiment
[0067] First, the concept of distributed distance
[0068] The present invention proposes a distributed distance (distribution distance metrics) ddm metric, as Figure 1 shown. The distributed distance is divided into 3 clusters (face, left and right hands). Figure 1It is the labeling sequence of the human skeleton Halle 136 dataset. The design significance of this clustering is that the face and the left and right hands contain a lot of detail points. Calculating the face and the left and right hands independently helps to improve the accuracy of the non-maximum suppression algorithm. Definition is a clustering set. Then the distribution distance can be calculated by the following formula:
[0069]
[0070] Where k is the key point (x, y) coordinate pair, N = 136 (the number of key points in the whole body). This distance is the mean distance, which calculates the distributed distance between id1 and id2 through weighted clusters.
[0071] Second, calculation of normalized weights
[0072]
[0073] Where Kx and Ky are the x and y coordinates of the merged key points. Norm(id) is the normalized weight of the current merge id.
[0074] and The coordinates of the skeleton key points of merge_id. The calculation rules of norm id can be found in step 8 of the fastmerge algorithm.
[0075] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0076] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0077] Although the present invention has been described above with reference to various embodiments, it should be understood that many changes and modifications can be made without departing from the scope of the present invention. Therefore, it is intended that the above detailed description is considered to be illustrative rather than restrictive, and it should be understood that the following claims (including all equivalents) are intended to limit the spirit and scope of the present invention. The above embodiments should be understood to be only used to illustrate the present invention and not to limit the scope of protection of the present invention. After reading the content of the record of the present invention, the technician can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall into the scope defined by the claims of the present invention.
Claims
1. A non-maximum suppression method for skeleton key points based on multi-person full-body key posture estimation, characterized in that: The skeleton key point non-maximum suppression method comprises the following steps: Step 1, initialize the human mask Human mask and human confidence Human Score, wherein the human mask is represented as an nx1 dimensional vector, n represents the current number of people, the human mask takes values of True and False, all human masks are initially set to True, and the human confidence Human Score is the value corresponding to the pre-input skeleton key point prediction confidence array; then initialize the threshold gamma and the matching threshold match_thresh and merge threshold merge_thresh, and keep looping when an element of the human mask array is True, and skip the point without traversal if the mask value is False; Step 2: Select the index with the highest confidence level of Human Score as Pick_id. Calculate the distribution weighted distance dist(id) of the index Pick_id from other unmatched ids, where the number of matching key points num_matches(id) is calculated for each given distribution weighted distance dist(id), and then traverse the ids of all unmatched skeletal key points. If the distribution weighted distance dist(id) of the key point is less than the threshold gamma or the number of matching key points num_matches(id) is greater than the matching threshold, add the index id to the merge array and set the human mask of the id to the False value, that is, Human mask[id] = False, where the definition is a cluster set, the distribution distance can be calculated by the following formula: Among them, k is the key point (x, y) coordinate pair, N = 136 is the set number of whole body key points, and the distributed distance is the mean distance, which calculates the distributed distance between id1 and id2 through weighted clusters; Step 3, traverse each element in the selection array, wherein the selection array is used to store the currently selected detection person ID, execute the fast merge algorithm to obtain the coordinates and confidence values of the skeleton key points after non-maximum suppression, wherein the execution of the fast merge algorithm further includes: Step 301, initialize the parameters, traverse the element merge point ID in the merge array merge_arr of all currently selected detected person IDs, traverse the 136 skeleton key point elements, and calculate the Euclidean distance dist between the skeleton key point of the currently selected detected person ID and the element merge point ID. If the Euclidean distance dist is less than a preset distance threshold, set the mask of the skeleton key point to 1, otherwise set it to 0; Step 302, calculate the value of the normalized weight norm_scores, merge the merge_id of the merge point ID of all elements of the skeleton key point prediction coordinate array according to the normalized weight norm_scores to obtain the final weighted merge point result, and then merge the merge_id of the merge point ID of all elements of the skeleton key point prediction confidence array according to the normalized weight norm_scores to obtain the final weighted merge point result, wherein the calculation formula of the normalized weight is: Among them, Kx and Ky are the x and y coordinates of the merged key points, and the norm(id) is the normalized weight of the current merge id. and The coordinates of the skeleton key points of merge_id; Step 303, adding the predicted point coordinates obtained by the calculation result of step 302 to the skeleton key point coordinate array after non-maximum suppression; Step 4: If the maximum value of the confidence of the skeleton key points after non-maximum suppression is less than the merge threshold, skip this loop and obtain the coordinates and confidence of the skeleton key points after non-maximum suppression and add them to the corresponding array, and return the output result at the same time.
2. A method for non-maximum suppression of skeleton key points based on multi-person whole-body key posture estimation as claimed in claim 1, characterized in that: The distributed distances are divided into three clusters, where the three clusters are the face and the left and right hands of the human body.
3. A method for non-maximum suppression of skeleton key points based on multi-person whole-body key posture estimation as claimed in claim 1, characterized in that: The step 1 further includes: inputting a human detection box array, a skeleton key point prediction coordinate array, and a skeleton key point prediction confidence array, and sorting the scores of the skeleton key point prediction confidence array in descending order.
4. An electronic device, characterized in that: include: processor; as well as, A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the non-maximum suppression method of skeletal key points based on multi-person full-body key posture estimation as described in any one of claims 1-3 by executing the executable instructions.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the non-maximum suppression method of skeletal key points based on multi-person full-body key posture estimation described in any one of claims 1 to 3 is implemented.
