A multi-person human pose estimation method based on deep learning

Abstract

The present invention aims to provide a deep learning-based human pose estimation method, belonging to the field of computer vision. The method comprises: inputting an image into a feature extraction network to obtain a heat map of key points on multiple people; decoding the heat map to obtain the position information of all key points on multiple people; clustering each key point to group key points of the same person together; and combining the key point position information of each person to determine the pose of each person in the image. This invention solves the current problem of low accuracy in human pose estimation and can accurately detect and analyze the pose of multiple people.

Description

Technical Field

[0001] The present invention mainly relates to the field of computer vision, and in particular to a human posture estimation method based on deep learning. Background Art

[0002] The goal of human pose estimation is to detect key human nodes in the data, such as the head, shoulders, and knees. This keypoint information can be used to determine the spatial position of the person within the current image data. The human pose, composed of these keypoints, is an important representation of human behavior. However, due to the considerable flexibility of the human body, it can have various postures and shapes. Its appearance is significantly affected by factors such as clothing, posture, and viewing angle. It is also subject to the influence of factors such as occlusion and lighting. Furthermore, there are significant visual differences between 2D and 3D human keypoints, and different parts of the body can appear visually shortened, making human pose estimation a highly challenging task in computer vision.

[0003] Human posture estimation has applications in many fields, including motion recognition, human-computer interaction, athlete training, and augmented reality. By capturing human posture information over time using a camera, human movements can be determined. This allows for timely detection of abnormal behaviors, such as falls and fights. It can also determine whether movements are standard and provide statistical data to aid in scientific training.

[0004] With the recent development of neural networks and deep learning, human skeleton keypoint detection algorithms have gradually replaced template matching detection methods with convolutional neural networks. Based on some prior knowledge of human structure, human pose estimation methods mainly come in two forms: top-down and bottom-up. The top-down human pose estimation algorithm mainly consists of two parts: human body detection and single-person human keypoint detection. First, each person is detected through the target detection algorithm. Then, based on the human proposal box, human skeleton keypoint detection is performed for each person, i.e., human target detection + skeleton keypoint detection. The bottom-up method also consists of two parts: keypoint detection and keypoint clustering. First, all keypoints in the image need to be detected, and then all keypoints are classified into different individuals through relevant strategies.

[0005] In summary, based on the analysis of the practical application value of human posture estimation and the importance of model optimization, improving the accuracy of human posture estimation is a technical problem that technicians in this field urgently need to solve. Summary of the Invention

[0006] The purpose of the present invention is to provide a human posture estimation method based on deep learning, which solves the problem of low accuracy of current human posture estimation and can accurately detect and analyze the human postures of multiple people.

[0007] The present invention provides a multi-person human posture estimation method based on deep learning, comprising:

[0008] Input the image into the feature extraction network to obtain the key point heat map of multiple people;

[0009] Decode the key point heat map to obtain the position information of all key points on multiple people's bodies;

[0010] Perform clustering operations on each key point and classify the key points of the same person into one category;

[0011] Combine the key point position information of each person to get the posture of each person in the image.

[0012] Before inputting the image into the feature extraction network to obtain the heat map of multiple human key points, the feature extraction network must be trained:

[0013] By training the scale factor s and adaptively adjusting the Gaussian kernel standard deviation, a scaled heat map is generated for supervision information during the training process. The formula for generating the heat map based on the key point position information and the scale factor is as follows:

[0014]

[0015] In the above formula, k represents the kth key point of the pth person, x and y represent the position information of the key point, and σ0 represents the standard deviation of the initial Gaussian kernel;

[0016] Add a regularization loss to the predicted scale map as shown below:

[0017]

[0018] In the above formula Represents that only the area covered by the Gaussian kernel is retained,

[0019] The overall loss function is the L2 loss of the heat map plus the regularization loss of the scale map, as shown below:

[0020]

[0021] Decoding the key point heat map to obtain the position information of all key points on multiple people's bodies includes:

[0022] Decoding formula:

[0023] μ=m-(D)(m) -1 D'(m)

[0024] Where m represents the coordinates of the maximum point in the heat map, D” and D’ represent the second-order derivative and first-order derivative of the heat map at point m, respectively, and μ is the final coordinate of the key point position of the human body.

[0025] Perform clustering operations on each key point and classify the key points of the same person into one category:

[0026] Each key point is assigned a label to indicate the cluster assignment. The distance between the label values ​​of all key points of the same person is small, while the distance between the label values ​​of key points of different people is large. The loss function is shown as follows:

[0027]

[0028] In the above formula, h represents the value of a point on the label map, n represents the number of people in the image, and k represents the number of key points.

[0029] This method fully utilizes the information surrounding key points of the human body, solving the semantic distinction problem of key points at different scales and the ambiguity of the key points' own annotations, effectively improving the accuracy of human pose estimation. Furthermore, the method's inference speed does not increase with the number of people in the image, making it effective in dense crowd scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 This is a flowchart of a human posture estimation method based on deep learning proposed by the present invention;

[0033] Figure 2 This is a feature extraction network structure diagram of a human posture estimation method based on deep learning proposed by the present invention;

[0034] Figure 3 This is a schematic diagram of the adaptive scaling heat map process of the deep learning-based human posture estimation method proposed in the present invention;

[0035] Figure 4 Generate a human body key point label value distribution map for the association embedding branch of a human body posture estimation method based on deep learning proposed in the present invention;

[0036] Figure 5 This is a human body posture and key point marking diagram of a human body posture estimation method based on deep learning proposed by the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0039] In addition, the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0040] The present invention provides a multi-person human posture estimation method based on deep learning, comprising:

[0041] S100, inputting the image into a feature extraction network to obtain a heat map of key points of multiple human bodies;

[0042] S200, decoding the key point heat map to obtain the position information of all key points of multiple people's bodies;

[0043] S300, performing a clustering operation on each key point, and classifying the key points of the same person into one category;

[0044] S400, combining the key point position information of each person to obtain the posture of each person in the image.

[0045] This method fully utilizes the information surrounding key points of the human body, solving the semantic distinction problem of key points at different scales and the ambiguity of the key points' own annotations, effectively improving the accuracy of human pose estimation. Furthermore, the method's inference speed does not increase with the number of people in the image, making it effective in dense crowd scenes.

[0046] Before the S100 inputs the image into the feature extraction network to obtain the heat map of multiple human key points, it also includes training the feature extraction network:

[0047] By training the scale factor s and adaptively adjusting the Gaussian kernel standard deviation, a scaled heat map is generated for supervision information during the training process. The formula for generating the heat map based on the key point position information and the scale factor is as follows:

[0048]

[0049] In the above formula, k represents the kth key point of the pth person, x and y represent the position information of the key point, and σ0 represents the standard deviation of the initial Gaussian kernel;

[0050] Add a regularization loss to the predicted scale map as shown below:

[0051]

[0052] In the above formula Represents that only the area covered by the Gaussian kernel is retained,

[0053] The overall loss function is the L2 loss of the heat map plus the regularization loss of the scale map, as shown below:

[0054]

[0055] S200 decodes the key point heat map to obtain the position information of all key points on multiple people's bodies, including:

[0056] Decoding formula:

[0057] μ=m-(D)(m) -1 D'(m)

[0058] Where m represents the coordinates of the maximum point in the heat map, D” and D’ represent the second-order derivative and first-order derivative of the heat map at point m, respectively, and μ is the final coordinate of the key point position of the human body.

[0059] S300 performs a clustering operation on each key point, classifying the key points of the same person into one category. Each key point is assigned a label to indicate the cluster assignment. The distance between the label values ​​of all key points of the same person is small, while the distance between the label values ​​of key points of different people is large. The loss function is shown in the following formula:

[0060]

[0061] In the above formula, h represents the value of a point on the label map, n represents the number of people in the image, and k represents the number of key points.

[0062] This method fully utilizes the information surrounding key points of the human body, solving the semantic distinction problem of key points at different scales and the ambiguity of the key points' own annotations, effectively improving the accuracy of human pose estimation. Furthermore, the method's inference speed does not increase with the number of people in the image, making it effective in dense crowd scenes.

[0063] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A multi-person human pose estimation method based on deep learning, characterized in that: include: Input the image into the feature extraction network to obtain the key point heat map of multiple people; Decode the key point heat map to obtain the position information of all key points on multiple people's bodies; Perform clustering operations on each key point and classify the key points of the same person into one category; Combine the key point position information of each person to obtain the posture of each person in the image; Before inputting the image into the feature extraction network to obtain the multi-person human key point heat map, the feature extraction network is trained: By training the scale factor s and adaptively adjusting the Gaussian kernel standard deviation, a scaled heat map is generated for supervision information during the training process. The formula for generating the heat map based on the key point position information and the scale factor is as follows: ； i and j represent the horizontal and vertical coordinates in the heat map, p represents the pth person, and k represents the kth key point. 、 They represent the horizontal and vertical coordinate positions of the kth key point of the pth person, σ0 represents the initial Gaussian kernel standard deviation, and s is the scale factor; Add a regularization loss to the predicted scale map as shown below: ； Represents that only the area covered by the Gaussian kernel is retained, ; The overall loss function is the L2 loss of the heat map plus the regularization loss of the scale map, as shown below: ； Where P is the true value heat map, H is the heat map predicted by the model, is the weight of the regularization loss; The clustering operation of each key point to classify the key points of the same person into one category includes: Each key point is assigned a label to indicate the cluster assignment. The distance between the label values ​​of all key points of the same person is small, while the distance between the label values ​​of key points of different people is large. The loss function is shown as follows: ； In the above formula, N represents the number of people in the image. The label value heat map representing the k-th key point of the n-th person at pixel position x, Represents the average value of the label values ​​of each joint point of the n-th person, k represents the k-th key point, n represents the n-th person in the image, and n' represents other people except the n-th person.

2. The method for multi-person human posture estimation based on deep learning according to claim 1, characterized in that: Decoding the key point heat map to obtain the position information of all key points of multiple people's bodies includes: Decoding formula: ； Where m represents the coordinates of the maximum point in the heat map, 、 They represent the second-order derivative and first-order derivative of the heat map at point m, and μ is the final position coordinate of the key points of the human body.

Images