3D multi-person body posture estimation method based on RTMW and 3DMPPE

Abstract

The invention provides a 3D multi-person body posture estimation method based on RTMW and 3DMPPE, and relates to the technical field of posture estimation. When the pelvic node depth is estimated, not only is pelvic node depth information obtained through direct regression based on a RootNet method, but also the depth of the pelvic node is further reasoned and optimized by using geometric information, the depth can be reasoned only through 2.5 D attitude and geometric information, and the problem of depth blur caused by occlusion in an image is avoided. And the obtained pelvis node depth is combined with the relative coordinates of the human body postures to obtain the 3D absolute coordinates of the human body postures of multiple persons. Meanwhile, in the 2D-to-3D lifting process, the elevation angle of the human pelvis node relative to the camera is predicted, and rotation and elevation angle compensation are performed on the human body posture through the value, so that the human body posture in a real scene is restored.

Description

Technical Field

[0001] The present invention relates to the technical field of posture estimation, and in particular to a 3D multi-person human posture estimation method based on RTMW and 3DMPPE. Background Art

[0002] 3D multi-person human pose estimation aims to obtain 3D pose information of the human body in the camera coordinate system from provided data. It is a key technology in fields such as security monitoring, sports health monitoring, and motion capture. In the security monitoring field, human pose recognition can help security personnel predict potentially dangerous actions (such as climbing over railings) in public places such as airports and shopping malls, thereby improving the processing speed of such incidents. In the field of sports health monitoring, human pose recognition can assist in sports training guidance or fall detection, enabling more convenient and lower-cost posture correction and more accurate fall warnings. In the field of motion capture, markerless intelligent human pose detection eliminates the need for expensive wearable equipment required for traditional motion capture, significantly reducing filming costs and becoming one of the current development directions of motion capture technology.

[0003] Human pose estimation can be categorized as single-person or multi-person, depending on the number of people involved. Single-person pose estimation typically uses relative coordinates centered on the pelvis, which fails to reflect the positional relationships between multiple people in the image. 3DMPPE, proposed by Moon et al. in 2019, estimates the camera-centered coordinates of each pelvic node, integrating the original single-person pose into the same camera coordinate system, thus enabling multi-person pose estimation.

[0004] Currently, 3D multi-person human pose estimation is divided into two methods: top-down and bottom-up. The top-down method first uses the target detection method to segment each human body region and uses the single-person 3D pose estimation model to estimate the single-person pose, and finally merges them into multi-person human poses. When the single-person human pose estimation model performs well, the top-down method can achieve higher estimation accuracy. The bottom-up method first detects the key points of all people in the image, and then groups the detected key points and assigns them to different human instances. This method is computationally efficient compared to the top-down method, but there may be errors in the key point grouping process and it is not easy to handle complex poses.

[0005] Current top-down multi-person pose estimation methods are primarily limited by the estimation efficiency of single-person pose estimation models, resulting in slow response times in densely populated scenes. The RTMW model proposed by Jiang et al. first replaces the traditional heatmap regression method with the SimCC module. By predicting pose coordinates in three dimensions separately, it reduces quantization error while also reducing structural complexity and model computation. Compared to the traditional self-attention module, the GAU module utilizes low-dimensional features and adds nonlinear features during calculation, reducing computational overhead while also enhancing the model's expressiveness. The RTMW model significantly reduces the computational complexity of single-person pose estimation using SimCC and GAU modules, addressing the slow response times of top-down methods in densely populated scenes. However, the RTMW model has not yet achieved true multi-person pose estimation. Its final output coordinates are relative, rather than global absolute, which limits the generalizability of pose inference results across different scenarios.

[0006] While ensuring computational efficiency, multi-person human pose estimation also needs to avoid complex occlusions to ensure estimation accuracy. The 3DMPPE model proposed by Moon et al. uses a separate RootNet network to predict the depth value of the root node by combining 2D image features and geometric constraints, thereby solving the problem of missing depth information caused by occlusion. This method of decomposing 3D pose into absolute position and relative pose can reduce the impact of occlusion on overall pose estimation. However, the 3DMPPE model cannot meet the requirements of existing application scenarios in terms of accuracy and computational efficiency, and its method of directly regressing the depth of the pelvic node cannot avoid the complex occlusion problem in multi-person scenes.

[0007] The task of estimating 3D human pose from 2D images is also affected by the camera angle at the time of capture. For example, two people of the same height standing on the same horizontal plane may not be on the same horizontal line in the captured 2D image. Currently, neither the RTMW nor the 3DMPPE models address the estimation errors caused by the camera angle. Given the limitations of existing methods, further research is urgently needed to develop a model that can directly predict the absolute coordinates of 3D multi-person poses while minimizing the estimation errors caused by depth ambiguity and camera angle, thereby improving its feasibility and generalization capabilities in real-world applications. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a 3D multi-person human pose estimation method based on RTMW and 3DMPPE. The method is optimized based on the existing RTMW and RootNet processes, and can estimate the 3D camera center coordinates of the human pose while reducing the influence of depth blur and lens distortion on pose estimation.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0010] A 3D multi-person human pose estimation method based on RTMW and 3DMPPE includes the following steps:

[0011] Step 1: Obtain the 3D and 2D datasets for the current human pose estimation task, and calculate the average human torso length of the datasets before training;

[0012] Step 2: Use the RTMW model to obtain the 2.5D pose estimation of each human body, that is, the relative coordinates of each human body's 3D pose estimation;

[0013] Step 3: Estimate the absolute camera depth of the pelvic node and calculate the camera center coordinates of each pelvic node. Depth is estimated using both direct regression and geometric reasoning. The direct regression method is implemented using the RootNet network of the 3DMPPE method. The geometric reasoning method uses geometric information to further optimize the pelvic node depth information based on the direct regression method. During the RootNet regression of the pelvic node depth information, the image feature information extracted by the backbone network is used to simultaneously regress the camera elevation angle estimate required in step 4.

[0014] Step 4: To address the impact of the shooting angle on pose estimation during RGB image capture, first use the elevation angle estimated in step 3 to perform rotation compensation on the pose, then merge the pose estimates of multiple people and perform elevation compensation on the merged pose. Based on the error between the predicted value and the true value, the network parameters are updated through back propagation to make the final synthesized pose closer to the real scene, and finally generate the absolute coordinates of the joint points of the multi-person pose.

[0015] Furthermore, the specific method of step 1 is:

[0016] Step 1.1: Obtain the public datasets Human3.6M and 3DPW for the human pose estimation task, and divide the two datasets into training and test sets. Use sequences S1, S5, S6, S7, and S8 provided by the 3D single-person dataset Human3.6M as the training set content, and use S9 and S11 as the test set content. At the same time, for the 3D multi-person dataset 3DPW, use the data in the provided train folder as the content of the model training set, and use the data in the provided validation and test files as the content of the test set; randomly sample 20% of the Human3.6M training set data and 80% of the 3DPW training set data as the final model training set;

[0017] Step 1.2: Calculate the average torso length of the training set; use the custom Python function load_json_data to load the JSON file storing the labeled data into a dictionary object, then read the keypoints attribute of each element in the dictionary in turn to obtain the 3D coordinates of the pelvic node and the neck node; then use the np.linalg.norm function provided by the numpy package to calculate the distance between the pelvic node and the neck node, and record the calculated distance value in the empty array torso_lengths; after traversing the labeled data, use the np.mean function provided by the numpy package to calculate the average of all torso lengths.

[0018] Furthermore, the specific method of step 2 is:

[0019] Use the init_model function provided by MMpose to load the model det_model for target detection and the model key_model for pose estimation; use the cv2.imread function provided in the OpenCV package to load the image, and then input it into the inference_detector function provided by MMpose to obtain the target detection result det_result; use the cpu function provided by the Pytorch package and the numpy function provided by the numpy package to convert the detection result into a numpy array form, and obtain its target detection box attribute value, that is, obtain the target detection box result of the target detection; input the target detection box value obtained by detection, the image address, and the pose estimation model together into the inference_topdown function provided by MMpose to obtain the target detection result, which includes the 2D pose detection results of multiple people in the image and the 3D pose detection results of each person.

[0020] Furthermore, in step 3, the specific content of the direct regression method is as follows:

[0021] Define the generate_patch_image function, which takes as input the previously obtained target detection bounding box data and the original image numpy array representation, and outputs an array of the image cropped according to the target detection bounding box values. The numpy array of the original image is obtained using the cv2.imread function provided by the OpenCV package. Based on the principle of pinhole imaging, the approximate distance from the camera to the person is calculated based on the camera's intrinsic parameters and information about the image area where the person is located.

[0022] Let d be the distance between the camera and the human pelvic joint, in mm; f is the focal length of the camera, in mm; is the length of the human body on the image sensor, in mm; The definition of the function yields:

[0023] ;

[0024] in, is the angle of the human body in the x-axis direction in the camera's perspective, is the real physical length of the human body in the x-axis direction, The length of the human body in the x-axis direction projected onto the image sensor;

[0025] set up is the unit pixel factor on the x-axis, so we get:

[0026] ;

[0027] in, is the pixel length of the image in the x direction;

[0028] Similarly, on the y-axis we get:

[0029] ;

[0030] in, is the real physical length of the human body in the y-axis direction, is the length of the human body in the y-axis direction projected onto the image sensor. is the pixel length of the image in the y direction;

[0031] So we get:

[0032] ;

[0033] in, 、 is the camera internal parameter, 、 is assumed to be a fixed value;

[0034] Define the root_model function, which takes as input the cropped image array and the calculated distance d between the camera and the human pelvic joint, and outputs the corrected pelvic node depth. In the root_model function, the ResNet network is first used to extract the useful global features of the input image, and then global average pooling is applied to these features. The pooled feature map is then passed through the Convolution, output a correction factor ; Describes the degree of influence of the current human posture or body shape on the bbox area; the absolute depth of the final pelvic node is obtained by the following formula:

[0035] ;

[0036] in, represents the absolute depth of the pelvic node obtained by regression method;

[0037] The global features extracted by ResNet will also be input into two layers of fully connected layers with an output dimension of 512 and a ReLu function for feature extraction. The extracted features related to elevation angle are then mapped to a scalar output elevation angle using the nn.Linear function provided by the Pytorch package. .

[0038] Furthermore, in step 3, the specific content of the geometric reasoning method is as follows:

[0039] Obtain the previously obtained 2.5D human body posture, and use the distance between the human pelvis node and the neck node to approximately equal the length of the human torso to derive the following inference:

[0040] ;

[0041] in, is the absolute depth of the pelvic node finally obtained through geometric reasoning, is the absolute depth variable of the pelvic node to be solved, Indicates the distance between the pelvis node and the neck node in the camera center coordinate system; is the 2.5D coordinate of the neck node, is the 2.5D coordinate of the pelvic node, Indicates that the absolute coordinates of the corresponding node are calculated by the absolute coordinates of the pelvic node and the 2.5D coordinates of the corresponding node. is the average trunk length of the training set samples;

[0042] Solving the above equation yields the following results:

[0043] ;

[0044] in,

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] ;

[0050] in, and is the focal length of the camera along the x-axis and y-axis, , is the coordinate of the camera optical center, is the 2D coordinate of the pelvic node, is the 3D camera center coordinate of the pelvic node obtained by regression;

[0051] Therefore, define the function optimize_root_depth and input the pelvic node depth obtained by regression. , 2D coordinates of the pelvic node, average torso length and camera internal parameters; use the above formula to calculate the current pelvic node depth in the function The value of , which will then be used as the next round of calculation and Used in At the end of each round of calculation, the np.abs function provided by numpy is used to calculate the current value. Value and When the difference is less than the specified threshold, the judgment function converges. The value of is the pelvic node depth value finally obtained through geometric reasoning; otherwise, the calculation is repeated until the number of iterations reaches the maximum number of iterations.

[0052] Furthermore, in step 4, the specific method of performing rotation compensation on the posture using the elevation angle estimated in step 3 is:

[0053] For the previously obtained single-person 3D pose and the corresponding 2D pose coordinates, use the rotation matrix to rotate the 3D pose so that its rotation angle matches the 2D projection; for each 3D joint coordinate, apply the following rotation matrix:

[0054] ;

[0055] ;

[0056] in, is the 3D joint point coordinate before rotation, is the 3D joint point coordinate after rotation, is the elevation angle value obtained by direct regression in step 3.

[0057] Furthermore, in step 4, the specific method of merging the postures of multiple people and performing elevation compensation on the merged postures is:

[0058] Step 4.2.1: Obtain the absolute coordinates of each human joint point based on the obtained camera center coordinates of the human pelvic node and the previously predicted relative coordinates of the single-person 3D joint points;

[0059] For the depth value of each joint point in the single-person 3D pose pose_3d, first subtract the depth value of the pelvic node pose_3d[:, 2][0], and then add the depth value of the pelvic node obtained by previous inference , that is, the 3D absolute coordinates of each human joint point are obtained; at the same time, according to the vertical height of the first human pelvic node, the pelvic nodes of other human bodies are aligned with the human body;

[0060] Step 4.2.2: For the predicted multiple human poses, according to the distance between the pelvic node and the camera and the elevation angle obtained by direct regression in step 3 , calculate the vertical offset between multiple human bodies , the calculation formula is as follows:

[0061] ;

[0062] in, 、 are the pelvic node depth values ​​of the first and second human bodies respectively; 、 are the elevation angles of the first and second human bodies obtained by direct regression in step 3, respectively;

[0063] According to the calculated vertical offset between the other human bodies and the first human body, the other human bodies are moved to the corresponding vertical height, that is, the y-axis coordinates of all joint points of the other human bodies are superimposed. Finally, we need to scale each human pose so that its feet fall on the y-plane.

[0064] Furthermore, in step 4, the specific method of updating the network parameters by back propagation according to the error between the predicted value and the true value is:

[0065] Step 4.3.1: Calculate the error between the predicted pose and the true pose , the calculation formula is as follows:

[0066] ;

[0067] in, is the predicted posture of the i-th person, is the true posture of the i-th person;

[0068] Step 4.3.2: Calculate the error between the predicted translation value and the actual translation value , the calculation formula is as follows:

[0069] ;

[0070] in, 、 They represent the translation value of the i-th person in the predicted scene and the translation value in the real scene respectively;

[0071] Step 4.3.3: Calculate the error between the predicted pelvic node and the true pelvic node , the calculation formula is as follows:

[0072] ;

[0073] in, 、 They represent the predicted pelvic node and the real pelvic node of the i-th person respectively;

[0074] Step 4.3.4: Get the overall error ;

[0075] Step 4.3.5: Update the network parameters based on the error through backpropagation. First, call the loss.backward() function provided by the PyTorch package to automatically derive the gradient of each parameter in the neural network. Then, use the optimizer.step() function to update the model parameters based on these gradients, so that the loss decreases and the model is gradually optimized.

[0076] The beneficial effect of adopting the above technical solution is that the 3D multi-person human posture estimation method based on RTMW and 3DMPPE provided by the present invention, when estimating the depth of the pelvic node, not only obtains the pelvic node depth information through direct regression based on the RootNet method, but also uses geometric information to further infer and optimize the depth of the pelvic node. This method only requires 2.5D posture and geometric information to infer the depth, avoiding the depth ambiguity problem caused by occlusion in the image. The obtained pelvic node depth is combined with the relative coordinates of the human body posture to obtain the 3D absolute coordinates of the multi-person human body posture. At the same time, in the 2D to 3D lifting process, the present invention predicts the elevation angle of the human pelvic node relative to the camera, and uses this value to rotate and compensate the human body posture for the elevation angle, thereby restoring the human body posture in the real scene. The optimized method can generate the camera center coordinates of multi-person human body postures under occlusion conditions, and avoids the influence of the lens angle to restore the relative relationship between human body postures in the real scene. On the basis of maintaining the lightweight characteristics of RTMW, the accuracy of the estimation results is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Flowchart of a 3D multi-person human pose estimation method based on RTMW and 3DMPPE provided in an embodiment of the present invention;

[0078] Figure 2These are experimental pictures and 3D multi-person human pose estimation renderings provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0079] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0080] like Figure 1 As shown, the method of this embodiment is as follows.

[0081] Step 1: Obtain an existing dataset and calculate the average length of the human torso.

[0082] Step 1.1: Obtain public datasets commonly used for human pose estimation tasks, such as Human3.6M and 3DPW, and divide the two datasets into training and test sets. Human3.6M and 3DPW are commonly used 3D human pose datasets. Human3.6M contains 3.6 million frames of 3D pose data from 11 actors performing 17 different actions, shot indoors. 3DPW collects over 60,000 frames of data from seven different scenes, including five outdoor shooting scenes such as natural environments and dynamic scenes. Sequences S1, S5, S6, S7, and S8 provided by the 3D single-person dataset Human3.6M are used as the training set content, and S9 and S11 are used as the test set content. For the 3D multi-person dataset 3DPW, use the data in the provided train folder as the model training set content, and use the data in the provided validation and test files as the test set content. In order to fully utilize the characteristics of each dataset and avoid training bias caused by data imbalance, 20% of the Human3.6M training set data and 80% of the 3DPW training set data are randomly sampled as the final model training set.

[0083] Step 1.2: Calculate the mean torso length for the training set. Use the custom Python function load_json_data to load the JSON file containing the annotated data into a dictionary object. Then, read the keypoints attribute of each element in the dictionary to obtain the 3D coordinates of the pelvic and neck nodes. Next, use the np.linalg.norm function provided by the numpy package to calculate the distance between the pelvic and neck nodes and store the calculated distance in the empty array torso_lengths. After iterating through the annotated data, use the np.mean function provided by the numpy package to calculate the mean of all torso lengths.

[0084] Step 2: Use the RTMpose model to obtain the 2.5D pose estimate for each person, that is, the relative coordinates of each person's 3D pose estimate. Use the init_model function provided by MMpose to load the det_model model for object detection and the key_model model for pose estimation. After loading the image using the cv2.imread function provided by OpenCV, input it into the inference_detector function provided by MMpose to obtain the object detection result det_result. Use the cpu function provided by the Pytorch package and the numpy function provided by the numpy package to convert the detection result into a NumPy array and obtain the object detection box attribute value, thus obtaining the object detection box result. Input the detected object detection box value, along with the image address and pose estimation model, into the inference_topdown function provided by MMpose to obtain the object detection result, which includes the 2D pose detection results for multiple people in the image and the 3D pose detection results for each individual person.

[0085] Step 3: Calculate the camera center coordinates of each human pelvic node.

[0086] Step 3.1: Regress the absolute depth of the human pelvic node and the camera elevation angle. Define the generate_patch_image function, which takes as input the previously obtained target detection bounding box data and the original image numpy array representation, and outputs an image array cropped according to the target detection bounding box values. The original image numpy array is obtained using the cv2.imread function provided by the OpenCV package. Based on the principle of pinhole imaging, the approximate distance from the camera to the person is calculated based on the camera's intrinsic parameters and information about the image region where the person is located.

[0087] The principle of pinhole imaging is as follows: let d be the distance between the camera and the human pelvic joint (mm), f be the focal length of the camera (mm), is the length of the human body on the image sensor (mm). The definition of the function yields:

[0088] ;

[0089] in, is the angle of the human body in the x-axis direction in the camera's perspective, is the real physical length of the human body in the x-axis direction, The length of the human body in the x-axis direction projected onto the image sensor.

[0090] set up is the unit pixel factor on the x-axis, so we get:

[0091] ;

[0092] in, is the pixel length of the image in the x direction.

[0093] Similarly, we can get the following on the y-axis:

[0094] ;

[0095] in, is the real physical length of the human body in the y-axis direction, is the length of the human body in the y-axis direction projected onto the image sensor. is the pixel length of the image in the y direction.

[0096] So we get:

[0097] ;

[0098] in, 、 is the camera internal parameter, 、 is assumed to be fixed.

[0099] Define the root_model function, which takes as input the cropped image array and the calculated distance d between the camera and the human pelvic joint, and outputs the corrected pelvic node depth. In this function, the ResNet network is first used to extract the useful global features of the input image, and then global average pooling is applied to these features. The pooled feature map is then passed through Convolution, output a correction factor . Describes the degree of influence of the current human posture or body shape on the bbox area. The absolute depth of the pelvic node is finally obtained by the regression method. It can be obtained by the following formula:

[0100] ;

[0101] The global features extracted by ResNet will also be input into two layers of fully connected layers with an output dimension of 512 and a ReLu function for feature extraction. The extracted features related to elevation angle are then mapped to a scalar output elevation angle using the nn.Linear function provided by the Pytorch package. .

[0102] Step 3.2: Further optimize the absolute depth of the pelvic node through geometric reasoning. Obtain the previously obtained 2.5D human pose and use the fact that the distance between the pelvic node and the neck node is approximately equal to the length of the human torso to obtain the following reasoning:

[0103] ;

[0104] in, is the absolute depth of the pelvic node finally obtained through geometric reasoning, is the absolute depth variable of the pelvic node to be solved, Indicates the distance between the pelvis node and the neck node in the camera center coordinate system; is the 2.5D coordinate of the neck node, is the 2.5D coordinate of the pelvic node, Indicates that the absolute coordinates of the corresponding node are calculated by the absolute coordinates of the pelvic node and the 2.5D coordinates of the corresponding node. is the average trunk length of the training set samples. Solving the above equation yields the following results:

[0105] ;

[0106] ;

[0107] ;

[0108] .

[0109] ;

[0110] .

[0111] in, and is the focal length of the camera along the x-axis and y-axis, , is the coordinate of the camera optical center, is the 2D coordinate of the pelvic node, is the 3D camera center coordinate of the pelvic node obtained by regression.

[0112] Therefore, define the function optimize_root_depth and input the depth obtained by regression , 2D coordinates of the pelvic node, average torso length and camera internal parameters. First, use the above formula to calculate the current The value of , which will then be used as the next round of calculation and Used in At the end of each round of calculation, the np.abs function provided by numpy is used to calculate the current value. Value and When the difference is less than the specified threshold, the judgment function converges. The value of is the pelvic node depth value finally obtained through geometric reasoning; otherwise, the calculation is repeated until the number of iterations reaches the maximum number of iterations.

[0113] Step 4: Generate the absolute coordinates of the joint points of multiple human body postures.

[0114] Step 4.1: For the previously obtained single-person 3D pose and the corresponding 2D pose coordinates, use the rotation matrix to rotate the 3D pose so that its rotation angle matches the 2D projection. For each 3D joint coordinate, apply the rotation matrix as follows:

[0115] ;

[0116] ;

[0117] in, is the 3D joint point coordinate before rotation, is the 3D joint point coordinate after rotation, is the elevation angle value obtained by direct regression in step 3.

[0118] Step 4.2: Merge multiple poses and perform elevation compensation on the merged poses.

[0119] Step 4.2.1: Based on the obtained camera center coordinates of the human pelvic node and the relative coordinates of the single-person 3D joint points predicted in the early stage, obtain the absolute coordinates of each human joint point. For the depth value of each joint point in the single-person 3D pose pose_3d, first subtract the depth value of the pelvic node pose_3d[:, 2][0], and then add the depth value of the pelvic node obtained by previous inference The 3D absolute coordinates of each human joint point can be obtained. At the same time, the pelvic nodes of other human bodies are aligned with the vertical height of the first human body pelvic node.

[0120] Step 4.2.2: For the predicted multiple human poses, according to the distance between the pelvic node and the camera and the elevation angle obtained by direct regression in step 3 , the vertical offset between multiple human bodies can be calculated , the calculation formula is as follows:

[0121] ;

[0122] in, 、 are the pelvic node depth values ​​of the first and second human bodies respectively; 、 are the elevation angles of the first and second human bodies directly regressed in step 3, respectively.

[0123] According to the calculated vertical offset between the other human bodies and the first human body, the other human bodies are moved to the corresponding vertical height, that is, the y-axis coordinates of all joint points of the other human bodies are superimposed. Finally, we need to scale each human pose so that the feet of each pose fall on the y-plane.

[0124] Step 4.3: Calculate the error.

[0125] Step 4.3.1: Calculate the error between the predicted pose and the true pose , the calculation formula is as follows:

[0126] ;

[0127] in, is the predicted posture of the i-th person, is the true posture of the i-th person.

[0128] Step 4.3.2: Calculate the error between the predicted translation value and the actual translation value , the calculation formula is as follows:

[0129] ;

[0130] in, 、 They represent the translation value of the i-th person in the predicted scene and the translation value in the real scene respectively.

[0131] Step 4.3.3: Calculate the error between the predicted pelvic node and the true pelvic node , the calculation formula is as follows:

[0132] ;

[0133] in, 、 They represent the predicted pelvic node and the real pelvic node of the i-th person respectively.

[0134] Step 4.3.4: Get the overall error .

[0135] Step 4.4: Update the network parameters using backpropagation based on the error. First, call the loss.backward() function provided by the PyTorch package to automatically derive the gradient of each parameter in the neural network. Then, use the optimizer.step() function to update the model parameters based on these gradients, reducing the loss and gradually optimizing the model.

[0136] The method of this embodiment is suitable for estimating the absolute coordinates of multiple 3D human bodies from a monocular RGB image. By adding a depth estimation module for the pelvic node and performing elevation and rotation compensation for the posture, the absolute coordinates of the 3D posture are obtained with higher accuracy.

[0137] like Figure 2 As shown, the method of this embodiment is used to estimate 3D multi-person human poses on the experimental image. Figure 2 As shown in the renderings, this embodiment effectively restores the 3D pose information of multiple people from a single RGB image. The model not only accurately estimates the spatial position of each individual's joints, but also clearly restores the relative positions and spatial hierarchical relationships between people, demonstrating excellent multi-person pose separation and reconstruction capabilities. Furthermore, the overall estimation results are highly consistent with the real scene, validating the model's effectiveness and practical value in achieving high-precision 3D multi-person pose estimation using monocular imagery.

[0138] This example uses the public dataset 3DPW for evaluation and comparison. The improved evaluation results are shown in Table 1. As can be seen from Table 1, the MPJPE of this example method on the 3DPW dataset is 54.62 mm, which is 2.68 mm lower than the 3DMPPE of 57.3 mm. This shows that the method of this example has better performance in terms of overall 3D joint point positioning accuracy and can more accurately restore the three-dimensional posture of the human body.

[0139] Table 1 Comparison of MPJPE error between this embodiment and 3DMPPE on 3DPW dataset method MPJPE 3DMPPE 57.3 This embodiment 54.62

[0140] Table 2 shows a comparison of the root node depth estimation errors before and after the improvement. The MPJPE of this embodiment is 5.875 mm, significantly lower than the 3D MPPE of 7.499 mm, representing a 1.624 mm reduction in error. This demonstrates that this embodiment achieves higher accuracy in root node depth estimation, helping to improve the spatial consistency and stability of overall pose reconstruction.

[0141] Table 2 Comparison of root node depth error MPJPE before and after improvement method MPJPE 3DMPPE 7.499 This embodiment 5.875

[0142] In summary, this embodiment outperforms the existing method 3DMPPE in terms of overall pose estimation accuracy and key depth estimation, verifying the effectiveness of the proposed improvement strategy in the task of 3D multi-person human pose estimation.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A 3D multi-person human pose estimation method based on RTMW and 3DMPPE, characterized by: The method comprises the following steps: Step 1: Obtain the 3D and 2D datasets for the current human pose estimation task, and calculate the average human torso length of the datasets before training; Step 2: Use the RTMW model to obtain the 2.5D pose estimation of each human body, that is, the relative coordinates of each human body's 3D pose estimation; Step 3: Estimate the absolute camera depth of the pelvic node and calculate the camera center coordinates of each pelvic node. Depth is estimated using both direct regression and geometric reasoning. The direct regression method is implemented using the RootNet network of the 3DMPPE method. The geometric reasoning method uses geometric information to further optimize the pelvic node depth information based on the direct regression method. During the RootNet regression of the pelvic node depth information, the image feature information extracted by the backbone network is used to simultaneously regress the camera elevation angle estimate required in step 4. Step 4: To address the impact of the shooting angle on pose estimation during RGB image capture, first use the elevation angle estimated in step 3 to perform rotation compensation on the pose, then merge the pose estimates of multiple people and perform elevation compensation on the merged pose. Based on the error between the predicted value and the true value, the network parameters are updated through back propagation to make the final synthesized pose closer to the real scene, and finally generate the absolute coordinates of the joint points of the multi-person pose.

2. The 3D multi-person human pose estimation method based on RTMW and 3DMPPE according to claim 1, characterized in that: The specific method of step 1 is: Step 1.1: Obtain the public datasets Human3.6M and 3DPW for the human pose estimation task, and divide the two datasets into training and test sets. Use sequences S1, S5, S6, S7, and S8 provided by the 3D single-person dataset Human3.6M as the training set content, and use S9 and S11 as the test set content. At the same time, for the 3D multi-person dataset 3DPW, use the data in the provided train folder as the content of the model training set, and use the data in the provided validation and test files as the content of the test set; randomly sample 20% of the Human3.6M training set data and 80% of the 3DPW training set data as the final model training set; Step 1.2: Calculate the average torso length of the training set; use the custom Python function load_json_data to load the JSON file storing the labeled data into a dictionary object, then read the keypoints attribute of each element in the dictionary in turn to obtain the 3D coordinates of the pelvic node and the neck node; then use the np.linalg.norm function provided by the numpy package to calculate the distance between the pelvic node and the neck node, and record the calculated distance value in the empty array torso_lengths; after traversing the labeled data, use the np.mean function provided by the numpy package to calculate the average of all torso lengths.

3. The 3D multi-person human pose estimation method based on RTMW and 3DMPPE according to claim 2, characterized in that: The specific method of step 2 is: Use the init_model function provided by MMpose to load the model det_model for target detection and the model key_model for pose estimation; use the cv2.imread function provided in the OpenCV package to load the image, and then input it into the inference_detector function provided by MMpose to obtain the target detection result det_result; use the cpu function provided by the Pytorch package and the numpy function provided by the numpy package to convert the detection result into a numpy array form, and obtain its target detection box attribute value, that is, obtain the target detection box result of the target detection; input the target detection box value obtained by detection, the image address, and the pose estimation model together into the inference_topdown function provided by MMpose to obtain the target detection result, which includes the 2D pose detection results of multiple people in the image and the 3D pose detection results of each person.

4. The 3D multi-person human pose estimation method based on RTMW and 3DMPPE according to claim 3, characterized in that: In step 3, the specific content of the direct regression method is as follows: Define the generate_patch_image function, which takes as input the previously obtained target detection bounding box data and the original image numpy array representation, and outputs an image array cropped according to the target detection bounding box values. The original image numpy array is obtained using the cv2.imread function provided by the OpenCV package. Based on the pinhole imaging principle, the approximate distance from the camera to the person is calculated based on the camera's internal parameters and the relevant information of the image area where the person is located; Let d be the distance between the camera and the human pelvic joint, in mm; f is the focal length of the camera, in mm; is the length of the human body on the image sensor, in mm; The definition of the function yields: ； in, is the angle of the human body in the x-axis direction in the camera's perspective, is the real physical length of the human body in the x-axis direction, The length of the human body in the x-axis direction projected onto the image sensor; set up is the unit pixel factor on the x-axis, so we get: ； in, is the pixel length of the image in the x direction; Similarly, on the y-axis we get: ； in, is the real physical length of the human body in the y-axis direction, is the length of the human body in the y-axis direction projected onto the image sensor. is the pixel length of the image in the y direction; So we get: ； in, 、 is the camera internal parameter, 、 is assumed to be a fixed value; Define the root_model function, which takes as input the cropped image array and the calculated distance d between the camera and the human pelvic joint, and outputs the corrected pelvic node depth. In the root_model function, the ResNet network is first used to extract the useful global features of the input image, and then global average pooling is applied to these features. The pooled feature map is then passed through the Convolution, output a correction factor ; Describes the degree of influence of the current human posture or body shape on the bbox area; the absolute depth of the final pelvic node is obtained by the following formula: ； in, represents the absolute depth of the pelvic node obtained by regression method; The global features extracted by ResNet will also be input into two layers of fully connected layers with an output dimension of 512 and a ReLu function for feature extraction. The extracted features related to elevation angle are then mapped to a scalar output elevation angle using the nn.Linear function provided by the Pytorch package. .

5. The 3D multi-person human pose estimation method based on RTMW and 3DMPPE according to claim 4, characterized in that: In step 3, the specific content of the geometric reasoning method is as follows: Obtain the previously obtained 2.5D human body posture, and use the distance between the human pelvis node and the neck node to approximately equal the length of the human torso to derive the following inference: ； in, is the absolute depth of the pelvic node finally obtained through geometric reasoning, is the absolute depth variable of the pelvic node to be solved, Indicates the distance between the pelvis node and the neck node in the camera center coordinate system; is the 2.5D coordinate of the neck node, is the 2.5D coordinate of the pelvic node, Indicates that the absolute coordinates of the corresponding node are calculated by the absolute coordinates of the pelvic node and the 2.5D coordinates of the corresponding node. is the average trunk length of the training set samples; Solving the above equation yields the following results: ； in, ； ； ； ； ； in, and is the focal length of the camera along the x-axis and y-axis, , is the coordinate of the camera optical center, is the 2D coordinate of the pelvic node, is the 3D camera center coordinate of the pelvic node obtained by regression; Therefore, define the function optimize_root_depth and input the pelvic node depth obtained by regression. , 2D coordinates of the pelvic node, average torso length and camera internal parameters; use the above formula to calculate the current pelvic node depth in the function The value of , which will then be used as the next round of calculation and Used in At the end of each round of calculation, the np.abs function provided by numpy is used to calculate the current value. Value and When the difference is less than the specified threshold, the judgment function converges. The value of is the pelvic node depth value finally obtained through geometric reasoning; otherwise, the calculation is repeated until the number of iterations reaches the maximum number of iterations.

6. The 3D multi-person human pose estimation method based on RTMW and 3DMPPE according to claim 5, characterized in that: In step 4, the specific method of performing rotation compensation on the attitude using the elevation angle estimated in step 3 is: For the previously obtained single-person 3D pose and the corresponding 2D pose coordinates, use the rotation matrix to rotate the 3D pose so that its rotation angle matches the 2D projection; for each 3D joint coordinate, apply the following rotation matrix: ； ； in, is the 3D joint point coordinate before rotation, is the 3D joint point coordinate after rotation, is the elevation angle value obtained by direct regression in step 3.

7. The 3D multi-person human pose estimation method based on RTMW and 3DMPPE according to claim 6, characterized in that: In step 4, the specific method of merging the postures of multiple people and performing elevation compensation on the merged postures is: Step 4.2.1: Obtain the absolute coordinates of each human joint point based on the obtained camera center coordinates of the human pelvic node and the previously predicted relative coordinates of the single-person 3D joint points; For the depth value of each joint point in the single-person 3D pose pose_3d, first subtract the depth value of the pelvic node pose_3d[:, 2][0], and then add the depth value of the pelvic node obtained by previous inference , that is, the 3D absolute coordinates of each human joint point are obtained; at the same time, according to the vertical height of the first human pelvic node, the pelvic nodes of other human bodies are aligned with the human body; Step 4.2.2: For the predicted multiple human poses, according to the distance between the pelvic node and the camera and the elevation angle obtained by direct regression in step 3 , calculate the vertical offset between multiple human bodies , the calculation formula is as follows: ； in, 、 are the pelvic node depth values ​​of the first and second human bodies respectively; 、 are the elevation angles of the first and second human bodies obtained by direct regression in step 3, respectively; According to the calculated vertical offset between the other human bodies and the first human body, the other human bodies are moved to the corresponding vertical height, that is, the y-axis coordinates of all joint points of the other human bodies are superimposed. Finally, we need to scale each human pose so that its feet fall on the y-plane.

8. The 3D multi-person human pose estimation method based on RTMW and 3DMPPE according to claim 7, characterized in that: In step 4, the specific method of updating the network parameters by back propagation according to the error between the predicted value and the true value is: Step 4.3.1: Calculate the error between the predicted pose and the true pose , the calculation formula is as follows: ； in, is the predicted posture of the i-th person, is the true posture of the i-th person; Step 4.3.2: Calculate the error between the predicted translation value and the actual translation value , the calculation formula is as follows: ； in, 、 They represent the translation value of the i-th person in the predicted scene and the translation value in the real scene respectively; Step 4.3.3: Calculate the error between the predicted pelvic node and the true pelvic node , the calculation formula is as follows: ； in, 、 They represent the predicted pelvic node and the real pelvic node of the i-th person respectively; Step 4.3.4: Get the overall error ; Step 4.3.5: Update the network parameters based on the error through backpropagation. First, call the loss.backward() function provided by the PyTorch package to automatically derive the gradient of each parameter in the neural network. Then, use the optimizer.step() function to update the model parameters based on these gradients, so that the loss decreases and the model is gradually optimized.

Images