Human pose estimation method, controller, and computer storage medium

Abstract

This application relates to the field of 3D human pose estimation technology, specifically providing a human pose estimation method, controller, and computer storage medium, aiming to solve the problem of reducing uncertainties in intermediate joints and achieving high-precision human pose estimation. To this end, the method of this application includes: acquiring the 3D position of anchor joints, head pose trajectory, and video images; for each image frame, determining the 2D unit projection direction of the limb containing the anchor joint in the image frame based on a limb orientation prediction network; generating a 3D human pose sequence based on the head pose trajectory; for each pose to be optimized, determining the orientation constraint residual based on the 2D unit projection direction to guide optimization, and obtaining the human pose estimation result. This application introduces the 2D projection direction from the video image into the pose optimization process, constraining the orientation of the limb containing the anchor joint under the premise of known anchor joint positions, effectively reducing ambiguity in intermediate joints and improving the accuracy of human pose estimation.

Description

Technical Field

[0001] This application relates to the field of three-dimensional human pose estimation technology, specifically providing a human pose estimation method, a controller, and a computer storage medium. Background Technology

[0002] In fields such as augmented reality and virtual reality, how to recover the wearer's full-body three-dimensional posture from their own perspective or head sensor data has become a key technology in areas such as human-computer interaction, social virtual avatars, motion analysis, and remote collaboration.

[0003] In existing technologies, most motion priors are based on diffusion models, using the six-DOF pose trajectory of the head as conditional input, and generating statistically reasonable whole-body pose sequences through progressive denoising. However, the sampling process of diffusion models typically requires the addition of foot contact constraints, hand position constraints, or hand pose tracking constraints to ensure that foot or hand movements are consistent with the observed results. However, this constraint, which only utilizes the end positions of kinetic chains such as the wrist, still has significant limitations. For example, even when the shoulder and wrist positions of the human arm are determined, the elbow joint may still be located at multiple candidate positions with the shoulder-wrist line as the axis, leading to ambiguity in the arm bending plane. Furthermore, there is a lack of direct constraints on instance-level details such as the arm bending inward or outward and the elbow orientation in specific image frames. Similarly, when only the ankle position is constrained for the lower limbs, the knee joint orientation may also be uncertain. Therefore, existing technologies cannot achieve high-precision, low-ambiguity human pose estimation.

[0004] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0005] This application aims to solve the aforementioned technical problem, namely, how to reduce the uncertainty of intermediate joints and achieve high-precision human posture estimation.

[0006] In a first aspect, this application provides a human pose estimation method, the method comprising: Based on a head-mounted device, the three-dimensional position of at least one anchor joint on the target human body, the head posture trajectory, and a video image containing the anchor joint are acquired. For each image frame in the video image, the two-dimensional unit projection direction of the limb where the anchor joint is located in the image frame is determined based on the three-dimensional position of the anchor joint and the pre-trained limb direction prediction network. Based on the head position trajectory, pose estimation is performed to generate a three-dimensional human pose sequence of the target human body; For each pose to be optimized in the three-dimensional human pose sequence, the directional constraint residual of the pose to be optimized is determined according to the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized. The pose to be optimized is then guided and optimized according to the directional constraint residual to obtain the optimized human pose estimation result.

[0007] In one technical solution of the above-mentioned human pose estimation method, determining the orientation constraint residual of the pose to be optimized based on the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized includes: The detection confidence level corresponding to each image frame is determined based on the two-dimensional unit projection direction of each image frame. For each pose to be optimized in the three-dimensional human pose sequence, the two-dimensional projection position of the target joint and the two-dimensional projection position of the anchor joint on the corresponding image frame are determined. The target joint is the middle joint of the limb where the anchor joint is located. Based on the detection confidence of the corresponding image frame, the two-dimensional projection position of the target joint, the two-dimensional projection position of the anchor joint, and the two-dimensional unit projection direction, the directional constraint residual corresponding to each posture to be optimized is determined.

[0008] In one technical solution of the above-mentioned human pose estimation method, determining the orientation constraint residual corresponding to each pose to be optimized includes: The directional constraint residual for each pose to be optimized is determined using the following formula: in, The orientation constraint residual of the attitude to be optimized is... The detection confidence score is the image frame corresponding to the pose to be optimized. The preset directional constraint weights, The two-dimensional projection position of the target joint on the corresponding image frame represents the pose to be optimized. Let represent the two-dimensional projection position of the anchor joint in the corresponding image frame for the pose to be optimized. This is a preset constant value. The orientation is the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized.

[0009] In one technical solution of the above-mentioned human pose estimation method, determining the detection confidence level corresponding to each image frame based on the two-dimensional unit projection direction of each image frame includes: For each image frame, if there is a prior position of the target joint, the prior direction is determined based on the prior position of the target joint and the three-dimensional position of the anchor joint; the absolute value of the dot product between the two-dimensional unit projection direction of each image frame and the prior direction is determined as the detection confidence. If there is no prior location for the target node, the detection confidence level is set to a preset value. The prior position of the target joint includes any one of the following: the three-dimensional position of the target joint in the pose to be optimized corresponding to the previous image frame, the three-dimensional position of the target joint in the current image frame obtained from the motion prior, and the three-dimensional position of the target joint in the pose to be optimized corresponding to the current image frame.

[0010] In one technical solution of the above-mentioned human pose estimation method, the method further includes: When the dot product between the two-dimensional unit projection direction of the image frame and the prior direction is negative, the corresponding two-dimensional unit projection direction is reversed to eliminate the 180-degree directional ambiguity.

[0011] In one technical solution of the above-mentioned human pose estimation method, determining the two-dimensional unit projection direction of the limb where the anchor joint is located in the image frame based on the three-dimensional position of the anchor joint and a pre-trained limb orientation prediction network includes: Based on the three-dimensional position of the anchor joint, determine the two-dimensional projection position of the anchor joint on the image frame; The image frame is cropped according to the two-dimensional projection position of the anchor joint to obtain an image region of a preset size; Based on the anchor joint, a limb identifier is determined, wherein the limb identifier is an identifier used to indicate the limb where the anchor joint is located; Based on a pre-trained limb orientation prediction network, the two-dimensional unit projection orientation is determined according to the image region and the limb identifier.

[0012] In one technical solution of the above-mentioned human pose estimation method, determining the two-dimensional projection position of the anchor joint on the image frame based on the three-dimensional position of the anchor joint includes: Obtain the camera intrinsic parameters of the head-mounted device and the camera extrinsic parameters corresponding to the image frame; Based on the camera intrinsic and extrinsic parameters, the three-dimensional position of the anchor joint is projected from perspective to obtain the corresponding two-dimensional pixel coordinates as the two-dimensional projection position of the anchor joint.

[0013] In one technical solution of the above-mentioned human pose estimation method, before performing perspective projection on the three-dimensional position of the anchor joint, the method further includes: Based on the camera extrinsic parameters, the three-dimensional position of the anchor joint is converted into three-dimensional coordinates in the camera coordinate system; Based on the ordinate in the three-dimensional coordinate system, determine whether the three-dimensional position of the anchor joint is valid; If effective, proceed to step "project the three-dimensional position of the anchor joint from a perspective view". If invalid, the image frame is determined to be unusable, and the two-dimensional projection position of the next image frame is obtained.

[0014] In one technical solution of the above-mentioned human pose estimation method, the limb orientation prediction network includes a shared visual encoder and a multilayer perceptron prediction head; The pre-trained limb orientation prediction network determines the two-dimensional unit projection orientation based on the image region and the limb identifier, including: Based on the shared visual encoder, feature extraction is performed on the image region to obtain the visual feature vector of the image region; The limb identifier is one-hot encoded to obtain a multi-dimensional feature vector, and the multi-dimensional feature vector is concatenated with the visual feature vector to obtain a conditional feature vector; The conditional feature vector is input into the multilayer perceptron prediction head to obtain a two-dimensional vector, and the two-dimensional vector is L2 normalized to obtain the two-dimensional unit projection direction. Wherein, the two-dimensional unit projection direction is a unit direction vector used to represent the two-dimensional projection position from the two-dimensional projection position of the anchor joint to the two-dimensional projection position of the target joint, and the target joint is the intermediate joint of the limb where the anchor joint is located.

[0015] In one technical solution of the above-mentioned human pose estimation method, the step of cropping the image frame based on the two-dimensional projection position of the anchor joint to obtain an image region of a preset size includes: The image frame is cropped with the two-dimensional projection position of the anchor joint as the center to obtain an image area of ​​a preset size.

[0016] In one technical solution of the above-mentioned human pose estimation method, "cropping the image frame" further includes: Based on the preset size, determine whether the minimum distance between the two-dimensional projection position of the anchor joint and the boundary of the image frame is less than a preset distance threshold. If so, after filling the boundaries of the image frame, the image region is cropped with the two-dimensional projection position of the anchor joint as the center to obtain an image region of a preset size; The image region of the preset size is preprocessed to obtain the final image region corresponding to the anchor joint. The preprocessing includes at least one of format conversion, normalization, and standardization.

[0017] In one technical solution of the above-mentioned human pose estimation method, the step of estimating the pose based on the head position trajectory to generate a three-dimensional human pose sequence of the target human body includes: Based on the diffusion model, the pose is estimated according to the head position trajectory, and a three-dimensional human pose sequence of the target human body is generated.

[0018] In one technical solution of the above-mentioned human pose estimation method, the step of guiding the optimization of the pose to be optimized based on the orientation constraint residual to obtain the optimized human pose estimation result includes: Add the orientation constraint residual of each attitude to be optimized to the nonlinear least squares optimizer; Based on the nonlinear least squares optimizer, with the goal of minimizing the directional constraint residual, the three-dimensional human pose sequence output by the diffusion model is guided to optimize, and the optimized human pose estimation result is obtained.

[0019] In a second aspect, a controller is provided, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described in any of the above-described technical solutions for human pose estimation.

[0020] In a third aspect, a computer storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described technical solutions for human pose estimation.

[0021] By employing the above technical solution, this application can acquire the three-dimensional position of at least one anchor joint on the target human body, the head pose trajectory, and video images containing the anchor joint; for each image frame in the video image, based on the three-dimensional position of the anchor joint and a pre-trained limb orientation prediction network, the two-dimensional unit projection direction of the limb containing the anchor joint in the image frame is determined; pose estimation is performed based on the head pose trajectory to generate a three-dimensional human pose sequence of the target human body; for each pose to be optimized in the three-dimensional human pose sequence, the orientation constraint residual of the pose to be optimized is determined based on the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized, so as to guide the optimization of the pose to be optimized according to the orientation constraint residual, and obtain the optimized human pose estimation result.

[0022] Through the above configuration, this application can fully utilize the acquired video images and the three-dimensional positions of the anchor joints to obtain the two-dimensional unit projection direction of the anchor joints in each image frame. Then, based on the two-dimensional unit projection direction, it generates the directional constraint residual of the posture to be optimized for each image frame, thereby guiding the optimization of the posture during the guided optimization process. In this way, based on the two-dimensional unit projection direction of the limb where the anchor joint is located, the two-dimensional limb direction signal in the video image can be introduced into the three-dimensional human posture optimization process. This allows for constraint of the direction of the limb (such as the forearm or lower leg) where the anchor joint is located, even when the anchor joint positions are the same (e.g., wrist or ankle), and further constraint of the bending direction of intermediate joints such as the elbow or knee joints, effectively reducing ambiguity of intermediate joints and improving the accuracy of human posture estimation. Attached Figure Description

[0023] The preferred embodiments of this application are described below with reference to the accompanying drawings, in which: Figure 1 This is a schematic flowchart of the main steps of a human pose estimation method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the main steps of determining the directional constraint residual according to one embodiment of this application. Figure 3 This is a schematic flowchart of the main steps of a human pose estimation method according to another embodiment of this application; Figure 4 This is a schematic flowchart illustrating the main steps of obtaining the two-dimensional unit projection direction in one embodiment of this application. Detailed Implementation

[0024] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0025] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0026] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a human pose estimation method according to an embodiment of this application. Figure 1 As shown, the human pose estimation method in this application embodiment mainly includes the following steps S101 to S104: Step S101: Based on the head-mounted device, acquire the three-dimensional position of at least one anchor joint on the target human body, the head posture trajectory, and a video image containing the anchor joint.

[0027] Step S102: For each image frame in the video image, determine the two-dimensional unit projection direction of the limb where the anchor joint is located in the image frame based on the three-dimensional position of the anchor joint and the pre-trained limb orientation prediction network.

[0028] Step S103: Estimate the pose based on the head position trajectory to generate a three-dimensional human pose sequence of the target human body.

[0029] Step S104: For each pose to be optimized in the three-dimensional human pose sequence, determine the directional constraint residual of the pose to be optimized according to the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized, so as to guide the optimization of the pose to be optimized according to the directional constraint residual, and obtain the optimized human pose estimation result.

[0030] In this embodiment, the "head-mounted device" in step S101 refers to an electronic device that is fixed to the head and worn through a structure such as a headband or headband. The "anchor joint" refers to the distal joints of the human limbs, such as the wrist joint and ankle joint.

[0031] In one embodiment, the head-mounted device may be equipped with a Simultaneous Localization and Mapping (SLAM) system to obtain head pose trajectory; it may also be equipped with a hand tracking system or other sensors to obtain the "three-dimensional position of at least one anchor joint" in step S101, such as the three-dimensional position of the left and right wrists; and it may also be equipped with a camera or video recording camera to obtain the "video image containing the anchor joint" in step S101.

[0032] In this embodiment, the "head pose trajectory" in step S101 refers to the sequence of changes in the three-dimensional position and three-dimensional pose of a sensor (such as a camera built into a VR / AR headset) over time relative to the starting point or world coordinate system during movement. For example, the head pose trajectory can specifically be a six-DOF head pose trajectory, which includes three translational degrees of freedom to describe the head's position coordinates in space, and three rotational degrees of freedom to describe the head's orientation.

[0033] In this embodiment, the "LimbAngleNet" in step S102 is a lightweight convolutional neural network used to predict the two-dimensional projection orientation of limbs (such as forearms and lower legs) from a cropped region of a monocular image.

[0034] In this embodiment, the "three-dimensional human posture sequence" in step S103 refers to a data set arranged in chronological order that describes the continuous changes of human joints in three-dimensional spatial coordinates. The three-dimensional human posture sequence includes multiple postures to be optimized, and each posture corresponds to an image frame in the video image.

[0035] In this embodiment, the "direction constraint residual" in step S104 refers to the difference between the two-dimensional projection direction calculated by the human posture to be optimized and the two-dimensional unit projection direction observed by the limb direction prediction network. The rotation error in posture estimation can be corrected by minimizing this direction constraint residual.

[0036] In one embodiment, step S102 may further include steps S1021 to S1024: Step S1021: Determine the two-dimensional projection position of the anchor joint on the image frame based on the three-dimensional position of the anchor joint.

[0037] Step S1022: Crop the image frame according to the two-dimensional projection position of the anchor joint to obtain an image area of ​​a preset size.

[0038] Step S1023: Determine limb identifiers based on anchor joints.

[0039] Step S1024: Based on the pre-trained limb orientation prediction network, determine the two-dimensional unit projection orientation according to the image region and limb identifier.

[0040] In step S1023, "limb identifier" refers to the identifier used to represent the limb where the anchor joint is located. For example, the limb identifier can be set to 0, 1, 2, and 3, representing the left forearm, right forearm, left lower leg, and right lower leg, respectively. As an example, if the anchor joint is the wrist joint of the left forearm, the determined limb identifier is 0.

[0041] In one embodiment, step S1021 may further include steps S10211 and S10212: Step S10211: Obtain the camera intrinsic parameters of the head-mounted device and the camera extrinsic parameters corresponding to the image frames.

[0042] Step S10212: Based on the camera intrinsic and extrinsic parameters, perform perspective projection on the three-dimensional position of the anchor joint to obtain the corresponding two-dimensional pixel coordinates as the two-dimensional projection position of the anchor joint.

[0043] In step S10211, "camera intrinsic parameters" are parameters describing the internal optical characteristics of the camera, used to map points from three-dimensional space to points on a two-dimensional image plane; "camera extrinsic parameters" are parameters describing the position and orientation of the camera in three-dimensional space, used to determine the transformation relationship between the camera coordinate system and the world coordinate system.

[0044] In one embodiment, prior to step S10212, the method may further include steps S10213 to S10215: Step S10213: Based on the camera extrinsic parameters, convert the three-dimensional position of the anchor joint into the three-dimensional coordinates of the camera coordinate system.

[0045] Step S10214: Based on the ordinate in the three-dimensional coordinate system, determine whether the three-dimensional position of the anchor joint is valid. If valid, proceed to step S10212; if invalid, proceed to step S10215.

[0046] Step S10215: Determine that the image frame is unavailable, and proceed to obtain the two-dimensional projection position of the next image frame.

[0047] In step S10213, the "camera coordinate system" refers to a three-dimensional Cartesian coordinate system established with the camera's optical center as the origin, used to describe the orientation and distance of the three-dimensional position relative to the camera itself. Specifically, the ordinate of the three-dimensional coordinates in the camera coordinate system refers to the distance of the three-dimensional position along the Z-axis in the camera coordinate system, that is, the projection of the straight-line distance from the three-dimensional position to the camera's optical center along the line of sight. In one embodiment, step S10214 can specifically determine whether the vertical coordinate in the three-dimensional coordinate system is greater than a preset depth threshold. If it is greater, the three-dimensional position of the anchor joint of the image frame is determined to be valid. If it is less than or equal to the threshold, the three-dimensional position of the anchor joint of the image frame is determined to be invalid, and the image frame needs to be skipped.

[0048] For example, the preset depth threshold can be equal to 0.05 meters.

[0049] In other possible implementations, during the process of "perspectively projecting the three-dimensional position of the anchor joint" in step S10212, it can be further determined whether the actual resolution of the image frame has changed relative to the reference resolution. If so, the focal length and principal point parameters in the camera intrinsic parameters can be scaled according to the changed width and height ratio to accurately project the three-dimensional position of the anchor joint into two dimensions.

[0050] In one embodiment, step S1022 may further include step S10221: Step S10221: Using the two-dimensional projection position of the anchor joint as the center, crop the image frame to obtain an image area of ​​a preset size.

[0051] In one embodiment, step S10221 may further include steps S102211 to S10223: Step S102211: Based on the preset size, determine whether the minimum distance between the two-dimensional projection position of the anchor joint and the boundary of the image frame is less than the preset distance threshold. If so, proceed to step S102212.

[0052] Step S102212: After filling the boundary of the image frame, crop the image area with the two-dimensional projection position of the anchor joint as the center to obtain an image area of ​​a preset size. Step S102213: Preprocess the image region of the preset size to obtain the final image region corresponding to the anchor joint.

[0053] In step S102212, the method of "filling the boundary of the image frame" can specifically adopt zero filling or other boundary filling methods.

[0054] The "preprocessing" in step S102213 may include at least one of format conversion, normalization, and standardization. For example, the cropped image region may first be converted from its original format to a three-primary-color (Red, Green, Blue, RGB) format, and the image pixels in the image region may be normalized to [0,1]. Then, a preset mean and standard deviation may be used for standardization, and finally, the image region may be converted into a tensor format suitable for neural network input.

[0055] In one implementation, the "limb orientation prediction network" in step S1024 may include a shared visual encoder and a multilayer perceptron prediction head, then step S1024 may further include steps S10241 to S10243: Step S10241: Based on the shared visual encoder, perform feature extraction on the image region to obtain the visual feature vector of the image region.

[0056] Step S10242: Perform one-hot encoding on the limb identifier to obtain a multi-dimensional feature vector, and concatenate the multi-dimensional feature vector with the visual feature vector to obtain a conditional feature vector.

[0057] Step S10243: Input the conditional feature vector into the multilayer perceptron prediction head to obtain a two-dimensional vector, and perform L2 normalization on the two-dimensional vector to obtain the two-dimensional unit projection direction.

[0058] In step S10243, "two-dimensional unit projection direction" is used to represent the unit direction vector from the two-dimensional projection position of the anchor joint to the two-dimensional projection position of the target joint, where the target joint is the intermediate joint of the limb where the anchor joint is located.

[0059] For example, if the anchor joint is the wrist joint, then the target joint is the elbow joint, and the two-dimensional unit projection direction is from the two-dimensional projection position of the wrist joint to the two-dimensional projection position of the elbow joint; when the anchor joint is the ankle joint, then the target joint is the knee joint, and the two-dimensional unit projection direction is from the two-dimensional projection position of the ankle joint to the two-dimensional projection position of the knee joint.

[0060] In step S10241, the “shared visual encoder” is a neural network module used to encode the input image region into a compact feature representation.

[0061] Among them, "One-Hot Encoding" in step S10242 is an encoding method that converts categorical variables into binary numerical vectors.

[0062] In step S10243, the "multilayer perceptron prediction head" is an end module in a deep learning model composed of multiple stacked fully connected layers, used to receive the feature vector output by the shared visual encoder and complete the final classification or regression prediction task; "L2 normalization" refers to the operation of dividing a two-dimensional vector by its L2 norm to make it a unit vector.

[0063] In one implementation, step S103 may further include step S1031: Step S1031: Based on the diffusion model, perform pose estimation according to the head position trajectory to generate a three-dimensional human pose sequence of the target human body.

[0064] Among them, diffusion models are generative models that generate data from Gaussian noise through progressive denoising. For example, a specific diffusion model can be the Denoising Diffusion Implicit Model (DDIM), which is a deterministic sampling method that accelerates diffusion model sampling.

[0065] In one implementation, see Appendix Figure 2 , attached Figure 2 This is a flowchart illustrating the main steps of determining the directional constraint residual according to one embodiment of this application. Figure 2 As shown, step S104, "determining the orientation constraint residual of the pose to be optimized based on the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized," can further include steps S1041 to S1043: Step S1041: Determine the detection confidence level corresponding to each image frame based on the two-dimensional unit projection direction of each image frame.

[0066] Step S1042: For each pose to be optimized in the three-dimensional human pose sequence, determine the two-dimensional projection position of the target joint and the two-dimensional projection position of the anchor joint on the corresponding image frame for each pose to be optimized.

[0067] Step S1043: Determine the directional constraint residual corresponding to each pose to be optimized based on the detection confidence of the corresponding image frame, the two-dimensional projection position of the target joint, the two-dimensional projection position of the anchor joint, and the two-dimensional unit projection direction.

[0068] The target joint is the intermediate joint of the limb where the anchor joint is located.

[0069] In one embodiment, step S1041 may further include steps S10411 to S10413: Step S10411: For each image frame, determine whether there is a prior position of the target joint in the image frame. If there is, proceed to step S10412; if not, proceed to step S10413.

[0070] Step S10412: Determine the prior direction based on the prior position of the target joint and the three-dimensional position of the anchor joint; determine the absolute value of the dot product between the two-dimensional unit projection direction of each image frame and the prior direction as the detection confidence.

[0071] Step S10413: Determine the detection confidence level as a preset value.

[0072] The "prior position of the target joint" in step S10411 may include any one of the following: the three-dimensional position of the target joint in the pose to be optimized corresponding to the previous image frame, the three-dimensional position of the target joint in the current image frame obtained from the motion prior, and the three-dimensional position of the target joint in the pose to be optimized corresponding to the current image frame.

[0073] Among them, motion priors are general rules and constraints about dynamic sequence changes learned from a large amount of motion data, which are used to guide the generation of motion results that conform to real logic.

[0074] Specifically, the "prior direction" in step S10412 refers to the two-dimensional projection direction of the projection from the three-dimensional position of the anchor joint to the prior position of the target joint onto the image frame.

[0075] In one embodiment, step S10413 may specifically include: if there is no prior position of the target joint, the detection confidence level can be set to 1.0 or other preset constants.

[0076] In other possible implementations, if there are situations such as invisible anchor joints, invalid cropping, abnormal network output, or severe image occlusion in the image frame, the detection confidence can be set to 0 or other low values.

[0077] In one embodiment, step S10412 may further include step S104121: Step S104121: When the dot product between the two-dimensional unit projection direction of the image frame and the prior direction is negative, reverse the corresponding two-dimensional unit projection direction to eliminate the 180-degree directional ambiguity.

[0078] In one embodiment, step S1042 may specifically include steps S10421 to S10423: Step S10421: Based on forward kinematics, determine the positions of the anchor joint and the target joint in three-dimensional space according to each posture to be optimized.

[0079] Step S10422: Based on the camera extrinsic parameters of each frame, transform the positions of the anchor joint and the target joint in three-dimensional space to the camera coordinate system to obtain the three-dimensional coordinates of the anchor joint and the target joint in the camera coordinate system.

[0080] Step S10423: Using the camera intrinsic parameters, project the three-dimensional coordinates of the anchor joint and the target joint onto the image plane corresponding to the image frame, respectively, to obtain the two-dimensional projection positions of the target joint and the anchor joint on the corresponding image frame.

[0081] Forward kinematics is a fundamental mapping method in robotics for solving the pose of an end effector using known joint parameters. It can directly calculate the position and orientation of the anchor joint in space through a series of coordinate system transformations, given the angles or displacements of each joint.

[0082] In one embodiment, step S1043 may further include step S10431: Step S10431: Determine the directional constraint residuals for each pose to be optimized according to the following formula (1): (1) in, The orientation constraint residuals for the attitude to be optimized are... Let be the detection confidence of the image frame corresponding to the pose to be optimized. The preset directional constraint weights, Let be the two-dimensional projection position of the target joint in the corresponding image frame, representing the pose to be optimized. Let represent the two-dimensional projection position of the anchor joint in the corresponding image frame for the pose to be optimized. This is a preset constant value. The orientation is the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized.

[0083] in, It can be a very small positive number, specifically used to prevent the denominator from being zero and to increase the stability of numerical calculations; for example, it can be equal to 1e-8. It is the L2 norm (i.e., the magnitude) of the vector. It is normalized by division to obtain the two-dimensional projection direction corresponding to the current pose to be optimized.

[0084] In one implementation, step S104, "guided optimization of the pose to be optimized based on the orientation constraint residual to obtain the optimized human pose estimation result," may further include steps S1044 and S1045: Step S1044: Add the orientation constraint residuals of each pose to be optimized to the nonlinear least squares optimizer.

[0085] Step S1045: Based on the nonlinear least squares optimizer, with the goal of minimizing the directional constraint residual, the three-dimensional human pose sequence output by the diffusion model is guided to optimize, and the optimized human pose estimation result is obtained.

[0086] Nonlinear least squares optimizers are a class of specialized numerical computation tools for solving nonlinear least squares problems. They minimize the sum of squares of multiple nonlinear residual functions through iterative optimization, ultimately obtaining the optimal estimate of the attitude to be optimized. For example, a specific nonlinear least squares optimizer can be the Levenberg-Marquardt (LM) optimizer, which employs a nonlinear least squares optimization algorithm that lies between the Gauss-Newton method and gradient descent.

[0087] In another embodiment, see the appendix for details. Figure 3 , Figure 3 This is a schematic flowchart illustrating the main steps of a human pose estimation method according to another embodiment of this application. Figure 3 As shown, this method may specifically include steps S201 to S206: Step S201: Acquire input data based on the head-mounted device.

[0088] Step S202: Based on the camera intrinsic parameters and the camera extrinsic parameters of each image frame, project the three-dimensional position of the anchor joint onto the image plane of the corresponding image frame to obtain the two-dimensional coordinates of the anchor point.

[0089] Step S203: Using the two-dimensional coordinates of the anchor point as the center, crop a fixed-size image region from the image frame and preprocess the image region.

[0090] Step S204: Input the preprocessed image region and the limb identifier corresponding to the anchor joint into the limb orientation prediction network to obtain the two-dimensional unit projection orientation of the limb where the anchor joint is located in the image plane.

[0091] Step S205: Based on the diffusion model, generate a three-dimensional human pose sequence according to the head position trajectory.

[0092] Step S206: In the guided optimization stage after sampling of the diffusion model, the two-dimensional unit projection direction is added as the directional constraint residual to the nonlinear least squares optimizer to optimize the three-dimensional human pose sequence and obtain the optimized human pose estimation result.

[0093] Specifically, the "input data" in step S201 may include video images acquired by the head-mounted device, the head six-degree-of-freedom pose trajectory output by the set SLAM system, the camera intrinsic parameter matrix, the camera extrinsic parameters corresponding to each image frame, and the three-dimensional positions of the left and right wrists obtained by the hand tracking system or other sensors on the head-mounted device.

[0094] In other implementations, if the actual application scenario also involves human lower limb pose estimation, the input data can also include the three-dimensional position of the left and right ankles or the ankle position inferred from the prior of the current pose.

[0095] In step S202, the “anchor point two-dimensional coordinates” is equivalent to the “anchor point joint two-dimensional projection position” in the above embodiment.

[0096] Optionally, the "fixed size" in step S203 can be specifically 192×192 pixels.

[0097] As an example, in step S203, for the left and right forearms, the cropping can be centered on the projection points of the left and right wrists, respectively, and different limb markers can be determined. For the lower leg, the cropping can be centered on the projection points of the left and right ankles, respectively.

[0098] In step S204, the "limb orientation prediction network" can utilize synthetic data, real-world data, and other data as training data during pre-training. Synthetic data can be obtained by labeling the two-dimensional projection positions of target joints using existing 3D motion capture sequences, parametric human models, and camera projections. Real-world data can be obtained by labeling the two-dimensional projection positions of target joints using multi-view calibration, 3D tracking systems, or manual / semi-automatic annotation. In a possible implementation, the limb orientation can also be obtained by subtracting the two-dimensional coordinates of the anchor joint from the two-dimensional coordinates of the target joint and then normalizing the result, serving as validation data for the network.

[0099] In other possible implementations, data augmentation techniques such as horizontal flipping, color perturbation, cropping center perturbation, and scale perturbation can be used during the training process of the limb orientation prediction network to generate new training data to expand the training data and improve the generalization ability of the limb orientation prediction network. Among them, during horizontal flipping, the left and right limb labels can be exchanged simultaneously, and the horizontal component of the limb orientation vector can be adjusted accordingly.

[0100] In one implementation, the limb orientation prediction network may further include a shared visual encoder and a multilayer perceptron prediction head. See the appendix for details. Figure 4 , attached Figure 4 This is a schematic flowchart illustrating the main steps of obtaining the two-dimensional unit projection direction in one embodiment of this application. Figure 4 As shown, step S204 may specifically include steps S2041 to S2048: Step S2041: Input the image region into the shared visual encoder.

[0101] Step S2042: The shared visual encoder uses multiple convolutional layers to sample the image region and performs adaptive global average pooling to obtain the visual feature vector.

[0102] Step S2043: Determine the limb identifier corresponding to the anchor joint.

[0103] Step S2044: Perform one-hot encoding on the limb identifiers to obtain multidimensional feature vectors.

[0104] Step S2045: Concatenate the visual feature vector and the multidimensional feature vector to obtain the conditional feature vector.

[0105] Step S2046: Input the concatenated conditional feature vector into the multilayer perceptron prediction head.

[0106] Step S2047: Perform L2 normalization on the two-dimensional vector output by the multilayer perceptron prediction head.

[0107] Step S2048: Based on the normalization results, obtain the two-dimensional unit projection direction of the limb where the anchor joint is located.

[0108] In a specific implementation scenario, steps S2041 and S2042 may include: inputting the image region into the shared visual encoder in a tensor format with dimensions of "3 channels × 192 pixels × 192 pixels". The shared visual encoder may sequentially include: a first convolutional layer that maps the 3-channel input to 32-channel features and performs downsampling with a stride of 2; multiple depthwise separable convolutional modules whose channel numbers can be successively changed to 64, 128, 128, 256, 256, 512, 512, and 256, and performing spatial downsampling with a stride of 2 in several layers; and finally obtaining a 256-dimensional visual feature vector through adaptive global average pooling.

[0109] Each depthwise separable convolutional module consists of 3×3 channel-wise depthwise convolutions and 1×1 pointwise convolutions, combined with batch normalization and the ReLU6 activation function. The ReLU6 activation function is a piecewise function designed specifically for low-precision computing scenarios on mobile devices, which can strictly limit the output to the range of 0 to 6.

[0110] In a specific implementation scenario, steps S2044 and S2048 may include: obtaining a 4-dimensional vector from the limb identifier through one-hot encoding; concatenating this 4-dimensional vector with 256-dimensional visual features to obtain 260-dimensional conditional features; and inputting these features into a multilayer perceptron prediction head, which may include a first fully connected layer, ReLU activation, a Dropout layer, and a second fully connected layer. The second fully connected layer of the multilayer perceptron prediction head outputs a 2-dimensional vector, which is then L2 normalized to obtain a 2-dimensional unit projection direction. This 2-dimensional unit projection direction represents the direction from the 2-dimensional projection position of the anchor joint to the 2-dimensional projection position of the target joint, and is represented by cosine and sinine components.

[0111] Specifically, the "diffusion model" in step S205 can be a denoised diffusion implicit model. Step S205 can specifically include: using the head pose trajectory as input, and generating a packaged human motion trajectory representation from Gaussian noise step-by-step based on the denoised diffusion implicit model. For long sequences, the entire sequence can be divided into multiple windows with overlapping regions. Each window is denoised separately, and weighted fusion is performed in the overlapping regions to obtain a complete human 3D pose sequence. The output of the diffusion model can include the human root node pose, local rotations of various joints, and other motion state variables.

[0112] In other possible implementations, the diffusion model is not limited to a specific network structure; it can use Transformer, temporal convolutional networks, or other denoising networks as diffusion denoisers. The sampling method can employ Denoising Diffusion Probabilistic Models (DDPM) or other diffusion sampling methods.

[0113] In a specific application scenario, during the guided optimization phase, the LM optimizer can be used as a nonlinear least squares optimizer. Step S206 can specifically include: the LM optimizer uses the 3D human pose sequence output by the diffusion model as initial values ​​to minimize the sum of squares of multiple observations and prior residuals. These residuals can include orientation constraint residuals, foot contact residuals, hand position residuals, pose smoothing residuals, or other task-related residuals.

[0114] In step S206, for each frame with valid orientation detection and each valid limb, the process of obtaining the orientation constraint residual can specifically include: performing forward kinematics based on the optimization variables of each posture to be optimized in the three-dimensional human posture sequence, calculating the three-dimensional positions of the anchor joint and the target joint; subtracting the centroid offset from the Z-up centroid coordinate system inside the head-mounted device, and transforming it to the Y-up original world coordinate system through a coordinate transformation matrix; transforming it to the camera coordinate system using the camera extrinsic parameters of the corresponding frame; performing perspective projection through the camera intrinsic parameters to obtain the two-dimensional pixel positions of the anchor joint and the target joint; subtracting the two-dimensional pixel position of the anchor joint from the two-dimensional pixel position of the target joint and normalizing it to obtain the current posture prediction direction; multiplying the difference between the prediction direction and the two-dimensional unit projection direction by the detection confidence and the preset orientation constraint weight to obtain the orientation constraint residual.

[0115] The optimization variables can include local rotations of body joints in the Skinned Multi-Person Linear Model with Hands (SMPL-H) or other parametric human models, expressed for example as SO(3) quaternions or other rotational representations. Specifically, the SMPL-H model can add 15 independent joints to each hand, based on the 24 trunk joints of the basic SMPL series parametric human models, resulting in a total of 52-54 joints, enabling precise hand gesture reproduction; SO(3) quaternions are a mathematical representation of elements in a three-dimensional special orthogonal group using unit quaternions. Step S206 also includes a process for determining the detection confidence level, which is the same as the method for determining the detection confidence level in the above embodiments, and will not be described in detail here.

[0116] In one possible implementation, data can be stored according to the image frame index and a pre-defined data structure to construct directional constraint residuals for each valid frame during the subsequent guided optimization phase. For example, the data structure may include at least the valid image frame index, left or right limb direction, corresponding detection confidence, camera extrinsic parameters for each frame, camera intrinsic parameters, and coordinate system centroid offset.

[0117] The preset directional constraint weights can be set according to task requirements. For example, when the limb representation corresponds to the forearm, the directional constraint weight is set to 5.0. Furthermore, the number of optimization iterations of the LM optimizer can be set to 20, and the initial damping coefficient of the LM optimizer can be set to 0.1. In other possible implementations, those skilled in the art can also adjust the above parameters according to data noise and real-time requirements.

[0118] Through the above configuration, this application can fully utilize the acquired video images and the three-dimensional positions of the anchor joints to obtain the two-dimensional unit projection direction of the anchor joints in each image frame. Then, based on the two-dimensional unit projection direction, it generates the directional constraint residual of the posture to be optimized for each image frame, thereby guiding the optimization of the posture during the guided optimization process. In this way, based on the two-dimensional unit projection direction of the limb where the anchor joint is located, the two-dimensional limb direction signal in the video image can be introduced into the three-dimensional human posture optimization process. This allows for constraint of the direction of the limb (such as the forearm or lower leg) where the anchor joint is located, even when the anchor joint positions are the same (e.g., wrist or ankle), and further constraint of the bending direction of intermediate joints such as the elbow or knee joints, effectively reducing ambiguity of intermediate joints and improving the accuracy of human posture estimation.

[0119] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0120] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0121] Another aspect of this application provides a computer-readable storage medium.

[0122] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium may be configured to store a program that performs the human pose estimation method of the above-described method embodiments. This program may be loaded and run by a processor to implement the above-described human pose estimation method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium may be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0123] Another aspect of this application provides a controller.

[0124] In one embodiment of a controller according to this application, the controller may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the human pose estimation method described in any of the above embodiments.

[0125] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for estimating human pose, characterized in that, The method includes: Based on a head-mounted device, the three-dimensional position of at least one anchor joint on the target human body, the head posture trajectory, and a video image containing the anchor joint are acquired. For each image frame in the video image, the two-dimensional unit projection direction of the limb where the anchor joint is located in the image frame is determined based on the three-dimensional position of the anchor joint and the pre-trained limb direction prediction network. Based on the head position trajectory, pose estimation is performed to generate a three-dimensional human pose sequence of the target human body; For each pose to be optimized in the three-dimensional human pose sequence, the directional constraint residual of the pose to be optimized is determined according to the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized. The pose to be optimized is then guided and optimized according to the directional constraint residual to obtain the optimized human pose estimation result.

2. The human posture estimation method according to claim 1, characterized in that, The step of determining the orientation constraint residual of the pose to be optimized based on the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized includes: The detection confidence level corresponding to each image frame is determined based on the two-dimensional unit projection direction of each image frame. For each pose to be optimized in the three-dimensional human pose sequence, the two-dimensional projection position of the target joint and the two-dimensional projection position of the anchor joint on the corresponding image frame are determined. The target joint is the middle joint of the limb where the anchor joint is located. Based on the detection confidence of the corresponding image frame, the two-dimensional projection position of the target joint, the two-dimensional projection position of the anchor joint, and the two-dimensional unit projection direction, the directional constraint residual corresponding to each posture to be optimized is determined.

3. The human posture estimation method according to claim 2, characterized in that, The determination of the directional constraint residual corresponding to each posture to be optimized includes: The directional constraint residual for each pose to be optimized is determined using the following formula: ， in, The orientation constraint residual of the attitude to be optimized is... The detection confidence score is the image frame corresponding to the pose to be optimized. The preset directional constraint weights, The two-dimensional projection position of the target joint on the corresponding image frame represents the pose to be optimized. Let represent the two-dimensional projection position of the anchor joint in the corresponding image frame for the pose to be optimized. This is a preset constant value. The orientation is the two-dimensional unit projection direction of the image frame corresponding to the pose to be optimized.

4. The human posture estimation method according to claim 2, characterized in that, The step of determining the detection confidence level corresponding to each image frame based on the two-dimensional unit projection direction of each image frame includes: For each image frame, if there is a prior position of the target joint, the prior direction is determined based on the prior position of the target joint and the three-dimensional position of the anchor joint; the absolute value of the dot product between the two-dimensional unit projection direction of each image frame and the prior direction is determined as the detection confidence. If there is no prior location for the target node, the detection confidence level is set to a preset value. The prior position of the target joint includes any one of the following: the three-dimensional position of the target joint in the pose to be optimized corresponding to the previous image frame, the three-dimensional position of the target joint in the current image frame obtained from the motion prior, and the three-dimensional position of the target joint in the pose to be optimized corresponding to the current image frame.

5. The human posture estimation method according to claim 4, characterized in that, The method further includes: When the dot product between the two-dimensional unit projection direction of the image frame and the prior direction is negative, the corresponding two-dimensional unit projection direction is reversed to eliminate the 180-degree directional ambiguity.

6. The human pose estimation method according to claim 1, characterized in that, The step of determining the two-dimensional unit projection direction of the limb containing the anchor joint in the image frame based on the three-dimensional position of the anchor joint and a pre-trained limb orientation prediction network includes: Based on the three-dimensional position of the anchor joint, determine the two-dimensional projection position of the anchor joint on the image frame; The image frame is cropped according to the two-dimensional projection position of the anchor joint to obtain an image region of a preset size; Based on the anchor joint, a limb identifier is determined, wherein the limb identifier is an identifier used to indicate the limb where the anchor joint is located; Based on a pre-trained limb orientation prediction network, the two-dimensional unit projection orientation is determined according to the image region and the limb identifier.

7. The human posture estimation method according to claim 6, characterized in that, Determining the two-dimensional projection position of the anchor joint on the image frame based on the three-dimensional position of the anchor joint includes: Obtain the camera intrinsic parameters of the head-mounted device and the camera extrinsic parameters corresponding to the image frame; Based on the camera intrinsic and extrinsic parameters, the three-dimensional position of the anchor joint is projected from perspective to obtain the corresponding two-dimensional pixel coordinates as the two-dimensional projection position of the anchor joint.

8. The human posture estimation method according to claim 7, characterized in that, Before performing perspective projection on the three-dimensional position of the anchor joint, the method further includes: Based on the camera extrinsic parameters, the three-dimensional position of the anchor joint is converted into three-dimensional coordinates in the camera coordinate system; Based on the ordinate in the three-dimensional coordinate system, determine whether the three-dimensional position of the anchor joint is valid; If effective, proceed to step "project the three-dimensional position of the anchor joint from a perspective view". If invalid, the image frame is determined to be unusable, and the two-dimensional projection position of the next image frame is obtained.

9. The human posture estimation method according to claim 6, characterized in that, The limb orientation prediction network includes a shared visual encoder and a multilayer perceptron prediction head; The pre-trained limb orientation prediction network determines the two-dimensional unit projection orientation based on the image region and the limb identifier, including: Based on the shared visual encoder, feature extraction is performed on the image region to obtain the visual feature vector of the image region; The limb identifier is one-hot encoded to obtain a multi-dimensional feature vector, and the multi-dimensional feature vector is concatenated with the visual feature vector to obtain a conditional feature vector; The conditional feature vector is input into the multilayer perceptron prediction head to obtain a two-dimensional vector, and the two-dimensional vector is L2 normalized to obtain the two-dimensional unit projection direction. Wherein, the two-dimensional unit projection direction is a unit direction vector used to represent the two-dimensional projection position from the two-dimensional projection position of the anchor joint to the two-dimensional projection position of the target joint, and the target joint is the intermediate joint of the limb where the anchor joint is located.

10. The human pose estimation method according to claim 6, characterized in that, The step of cropping the image frame based on the two-dimensional projection position of the anchor joint to obtain an image region of a preset size includes: The image frame is cropped with the two-dimensional projection position of the anchor joint as the center to obtain an image area of ​​a preset size.

11. The human pose estimation method according to claim 10, characterized in that, "Cropping the image frame" further includes: Based on the preset size, determine whether the minimum distance between the two-dimensional projection position of the anchor joint and the boundary of the image frame is less than a preset distance threshold. If so, after filling the boundaries of the image frame, the image region is cropped with the two-dimensional projection position of the anchor joint as the center to obtain an image region of a preset size; The image region of the preset size is preprocessed to obtain the final image region corresponding to the anchor joint. The preprocessing includes at least one of format conversion, normalization, and standardization.

12. The human pose estimation method according to claim 1, characterized in that, The step of estimating the pose based on the head position trajectory to generate a three-dimensional human pose sequence for the target human body includes: Based on the diffusion model, the pose is estimated according to the head position trajectory, and a three-dimensional human pose sequence of the target human body is generated.

13. The human posture estimation method according to claim 12, characterized in that, The step of guiding and optimizing the pose to be optimized based on the directional constraint residual to obtain the optimized human pose estimation result includes: Add the orientation constraint residual of each attitude to be optimized to the nonlinear least squares optimizer; Based on the nonlinear least squares optimizer, with the goal of minimizing the directional constraint residual, the three-dimensional human pose sequence output by the diffusion model is guided to optimize, and the optimized human pose estimation result is obtained.

14. A controller, characterized in that, The controller includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the human pose estimation method according to any one of claims 1 to 13.

15. A computer storage medium storing a plurality of program codes, characterized in that... The program code described herein is adapted to be loaded and run by a processor to perform the human pose estimation method according to any one of claims 1 to 13.

Images