Human body posture estimation method and device, electronic equipment and storage medium
The human pose estimation model trained by model predictive control algorithm and physical constraints solves the problems of accuracy and stability of pose estimation in complex dynamic scenarios and generates pose results that conform to the laws of human movement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-03-31
Smart Images
Figure CN121768066A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a human pose estimation method, apparatus, electronic device, and storage medium. Background Technology
[0002] Human pose estimation, as one of the core technologies in computer vision, aims to accurately locate the positions of key points on the human body from input images or videos, thereby depicting the human body's pose structure. In recent years, with the rapid development of deep learning technology, human pose estimation methods based on convolutional neural network models have become the mainstream technical approach. These methods achieve good estimation results in static scenes with simple backgrounds, but still face many challenges in dynamic and complex scenes.
[0003] Specifically, these purely data-driven models rely entirely on visual features for human pose prediction, lacking an intrinsic understanding of the laws governing human movement. Therefore, when faced with complex dynamic scenes such as occlusion, sudden changes in lighting, or rapid movement, their pose estimation results often contain significant noise and errors, and may even generate human poses that do not conform to the laws of human kinematics and are physically unreasonable.
[0004] Therefore, how to obtain posture estimation results that conform to the laws of human movement in physics and improve the accuracy and stability of human posture estimation in complex dynamic scenarios are technical problems that urgently need to be solved. Summary of the Invention
[0005] This application provides a human posture estimation method, device, electronic device, and storage medium, which can obtain posture estimation results that physically conform to the laws of human movement, and improve the accuracy and stability of human posture estimation in complex dynamic scenarios.
[0006] In a first aspect, embodiments of this application provide a human pose estimation method, including:
[0007] Obtain the image to be processed;
[0008] Human detection is performed on the image to be processed to obtain a human detection box;
[0009] The human body detection box is input into the human posture estimation model to obtain a skeleton heatmap. The human posture estimation model is obtained by training guided by a model predictive control algorithm using a dynamic model and physical constraints.
[0010] Based on the skeleton heatmap, human pose estimation results are generated.
[0011] In one possible implementation of the first aspect, the training of the human pose estimation model includes:
[0012] Obtain the training image sequence containing the real labels;
[0013] The training image sequence is input into the human pose estimation model to be trained to obtain the initial skeleton heatmap and initial key point coordinates.
[0014] The initial keypoint coordinates are optimized across frames using a model predictive control algorithm to obtain optimized keypoint coordinates. The model predictive control algorithm is based on a dynamic model and physical constraints for optimization.
[0015] Based on the optimized key point coordinates, an optimized skeleton heatmap is generated;
[0016] Based on the initial skeleton heatmap, the optimized skeleton heatmap, and the real labels, a synthesis loss function is constructed and calculated;
[0017] Based on the synthetic loss function, the parameters of the human pose estimation model to be trained are adjusted until the human pose estimation model to be trained converges, thus obtaining the trained human pose estimation model.
[0018] In one possible implementation of the first aspect, the cross-frame optimization of the initial keypoint coordinates using a model predictive control algorithm includes:
[0019] The key points of multiple frames of images are jointly optimized within a preset prediction time domain, which includes the current frame and several future frames.
[0020] In one possible implementation of the first aspect, the model predictive control algorithm performs optimization based on a dynamic model and physical constraints, including:
[0021] A dynamic model is applied to predict the state of key points. The dynamic model describes the evolution of human motion state over time through a state transition function. The state transition function is characterized as a functional relationship between the state of key points at the next moment, the state of key points at the current moment, and the control input.
[0022] Under the physical constraints, optimization is performed within the prediction time domain with the objective of minimizing the preset cost function, resulting in an optimized sequence of keypoint coordinates. The physical constraints include keypoint state constraints, structural constraints, and control constraints. The preset cost function is used to calculate the error between the predicted keypoint state and the reference keypoint state, the magnitude of the control input, and the weighted sum of the errors between the keypoint state and the corresponding reference keypoint state at the end of the prediction time domain. The reference keypoint state is determined based on the initial keypoint coordinates.
[0023] In one possible implementation of the first aspect, the keypoint state constraints limit the keypoint position and velocity to a physiologically feasible range; the structural constraints are based on human body structure, including angular constraints of human joints and / or relative positional constraints between adjacent keypoints; and the control constraints limit the range of variation of the control input.
[0024] In one possible implementation of the first aspect, the preset cost function includes a state tracking error term and a control smoothness penalty term, wherein the state tracking error term is used to measure the difference between the predicted key point state and the reference key point state; and the control smoothness penalty term is used to measure the degree of change of the control input.
[0025] In one possible implementation of the first aspect, constructing and calculating the synthesis loss function based on the initial skeleton heatmap, the optimized skeleton heatmap, and the ground truth labels includes:
[0026] The original data loss is obtained based on the error between the initial skeleton heatmap and the real heatmap generated from the real labels;
[0027] The model predictive control optimization loss is obtained based on the error between the optimized skeleton heatmap and the initial skeleton heatmap.
[0028] The synthetic loss function is determined based on the original data loss and the model prediction control optimization loss.
[0029] Secondly, embodiments of this application provide a human posture estimation device, comprising:
[0030] An image acquisition unit is used to acquire the image to be processed.
[0031] A human body detection unit is used to perform human body detection on the image to be processed and obtain a human body detection box.
[0032] The heatmap acquisition unit is used to input the human body detection box into the human pose estimation model to obtain a skeleton heatmap. The human pose estimation model is obtained by training guided by a model prediction control algorithm.
[0033] The estimation result generation unit is used to generate human posture estimation results based on the skeleton heatmap.
[0034] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the human pose estimation method as described in the first aspect above.
[0035] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the human pose estimation method as described in the first aspect above.
[0036] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the human pose estimation method as described in the first aspect above.
[0037] In this embodiment, firstly, human detection is performed on the acquired image to be processed to obtain a human detection box. This accurately locates the human body region in the image, effectively eliminating the interference of background noise on subsequent pose estimation. This provides a focused and reliable input basis for subsequent model inference and effectively avoids key point prediction deviations caused by ambiguous human body region localization. Then, the human detection box is input into a specifically trained human pose estimation model to obtain a skeleton heatmap. This human pose estimation model is trained by a model prediction control algorithm. Since the model has learned the laws of motion through dynamic models and physical constraints during training, it automatically incorporates human dynamic rationality judgment when outputting the skeleton heatmap. Therefore, it can effectively avoid generating poses that do not conform to physiological structure, such as reverse joints, and effectively improve the accuracy and stability of human pose estimation in complex dynamic scenes. Finally, based on this skeleton heatmap that combines accuracy and rationality, an actual pose that can truly reflect the laws of human movement is generated, improving the reliability of the estimation results. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating the implementation of the human pose estimation method provided in the embodiments of this application;
[0040] Figure 2 This is a flowchart illustrating a specific implementation of training a human pose estimation model in the human pose estimation method provided in this application embodiment;
[0041] Figure 3 This is a flowchart illustrating a specific implementation of the human pose estimation method provided in this application, which optimizes the solution based on a dynamic model and physical constraints.
[0042] Figure 4This is a flowchart illustrating a specific implementation of the motion trajectory in step S205 of the human pose estimation method provided in this application embodiment;
[0043] Figure 5 This is a structural block diagram of the human pose estimation device provided in the embodiments of this application;
[0044] Figure 6 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0045] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0046] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0047] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0048] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0049] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0050] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0051] By way of example and not limitation, the human pose estimation method provided in this application is applicable to various types of electronic devices, including mobile phones, tablets, wearable devices, laptops, ultra-mobile personal computers (UMPCs), desktop computers, and servers. This application does not impose any limitations on the specific type of electronic device.
[0052] Figure 1 The implementation flow of the human pose estimation method provided in this application embodiment is illustrated. The method flow includes steps S101 to S104. The specific implementation principle of each step is as follows:
[0053] Step S101: Obtain the image to be processed.
[0054] The image to be processed contains the target human body and represents the raw visual information for which human pose estimation is needed. It can be a single static image or a frame from a video sequence. The image to be processed can include a single human body or multiple human bodies. The image format of the image to be processed includes, but is not limited to, JPEG, PNG, and BMP, and its resolution can vary depending on the image acquisition device and the application scenario.
[0055] Acquiring the image to be processed includes active acquisition and passive reception. In one possible implementation, active acquisition of the image to be processed includes: the electronic device acquiring the image to be processed from a video stream captured in real time by an image sensor (such as a CMOS or CCD camera) directly connected to it, or the electronic device acquiring the image to be processed from image data packets received from a network interface. Passive acquisition of the image to be processed includes: the electronic device reading a pre-stored image file as the image to be processed from local memory or cloud storage.
[0056] For example, in one application scenario, a surveillance camera deployed on-site continuously captures 1080p resolution video data at a rate of 25 frames per second. This video data is received in real time through a local area network interface and decoded into a series of consecutive RGB image frames. Each decoded image frame then constitutes the image to be processed described above.
[0057] In one possible implementation, the acquired image to be processed is preprocessed to improve the accuracy of subsequent human pose estimation. The preprocessing includes, but is not limited to, denoising (e.g., Gaussian filtering), size normalization (adjusting to the input size suitable for the human pose estimation model), and illumination equalization.
[0058] Step S102: Perform human detection on the image to be processed to obtain a human detection box.
[0059] Human detection refers to the process of identifying and locating human targets in an image using object detection technology. The purpose of human detection is to separate the human body region from the image background, eliminating interference from non-human targets (such as vehicles, buildings, furniture, etc.) for subsequent pose estimation. The human detection bounding box is the output of human detection; it is a bounding box used to define the human body region, typically a rectangle. The human detection bounding box is defined by a set of coordinate parameters, including the bounding box's position information in the image pixel coordinate system (e.g., represented by the coordinates of the top-left vertex (x1, y1) and the bottom-right vertex (x2, y2)), as well as the bounding box's width w and height h.
[0060] In one possible implementation, to adapt to the application requirements of complex dynamic scenes, a pre-trained human detection model is used to identify human targets in different poses, clothing, and partial occlusion conditions in the image to be processed.
[0061] In one possible implementation, the obtained human detection bounding box also includes a detection confidence parameter, which quantifies the reliability of the human detection result and ranges from 0 to 1. The closer the value is to 1, the higher the probability that the region is a human body. When multiple candidate human detection bounding boxes are output for the image to be processed, they are filtered based on a preset confidence threshold, and candidate human detection bounding boxes with a confidence level lower than the preset confidence threshold are removed to avoid misidentifying non-human targets as human bodies.
[0062] In one possible implementation, when the same human body is detected with overlapping bounding boxes multiple times, the human body detection box with the highest confidence is retained by the non-maximum suppression (NMS) algorithm to ensure that each human body corresponds to a unique valid human body detection box.
[0063] In one possible implementation, when the image to be processed contains multiple human targets, multiple non-overlapping human detection boxes are output simultaneously. Each human detection box corresponds to a human target, and each human detection box carries independent position information and confidence parameters, providing an independent input region for subsequent pose estimation of a single human body, so as to adapt to complex scenarios of multi-person interaction (such as group movement, social scenarios, etc.).
[0064] In this embodiment, accurate human detection and human detection bounding box output provide focused input for subsequent pose estimation model inference, effectively eliminating the influence of background interference on pose estimation, avoiding redundant calculations of the pose estimation model across the entire image, saving computing resources, and improving the robustness and accuracy of pose estimation.
[0065] Step S103: Input the human body detection box into the human posture estimation model to obtain the skeleton heatmap. The human posture estimation model is obtained by training guided by the model prediction control algorithm using dynamic model and physical constraints.
[0066] Human pose estimation models are deep learning models used to predict the positions of key points on a human body from a human detection bounding box. Unlike existing technologies, the human pose estimation model in this embodiment is obtained through training guided by a model predictive control (MPC) algorithm using a dynamic model and physical constraints. The model predictive control algorithm is a control theory method based on dynamic models for prediction and optimization. A dynamic model describes the evolution of human motion states, including the position, velocity, and other state parameters of key points on the body (such as the head, elbows, and knees), as well as the relationships between these parameters. It is used to quantify the continuity and correlation of human motion, and the dynamic model can predict the trend of human pose changes at several future moments based on the current state. Physical constraints are constraint rules determined based on human anatomy and kinematics.
[0067] In this embodiment, the model prediction control algorithm serves as a guide during the training phase of the human posture estimation model. It uses the dynamic model as the basis for motion laws and physical constraints as the rationality criteria. After the model outputs the initial posture estimation result, it predicts the future posture change trend through the dynamic model and judges whether the initial posture estimation result is reasonable by combining the physical constraints. Then, it generates a correction signal and propagates it back to the human posture estimation model, so that the human posture estimation model gradually learns the feature extraction and prediction methods that conform to the laws of human motion, and avoids the human posture estimation model learning posture features that violate physical common sense.
[0068] The skeleton heatmap is the output of the human pose estimation model, and is a pixel-level probability distribution map. This skeleton heatmap used for pose estimation is a standard multi-channel two-dimensional array, with each channel corresponding to a specific human keypoint. The value of each pixel within the same channel represents the probability that the location corresponds to a human keypoint; the higher the probability, the brighter the pixel, indicating a greater likelihood that the human keypoint is located there. The coordinates of the keypoints can be obtained by locating the probability peaks in the skeleton heatmap. For example, for a given frame of image, the human pose estimation model outputs a skeleton heatmap containing 17 channels, each corresponding to one of the 17 human keypoints.
[0069] Because the human posture estimation model incorporates the laws of human movement and physical constraints through model prediction and control algorithms during training, the probability distribution of each key point in the output skeleton heat map is naturally more reasonable and in line with the logic of human structure. For example, the heat peak of the knee joint key point will not appear in a position far away from the connection area between the thigh and the calf.
[0070] As one possible implementation of this application Figure 2 The following is a detailed description of a specific implementation process for training a human pose estimation model in the human pose estimation method provided in this application embodiment:
[0071] Step S201: Obtain the training image sequence containing the real labels.
[0072] The training image sequence is a dataset composed of multiple consecutively related images. Unlike single-frame static images, the training image sequence contains the temporal correlation of human motion, which can accurately reproduce the posture changes of the human body in dynamic scenes. The ground truth labels are the human posture baseline data corresponding to each frame in the training image sequence, used to supervise model training.
[0073] The content of the real-world label includes the pixel coordinates of key points of the human body (such as the top of the head, shoulder joint, elbow joint, hip joint, etc.), the relationship between key points, and the confidence level. Among them, the confidence level is used to indicate whether the key point is occluded (fully visible is labeled as 1.0, partially occluded is labeled as 0.5 to 0.9, and completely occluded is labeled as 0).
[0074] In one possible implementation, the time interval between image frames in the training image sequence is adapted to common dynamic capture frame rates, for example, it can be controlled within the range of 10ms to 33ms.
[0075] The training image sequence includes human samples of different ages, body types, and clothing, covering different shooting angles (front, side, and back) and shooting distances to avoid overfitting due to a single sample. In one possible implementation, the training image sequence prioritizes complex dynamic scene samples, with a higher proportion of dynamic scene samples than static scene samples in the dataset. Dynamic scene samples cover various real-world application scenarios, including rapid human movement (such as running and jumping), partial occlusion (such as limb overlap in group interactions), and lighting changes (such as indoor and outdoor light switching), ensuring a high degree of match between the sample distribution and the real-world scenarios during model inference, thereby improving the model's generalization ability. The sequence length of this training image sequence is set according to the complexity of the motion; simple action sequences (such as walking) have a length of 50 to 100 frames, while complex action sequences (such as dancing and ball sports) have a length of 200 to 500 frames.
[0076] Step S202: Input the training image sequence into the human pose estimation model to be trained to obtain the initial skeleton heatmap and initial key point coordinates.
[0077] The human pose estimation model to be trained is a deep learning model that has not been optimized by the model prediction control algorithm and only has basic pose estimation capabilities. Its network structure can be based on convolutional neural networks. The human pose estimation model to be trained can extract image features and output key point related prediction results, but it has not yet learned the dynamic laws and physical constraints of human movement.
[0078] The initial skeleton heatmap is the initial pose estimation result of the human pose estimation model to be trained on a single frame of the training image sequence. It is a pixel-level probability distribution map without optimization by the model's predictive control algorithm. The initial keypoint coordinates are keypoint location data extracted from the initial skeleton heatmap by locating the probability peaks of each channel. Specifically, they are obtained by obtaining the pixel coordinates corresponding to the maximum probability value in each channel. Each initial keypoint coordinate corresponds to the pixel position of the probability peak of a channel in the initial skeleton heatmap. For example, for a certain frame, the human pose estimation model outputs an initial skeleton heatmap containing 17 channels, corresponding to 17 human keypoints.
[0079] The human pose estimation model in this application supports batch input in units of sequences, and can process multiple frames of images in the training image sequence at the same time. The model initially captures the correlation of joint motion between frames through the temporal feature extraction layer inside the model, avoiding the break in motion law learning caused by single frame input, so that the initial pose estimation result has basic temporal coherence.
[0080] Step S203: Use the model predictive control algorithm to optimize the initial key point coordinates across frames to obtain the optimized key point coordinates. The model predictive control algorithm is based on dynamic models and physical constraints for optimization.
[0081] The model predictive control algorithm adopts a rolling time-domain optimization control strategy. In each control cycle, based on the current state of human key points and dynamic model, it iteratively solves an optimization problem within a finite prediction time domain, generating a supervision signal to guide the optimization of human pose estimation model parameters.
[0082] Cross-frame optimization refers to the optimization of image sequences using model predictive control algorithms, which simultaneously consider pose estimation results from multiple consecutive video frames, rather than processing a single frame in isolation. One possible implementation involves joint optimization of key points across multiple frames within a preset prediction time domain, which includes the current frame and several future frames. The prediction time domain refers to the sum of future time steps considered by the model predictive control algorithm when optimizing at the current moment. Specifically, a finite optimization window is defined, starting from the current frame and extending to several future frames. For example, if the prediction time domain is N, then when processing the k-th frame, the algorithm will simultaneously consider the sequence of frames k, k+1, ..., k+N.
[0083] In this embodiment, the pose of multiple frames within the prediction time domain is jointly optimized as a whole, which improves the pose estimation from the static image space to the dynamic spatiotemporal space. By fusing multi-frame image information, the prediction error of a single frame and the jitter across frames are reduced, which can reduce the jitter of key point prediction in the time dimension.
[0084] As one possible implementation of this application, such as Figure 3 As shown, the model predictive control algorithm optimizes the solution based on a dynamic model and physical constraints, including:
[0085] A1: Apply a dynamic model to predict the state of key points. The dynamic model describes the evolution of human motion state over time through a state transition function.
[0086] The state transition function represents the functional relationship between the key point state at the next time step and the key point state and control input at the current time step.
[0087] Dynamic models focus on capturing the continuity of motion. Human motion states are a set of parameters describing joint motion characteristics, including at least keypoint position and velocity information. In one possible implementation, human motion states also include acceleration information. Model predictive control algorithms employing dynamic model optimization processes must satisfy system dynamic constraints, which are specifically implemented through state transition functions. The state transition function defines the evolution of keypoint states over time, where the control input is the variable driving the changes in keypoint states. For example, the dynamic model is represented as x. k+1 =f(x) k uk ), x k x represents the state of the key point at time k. k x represents the control input at time k. k+1 This represents the keypoint state at time k+1. The dynamic model uses the keypoint state x at the current time k. k The control input at time k is the independent variable u. k Output the key point state x at time k+1. k+1 (i.e., the predicted state).
[0088] A2: Under the physical constraints, with the goal of minimizing the preset cost function, optimization is performed within the prediction time domain to obtain the optimized key point coordinate sequence.
[0089] Physical constraints are a set of limiting conditions that ensure the optimization results conform to human physiological characteristics. In one possible implementation, physical constraints include key point state constraints, structural constraints, and control constraints.
[0090] The key point state constraints limit the key point position and velocity to a physiologically feasible range, for example, the knee joint extension angle does not exceed 180° and the elbow joint flexion angle is not less than 30°; the structural constraints are based on human body structure, including angular constraints of human joints and / or relative position constraints between adjacent key points, for example, the spatial distance range between the shoulder joint and the elbow joint; the control constraints limit the range of changes in the control input to ensure motion continuity.
[0091] A preset cost function is used to calculate the error between the predicted key point state and the reference key point state, the magnitude of the control input, and the weighted sum of the errors between the key point state at the end of the predicted time domain and the corresponding reference key point state; wherein, the reference key point state is determined based on the initial key point coordinates.
[0092] In one possible implementation, the preset cost function includes a state tracking error term and a control smoothness penalty term. The state tracking error term measures the difference between the predicted key point state and the reference key point state. The control smoothness penalty term measures the degree of change in the control input.
[0093] In one possible implementation, the preset cost function further includes a terminal state error term. The terminal state error term measures the difference between the state of a key point at the end of the predicted time domain and the state of the reference key point.
[0094] For example, under physical constraints, with the objective of minimizing the cost function, the optimal control sequence U*=[uk,uk+1,...,uk+N-1] is solved, and the state transition function is represented as x.k+i+1 =f(x) k+i u k+i ), where i = 0, 1, ..., N-1;
[0095] Key point state constraints: x min ≤x k+i ≤x max Control constraints: u min ≤u k+i ≤u max ;
[0096] Structural constraints include angular constraints and positional constraints: angular constraint θ min ≤angle(p a p b p c )≤θ max , (p a p b p c () represents three adjacent key points on the human body, such as shoulder, elbow, and wrist; angle(shoulder, elbow, wrist) represents the flexion angle of the elbow joint; θ min and max This is based on a reasonable angular range between key points on the human body; relative position constraint d min ≤||p m —p n ||≤d max , representing adjacent human body key points p m With p n A fixed distance between and, where d min and d max This is based on a reasonable length range between key points on the human body.
[0097] The cost function expression is as follows: (1)
[0098]
[0099] Where, x k+i Let be the key point state vector at time k+i. Let u be the reference keypoint state vector corresponding to the keypoint state vector at time k+i. k+i Let N be the control input vector at time k+i, N be the prediction time domain length, and Q, R, and P be the weight matrices.
[0100] The state tracking error term is used to ensure that the state of the key points predicted by the dynamic model closely follows the state of the reference key points. The weight Q is used to determine the importance of tracking accuracy. The larger the Q is, the closer the predicted state of the key points follows the state of the reference key points. This represents the control smoothness penalty term, used to penalize drastic control actions and ensure the smoothness of the motion. The larger the weight matrix R, the stronger the penalty for drastic motion, and the smoother the output trajectory. This represents the terminal state error term, used to ensure that the attitude at the end of the prediction time domain is close to the reference value, thereby improving the stability and accuracy of long-term prediction.
[0101] This application constructs a complete model predictive control algorithm optimization solution framework. Based on the predictive capability of the dynamic model, the motion trajectory is adjusted in advance to avoid unreasonable posture. The motion prediction capability is established through the dynamic model. Physical constraints ensure that the results conform to human physiological laws. The tracking accuracy and motion smoothness are balanced through a preset cost function. Through joint optimization in the prediction time domain N, inter-frame jitter is eliminated, which fundamentally improves the physical rationality and temporal consistency of posture estimation.
[0102] For example, when processing walking video sequences, a dynamic model is first used to predict the lower limb movement trajectory for the next few frames based on the hip and knee joint states of the current frame. When the predicted knee flexion angle exceeds the physiological limit of 120 degrees, structural constraints limit it to a reasonable range; when the hip joint displacement between adjacent frames is too large, causing the calculation speed to exceed the human body limit, key point state constraints correct it to a reasonable range; at the same time, the state tracking error term in the cost function ensures that the optimized trajectory does not deviate from the visual observation, the smoothness penalty term suppresses motion jitter, and the terminal state error term ensures that in the last frame of the prediction period, the lower limb posture can accurately converge to a reasonable gait cycle endpoint, ultimately outputting a natural walking posture sequence that conforms to human physiological laws.
[0103] Step S204: Generate an optimized skeleton heatmap based on the optimized key point coordinates.
[0104] The optimized keypoint coordinates are the output results of cross-frame optimization by the model predictive control algorithm, representing the positions of each keypoint of the human body in the image pixel coordinate system. These optimized keypoint coordinates conform to the dynamic laws of the human body and physical constraints.
[0105] The optimized skeleton heatmap is a pixel-level probability distribution map obtained by transforming the coordinates of the optimized key points. The optimized skeleton heatmap has the same format as the initial skeleton heatmap, and the probability peak positions in the optimized skeleton heatmap are precisely matched with the coordinates of the optimized key points.
[0106] Step S205: Based on the initial skeleton heatmap, the optimized skeleton heatmap, and the real labels, construct and calculate the synthesis loss function.
[0107] The initial skeleton heatmap represents the original predictive ability of the human pose estimation model under training. The optimized skeleton heatmap represents the current predictive ability of the human pose estimation model after optimization guided by the model prediction control algorithm. The ground truth labels represent the real data of the training image sequence.
[0108] As one possible implementation of this application Figure 4 A specific implementation flow of step S205 in the human pose estimation method provided in this application embodiment is shown below:
[0109] B1: The raw data loss is obtained based on the error between the initial skeleton heatmap and the real heatmap generated from the real labels. The raw data loss is used to quantify the difference between the initial pose estimation result and the true pose.
[0110] B2: Based on the error between the optimized skeleton heatmap and the initial skeleton heatmap, the model predictive control optimization loss is obtained. The model predictive control optimization loss is used to quantify the difference between the initial attitude estimation result and the attitude estimation result after optimization guided by the model predictive control algorithm.
[0111] In one possible implementation, the original data loss and the model predictive control optimization loss are calculated using mean squared error (MSE) or peak signal-to-noise ratio (PSNR).
[0112] B3: Determine the synthetic loss function based on the original data loss and the model prediction control optimization loss.
[0113] In one possible implementation, the synthetic loss function is a weighted sum of the original data loss and the model predictive control optimization loss, and the synthetic loss function is as follows (2):
[0114] L=ω1×L1+ω2×L2 (2)
[0115] Where L1 is the original data loss, L2 is the model prediction control optimization loss, and ω1 and ω2 are the fusion weights, which can be dynamically adjusted according to the training stage.
[0116] This application embodiment constructs a synthetic loss function that integrates data authenticity and physical rationality to ensure that the loss value truly reflects the model accuracy.
[0117] Step S206: Adjust the parameters of the human pose estimation model to be trained according to the synthesis loss function until the human pose estimation model to be trained converges, and obtain the trained human pose estimation model.
[0118] In this embodiment, based on the gradient information of the synthetic loss function, the trainable parameters in the human pose estimation model are updated by the gradient backpropagation algorithm. After multiple iterations of training, when the value of the synthetic loss function stabilizes within a specific range and the performance of the model on the validation set does not improve within a preset number of iterations (when the performance no longer improves significantly), the human pose estimation model to be trained converges, and the trained human pose estimation model is obtained.
[0119] During the model training phase, cross-frame optimization is performed using a model prediction control algorithm; during the model application phase (inference phase), the trained human pose estimation model is directly used to infer the skeleton heatmap.
[0120] Step S104: Based on the skeleton heatmap, generate human pose estimation results.
[0121] The result of human pose estimation is a structured data set that integrates key point coordinates, joint connections and confidence scores. Specifically, it includes: two-dimensional pixel coordinates of each human key point, joint-related skeleton based on human structure (such as the upper limb skeletal chain of shoulder joint → elbow joint → wrist joint), and single key point confidence scores (measuring the reliability of the key point's localization) and overall pose confidence scores (measuring the rationality of the entire skeleton).
[0122] In one possible implementation, the human pose estimation result includes a human skeleton connection topology map formed based on the connection of key point coordinates.
[0123] In this embodiment of the application, by decoding the probabilistic form of the skeleton heatmap into a deterministic key point coordinate and skeleton connection topology map, the posture information becomes structured data that can be directly used by upper-layer applications such as action recognition and human-computer interaction.
[0124] As can be seen from the above, in this embodiment, firstly, human detection is performed on the acquired image to be processed to obtain a human detection box, which can accurately locate the human body region in the image, effectively eliminate the interference of background noise on subsequent pose estimation, and provide a focused and reliable input basis for subsequent model inference. This effectively avoids the key point prediction deviation caused by the fuzzy localization of the human body region. Then, the human detection box is input into a specifically trained human pose estimation model to obtain a skeleton heatmap. This human pose estimation model is trained by a model prediction control algorithm. Since the model has learned the motion law through dynamic models and physical constraints during training, it automatically incorporates the human body dynamic rationality judgment when outputting the skeleton heatmap. Therefore, it can effectively avoid generating poses that do not conform to physiological structure, such as reverse joints, and effectively improve the accuracy and stability of human pose estimation in complex dynamic scenes. Finally, based on the skeleton heatmap that combines accuracy and rationality, an actual pose that can truly reflect the human body's motion law is generated, improving the reliability of the estimation results.
[0125] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0126] Corresponding to the human pose estimation method described in the above embodiments, Figure 5 A structural block diagram of the human posture estimation device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.
[0127] Reference Figure 5 The human pose estimation device includes: an image acquisition unit 51, a human detection unit 52, a heatmap acquisition unit 53, and an estimation result generation unit 54, wherein:
[0128] Image acquisition unit 51 is used to acquire the image to be processed;
[0129] Human body detection unit 52 is used to perform human body detection on the image to be processed and obtain a human body detection box;
[0130] The heatmap acquisition unit 53 is used to input the human body detection box into the human posture estimation model to obtain a skeleton heatmap. The human posture estimation model is obtained by training guided by a model prediction control algorithm.
[0131] The estimation result generation unit 54 is used to generate human posture estimation results based on the skeleton heatmap.
[0132] As one possible implementation of this application, the human posture estimation device further includes:
[0133] The training image acquisition unit is used to acquire training image sequences containing real labels;
[0134] The model training unit is used to input the training image sequence into the human pose estimation model to be trained to obtain an initial skeleton heatmap and initial keypoint coordinates; to optimize the initial keypoint coordinates across frames using a model predictive control algorithm to obtain optimized keypoint coordinates, wherein the model predictive control algorithm is based on a dynamic model and physical constraints for optimization; to generate an optimized skeleton heatmap based on the optimized keypoint coordinates; to construct and calculate a synthesis loss function based on the initial skeleton heatmap, the optimized skeleton heatmap, and the ground truth labels; and to adjust the parameters of the human pose estimation model to be trained according to the synthesis loss function until the human pose estimation model to be trained converges to obtain a trained human pose estimation model.
[0135] As one possible implementation of this application, the cross-frame optimization of the initial keypoint coordinates using a model predictive control algorithm includes:
[0136] The key points of multiple frames of images are jointly optimized within a preset prediction time domain, which includes the current frame and several future frames.
[0137] As one possible implementation of this application, the model predictive control algorithm optimizes the solution based on a dynamic model and physical constraints, including:
[0138] A dynamic model is applied to predict the state of key points. The dynamic model describes the evolution of human motion state over time through a state transition function. The human motion state includes at least key point position information and velocity information. The state transition function is a functional relationship between the key point state at the next moment and the key point state at the current moment and the control input.
[0139] Under the physical constraints, optimization is performed within the prediction time domain with the objective of minimizing the preset cost function, resulting in an optimized sequence of keypoint coordinates. The physical constraints include keypoint state constraints, structural constraints, and control constraints. The preset cost function is used to calculate the error between the predicted keypoint state and the reference keypoint state, the magnitude of the control input, and the weighted sum of the errors between the keypoint state and the corresponding reference keypoint state at the end of the prediction time domain. The reference keypoint state is determined based on the initial keypoint coordinates.
[0140] As one possible implementation of this application, the key point state constraints limit the key point position and velocity to a physiologically feasible range; the structural constraints are based on human body structure, including angular constraints of human joints and / or relative position constraints between adjacent key points; the control constraints limit the range of change of the control input.
[0141] As one possible implementation of this application, the preset cost function includes a state tracking error term and a control smoothness penalty term. The state tracking error term is used to measure the difference between the predicted key point state and the reference key point state; the control smoothness penalty term is used to measure the degree of change of the control input.
[0142] As one possible implementation of this application, the step of constructing and calculating a synthesis loss function based on the initial skeleton heatmap, the optimized skeleton heatmap, and the real labels includes:
[0143] The original data loss is obtained based on the error between the initial skeleton heatmap and the real heatmap generated from the real labels;
[0144] The model predictive control optimization loss is obtained based on the error between the optimized skeleton heatmap and the initial skeleton heatmap.
[0145] The synthetic loss function is determined based on the original data loss and the model prediction control optimization loss.
[0146] As can be seen from the above, in this embodiment, firstly, human detection is performed on the acquired image to be processed to obtain a human detection box, which can accurately locate the human body region in the image, effectively eliminate the interference of background noise on subsequent pose estimation, and provide a focused and reliable input basis for subsequent model inference. This effectively avoids the key point prediction deviation caused by the fuzzy localization of the human body region. Then, the human detection box is input into a specifically trained human pose estimation model to obtain a skeleton heatmap. This human pose estimation model is trained by a model prediction control algorithm. Since the model has learned the motion law through dynamic models and physical constraints during training, it automatically incorporates the human body dynamic rationality judgment when outputting the skeleton heatmap. Therefore, it can effectively avoid generating poses that do not conform to physiological structure, such as reverse joints, and effectively improve the accuracy and stability of human pose estimation in complex dynamic scenes. Finally, based on the skeleton heatmap that combines accuracy and rationality, an actual pose that can truly reflect the human body's motion law is generated, improving the reliability of the estimation results.
[0147] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0148] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figures 1 to 4 The steps of any human pose estimation method are represented.
[0149] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figures 1 to 4 The steps of any human pose estimation method are represented.
[0150] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the following: Figures 1 to 4 The steps of any human pose estimation method are represented.
[0151] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps described in the various human pose estimation method embodiments above, for example... Figure 1 Steps S101 to S104 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of units 51 to 54 are shown.
[0152] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the electronic device 6.
[0153] The electronic device 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 6 may also include input / output devices, network access devices, buses, etc.
[0154] The processor 60 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0155] The memory 61 can be an internal storage unit of the electronic device 6, such as a hard disk or memory. The memory 61 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 61 can include both internal and external storage units of the electronic device 6. The memory 61 is used to store the computer program and other programs and data required by the electronic device. The memory 61 can also be used to temporarily store data that has been output or will be output.
[0156] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0157] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0159] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0160] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A human pose estimation method, characterized by, The method comprises: obtaining a to-be-processed image; performing human body detection on the to-be-processed image to obtain a human body detection frame; inputting the human body detection frame into a human body pose estimation model to obtain a skeleton heat map, the human body pose estimation model being obtained by using a model predictive control algorithm to guide training of a dynamic model and a physical constraint condition; generating a human body pose estimation result based on the skeleton heat map.
2. The method of claim 1, wherein, The training of the human body pose estimation model comprises: obtaining a training image sequence containing real labels; inputting the training image sequence into a to-be-trained human body pose estimation model to obtain initial skeleton heat maps and initial key point coordinates; performing cross-frame optimization on the initial key point coordinates by using a model predictive control algorithm to obtain optimized key point coordinates, wherein the model predictive control algorithm is used to optimize and solve based on a dynamic model and a physical constraint condition; generating an optimized skeleton heat map according to the optimized key point coordinates; constructing and calculating a synthetic loss function based on the initial skeleton heat map, the optimized skeleton heat map and the real labels; adjusting parameters of the to-be-trained human body pose estimation model according to the synthetic loss function until the to-be-trained human body pose estimation model converges, thereby obtaining a trained human body pose estimation model.
3. The method of claim 2, wherein, The cross-frame optimization of the initial key point coordinates by using the model predictive control algorithm comprises: jointly optimizing key points of multiple images within a preset prediction time domain range, the prediction time domain range including a current frame and a plurality of future frames.
4. The method of claim 3, wherein, The model predictive control algorithm is used to optimize and solve based on a dynamic model and a physical constraint condition, comprising: applying a dynamic model to predict key point states, the dynamic model describing the evolution relationship of human body motion states over time through a state transition function, the state transition function being represented as a functional relationship between a key point state at a next time and a key point state at a current time and a control input; under the physical constraint condition, optimizing and solving within the prediction time domain range to obtain an optimized key point coordinate sequence, with the goal of minimizing a preset cost function; the physical constraint condition includes key point state constraints, structure constraints and control constraints; the preset cost function is used to calculate the weighted sum of the error between the predicted key point state and the reference key point state, the size of the control input and the error between the key point state at the end of the prediction time domain and the corresponding reference key point state; wherein the reference key point state is determined based on the initial key point coordinates.
5. The method of claim 4, wherein, The key point state constraints limit the positions and velocities of the key points within a physiological feasible range; the structure constraints are based on the human body structure and include angle constraints of human body joints and / or relative position constraints between adjacent key points; The control constraints limit the variation amplitude of the control input.
6. The method of claim 4, wherein, The preset cost function includes a state tracking error term and a control smoothness penalty term, the state tracking error term being used to measure the difference between the predicted key point state and the reference key point state; the control smoothness penalty term being used to measure the degree of variation of the control input.
7. The method of claim 2, wherein, The synthetic loss function is constructed and calculated based on the initial skeleton heat map, the optimized skeleton heat map and the true label, including: According to the error between the initial skeleton heat map and the real heat map generated by the true label, the original data loss is obtained; According to the error between the optimized skeleton heat map and the initial skeleton heat map, the model predictive control optimization loss is obtained; Based on the original data loss and the model predictive control optimization loss, the synthetic loss function is determined.
8. A human pose estimation apparatus characterized by comprising: It comprises: An image acquisition unit is configured to acquire a to-be-processed image; A human body detection unit is configured to perform human body detection on the to-be-processed image to obtain a human body detection frame; A heat map acquisition unit is configured to input the human body detection frame into a human body pose estimation model to obtain a skeleton heat map, wherein the human body pose estimation model is obtained by model predictive control algorithm guided training; An estimation result generation unit is configured to generate a human body pose estimation result based on the skeleton heat map.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the human body pose estimation method of any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to realize the human body pose estimation method of any one of claims 1 to 7.