Human body posture estimation method based on motion and appearance time consistency
By constructing a neural network model that collaboratively utilizes the temporal consistency and motion continuity of human appearance, the robustness problem of video human pose estimation in complex scenarios is solved, and efficient and accurate human keypoint detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-03-17
AI Technical Summary
Existing video human pose estimation methods are not robust enough in motion blur, occlusion and complex backgrounds, cannot effectively utilize appearance information, have high computational costs, and are sensitive to noise and illumination changes.
A neural network model that collaboratively utilizes the temporal consistency of human appearance and the temporal continuity of motion is constructed. Through heatmap sequence extraction, appearance knowledge transfer, motion information extraction, and multi-scale feature fusion, accurate coordinates of human key points are generated.
It improves the robustness of pose estimation in complex scenarios such as occlusion and motion blur, reduces computational costs, and provides more reliable pose data support.
Smart Images
Figure CN121686567A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of video human key point detection, and particularly relates to a human pose estimation method based on motion and appearance time consistency. BACKGROUND
[0002] Human pose estimation is a fundamental and challenging task in computer vision, which plays a key role in applications such as motion capture and behavior analysis. Early human pose estimation methods mainly focus on image-based methods and have achieved good results. However, when applied to video sequences, their performance decreases significantly because they cannot capture temporal dynamics and effectively utilize inter-frame correlation.
[0003] To solve this problem, early video-based pose estimation methods proposed intuitive solutions, such as using recurrent neural networks (RNN) to capture inter-frame spatial consistency. Although RNN-based methods perform reasonably well on sparse single-person video sequences, they encounter significant limitations when extended to multi-person scenarios. Another approach uses optical flow to track joint motion trajectories across frames, however, these methods are usually computationally expensive and sensitive to image quality changes.
[0004] Recent research explores the continuity of human motion in time to extract temporal information and improve pose prediction accuracy. Although the above methods have made great progress in pose estimation results, they cannot solve the following problems: 1) only focus on modeling motion continuity, cannot utilize appearance information to enhance robustness; 2) performance significantly decreases in difficult scenarios such as motion blur and complex background; 3) sensitive to noise, lighting changes and other interference factors. Therefore, there is an urgent need to design a pose estimation method that can utilize both motion and appearance information and has stronger robustness. SUMMARY
[0005] The present application aims to address the deficiencies of the prior art and provides a video human pose estimation method based on motion and appearance time consistency. The present application constructs a neural network model that can cooperatively utilize human appearance temporal consistency and human motion temporal continuity to obtain more accurate and robust human key point coordinate information from video sequences, thereby solving the problems of insufficient pose estimation accuracy, severe temporal jitter, and other issues in the prior art under the influence of motion blur, severe occlusion, and complex background.
[0006] The technical solution to achieve the present application is a human pose estimation method based on motion and appearance time consistency, which comprises:
[0007] Step 1, constructing a human pose estimation dataset, which includes at least one input video sequence and the corresponding real key point heat map of each frame image of the input video sequence;
[0008] Step 2: Construct a human body key point coordinate prediction model. This model is a neural network model that collaboratively utilizes the temporal consistency of human appearance and the temporal continuity of human movement.
[0009] Step 3: Based on the human pose estimation dataset, perform end-to-end training on the human keypoint coordinate prediction model to generate the trained human keypoint coordinate prediction model.
[0010] Step 4: For the video sequence to be estimated, input it into the trained human keypoint coordinate prediction model and output the human keypoint coordinate detection results.
[0011] Step 5: Estimate human pose based on the detection results of human key point coordinates.
[0012] Furthermore, the human body key point coordinate prediction model in step 2 specifically includes:
[0013] Heatmap sequence extraction module: used to randomly extract the initial key point heatmaps of three consecutive frames in a video sequence to form a heatmap sequence;
[0014] Appearance knowledge transfer module: Based on the data of the two frames before and after the current frame, it performs joint-level knowledge matching and propagation on the initial keypoint heatmap of the current frame to generate appearance embedding features. ;
[0015] Motion information extraction module: Used to extract motion information features of the current frame based on data from the two frames before and after the current frame. ;
[0016] Multi-scale feature fusion module: used for embedding appearance features and motion information features Perform multi-scale fusion and output the final human keypoint coordinate prediction result for the current frame.
[0017] Furthermore, the heatmap sequence extraction module is used to randomly extract initial key point heatmaps from three consecutive frames of images in a video sequence to form a heatmap sequence, specifically including:
[0018] Randomly extract three consecutive frames from a video sequence and Then, crop and align the same person object i to obtain video clips. ;
[0019] Based on the video clip Using the pre-trained pose estimation backbone network HRNet-w48, the pose estimation backbone network HRNet-w48 is used to extract the pose of person object i from three frames of images. and Initial key point heatmap This forms a heatmap sequence.
[0020] Furthermore, the appearance knowledge transmission module is used to perform joint-level knowledge matching and propagation on the initial key point heatmap of the current frame based on the data of the two frames before and after the current frame, and generate appearance embedding features. Specifically, it includes:
[0021] Through a lightweight attitude nuclear distillation apparatus From the current frame adjacent frames and The attitude-aware features are distilled from the sample and encoded into attitude kernels. and ;
[0022] After optimizing the pose kernel using low-rank decomposition, a sliding window strategy is applied in the current frame. Joint-level knowledge matching and propagation are performed on the initial keypoint heatmap to generate appearance embedding features. .
[0023] Furthermore, the attitude core and They are represented as follows:
[0024]
[0025]
[0026] The optimization of the attitude kernel using low-rank decomposition specifically includes:
[0027] Full-size attitude core Decomposed into space nuclear base With channel level coefficient matrix The decomposition formula is:
[0028]
[0029] The attitude kernel is optimized using the following optimization objectives:
[0030]
[0031] In the formula, It is the kernel size. It is the number of channels. It refers to the number of key points in the human body. It is the Frobenius norm. for Norm, Represents convolution. This represents channel-level convolution. is a coefficient used to control the trade-off between reconstruction accuracy and rank sparsity.
[0032] Furthermore, the sliding window strategy in the current frame Joint-level knowledge matching and propagation are performed on the initial keypoint heatmap to generate appearance embedding features. Specifically, it includes:
[0033] For each keypoint, the following formula is used from the current frame The spatial regions closest to each keypoint (joint) are retrieved from adjacent frames, thereby obtaining a set of joint-level matching features:
[0034]
[0035]
[0036] In the formula, Represents convolution. These respectively represent the transmission through adjacent frames. and The current frame obtained by matching The Middle The first person Appearance information of key points; , These represent the previous frame respectively. and the next frame The attitude core;
[0037] The joint-level matching features corresponding to all key points are concatenated and aggregated to form appearance embedding features. :
[0038]
[0039] In the formula, This indicates an aggregation operation that iterates through all joints and splices them together. This indicates a splicing operation.
[0040] Furthermore, the motion information extraction module is used to extract motion information features of the current frame based on the data of the two frames preceding and following the current frame. Specifically, it includes:
[0041] The residual features of the heatmap sequences extracted by the heatmap sequence extraction module between adjacent frames are calculated and fused to obtain the fused residual features. ;
[0042] The heatmap sequences were grouped by joint to obtain joint temporal fusion features. ;
[0043] The fused residual features Joint temporal fusion features After processing the residual blocks, the data is aggregated and then modeled using a spatiotemporal attention mechanism to output motion information features. .
[0044] Furthermore, a linear distance weighting strategy is used to fuse the residual features, resulting in fused residual features. Represented as:
[0045]
[0046] in,
[0047]
[0048]
[0049] In the formula, express and The residual, express and The residual, For frame index, respectively representing the first frame. Frame, First Frame and the frame;
[0050] The heatmap sequence is grouped by joint to obtain joint temporal fusion features. Specifically, it includes:
[0051] Joint temporal fusion features are obtained by linear interpolation based on temporal distance. :
[0052]
[0053] In the formula, They represent the first The key point is in the first The first person Frame, First Frame and the Heatmap of key points in a frame.
[0054] Furthermore, the fused residual features Joint temporal fusion features After processing the residual blocks, the data is aggregated and then modeled using a spatiotemporal attention mechanism to output motion information features. Specifically, it includes:
[0055] (1) The fused residual features Joint temporal fusion features Fusion yields motion information features. :
[0056]
[0057] In the formula, As a group One residual block;
[0058] (2) Stacked motion information Divided into One block, of which Feature map for each key point, For the domain of the feature map, The height of the feature map, The width of the feature map. The height of the block. The width of the block;
[0059] Each block is stacked in the same position to form 3D Embedded Feature ,in express , express ;
[0060] (3) The projection is a query, key, and value, which are then input into the multi-head self-attention layer MSA for computation:
[0061]
[0062] In the formula, Representation layer normalization, This is the output of the multi-head self-attention layer MSA;
[0063] (4) Obtain updated motion information features through a multilayer perceptron. :
[0064] .
[0065] Furthermore, the multi-scale feature fusion module is used to embed appearance features. and motion information features Perform multi-scale fusion and output the final human keypoint coordinate prediction result for the current frame, specifically including:
[0066] Motion information features of the current frame By using deformable convolutions with multiple sets of different expansion rates d, offset maps corresponding to each expansion rate d are generated. With modulation mask ;
[0067] For each expansion rate d, embed the appearance features. Offset diagram and modulation mask Input to a DCNv2 deformable convolution, output the refined keypoint heatmap of the current frame corresponding to the dilation rate d. ;
[0068] Heatmap of all expansion rates d corresponding to the key refinement points Perform weighted average fusion to obtain the first Heatmap of key points for each character Thus, the first The final predicted coordinates of key points on the human body for each individual.
[0069] Further, step 3, which involves training the human keypoint coordinate prediction model end-to-end based on the human pose estimation dataset to generate the trained human keypoint coordinate prediction model, specifically includes:
[0070] Constructing the loss function :
[0071]
[0072] In the formula, It refers to the number of key points in the human body. For the first Visibility indicators of key points and They represent the first Predicted heatmaps and actual heatmaps for key points;
[0073] Based on the loss function Using real key point heatmaps as supervision signals, the human body key point coordinate prediction model is trained end-to-end to generate the trained human body key point coordinate prediction model.
[0074] Furthermore, the method also includes the following steps performed after step 4 and before step 5:
[0075] Perform time smoothing on the keypoint sequence output from step 4.
[0076] Compared with the prior art, the significant advantages of this invention are:
[0077] (1) By coordinating the use of temporal appearance consistency and motion continuity, the robustness of pose estimation in complex scenarios such as occlusion and motion blur is effectively improved.
[0078] (2) An efficient attitude kernel distillation and low-rank decomposition mechanism is introduced to significantly reduce computational costs while realizing cross-frame knowledge transfer, thus balancing high precision and high efficiency.
[0079] (3) Provides a complete technical solution from feature matching to multi-scale refinement, providing more reliable pose data support for upper-level applications such as anomaly detection.
[0080] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description
[0081] Figure 1 This is a schematic diagram of a human pose estimation method based on the temporal consistency of motion and appearance in one embodiment.
[0082] Figure 2 This is a schematic diagram of the human body key point coordinate prediction model in one embodiment.
[0083] Figure 3 This is a schematic diagram of Xavier initialization in one embodiment.
[0084] Figure 4 This is a histogram of human joint velocity distribution in one embodiment.
[0085] Figure 5 This is a visual comparison diagram of the method of the present invention and a prior art method in one embodiment, wherein... Figure 5 (a) in the figure represents a set of attitude estimation results corresponding to the algorithm of this invention. Figure 5 (b) in the table represents a set of attitude estimation results corresponding to the FAMI-Pose algorithm. Detailed Implementation
[0086] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0087] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0088] In one embodiment, combinedFigure 1 and Figure 2 This paper presents a human pose estimation method based on the temporal consistency of motion and appearance. This method improves the accuracy and robustness of human pose estimation in videos by jointly utilizing the temporal consistency of human appearance and the continuity of motion. The method includes:
[0089] Step 1: Construct a human pose estimation dataset, which includes at least one input video sequence and a heatmap of real key points corresponding to each frame of each input video sequence.
[0090] Here, the human pose estimation dataset can also directly use existing pose tracking datasets such as the PoseTrack dataset.
[0091] Step 2: Construct a human body key point coordinate prediction model. This model is a neural network model that collaboratively utilizes the temporal consistency of human appearance and the temporal continuity of human movement.
[0092] Step 3: Based on the human pose estimation dataset, perform end-to-end training on the human keypoint coordinate prediction model to generate the trained human keypoint coordinate prediction model.
[0093] Step 4: For the video sequence to be estimated, input it into the trained human keypoint coordinate prediction model and output the human keypoint coordinate detection results.
[0094] Step 5: Estimate human pose based on the detection results of human key point coordinates.
[0095] Furthermore, in one embodiment, the human body key point coordinate prediction model in step 2 specifically includes:
[0096] Heatmap sequence extraction module: used to randomly extract the initial key point heatmaps of three consecutive frames in a video sequence to form a heatmap sequence;
[0097] The Appearance Knowledge Transfer (AKT) module is used to perform joint-level knowledge matching and propagation on the initial keypoint heatmap of the current frame based on the data of the two frames before and after the current frame, generating appearance embedding features. ;
[0098] Motion Information Extraction Module (MIE): Used to extract motion information features of the current frame based on data from the two frames before and after the current frame. ;
[0099] Multi-scale feature fusion module: used for embedding appearance features and motion information features Perform multi-scale fusion and output the final human keypoint coordinate prediction result for the current frame.
[0100] Preferably, in some embodiments, the heatmap sequence extraction module is used to randomly extract initial key point heatmaps from three consecutive frames of images in a video sequence, specifically including:
[0101] Randomly extract three consecutive frames from a video sequence and (representing past frames, current frames, and future frames respectively), and cropping and aligning the same person object i within them to obtain video clips. ;
[0102] Based on the video clip Using the pre-trained pose estimation backbone network HRNet-w48, the pose estimation backbone network HRNet-w48 is used to extract the pose of person object i from three frames of images. and Initial key point heatmap .
[0103] Preferably, in the current frame The algorithm uses a human detector (such as Faster R-CNN) to obtain the bounding box of each human figure and expands it by a certain percentage, for example, 25%, to accommodate spatial changes caused by limb movements. Then, the expanded bounding boxes are used in... and The corresponding characters are then cropped to form aligned video clips, denoted as... ,in Representing the Each character segment is aligned. This alignment operation effectively alleviates the inconsistency in regions caused by character movement, providing spatially consistent input for subsequent cross-frame information fusion. After obtaining the aligned character segments, they are input into the pre-trained HRNet-w48 backbone network to extract the keypoint heatmap of each character in each frame. Here, HRNet-w48, as a high-resolution pose estimation network, can preserve the spatial details of the feature maps throughout the forward pass, thereby achieving accurate keypoint localization on static images. Each heatmap represents the possible locations of each keypoint (such as shoulder, elbow, wrist, etc.) in the image in the form of a two-dimensional probability distribution, providing high-quality initial pose estimation for subsequent modules.
[0104] Preferably, in some embodiments, the appearance knowledge transmission module is used to perform joint-level knowledge matching and propagation on the initial key point heatmap of the current frame based on the data of the two frames before and after the current frame, and generate appearance embedding features. Specifically, it includes:
[0105] Through a lightweight attitude nuclear distillation apparatus From the current frame adjacent frames and The attitude-aware features are distilled from the sample and encoded into attitude kernels. and ;
[0106] After optimizing the pose kernel using low-rank decomposition, a sliding window strategy is applied in the current frame. Joint-level knowledge matching and propagation are performed on the initial keypoint heatmap to generate appearance embedding features. .
[0107] The specific process is as follows:
[0108] 2.1) Attitude nucleus distillation
[0109] To effectively transmit the appearance information of human joints between frames, a pose kernel distillation based on a lightweight CNN is adopted. Distill pose-aware features from adjacent frames and encode them into the current frame. The attitude kernel. Specifically, given the HRNet-w48 backbone network from the first... The heatmaps extracted from the previous, current, and next frames of a character are denoted as follows: The attitude kernels for the previous and next frames are calculated as follows: and :
[0110]
[0111] In the formula, , It is the kernel size. It is the number of channels. It refers to the number of key points on the human body;
[0112] This invention decomposes full-size convolutional kernels into low-rank representations to reduce redundancy and improve efficiency. Each kernel is decomposed into a basic kernel and two coefficient matrices:
[0113]
[0114] in, Indicates space nuclear base, For channel-level coefficients, Represents convolution. This represents channel-level convolution. This decomposition enables the distilled kernel to efficiently encode joint-level appearance patterns. To enforce a low-rank structure and promote the sparsity of the coefficient matrix, this invention proposes the following optimization objectives:
[0115]
[0116] In the formula, It is the kernel size. It is the number of channels. It refers to the number of key points in the human body. It is the Frobenius norm. for Norm, Represents convolution. This represents channel-level convolution. is a coefficient used to control the trade-off between reconstruction accuracy and rank sparsity.
[0117] To improve the stability and convergence speed of model training, the Xavier initialization method is used to initialize the network parameters. Xavier initialization maintains consistency in variance between input and output, thereby mitigating the vanishing or exploding gradient problem. A graph illustrating the probability density function of this initialization strategy is shown below. Figure 3 .
[0118] Furthermore, the proposed kernel decomposition significantly reduces computational complexity. The complexity of the original kernel is:
[0119]
[0120] After decomposition, it is reduced to:
[0121]
[0122] in and These are the input and output channel dimensions, respectively. and For decomposition matrix and The rank of. Assume The theoretical speedup ratio is approximately:
[0123] ,
[0124] This confirms that low-rank constraints not only preserve expressive power but also significantly improve computational efficiency.
[0125] 2.2) Joint knowledge matching
[0126] To transfer joint-level appearance information from adjacent frames to the current frame, this invention performs spatial matching on joint patterns. Specifically, a sliding window strategy is used to distill the pose kernels obtained from the previous and next frames. and Heatmap applied to the current frame This operation retrieves the spatial region closest to each joint pattern from adjacent frames:
[0127]
[0128]
[0129] In the formula, Represents convolution. These respectively represent the transmission through adjacent frames. and The current frame obtained by matching The Middle The first person Appearance information of key points; , These represent the previous frame respectively. and the next frame The attitude core;
[0130] The joint-level matching features corresponding to all key points are concatenated and aggregated to form the first... Individual appearance embedded features :
[0131]
[0132] In the formula, This indicates an aggregation operation that iterates through all joints and splices them together. This indicates a splicing operation.
[0133] This joint-level matching process effectively encodes the spatial consistency of human appearance over time, enhancing robustness to occlusion, motion blur, and complex backgrounds, especially when visual cues in the current frame are weak or missing.
[0134] Preferably, in some embodiments, the motion information extraction module is used to extract motion information features of the current frame based on data from the two frames preceding and following the current frame. Specifically, it includes:
[0135] The residual features of the heatmap sequences extracted by the heatmap sequence extraction module between adjacent frames are calculated and fused to obtain the fused residual features. ;
[0136] The heatmap sequences were grouped by joint to obtain joint temporal fusion features. ;
[0137] The fused residual features Joint temporal fusion features After processing the residual blocks, the data is aggregated and then modeled using a spatiotemporal attention mechanism to output motion information features. .
[0138] The specific process is as follows:
[0139] 3.1) Heatmap Residual Fusion
[0140] Inter-frame heatmap residuals are used as motion cues to describe the dynamic changes of body joints. Given keypoint heatmaps from the previous, current, and next frames... and The residuals are calculated as follows:
[0141]
[0142]
[0143] These residuals are further modulated using a linear distance-weighted strategy to emphasize the contribution of temporally closer frames. The original residuals are then concatenated with the weighted residuals.
[0144]
[0145] in and For frame index, This indicates a splicing operation. The original pose residual is concatenated with its weighted version. This method ensures that bidirectional motion modes are fused with appropriate importance, thereby enhancing the modeling of inter-frame transitions.
[0146] Although this linear time interpolation strategy has proven effective, its initial design lacked a clear theoretical basis. To better support this scheme, this invention performs statistical analysis on the human motion velocity in the training video sequence. Given the centroid position of a person at frame t... The inter-frame motion speed is defined as:
[0147]
[0148] Figure 4 Empirical measurement results of the PoseTrack2017 validation set (JSON annotation) show that... The distribution exhibits a heavy-tailed characteristic. Low-speed instances, i.e., those close to zero, constitute a significant proportion, while high-speed instances gradually decrease. This skewed distribution aligns with the inherent temporal smoothness of human motion in real-world videos, where abrupt displacements are rare, and most inter-frame transitions are subtle and gradual pose changes. The statistical dominance of low-speed samples further supports the linear weighting strategy employed in this invention. Specifically, frames adjacent to the current frame are more likely to exhibit minimal motion deviations, thus their appearance and joint configurations are highly reliable for the interpolation process. Based on this, linearly decreasing weights prioritize frames that are closer in time, thereby optimally utilizing this motion continuity. Simultaneously, this strategy avoids overfitting due to sparse high-speed outliers, which introduce noise. This theoretical foundation ensures the method's robustness in scenarios dominated by slow motion (e.g., walking or gestures) while maintaining processing efficiency.
[0149] 3.2) Joint-time fusion
[0150] To better model the temporal continuity of each joint, a heatmap will be used. Grouped by joint, and linear interpolated based on time distance:
[0151]
[0152] in Indicates the first The key point is in the first The first of the individuals Frame (previous frame), number Frame (current frame) and the Heatmap of the next frame. Given the number of existing joints in the human body, a linear weighting scheme assigns greater weight to frames closer to the current frame, effectively capturing the temporal continuity of human motion.
[0153] Obtaining attitude residuals and pose aggregation results Then, input them into a set In each residual block (RSB), redundant and interfering joints are eliminated, and then the resulting fused motion information features are obtained. :
[0154]
[0155] 3.3) Time Management
[0156] First, the stacked motion information Divided into One block, of which Feature map for each key point, For the domain of the feature map, The height of the feature map, The width of the feature map. The height of the block. The width of the block. Preferably, the width is... The block (i.e.) This method is considered the optimal configuration. It treats each pixel as an independent block, maximizing spatial detail preservation while maintaining computational efficiency.
[0157] Each block is stacked in the same position to form 3D Embedded Feature ,in express , express Projected as (Query) (key) and (Value), then input into the multi-head self-attention layer (MSA) for calculation:
[0158]
[0159] Following this step, the output is processed by an MLP (Multilayer Perceptron) to obtain updated motion information features. :
[0160]
[0161] In the above configuration, layer normalization (LN) is applied before each MSA and MLP block. This design ensures comprehensive modeling of temporal dependencies while maintaining the finest spatial granularity for accurate pose estimation.
[0162] Preferably, in some embodiments, the multi-scale feature fusion module is used to embed appearance features. and motion information features Perform multi-scale fusion and output the final human keypoint coordinate prediction result for the current frame, specifically including:
[0163] Motion information features of the current frame By using deformable convolutions with multiple sets of different expansion rates d, offset maps corresponding to each expansion rate d are generated. With modulation mask ;
[0164] For each expansion rate d, embed the appearance features. Offset diagram and modulation mask Input to a DCNv2 deformable convolution, output the refined keypoint heatmap of the current frame corresponding to the dilation rate d. ;
[0165] Heatmap of all expansion rates d corresponding to the key refinement points Perform weighted average fusion to obtain the first Heatmap of key points for each character Thus, the first The final predicted coordinates of key points on the human body for each individual.
[0166] Preferably, the expansion rate is used. The dilated convolution generates five sets of offset maps. and modulation mask :
[0167]
[0168] Adaptive sampling and pose refinement are performed using DCNv2 deformable convolution, outputting a refined pose heatmap for the current frame. :
[0169]
[0170] Finally, the five sets of refinement results corresponding to different expansion rates were fused by weighted averaging to generate the first set. The individual's final heatmap :
[0171]
[0172] Preferably, in some embodiments, step 3, which involves end-to-end training of the human keypoint coordinate prediction model based on the human pose estimation dataset to generate a trained human keypoint coordinate prediction model, specifically includes:
[0173] To supervise the model training process, the mean squared error (MSE) loss function is used, which measures the difference between the predicted heatmap and the actual heatmap at each joint. The loss is calculated over all joints as follows:
[0174]
[0175] In the formula, It refers to the number of key points in the human body. For the first Visibility indicators of key points and They represent the first The predicted heatmap and the actual heatmap for each keypoint; this formula ensures that the model focuses only on accurately estimating visible joints during training, ignoring occluded or unlabeled joints. Using per-joint MSE provides a direct and effective way to guide the network to minimize localization error.
[0176] Based on the loss function Using real key point heatmaps as supervision signals, the human body key point coordinate prediction model is trained end-to-end to generate the trained human body key point coordinate prediction model.
[0177] Furthermore, in one embodiment, the method further includes performing temporal smoothing on the keypoint sequence output from step 4 after step 4 and before step 5.
[0178] Here, although the framework proposed in this invention can effectively integrate appearance and motion cues to achieve robust pose estimation, the predicted keypoints may still exhibit temporal jitter in complex motion scenes or under occlusion. To further improve the temporal stability and visual consistency of the estimated pose, a lightweight post-processing strategy inspired by SmoothNet is adopted.
[0179] SmoothNet aims to model the long-term temporal dependencies of individual joints, thereby generating smooth and consistent motion trajectories. It is not integrated into the end-to-end training process, but rather serves as a plug-and-play module, independently applied during the inference phase to the keypoint sequences generated by the main model of this invention.
[0180] Given a predicted sequence of joint coordinates ,in Indicates the number of frames. Indicates the number of joints. Representing the spatial dimension, SmoothNet is applied temporally to each joint trajectory. This module consists of multiple residual 1D convolutional blocks with instance normalization and GELU activation, enabling it to capture temporal patterns while preserving local joint motion.
[0181] In terms of form, for each joint SmoothNet input sequence And output a smooth version ,in:
[0182]
[0183] Here, Control the time window size. Here, an empirical window size of 13 is used, consistent with previous work, and all joints are handled independently to maximize modularity and scalability.
[0184] In one embodiment, a human pose estimation system based on the temporal consistency of motion and appearance is provided, the system comprising sequentially executed:
[0185] The first module is used to: construct a human pose estimation dataset, which includes at least one input video sequence and a heatmap of real key points corresponding to each frame of the input video sequence;
[0186] The second module is used to implement: constructing a human key point coordinate prediction model, which is a neural network model that collaboratively utilizes the temporal consistency of human appearance and the temporal continuity of human movement.
[0187] The third module is used to: perform end-to-end training on the human keypoint coordinate prediction model based on the human pose estimation dataset, and generate the trained human keypoint coordinate prediction model.
[0188] The fourth module is used to: input the video sequence to be estimated into the trained human keypoint coordinate prediction model, and output the human keypoint coordinate detection results;
[0189] The fifth module is used to estimate human posture based on the detection results of human key point coordinates.
[0190] Furthermore, the human pose estimation system also includes a module located between the fourth and fifth modules:
[0191] The sixth module is used to perform time smoothing on the key point sequence output by the fourth module.
[0192] Specific limitations regarding the human pose estimation system based on the temporal consistency of motion and appearance can be found in the limitations of the human pose estimation method based on the temporal consistency of motion and appearance mentioned above, and will not be repeated here. Each module in the aforementioned human pose estimation system based on the temporal consistency of motion and appearance can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.
[0193] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:
[0194] Step 1: Construct a human pose estimation dataset, which includes at least one input video sequence and a heatmap of real key points corresponding to each frame of the input video sequence.
[0195] Step 2: Construct a human body key point coordinate prediction model. This model is a neural network model that collaboratively utilizes the temporal consistency of human appearance and the temporal continuity of human movement.
[0196] Step 3: Based on the human pose estimation dataset, perform end-to-end training on the human keypoint coordinate prediction model to generate the trained human keypoint coordinate prediction model.
[0197] Step 4: For the video sequence to be estimated, input it into the trained human keypoint coordinate prediction model and output the human keypoint coordinate detection results.
[0198] Step 5: Perform time smoothing on the key point sequence output in Step 4;
[0199] Step 6: Estimate human pose based on the human key point coordinate detection results.
[0200] For specific constraints on each step, please refer to the constraints on human pose estimation methods based on the temporal consistency of motion and appearance mentioned above, which will not be repeated here.
[0201] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:
[0202] Step 1: Construct a human pose estimation dataset, which includes at least one input video sequence and a heatmap of real key points corresponding to each frame of the input video sequence.
[0203] Step 2: Construct a human body key point coordinate prediction model. This model is a neural network model that collaboratively utilizes the temporal consistency of human appearance and the temporal continuity of human movement.
[0204] Step 3: Based on the human pose estimation dataset, perform end-to-end training on the human keypoint coordinate prediction model to generate the trained human keypoint coordinate prediction model.
[0205] Step 4: For the video sequence to be estimated, input it into the trained human keypoint coordinate prediction model and output the human keypoint coordinate detection results.
[0206] Step 5: Perform time smoothing on the key point sequence output in Step 4;
[0207] Step 6: Estimate human pose based on the human key point coordinate detection results.
[0208] For specific constraints on each step, please refer to the constraints on human pose estimation methods based on the temporal consistency of motion and appearance mentioned above, which will not be repeated here.
[0209] As a specific example, the present invention will be further verified and illustrated.
[0210] Table 1 compares the results of the proposed method with other methods on the PoseTrack2017 validation set. The evaluation metric mAP score, representing the mean precision, comprehensively evaluates the accuracy and recall of keypoint detection. It is the most critical performance indicator in multi-person pose estimation, and a higher value indicates better performance.
[0211] Table 1 shows the results of the algorithm of this invention on the PoseTrack2017 validation set.
[0212]
[0213] As can be seen, this invention achieved the highest ranking in mAP score, which fully demonstrates the accuracy of this method in locating key points of the human body in complex scenarios.
[0214] Furthermore, Figure 5This paper compares the pose estimation performance of the algorithm proposed in this invention with that of the FAMI-Pose algorithm. It can be seen that the method of this invention exhibits significant detection performance in challenging scenarios such as rapid movement and occlusion.
[0215] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.
Claims
1. A human pose estimation method based on motion and appearance temporal consistency, characterized in that, The method comprises: Step 1, constructing a human pose estimation data set, the data set comprising at least one input video sequence, and a real key point heat map corresponding to each frame image of the input video sequence; Step 2, constructing a human key point coordinate prediction model, the model being a neural network model utilizing human appearance temporal consistency and human motion temporal continuity cooperatively; Step 3, based on the human pose estimation data set, performing end-to-end training on the human key point coordinate prediction model to generate a trained human key point coordinate prediction model; Step 4, inputting a video sequence to be estimated into the trained human key point coordinate prediction model to output a human key point coordinate detection result; Step 5, estimating a human pose based on the human key point coordinate detection result.
2. The human pose estimation method based on motion and appearance temporal consistency according to claim 1, wherein, The human key point coordinate prediction model in step 2 specifically comprises: A heat map sequence extraction module for randomly extracting initial key point heat maps of three continuous frames of images in a video sequence to form a heat map sequence; The appearance knowledge transmission module is configured to perform joint-level knowledge matching and propagation on an initial key point heat map of the current frame based on front and back frame data of the current frame, to generate appearance embedding features ; The motion information extraction module is configured to extract motion information features of the current frame based on the front and rear frame data of the current frame ; Multi-scale feature fusion module: for appearance embedding features and motion information features Multi-scale fusion is performed, and the final human key point coordinate prediction result of the current frame is output.
3. The human pose estimation method based on motion and appearance temporal consistency according to claim 2, characterized in that, The heat map sequence extraction module is configured to randomly extract initial key point heat maps of three continuous frames of images in a video sequence to form a heat map sequence, and specifically comprises: Randomly extract three consecutive images in a video sequence and Crop and align the same person object i in them to obtain a video clip ; Based on the video segment , by using a pre-trained pose estimation backbone network HRNet-w48, the initial key point heat maps of the person object i in three frames of images and are extracted respectively to form a heat map sequence.
4. The human pose estimation method based on motion and appearance temporal consistency according to claim 3, characterized in that, The appearance knowledge transmission module is configured to perform joint-level knowledge matching and propagation on an initial key point heat map of the current frame based on front and back frame data of the current frame, and generate appearance embedding features , and specifically comprises: Lightweight pose kernel distiller From the current frame of adjacent frames and distill pose-aware features and encode them into pose kernels and respectively; After optimizing the pose kernel by low-rank decomposition, the sliding window strategy is used to perform joint-level knowledge matching and propagation on the initial keypoint heat map of the current frame to generate appearance embedding features .
5. The human pose estimation method based on motion and appearance temporal consistency according to claim 4, characterized in that, The posture core and are respectively expressed as: ； ； The optimization of the pose kernel by low-rank decomposition specifically comprises: decompose the full-size pose kernel into spatial kernel bases and channel-wise coefficient matrices The decomposition formula is: ； The pose kernel is optimized by an optimization target constructed as follows: ； wherein, is the size of the convolution kernel, is the number of channels, is the number of human key points, is the Frobenius norm, is is the norm, denotes convolution, denotes channel-wise convolution, is a coefficient for controlling the trade-off between reconstruction accuracy and rank sparsity; The joint-level knowledge matching and propagation is performed on the initial key point heat map of the current frame through a sliding window strategy to generate appearance embedding features , specifically including: For each keypoint, the closest spatial region to each keypoint, i.e. the joint, is retrieved from the adjacent frame of the current frame by Thus, a set of joint-level matching features is obtained. ； ； In the formula, denotes convolution, denotes appearance information of the i-th key point of the personal object in the current frame and obtained by matching with adjacent frames respectively; , denote pose kernels of the previous frame and the next frame respectively. The joint-level matching features corresponding to all the key points are spliced and aggregated to form the appearance embedding features : ； In the formula, denotes an aggregate operation that traverses all joints and concatenates them; denotes a concatenation operation.
6. The human pose estimation method based on motion and appearance temporal consistency according to claim 5, characterized in that, The motion information extraction module is configured to extract motion information features of the current frame based on the front and rear frame data of the current frame , and specifically comprises: The residual features of the heat map sequence extracted by the heat map sequence extraction module are fused to obtain fused residual features ; Grouping the thermogram sequence by joints to obtain joint timing fusion features ; The fused residual features joint temporal fusion features After residual block processing, aggregation is performed, and a spatio-temporal attention mechanism is used for modeling to output motion information features .
7. The human pose estimation method based on motion and appearance temporal consistency according to claim 6, characterized in that, The residual features are fused by using a linear distance weighting strategy, and the fused residual features are represented as: ； Wherein, ； ； In the formula, denotes the residual of and denotes the residual of and denotes the residual of and is a frame index, respectively denoting the first frame, the first frame and the first frame; The heat map sequence is grouped by joints to obtain joint time sequence fusion features , and specifically comprises: Linearly interpolating based on time distance to obtain joint timing fusion features : ； In the formula, respectively represent the first key points of the first person in the first frame, the second frame, and the third frame key point heat map; The fused residual feature Joint temporal fusion feature After residual block processing, aggregation is performed, and a spatio-temporal attention mechanism is used for modeling to output motion information features Specifically, the method comprises the following steps: (1) the fused residual feature is fused with the joint temporal fusion feature to obtain a motion information feature : ； In the formula, is a set of residual blocks; (2) stack the motion information into blocks, wherein is a feature map for each keypoint, is a domain of the feature map, is a height of the feature map, is a width of the feature map, is a height of the block, is a width of the block; Each block is stacked in the same position to form The embedded features wherein denotes , denotes ; (3) will be The projection is a query, key and value, and then input multi-head self-attention layer MSA for calculation: ； wherein denotes the layer normalization, is the output of the multi-head self-attention layer MSA; (4) Obtain updated motion information features by a multi-layer perceptron : 。 8. The human pose estimation method based on motion and appearance temporal consistency according to claim 2, characterized in that, The multi-scale feature fusion module is configured to perform multi-scale fusion on the appearance embedding features and motion information features to output a final human key point coordinate prediction result of the current frame, and specifically includes: Motion information features for the current frame , respectively generate offset maps corresponding to each inflation rate d by using multiple groups of deformable convolution with different inflation rates d and modulation mask ; For each dilation rate d, the appearance embedding features , offset map , and modulation mask are input to a DCNv2 deformable convolution, outputting a refined keypoint heatmap for the current frame corresponding to the dilation rate d ; Refine the key point heat map of all the expansion rates d Weighted average fusion is performed to obtain the first The final key point heat map of the individual object , thereby obtaining the first The final human key point coordinate prediction result of the individual object.
9. The human pose estimation method based on motion and appearance temporal consistency according to claim 1, characterized in that, Step 3, based on the human pose estimation data set, performing end-to-end training on the human key point coordinate prediction model to generate a trained human key point coordinate prediction model, specifically comprises: Constructing loss function : ； In the formula, is the number of human body key points, is the visibility flag of the th key point, and respectively represent the predicted heat map and the real heat map of the th key point. based on the loss function An end-to-end training is performed on the human key point coordinate prediction model with the real key point heat map as a supervision signal to generate the trained human key point coordinate prediction model.
10. The human pose estimation method based on motion and appearance temporal consistency according to claim 1, characterized in that, The method comprises the following steps performed after step 4 and before step 5: Time smoothing is performed on the key point sequence output by step 4.