A lightweight algorithm for estimating 3D human pose from video

Abstract

The application discloses to the technical field of deep learning and artificial intelligence, and particularly relates to a lightweight algorithm for estimating 3D human posture from a video, which comprises the following specific steps: S1, a first stage: inputting an RGB image, extracting 2D key point features by using a deep learning model, predicting 2D coordinates of human key points, and representing the 2D coordinates in the form of a heat map to accurately identify the positions of the key points in a 2D image; S2, a second stage: based on the 2D key points, using a LitePoseFormer algorithm to perform 3D posture reconstruction through geometric constraints and kinematic constraints, and combining camera parameters or depth information to optimize the estimation of three-dimensional coordinates. The application can not only reduce memory consumption and calculation cost, but also improve the pertinence of each Transformer, avoid information loss, and improve the accuracy of central frame posture estimation by using the LitePoseFormer algorithm to perform 3D posture reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and artificial intelligence, specifically to a lightweight algorithm for estimating 3D human pose from video. Background Technology

[0002] In 3D human pose estimation algorithms, most researchers use graph convolutional neural networks (GCNNs) to improve accuracy in complex scenes. Graph convolution can model local and global dependencies between joints, flexibly capture complex spatial-temporal patterns in skeleton sequences, and improve the algorithm's adaptability to complex actions and occluded scenes. Currently, the skeleton sequence is usually treated as a spatiotemporal graph and input into the graph convolutional network to model the local and global dependencies between joints. This strategy can flexibly capture complex spatiotemporal patterns in the skeleton sequence and enhance the adaptability of pose estimation algorithms to complex actions and occluded scenes. However, graph convolution methods heavily rely on the graph structure and cannot dynamically adjust the graph structure to adapt to different action patterns. At the same time, the receptive field of graph convolutional filters is limited, making it difficult to effectively integrate information from distant nodes. Transformers, on the other hand, can capture long-range temporal dependencies in skeleton sequences both spatially and temporally. PoseFormer is a typical algorithm that was the first to use Transformers to replace graph convolutions for human pose estimation.

[0003] While PoseFormer achieves high performance, it incurs significant memory and computational overhead as frame length increases. Specifically, existing two-stage 3D human pose estimation algorithms typically first detect 2D human keypoints in the first stage, and then map these 2D keypoints to 3D space in the second stage to estimate the 3D pose. The process involves extracting the 2D coordinates of keypoints from the input image using a 2D pose detector (such as HRNet or Cascaded Pyramid Network), and then using a 3D pose estimation model combined with geometric constraints, depth information, or a pre-trained model to convert the 2D coordinates into 3D joint positions, optimizing spatial consistency and motion smoothness. PoseFormer, as a Transformer-based model, plays a crucial role in the second stage. It utilizes a self-attention mechanism to capture the spatiotemporal dependencies of 2D keypoints, models long-term contextual information between joints through spatial and temporal Transformer encoders, and combines time-series data (such as video frames) to improve the accuracy and robustness of 3D pose estimation, especially performing exceptionally well when handling occlusion or complex movements, significantly outperforming traditional convolutional or recurrent network methods. However, setting the spatial dimension to 32 and the temporal dimension to 544 for PoseFormerV1 results in significant memory consumption and computational cost. Furthermore, simply stacking spatial and temporal Transformers fails to focus specific information, making each Transformer less targeted. Additionally, its use of average pooling to extract features from the center frame leads to information loss, thus affecting the accuracy of center frame pose estimation. Therefore, a lightweight algorithm for estimating 3D human pose from video is invented. Summary of the Invention

[0004] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0005] A lightweight algorithm for estimating 3D human pose from video includes the following specific steps:

[0006] S1, First stage: Input RGB image, use deep learning model to extract 2D key point features and predict the 2D coordinates of human key points, and represent them in the form of heat map to accurately identify the position of key points in 2D image;

[0007] S2, Second Stage: Based on 2D key points, the LitePoseFormer algorithm is used to reconstruct 3D pose through geometric and kinematic constraints, and the estimation of 3D coordinates is optimized by combining camera parameters or depth information.

[0008] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the steps for extracting 2D keypoint features in S1 are as follows:

[0009] S11: Starting from the input image or video frame, using the HRNet-W48 model, detect the 2D pixel coordinates of 17 keypoints using weights pre-trained on the COCO dataset;

[0010] S12: Outputs a heatmap with a resolution of 384×288. Each heatmap represents the key point location using a Gaussian distribution, and the confidence threshold is set to 0.5 to filter out low-quality detection results.

[0011] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the heatmap generation steps in S1 are as follows:

[0012] For a key point, its true coordinates are (x... c ,y c The value at any pixel position (x, y) on the heatmap is calculated using a two-dimensional Gaussian distribution, as shown in the formula:

[0013]

[0014] Where H(x,y) represents the probability value of location (x,y) on the heatmap, ranging from [0,1], with a larger value indicating a higher likelihood of it being a key point; (x c ,y c () represents the actual pixel coordinates of the keypoint; σ represents the standard deviation of the Gaussian distribution, controlling the spread of the distribution, and can be empirically set based on the heatmap resolution and keypoint size; (xx c ) 2 +(yy c ) 2 It represents the squared Euclidean distance from pixel (x,y) to the center of the keypoint.

[0015] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video as described in this invention, in step S1, when processing multi-person scenes, YOLOv5 is first used to detect human bounding boxes, and then HRNet is run within the boxes to extract key points. The key points are then assigned to the corresponding individuals using the Hungarian algorithm based on Euclidean distance.

[0016] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the processing flow of the Hungarian algorithm is as follows:

[0017] Step 1, Input Preparation: Assume there are N human bodies. First, use a YOLOv5 detector to provide bounding boxes, with the center coordinates of each bounding box being C.i =(x i ,y i ) represents the reference position of the i-th person; then, HRNet is used to detect the 2D coordinate set P of M key points of a certain type. j =(p j,x ,p j,y ), where j = 1, 2, ..., M; then assign the M key points to N individuals, usually M ≥ N;

[0018] Step 2, Construct the cost matrix: Create an M×N cost matrix C. If M>N or M<N, matrix C will be padded to a square matrix by adding virtual nodes. The element C[j,i] represents the cost of connecting the j-th keypoint to the i-th person, and the cost is based on the Euclidean distance, which is used to calculate the keypoint coordinates P. j Center C of the human body bounding box i The distance between them, C[j,i], is defined as:

[0019]

[0020] Step 3, applying the Hungarian algorithm: First, use the Hungarian algorithm to find the optimal match on the cost matrix, minimizing the total cost of allocation, i.e., the sum of Euclidean distances. The Hungarian algorithm is then used to iteratively adjust the matrix, ultimately outputting a matching pair between each keypoint and the human body.

[0021] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the LitePoseFormer algorithm in S2 includes a spatial Transformer with a skeletal attention mechanism, a temporal Transformer with a skeletal attention mechanism, and a keypoint Transformer; the spatial Transformer is used to model the spatial positional relationship of human skeletons in each frame; the temporal Transformer is used to capture long-range dependencies between frames; the keypoint Transformer is used to decouple temporal features and spatial information, improve the accuracy and robustness of 3D pose estimation, and at the same time, a regression head is applied to the optimized features generated by the keypoint Transformer to generate the accurate 3D coordinates of the center frame.

[0022] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the specific steps of the LitePoseFormer algorithm are as follows:

[0023] Step 1: Use a matrix M∈R 2×D pass Each joint Mapped to And Z∈R F×J×D Defined as The splicing process is called pose embedding;

[0024] Step 2: Input the obtained output Z into the spatial Transformer to encode the spatial features of the joints. After obtaining the spatial features, they are embedded into the temporal dimension D through a temporal embedding operation. T To obtain the input of the time Transformer When the input to the time Transformer is obtained Then, the temporal dependencies Z between joints will be obtained through temporal Transformer encoding. T The time embedding operation is represented as:

[0025]

[0026] Step 3: Obtain the spatial features Z through spatial Transformer and temporal Transformer modeling respectively. S With time feature Z T And then model the spatiotemporal constraint relationship P through the skeletal attention mechanism. S and P T ;

[0027] Step 4: The key point Transformer analyzes the spatiotemporal constraints between joints to accurately reconstruct the 3D human pose.

[0028] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the specific steps of the spatial Transformer are as follows:

[0029] Step 1: Introduce distance and location embedding;

[0030] Step 2: Group the joints according to their distance from the joint to the pelvis as follows:

[0031] G1 = [Spine, Left Hip, Right Hip];

[0032] G2 = [throat, left knee, right knee];

[0033] G3 = [neck, left shoulder, right shoulder, left foot, right foot];

[0034] G4 = [Head, Left Elbow, Right Elbow];

[0035] G5=[left wrist, right wrist];

[0036] Gd represents the jump distance d from a certain joint to the pelvis;

[0037] Joints belonging to the same group are encoded with the same distance embedding. To embed and splice as

[0038] Step 3: Update Z to ZD using the following formula:

[0039]

[0040] Where φ(j) represents the jump distance between the j-th joint and the pelvis;

[0041] Step 4: The ZD is then input into three Transformer encoders to model the global spatial features.

[0042] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the specific steps of the temporal Transformer are as follows:

[0043] Step 1: First, introduce time-based position embedding to encode the temporal relationship between different time frames;

[0044] Step 2: To reduce redundant information and computational costs caused by high similarity between frames, a frame attention mechanism is proposed to extract key frame information and solve the redundancy problem. The frame attention mechanism includes two stages: the first stage extracts the initial temporal relationship through frame pooling, and the second stage eliminates redundancy through frame upsampling and feature refinement.

[0045] As a preferred embodiment of the lightweight algorithm for estimating 3D human pose from video according to the present invention, the keypoint Transformer includes a multi-head self-attention mechanism for decoupling keypoint skeletal constraints, a multi-head cross-attention mechanism for decoupling temporal constraints, and a deep separable feedforward convolutional neural network for strong feature representation.

[0046] The multi-head self-attention mechanism uses an initial all-zero tensor representing the keypoint labeling. Using the motion relationships between joints as input, the focus is on capturing the spatial structure P of joints within a frame. S And decouple complex skeleton constraints;

[0047] The multi-head cross-attention mechanism uses keypoint marker P. K Implicit temporal correlation P T As input, the modeling of dynamic changes and temporal dependencies between frames is used, while spatiotemporal information is integrated to improve the understanding of joint motion patterns.

[0048] The depthwise separable feedforward convolutional neural network utilizes depthwise separable convolution to efficiently extract local and global features, reducing computational costs, while enhancing the nonlinear expressive power of key point features, ultimately contributing to the final output of the key point Transformer.

[0049] Compared with existing technologies:

[0050] This invention utilizes the LitePoseFormer algorithm for 3D pose reconstruction, which not only reduces memory consumption and computational costs but also improves the specificity of each Transformer, avoids information loss, and enhances the accuracy of center frame pose estimation. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the attention mechanism at the joint site in this invention;

[0052] Figure 2 This is a schematic diagram of the frame attention mechanism of the present invention;

[0053] Figure 3 This is a schematic diagram of the skeleton attention mechanism of the present invention;

[0054] Figure 4 This is a schematic diagram of the specific network architecture of the Transformer, a key component of this invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0056] This invention provides a lightweight algorithm for estimating 3D human pose from video. Please refer to [link / reference]. Figure 1-4 The specific steps are as follows:

[0057] S1, First stage: Input RGB image, use deep learning model to extract 2D key point features and predict the 2D coordinates of human key points, and represent them in the form of heat map to accurately identify the position of key points in 2D image;

[0058] S2, Second Stage: Based on 2D key points, the LitePoseFormer algorithm is used to reconstruct 3D pose through geometric and kinematic constraints, and the estimation of 3D coordinates is optimized by combining camera parameters or depth information.

[0059] The process also includes post-processing steps, such as pose smoothing or temporal consistency adjustment, to improve the robustness and accuracy of the estimation, and finally outputs a complete 3D human pose representation of the center frame, which is usually a set of three-dimensional coordinates of key points.

[0060] In 3D human pose estimation, data preprocessing is a crucial step, especially the 2D joint extraction part, which provides the basic input for subsequent 3D pose reconstruction. This typically involves the process of accurately detecting key points on the human body from images or videos. The steps for 2D key point feature extraction in S1 are as follows:

[0061] S11: Starting from the input image or video frame, using the HRNet-W48 model, detect the 2D pixel coordinates of 17 keypoints (including head, shoulder, elbow, wrist, hip, knee, ankle, etc.) by weights pre-trained on the COCO dataset;

[0062] S12: Outputs a heatmap with a resolution of 384×288. Each heatmap represents the key point location using a Gaussian distribution, and the confidence threshold is set to 0.5 to filter out low-quality detection results.

[0063] The steps for generating the heatmap in S1 are as follows:

[0064] The true coordinates of keypoints (e.g., labeled (x, y)) are used as the mean of a Gaussian distribution to calculate the probability value for each pixel location on the heatmap. Higher heatmap resolution results in a more refined representation of the Gaussian distribution, but also increases computational cost. The standard deviation σ controls the spread of the Gaussian distribution, determining the size of the probability region; a typical value is between 2 and 10 pixels, and needs to be adjusted based on the heatmap resolution and keypoint size. For a keypoint, its true coordinates are (x, y)... c ,y c The value at any pixel position (x, y) on the heatmap is calculated using a two-dimensional Gaussian distribution, as shown in the formula:

[0065]

[0066] Where H(x,y) represents the probability value of location (x,y) on the heatmap, ranging from [0,1], with a larger value indicating a higher likelihood of it being a key point; (x c ,y c () represents the actual pixel coordinates of the keypoint; σ represents the standard deviation of the Gaussian distribution, controlling the spread of the distribution, and can be empirically set based on the heatmap resolution and keypoint size; (xx c ) 2 +(yy c ) 2 It represents the squared Euclidean distance from pixel (x,y) to the center of the keypoint.

[0067] When processing multi-person scenes, S1 first uses YOLOv5 to detect human bounding boxes, then runs HRNet to extract key points within the boxes, and uses the Hungarian algorithm based on Euclidean distance to assign the key points to the corresponding individuals.

[0068] The Hungarian algorithm, based on Euclidean distance, is used in 2D joint extraction for 3D human pose estimation to solve the keypoint assignment problem in multi-person scenes, that is, to correctly match detected keypoints to their corresponding human individuals. The Hungarian algorithm is a classic bipartite graph maximum matching algorithm. In this scenario, it optimizes the keypoint-individual assignment by minimizing a cost matrix based on Euclidean distance. In multi-person pose estimation, 2D keypoint detection (e.g., using HRNet) typically outputs several sets of keypoints, but these keypoints are not associated with specific individuals. The goal is to assign these keypoints to N detected human bodies (usually located by bounding boxes), ensuring that each keypoint matches the correct human body, and that each individual is assigned only one set of keypoints. The Hungarian algorithm constructs a cost matrix, calculates the matching cost between keypoints and human bodies based on Euclidean distance, and then finds the globally optimal assignment to minimize the total cost. The processing flow of the Hungarian algorithm is as follows:

[0069] Step 1, Input Preparation: Assume there are N human bodies. First, use a YOLOv5 detector to provide bounding boxes, with the center coordinates of each bounding box being C. i =(x i ,y i ) represents the reference position of the i-th person; then, HRNet is used to detect the 2D coordinate set P of M key points of a certain type. j =(p j,x ,p j,y ), where j = 1, 2, ..., M; then assign the M key points to N individuals, usually M ≥ N;

[0070] Step 2, Construct the cost matrix: Create an M×N cost matrix C. If M>N or M<N, matrix C will be padded to a square matrix by adding virtual nodes. The element C[j,i] represents the cost of connecting the j-th keypoint to the i-th person, and the cost is based on the Euclidean distance, which is used to calculate the keypoint coordinates P. j Center C of the human body bounding box i The distance between them, C[j,i], is defined as:

[0071]

[0072] Step 3, applying the Hungarian algorithm: First, use the Hungarian algorithm to find the optimal match on the cost matrix, minimizing the total cost of allocation, i.e., the sum of Euclidean distances. The Hungarian algorithm is then used to iteratively adjust the matrix (by reducing rows and columns, finding the minimum coverage line, etc.) and finally output the matching pair of each key point to the human body.

[0073] The core of the Hungarian algorithm is to find the minimum cost match through matrix operations. The specific steps are as follows: Initialize the cost matrix: Calculate C[j,i] based on the Euclidean distance. Then perform row reduction: Subtract the minimum value of each row to ensure each row contains at least one 0; perform column reduction: Subtract the minimum value of each column to ensure each column contains at least one 0. Next, cover all 0 elements in the matrix using the minimum covering line: If the number of covering lines equals the matrix dimension (i.e., N or M), then the current 0 corresponds to the optimal match; otherwise, adjust the matrix: Find the minimum value of the uncovered elements, subtract the uncovered elements, add the intersection element, and repeat this step. Finally, extract the matches: Extract matching pairs from the 0 elements of the final matrix, ensuring that only one 0 is selected per row and column.

[0074] The LitePoseFormer algorithm in S2 includes a spatial Transformer with a skeletal attention mechanism, a temporal Transformer with a skeletal attention mechanism, and a keypoint Transformer. The spatial Transformer is used to model the spatial positional relationships of the human skeleton in each frame. The temporal Transformer is used to capture long-range dependencies between frames. The keypoint Transformer is used to decouple temporal features and spatial information, improve the accuracy and robustness of 3D pose estimation, and apply a regression head to the optimized features generated by the keypoint Transformer to generate accurate 3D coordinates of the center frame.

[0075] The specific steps of the LitePoseFormer algorithm are as follows:

[0076] Step 1: Use a matrix M∈R 2×D pass Each joint Mapped to And Z∈R F×J×D Defined as The splicing process is called pose embedding;

[0077] Step 2: Input the obtained output Z into the spatial Transformer to encode the spatial features of the joints. After obtaining the spatial features, they are embedded into the temporal dimension D through a temporal embedding operation. T To obtain the input of the time Transformer When the input to the time Transformer is obtained Then, the temporal dependencies Z between joints will be obtained through temporal Transformer encoding. T The time embedding operation is represented as:

[0078]

[0079] Step 3: Obtain the spatial features Z through spatial Transformer and temporal Transformer modeling respectively. S With time feature Z T And then model the spatiotemporal constraint relationship P through the skeletal attention mechanism. S and P T ;

[0080] Step 4: The key point Transformer analyzes the spatiotemporal constraints between joints to accurately reconstruct the 3D human pose.

[0081] In the Spatial Transformer, the introduction of distance and part embedding enables a more accurate understanding of the relative positions between joints and the kinematic relationships between parts, thereby enhancing its ability to perceive overall posture structure and body parts. The specific steps of the Spatial Transformer are as follows:

[0082] Step 1: Introduce distance and location embedding;

[0083] Step 2: Group the joints according to their distance from the joint to the pelvis as follows:

[0084] G1 = [Spine, Left Hip, Right Hip];

[0085] G2 = [throat, left knee, right knee];

[0086] G3 = [neck, left shoulder, right shoulder, left foot, right foot];

[0087] G4 = [Head, Left Elbow, Right Elbow];

[0088] G5=[left wrist, right wrist];

[0089] Gd represents the jump distance d from a certain joint to the pelvis;

[0090] Joints belonging to the same group are encoded with the same distance embedding. To embed and splice as

[0091] Step 3: Update Z to ZD using the following formula:

[0092]

[0093] Where φ(j) represents the jump distance between the j-th joint and the pelvis; a learnable distance embedding E is used. D Instead of traditional spatial location embedding, this strategy enables the spatial Transformer to better handle spatial relationships between joints;

[0094] Step 4: The ZD is then input into three Transformer encoders to model the global spatial features.

[0095] The first encoder extracts the initial local spatial features. Furthermore, joint pooling is used to fuse joints of the same body part to obtain features representing that part.

[0096] To prevent joint pooling from disrupting the original joint position information, the human skeleton is reclassified according to the location of each joint:

[0097] P1 = [hips, spine, throat, pelvis, head];

[0098] P2 = [left hip, left knee, left foot];

[0099] P3 = [right hip, right knee, right foot];

[0100] P4 = [left shoulder, left elbow, left wrist];

[0101] P5 = [right shoulder, right elbow, right wrist];

[0102] P1, P2, P3, P4, and P5 represent the central joint, left foot, right foot, left hand, and right hand, respectively. Next, joint embeddings are added for joints in the same area. This allows for the embedding of specific location information for each joint, and the embedding results for each joint are then concatenated into a ZP (Zero-Plate).

[0103] The second Transformer encoder integrates the correlation features and interdependencies between different parts, and the output is...

[0104] The third encoder pair from Pooling features obtained through joint pooling Encode to generate global spatial features Capture the overall structure and distribution of the entire skeleton. This process can be represented as:

[0105]

[0106] Next, joint upsampling and joint attention mechanisms are implemented on the global spatial features. Specifically, firstly, joint upsampling is performed on the joint features. Global spatial information is transmitted to each body part. Then, through...

[0107]

[0108] Upsample global features Local features of each body part Fusion. This method can simultaneously utilize global structural information and local detailed features, thereby comprehensively modeling the interrelationships and dependencies between body parts. A tensor representing joint features, filled with zeros, is then initialized. And through the attention mechanism of joint areas This information is disseminated to every joint. Taking the left hand as an example, an improved cross-attention mechanism integrates location information into every joint of the left hand.

[0109] The joint attention mechanism is an improved cross-attention mechanism. Specifically, for each joint j, a distance bias is introduced when fusing part features using the scaled dot product attention mechanism. This bias is based on the distance from the central joint of each part to the pelvis. Therefore, the modified scaled dot product attention mechanism is as follows:

[0110]

[0111] Here, Φ is an MLP network that maps distance bias to an H-dimensional vector, where H is the number of heads in the multi-head cross-attention mechanism. The joint-part attention mechanism integrates coordination features and global features between different body parts into each joint, enabling each joint to interact not only with other joints within the same body part but also to establish connections with joints in other body parts. Therefore, joints closer to the central joint receive higher attention weights, while joints farther away are assigned lower weights. This adjustment allows the model to more effectively focus on key features relevant to the current joint while suppressing irrelevant or redundant information. The final output of the spatial Transformer is expressed as:

[0112]

[0113] The joint attention mechanism integrates the coordination features and global features between different body parts into each joint, enabling each joint to not only interact with other joints in its own body part, but also to establish connections with joints in other body parts.

[0114] The specific steps of the Time Transformer are as follows:

[0115] Step 1: First, introduce time-based position embedding to encode the temporal relationship between different time frames;

[0116] Step 2: To reduce redundant information and computational costs caused by high similarity between frames, a frame attention mechanism is proposed to extract key frame information and solve the redundancy problem. The frame attention mechanism includes two stages: the first stage extracts the initial temporal relationship through frame pooling, and the second stage eliminates redundancy through frame upsampling and feature refinement.

[0117] The process of frame attention mechanism:

[0118] Taking a 9-frame case as an example, the first step is to embed the temporal position of the original features of the 9 frames into E. T The algorithm is designed to recognize frame order and temporal dependencies. A Transformer encoder is then used to model the temporal relationships between nine frames. To effectively shorten the frame sequence length, frame pooling is employed, merging every three consecutive frames into one to reduce redundancy. After pooling the nine-frame sequence into three frames, a Transformer encoder is used again to model the temporal relationships between these three frames. This process is iterated until the frame length becomes one, and a Transformer encoder is used again to model the single-frame features. In the case of nine frames, this ultimately yields three inter-frame features of different lengths:

[0119] Since the frame pooling in the first stage may lead to feature redundancy, a second stage is used to optimize the features. Deconvolutional layers are used to process the single-frame features obtained in the first stage. Upsampled to 3 frames These upsampled features were then combined with the three original features encoded in the first stage. Feature refinement is performed to extract key inter-frame information and remove redundancy between the three frames. Feature refinement primarily involves element-wise subtraction between tensors. This process is iterated until the frame length is reduced to its original length, which is 9 in this example. The overall flow of the frame attention mechanism in the 9-frame case is as follows:

[0120]

[0121] Where Γ represents element-wise subtraction.

[0122] The second stage aims to gradually recover the temporal details of the frames through deconvolution, while optimizing features in each iteration to better capture the dynamics and temporal dependencies between frames. Through iterative feature optimization, the model can effectively reconstruct the temporal structure of the original sequence, avoid error propagation caused by redundant information, and enhance the ability of the temporal Transformer to model complex motion dynamics. In summary, when the frame length is F, the number of iterations for both the first and second stages is log3F.

[0123] In 3D human pose estimation tasks, the human skeleton serves as crucial prior knowledge. The relative positions and ranges of motion of joints provide spatial constraints, ensuring that the estimated coordinates conform to human kinematics; while joint motion trajectories provide temporal constraints to maintain inter-frame consistency and reduce errors. Therefore, a skeleton attention mechanism is proposed to model the skeleton prior and use it as constraint cues to improve prediction accuracy. Specifically, the spatial features Z obtained through spatial and temporal Transformer modeling are... S With time feature Z T The spatiotemporal constraints are modeled using a skeletal attention mechanism. By modeling spatial features, the skeletal attention mechanism tracks the relationships between adjacent and non-adjacent joints, capturing local and global spatial constraints P. S Similarly, by modeling temporal features, the skeleton attention mechanism tracks the temporal correlation between adjacent and non-adjacent frames, providing robust temporal constraints P for dynamic motion modeling. T The skeleton attention mechanism is also a cross-attention mechanism. It accepts the initial spatial and temporal constraints as queries, and the spatial and temporal features obtained from modeling as keys and values.

[0124] The skeleton attention mechanism captures the implicit motion relationships between adjacent and non-adjacent joints within a frame, as well as the implicit temporal correlations between adjacent and non-adjacent frames. Based on these spatiotemporal constraints, a keypoint Transformer is proposed to further extract high-dimensional coordinate information related to 3D position.

[0125] The keypoint Transformer takes initialized zero keypoint markers as input, which are designed to capture features related to the joint coordinates of the center frame and interact with the spatiotemporal correlations modeled by the skeleton attention mechanism, ultimately generating the coordinate features of the center frame.

[0126] The keypoint Transformer includes a multi-head self-attention mechanism for decoupling keypoint skeleton constraints, a multi-head cross-attention mechanism for decoupling temporal constraints, and a deep separable feedforward convolutional neural network for strong feature representation.

[0127] The multi-head self-attention mechanism uses an initial all-zero tensor representing the keypoint labeling. Using the motion relationships between joints as input, the focus is on capturing the spatial structure P of joints within a frame. S And decouple complex skeleton constraints;

[0128] The multi-head cross-attention mechanism uses keypoint marker P. K Implicit temporal correlation P T As input, the modeling of dynamic changes and temporal dependencies between frames is used, while spatiotemporal information is integrated to improve the understanding of joint motion patterns.

[0129] The depthwise separable feedforward convolutional neural network utilizes depthwise separable convolution to efficiently extract local and global features, reducing computational costs, while enhancing the nonlinear expressive power of key point features, ultimately contributing to the final output of the key point Transformer.

[0130] Specifically, to ensure the correct joint arrangement order and consider the temporal order of joint trajectories, spatial position encoding (ES) and temporal position encoding (ET) are embedded in the MHSA and MHCA stages. These components together enable the keypoint Transformer to accurately extract high-dimensional features related to 3D pose and perform fine-grained feature modeling.

[0131] Finally, at the key point, the final output of the Transformer Apply a linear transformation matrix to the above Obtain the 3D joint coordinates of the center frame

[0132] The effects of the invention

[0133] The LitePoseFormer algorithm is compared with some mainstream methods in terms of performance. The LitePoseFormer algorithm has only 17% of the parameters and 12% of the MFLOPs of PoseFormer, while its accuracy (MPJPE) decreases by only 0.85%. This is because instead of simply stacking Transformer encoders like PoseFormer, it uses multiple encoders focused on modeling key features such as joints, parts, frames, and coordinates. This allows fewer encoders to extract more refined features. Furthermore, a skeletal attention mechanism is employed to model spatiotemporal constraints, thus ensuring high accuracy.

[0134] Parameters (M) FLOPs(M) MPJPE (millimeters) MHFormer 18.92 1030 45.9 StridedFormer 4.01 128 46.9 PoseFormerV1 9.58 462 47.0 PoseFormerV2 - 77.2 48.7 LitePoseFormer 1.72 57.9 47.4

[0135] Example

[0136] LitePoseFormer is designed for real-time 3D human pose estimation tasks and is suitable for scenarios such as motion capture and virtual reality. It takes real-time video streams as input and aims to accurately predict the 3D joint coordinates of each frame. The system employs an efficient multi-stage processing flow: First, YOLOv8 is used to quickly detect human bounding boxes in each frame, combined with the SORT algorithm for cross-frame tracking to ensure target consistency; next, HRNet is used for high-precision 2D keypoint detection to generate reliable joint coordinates; then, LitePoseFormer uses powerful spatiotemporal modeling capabilities to upscale the 2D coordinates to 3D space, capturing spatial relationships between joints and inter-frame motion information; finally, camera parameters are combined to convert the 3D coordinates to the world coordinate system, outputting accurate pose data. Its lightweight design and optimization mechanisms ensure low latency and high robustness, adapting to complex scenes and edge devices, providing an efficient solution for real-time applications.

[0137] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A lightweight algorithm for estimating 3D human pose from video, characterized in that, The specific steps are as follows: S1, First stage: Input RGB image, use deep learning model to extract 2D key point features and predict the 2D coordinates of human key points, and represent them in the form of heat map to accurately identify the position of key points in 2D image; S2, Second Stage: Based on 2D key points, the LitePoseFormer algorithm is used to reconstruct 3D pose through geometric and kinematic constraints, and the estimation of 3D coordinates is optimized by combining camera parameters or depth information. The LitePoseFormer algorithm in S2 includes a spatial Transformer with a skeletal attention mechanism, a temporal Transformer with a skeletal attention mechanism, and a keypoint Transformer. The spatial Transformer is used to model the spatial positional relationships of the human skeleton in each frame. The temporal Transformer is used to capture long-range dependencies between frames. The keypoint Transformer is used to decouple temporal features and spatial information, improve the accuracy and robustness of 3D pose estimation, and apply a regression head to the optimized features generated by the keypoint Transformer to generate accurate 3D coordinates of the center frame. The specific steps of the LitePoseFormer algorithm are as follows: Step 1: Use a matrix pass Each joint Mapped to and will Defined as The splicing process is called pose embedding; Step 2: Obtain the output The input is fed into a spatial Transformer to encode the spatial features of the joints. After obtaining the spatial features, they are embedded into the temporal dimension through a temporal embedding operation. To obtain the input of the time Transformer When the input of the time Transformer is obtained Then, the temporal dependencies between joints are obtained through temporal Transformer encoding. The time embedding operation is represented as: Step 3: Spatial features obtained through spatial Transformer and temporal Transformer modeling respectively. With time characteristics And then model the spatiotemporal constraints through the skeletal attention mechanism. and ; Step 4: The key point Transformer analyzes the spatiotemporal constraints between joints to accurately reconstruct the 3D human pose.

2. The lightweight algorithm for estimating 3D human pose from video according to claim 1, characterized in that, The steps for extracting 2D key point features in S1 are as follows: S11: Starting from the input image or video frame, using the HRNet-W48 model, detect the 2D pixel coordinates of 17 keypoints using weights pre-trained on the COCO dataset; S12: Outputs a heatmap with a resolution of 384×288. Each heatmap represents the key point location using a Gaussian distribution, and the confidence threshold is set to 0.5 to filter out low-quality detection results.

3. The lightweight algorithm for estimating 3D human pose from video according to claim 1, characterized in that, The steps for generating the heatmap in S1 are as follows: For a key point, its true coordinates are Any pixel position on the heatmap The value is calculated using a two-dimensional Gaussian distribution, with the following formula: in, Indicates the location on the heat map The probability value, ranging from A larger value indicates that it is more likely to be a key point; Represents the actual pixel coordinates of key points; It represents the standard deviation of the Gaussian distribution, controls the spread of the distribution, and can be set empirically based on the heatmap resolution and key point size; Represents pixels The square of the Euclidean distance to the center of the critical point.

4. The lightweight algorithm for estimating 3D human pose from video according to claim 2, characterized in that, When processing multi-person scenes, S1 first uses YOLOv5 to detect human bounding boxes, then runs HRNet to extract key points within the boxes, and uses the Hungarian algorithm based on Euclidean distance to assign the key points to the corresponding individuals.

5. The lightweight algorithm for estimating 3D human pose from video according to claim 4, characterized in that, The processing flow of the Hungarian algorithm is as follows: Step 1, Input Preparation: Assume there are... For each individual, bounding boxes are first provided using a YOLOv5 detector, with the center coordinates of each bounding box being... Indicates the first The reference position of the individual body; then, HRNet is used to detect... A set of 2D coordinates of a certain type of key point ,in ; then will Key points were allocated to Individual body, usually ; Step 2, Constructing a cost matrix: Create a Cost matrix ,if or This will cause matrix C to be completed into a square matrix by adding virtual nodes, where the elements Indicates the first The key point to the first The cost per individual is based on Euclidean distance, which is used to calculate the coordinates of key points. Center of the human body bounding box The distance between them Defined as: Step 3, applying the Hungarian algorithm: First, use the Hungarian algorithm to find the optimal match on the cost matrix, minimizing the total cost of allocation, i.e., the sum of Euclidean distances. Then, the Hungarian algorithm iteratively adjusts the matrix to finally output the matching pair of each key point to the human body.

6. The lightweight algorithm for estimating 3D human pose from video according to claim 1, characterized in that, The specific steps of the spatial Transformer are as follows: Step 1: Introduce distance and location embedding; Step 2: Group the joints according to their distance from the joint to the pelvis as follows: G1 = [Spine, Left Hip, Right Hip]; G2 = [throat, left knee, right knee]; G3 = [neck, left shoulder, right shoulder, left foot, right foot]; G4 = [Head, Left Elbow, Right Elbow]; G5=[left wrist, right wrist]; Gd represents the jump distance d from a certain joint to the pelvis; Joints belonging to the same group are encoded with the same distance embedding. , To splice the embedded parts into ; Step 3: Update Z to ZD using the following formula: ， in This represents the jump distance between the j-th joint and the pelvis; Step 4: The ZD is then input into three Transformer encoders to model the global spatial features.

7. The lightweight algorithm for estimating 3D human pose from video according to claim 1, characterized in that, The specific steps of the Time Transformer are as follows: Step 1: First, introduce time-based position embedding to encode the temporal relationship between different time frames; Step 2: To reduce redundant information and computational costs caused by high similarity between frames, a frame attention mechanism is proposed to extract key frame information and solve the redundancy problem. The frame attention mechanism includes two stages: the first stage extracts the initial temporal relationship through frame pooling, and the second stage eliminates redundancy through frame upsampling and feature refinement.

8. The lightweight algorithm for estimating 3D human pose from video according to claim 1, characterized in that, The keypoint Transformer includes a multi-head self-attention mechanism for decoupling keypoint skeleton constraints, a multi-head cross-attention mechanism for decoupling temporal constraints, and a deep separable feedforward convolutional neural network for strong feature representation. The multi-head self-attention mechanism uses an initial all-zero tensor representing the keypoint labeling. Using the motion relationships between joints as input, the focus is on capturing the spatial structure of joints within a frame. And decouple complex skeleton constraints; The multi-head cross-attention mechanism uses key point marking. Implicit temporal correlation As input, the modeling of dynamic changes and temporal dependencies between frames is used, while spatiotemporal information is integrated to improve the understanding of joint motion patterns. The depthwise separable feedforward convolutional neural network utilizes depthwise separable convolution to efficiently extract local and global features, reducing computational costs, while enhancing the nonlinear expressive power of key point features, ultimately contributing to the final output of the key point Transformer.

Images