Lightweight human 3d pose estimation method and system based on multi-dimensional encoder

Abstract

The application discloses a lightweight human three-dimensional posture estimation method and system based on a multi-dimensional encoder. The method first inputs a human two-dimensional posture sequence, constructs a space topology modeling branch through a state space encoder, and extracts a space feature vector of the two-dimensional posture. Secondly, the human two-dimensional posture sequence is synchronously subjected to a dimensional supplementing encoder to construct a dimensional supplementing modeling branch, and a supplementing feature vector of the two-dimensional posture is extracted. Then, the space feature vector and the supplementing feature vector are subjected to feature splicing and fusion to form a global vector. Finally, the global feature vector is subjected to a regression operation modeling to a three-dimensional coordinate system to form a complete three-dimensional posture, and three-dimensional human posture modeling is completed. The application enhances the adaptability to occlusion, jitter and noise key nodes, and realizes lightweight and accuracy improvement of human three-dimensional posture estimation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence algorithms, specifically relating to a lightweight human 3D pose estimation method and system based on a multi-dimensional encoder, which is used to achieve more convenient and accurate 3D pose modeling. Background Technology

[0002] Human 3D pose estimation is a technique that uses computer vision and deep learning methods to infer the 3D pose information of the human body from images or videos. This technology is currently widely used in gait recognition, motion analysis, and telemedicine. Compared with existing solutions that rely on marker points or wearable devices, vision-based 3D pose estimation can obtain 2D pose key points from image sequences using a 2D estimator, and then model the 2D pose to 3D pose using a spatiotemporal network. It has advantages such as being non-contact, low-cost, and easy to deploy, and can continuously provide accurate and continuous joint trajectories, providing important support for the development of medical clinical applications such as motion quality assessment, abnormal posture early warning, and personalized intervention strategies.

[0003] Currently, Transformer-based models are relatively mature and widely used in human pose estimation tasks. However, the models have a large computational load, which increases the time required for pose estimation and affects model performance. Mamba-based models, based on their unique internal state space structure, have significantly improved the speed of pose estimation. However, they generally lack a sufficient understanding of the connections between human skeletons and still have certain deficiencies in the ability to compute and model various complex pose types. Summary of the Invention

[0004] To alleviate the limitations of current model modules, this invention proposes a lightweight human 3D pose estimation method and system based on a multi-dimensional encoder. This invention reduces redundant computation while ensuring detection accuracy by introducing state-space operations, tracking function layer normalization, and parallel supplementary encoders. Specifically, this invention employs a state-space layer processing method to efficiently extract spatial relationships while avoiding large-scale convolution and attention parameters; it uses tracking function layer normalization to reduce the computational load required by the model, effectively reducing model training time; and it introduces a parallel encoder for information supplementation to ensure accurate detection.

[0005] In one aspect, this invention provides a lightweight human 3D pose estimation method based on a multi-dimensional encoder, comprising the following steps: Step 1: Input the human body's two-dimensional pose sequence, construct a spatial topology modeling branch, and extract the spatial feature vector of the two-dimensional pose.

[0006] The preprocessed human 2D pose sequence is input into the spatial topology modeling branch. The spatial topology modeling branch is used to learn the structural dependencies between joints in the human 2D pose.

[0007] First, select several consecutive frames located at the very center of the input 2D pose sequence as the modeling object and perform appropriate dimensional transformation. Then, based on each transformed pose joint, perform independent linear projection on each pose joint using given equal-dimensional joint embedding weights and biases. Next, sum the given equal-dimensional joint spatial position codes and process them through a normalization module to obtain the spatial input features. The spatial input features formed by the above operations preserve the relative position information of human joints within the skeletal structure.

[0008] Simultaneously, based on this two-dimensional attitude sequence, the center of the input sequence is constructed synchronously. The observation reliability parameters for each keypoint in the frame are used to quantitatively express the impact of noise and other factors on the keypoints. The method for constructing the observation reliability parameters is as follows: For the first frame... t The first frame of the image p For each joint, its motion velocity and acceleration relative to the previous frame are calculated. Based on the stability of motion acceleration and bone length changes across multiple image frames, the observation reliability coefficient of the joint at the current moment is determined and fused with spatial input features to generate spatial input features containing observation reliability parameters, which are then input to the state spatial feature encoding module.

[0009] The state-space feature encoding module includes a state-space window segmentation unit and a state-space window pose processing layer unit. Each layer of the state-space window pose processing layer contains a state-space transformation function unit and a chain of state-space multilayer perceptron units. Its processing can be represented as follows: In the state space window segmentation unit, the size of the local window is first initialized and the window segmentation operation is performed. The spatial input features are converted into a sequence and sent into each window for processing one by one.

[0010] First, the sequences within each local window are normalized to obtain a three-dimensional normalized window feature matrix with batch, sequence, and feature dimensions. A trainable dual-branch feature projection matrix is ​​then defined for matrix multiplication, outputting two sub-feature information segments according to a preset channel division ratio. These segments are responsible for modeling the global and local detail information of the sequence within the window, respectively. Subsequently, a transpose, one-dimensional channel-wise convolution module and a SiLU activation function module are used to extract local temporal features from the two sub-feature information segments.

[0011] Subsequently, two sub-feature information points are input into the state-space equations concerning attitude and motion. One sub-feature information point is used to extract attitude and motion features and is directly input into the state-space equation; the other sub-feature information point is used to extract non-state compensation features and is not directly input into the state-space equation, but is only concatenated with the output of the state-space equation in a subsequent step. These state-space equations are respectively the state-space equation for attitude and the state-space equation for motion, and they are concatenated end-to-end. These two state-space equations are used to describe the attitude changes and motion changes between adjacent feature locations, respectively.

[0012] Subsequently, based on the attitude output, attitude state, and motion output generated by the two state-space equations, the attitude output, motion output, and attitude state are concatenated with the corresponding state observation matrix to generate state selection weights. The attitude trend state and motion trend state are then adaptively combined to obtain the state-space output feature of the first feature channel. This state-space output feature is concatenated with the second sub-feature information and then processed through state-space perceptron encoding, window restoration, and dimensional transformation methods to ultimately transform it into a two-dimensional attitude spatial feature vector containing a complete spatial topology. This feature vector includes information such as the local limb structure of the human body, the coordination relationship between the left and right limbs, and the spatial connections between key joints, providing local attitude information for subsequent three-dimensional attitude generation.

[0013] Step 2: For the input human 2D pose sequence, simultaneously construct a dimension augmentation modeling branch through a dimension augmentation encoder to extract the augmentation feature vector of the 2D pose; The preprocessed 2D human pose sequence is synchronously input into the dimension augmentation modeling branch. This branch models and extracts global motion relationships and global dynamic features from the input video sequence. Specifically, firstly, a dimensional transformation operation is performed on the 2D human pose sequence along the time dimension, and the horizontal and vertical coordinates of the human joints in the sequence are subjected to discrete cosine transform operation, and the horizontal and vertical coordinates are represented by horizontal and vertical frequency coefficients.

[0014] Furthermore, all the retained frequency coefficients are processed by a frequency order gain module. This module sets frequency gain parameters for different low-frequency orders at different joints based on their frequency orders, and multiplies the frequency gain parameters by the frequency coefficients to obtain the gain. Then, the frequency coefficients are arranged in ascending order according to their corresponding frequency order numbers, and the earlier frequency orders are truncated to obtain the spectrum sequence.

[0015] Subsequently, the spectral sequence is input into the dimension-enhancing feature encoding module. The dimension-enhancing feature encoding module includes an attention layer, a feedforward network layer, a normalization layer, and a residual connection layer, which are used to model the global correlation between different key points, different dynamic frequency bands, and different action states, and output the pose enhancement feature vector.

[0016] Step 3: Perform feature splicing and fusion on the spatial topological feature vector and the dimension-addition feature vector to form a fused feature vector; The spatial feature vector and the dimension-enhanced feature vector are input into the feature fusion module for feature dimension alignment. Since the two types of features have different dimensions, a one-dimensional convolution operation is first used to unify the dimensions of the two types of features.

[0017] The spatial topological pose features after dimensional unification and the dimensional supplementary pose features are fused by channel splicing to obtain a fused feature vector.

[0018] Step 4: Perform regression operations on the global feature vectors to model them in a 3D coordinate system, forming a complete 3D pose: The obtained fused feature vector is input into the pose regression module. This pose regression decoding module consists of a normalization layer and a linear output layer, which outputs the coordinates of the human body's 3D joints corresponding to the center frame.

[0019] For a single-frame image, the output can be denoted as: That is, the first to the second frames in this image. P The x, y, and z coordinates of each 3D joint are used as the human skeleton joints. According to the preset joint connection relationship, a skeletal connection is established between each joint and all the joints it is connected to, resulting in the corresponding 3D human skeleton model.

[0020] Step 5: Based on the above steps, construct a deep learning model with the corresponding structure, and perform model training, optimization, and actual modeling: The model structure constructed according to the above steps serves as the actual deep learning model structure. During the training phase, a 2D human pose from a standard dataset is used as the model input. This input first passes through both a spatial topology modeling branch and a dimensionality augmentation modeling branch, then sequentially through a feature fusion module and a pose regression module. Finally, the model outputs the 3D human pose coordinates of all joints within the image.

[0021] The 3D human pose coordinates of each image output by the model are compared with the standard 3D human pose coordinates of each image in the official standard dataset. Indicators such as joint position error, normalized joint position error, or joint velocity error are calculated. The expression for the error calculation serves as the loss function.

[0022] Subsequently, the gradient of each trainable parameter is calculated using the backpropagation algorithm to further update the trainable parameters within the entire model. Furthermore, during training, after each training round, the 3D human pose estimation error is simultaneously calculated on the test set. The model parameter set with the lowest error on the test set is selected as the current optimal model parameter set and replaces the optimal model parameter set determined in the previous training round.

[0023] Using the trained model, 3D human pose modeling is performed on actual videos. First, 2D pose detection is performed on the input video as data preprocessing. Then, according to the processing length set during model training, and the length of the central frames used for actual modeling within that processing length, 3D human pose information is extracted and modeled from the video using this method. For video frames located at the beginning or end of the video and with insufficient adjacent frames, padding is used to extend the length.

[0024] The model extracts human joint structure features from multiple frames via a spatial topology modeling branch, while simultaneously extracting global motion trends and dynamic features from the input sequence via a dimensionality augmentation modeling branch. A feature fusion module then obtains fused pose information, and finally, a pose regression module outputs the corresponding 3D human joint coordinates. Subsequently, based on the predicted 3D joint coordinates, a 3D coordinate system and a 3D human skeleton are plotted, generating frame-by-frame 2D-3D pose comparison images. The output includes 3D joint coordinates, frame-by-frame visualizations, and a synthesized video file. Continuous 3D human pose videos are not only concise and clear but also suitable for tasks such as motion recognition and behavior analysis.

[0025] In another aspect, the present invention also provides a lightweight human 3D pose estimation system based on a multi-dimensional encoder, comprising the following modules: The spatial feature module is used to extract spatial feature vectors of the two-dimensional human pose sequence by constructing a spatial topology modeling branch through a state space encoder.

[0026] The supplementary feature module is used to simultaneously construct a dimensional supplementary modeling branch through a dimensional supplementary encoder for human two-dimensional pose sequences, and extract supplementary feature vectors of the two-dimensional poses.

[0027] The global module is used to concatenate and fuse spatial feature vectors and supplementary feature vectors to form a global vector.

[0028] The 3D pose module is used to perform regression operations on global feature vectors to model them in a 3D coordinate system, forming a complete 3D pose.

[0029] The training optimization module is used to construct the loss function, perform training tuning, and complete 3D human pose modeling.

[0030] Invention effects: This invention constructs a spatial topology modeling branch and a dimension-supplementing modeling branch, and introduces observation reliability parameters and frequency order gain mechanisms to collaboratively and complementaryly extract the local structure and global features of the posture. This reduces redundant computation and enhances the model's adaptability to occlusion, jitter, and noisy key points. It solves the problems of long model training and evaluation time and difficulty in simultaneously considering the local structure and global dynamics of the posture, thus achieving lightweight and improved accuracy in human 3D posture estimation. Attached Figure Description

[0031] The flowchart for this evaluation method is shown in the figure: Figure 1 This is the overall flowchart of this method; Figure 2 This is the overall flowchart of the state-space encoder. Figure 3 Images used to visualize and compare with other methods on the standard Human3.6M dataset for 3D human pose estimation using the proposed model; Figure 4 Images are used to visualize and compare the proposed model with other methods on a self-built dataset for 3D human pose estimation. Detailed Implementation

[0032] The following examples, combined with... Figure 1 and Figure 2 The present invention will be further described below.

[0033] Example 1: This embodiment relates to a three-dimensional human pose estimation method based on multi-dimensional feature information, such as... Figure 1 As shown, the specific steps are as follows.

[0034] The following describes a three-dimensional human pose estimation method based on the fusion of spatial topology and time-frequency domain features according to a specific embodiment of the present invention. This embodiment uses the estimation of the three-dimensional human pose in the center frame of a video sequence as an example; that is, it inputs a sequence of multiple consecutive two-dimensional human pose frames and outputs the coordinates of the three-dimensional joint points of the human body corresponding to the center frame of the consecutive sequence. It should be understood that this embodiment is only used to illustrate the specific implementation of the present invention and is not intended to limit the scope of protection of the present invention.

[0035] The method described in this embodiment can be implemented on a computing device equipped with a graphics processor. The computing device deploys a two-dimensional human keypoint detection module, a three-dimensional human pose estimation model training module, and a three-dimensional human pose inference module. The two-dimensional human keypoint detection module is used to extract the coordinates of two-dimensional human joints in consecutive frames from the input video; the three-dimensional human pose estimation model training module is used to train model parameters based on the two-dimensional human pose sequence and corresponding three-dimensional pose annotations; the three-dimensional human pose inference module is used to perform center frame three-dimensional human pose estimation on the video to be tested after training.

[0036] Step 1: First, perform 2D human keypoint detection frame by frame on the input complete video to obtain the 2D coordinates of all human joints in each frame of the video. This is the 2D human pose sequence input to the model, represented as: ,in B For batch, F For each batch of input frames, P 2 represents the number of joints in the human body, and 2 represents the coordinate dimension (two-dimensional).

[0037] The above two-dimensional human pose sequence is input into the spatial topology modeling branch. This branch is used to learn the structural dependencies between the joints of the human body.

[0038] From the input two-dimensional human pose sequence, firstly, select the center of the sequence. K frame Then, for each human joint in each frame, weights are embedded for each joint. With bias This enables the two-dimensional coordinates of each human body joint to be displayed. Projecting as joint embedding features Subsequently, the spatial location is encoded. By adding the joint embedding features to the spatial location encoding, the spatial input features can be obtained. This spatial input feature Preserve the relative position information of human joints within the skeletal structure.

[0039] Meanwhile, based on the above input sequence Simultaneously construct the center of the input sequence The observation reliability parameters for each keypoint in a frame are used to characterize whether the keypoint may be affected by occlusion, jitter, noise, etc. First, for the first keypoint in the sequence... B Batch, No. The first frame of the image P Each keypoint is used to calculate its position relative to the previous frame. Calculate the acceleration stability of the joint by analyzing the changes in motion velocity and acceleration at the corresponding joint point in the frame. For a single joint, instantaneous motion acceleration is acquired. The acceleration offset is obtained by subtracting the instantaneous motion acceleration at the joint from the median instantaneous motion acceleration. The L2 norm of the acceleration offset is calculated, and the standardized L2 result is used as the joint acceleration stability. .

[0040] Based on the acceleration stability of the joints, determine the reliability coefficient of the joints in the current observation. ,in It is an exponential function with base e. Scaling factor: An observation reliability matrix is ​​constructed by combining all observation reliability coefficients according to the same batch, center length, and key point order. The form, and the aforementioned spatial input features The fusion process yields spatial input features containing observation reliability parameters. Then, the window size is initialized within the window segmentation unit. This initialization process can be represented as: defining the window... s Width is W The height is H Furthermore, the window edges are padded with blank features (if necessary) to ensure reversibility of subsequent operations; through window segmentation, the... Convert to window sequence In this format, the number of windows generated is ,in , .

[0041] Then, within the state-space transformation function unit, for ease of calculation, let... , , window sequence Converted into a three-dimensional normalized window feature matrix. And define a trainable dual-branch feature projection matrix. and Perform matrix multiplication: in It is a matrix that can be divided into blocks in a 1:1 ratio, with the dimensions of the two partitioned matrices being respectively... and These two matrices are used to generate two sub-feature information sequences after matrix multiplication. and Subsequently, transposition, one-dimensional channel-wise convolution, and SiLU activation function modules are used sequentially to extract local temporal features from the two sub-feature information. The two sub-feature information processed in this step are denoted as follows: and : Then, the two sub-feature information are input into the state-space equations concerning attitude and motion. Used to extract pose and motion features, its internal tokens are directly input into the state space equation; This is used to extract non-state compensation features, without directly inputting the state-space equation; it is only concatenated with the output of the state-space equation in subsequent steps. The first i The first window n Each block is denoted as Each corresponding observed coefficient (in For trainable observation coefficient matrix, Indicates the first i The first window n The observation coefficients of each block are multiplied by a matrix and then input into the state-space equation for attitude, as shown below, to obtain the attitude output. : Wherein, discrete state matrix Denotes the training parameters, where The step size parameter for attitude; input mapping parameters and output mapping parameters and define The pose state defined for this Token is used to describe the pose changes between adjacent feature locations. The definition is as follows: and It is a trainable weight and bias matrix with the same dimension as the input features, used to control the step size parameter, input mapping parameter, and output mapping parameter, such as... Figure 2 As shown.

[0042] After obtaining the pose state, the output at the current feature position is predicted based on the pose trend state of the previous feature position. And calculate the difference between the predicted and actual results, for The first i The first window n Each block yields the motion trend state. The observation reliability coefficient corresponding to this token After matrix multiplication, the input is first fed into the state-space equations for motion shown below: Wherein, discrete state matrix Denotes the training parameters, where The step size parameter of the motion; input mapping parameters and output mapping parameters and define The motion state defined for this Token is used to describe the motion changes between adjacent feature locations. The definition includes... and It is a weight and bias matrix with the same dimension as the input features, used to control the step size parameter, input mapping parameter, and output mapping parameter.

[0043] Then, for each token, output according to the generated pose. With motion output ,according to The adjustment method is handled as follows, among which , where defined The matrix is ​​a concatenation of the attitude output, motion output, attitude state, and observation state matrices. It is a weight and bias matrix with the same dimension as the input features; by combining all the tokens in this way, the first sub-feature information is generated. The final output .

[0044] Then, this output is combined with the second sub-feature information. The data are concatenated and then processed sequentially using state-space perceptron encoding, window restoration, and dimensional transformation methods to ultimately transform the data into a two-dimensional pose spatial feature vector containing a complete spatial topology. ,in D = P × C .

[0045] This feature vector contains information such as the local limb structure of the human body, the coordination relationship between the left and right limbs, and the spatial connection between key joints, providing local structural information for subsequent generation of 3D pose.

[0046] Step 2: Sequence of two-dimensional human poses Synchronous input dimension augmentation modeling branch. The operations in this branch are as follows: First, regarding this pose sequence, the first... b The first batch, the... t A video frame (t∈1, 2, ..., F) in the time dimension, the th... p Two-dimensional coordinates of each key point Perform a discrete cosine transform on the sequence to obtain the transformed sequence. kFrequency coefficients on the horizontal and vertical axes at each frequency order At the same time, the high-frequency coefficients after the transformation are discarded, leaving only the high-frequency coefficients. M A low-frequency coefficient. This process can be represented as follows: in, These are the weighting coefficients for the discrete cosine transform.

[0047] Then, all frequency coefficients are numbered according to their corresponding frequency orders. k The components are arranged from smallest to largest through a frequency order gain module. This module sets the gain level based on the frequency order of the key points. p The first joint point in the k Frequency gain parameters at low frequency orders ,in This is the gain coefficient. It involves dynamically adjusting the parameters. The frequency gain parameter is compared with a two-dimensional vector. Multiplying by the gain, we have: Then, through dimensional transformation, Numbered according to the frequency order corresponding to each frequency coefficient k Arrange from smallest to largest, and truncate the first few bytes. M Each frequency order is transformed into a dimensionless form. Obtain the spectrum token sequence (where D = P × C ); and set the corresponding spectral weights. and bias Perform a linear mapping and add location information. Finally, the token sequence is obtained. : Subsequently, the token sequence Further input is made into a dimension-enhancing feature encoding module. This module includes an attention layer, a feedforward network layer, a normalization layer, and a residual connection layer, used to model the correlation between different key points and coefficients in different frequency domains.

[0048] Through the above steps, the final result is a dimension-enhanced feature vector containing complete global information. This feature vector mainly contains global change information of the overall attitude, and can suppress high-frequency noise in the attitude information.

[0049] Step 3: Convert the spatial feature vector With dimension-enhanced feature vectors Input feature fusion module. Since the two types of features have different dimensions, a one-dimensional convolution operation is first used to unify the dimensions of the two types of features. The resulting spatial topological pose features are then processed. and dimension-enhanced pose features They are represented as follows: Spatial topological pose features after unifying the above dimensions and dimension-enhanced pose features The fusion operation is performed by concatenating channels to obtain the fused feature vector. This process can be represented as , Represents the weight coefficient matrix. This represents the bias matrix.

[0050] Step 4: Obtain the fused feature vector Input pose regression module. This pose regression decoding module consists of a normalization layer and a linear output layer, outputting the coordinates of the human body's 3D joint points corresponding to the center frame. ,in T' This indicates the number of output frames, and 3 represents the three-dimensional coordinate dimension. The process can be represented as: The specific form of the three-dimensional joint coordinates is as follows: Using the coordinates of these three-dimensional joints of the human body as the joints of the human skeleton, and according to the preset joint connection relationships (such as pelvis-spine, pelvis-left thigh, pelvis-right thigh, cervical spine-spine, cervical spine-left upper arm, cervical spine-right upper arm, etc.), a skeletal connection is established between each joint and all the joints it is connected to, thus obtaining the corresponding three-dimensional human skeleton model.

[0051] Step 5: Build and train the model, following the steps outlined above as the model structure. During the training phase, the 2D human pose data from the Human 3.6M dataset is used as the model input. This input will simultaneously pass through the spatial topology modeling branch and the dimensionality augmentation modeling branch, followed by the feature fusion module and the pose regression module. The final output of the model is the 3D human pose coordinates.

[0052] The 3D human pose coordinates output by the model are compared with the officially labeled 3D human pose coordinates of the Human 3.6M dataset to calculate the joint position error. The expression for this error is used as the loss function. In this embodiment, the average joint position error of the center frame is used. The expression for the loss function is as follows: Subsequently, the gradient of each trainable parameter is calculated using the backpropagation algorithm, and the trainable parameters throughout the model are updated using the Adam optimizer. During training, after each round, the 3D human pose estimation error is calculated synchronously on the test set, and the model parameter set with the lowest error on the test set is selected as the optimal model parameter set.

[0053] Subsequently, the trained model is used to perform 3D human pose modeling on the actual video. The video input can be a video file of any format (such as .mp4, .mov, etc.). First, 2D pose detection is performed on the input video as data preprocessing. Then, according to the processing length set during model training, and the length of the central multi-frames used for actual modeling within this processing length, the 3D human pose information is extracted and modeled from the video. The model extracts the human joint structure features of the central multi-frames through a spatial topology modeling branch, and simultaneously extracts the global motion trend and dynamic features of the input sequence through a dimension augmentation modeling branch. Then, the feature fusion module obtains the fused pose information, and finally, the pose regression module outputs the coordinates of the corresponding 3D human joint points.

[0054] For a complete video, a sliding method is used in actual modeling to construct a sequence of center frames frame by frame. This sequence of center frames can model the 3D pose of the human body in the most central frame. By sliding, the 3D pose of the human body in each frame of the video is gradually obtained. The final output contains the coordinates of the human body's 3D skeleton in each frame and the overall 3D human pose and movement corresponding to the original video. Continuous 3D human pose and movement videos are not only concise and clear, but also more suitable for tasks such as motion recognition and behavior analysis.

[0055] Example 2: This embodiment provides a lightweight human 3D pose estimation system based on a multi-dimensional encoder, used to implement the aforementioned lightweight human 3D pose estimation method, including a spatial feature module, an augmented feature module, a global module, a 3D pose module, and a training optimization module: The spatial feature module is used to extract spatial feature vectors of the two-dimensional human pose sequence by constructing a spatial topology modeling branch through a state space encoder.

[0056] The supplementary feature module is used to simultaneously construct a dimensional supplementary modeling branch through a dimensional supplementary encoder on a two-dimensional human posture sequence, and extract supplementary feature vectors of the two-dimensional posture.

[0057] The global module is used to perform feature splicing and fusion on the spatial feature vector and the supplementary feature vector to form a global vector.

[0058] The three-dimensional pose module is used to perform regression operations on the global feature vectors to model them in a three-dimensional coordinate system, forming a complete three-dimensional pose.

[0059] The training optimization module is used to construct a loss function, perform training tuning, and complete 3D human pose modeling.

[0060] To further illustrate the technical effects of the method of this invention, the 3D human pose estimation method described in this invention is compared with the existing related method MotionAGFormer. MotionAGFormer is also a modeling method that applies time-space technology to 3D human pose estimation. It models all joints of the human body through Transformer branches and enhances information between adjacent joints through graph structure branches, thereby improving the accuracy of 3D pose estimation. Experimental results are as follows... Figure 3 and Figure 4 As shown.

[0061] In this comparison, the Human 3.6M dataset was used, with S1, S5, S6, S7, and S8 divided into training sets and S9 and S11 into test sets. The detection accuracy of the algorithm was evaluated using two metrics: Mean Joint Error (MPJPE) and Normalized Mean Joint Error (P-MPJPE). During training, all input sequences used for evaluation were two-dimensional pose sequences, with a length of 243 frames, a central portion of 27 frames, and 400 training iterations.

[0062] Under the same hardware, dataset partitioning, 2D pose sequence input, training epochs, and evaluation protocol, the proposed method and the MotionAGFormer method were experimentally compared. Evaluation metrics included mean joint position error (MPJPE) and normalized mean joint position error (P-MPJPE). Experimental results show that the proposed method achieves better or comparable pose estimation accuracy in the center-frame 3D human pose estimation task. On all test sets of the Human 3.6M dataset, MotionAGFormer's MPJPE, P-MPJPE, and model evaluation speed were 38.4mm, 32.6mm, and 7.87 FPS, respectively; while the proposed method's MPJPE, P-MPJPE, and model evaluation speed were 38.1mm, 30.7mm, and 8.05 FPS, respectively. To reflect overall accuracy, the corresponding metric is the average of all evaluated actions across all test sets. Compared with MotionAGFormer, the method of the present invention reduces MPJPE by 0.3mm and P-MPJPE by 1.9mm, and improves model evaluation speed by 2.28%, indicating that the method of the present invention can further improve the accuracy of center frame 3D pose estimation while maintaining the ability to model lightweight human body space structures.

Claims

1. A lightweight human 3D pose estimation method based on a multi-dimensional encoder, characterized in that, Includes the following steps: Step 1: Input the human body 2D pose sequence, construct a spatial topology modeling branch through a state space encoder, and extract the spatial feature vector of the 2D pose; Step 2: For the human body 2D pose sequence, simultaneously construct a dimensional augmentation modeling branch through a dimensional augmentation encoder to extract the augmentation feature vector of the 2D pose; Step 3: Perform feature concatenation and fusion on the spatial feature vector and the supplementary feature vector to form a global vector; Step 4: Perform regression operation on the global feature vectors to model them in a three-dimensional coordinate system, forming a complete three-dimensional pose; Step 5: Construct the loss function, perform training and optimization, and complete the 3D human pose modeling.

2. The lightweight human 3D pose estimation method based on a multi-dimensional encoder according to claim 1, characterized in that, Step 1 is specifically implemented as follows: Step 1.1: The input video is processed through a spatial topology modeling branch to obtain spatial input features, and the observation reliability parameters of the key points are calculated simultaneously. Step 1.2: Determine the current observation reliability coefficient of the joint based on the joint acceleration stability, fuse it with the spatial input features, and then perform local temporal feature extraction to obtain sub-feature information; Step 1.3: Sub-feature information is input into the state space equations of attitude and motion to obtain attitude output and motion output, which are then transformed into a two-dimensional attitude spatial feature vector containing a complete spatial topology.

3. The lightweight human 3D pose estimation method based on a multi-dimensional encoder according to claim 2, characterized in that, Step 1.1 is specifically implemented as follows: First, perform 2D human keypoint detection frame by frame on the input complete video to obtain the 2D coordinates of all human joints in each frame of the video. This is the input 2D human pose sequence, represented as follows: ; The two-dimensional human pose sequence is input into the spatial topology modeling branch to learn the structural dependencies between various joints of the human body: dimensional transformation is performed on the input two-dimensional human pose sequence, and the central joint of the sequence is selected. K The frame is treated as the processing object, and projected as joint embedding features. The joint embedding features are added to the spatial location encoding to obtain the spatial input features. ; Meanwhile, based on the input sequence Simultaneously construct the center of the input sequence The observation reliability parameters for each key point in the frame are as follows: For each key point, instantaneous motion acceleration is acquired. The acceleration offset is obtained by subtracting the instantaneous motion acceleration at the key point from the median instantaneous motion acceleration. The L2 norm of the acceleration offset is calculated, and the standardized L2 result is used as the key point acceleration stability. Based on the acceleration stability of the joints, the reliability coefficient of the joints in the current observation is determined using an exponential function. .

4. The lightweight human 3D pose estimation method based on a multi-dimensional encoder according to claim 3, characterized in that, Step 1.2 is specifically implemented as follows: Observation reliability characteristics are constructed by combining all observation reliability coefficients according to the same batch, center length, and key point order. , and spatial input features By fusing the data, spatial input features containing observation reliability parameters are obtained. Then, by splitting the window, Convert to window sequence ; Then, within the state-space transformation function unit, the window sequence is... Converted into a three-dimensional normalized window feature matrix. And define a trainable dual-branch feature projection matrix and Perform matrix multiplication separately to generate two sub-feature information. and Subsequently, transposition, one-dimensional channel-wise convolution, and SiLU activation function modules are used sequentially to extract local temporal features from the two sub-feature information, resulting in two sub-feature information denoted as... and .

5. The lightweight human 3D pose estimation method based on a multi-dimensional encoder according to claim 4, characterized in that, Step 1.3 is specifically implemented as follows: Two sub-features are input into the state-space equations relating pose and motion, where, Used to extract pose and motion features; Used to extract non-state compensation features; for The first i The first window n Each block is denoted as Each corresponding observed coefficient After performing matrix multiplication, the input is first into the state-space equation regarding the attitude, and then the attitude output is obtained. ,in For trainable observation coefficient matrix, Indicates the first i The first window n The observation coefficients of each block; based on the attitude state in the state-space equation, predict the output of the current feature position. ,pass and By subtracting the values, we obtain the trend state of motion. ; Movement trend status With observed coefficients After performing matrix multiplication, the input is given to the state-space equations for motion to obtain the motion output. ,according to The adjustment method is used to generate the first sub-feature information. The final output ,in ,in For matrix concatenation operations, It is a weight and bias matrix with the same dimension as the input features; Then, and The data are then concatenated and successively processed through state-space perceptron encoding, window restoration, and dimensional transformation operations, ultimately transforming the data into a two-dimensional pose spatial feature vector containing a complete spatial topology. .

6. The lightweight human 3D pose estimation method based on a multi-dimensional encoder according to claim 5, characterized in that, Step 2 is specifically implemented as follows: Two-dimensional human pose sequence The synchronous input dimension supplementation modeling branch is operated as follows: First, for the two-dimensional coordinates of the joints Perform a discrete cosine transform operation to obtain a two-dimensional vector containing the frequency coefficients represented by the horizontal and vertical coordinates. ; Set frequency gain parameters , and two-dimensional vector Perform gain multiplication to obtain the features ; Then, through dimensional transformation, Numbered according to the frequency order corresponding to each frequency coefficient k Arrange from smallest to largest, and truncate the first few bytes. M Each frequency order yields a spectral sequence. Perform a linear mapping and add location information. Finally, the sequence is obtained. : Subsequently, the sequence Input dimension augmentation feature encoding module; The dimension-enhanced feature encoding module includes an attention layer, a feedforward network layer, a normalization layer, and a residual connection layer, ultimately converting the feature into a dimension-enhanced feature vector containing complete global information. .

7. The lightweight human 3D pose estimation method based on a multi-dimensional encoder according to claim 6, characterized in that, Step 3 is specifically implemented as follows: Then, the spatial feature vector With dimension-enhanced feature vectors Input feature fusion module; firstly, it performs dimensionality unification on the two types of features through a one-dimensional convolution operation, resulting in spatial topological pose features after dimensionality unification. and dimension-enhanced pose features They are represented as follows: Spatial topological pose features after unifying the above dimensions and dimension-enhanced pose features The fusion operation is performed by concatenating channels to obtain the fused feature vector. .

8. The lightweight human 3D pose estimation method based on a multi-dimensional encoder according to claim 7, characterized in that, Step 4 is specifically implemented as follows: The obtained fused feature vector The input pose regression module, whose structure includes a normalization layer and a linear output layer, outputs the coordinates of the human body's 3D joints corresponding to the center frame. ; Using the coordinates of these three-dimensional human joints as the joints of the human skeleton, and according to the preset joint connection relationship, establish skeletal connections between each joint and all the joints it is connected to, to obtain the corresponding three-dimensional human skeleton model.

9. A lightweight human 3D pose estimation system based on a multi-dimensional encoder, used to implement the lightweight human 3D pose estimation method according to any one of claims 1 to 8, characterized in that, Includes the following modules: The spatial feature module is used to extract spatial feature vectors of the two-dimensional human pose sequence by constructing a spatial topology modeling branch through a state space encoder. The supplementary feature module is used to simultaneously construct a dimension-supplement modeling branch through a dimension-supplement encoder on the human two-dimensional pose sequence to extract the supplementary feature vector of the two-dimensional pose. The global module is used to concatenate and fuse spatial feature vectors and supplementary feature vectors to form a global vector. The 3D pose module is used to perform regression operations on global feature vectors and model them in a 3D coordinate system to form a complete 3D pose. The training optimization module is used to construct the loss function, perform training tuning, and complete 3D human pose modeling.

Images