A 3D human pose estimation method, system and application based on a Transformer

Abstract

This invention discloses a Transformer-based 3D human pose estimation method, comprising: obtaining the 2D coordinates and feature maps of human keypoints in all frame sequences using a 2D human pose estimation model, extracting sequence frames, flattening and concatenating the feature maps from the sampled sequence frames with the embedding information of the penultimate layer, and obtaining the image embedding through linear projection; encoding the image embedding through a Transformer encoder, using a self-attention mechanism to capture the spatial dependencies between human keypoints, and enhancing the spatial modeling between keypoints through positional encoding; extracting frequency domain features through Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT) respectively; concatenating the temporal domain features of the sampled image frame sequence and the transformed frequency domain features of all sequence frames, and then fusing the temporal and frequency domain features through Transformer encoding; and outputting the predicted 3D human pose estimation result through a regression head. This invention also discloses an estimation system with broad application value.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and relates to a 3D human pose estimation method, system and application based on Transformer. Background Technology

[0002] Currently, human pose estimation methods in monocular RGB image scenes are mainly divided into 2D and 3D methods. 2D human pose estimation detects the key points of the human body on a two-dimensional plane. With the development of deep learning technology in recent years, significant progress and excellent performance have been achieved in 2D human pose estimation in images and videos, and it is relatively mature. In contrast, 3D human pose estimation is much more difficult due to the challenge of depth blur. An important direction for 3D human pose estimation is based on 2D-to-3D conversion methods, that is, first obtaining the 2D coordinates of key points using mature 2D pose estimation methods, and then inferring the 3D coordinates using a 3D human model and optimization algorithms. Some major problems in this direction include:

[0003] The lack of original image feature information: Images may have problems such as complex background interference, multiple people in the scene and diverse poses, and 2D pose estimation methods may not be able to fully capture and process this information;

[0004] Multimodal information fusion: Most methods fail to fully utilize intermediate feature information in 2D models;

[0005] Accumulation of model error: During the 2D to 3D conversion process, the error in 2D pose estimation will be propagated and amplified into 3D pose estimation, resulting in inaccurate final 3D pose estimation.

[0006] Limitations of model structure: In this scenario, many models are not capable enough in handling global features and medium-to-long-distance dependencies. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide a 3D human pose estimation method based on Transformer. This method inputs the 2D model's input and intermediate generated features, along with its output information, into a new model. Through its self-attention mechanism to capture global features and multi-scale feature fusion capabilities, as well as its flexible structural design, the method aims to solve the aforementioned problems, improve estimation accuracy and robustness, while maintaining high computational efficiency.

[0008] This invention provides a 3D human pose estimation method based on Transformer, the estimation method comprising the following steps:

[0009] Step 1: Use a 2D human pose estimation model to obtain the 2D coordinates and feature maps of human key points in all frame sequences, and sample sequence frames through a sliding time window.

[0010] Step 2: Flatten and stitch the feature map in the sampled sequence frame with the embedding information of the penultimate layer, and obtain the image embedding through linear projection; encode the image embedding through the Transformer encoder, use the self-attention mechanism to capture the spatial dependencies between human key points, and enhance the spatial modeling between key points through position encoding.

[0011] and,

[0012] The 2D coordinates and features of human key points in all sequences are transformed from the time domain to the frequency domain (i.e., transformed from the time domain to the frequency domain), and the frequency domain features are extracted by Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT), respectively.

[0013] Step 3: Concatenate the temporal and frequency domain features of the sampled image frame sequence and the transformed frequency domain features of all sequence frames, and then use Transformer encoding to fuse the temporal and frequency domain features.

[0014] Step 4: Capture information from the input feature sequence through 1D convolution, compress the sequence information into a single intermediate frame, and output the predicted 3D human pose estimation result.

[0015] In step one, by inputting a monocular RGB image into the existing mainstream 2D ​​human pose estimation model, the model outputs the 2D coordinates and feature maps of all human key points; the 2D coordinate information of the human key points is represented as p0(x0,y0), p1(x1,y1), ..., p i (x i ,y i By employing a sliding time window strategy, frame sequences are extracted, and 2D coordinates and feature maps are obtained in the same way.

[0016] In step two, the coordinates and feature maps obtained from the extracted sequence frames are processed as follows:

[0017] The flattened representation of the feature map is as follows:

[0018] Where Feat is the feature map, H F W F and C F These are height, width, and number of channels, respectively.

[0019] The second-to-last layer of embedding information is flattened as follows:

[0020] Among them, En-1 It's the second-to-last layer of embedding, L 2d It is the sequence length, D 2d It is the embedding dimension;

[0021] The flattened feature map and the flattened second-to-last layer embedding information are concatenated to obtain the concatenated feature, represented as:

[0022] The concatenated feature vectors are linearly projected, and then mapped to a fixed-size embedding space through a fully connected layer to obtain the original image embedding, denoted as: E feat =E concat W concat +b concat ;

[0023] in, It is a weight matrix. It is the bias vector, D f It is the dimension of the final embedding.

[0024] In step two, each keypoint coordinate (xi, yi) in each frame is also treated as a patch, and linear projection is used to map it to a high-dimensional patch embedding space, represented as: Pe = X p W p +b p ;

[0025] Among them, X p This represents the input image patch or feature, with shape (N). p ,P p C in );W p This represents the learnable weight matrix, with shape (C). in C out );b p This represents the bias term, with the shape (C). out ); P e It is the output patch embedding, with shape (N) p ,P p C out );

[0026] N p It is the batch size, P p It is the number of patches, C in It is the number of input channels, C out This refers to the number of output channels;

[0027] The transformation feature of 2D keypoint coordinates is represented by E. J =Pe+PE, where Pe represents the output patch embedding and PE represents the positional encoding embedding;

[0028] And / or,

[0029] During the keypoint embedding process, a positional encoding embedding is added to retain the spatial position information of each keypoint for relative position identification; the corresponding positional encoding is performed using a fixed sine and cosine method, as shown below:

[0030]

[0031] Where pos represents the position index, pos∈[0,l-1], l represents the sequence length, and i represents the dimension index. D pos This indicates the dimension of the location encoding embedding.

[0032] In step two, the original image embedding and keypoint embedding are also used as input, and after passing through the Transformer encoder, spatial two-dimensional coded transformation features are generated.

[0033] The self-attention mechanism calculates the correlation between the embeddings of various key points in the input sequence to obtain the interdependencies between different key points. Multi-head attention enables the model to capture feature information from multiple different perspectives in parallel. Each head performs self-attention calculation independently, and the results are then combined, as shown below:

[0034] MultiHead(Q,K,V)=Concat(head1,…,head h W O ,

[0035] in, The weight matrix represents various linear transformations, and Q, K, and V represent the query, key, and value matrices in the self-attention mechanism, respectively.

[0036] The output of the attention mechanism is processed by a feedforward neural network, and a further nonlinear transformation is performed on the embedding of each keypoint, as follows: FFN(x) = ReLU(xW1+b1)W2+b2.

[0037] Where x represents the output of the attention mechanism, W1 and W2 represent the weights in the feedforward neural network FFN, and b1 and b2 represent the biases in the feedforward neural network FFN.

[0038] And / or,

[0039] Using layer normalization and residual connections, the output of the Transformer layer is obtained as follows:

[0040] E=LayerNorm(FFN(MultiHead(X,X,X))+X),

[0041] Where X represents the transformed features of the input feature map and the transformed features of the penultimate layer features, denoted as [E feat E J ], E feat E is the transformation feature of the feature map. J Transformation features for key point coordinates;

[0042] The transformation features E are obtained by stacking the transformations multiple times in the Transformer computation and then encoding them in the spatial Transformer. s ;

[0043] In the stacked computation process, each input is the output of the previous layer, represented as:

[0044] E n =LayerNorm(FFN(MultiHead(E n-1 E n-1 E n-1 ))+E n-1 ), n=1,2…N,

[0045] The final transformed feature E output after spatial Transformer encoding s =E N .

[0046] In step two, it is necessary to calculate the coefficients of the Discrete Cosine Transform (DCT) and the Discrete Wavelet Transform (DWT).

[0047] The 2D keypoint coordinates on the complete frame sequence F are converted into DCT coefficients, and the first N values ​​are retained using a low-pass filter. DCT The coefficients are (<<F), flattened, and then mapped to a fixed-size embedding space through linear projection, as shown below:

[0048] C = DCT(x), where

[0049] Where i = 0, ..., N DCT -1;j=0,…,J-1;k=0,1,

[0050]

[0051] E DCT =W DCT C flat +b DCT ,

[0052] Where x represents the coordinates of 2D keypoints on the complete frame sequence F, J represents the number of human body key nodes, i represents the number of frames or the number of human body key nodes, and N represents the number of frames or the number of human body key nodes. DCT This indicates the preset coefficient retention factor, which is much smaller than F and C. flat E represents the transformation of a multidimensional matrix into a one-dimensional vector. DCT Indicate C flat The embedding, W DCT Let b be the weight matrix. DCT For bias;

[0053] And / or,

[0054] Perform DWT transformation on the 2D keypoint coordinates of the complete frame sequence F, denoted as Y. DWT All low-frequency approximation coefficients are filtered and saved. High-frequency detail coefficients are retained using a multi-level selection and soft thresholding method. These coefficients are then flattened and mapped to a fixed-size embedding space via linear projection, as shown below:

[0055] Y DWT =DWT(x)={A L D L D L-1 ,…,D1}, where

[0056] A′ L =A L ,

[0057]

[0058] Y selected ={A′ L ,D′,D′ -1 ,…,D′ -K+1},

[0059] Y flat =flatten(Y) selected ),

[0060] E DWT =W DWT Y flat +b DWT ,

[0061] Where x represents the 2D keypoint coordinates on the complete frame sequence F, L represents the decomposition level, A represents the low-frequency approximation coefficient, D represents the high-frequency detail coefficient, J represents the number of human body key nodes, λ represents the threshold, and Y... selected Y represents the coefficient that DWT ultimately selected. flat e represents the transformation of a multidimensional matrix into a one-dimensional vector. DWT Y represents flat Embedding of linear projection mapping, W DWT Let b be the weight matrix. DWT For bias;

[0062] In step three, a sliding time window sampling strategy is adopted to further learn and utilize the feature information in the time dimension;

[0063] First, a subsequence of length f is sampled from the complete sequence F. For each frame in this subsequence, its spatial transformation feature E is obtained. s0 To E s(f-1) Add the corresponding location code. arrive Used to preserve the temporal information of the frame sequence; the transformation features and position codes are added together E′ si =E si +PE si , i = 0, 1, ..., f-1;

[0064] The processed feature E′ si ∈R f×J×c As a time series feature, predict the coordinates of 3D human key points in intermediate frames;

[0065] And / or, E for time-frequency domain Transformer transformation DCT e DWT Position encoding is also added to the features:

[0066] E′ DCTi =E DCTi +PE ci i = 0, 1, ..., N DCT -1,

[0067] E′ DWTi =E DWTi +PE di i = 0, 1, ..., N DWT -1, where i represents the frame sequence encoding;

[0068] In one specific implementation, i = 0, 1, ..., 8.

[0069] Time-domain feature E′ si and two frequency domain features E′ DCTi E′DWTi Concatenating them together, we get: z = [E′ si ;E′ DCT ;E′ DWT It serves as the input to the time-domain and frequency-domain feature fusion Transformer.

[0070] In step four, a 1D convolutional layer is used to capture and fuse information in the temporal dimension, represented as: Z = Conv1D(z K ),in, This is the output of the Transformer module for time-domain and frequency-domain feature fusion, where J is the number of key points on the human body and 3 is the feature dimension of each key point.

[0071] Applying a weighted average operation along the frame dimension compresses the sequence information into a single center frame, as shown below:

[0072]

[0073] Among them, w i The weights are learned and satisfy...

[0074] Using an MLP block that includes layer normalization and a linear layer, it can be represented as follows:

[0075] Z norm =LayerNorm(Z) avg ),

[0076] y = LinearLayer(Z) norm ),

[0077] Final output These are the predicted 3D coordinates of the center frame.

[0078] The present invention also provides an estimation system for implementing the above estimation method, the estimation system comprising: an input and 2D pose estimation module, a feature encoding and spatial Transformer module, a time-frequency domain feature and fusion encoding module, and a 3D pose prediction and optimization module;

[0079] The input and 2D pose estimation module receives a monocular RGB image and uses a pre-trained 2D pose estimation model to extract the 2D coordinates and feature maps of human key points.

[0080] The feature encoding and spatial Transformer module concatenates the feature map output by the 2D pose estimation model with the penultimate layer embedding information to generate the image feature vector; and encodes the feature vector through the spatial Transformer model to capture the spatial dependencies between human key points, and uses self-attention mechanism and position encoding to enhance the understanding of human pose.

[0081] The time-frequency domain feature and fusion coding module performs time-domain and frequency-domain transformations on the input 2D coordinates and feature maps to obtain time-domain and frequency-domain features, and encodes them through a time-domain and frequency-domain fusion Transformer to capture the spatiotemporal dependencies between key points.

[0082] The 3D pose prediction and optimization module captures temporal information in the sequence through 1D convolution, compresses it into intermediate frames, outputs the predicted 3D human pose key point coordinates, and uses the average joint position error for model optimization.

[0083] The present invention also provides the above-mentioned estimation method or estimation system for application in human posture analysis, motion recognition and monitoring, human-computer interaction and intelligent monitoring.

[0084] The beneficial effects of this invention include:

[0085] This invention combines 2D model output encoding, spatial transformer encoding, time-frequency domain transform encoding, and time-frequency domain feature fusion transformer encoding to propose a novel multimodal 3D pose estimation method. This method not only makes full use of the original features extracted from the graph in the 2D human pose estimation model, but also effectively improves the model's performance in complex scenes through the co-encoding of spatial and time-frequency domain features. The results are greatly improved without increasing the amount of computation or reducing the performance too much.

[0086] The human pose estimation method in this invention utilizes multiple layers of feature information from a 2D model, including key point coordinates, feature maps (including low-level visual features such as edges and textures), and the penultimate layer embedding containing high-level semantic information. These features are flattened and stitched together to ensure that the local details and global semantic information of the original image are preserved, thus solving the problem of missing original feature information of images in the prior art.

[0087] The human pose estimation method in this invention designs a complete feature fusion process. It encodes the input, intermediate features and output information of the 2D model in a unified manner, then fuses the features of different modalities through the multi-head attention mechanism of the spatial Transformer, combines two time-frequency domain transformations, DCT and DWT, to extract and fuse features from the frequency domain perspective, and finally uses the time-frequency domain feature fusion Transformer to perform the final feature integration. This allows the method of this invention to make full use of multimodal fusion information.

[0088] The human pose estimation method in this invention uses DCT to capture local and global features of the image, employs DWT for multi-scale analysis to provide a more robust feature representation, and improves robustness to inaccurate joint detection by introducing time-frequency domain transform coding; it also considers temporal information and uses a sliding window (4 frames before and after) to reduce the error of single-frame prediction and reduce the accumulation of model error.

[0089] The human pose estimation method in this invention adopts an improved spatial Transformer structure. By introducing a multi-scale attention mechanism, it can simultaneously capture local and global spatial relationships. The method uses relative position encoding, which can better model the spatial relationships between key points. It also designs an adaptive number of layers, which dynamically adjusts the number of Transformer layers according to the input complexity. The method of this invention enhances the model's ability to process global features and model long-distance dependencies through multi-level feature extraction and fusion, and improves the problem of insufficient ability of existing models to process global features and medium-to-long-distance dependencies. Attached Figure Description

[0090] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0091] Figure 1 This is a schematic diagram of the complete 3D human pose estimation model structure of the present invention.

[0092] Figure 2 This is a schematic diagram of the spatial Transformer model structure of the present invention.

[0093] Figure 3 This is a schematic diagram of the Transformer model structure for time-domain and frequency-domain feature fusion according to the present invention. Detailed Implementation

[0094] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0095] This invention proposes a Transformer-based 3D human pose estimation method, comprising: acquiring the 2D coordinates and feature maps of human keypoints in all frame sequences using a 2D human pose estimation model; sampling sequence frames; flattening and concatenating the feature maps from the sampled sequence frames with the embedding information of the penultimate layer; obtaining the image embedding through linear projection; encoding the image embedding through a Transformer encoder; capturing the spatial dependencies between human keypoints using a self-attention mechanism; and enhancing the spatial modeling between keypoints through positional encoding; simultaneously performing time-domain and frequency-domain transformations on the 2D coordinates and features of human keypoints in all sequences, extracting frequency-domain features through Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT), respectively; concatenating the time-domain features from the sampled image frame sequence with the frequency-domain features after transformation of all sequence frames; and then encoding the time-domain and frequency-domain features through Transformer encoding; capturing information from the input feature sequence through 1D convolution (regression head); compressing the sequence information into a single intermediate frame; and outputting the predicted 3D human pose estimation result. This invention also proposes an estimation system with broad application value.

[0096] This invention provides a 3D human pose estimation method based on Transformer, such as... Figure 1 As shown, the estimation method includes the following steps:

[0097] Step 1: Use a 2D human pose estimation model to obtain the 2D coordinates and feature maps of human key points in all frame sequences, and sample the sequence frames.

[0098] Step 2: Flatten and stitch the feature map in the sampled sequence frame with the embedding information of the penultimate layer, and obtain the image embedding through linear projection; encode the image embedding through the Transformer encoder, use the self-attention mechanism to capture the spatial dependencies between human key points, and enhance the spatial modeling between key points through position encoding.

[0099] and,

[0100] The 2D coordinates and features of human key points in all sequences are transformed in the time domain and frequency domain, and the frequency domain features are extracted by Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT), respectively.

[0101] Step 3: Concatenate the temporal and frequency domain features of the sampled image frame sequence and the transformed frequency domain features of all sequence frames, and then use Transformer encoding to fuse the temporal and frequency domain features.

[0102] Step 4: Capture information from the input feature sequence through 1D convolution, compress the sequence information into a single intermediate frame, and output the predicted 3D human pose estimation result.

[0103] This invention mainly solves the following technical problems:

[0104] Insufficient Accuracy in 3D Pose Estimation: Existing 3D human pose estimation methods based on monocular RGB images suffer from insufficient accuracy when handling complex backgrounds, occlusion, and scenes with multiple characters. This invention improves the accuracy of 3D pose estimation by introducing a Transformer-based structure and feature data from the previous stage model, thereby enhancing global feature capture and multi-scale feature fusion capabilities. The goal is to reduce the mean joint position error (MPJPE) by more than 15%.

[0105] Poor real-time performance: Many existing methods struggle to balance high-precision 3D pose estimation and real-time processing. This invention achieves a combination of high accuracy and real-time performance on high-performance hardware by optimizing model structure and computational efficiency, aiming to achieve processing speeds of over 30 FPS on mainstream GPUs while maintaining high accuracy.

[0106] Lack of robustness: Existing methods are not robust to complex backgrounds, occlusion, and multiple person poses, and are easily affected by external interference. This invention utilizes the global attention mechanism of Transformer and multi-scale feature fusion technology to significantly enhance the robustness of the model under various complex conditions.

[0107] Data privacy and security: In rehabilitation settings, the privacy and security of patient data are paramount. By performing data processing and pose estimation locally, this invention reduces the frequency and distance of data transmission, thereby improving data privacy and security.

[0108] Multimodal information fusion: effectively integrates the input, intermediate features and output information of 2D models to improve the accuracy of 3D pose estimation.

[0109] Adaptability and scalability: Through the flexible Transformer architecture, it supports personalized customization and expansion, and can better adapt to the needs of different application scenarios.

[0110] Specifically, the implementation of the 3D human pose estimation method of the present invention mainly includes the following parts:

[0111] ●2D model output encoding

[0112] This invention designs a flexible 2D model output encoding method that is compatible with various mainstream 2D ​​human pose estimation models (such as OpenPose, HRNet, etc.). The following information is obtained through the 2D human pose estimation model:

[0113] ○Key point coordinates: Directly reflects the 2D coordinate positions of various key points on the human body;

[0114] ○Feature map: Contains rich low-level visual features, such as edges and textures;

[0115] ○ The second-to-last layer of embedding: captures high-level semantic information and abstract features;

[0116] The 2D to 3D conversion method fully utilizes the feature information of the 2D model. For each frame of image, the 2D model outputs information on J (in one specific embodiment of the invention, J = 21) key points, denoted as p0, p1, ..., p... (The 2D model outputs information on J key points, denoted as p0, p1, ..., p...). 20 The coordinate values ​​are denoted as (x0, y0), (x1, y1), ..., (x 20 ,y 20 In 2D network models, feature maps and the embedding information of the penultimate layer often contain important original features and abstract features of the 2D model, which play a crucial role in the 2D to 3D conversion. Specifically, feature maps contain low-level visual features of various local regions in the input image, while embedding information captures higher-level abstract semantic information.

[0117] To better utilize this information, the feature maps and the embedding information from the penultimate layer are flattened, transforming the multidimensional matrix into a one-dimensional vector, and then concatenated. This process ensures that both local details and global semantic information of the original image are preserved. Next, these concatenated feature vectors are linearly projected, mapping the high-dimensional feature vectors to a fixed-size embedding space through a fully connected layer. This set of linearly projected feature vectors represents the embedding of the original image.

[0118] Drawing inspiration from ViT (Vision Transformer), the coordinates (x, y) of each keypoint in each frame are... i ,y iThe keypoint is treated as a patch and mapped onto a high-dimensional patch embedding. This mapping is a trainable linear process. Simultaneously, to preserve the positional information between keypoint coordinates, a position encoding embedding is generated for each keypoint coordinate, retaining spatial positional information. The patch embedding and position encoding embedding are combined and fed into the next encoder for further processing and modeling. This part (including patch embedding and position embedding) of the feature vector represents the keypoint embedding.

[0119] The specific modeling process is shown in the attached figure. Figure 2 The spatial encoder is illustrated in the following diagram, and the formal expressions of some of the relevant parts are as follows:

[0120] The original image embedding (transformed feature) obtained by flattening, stitching, and projecting the feature map and the penultimate layer feature map can be represented as E. feat Specifically:

[0121] ■Feature map flattening: Where Feat is the feature map, and H... F W F and C F These are height, width, and number of channels, respectively.

[0122] ■ Flatten the second-to-last layer of features: Where E n-1 It's the second-to-last layer of embedding, L 2d It is the sequence length, D 2d It is the embedding dimension;

[0123] ■Jointing features:

[0124] ■ After linear projection and mapping, the feature map of the final 2D model and the transformed features of the penultimate layer: E feat =E concat W concat +b concat ,in It is a weight matrix. It is the bias vector, D f It is the dimension of the final embedding. This is the final original image embedding;

[0125] The transformation characteristics of keypoint coordinates in 2D can be represented as: E J=Pe + PE, where Pe represents the output patch embedding and PE represents the positional encoding embedding;

[0126] ■ Patch embedding can be represented as Pe = X p W p +b p ,in,

[0127] ●X p It is the input image patch or feature, with shape (N) p ,P p C in In this invention, the key point coordinates (x, y) of the human body are... i ,y i );

[0128] ●W p It is a learnable weight matrix with shape (C in C out );

[0129] ●b p It is a bias term with shape (C out );

[0130] ●Pe is the output patch embedding, with shape (N) p ,P p C out );

[0131] ●N p It is the batch size, P p It is the number of patches, C in It is the number of input channels, C out This refers to the number of output channels;

[0132] ■ Position embedding: This invention uses a fixed sine and cosine method for encoding, namely:

[0133]

[0134] Where: pos is the position index, pos∈[0,l-1], l represents the sequence length, and i is the dimension index. D pos It is the dimension of the position encoding embedding;

[0135] ● Spatial Transformer Coding

[0136] This invention also designs an improved spatial Transformer structure, which has the following characteristics:

[0137] ○ Multi-scale attention mechanism: By setting receptive fields of different sizes, it captures local and global spatial relationships;

[0138] ○Relative position encoding: Introducing relative position information to better model the spatial relationships between key points;

[0139] ○ Adaptive layer count: Dynamically adjusts the number of Transformer layers based on input complexity to balance performance and computational efficiency;

[0140] The specific steps involve inputting the 2D output model encoding of a sampled frame sequence into a spatial Transformer. The spatial Transformer is an attention-based model, incorporating multi-head attention, capable of capturing long-range dependencies and adept at capturing spatial relationships in images. In this invention, the spatial Transformer extracts spatial features from the image by encoding each keypoint in the frame. Specifically, the embeddings representing the original image and the keypoints output by the 2D model are used as input, and after passing through the Transformer encoder, spatial two-dimensional encoded transformation features are generated. (See attached diagram) Figure 2 As shown.

[0141] The specific modeling process of the spatial Transformer is shown in the attached figure. Figure 2 As shown, the formula is as follows:

[0142] ○ Spatial Transformer Formula

[0143]

[0144] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0145]

[0146] FFN(x)=ReLU(xW1+b1)W2+b2

[0147] E=LayerNorm(FFN(MultiHead(X,X,X))+X)

[0148] in,

[0149] Attention: The formula for attention mechanisms;

[0150] MultiHead: Multi-head attention mechanism;

[0151] Q,K,V: Query, key, and value matrix;

[0152] h: Number of attention heads;

[0153] W O , Weight matrices for various linear transformations;

[0154] d k : The dimension of the key vector;

[0155] FFN: Feedforward Neural Network;

[0156] W1, W2, b1, b2: Weights and biases in FFN;

[0157] X: Input to the Transformer, which in this invention is the transformation feature E of the feature map and the penultimate layer features. feat Transformation feature E of 2D keypoint coordinates J That is, [E] feat E J ];

[0158] LayerNorm: Layer Normalization

[0159] E: Output of the Transformer layer

[0160] ■ It is worth noting that X = [E feat E J The input Transformer is used to compute the representation of layer 0. This process stacks N layers, with each input being the output of the previous layer.

[0161] E n =LayerNorm(FFN(MultiHead(E n-1 E n-1 E n-1 ))+E n-1 )

[0162] n = 1, 2, ..., N

[0163] ■ The final transformation feature E output after spatial Transformer encoding s =E N ;

[0164] ●Time-frequency domain transform coding

[0165] In this invention, a time-domain frequency-domain transform encoding operation is performed on the entire frame sequence of human body key points. This operation can effectively solve the following two problems:

[0166] ○ More efficient at processing long sequences: Frequency domain representation can process long sequence data more effectively, which is especially important when dealing with large amounts of motion data;

[0167] ○ More robust to inaccurate joint detection: Frequency domain representation is more robust to noise and outliers (such as human key detection errors), enabling the model to perform better even when faced with imperfect data;

[0168] In this step, two time-domain and frequency-domain transformation methods are used to encode the 2D output model of the frame sequence: Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT). DCT is a transformation method that converts signals from the time domain to the frequency domain and is widely used in image compression and signal processing. DCT can obtain the frequency domain features of each frame, reflecting local details and global structure in the image. The other method is Discrete Wavelet Transform (DWT), a discretized version of wavelet transform that decomposes the signal into high-frequency and low-frequency components. In image recognition, DWT is often used for multi-scale analysis and feature extraction. By performing DWT on an image, sub-band images of different resolutions can be obtained, containing details and approximate information of the image. This decomposition helps capture features such as texture and edges while reducing computational complexity. DWT performs well in tasks such as image compression, denoising, edge detection, and texture analysis, providing an effective preprocessing and feature representation method for deep learning models, improving the accuracy and efficiency of image recognition.

[0169] Specifically, the 2D output model encoding (i.e., 2D keypoint output coordinates) of each frame is first subjected to DCT and DWT transformations, and the correlation coefficient of each frame is calculated. The DCT correlation coefficient represents the weight of different frequency components in the image and can effectively capture local and global features in the image. The DWT correlation coefficient is divided into low-frequency approximation coefficient and high-frequency detail coefficient. The low-frequency approximation coefficient represents the overall trend and contour of the input signal and contains the main energy of the signal. The high-frequency detail coefficient represents the details and rapid changes of the signal and contains high-frequency information such as edges and textures.

[0170] For DCT, the 2D keypoint coordinates on the complete frame sequence F After conversion to DCT coefficients (J is the number of key nodes in the human body; in one specific embodiment of this invention, J = 21), it is expressed as follows: Then, for each keypoint trajectory, a low-pass filter is used to retain only the first N points. DCT (<<F) coefficients Then these correlation coefficients are flattened, that is, the multidimensional matrix is ​​transformed into a one-dimensional vector C. flat It is then mapped to a fixed-size embedding space via linear projection. (where c is the vector dimension of the embedding); the specific formula is as follows:

[0171] ○DCT Formula Process

[0172] C = DCT(x)

[0173] Where i = 0, ..., N DCT -1; j = 0, ..., J-1; k = 0, 1

[0174] C flat =flatten(C′)

[0175] E DCT =W DCT C flat +b DCT

[0176] Among them, W DCT Let b be the weight matrix. DCT For bias;

[0177] For DWT, the 2D keypoint coordinates on the complete frame sequence F After DWT conversion, denoted as Y DWT After screening, low-frequency approximation coefficients and some high-frequency detail coefficients are retained. Let L be the decomposition level, A be the low-frequency approximation coefficients, and D be the high-frequency detail coefficients. The specific screening principle is as follows: low-frequency approximation coefficients A L All are retained, i.e., A′ L =A L For high-frequency detail coefficients D i By combining multi-level selection and soft thresholding, important information is effectively preserved while noise is reduced, retaining levels L, L-1, ..., L-K+1, and D′. i =sign(D i )max(|D i |-λ i ,0), where i=L,L-1,…,L-K+1, and the threshold λ i The coefficients can be adjusted according to task requirements and signal characteristics. Therefore, the final coefficient selected for DWT is Y. selected ={A′ L ,D′ L ,D′ L-1 ,…,D′ L-K+1 Similarly, these correlation coefficients are flattened, that is, the multidimensional matrix is ​​transformed into a one-dimensional vector Y. flat It is then mapped to a fixed-size embedding space via linear projection. (N DWT The number of DWT coefficients to be retained (where c is the vector dimension of the embedding); the specific formula is as follows:

[0178] ○DWT Formula Process

[0179] Y DWT =DWT(x)={A L D L D L-1 ,…,D1}

[0180] A′ L =A L

[0181]

[0182] Y selected ={A′ L ,D′ L ,D′ L-1 ,…,D′ L-K+1}

[0183] Y flat =flatten(Y) selected )

[0184] E DWT =W DWT Y flat +b DWT

[0185] Among them, W DWT Let b be the weight matrix. DWT For bias.

[0186] ●Time-domain and frequency-domain feature fusion Transformer coding

[0187] In the process of transforming 2D to 3D coordinates, temporal features also need to be considered. Although the spatial Transformer outputs the transformation features E of a single frame after encoding, s However, this is still insufficient to fully utilize information in the time dimension. Therefore, this invention designs a sliding time window sampling strategy, taking a specific implementation as an example:

[0188] ○ Sample a subsequence of length f = 9 (denoted as frame 0-8) from the complete sequence F.

[0189] For each frame in this subsequence:

[0190] ■ Obtain its spatial transformation characteristics E s0 To E s(f-1) ;

[0191] ■ Add the corresponding location code arrive Used to preserve the timing information of the frame sequence;

[0192] ■ Add the transformation feature and position code E′ si =E si +PE si , i = 0, 1, ..., f-1;

[0193] ○ The processed feature E′ si ∈R f×J×c The input is fed into the subsequent structure as a time series feature to predict the coordinates of 3D human key points in the middle frame f4 of the sliding time window;

[0194] This sampling method reduces computational complexity while effectively learning and utilizing feature information in the time dimension.

[0195] Similarly, the E of the time-frequency domain Transformer transformation DCT E DWT In terms of features, considering long-distance dependent features, this invention also incorporates position embedding, which can be expressed by the formula:

[0196] E′ DCTi =E DCTi +PE ci i = 0, 1, ..., N DCT -1;

[0197] E′ DWTi =E DWTi +PE di i = 0, 1, ..., N DWT -1;

[0198] Note the position encoding PE mentioned in the above process. si PE ci PE di This method is consistent with the fixed sine and cosine method mentioned above in this invention, and the specific details will not be repeated.

[0199] The next step is to apply the time-domain feature E′ si and two frequency domain features E′ DCT ,E′ DWT When pieced together, it means:

[0200] z = [E′ si ;E′ DCT ;E′ DWT ]

[0201] As input to the next step of the Transformer process for time-domain and frequency-domain feature fusion;

[0202] Because the method in this invention combines time-domain and frequency-domain features to improve model performance, the Transformer model is improved to better integrate time-domain and frequency-domain features. The structure of the improved time-domain and frequency-domain feature fusion Transformer is shown below:

[0203] ○ Shared self-attention mechanism: Temporal and frequency domain features share the same self-attention mechanism to capture the correlation between them.

[0204] ○ Using different feedforward networks: For time-domain features, specially designed MLP-Freq-C and MLP-Freq-W are used to better handle time-domain information:

[0205] ■MLP-Freq-C achieves frequency domain processing of time-domain features by applying Discrete Cosine Transform (DCT) and Inverse Discrete Cosine Transform (IDCT) before and after a regular MLP. The principle of this method is:

[0206] ● High-frequency noise can be removed from the frequency domain characteristics using a low-pass filter;

[0207] However, some important details (such as rapid local motion) may be mistaken for noise and filtered out.

[0208] ● To avoid this situation, FreqMLP, as a trainable frequency domain filter, can adaptively adjust the weights of different frequency components to retain important information;

[0209] ■MLP-Freq-W utilizes DWT (Discrete Wavelet Transform) and IDWT (Inverse Discrete Wavelet Transform) before and after ordinary MLP, respectively, to achieve frequency domain processing of time-domain features. The principle of this method is:

[0210] ●DWT can decompose a signal into components of different frequencies and time scales, providing the ability for multi-resolution analysis;

[0211] ● Compared to DCT, DWT is better suited for capturing local time-frequency features, which is especially effective for features such as human motion, which have local and multi-scale characteristics;

[0212] ● By applying a trainable MLP in the frequency domain, the weights of different frequency and scale components can be adaptively adjusted, which allows the model to effectively filter out noise while preserving important motion details.

[0213] ●IDWT converts the processed frequency domain features back to the time domain, preserving the integrity of the signal.

[0214] This improvement effectively fuses time-domain and frequency-domain features, thereby enhancing the model's performance and robustness. The improved Transformer is a time-domain and frequency-domain feature fusion Transformer, and its model structure is shown in the attached figure. Figure 3 As shown, the model structure including inputs and outputs is attached. Figure 1 As shown.

[0215] The process of inputting the time-domain and frequency-domain concatenated feature z into the time-domain and frequency-domain feature fusion Transformer can be expressed as a formula:

[0216] z′ k =MSA(z k )

[0217]

[0218] Where k = 0, 1, ..., K-1 is the number of stacked layers of the encoder, and MSA is a multi-head self-attention network.

[0219] ●Return to Head

[0220] In the final step, the output of the time-domain and frequency-domain feature fusion Transformer is fed into the regression head to fit the 3D coordinates of the center frame. The specific process is as follows:

[0221] First, 1D convolutional layers are used to capture and fuse information in the temporal dimension. This step can be represented as:

[0222] Z = Conv1D(z) K )

[0223] in This is the output of the Transformer module for time-domain and frequency-domain feature fusion, where J is the number of key points on the human body and 3 is the feature dimension of each key point.

[0224] Next, a weighted average operation is applied to the frame dimension to compress the sequence information into a single center frame:

[0225]

[0226] Where w i The weights are learned and satisfy...

[0227] Then, a simple MLP block is used, containing layer normalization and a linear layer:

[0228] Z norm =LayerNorm(Z) avg )

[0229] y = LinearLayer(Z) norm )

[0230] Final output The predicted 3D coordinates of the center frame

[0231] Furthermore, the standard MPJPE (Mean Joint Position Error) is used as the loss function:

[0232]

[0233] Where p k and These are the actual and predicted 3D coordinates of the j-th key point, respectively.

[0234] Ultimately, through the above series of processing steps, accurate prediction from 2D images to 3D coordinates was achieved, making full use of the feature information of the 2D model and the spatial, temporal, and frequency domain relationships in the frame sequence, providing an effective method for human pose detection and screening.

[0235] 1) Implementation of Special Case 1: Basic Real-Time Monitoring Version

[0236] Hardware environment:

[0237] GPU: NVIDIA RTX 3060

[0238] Memory: 16GB; CPU: Intel i7 processor

[0239] • Implementation details:

[0240] o 2D pose estimation model selected: HRNet

[0241] Number of key points: 21

[0242] o Sliding window size: 9 frames (current frame plus 4 frames before and after)

[0243] o Transformer structure:

[0244] ■ 4-layer standard Transformer layer

[0245] ■8 points to focus on

[0246] ■Embedding dimension: 256

[0247] o Time-frequency domain transformation: Using only DCT transform

[0248] Real-time performance: Up to 30 FPS

[0249] • Application scenarios:

[0250] Real-time gym workout monitoring

[0251] Home-based rehabilitation training

[0252] o Exercise posture guidance

[0253] Features:

[0254] o Low computing resource requirements

[0255] Good real-time performance

[0256] moderate accuracy

[0257] Suitable for solo use.

[0258] 2) Implementation of Special Case 2: High-Precision Research Version

[0259] Hardware environment:

[0260] oGPU: NVIDIA RTX 4090

[0261] Memory: 32GB

[0262] CPU: Intel i9 processor

[0263] • Implementation details:

[0264] The o2D pose estimation model used is a fusion of OpenPose and HRNet models.

[0265] Number of key points: 21

[0266] Sliding window size: 17 frames (current frame plus 8 frames before and after).

[0267] oTransformer structure:

[0268] ■8 adaptive Transformer layers

[0269] ■ 16 multi-scale attention heads

[0270] ■Embedding dimension: 512

[0271] o Time-frequency domain transformation: Simultaneous use of both DCT and DWT dual transformations

[0272] Processing speed: Approximately 15 FPS

[0273] • Application scenarios:

[0274] o Professional sports movement analysis

[0275] o Medical rehabilitation assessment

[0276] o Motion capture research

[0277] o Multi-person complex scene analysis

[0278] Features:

[0279] o Higher estimation accuracy (MPJPE reduced by more than 20%)

[0280] o Greater robustness

[0281] o Supports complex scenarios

[0282] o Better occlusion handling capabilities

[0283] These two special cases are designed for different application scenarios and performance requirements. Implementation case 1 focuses on real-time performance and practicality, and is suitable for general application scenarios; implementation case 2 pursues higher accuracy and stronger processing capabilities, and is suitable for professional research and analysis scenarios.

[0284] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0285] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0286] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0287] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0288] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0289] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0290] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A 3D human pose estimation method based on Transformer, characterized in that, The estimation method includes the following steps: Step 1: Use a 2D human pose estimation model to obtain the 2D coordinates and feature maps of human key points in all frame sequences, and sample sequence frames through a sliding time window. Step 2: Flatten and stitch the feature map in the sampled sequence frame with the embedding information of the penultimate layer, and obtain the image embedding through linear projection; encode the image embedding through the Transformer encoder, use the self-attention mechanism to capture the spatial dependencies between human key points, and enhance the spatial modeling between key points through position encoding. and, The 2D coordinates and features of human key points in all sequences are transformed in the time domain and frequency domain, and the frequency domain features are extracted by Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT), respectively. Step 3: Concatenate the temporal features of the sampled image frame sequence and the frequency domain features of all transformed frames, and then fuse the temporal and frequency domain features using Transformer encoding. Step 4: Capture information from the input feature sequence through 1D convolution, compress the sequence information into a single intermediate frame, and output the predicted 3D human pose estimation result.

2. The estimation method as described in claim 1, characterized in that, In step one, by inputting a monocular RGB image into the 2D human pose estimation model, the 2D coordinates and feature maps of all human key points are output, and a frame sequence is extracted using a sliding time window strategy. The 2D coordinate information of the key points of the human body is represented as p0(x0,y0), p1(x1,y1), ..., p i (x i ,y i ).

3. The estimation method as described in claim 1, characterized in that, In step two, the coordinates and feature maps obtained from the extracted sequence frames are processed as follows: The feature map is flattened as follows: Where Feat is the feature map, H F W F and C F These are height, width, and number of channels, respectively. The second-to-last layer of embedding information is flattened as follows: Among them, E n-1 It's the second-to-last layer of embedding, L 2d It is the sequence length, D 2d It is the embedding dimension; The flattened feature map and the flattened second-to-last layer embedding information are concatenated to obtain the concatenated feature, represented as: And / or, The concatenated feature vectors are linearly projected, and then mapped to a fixed-size embedding space through a fully connected layer to obtain the original image embedding, denoted as: E feat =E concat W concat +b concat ; in, It is a weight matrix. It is the bias vector, D f It is the dimension of the final embedding; and / or, Step two also includes treating each keypoint coordinate (xi, yi) in each frame as a patch and mapping it to a high-dimensional patch embedding space using linear projection, represented as: Pe = X p W p +b p ; Among them, X p This represents the input image patch or feature, with shape (N). p ,P p C in );W p This represents the learnable weight matrix, with shape (C). in C out );b p This represents the bias term, with the shape (C). out Pe is the output patch embedding, with shape (N) p ,P p C out ); N p It is the batch size, P p It is the number of patches, C in It is the number of input channels, C out This refers to the number of output channels; The transformation feature of 2D keypoint coordinates is represented by E. J =Pe+PE, where Pe represents the output patch embedding and PE represents the positional encoding embedding; And / or, During the keypoint embedding process, a positional encoding embedding is added to retain the spatial position information of each keypoint for relative position identification; the corresponding positional encoding is performed using a fixed sine and cosine method, as shown below: Where pos represents the position index, pos∈[0,l-1], l represents the sequence length, and i represents the dimension index. D pos This indicates the dimension of the location encoding embedding.

4. The estimation method as described in claim 3, characterized in that, In step two, the original image embedding and keypoint embedding are used as input, and after passing through the Transformer encoder, spatial two-dimensional coded transformation features are generated. The self-attention mechanism calculates the correlation between the embeddings of various key points in the input sequence to obtain the interdependencies between different key points. Multi-head attention enables the model to capture feature information from multiple different perspectives in parallel. Each head performs self-attention calculation independently, and the results are then combined, as shown below: MultiHead(Q,K,V)=Concat(head1,…,head h )W O , Among them, head i =Attention(QW i Q ,KW i K VW i V ), W O W i Q W i K W i V The weight matrix represents various linear transformations, and Q, K, and V represent the query, key, and value matrices in the self-attention mechanism, respectively.

5. The estimation method as described in claim 4, characterized in that, The output of the attention mechanism is processed by a feedforward neural network, and a further nonlinear transformation is performed on the embedding of each keypoint, as follows: FFN(x) = ReLU(xW1+b1)W2+b2. Where x represents the output of the attention mechanism, W1 and W2 represent the weights in the feedforward neural network FFN, and b1 and b2 represent the biases in the feedforward neural network FFN. And / or, Using layer normalization and residual connections, the output of the Transformer layer is obtained as follows: E=LayerNorm(FFN(MultiHead(X,X,X))+X), Where X represents the transformed features of the input feature map and the transformed features of the penultimate layer features, denoted as [E feat E J ], E feat E is the transformation feature of the feature map. J Transformation features for keypoint coordinates; The transformation features E are obtained by stacking the transformations multiple times in the Transformer computation and then encoding them in the spatial Transformer. s ; In the stacked computation process, each input is the output of the previous layer, represented as: IN n =LayerNorm(FFN(MultiHead(E n-1 ,IN n-1 ,IN n-1 ))+E n-1 ),n=1,2...N, The final transformed feature E output after spatial Transformer encoding s =E N .

6. The estimation method as described in claim 1, characterized in that, In step two, the coefficients of the Discrete Cosine Transform (DCT) and the Discrete Wavelet Transform (DWT) are calculated; The 2D keypoint coordinates on the complete frame sequence F are converted into DCT coefficients, and the first N values ​​are retained using a low-pass filter. DCT Each coefficient is flattened and then mapped to a fixed-size embedding space via linear projection, as shown below: C = DCT(x), where Where i = 0, ..., N DCT -1;j=0,…,J-1;k=0,1, E DCT =W DCT C flat +b DCT , Where x represents the coordinates of 2D keypoints on the complete frame sequence F, J represents the number of human body key nodes, i represents the number of frames or the number of human body key nodes, and N represents the number of frames or the number of human body key nodes. DCT C represents the preset coefficient retention factor. flat E represents the transformation of a multidimensional matrix into a one-dimensional vector. DCT Indicate C flat The embedding, W DCT Let b be the weight matrix. DCT For bias; And / or, Perform DWT transformation on the 2D keypoint coordinates of the complete frame sequence F, denoted as Y. DWT All low-frequency approximation coefficients are filtered and saved. High-frequency detail coefficients are retained using a multi-level selection and soft thresholding method. These coefficients are then flattened and mapped to a fixed-size embedding space via linear projection, as shown below: Y DWT =DWT(x)={A L D L D L-1 ,…,D1}, where A′ L =A L , Y selected ={A′ L ,D′ L ,D′ L-1 ,…,D′ L-K+1 }, AND flat =flatten(Y selected ), E DWT =W DWT Y flat +b DWT , Where x represents the 2D keypoint coordinates on the complete frame sequence F, L represents the decomposition level, A represents the low-frequency approximation coefficient, D represents the high-frequency detail coefficient, J represents the number of human body key nodes, λ represents the threshold, and Y... selected Y represents the coefficient that DWT ultimately selected. flat E represents the transformation of a multidimensional matrix into a one-dimensional vector. DWT Y represents flat embedding of linear projection mapping, W DWT Let b be the weight matrix. DWT For bias.

7. The estimation method as described in claim 1, characterized in that, In step three, a sliding time window sampling strategy is adopted to further learn and utilize the feature information in the time dimension; From the complete sequence F, sample a subsequence of length f. For each frame in this subsequence: obtain its spatial transformation feature E. s0 To E s(f-1) Add the corresponding location code. arrive Used to preserve the temporal information of the frame sequence; the transformation features and position codes are added together E′ si =E si +PE si , i = 0, 1, ..., f-1, are used as time series features; And / or, E for time-frequency domain transformation DCT E DWT Position encoding is added to all features: HAVE BEEN' DCTi =E DCTi +PE ci ,i=0,1,…,N DCT -1, E′ DWTi =E DWTi +PE di i = 0, 1, ..., N DWT -1, where i represents the frame sequence encoding; Time-domain feature E′ si and two frequency domain features E′ DCTi E′ DWTi Concatenating them together, we get: z = [E′ si ;E′ DCT ;E′ DWT It serves as the input to the time-domain and frequency-domain feature fusion Transformer.

8. The estimation method as described in claim 1, characterized in that, In step four, a 1D convolutional layer is used to capture and fuse information in the temporal dimension, represented as: Z = Conv1D(z K ),in, This is the output of the Transformer module for time-domain and frequency-domain feature fusion, where J is the number of key points on the human body and 3 is the feature dimension of each key point. Applying a weighted average operation along the frame dimension compresses the sequence information into a single center frame, as shown below: Among them, w i The weights are learned and satisfy... Using an MLP block that includes layer normalization and a linear layer, it can be represented as follows: y=LinearLayer(Z norm ), Final output These are the predicted 3D coordinates of the center frame.

9. An estimation system that implements the estimation method as described in any one of claims 1-8, characterized in that, The estimation system includes: an input and 2D pose estimation module, a feature encoding and spatial Transformer module, a time-frequency domain feature and fusion encoding module, and a 3D pose prediction and optimization module; The input and 2D pose estimation module receives a monocular RGB image and uses a pre-trained 2D pose estimation model to extract the 2D coordinates and feature maps of human key points. The feature encoding and spatial Transformer module concatenates the feature map output by the 2D pose estimation model with the penultimate layer embedding information to generate the image feature vector; and encodes the feature vector through the spatial Transformer model to capture the spatial dependencies between human key points, and uses self-attention mechanism and position encoding to enhance the understanding of human pose. The time-frequency domain feature and fusion coding module performs time-domain and frequency-domain transformations on the input 2D coordinates and feature maps to obtain time-domain and frequency-domain features, and encodes them through a time-domain and frequency-domain fusion Transformer to capture the spatiotemporal dependencies between key points. The 3D pose prediction and optimization module captures temporal information in the sequence through 1D convolution, compresses it into intermediate frames, outputs the predicted 3D human pose key point coordinates, and uses the average joint position error for model optimization.

10. The estimation method as described in any one of claims 1-8, or the estimation system as described in claim 9, is applied in human posture analysis, motion recognition and monitoring, human-computer interaction and intelligent monitoring.

Images