A 3D human pose estimation method integrating multi-scale spatiotemporal features

Abstract

The present invention relates to a three-dimensional human posture estimation method integrating multi-scale spatiotemporal features, and belongs to the field of computer vision and graphics and images. The present invention first performs posture recognition on a two-dimensional posture sequence in a human motion video data set, generates three scales of joints, limbs and upper / lower body describing human posture, and maps them into high-dimensional vectors of different sizes; secondly, constructs a spatial multi-scale feature extraction module, and uses multi-head attention to extract human posture features of joints, limbs and upper / lower body at three scales in parallel; then, calculates the query, key and value matrices of the three features, and obtains a three-dimensional human posture integrating spatial multi-scale features; finally, performs multi-scale temporal information encoding on the fused different spatial multi-scale feature frames, and finally generates a refined three-dimensional human posture. The present invention improves the accuracy and efficiency of three-dimensional human posture estimation of two-dimensional video input by integrating multi-scale spatial and temporal features.

Description

Technical Field

[0001] This invention relates to a three-dimensional human pose estimation method that integrates multi-scale spatiotemporal features, belonging to the fields of computer vision and graphics. Background Technology

[0002] The goal of 3D human pose estimation is to estimate the coordinates of key points on a target human body in 3D space. As an upstream task in fields such as human motion analysis, human-computer interaction, and robotics, it has received increasing attention in computer vision and graphics in recent years. However, 3D human pose estimation faces an ill-posed problem: the lack of depth information from 2D to 3D, and the fact that one 3D pose can correspond to multiple 2D poses. Furthermore, captured images of the human body are often occluded by objects or parts of the body itself, and different movements are typically performed at different human spatial scales. For example, when waving, significant motion features are concentrated at a few key points near the arm; performing a single global feature extraction will lose this important local semantic information. When running, performing only local feature extraction will ignore the global motion state information of the human body. When performing 3D human pose estimation based on video input, the human body typically performs various movements at different speeds; for example, before starting a run, the human body maintains a fixed posture for several seconds, while after starting, it makes continuous and large-scale changes in movement within a very short period. Currently, known methods typically extract motion features at a constant time scale, leading to uneven distribution of extracted feature density, redundancy of many similar features, and omission of a few key features, resulting in insufficient representation of motion posture. Furthermore, these methods are limited by a finite receptive field, making it difficult to aggregate long-term temporal information and prone to significant jitter issues.

[0003] Commonly known methods typically focus only on feature extraction at a single scale or only on features in the spatial domain, neglecting information at the temporal scale. For example, Zheng (<IEEE / CVF International Conference on ComputerVision> Patent CN 114565938 A utilizes the Transformer architecture, taking a single human joint as input in the spatial domain and the 2D pose of each frame as input in the temporal domain. It extracts spatiotemporal hybrid features of human joints using two Transformer modules. However, it extracts single-scale features in both the temporal and spatial domains, making it difficult to accurately represent complex human motion patterns. Patent CN 114565938 A extracts global and local features of human pose using fully connected and grouped connections respectively, fuses the obtained feature vectors, and then regresses to obtain the 3D human pose. (Zhang)<IEEE / CVF International Conference on Computer Vision> Patents (2022, 13222-13232) point out that the motion trajectories of different joints differ significantly. They suggest using the Transformer module alternately to encode the trajectory information of the joints, achieving sequence input to sequence output and overcoming the receptive field limitations of video-based input methods. However, this method still extracts fine-grained spatiotemporal information at a single joint scale, neglecting the impact of higher-scale human limb and half-body motion features on accurate 3D human pose estimation. Patent CN 113989854 A constructs a graph convolutional neural network based on the graph topology of the human body, from sparse to dense, for multi-scale feature extraction, and uses a fusion module to achieve information interaction at different scales. Both of these patents only extract features at different scales within the human body's topological space, without considering the significant impact of contextual information provided by preceding and following frames in the temporal domain on refining the estimation results. To better represent complex human motion postures, this invention designs a Transformer-based multi-scale spatiotemporal feature fusion network, extracting multi-scale human motion features in parallel in the spatiotemporal domain to improve the accuracy and inference speed of 3D human pose estimation. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a three-dimensional human pose estimation method that integrates multi-scale spatiotemporal features. This method can extract multi-scale human motion features in parallel in the spatiotemporal domain, thereby improving the accuracy and inference speed of three-dimensional human pose estimation and meeting the need to quickly obtain accurate three-dimensional human pose estimation results.

[0005] The technical solution of this invention is: a three-dimensional human pose estimation method that integrates multi-scale spatiotemporal features.

[0006] Step 1: Based on the human body topology, perform pose recognition on the two-dimensional pose sequences in the input human motion video dataset, generate three scales describing human pose: joints, limbs, and upper / lower body, and map them into high-dimensional vectors of different sizes.

[0007] Step 2: Construct a spatial multi-scale feature extraction module based on Transformer, and use multi-head attention to extract joint points, limbs and upper / lower body human posture features at three scales in parallel.

[0008] Step 3: By defining a spatial multi-scale feature fusion module that combines self-attention and cross-attention, calculate the query, key, and value matrices of the three features to obtain the three-dimensional human pose that incorporates spatial multi-scale features.

[0009] Step 4: By establishing a multi-scale temporal coding module, multi-scale temporal information is encoded between different spatial multi-scale feature frames after fusion, and finally a refined three-dimensional human pose is generated.

[0010] Step 1 is described in detail below:

[0011] Step 1.1: First, perform two-dimensional human pose estimation on the input human motion video dataset Γ to obtain the two-dimensional human joint point sequence {X} for each video frame. i ∈R 1×(J·2) |i=1,2,...,F}, where F is the number of input frames, J is the number of key points in each video frame, and R represents the real number field.

[0012] Step 1.2: Then, based on the human body topology, the two-dimensional human joint point sequence of each video frame is described as three different scale sets, including: joint point scale description. Limb Scale Description in Upper / lower body size description in Among them, M1 represents the buttocks, M2 the right buttocks, M3 the right knee, M4 the right ankle, M5 the left buttocks, M6 the left knee, M7 the left ankle, M8 the spine, M9 the chest, and M... 10 For the neck, M 11 For the head, M 12 For the left shoulder, M 13 For the left elbow, M 14 For the left wrist, M 15 For the right shoulder, M 16 For the right elbow, M 17 Right wrist;

[0013] Step 1.3 Finally, the three scale descriptions are linearly projected to a high-dimensional space, and an embedding matrix with spatial location information is added: t = 1, 2, 3, resulting in three high-dimensional vectors, representing the key points. limbs upper / lower body Where Linear represents linear projection, E spos The matrix represents the spatial location embedding matrix, where c1, c2, and c3 are all projection dimensions, and c1 < c2 < c3.

[0014] Step 2 is described in detail below:

[0015] Step 2.1: First, separate the high-dimensional vectors of joints, limbs, and upper / lower body in the temporal domain for each frame to obtain the pose feature matrix of each frame at three scales: i = 1, 2, ..., F;

[0016] Step 2.2: Then, using Transformer, a multi-scale feature extraction module containing three self-attention heads and a feedforward neural network is constructed. The self-attention operation is defined as... Where Q, K, and V are the query, key, and value matrices obtained by linear transformation of the feature matrix, respectively, Softmax is the normalization function, and c is the projection dimension.

[0017] Step 2.3 Finally, extract the joint, limb, and upper / lower body pose feature matrices for each frame. Parallel input is fed to three self-attention heads in the multi-scale feature extraction module to extract keypoint feature matrices at three scales. Limb feature matrix Upper / lower body feature matrix Then, a 1×1 convolutional layer is added after each self-attention head to unify the dimensions c1, c2, and c3 in the feature matrix to c.

[0018] Step 3 is described in detail below:

[0019] Step 3.1: First, extract the key feature matrix. Limb feature matrix Upper / lower body feature matrix Each matrix is ​​linearly projected to generate a new set of Q, K, and V matrices;

[0020] Step 3.2: Next, define a spatial multi-scale feature fusion module by combining self-attention and cross-attention mechanisms. Input the Q, K, and V generated by the three features into this module and calculate the bidirectional attention matrix between the associated features respectively.

[0021] The bidirectional attention matrix between key points and limbs is represented as follows: in W V ∈R c×c For a linear layer, d k This is the intermediate dimension of matrix multiplication, where k∈{1,2,3} represents three spatial scales, and l is the current spatial fusion layer number. Similarly, the bidirectional attention matrix between key points and upper / lower body, and the bidirectional attention matrix between limbs and upper / lower body can be represented as follows:

[0022] Step 3.3: Then, combine the three bidirectional attention matrices. The feature dimensions are recovered through linear layers, then input into the feedforward neural network in the Transformer, and after normalization, they are used as the input to the (l+1)th layer spatial multi-scale feature fusion module.

[0023] Step 3.4 Finally, repeat Step 3.1-Step 3.3 L times to obtain a three-dimensional human pose that incorporates multi-scale spatial features. i = 1, 2, ..., F, where L is a hyperparameter.

[0024] Step 4 is described in detail below:

[0025] Step 4.1: First, integrate the spatial multi-scale features of each frame of the 3D human pose. The numbers i = 1, 2, ..., F are concatenated and then an embedding matrix E containing temporal and positional information is added. tpos The input is fed into a multi-scale temporal coding module consisting of N multi-head self-attention layers, N frame fusion layers, and a temporal scale merging layer to obtain the temporal feature matrix Y. 0 ∈R F×(J+6+2)×c , where N is a hyperparameter that can be set by the user;

[0026] Step 4.2: Next, each multi-head self-attention layer and a frame fusion layer are combined into a stage of multi-scale temporal coding. In the first stage, Y... 0 Input a multi-head self-attention layer to calculate the similarity matrix between all frames at the current temporal scale to model global motion relationships. Input a frame fusion layer consisting of a temporal convolution with a kernel size of 3 and a stride of 2, and output the result. Repeat the above steps after each stage to obtain temporal multi-scale features, until the Nth stage output is: F is the number of input frames, c is the projection dimension, and J is the number of key points in each video frame.

[0027] Step 4.3: Then, the output of each stage is input into the temporal scale merging layer constructed based on the linear layer and upsampling, and the feature matrix of each temporal scale is reshaped into Y. n′ =UpSample k (Y n-1 W n , where Y n′ ∈R F×D n = 1, 2, ..., N, D is a unified temporal feature dimension, UpSample k This indicates that the upsampling rate is k=2. N-1 Upsampling,

[0028] Step 4.4 Finally, the outputs of each stage after time-series fusion are concatenated to obtain the final result. Then, a weighted average operation is performed in the time domain to reduce the number of output frames to 1. Finally, the image is passed through a fully connected layer with N·D neurons to obtain a refined 3D human pose.

[0029] The beneficial effects of this invention are:

[0030] 1. Most known methods extract human features only at a single scale, without considering the multi-level feature representation of a moving human body. This invention, by performing multi-scale representation of the human body in both the temporal and spatial domains, can obtain more accurate 3D human pose estimation results.

[0031] 2. Existing methods mainly use complete residual blocks or Transformer blocks for human feature extraction, resulting in the extraction of excessive redundant keypoint information and significantly increasing network complexity, making it difficult for the neural network to converge and reducing inference speed. This invention utilizes the parallel characteristics of multi-head attention mechanisms to simultaneously extract human features at different scales, reducing network complexity while maintaining accuracy, and effectively improving the speed of 3D human pose estimation from video input.

[0032] 3. Existing methods primarily treat the joints of the human body in motion as a whole, failing to consider the uneven and discontinuous distribution of joints in the time-space domain during complex movements. This results in limited representation capabilities for complex action scenarios. This invention decomposes human motion into descriptions at different scales in the time and space domains and utilizes multi-scale feature fusion to obtain more accurate estimation results, demonstrating excellent fitting capabilities for complex movements. Attached Figure Description

[0033] Figure 1 This is a flowchart of a three-dimensional human pose estimation method that integrates multi-scale spatiotemporal features according to the present invention.

[0034] Figure 2This is a schematic diagram of the posture recognition proposed in this invention;

[0035] Figure 3 This invention defines human joint points and three human body scales.

[0036] Figure 4 This is a schematic diagram of spatial multi-scale feature extraction according to the present invention;

[0037] Figure 5 This is a schematic diagram of spatial multi-scale feature fusion according to the present invention;

[0038] Figure 6 This is a schematic diagram of the multi-scale timing coding of the present invention;

[0039] Figure 7 This is an example image showing the 3D pose estimation results of this invention on the Human3.6M dataset;

[0040] Figure 8 This is an example diagram showing the 3D pose estimation results of the present invention on Internet video. Detailed Implementation

[0041] Example 1: As Figures 1-6 As shown, a three-dimensional human pose estimation method that integrates multi-scale spatiotemporal features includes:

[0042] Step 1: Based on the human body topology, perform pose recognition on the two-dimensional pose sequences in the input human motion video dataset, generate three scales describing human pose: joints, limbs, and upper / lower body, and map them into high-dimensional vectors of different sizes.

[0043] Step 2: Construct a spatial multi-scale feature extraction module based on Transformer, and use multi-head attention to extract joint points, limbs and upper / lower body human posture features at three scales in parallel.

[0044] Step 3: By defining a spatial multi-scale feature fusion module that combines self-attention and cross-attention, calculate the query, key, and value matrices of the three features to obtain the three-dimensional human pose that incorporates spatial multi-scale features.

[0045] Step 4: By establishing a multi-scale temporal coding module, multi-scale temporal information is encoded between different spatial multi-scale feature frames after fusion, and finally a refined three-dimensional human pose is generated.

[0046] Furthermore, Step 1 can be configured as follows:

[0047] Step 1.1: First, perform two-dimensional human pose estimation on the input human motion video dataset Γ to obtain the two-dimensional human joint point sequence {X} for each video frame. i∈R 1×(J·2) |i=1,2,...,F}, where F is the number of input frames, J is the number of key points in each video frame, and R represents the real number field.

[0048] Step 1.2: Then, based on the human body topology, the two-dimensional human joint point sequence of each video frame is described as three different scale sets, including: joint point scale description. Limb Scale Description in Upper / lower body size description in Among them, M1 represents the buttocks, M2 the right buttocks, M3 the right knee, M4 the right ankle, M5 the left buttocks, M6 the left knee, M7 the left ankle, M8 the spine, M9 the chest, and M... 10 For the neck, M 11 For the head, M 12 For the left shoulder, M 13 For the left elbow, M 14 For the left wrist, M 15 For the right shoulder, M 16 For the right elbow, M 17 The right wrist.

[0049] Step 1.3 Finally, the three scale descriptions are linearly projected to a high-dimensional space, and an embedding matrix with spatial location information is added: t = 1, 2, 3, resulting in three high-dimensional vectors, representing the key points. limbs upper / lower body Where Linear represents linear projection, E spos The matrix represents the spatial location embedding matrix, where c1, c2, and c3 are all projection dimensions, and c1 < c2 < c3.

[0050] Furthermore, Step 2 can be configured as follows:

[0051] Step 2.1: First, separate the high-dimensional vectors of joints, limbs, and upper / lower body in the temporal domain for each frame to obtain the pose feature matrix of each frame at three scales: i = 1, 2, ..., F.

[0052] Step 2.2: Then, using Transformer, a multi-scale feature extraction module containing three self-attention heads and a feedforward neural network is constructed. The self-attention operation is defined as... Where Q, K, and V are the query, key, and value matrices obtained by linear transformation of the feature matrix, respectively, Softmax is the normalization function, and c is the projection dimension.

[0053] Step 2.3 Finally, extract the joint, limb, and upper / lower body pose feature matrices for each frame. Parallel input is fed to three self-attention heads in the multi-scale feature extraction module to extract keypoint feature matrices at three scales. Limb feature matrix Upper / lower body feature matrix Then, a 1×1 convolutional layer is added after each self-attention head to unify the dimensions c1, c2, and c3 in the feature matrix to c.

[0054] Furthermore, Step 3 can be configured as follows:

[0055] Step 3.1: First, extract the key feature matrix. Limb feature matrix Upper / lower body feature matrix Each matrix is ​​linearly projected to generate a new set of Q, K, and V matrices.

[0056] Step 3.2: Next, define a spatial multi-scale feature fusion module by combining self-attention and cross-attention mechanisms. Input the Q, K, and V generated by the three features into this module and calculate the bidirectional attention matrix between the associated features respectively.

[0057] The bidirectional attention matrix between key points and limbs is represented as follows: in W V ∈R c×c For a linear layer, d k This is the intermediate dimension of matrix multiplication, where k∈{1,2,3} represents three spatial scales, and l is the current spatial fusion layer number. Similarly, the bidirectional attention matrix between key points and upper / lower body, and the bidirectional attention matrix between limbs and upper / lower body can be represented as follows:

[0058] Step 3.3: Then, combine the three bidirectional attention matrices. The feature dimensions are recovered through linear layers, then input into the feedforward neural network in the Transformer, and after normalization, they are used as the input to the (l+1)th layer spatial multi-scale feature fusion module.

[0059] Step 3.4 Finally, repeat Step 3.1-Step 3.3 L times to obtain a three-dimensional human pose that incorporates multi-scale spatial features. i = 1, 2, ..., F, where L is a hyperparameter.

[0060] Furthermore, Step 4 can be configured as follows:

[0061] Step 4.1: First, integrate the spatial multi-scale features of each frame of the 3D human pose. The numbers i = 1, 2, ..., F are concatenated and then an embedding matrix E containing temporal and positional information is added. tpos The input is fed into a multi-scale temporal coding module consisting of N multi-head self-attention layers, N frame fusion layers, and a temporal scale merging layer to obtain the temporal feature matrix Y. 0 ∈R F×(J+6+2)×c , where N is a hyperparameter that can be set by the user.

[0062] Step 4.2: Next, each multi-head self-attention layer and a frame fusion layer are combined into a stage of multi-scale temporal coding. In the first stage, Y... 0 Input a multi-head self-attention layer to calculate the similarity matrix between all frames at the current temporal scale to model global motion relationships. Input a frame fusion layer consisting of a temporal convolution with a kernel size of 3 and a stride of 2, and output the result. Repeat the above steps after each stage to obtain temporal multi-scale features, until the Nth stage output is: F is the number of input frames, c is the projection dimension, and J is the number of key points in each video frame.

[0063] Step 4.3: Then, the output of each stage is input into the temporal scale merging layer constructed based on the linear layer and upsampling, and the feature matrix of each temporal scale is reshaped into Y. n′ =UpSample k (Y n-1 W n , where Y n′ ∈R F×D n = 1, 2, ..., N, D is a unified temporal feature dimension, UpSample k This indicates that the upsampling rate is k=2. N-1 Upsampling,

[0064] Step 4.4 Finally, the outputs of each stage after time-series fusion are concatenated to obtain the final result. Then, a weighted average operation is performed in the time domain to reduce the number of output frames to 1. Finally, the image is passed through a fully connected layer with N·D neurons to obtain a refined 3D human pose.

[0065] Example 2: As Figures 1-6 As shown, a three-dimensional human pose estimation method that integrates multi-scale spatiotemporal features includes:

[0066] Step 1, as follows Figure 2 As shown, a segment of video from the human motion video dataset Γ is input into the 2D pose estimator HRNet to obtain the 2D human joint sequence {X} for each video frame. i ∈R 1×(J·2) |i=1,2,...,F}, where F is the number of input frames, J is the number of key points in each video frame, and R represents the real number field.

[0067] Then, as Figure 3 As shown, based on the human body topology, the two-dimensional human joint point sequence of each video frame is described as a set of three different scales, including: joint point scale description. Limb Scale Description in Upper / lower body size description in Among them, M1 represents the buttocks, M2 the right buttocks, M3 the right knee, M4 the right ankle, M5 the left buttocks, M6 the left knee, M7 the left ankle, M8 the spine, M9 the chest, and M... 10 For the neck, M 11 For the head, M 12 For the left shoulder, M 13 For the left elbow, M 14 For the left wrist, M 15 For the right shoulder, M 16 For the right elbow, M 17 The right wrist.

[0068] Finally, the three scale descriptions are linearly projected to a high-dimensional space, and an embedding matrix with spatial location information is added: t = 1, 2, 3, resulting in three high-dimensional vectors, representing the key points. limbs upper / lower body Where Linear represents linear projection, E spos The matrix represents the spatial location embedding matrix, where c1, c2, and c3 are all projection dimensions, and c1 < c2 < c3.

[0069] After Step 1, a high-dimensional vector describing the three scales is obtained for each video frame. This example uses the public dataset Human3.6M as the human motion video dataset Γ, where subjects S1, S5, S6, S7, and S8 are used for training, and S9 and S11 are used for validation. The experiment is conducted using PyTorch.

[0070] Step 2: First, separate the high-dimensional vectors of joints, limbs, and upper / lower body in the temporal domain for each frame to obtain the pose feature matrix of each frame at three scales: i = 1, 2, ..., F.

[0071] Then, as Figure 4 As shown, a multi-scale feature extraction module containing three self-attention heads and a feedforward neural network is constructed using Transformer. The self-attention operation is defined as... Where Q, K, and V are the query, key, and value matrices obtained by linear transformation of the feature matrix, respectively, Softmax is the normalization function, and c is the projection dimension.

[0072] Finally, the joint, limb, and upper / lower body pose feature matrices of each frame are analyzed separately. Parallel input is fed to three self-attention heads in the multi-scale feature extraction module to extract keypoint feature matrices at three scales. Limb feature matrix Upper / lower body feature matrix Then, a 1×1 convolutional layer is added after each self-attention head to unify the dimensions c1, c2, and c3 in the feature matrix to c.

[0073] Step 3: First, extract the key feature matrix. Limb feature matrix Upper / lower body feature matrix Each matrix is ​​linearly projected to generate a new set of Q, K, and V matrices.

[0074] Secondly, such as Figure 5 As shown, a spatial multi-scale feature fusion module is defined by combining self-attention and cross-attention mechanisms. The Q, K, and V generated by the three features are input into this module to calculate the bidirectional attention matrix between the associated features.

[0075] The bidirectional attention matrix between key points and limbs is represented as follows: in W V ∈R c×c For a linear layer, d k This is the intermediate dimension of matrix multiplication, where k∈{1,2,3} represents three spatial scales, and l is the current spatial fusion layer number. Similarly, the bidirectional attention matrix between key points and upper / lower body, and the bidirectional attention matrix between limbs and upper / lower body can be represented as follows:

[0076] Then, the three bidirectional attention matrices The feature dimensions are recovered through linear layers, then input into the feedforward neural network in the Transformer, and after normalization, they are used as the input to the (l+1)th layer spatial multi-scale feature fusion module.

[0077] Finally, repeat the above process in Step 3 L times to obtain a three-dimensional human pose that incorporates multi-scale spatial features. i = 1, 2, ..., F, the result is as follows Figure 5 As shown.

[0078] Step 4, First, as Figure 6 As shown, each frame of the 3D human pose incorporates multi-scale spatial features. The numbers i = 1, 2, ..., F are concatenated and then an embedding matrix E containing temporal and positional information is added. tpos The input is fed into a multi-scale temporal coding module consisting of N multi-head self-attention layers, N frame fusion layers, and a temporal scale merging layer to obtain the temporal feature matrix Y. 0 ∈R F×(J+6+2)×c Where N is a hyperparameter that can be set by the user. In the method of this invention, taking 27 frames of input as an example, the hyperparameter N is set to 3, the learning rate is 0.0001 and decreases at a decay rate of 0.99 after each epoch, the batch size is 1024, and a total of 140 epochs are trained.

[0079] Secondly, each multi-head self-attention layer and a frame fusion layer are combined into a stage of multi-scale temporal coding. In the first stage, Y... 0 Input a multi-head self-attention layer to calculate the similarity matrix between all frames at the current temporal scale to model global motion relationships. Input a frame fusion layer consisting of a temporal convolution with a kernel size of 3 and a stride of 2, and output the result. Repeat the above steps after each stage to obtain temporal multi-scale features, until the Nth stage output is: F is the number of input frames, c is the projection dimension, and J is the number of key points in each video frame.

[0080] Then, the output of each stage is fed into a temporal scale merging layer constructed based on linear layers and upsampling, and the feature matrix of each temporal scale is reshaped into Y. n′ =UpSample k (Y n-1 W n , where Y n′ ∈R F×D n = 1, 2, ..., N, D is a unified temporal feature dimension, UpSample k This indicates that the upsampling rate is k=2. N-1 Upsampling,

[0081] Finally, by concatenating the outputs of each stage after temporal fusion, we obtain... Then, a weighted average operation is performed in the time domain to reduce the number of output frames to 1. Finally, the image is passed through a fully connected layer with N·D neurons to obtain a refined 3D human pose. See Figure 6 .

[0082] Figure 7 , Figure 8 The figure shows an example of the 3D pose estimation results of this invention on the Human3.6M dataset and Internet videos. As can be seen from the figure, the invention has relatively accurate pose estimation results for complex poses with occlusion, strong fitting ability for various poses, and robust estimation across datasets.

[0083] The method of this invention was compared with other known methods, such as Poseformer and VideoPose3D, on the Human3.6M dataset. The results are shown in Table 1, where MPJPE is the average position error per joint, reflecting the error between the estimated result and the true value. Its calculation method is as follows: Where j is the joint number, Y (gt) The MPJPE represents the true value. P-MPJPE is performed by first rigidly transforming the joints (translation, rotation, and scaling) to align them with the true value before performing MPJPE. MPJVE is the joint velocity error used to evaluate the smoothness of the video result. It is calculated by first taking the first derivative of the 3D pose sequence and then calculating MPJPE.

[0084] Table 1 Comparison of Method Performance

[0085] method MPJPE (mm) P-MPJPE (mm) MPJVE(mm) VideoPose3D 46.8 36.5 2.79 Poseformer 44.3 34.6 2.83 Method of the present invention 43.6 34.1 2.74

[0086] As can be seen from Table 1, the MPJPE, P-MPJPE, and MPJVE of the method of the present invention are all lower than those of the two known methods, that is, the accuracy of three-dimensional attitude estimation is higher.

[0087] As shown in Table 2, the efficiency of the method of the present invention is compared with that of known methods in terms of parameters, computational complexity (FLOPs), and frames per second (FPS).

[0088] Table 2 Comparison of Method Efficiency

[0089] method Parameters (M) FLOPs(M) FPS VideoPose3D 16.95 34 863 Poseformer 9.60 1358 269 Method of the present invention 16.32 649 687

[0090] In summary, compared with related known methods, the method of the present invention has lower estimation error and higher efficiency, and better fitting ability for complex actions.

[0091] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A 3D human body posture estimation method integrating multi-scale spatiotemporal features, characterized by: The steps include: Step 1: Perform posture recognition on the two-dimensional posture sequence in the input human motion video dataset based on the human body topology structure, generate three scales describing the human body posture: joints, limbs, and upper / lower body, and map them into high-dimensional vectors of different sizes; The three scales describing human posture at joints, limbs, and upper / lower body include: Joint point scale description Limb size description in Upper / Lower Body Dimensions Description in Among them, M1 is the hip, M2 is the right hip, M3 is the right knee, M4 is the right ankle, M5 is the left hip, M6 is the left knee, M7 is the left ankle, M8 is the spine, M9 is the chest, M 10 For the neck, M 11 For the head, M 12 For left shoulder, M 13 For the left elbow, M 14 For left wrist, M 15 For right shoulder, M 16 For the right elbow, M 17 For the right wrist; Step 2: Build a Transformer-based spatial multi-scale feature extraction module, and use multi-head attention to extract joints, limbs, and upper / lower body posture features at three scales in parallel; Step 3: By defining a spatial multi-scale feature fusion module that combines self-attention and cross-attention, the query, key, and value matrices of the three features are calculated to obtain a 3D human body posture that integrates spatial multi-scale features. Step 4: By establishing a multi-scale temporal coding module, multi-scale temporal information is encoded between the fused multi-scale feature frames of different spaces, and finally a refined three-dimensional human posture is generated.

2. The 3D human body posture estimation method integrating multi-scale spatiotemporal features according to claim 1, characterized in that: The specific process of Step 1 is as follows: Step 1.1: First, perform 2D human posture estimation on the input human motion video dataset Γ to obtain the 2D human joint point sequence {X i ∈R 1×(J·2) |i=1,2,...,F}, where F is the number of input frames, J is the number of joint points in each video frame, and R represents the real number domain; Step 1.2, then, the 2D human joint point sequence of each video frame is described as three different scale sets according to the human topological structure; Step 1.

3. Finally, linearly project the three scale descriptions into a high-dimensional space and add an embedding matrix with spatial position information: Get three high-dimensional vectors, namely: joint points Limbs Upper / Lower Body Among them, Linear represents linear projection, E spos represents the spatial position embedding matrix, c1, c 2、 c3 are all projection dimensions, and c1<c2<c3.

3. The 3D human body posture estimation method integrating multi-scale spatiotemporal features according to claim 1, characterized in that: The specific process of Step 2 is as follows: Step 2.

1. First, separate the high-dimensional vectors of joints, limbs, and upper / lower body in the time domain for each frame to obtain the posture feature matrix of each frame at three scales: F is the number of input frames, J is the number of joint points in each video frame, c1, c2, and c3 are all projection dimensions; Step 2.2, then use Transformer to build a multi-scale feature extraction module consisting of three self-attention heads and a feedforward neural network. The self-attention operation is defined as Among them, Q, K, and V are query, key, and value matrices obtained by linear transformation of the feature matrix, Softmax is the normalization function, and c is the projection dimension; Step 2.3, finally, the joint points, limbs, upper / lower body posture feature matrices of each frame are The three self-attention heads in the multi-scale feature extraction module are input in parallel to extract the joint feature matrix at three scales. Limb feature matrix Upper / Lower Body Feature Matrix Then add a 1×1 convolution layer after each self-attention head to unify the dimensions c1, c2, and c3 in the feature matrix to c.

4. The 3D human body posture estimation method integrating multi-scale spatiotemporal features according to claim 1, characterized in that: The specific process of Step 3 is as follows: Step 3.

1. First, transform the joint feature matrix Limb feature matrix Upper / Lower Body Feature Matrix A new set of Q, K, and V matrices are generated respectively through linear projection; J is the number of joint points in each video frame, and c is the projection dimension; Step 3.2, secondly, define a spatial multi-scale feature fusion module by combining self-attention and cross-attention mechanisms, input Q, K, and V generated by the three features into the module, and calculate the bidirectional attention matrix between the associated features respectively; The bidirectional attention matrix of joints and limbs is expressed as: in W V ∈R c×c is a linear layer, d k is the intermediate dimension of matrix multiplication, k∈{1,2,3} represents three spatial scales, and l is the number of current spatial fusion layers; similarly, the bidirectional attention matrix between joints and upper / lower body, and the bidirectional attention matrix between limbs and upper / lower body are expressed as Step 3.3, then, the three bidirectional attention matrices The feature dimensions are restored through the linear layer, and then input into the feedforward neural network in the Transformer. After normalization, they are used as the input of the l+1th layer spatial multi-scale feature fusion module. Step 3.

4. Finally, repeat Step 3.1-Step 3.3 L times to obtain a 3D human body posture that integrates spatial multi-scale features. F is the number of input frames and L is a hyperparameter.

5. The 3D human body posture estimation method integrating multi-scale spatiotemporal features according to claim 1, characterized in that: The specific process of Step 4 is as follows: Step 4.

1. First, each frame of 3D human body posture that integrates spatial multi-scale features Concatenate them together and add the embedding matrix E with timing and position information tpos , input to the multi-scale temporal coding module consisting of N multi-head self-attention layers, N frame fusion layers and a temporal scale merging layer, and obtain the temporal feature matrix Y 0 ∈R F ×(J+6+2)×c , where N is a hyperparameter, which can be set by yourself; Step 4.

2. Secondly, each multi-head self-attention layer and a frame fusion layer form a stage of multi-scale temporal coding. In the first stage, Y 0 Input the multi-head self-attention layer, calculate the similarity matrix between all frames at the current time series scale to model the global motion relationship and input the frame fusion layer composed of a temporal convolution with a convolution kernel size of 3 and a step size of 2, and the output is The above operations in this step are repeated once every stage to obtain the temporal multi-scale features until the output of the Nth stage is: F is the number of input frames, c is the projection dimension, and J is the number of joint points in each video frame; Step 4.3, then, the output of each stage is input into the time-series scale merging layer based on the linear layer and upsampling, and the feature matrix of each time-series scale is reshaped into Y n′ =UpSample k (Y n-1 )W n , where Y n′ ∈R F×D , n=1,2,...,N, D is the unified time series feature dimension, UpSample k Indicates that the upsampling rate is k = 2 N-1 Upsampling of Step 4.4, finally, the output of each stage after time series fusion is spliced ​​to obtain A weighted average operation is performed in the time domain to reduce the number of output frames to 1, and then a fully connected layer with N·D neurons is passed to finally obtain a refined 3D human posture.

Images