A 3D human pose estimation method, device, equipment and readable storage medium

Abstract

The application discloses a 3D human posture estimation method, device and equipment and a readable storage medium. The method comprises the following steps: extracting a 2D human posture sequence from each video frame; generating an index sequence corresponding to a 3D human posture by using a discrete diffusion model according to the 2D human posture sequence; searching each codebook vector from a pre-constructed feature vector codebook according to the index sequence; and decoding each codebook vector by using a 3D human posture sequence decoder to obtain a reconstructed 3D human posture sequence. The application realizes posture quantization, limits a search space in a range conforming to physical constraints, and improves the accuracy of 3D human posture estimation.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a 3D human pose estimation method, apparatus, device, and readable storage medium. Background Technology

[0002] Traditional human pose estimation methods use wearable sensors to collect human motion information, but due to cost and complexity, they have not been widely used. Deep learning methods estimate human pose from monocular images and videos, which does not require the human to wear sensors, making them more convenient. Furthermore, 3D (Three-Dimensional) human pose estimation can provide rich depth information and has a wider range of applications compared to 2D human pose estimation, thus leading to its widespread use.

[0003] 3D human pose estimation is divided into single-stage and two-stage methods. Single-stage methods directly predict the 3D coordinates of human keypoints from the input image, without needing to perform 2D (Two-Dimensional) human pose estimation and then upscaling to 3D space. Compared to single-stage methods, two-stage methods have an advantage in accuracy because current 2D human pose estimation has achieved excellent performance, and the error in the 2D human pose estimation stage can be reduced by utilizing temporal information.

[0004] The two-stage approach consists of two parts. The first part is a 2D human pose estimation network, which outputs a 2D human pose sequence. This network is typically composed of models that perform well in 2D human pose estimation tasks, such as cascaded pyramid networks and high-resolution networks. The second part is a 2D-to-3D upscaling network, which is performed by a 3D human pose estimation model, primarily upscaling the 2D data to 3D space. Currently, 3D human pose estimation models are typically based on attention mechanisms, graph convolutional neural networks, and diffusion models. Attention-based methods use self-attention to capture global joint motion features in human motion; graph convolutional neural network-based methods construct an adjacency matrix based on human topology and excel at learning local joint motion features; and diffusion model-based methods generate and aggregate multiple hypotheses to alleviate motion ambiguity.

[0005] Existing sequence-to-sequence 3D human pose estimation methods typically model human pose sequences in a continuous space. Due to the uncertainty and ambiguity of pose, such as self-occlusion or object occlusion, existing methods may generate poses that do not conform to physical constraints.

[0006] In summary, how to effectively address the problem that current 3D human pose estimation methods generate poses that do not conform to physical constraints due to the uncertainty and ambiguity of pose is an urgent issue that needs to be solved by those skilled in the art. Summary of the Invention

[0007] The purpose of this application is to provide a 3D human pose estimation method that realizes pose quantization, restricts the search space to a range that conforms to physical constraints, and improves the accuracy of 3D human pose estimation. Another purpose of this application is to provide a 3D human pose estimation device, equipment, and computer-readable storage medium.

[0008] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0009] A 3D human pose estimation method, comprising:

[0010] Extract 2D human pose sequences from each video frame;

[0011] A discrete diffusion model is used to generate an index sequence corresponding to a 3D human pose based on the 2D human pose sequence;

[0012] Retrieve each codebook vector from the pre-constructed feature vector codebook according to the index sequence;

[0013] The 3D human pose sequence decoder is used to decode each codebook vector to obtain the reconstructed 3D human pose sequence.

[0014] In one specific embodiment of this application, a discrete diffusion model is used to generate an index sequence corresponding to a 3D human pose based on the 2D human pose sequence, including:

[0015] The 2D human pose sequence feature extraction module is used to extract 2D human pose sequence features from the 2D human pose sequence, and the discrete diffusion model is used to generate an index sequence corresponding to the 3D human pose based on the 2D human pose sequence features.

[0016] In one specific embodiment of this application, a 2D human pose sequence feature extraction module is used to extract 2D human pose sequence features from the 2D human pose sequence, including:

[0017] The first spatiotemporal feature of spatiotemporal self-attention flow is extracted from the 2D human posture sequence using the 2D human posture sequence feature extraction module.

[0018] The second spatiotemporal features of the spatiotemporal graph convolutional neural network stream are extracted from the 2D human pose sequence using the 2D human pose sequence feature extraction module.

[0019] The first spatiotemporal feature and the second spatiotemporal feature are fused to obtain the 2D human posture sequence features.

[0020] In one specific embodiment of this application, the first spatiotemporal feature of spatiotemporal self-attention flow is extracted from the 2D human pose sequence using the 2D human pose sequence feature extraction module, including:

[0021] The 2D human pose sequence is mapped to a preset dimension using the first linear layer of the 2D human pose sequence feature extraction module to obtain a pose representation after pose embedding.

[0022] The pose representation after pose embedding is reshaped to obtain the first reshaped pose representation.

[0023] Spatial self-attention calculation is performed on the first reshaped pose representation to capture the relationship between joints in the first reshaped pose representation and obtain the spatial self-attention pose calculation result.

[0024] The spatial self-attention pose calculation result is reshaped in the time dimension to obtain a second reshaped pose representation.

[0025] The first spatiotemporal feature is obtained by capturing the relationship between frames in the second reshaped pose representation through a self-attention mechanism.

[0026] In one specific embodiment of this application, the second spatiotemporal feature of the spatiotemporal graph convolutional neural network stream is extracted from the 2D human pose sequence using the 2D human pose sequence feature extraction module, including:

[0027] Spatiotemporal graph convolution is performed on the first reshaped pose representation to obtain the spatial graph convolution result.

[0028] The spatial graph convolution calculation result is reshaped in the time dimension to obtain a third reshaped pose representation.

[0029] The second spatiotemporal feature is obtained by performing temporal graph convolution on the third reshaped pose representation.

[0030] In one specific embodiment of this application, the discrete diffusion model is used to generate an index sequence corresponding to a 3D human pose based on the features of the 2D human pose sequence, including:

[0031] Obtain the preset 3D mask sequence;

[0032] The 2D human pose sequence features and the 3D mask sequence are concatenated along the feature dimension to obtain the initial 3D human pose features.

[0033] The initial 3D human pose features are denoised using a pose denoiser to obtain the index sequence corresponding to the 3D human pose.

[0034] In one specific embodiment of this application, a pose denoiser is used to denoise the initial 3D human pose features to obtain an index sequence corresponding to the 3D human pose, including:

[0035] The spatiotemporal self-attention module in the pose denoiser is used to perform spatiotemporal self-attention calculation on the initial 3D human pose features to obtain the denoised 3D human pose features.

[0036] The second linear layer in the pose denoiser maps the feature dimension of the denoised 3D human pose feature to the dimension of the feature vector codebook, and normalizes the obtained mapped 3D human pose feature to obtain the probability distribution corresponding to the index sequence.

[0037] By taking the maximum value index of the probability distribution, the index sequence corresponding to the 3D human pose is obtained.

[0038] A 3D human pose estimation device, comprising:

[0039] The 2D human pose sequence extraction unit is used to extract 2D human pose sequences from each video frame.

[0040] An index sequence generation unit is used to generate an index sequence corresponding to a 3D human pose based on the 2D human pose sequence using a discrete diffusion model.

[0041] The codebook vector retrieval unit is used to retrieve each codebook vector from the pre-constructed feature vector codebook according to the index sequence;

[0042] The 3D human pose sequence reconstruction unit is used to decode each codebook vector using a 3D human pose sequence decoder to obtain the reconstructed 3D human pose sequence.

[0043] A 3D human pose estimation device, comprising:

[0044] Memory, used to store computer programs;

[0045] A processor is used to implement the steps of the 3D human pose estimation method as described above when executing the computer program.

[0046] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the 3D human pose estimation method as described above.

[0047] The 3D human pose estimation method provided in this application includes: extracting 2D human pose sequences from each video frame; generating an index sequence corresponding to the 3D human pose based on the 2D human pose sequence using a discrete diffusion model; retrieving each codebook vector from a pre-constructed feature vector codebook based on the index sequence; and decoding each codebook vector using a 3D human pose sequence decoder to obtain the reconstructed 3D human pose sequence.

[0048] As can be seen from the above technical solution, by using a discrete diffusion model to generate an index sequence corresponding to a 3D human pose based on a 2D human pose sequence, and retrieving each codebook vector from a pre-constructed feature vector codebook based on the index sequence, pose quantization is achieved. This transforms the continuous pose space generation problem into a discrete sequence generation problem, limiting the search space to a range that conforms to physical constraints and improving the accuracy of 3D human pose estimation. Compared to 3D human pose estimation methods that go from sequence to single frame, this application directly generates a 3D human pose sequence, avoiding redundant computation caused by repeated processing of overlapping sequences in the sequence-to-frame approach.

[0049] Accordingly, this application also provides a 3D human pose estimation device, equipment, and computer-readable storage medium corresponding to the above-mentioned 3D human pose estimation method, which have the above-mentioned technical effects, and will not be repeated here. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating one implementation of the 3D human pose estimation method in this application.

[0052] Figure 2 This is a flowchart illustrating another implementation of the 3D human pose estimation method in this application.

[0053] Figure 3 This is a schematic diagram of an attitude quantization process in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of an index sequence generation process in an embodiment of this application;

[0055] Figure 5 This is a structural block diagram of a 3D human pose estimation device according to an embodiment of this application;

[0056] Figure 6This is a structural block diagram of a 3D human pose estimation device according to an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of the specific structure of a 3D human pose estimation device provided in an embodiment of this application. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0060] See Figure 1 , Figure 1 This is a flowchart of an implementation of a 3D human pose estimation method in this application. The method may include the following steps.

[0061] S101: Extract 2D human pose sequences from each video frame.

[0062] Acquire video frames, which can be either a video segment or a set of images arranged chronologically. Extract 2D human pose sequences from each video frame. A 2D human pose sequence is a set of multiple 2D human poses estimated consecutively within a video segment or a set of images arranged chronologically.

[0063] S102: Use the discrete diffusion model to generate the index sequence corresponding to the 3D human pose based on the 2D human pose sequence.

[0064] The model is pre-trained to obtain a discrete diffusion model. After extracting 2D human pose sequences from each video frame, the discrete diffusion model is used to generate index sequences corresponding to 3D human poses based on the 2D human pose sequences.

[0065] S103: Retrieve each codebook vector from the pre-constructed feature vector codebook according to the index sequence.

[0066] A feature vector codebook is pre-constructed, containing codebook vectors corresponding to each 3D human pose feature vector. After generating an index sequence corresponding to the 3D human pose based on the 2D human pose sequence using a discrete diffusion model, each codebook vector is retrieved from the pre-constructed feature vector codebook according to the index sequence.

[0067] S104: Use a 3D human pose sequence decoder to decode each codebook vector to obtain the reconstructed 3D human pose sequence.

[0068] After retrieving each codebook vector from the pre-constructed feature vector codebook according to the index sequence, the 3D human pose sequence decoder is used to decode each codebook vector to obtain the reconstructed 3D human pose sequence.

[0069] As can be seen from the above technical solution, by using a discrete diffusion model to generate an index sequence corresponding to a 3D human pose based on a 2D human pose sequence, and retrieving each codebook vector from a pre-constructed feature vector codebook based on the index sequence, pose quantization is achieved. This transforms the continuous pose space generation problem into a discrete sequence generation problem, limiting the search space to a range that conforms to physical constraints and improving the accuracy of 3D human pose estimation. Compared to 3D human pose estimation methods that go from sequence to single frame, this application directly generates a 3D human pose sequence, avoiding redundant computation caused by repeated processing of overlapping sequences in the sequence-to-frame approach.

[0070] It should be noted that, based on the above embodiments, this application also provides corresponding improvement solutions. In subsequent embodiments, steps that are the same as or corresponding to those in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other. These improvements will not be elaborated upon in the following improved embodiments.

[0071] See Figure 2 , Figure 2 This is another implementation flowchart of the 3D human pose estimation method in the embodiments of this application. The method may include the following steps.

[0072] S201: Extract 2D human pose sequences from each video frame.

[0073] S202: Use the 2D human posture sequence feature extraction module to extract 2D human posture sequence features from the 2D human posture sequence, and use the discrete diffusion model to generate the index sequence corresponding to the 3D human posture based on the 2D human posture sequence features.

[0074] After extracting 2D human pose sequences from each video frame, a 2D human pose sequence feature extraction module is used to extract 2D human pose sequence features, such as joint coordinates, bone vectors, joint angles, motion velocity, acceleration, and motion trajectory. A discrete diffusion model is then used to generate an index sequence corresponding to the 3D human pose based on these 2D human pose sequence features. By generating the index sequence corresponding to the 3D human pose based on the 2D human pose sequence features, accurate generation of the index sequence corresponding to the 3D human pose is achieved.

[0075] See Figure 3 , Figure 3 This is a schematic diagram of an attitude quantization process in an embodiment of this application. The training of the discrete diffusion model includes forward diffusion and backward propagation. A uniform sampling method is used to sample a time step s from the range 0-S, with each time step having an equal probability of being sampled:

[0076] ;

[0077] Where S is the total time step of discrete diffusion.

[0078] Based on the sampled time step t, a forward diffusion process is performed on the index sequence, where the single-step transition probability of the forward diffusion of the index sequence corresponding to the 3D human pose is defined as follows:

[0079] ;

[0080] In the formula, c(·) is the one-hot encoding function, which converts the index into a one-hot vector. Let be the transition matrix of the discrete diffusion process. Let be the probability distribution of the index sequence at the current time step, where the transition matrix is ​​defined as follows:

[0081] ;

[0082] in, The probability that the index remains unchanged. This represents the probability that an index will be replaced by another index (each index has the same probability). The probability that the index becomes a mask.

[0083] Backpropagation optimizes the discrete diffusion model by minimizing the variational lower bound loss, where the loss is defined as follows:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] in, To minimize the variational lower bound loss, For the s-th step of the reverse denoising process, the probability distribution of the index sequence is... For noise reduction conditions, For the parameters of the model, As a prior distribution, Let KL divergence function be used. Let S be the error at step S.

[0089] In one specific embodiment of this application, extracting 2D human pose sequence features from a 2D human pose sequence using a 2D human pose sequence feature extraction module may include the following steps:

[0090] Step 1: Use the 2D human pose sequence feature extraction module to extract the first spatiotemporal feature of spatiotemporal self-attention flow from the 2D human pose sequence;

[0091] Step 2: Use the 2D human pose sequence feature extraction module to extract the second spatiotemporal features of the spatiotemporal graph convolutional neural network stream from the 2D human pose sequence;

[0092] Step 3: Perform feature fusion on the first and second spatiotemporal features to obtain 2D human pose sequence features.

[0093] For ease of description, the three steps above can be combined for explanation.

[0094] After extracting 2D human pose sequences from each video frame, a 2D human pose sequence feature extraction module is used to extract the first spatiotemporal feature of the spatiotemporal self-attention flow from the 2D human pose sequence. The same module then extracts the second spatiotemporal feature of the spatiotemporal graph convolutional neural network flow from the 2D human pose sequence. The first and second spatiotemporal features are then fused to obtain the 2D human pose sequence features. By fusing the first spatiotemporal feature of the spatiotemporal self-attention flow and the second spatiotemporal feature of the spatiotemporal graph convolutional neural network flow, accurate extraction of the 2D human pose sequence features is achieved.

[0095] The first and second spatiotemporal features can be fused using the following formula:

[0096] ;

[0097] in, Features of 2D human pose sequences As the first spatiotemporal feature, As a second spatiotemporal feature, Element-wise multiplication, also known as Hadamard product, is performed. α is a learnable weight with a value range of (0, 1). Adaptive fusion can better integrate the global and local features of the pose sequence to obtain richer 2D human pose sequence features.

[0098] In one specific embodiment of this application, extracting the first spatiotemporal feature of spatiotemporal self-attention flow from a 2D human pose sequence using a 2D human pose sequence feature extraction module may include the following steps:

[0099] Step 1: Use the first linear layer of the 2D human pose sequence feature extraction module to map the 2D human pose sequence to a preset dimension to obtain the pose representation after pose embedding.

[0100] Step 2: Perform a reshaping operation on the pose representation after pose embedding to obtain the first reshaped pose representation;

[0101] Step 3: Perform spatial self-attention calculation on the first reshaped pose representation to capture the relationship between joints in the first reshaped pose representation and obtain the spatial self-attention pose calculation result;

[0102] Step 4: Reshape the spatial self-attention pose calculation results in the time dimension to obtain the second reshaped pose representation;

[0103] Step 5: Capture the relationship between frames in the second reconstructed pose representation through a self-attention mechanism to obtain the first spatiotemporal features.

[0104] For ease of description, the above five steps can be combined for explanation.

[0105] After extracting the 2D human pose sequence from each video frame, the first linear layer of the 2D human pose sequence feature extraction module is used to extract the 2D human pose sequence. Mapping to a preset dimension (d-dimensional), the original joint coordinates are projected into a high-dimensional space through a learnable linear transformation to obtain a 2D pose embedding. Its shape is B×T×J×d (B represents the batch size, J represents the number of joints, human pose consists of 17 joints, T represents the number of frames, which is set to 243 during training, and d represents the feature dimension):

[0106] ;

[0107] in, It is a linear transformation function.

[0108] The pose representation after pose embedding is reshaped to obtain the first reshaped pose representation. The pose representation after pose embedding can be reshaped using the following formula:

[0109] ;

[0110] in, This represents the first reshaped posture. This is a reshaping function.

[0111] Spatial self-attention calculation is performed on the first reshaped pose representation to capture the relationships between joints in the first reshaped pose representation, resulting in a spatial self-attention pose calculation. The spatial self-attention calculation for the first reshaped pose representation can be performed using the following formula:

[0112] , ;

[0113] in, The spatial self-attention pose calculation result is represented by Q, K, and V, which are the three main matrices in the spatial attention mechanism. Q is the spatial attention query matrix, K is the spatial attention key matrix, and V is the spatial attention value matrix. Q, K, and V are obtained by linear transformation of the encoded pose features. The linear transformation is achieved by multiplying with the weight matrix. q, k, and v are the indices of the weight matrix, representing the respective weight matrices of Q, K, and V. is the dimension of the feature vector.

[0114] Spatiotemporal self-attention flow leverages the attention mechanism's ability to capture global dependencies, dynamically modeling global dependencies between human joints in the spatial dimension through a self-attention mechanism.

[0115] The spatial self-attention pose calculation results are reshaped in the time dimension to obtain a second reshaped pose representation. The reshaping operation of the spatial self-attention pose calculation results in the time dimension can be performed using the following formula:

[0116] ;

[0117] in, The second reshaped pose is represented by the shape BJ×T×d.

[0118] By capturing the frame-to-frame relationships in the second reconstructed pose representation using a self-attention mechanism, robust first spatiotemporal features are generated by incorporating rich spatiotemporal information. The following formula can be used to capture the frame-to-frame relationships in the second reconstructed pose representation using a self-attention mechanism:

[0119] , ;

[0120] in, For a learnable parameter matrix, , , These are the three main matrices in the self-attention mechanism. Represents the self-attention query matrix. Represents the self-attention key matrix, Representative value matrix, , , It is obtained by linear transformation of the encoded pose features, which is achieved by multiplying with the weight matrix.

[0121] In one specific embodiment of this application, extracting the second spatiotemporal features of the spatiotemporal graph convolutional neural network stream from a 2D human pose sequence using a 2D human pose sequence feature extraction module may include the following steps:

[0122] Step 1: Perform spatiotemporal graph convolution calculation on the first reshaped pose representation to obtain the spatial graph convolution calculation result;

[0123] Step 2: Reshape the spatial graph convolution calculation results in the time dimension to obtain the third reshaped pose representation;

[0124] Step 3: Perform temporal graph convolution calculation on the third reshaped pose representation to obtain the second spatiotemporal feature.

[0125] For ease of description, the three steps above can be combined for explanation.

[0126] A reshaping operation is performed on the pose representation after pose embedding to obtain the first reshaped pose representation. Next, spatiotemporal graph convolution is performed on the first reshaped pose representation to obtain the spatial graph convolution result. The spatiotemporal graph convolution can be performed on the first reshaped pose representation using the following formula:

[0127] ;

[0128] in, The result of spatial graph convolution calculation. For activation function, The adjacency matrix in graph convolution calculation. Let J be a spatial adjacency matrix of shape J×J. Joints that are physically connected in space are represented by positions 1, and joints that are not physically connected in space are represented by positions 0. The matrix represents the self-connection of joints. It has a shape of J×J, and all elements on the diagonal of the matrix are 1, representing the self-connection of joints. This is the degree matrix of a Graph Convolutional Network (GCN). Adjacency matrix Element; and This represents the learnable weight matrix in a graph convolutional neural network.

[0129] The spatial graph convolution calculation results are reshaped in the time dimension to obtain the third reshaped pose representation. The reshaping operation of the spatial graph convolution calculation results in the time dimension can be performed using the following formula:

[0130] ;

[0131] in, The third reshaped pose is represented by the shape BJ×T×d.

[0132] The second spatiotemporal feature is obtained by performing temporal map convolution on the third reshaped pose representation. The temporal map convolution on the third reshaped pose representation can be calculated using the following formula:

[0133] ;

[0134] in, This is a second spatiotemporal feature.

[0135] The difference between temporal graph convolution and spatial graph convolution lies in the temporal adjacency matrix. The shape is T×T, representing the connection relationship between the human pose in each frame and other frames in the time dimension. If two frames are adjacent in the time dimension, the corresponding position is 1, otherwise it is 0. The dependencies of human pose in each frame are captured by time-map convolution. and This represents the learnable weight matrix in a graph convolutional neural network.

[0136] The core of graph convolutional neural networks (GNNs) lies in updating feature representations through dependencies between local neighboring nodes. As a graph structure, the human body's topology allows GNNs to capture the dependencies of local joints. Unlike self-attention mechanisms that compute global dependencies, GNNs update node representations through an adjacency matrix. Nodes in GNNs can only rely on their neighboring nodes without considering the influence of distant, non-adjacent nodes. Therefore, GNNs are better at capturing local dependencies.

[0137] In one specific embodiment of this application, generating an index sequence corresponding to a 3D human pose based on the features of a 2D human pose sequence using a discrete diffusion model may include the following steps:

[0138] Step 1: Obtain the preset 3D mask sequence;

[0139] Step 2: Concatenate the 2D human pose sequence features with the 3D mask sequence along the feature dimension to obtain the initial 3D human pose features;

[0140] Step 3: Use a pose denoiser to denoise the initial 3D human pose features to obtain the index sequence corresponding to the 3D human pose.

[0141] For ease of description, the three steps above can be combined for explanation.

[0142] Obtain a preset 3D mask sequence, and concatenate the 2D human pose sequence features with the 3D mask sequence along the feature dimension to obtain the initial 3D human pose features. The concatenation of the 2D human pose sequence features with the 3D mask sequence along the feature dimension can be achieved using the following formula:

[0143] ;

[0144] in, For the preset 3D mask sequence, The first part is a 2D human pose sequence feature, and the second part is an initial 3D human pose feature with a shape of B×T×J×2d.

[0145] After conditional embedding, the initial 3D human pose features It contains information about the features of 2D human pose sequences.

[0146] A pose denoising function is used to denoise the initial 3D human pose features, obtaining the corresponding index sequence. The pose denoising function, implemented using stacked spatiotemporal self-attention modules and a linear layer, is then used to reconstruct the index sequence corresponding to the 3D human pose. By denoising the initial 3D human pose features using the pose denoising function, accurate determination of the index sequence corresponding to the 3D human pose is achieved.

[0147] In one specific embodiment of this application, the initial 3D human pose features are denoised using a pose denoiser to obtain the index sequence corresponding to the 3D human pose, which may include the following steps:

[0148] Step 1: Use the spatiotemporal self-attention module in the pose denoiser to perform spatiotemporal self-attention calculation on the initial 3D human pose features to obtain the denoised 3D human pose features.

[0149] Step 2: Use the second linear layer in the pose denoising denoising 3D human pose features to map the feature dimension of the feature vector codebook, and normalize the obtained mapped 3D human pose features to obtain the probability distribution corresponding to the index sequence.

[0150] Step 3: Take the maximum value index of the probability distribution to obtain the index sequence corresponding to the 3D human pose.

[0151] For ease of description, the three steps above can be combined for explanation.

[0152] The spatiotemporal self-attention module in the pose denoiser is used to perform spatiotemporal self-attention calculation on the initial 3D human pose features to obtain the denoised 3D human pose features. The spatiotemporal self-attention calculation on the initial 3D human pose features can be performed using the following formula:

[0153] ;

[0154] in, The 3D human posture features after noise reduction.

[0155] The second linear layer in the pose denoising denoising derrick maps the feature dimension of the denoised 3D human pose features to the dimension of the feature vector codebook, and normalizes the obtained mapped 3D human pose features to obtain the probability distribution corresponding to the index sequence.

[0156] The probability distribution corresponding to the index sequence can be calculated using the following formula:

[0157] ;

[0158] in, This represents the probability distribution corresponding to the index sequence. This is the activation function.

[0159] By taking the index of the maximum value of the probability distribution, the index sequence corresponding to the 3D human pose is obtained. Accurate determination of the index sequence corresponding to the 3D human pose is achieved by using the probability distribution to calculate the index sequence.

[0160] The index sequence corresponding to a 3D human pose can be calculated using the following formula:

[0161] ;

[0162] in, This is the index sequence corresponding to the 3D human body pose. This is the index function for finding the maximum value.

[0163] S203: Retrieve each codebook vector from the pre-constructed feature vector codebook according to the index sequence.

[0164] A pose quantization network is used to discretize and encode 3D human pose sequences, and a feature vector codebook is constructed. The pose quantization network includes a 3D human pose sequence encoder, a feature vector codebook, and a 3D human pose sequence decoder. The 3D human pose sequence encoder and decoder consists of stacked spatiotemporal self-attention modules, used to process the input 3D human pose sequence... Encoded as a codebook vector And reconstructing 3D human pose sequences from codebook vectors, using a self-attention mechanism, the calculation formula for which is as follows:

[0165] ;

[0166] ;

[0167] , ;

[0168] in, For linear layer functions, For the position encoding matrix, For a learnable parameter matrix, It represents 3D human posture features.

[0169] The following formula can be used to retrieve each codebook vector from the pre-constructed feature vector codebook based on the index sequence:

[0170] ;

[0171] in, This refers to 3D human pose features, i.e., the retrieved codebook vector. This is a codebook vector retrieval function.

[0172] See Figure 4 , Figure 4 This is a schematic diagram of an index sequence generation process in an embodiment of this application. In the pose quantization network, the feature vector codebook is an embedding matrix of dimension K×d, where K represents the number of codebook vectors in the feature vector codebook, and d corresponds to the feature vector dimension. Each codebook vector represents a potential feature representation. The continuous features output by the encoder are mapped to the closest codebook vector in the feature vector codebook through nearest neighbor search, compressing high-dimensional continuous features into a low-dimensional index sequence. This reduces data redundancy while implicitly learning the structural representation in 3D human pose data. The representation of the feature vector codebook is as follows:

[0173] , ;

[0174] The 3D human pose features are quantized using the feature vector codebook Z. First, the distance L between each 3D human pose feature vector and each codebook vector is calculated, where the distance calculation formula is as follows:

[0175] ;

[0176] After calculating the distance between the 3D human pose feature vector and the codebook vector, nearest neighbor search is used to complete the quantization. The formula for nearest neighbor search is as follows:

[0177] ;

[0178] in, This is a 3D human pose feature vector. For codebook vectors, This is the quantization result of the 3D human pose feature vector. This is an index operation to find the minimum value.

[0179] Since the minimum value process in quantization is non-differentiable, this embodiment employs gradient pass-through technology, which directly copies the decoder gradient to the encoder output during backpropagation, ensuring that the encoder / decoder and feature vector codebook can be optimized synchronously. The formula for gradient pass-through technology is as follows:

[0180] ;

[0181] ;

[0182] Here, sg stands for stop gradient.

[0183] S204: The 3D human pose sequence decoder is used to decode each codebook vector to obtain the reconstructed 3D human pose sequence.

[0184] After retrieving the codebook vectors from the pre-constructed feature vector codebook based on the index sequence, a 3D human pose sequence decoder is used to decode each codebook vector to obtain the reconstructed 3D human pose sequence. The reconstructed 3D human pose sequence can include the 3D coordinates of 17 joints of the human body. This solves the problem of environmental occlusion and human self-occlusion affecting the accuracy of 3D human pose estimation in the task, and incorporates the inherent discreteness of human pose data into the diffusion model to achieve more accurate 3D human pose estimation.

[0185] The codebook vectors can be decoded using the 3D human pose sequence decoder using the following formula:

[0186] ;

[0187] in, For the reconstructed 3D human pose sequence. For linear layer functions in attitude quantization networks, This is the attitude decoder function in the attitude quantization network.

[0188] Corresponding to the above method embodiments, this application also provides a 3D human pose estimation device. The 3D human pose estimation device described below can be referred to in correspondence with the 3D human pose estimation method described above.

[0189] See Figure 5 , Figure 5 This is a structural block diagram of a 3D human pose estimation device according to an embodiment of this application. The device may include:

[0190] The 2D human pose sequence extraction unit 51 is used to extract 2D human pose sequences from each video frame.

[0191] The index sequence generation unit 52 is used to generate an index sequence corresponding to a 3D human pose based on a 2D human pose sequence using a discrete diffusion model.

[0192] The codebook vector retrieval unit 53 is used to retrieve each codebook vector from the pre-constructed feature vector codebook according to the index sequence;

[0193] The 3D human pose sequence reconstruction unit 54 is used to decode each codebook vector using a 3D human pose sequence decoder to obtain the reconstructed 3D human pose sequence.

[0194] As can be seen from the above technical solution, by using a discrete diffusion model to generate an index sequence corresponding to a 3D human pose based on a 2D human pose sequence, and retrieving each codebook vector from a pre-constructed feature vector codebook based on the index sequence, pose quantization is achieved. This transforms the continuous pose space generation problem into a discrete sequence generation problem, limiting the search space to a range that conforms to physical constraints and improving the accuracy of 3D human pose estimation. Compared to 3D human pose estimation methods that go from sequence to single frame, this application directly generates a 3D human pose sequence, avoiding redundant computation caused by repeated processing of overlapping sequences in the sequence-to-frame approach.

[0195] In one specific embodiment of this application, the index sequence generation unit 52 is specifically used to extract 2D human posture sequence features from the 2D human posture sequence using the 2D human posture sequence feature extraction module, and to generate an index sequence corresponding to the 3D human posture based on the 2D human posture sequence features using a discrete diffusion model.

[0196] In one specific embodiment of this application, the index sequence generation unit 52 may include:

[0197] The first spatiotemporal feature extraction subunit is used to extract the first spatiotemporal features of spatiotemporal self-attention flow from the 2D human posture sequence using the 2D human posture sequence feature extraction module.

[0198] The second spatiotemporal feature extraction subunit is used to extract the second spatiotemporal features of the spatiotemporal graph convolutional neural network stream from the 2D human pose sequence using the 2D human pose sequence feature extraction module.

[0199] The feature fusion subunit is used to fuse the first spatiotemporal features and the second spatiotemporal features to obtain 2D human pose sequence features.

[0200] In one specific embodiment of this application, the first spatiotemporal feature extraction subunit is specifically used to map the 2D human pose sequence to a preset dimension using the first linear layer of the 2D human pose sequence feature extraction module to obtain a pose representation after pose embedding; to perform a reshaping operation on the pose representation after pose embedding to obtain a first reshaped pose representation; to perform spatial self-attention calculation on the first reshaped pose representation to capture the relationship between joints in the first reshaped pose representation to obtain a spatial self-attention pose calculation result; to perform a reshaping operation on the spatial self-attention pose calculation result in the time dimension to obtain a second reshaped pose representation; and to capture the relationship between frames in the second reshaped pose representation through a self-attention mechanism to obtain the first spatiotemporal feature.

[0201] In one specific embodiment of this application, the second spatiotemporal feature extraction subunit is specifically used to perform spatiotemporal graph convolution calculation on the first reshaped pose representation to obtain a spatial graph convolution calculation result; to perform a reshaping operation on the spatial graph convolution calculation result in the time dimension to obtain a third reshaped pose representation; and to perform time graph convolution calculation on the third reshaped pose representation to obtain a second spatiotemporal feature.

[0202] In one specific embodiment of this application, the index sequence generation unit 52 may include:

[0203] The 3D mask sequence acquisition subunit is used to acquire a preset 3D mask sequence.

[0204] The initial 3D human pose feature acquisition sub-unit is used to concatenate the 2D human pose sequence features and the 3D mask sequence along the feature dimension to obtain the initial 3D human pose features.

[0205] The index sequence obtains a sub-unit, which is used to denoise the initial 3D human pose features using a pose denoiser to obtain the index sequence corresponding to the 3D human pose.

[0206] In one specific embodiment of this application, the index sequence obtaining sub-unit is specifically used to perform spatiotemporal self-attention calculation on the initial 3D human pose features using the spatiotemporal self-attention module in the pose denoiser to obtain denoised 3D human pose features; the second linear layer in the pose denoiser maps the feature dimension of the denoised 3D human pose features to the dimension of the feature vector codebook, and normalizes the obtained mapped 3D human pose features to obtain the probability distribution corresponding to the index sequence; the maximum value index of the probability distribution is taken to obtain the index sequence corresponding to the 3D human pose.

[0207] For the method embodiments described above, see [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the 3D human pose estimation device provided in this application. The device may include:

[0208] Memory 332 is used to store computer programs;

[0209] The processor 322 is used to implement the steps of the 3D human pose estimation method of the above method embodiment when executing a computer program.

[0210] For details, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the specific structure of a 3D human pose estimation device provided in this embodiment. The 3D human pose estimation device can vary significantly due to differences in configuration or performance. It may include a processor (central processing unit, CPU) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the 3D human pose estimation device 301.

[0211] The 3D human pose estimation device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0212] The steps in the 3D human pose estimation method described above can be implemented by the structure of a 3D human pose estimation device.

[0213] Corresponding to the above method embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:

[0214] Extract 2D human pose sequences from each video frame; generate 3D human pose index sequences based on the 2D human pose sequences using a discrete diffusion model; retrieve each codebook vector from a pre-constructed feature vector codebook based on the index sequences; decode each codebook vector using a 3D human pose sequence decoder to obtain the reconstructed 3D human pose sequence.

[0215] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0216] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0217] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0218] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A 3D human pose estimation method, characterized in that, include: Extract 2D human pose sequences from each video frame; A discrete diffusion model is used to generate an index sequence corresponding to a 3D human pose based on the 2D human pose sequence; Retrieve each codebook vector from the pre-constructed feature vector codebook according to the index sequence; The 3D human pose sequence decoder is used to decode each codebook vector to obtain the reconstructed 3D human pose sequence. The process of generating an index sequence corresponding to a 3D human pose based on the 2D human pose sequence using a discrete diffusion model includes: The 2D human pose sequence feature extraction module is used to extract 2D human pose sequence features from the 2D human pose sequence, and the discrete diffusion model is used to generate an index sequence corresponding to the 3D human pose based on the 2D human pose sequence features. The discrete diffusion model is used to generate an index sequence corresponding to a 3D human pose based on the features of the 2D human pose sequence, including: Obtain the preset 3D mask sequence; The 2D human pose sequence features and the 3D mask sequence are concatenated along the feature dimension to obtain the initial 3D human pose features. The initial 3D human pose features are denoised using a pose denoiser to obtain the index sequence corresponding to the 3D human pose.

2. The 3D human pose estimation method according to claim 1, characterized in that, Extracting 2D human pose sequence features from the 2D human pose sequence using a 2D human pose sequence feature extraction module includes: The first spatiotemporal feature of spatiotemporal self-attention flow is extracted from the 2D human posture sequence using the 2D human posture sequence feature extraction module. The second spatiotemporal features of the spatiotemporal graph convolutional neural network stream are extracted from the 2D human pose sequence using the 2D human pose sequence feature extraction module. The first spatiotemporal feature and the second spatiotemporal feature are fused to obtain the 2D human posture sequence features.

3. The 3D human pose estimation method according to claim 2, characterized in that, The first spatiotemporal feature of spatiotemporal self-attention flow is extracted from the 2D human pose sequence using the 2D human pose sequence feature extraction module, including: The 2D human pose sequence is mapped to a preset dimension using the first linear layer of the 2D human pose sequence feature extraction module to obtain a pose representation after pose embedding. The pose representation after pose embedding is reshaped to obtain the first reshaped pose representation. Spatial self-attention calculation is performed on the first reshaped pose representation to capture the relationship between joints in the first reshaped pose representation and obtain the spatial self-attention pose calculation result. The spatial self-attention pose calculation result is reshaped in the time dimension to obtain a second reshaped pose representation. The first spatiotemporal feature is obtained by capturing the relationship between frames in the second reshaped pose representation through a self-attention mechanism.

4. The 3D human pose estimation method according to claim 3, characterized in that, The second spatiotemporal feature of the spatiotemporal graph convolutional neural network stream is extracted from the 2D human pose sequence using the 2D human pose sequence feature extraction module, including: Spatiotemporal graph convolution is performed on the first reshaped pose representation to obtain the spatial graph convolution result. The spatial graph convolution calculation result is reshaped in the time dimension to obtain a third reshaped pose representation. The second spatiotemporal feature is obtained by performing temporal graph convolution on the third reshaped pose representation.

5. The 3D human pose estimation method according to claim 1, characterized in that, The initial 3D human pose features are denoised using a pose denoiser to obtain an index sequence corresponding to the 3D human pose, including: The spatiotemporal self-attention module in the pose denoiser is used to perform spatiotemporal self-attention calculation on the initial 3D human pose features to obtain the denoised 3D human pose features. The second linear layer in the pose denoiser maps the feature dimension of the denoised 3D human pose feature to the dimension of the feature vector codebook, and normalizes the obtained mapped 3D human pose feature to obtain the probability distribution corresponding to the index sequence. By taking the maximum value index of the probability distribution, the index sequence corresponding to the 3D human pose is obtained.

6. A 3D human pose estimation device, characterized in that, include: The 2D human pose sequence extraction unit is used to extract 2D human pose sequences from each video frame. An index sequence generation unit is used to generate an index sequence corresponding to a 3D human pose based on the 2D human pose sequence using a discrete diffusion model. The codebook vector retrieval unit is used to retrieve each codebook vector from the pre-constructed feature vector codebook according to the index sequence; The 3D human pose sequence reconstruction unit is used to decode each codebook vector using a 3D human pose sequence decoder to obtain the reconstructed 3D human pose sequence. The index sequence generation unit is specifically used to extract 2D human posture sequence features from the 2D human posture sequence using the 2D human posture sequence feature extraction module, and to generate the index sequence corresponding to the 3D human posture based on the 2D human posture sequence features using the discrete diffusion model. The index sequence generation unit includes: The 3D mask sequence acquisition subunit is used to acquire a preset 3D mask sequence. The initial 3D human pose feature acquisition sub-unit is used to concatenate the 2D human pose sequence features and the 3D mask sequence along the feature dimension to obtain the initial 3D human pose features. The index sequence obtains a sub-unit, which is used to denoise the initial 3D human pose features using a pose denoiser to obtain the index sequence corresponding to the 3D human pose.

7. A 3D human pose estimation device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the 3D human pose estimation method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the 3D human pose estimation method as described in any one of claims 1 to 5.

Images