Video three-dimensional human pose estimation method and device based on discretized latent space

Abstract

Embodiments of the present application provide a video three-dimensional human pose estimation method and device based on a discretized latent space. The method comprises: obtaining a training data set; training a pre-constructed synchronization network according to the training data set; constructing a 3D human pose reconstruction network and training it based on 3D human pose sequences in the training data set; constructing a 3D human pose estimation network based on a sampler network, a discretized latent space in the trained 3D human pose reconstruction network, and a decoder, and training the sampler network in the 3D human pose estimation network according to 2D human pose sequences in the training data set; and performing three-dimensional human pose estimation according to the pre-trained 2D human pose estimation network and the 3D human pose estimation network. The technical solution of the embodiments of the present application can make full use of the connection between 2D human key points and 3D human key points, and ensure the accuracy of the network 3D human pose estimation result.

Description

Technical Field

[0001] This application relates to the field of human pose estimation technology, and more specifically, to a video 3D human pose estimation method and apparatus based on discretized latent space. Background Technology

[0002] With the rapid development of computer vision and deep learning technologies, 3D human pose estimation, as one of the core tasks for understanding and analyzing human behavior, has received widespread attention. It has broad application prospects in many fields such as smart fitness, virtual reality, motion capture, security monitoring, and human-computer interaction. The goal of 3D human pose estimation is to accurately capture the joint positions of the human body in three-dimensional space by analyzing the positions of key points in images or videos using algorithms, providing a reliable foundation for motion evaluation and interaction.

[0003] Current technical solutions often employ a pre-trained 2D human pose estimation network to extract 2D human poses from video images, and then use the 2D human pose sequence to estimate the 3D human pose sequence, i.e., the 2D-to-3D Lifting method. However, this approach heavily relies on the accuracy of the 2D human pose estimation network, meaning that estimation errors may originate more from the network's own errors than from the process of estimating 3D human poses from 2D poses.

[0004] Therefore, how to fully utilize the connection between 2D and 3D human body key points to ensure the accuracy of the network's 3D human body pose estimation results has become an urgent technical problem to be solved. Summary of the Invention

[0005] The embodiments of this application provide a video 3D human pose estimation method and apparatus based on discretized latent space, which can at least to a certain extent make full use of the relationship between 2D human key points and 3D human key points, and ensure the accuracy of the network 3D human pose estimation results.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to one aspect of the embodiments of this application, a video 3D human pose estimation method based on discretized latent space is provided, including:

[0008] Obtain a training dataset, which contains several training data pairs, including positive sample pairs and negative sample pairs, and each training data pair contains a 2D human pose sequence and a 3D human pose sequence.

[0009] The pre-built synchronization network is trained based on the training dataset so that the synchronization network learns the potential connections between 2D human pose sequences and 3D human pose sequences.

[0010] A 3D human pose reconstruction network is constructed and trained on the 3D human pose sequences in the training dataset. The 3D human pose reconstruction network includes an encoder, a discretized latent space, and a decoder.

[0011] A 3D human pose estimation network is constructed based on a sampler network, the discretized latent space in the trained 3D human pose reconstruction network, and the decoder. The sampler network in the 3D human pose estimation network is trained according to the 2D human pose sequence in the training dataset. During the training process, the auxiliary loss is calculated through the synchronization network for weak supervision, so that the 3D human pose estimation network can estimate the corresponding 3D human pose based on the input 2D human pose sequence.

[0012] Three-dimensional human pose estimation is performed based on pre-trained 2D and 3D human pose estimation networks.

[0013] According to one aspect of the embodiments of this application, a video 3D human pose estimation device based on a discretized latent space is provided, comprising:

[0014] The acquisition module is used to acquire a training dataset, which contains several training data pairs, including positive sample pairs and negative sample pairs, and each training data pair contains a 2D human pose sequence and a 3D human pose sequence.

[0015] The first training module is used to train a pre-constructed synchronous network based on the training dataset, so that the synchronous network learns the potential relationship between 2D human pose sequences and 3D human pose sequences.

[0016] The second training module is used to construct a 3D human pose reconstruction network and train it based on the 3D human pose sequences in the training dataset. The 3D human pose reconstruction network includes an encoder, a discretized latent space, and a decoder.

[0017] The third training module is used to construct a 3D human pose estimation network based on a sampler network, the discretized latent space in the trained 3D human pose reconstruction network, and the decoder. The sampler network in the 3D human pose estimation network is trained according to the 2D human pose sequence in the training dataset. During the training process, the auxiliary loss is calculated through the synchronization network for weak supervision, so that the 3D human pose estimation network can estimate the corresponding 3D human pose based on the input 2D human pose sequence.

[0018] The processing module is used to perform three-dimensional human pose estimation based on the pre-trained 2D human pose estimation network and 3D human pose estimation network.

[0019] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video 3D human pose estimation method based on discretized latent space as described in the above embodiments.

[0020] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the video three-dimensional human pose estimation method based on discretized latent space as described in the above embodiments.

[0021] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video 3D human pose estimation method based on discretized latent space provided in the above embodiments.

[0022] In some embodiments of this application, the technical solutions involve: acquiring a training dataset containing several training data pairs, including positive and negative sample pairs, with each training data pair containing a 2D human pose sequence and a 3D human pose sequence; training a pre-constructed synchronous network based on the training dataset to learn the potential relationship between the 2D and 3D human pose sequences; constructing a 3D human pose reconstruction network and training it based on the 3D human pose sequences in the training dataset, the 3D human pose reconstruction network including an encoder, a discretized latent space, and a decoder; constructing a 3D human pose estimation network based on a sampler network, the discretized latent space in the trained 3D human pose reconstruction network, and the decoder; training the sampler network in the 3D human pose estimation network based on the 2D human pose sequences in the training dataset, and performing weak supervision by calculating an auxiliary loss through the synchronous network during training, so that the 3D human pose estimation network can estimate the corresponding 3D human pose based on the input 2D human pose sequence; and performing 3D human pose estimation based on the pre-trained 2D and 3D human pose estimation networks.

[0023] In this way, by synchronously learning the correlation between 2D and 3D human pose sequences through a network and injecting it into a 3D human pose estimation network, and by using a 3D human pose reconstruction network to reconstruct and train the 3D pose data, prior knowledge of the data can be extracted, thereby enabling more accurate estimation of 3D human pose and improving the accuracy of the estimation.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0026] Figure 1 A schematic diagram illustrating the construction of positive and negative sample pairs according to an embodiment of this application is shown;

[0027] Figure 2 A schematic diagram of the structure of a synchronization network according to an embodiment of this application is shown;

[0028] Figure 3 A schematic diagram of the structure of a 3D human pose reconstruction network according to an embodiment of this application is shown;

[0029] Figure 4 A schematic diagram of the structure of a 3D human pose estimation network according to an embodiment of this application is shown;

[0030] Figure 5 A block diagram of a video 3D human pose estimation device based on a discretized latent space according to an embodiment of this application is shown;

[0031] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0033] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] This application provides a video 3D human pose estimation method based on discretized latent space. This method can be applied to terminal devices or servers. The terminal devices may include, but are not limited to, one or more of smartphones, tablets, laptops, and desktop computers. The server may be a physical server or a cloud server.

[0037] In some embodiments of this application, the video 3D human pose estimation method based on discretized latent space includes the following steps:

[0038] S110. Obtain a training dataset, wherein the training dataset contains several training data pairs, the several training data pairs include positive sample pairs and negative sample pairs, and each training data pair contains a 2D human pose sequence and a 3D human pose sequence.

[0039] In one embodiment, obtaining the training dataset includes:

[0040] Preprocessing of video image sequences yields 2D human pose sequences and 3D human pose sequences.

[0041] Randomly misalign the 2D human pose sequence and the 3D human pose sequence to construct several training data pairs, which include positive sample pairs and negative sample pairs.

[0042] In this embodiment, the video image sequence may contain several video images. During preprocessing, video images that do not contain a person are filtered out and cropped to ensure that each video image contains only the full body of a person. The images are scaled to a size of 384*288, and then the data is enhanced, including 2D and 3D data enhancement. The 2D human keypoints are then flipped. In one example, there are 17 human keypoints: hip, right hip, right knee, right foot, left hip, left knee, left foot, spine, chest, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow, and right wrist.

[0043] Specifically, based on camera parameters and 3D data points (i.e., 3D human pose keypoints) in the world coordinate system, a coordinate transformation is first used to convert the 3D data points in the world coordinate system to 3D data points in the camera coordinate system, thus expanding the data. Then, based on the camera parameters, the 3D human pose keypoints are projected into 2D points, and subsequently converted to pixel coordinates of the image to obtain 2D human pose keypoints. According to the selected time window size, each data segment is cropped, dividing a video image sequence into several video segments with the same number of frames. At the selected video segment boundary points, excess frames are filled in; the figures in the filled frames are static. Finally, a reversal operation is performed, inverting the 2D and 3D data points, thereby doubling the data expansion to obtain the 2D and 3D human pose sequences.

[0044] Next, based on the preprocessed 2D and 3D human pose sequences, it is necessary to further enhance the connection between them and incorporate this connection into the subsequently trained network through a loss function. Specifically, 2D and 3D data with consistent timestamps can be considered as positive sample pairs, and then 3D data shifted by a certain time interval can be paired with 2D data with inconsistent timestamps to form negative sample pairs (e.g., ...). Figure 1 (As shown).

[0045] S120. Train the pre-constructed synchronization network according to the training dataset so that the synchronization network learns the potential relationship between 2D human pose sequences and 3D human pose sequences.

[0046] In this embodiment, after constructing the positive and negative sample pairs, the pre-constructed synchronization network can be trained based on them. Specifically, as follows: Figure 2As shown, in the synchronization network, 6 layers of Transformer Blocks are used to encode the 2D human pose sequence and the 3D human pose sequence, respectively, and both are encoded into feature vectors of the same dimension (i.e., 2D pose features and 3D pose features). The similarity distance between the 2D pose features and the 3D pose features is measured by the contrastive loss. The purpose is to bring the similarity distance between positive sample pairs closer and increase the similarity distance between negative sample pairs further. Then, the gradient descent algorithm is executed to optimize the synchronization network.

[0047] In one example, key parameter settings during the training of the synchronous network include: 100 training epochs, using the Adam optimizer, and a learning rate of 1e-4. The loss function of the synchronous network is defined as follows:

[0048]

[0049] Among them, y n This represents whether a pair of samples is positive (1 for positive, 0 for negative). N is the total number of samples, and d is the cosine similarity distance between 2D and 3D pose features. The goal is to minimize the distance between positive feature pairs and maximize the distance between negative feature pairs. The definition of d is as follows:

[0050]

[0051] in, ∈ represents the features of the 2D human pose sequence and the 3D human pose sequence after being encoded by a synchronous network, respectively, and ∈ represents a very small number.

[0052] S130. Construct a 3D human pose reconstruction network and train it based on the 3D human pose sequences in the training dataset. The 3D human pose reconstruction network includes an encoder, a discretized latent space, and a decoder.

[0053] In this embodiment, those skilled in the art can pre-construct a 3D human pose reconstruction network. This 3D human pose reconstruction network is built based on a VQ-VAE network and includes an encoder, a discretized latent space, and a decoder. Using a VQ-VAE network, the network learns the distribution of 3D human poses in the training dataset, compresses the network features into a discretized latent space, and uses a multi-head self-attention structure to encode the 3D human pose sequence into multiple embedding vectors. Then, the embedding vector with the smallest similarity distance is selected in the discretized latent space and replaced, and then input into the decoder to complete the decoding.

[0054] like Figure 3As shown, B is the batch size, T is the sequence length, and N is the number of key points. During the training process of the 3D human pose reconstruction network, the encoder encodes the key point coordinates in the 3D human pose sequence into a 1024-dimensional feature vector through a linear layer. The temporal convolutional network (TCN) is used to analyze the temporal dimension of the feature vector output by the linear layer, and temporal positional information is added through positional encoding. Then, it is converted into an embedding vector through 6 layers of Transformer Block.

[0055] The discretized latent space is designed as an [n,d] matrix containing n d-dimensional embedding vectors. The embedding vector with the highest similarity to the embedding vector output by the encoder is queried in the discretized latent space to replace the embedding vector output by the encoder.

[0056] The decoder transforms the replaced embedding vectors into 1024-dimensional features. A temporal convolutional network is used to analyze the temporal dimension of the features, and temporal positional information is added through positional encoding. Then, six Transformer Block layers and linear layers are used to transform the features back into their original data dimensions to complete the decoding. It should be noted that since the operation of querying and replacing features in the discretized latent space cannot be backpropagated, the Stop Gradient technique is used to update the encoder's gradient. By comparing the differences between the input 3D human pose sequence and the network-estimated 3D human pose sequence, gradient descent is used to optimize the network parameters.

[0057] Specifically, during training, key parameter settings include: setting the total number of embedding vectors n in the discretized latent space to 256, the feature dimension d to 64, the number of Transformer Block layers in the encoder to 6, the number of Transformer Block layers in the decoder to 6, the number of iterations to 80, using the AdamW optimizer, and the learning rate to 4e-5.

[0058] The total loss of the network consists of three parts: action loss, vector quantization loss, and embedding regression loss, defined as follows, where α and β are weight coefficients:

[0059] L = L motion +αL vq +βL reg

[0060] Action loss L in the total loss function motion The definition is as follows:

[0061]

[0062] Among them, L w It is WMPJPE loss, Lt It is TC loss, L m It is MPJVE loss. These are their respective weighting coefficients.

[0063] WMPJPE loss L w The definition is as follows:

[0064]

[0065] Where N is the number of human body keypoints, W is the set weight of each joint, T is the number of frames in a sequence, and p i,j and gt i,j is the predicted and true values ​​of the 3D position corresponding to the i-th joint in the j-th frame. ||·||2 represents the calculation of the L2 norm, with the goal of making the network predict the 3D human pose as accurately as possible.

[0066] TC loss L t The definition is as follows:

[0067]

[0068] Where, p i,j and p i,j-1 It is the predicted value of the 3D position corresponding to the i-th joint in the j-th frame and the (j-1)-th frame. The goal is to make the difference between each frame not too large, so that the network prediction results are smoother.

[0069] MPJVE loss L m The definition is as follows:

[0070]

[0071] The goal of this loss function is to focus on the motion velocity of the joints, ensuring smoothness and continuity in the time dimension, so that the model can better learn the dynamic changes of the action sequence.

[0072] Vector quantization loss L in the total loss function vq The definition is as follows:

[0073] L vq =||sg[z e (x)]-e k ||2

[0074] Among them, z e (x) is a continuous feature vector generated by the encoder, e k It is a discrete code vector stored in the discretized latent space. sg[·] represents the stopping gradient operation, indicating that z will not be updated during backpropagation. e (x). The goal is to narrow the distance between the encoder output and the nearest code vector, while the code vector only updates itself.

[0075] Embedded regression loss L in the total loss function reg The definition is as follows:

[0076] L reg =||z e (x)-sg[e k ]||2

[0077] Among them, z e (x) is a continuous feature vector generated by the encoder, e k It is a discrete code vector stored in the discretized latent space. sg[·] represents the stopping gradient operation, indicating that e will not be updated during backpropagation. k The target constrains the encoder embedding vector to prevent it from straying far from the nearest feature vector.

[0078] S140. A 3D human pose estimation network is constructed based on a sampler network, the discretized latent space in the trained 3D human pose reconstruction network, and the decoder. The sampler network in the 3D human pose estimation network is trained according to the 2D human pose sequence in the training dataset. During the training process, the auxiliary loss is calculated through the synchronization network for weak supervision, so that the 3D human pose estimation network can estimate the corresponding 3D human pose based on the input 2D human pose sequence.

[0079] In this embodiment, based on the trained 3D human pose reconstruction network, its discretized latent space and decoder network are used, and a sampler network is trained to construct a 3D human pose estimation network. Figure 4 As shown, the sampler network encodes the 2D human pose sequences in the training dataset into feature vectors with the same dimension as the embedding vectors in the discretized latent space. It then selects the vector with the smallest similarity distance in the discretized latent space for replacement. After obtaining the replaced embedding vector, the decoder network is used to decode it to obtain the 3D human pose.

[0080] Specifically, during the training of the 3D human pose estimation network, only the sampler network is trained, while the discretized latent space of the 3D human pose reconstruction network and the parameters of the decoder are frozen. Furthermore, the loss value is calculated using the frozen synchronous network after training as an auxiliary loss, and the correlation between the 2D and 3D human pose sequences learned by the synchronous network is injected into the 3D human pose estimation network, thereby enhancing data connectivity.

[0081] Key parameter settings include: 100 iterations, using the AdamW optimizer, and a learning rate of 4e-5. The network's total loss consists of three parts: regression loss, action loss, and synchronization loss, defined as follows, where α and β are weight coefficients:

[0082] L = L reg +γL motion +δL sync

[0083] Regression loss L reg The definition is as follows:

[0084]

[0085] in, These are features generated by the sampler network, e k It is a discrete embedding vector stored in the discretized latent space. sg[·] represents the stopping gradient operation, indicating that e will not be updated during backpropagation. k The goal is to narrow the distance between the sampler output and the nearest code vector while updating only the sampler network.

[0086] motion loss L motion The definition is consistent with that in the 3D human pose VQ-VAE reconstruction network. These are weighting coefficients, which will not be elaborated further:

[0087]

[0088] Synchronous loss L sync The definition is as follows:

[0089]

[0090] in, These represent the features of the 2D human pose sequence and the 3D human pose data generated by the 3D human pose estimation network after being encoded by the synchronous network. Since the 3D human pose estimation network only includes positive sample targets during training, the second term in the loss function of the synchronous network is discarded. The purpose is to inject the correlation prior knowledge of the synchronous network into the 3D human pose estimation network.

[0091] In the actual training process of the 3D human pose estimation network, the 2D human pose sequence is regarded as a matrix of [B,T,N,2], where B is the batch size, T is the sequence length, N is the number of human keypoints, and the last dimension represents the 2D coordinates of the human keypoints in the image. The sampler network maps the data to high-dimensional features [B,T,N,512] through linear layers, where 512 represents the feature dimension. The Transformer Block is used to analyze the spatial and temporal dimensions of the data. Specifically, in spatial analysis, a 2D human pose sequence feature [B,T,N,512] is regarded as multiple single-frame keypoints [B*T,N,1024], aiming to learn the connections between body keypoints in each frame. In temporal analysis, a 2D human pose sequence feature [B,T,N,512] is regarded as the temporal positional changes of multiple human keypoints [B*N,T,512], aiming to learn the connections between keypoints between frames, such as... Figure 4 As shown in the dashed box.

[0092] S150. Perform three-dimensional human pose estimation based on the pre-trained 2D human pose estimation network and 3D human pose estimation network.

[0093] In this embodiment, when the 3D human pose estimation network is trained, when 3D human pose estimation is required, the input image can be processed into an image containing only one person by using the aforementioned preprocessing operation. Then, the trained 2D human pose estimation network is used to estimate the 2D human pose sequence of the person. The 2D human pose sequence is then input into the 3D human pose estimation network so that it outputs the predicted 3D human pose estimate.

[0094] The following describes an apparatus embodiment of this application, which can be used to execute the video 3D human pose estimation method based on discretized latent space in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the above embodiments of the video 3D human pose estimation method based on discretized latent space of this application.

[0095] Figure 5 A block diagram of a video 3D human pose estimation device based on a discretized latent space according to an embodiment of this application is shown.

[0096] Reference Figure 5 As shown, a video 3D human pose estimation device based on discretized latent space according to an embodiment of this application includes:

[0097] The acquisition module is used to acquire a training dataset, which contains several training data pairs, including positive sample pairs and negative sample pairs, and each training data pair contains a 2D human pose sequence and a 3D human pose sequence.

[0098] The first training module is used to train a pre-constructed synchronous network based on the training dataset, so that the synchronous network learns the potential relationship between 2D human pose sequences and 3D human pose sequences.

[0099] The second training module is used to construct a 3D human pose reconstruction network and train it based on the 3D human pose sequences in the training dataset. The 3D human pose reconstruction network includes an encoder, a discretized latent space, and a decoder.

[0100] The third training module is used to construct a 3D human pose estimation network based on a sampler network, the discretized latent space in the trained 3D human pose reconstruction network, and the decoder. The sampler network in the 3D human pose estimation network is trained according to the 2D human pose sequence in the training dataset. During the training process, the auxiliary loss is calculated through the synchronization network for weak supervision, so that the 3D human pose estimation network can estimate the corresponding 3D human pose based on the input 2D human pose sequence.

[0101] The processing module is used to perform three-dimensional human pose estimation based on the pre-trained 2D human pose estimation network and 3D human pose estimation network.

[0102] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0103] It should be noted that, Figure 6 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 6As shown, the computer system includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from storage portion 608 into Random Access Memory (RAM) 603, such as performing the methods described in the above embodiments. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0105] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0106] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0107] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0110] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0111] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0112] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0113] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0114] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video 3D human pose estimation method based on discretized latent space, characterized in that, include: Obtain a training dataset, which contains several training data pairs, including positive sample pairs and negative sample pairs, and each training data pair contains a 2D human pose sequence and a 3D human pose sequence. The pre-built synchronization network is trained based on the training dataset so that the synchronization network learns the potential connections between 2D human pose sequences and 3D human pose sequences. A 3D human pose reconstruction network is constructed and trained on the 3D human pose sequences in the training dataset. The 3D human pose reconstruction network includes an encoder, a discretized latent space, and a decoder. A 3D human pose estimation network is constructed based on a sampler network, the discretized latent space in the trained 3D human pose reconstruction network, and the decoder. The sampler network in the 3D human pose estimation network is trained according to the 2D human pose sequence in the training dataset. During the training process, the auxiliary loss is calculated through the synchronization network for weak supervision, so that the 3D human pose estimation network can estimate the corresponding 3D human pose based on the input 2D human pose sequence. Three-dimensional human pose estimation is performed based on pre-trained 2D and 3D human pose estimation networks.

2. The method according to claim 1, characterized in that, During the training process of the 3D human pose reconstruction network, the encoder encodes the key point coordinates in the 3D human pose sequence into a 1024-dimensional feature vector through a linear layer. The temporal convolutional network is used to analyze the temporal dimension of the feature vector output by the linear layer, and temporal positional information is added through positional encoding. Then, it is converted into an embedding vector through 6 layers of Transformer Block. The discretized latent space is designed as an [n,d] matrix containing n d-dimensional embedding vectors. The embedding vector with the highest similarity to the embedding vector output by the encoder is queried in the discretized latent space to replace the embedding vector output by the encoder. The decoder converts the replaced embedding vector into 1024-dimensional features, uses a temporal convolutional network to analyze the temporal dimension of the features, adds temporal positional information through positional encoding, and then converts the features back into the original data dimension through 6 Transformer Blocks and linear layers to complete the decoding.

3. The method according to claim 2, characterized in that, The total loss function of the 3D human pose reconstruction network is calculated using the following formula: L=L motion +αL vq +βL reg Where α and β are coefficients; Action loss L in the total loss function motion The definition is as follows: L w It is WMPJPE loss, L t It is TC loss, L m It is MPJVE loss. These are weighting coefficients; The WMPJPE loss is defined as follows: N is the number of human body keypoints, W is the set weight of each joint, T is the number of frames in the input video sequence, and p i,j and gt i,j These are the predicted and actual values ​​of the 3D position corresponding to the i-th joint in the j-th frame, respectively, and ||·||2 indicates that the L2 norm is calculated. TC loss L t The definition is as follows: p i,j and p i,j-1 These are the predicted 3D positions of the i-th joint in the j-th and (j-1)-th frames, respectively. MPJVE loss L m The definition is as follows: Vector quantization loss L in the total loss function vq The definition is as follows: L vq =||sg[z e (x)]-e k ||2 z e (x) is a continuous feature vector generated by the encoder, e k It is a discrete code vector stored in the discretized latent space. sg[·] represents the stopping gradient operation, indicating that z will not be updated during backpropagation. e (x); Embedded regression loss L in the total loss function reg The definition is as follows: L reg =||z e (x)-sg[e k ]||2 Among them, z e (x) is a continuous feature vector generated by the encoder, e k It is a discrete code vector stored in the discretized latent space. sg[·] represents the stopping gradient operation, indicating that e will not be updated during backpropagation. k .

4. The method according to claim 1, characterized in that, During the training process of the 3D human pose estimation network, only the sampler network is trained so that it encodes the 2D human pose sequence into a feature vector with the same dimension as the embedding vector in the discretized latent space. The embedding vector with the smallest similarity distance is selected in the discretized latent space for replacement, and then the replaced embedding vector is decoded by the decoder to obtain the 3D human pose.

5. The method according to claim 4, characterized in that, The total loss of the 3D human pose estimation network is defined as follows: L=L reg +γL motion +δL sync Where γ and δ are coefficients, and the regression loss L reg The definition is as follows: in, These are features generated by the sampler network, e k It is a discrete embedding vector stored in the discretized latent space. sg[·] represents the stopping gradient operation, indicating that e will not be updated during backpropagation. k ; motion loss L motion The definition is as follows: Among them, L w It is WMPJPE loss, L t It is TC loss, L m It is MPJVE loss. These are weighting coefficients; Synchronous loss L sync The definition is as follows: in, These represent the features of the 2D and 3D human pose data generated by the prediction network after being encoded by the synchronous network.

6. The method according to claim 1, characterized in that, Obtain the training dataset, including: Preprocessing of video image sequences yields 2D human pose sequences and 3D human pose sequences. Randomly misalign the 2D human pose sequence and the 3D human pose sequence to construct several training data pairs, which include positive sample pairs and negative sample pairs.

7. A video 3D human pose estimation device based on discretized latent space, characterized in that, include: The acquisition module is used to acquire a training dataset, which contains several training data pairs, including positive sample pairs and negative sample pairs, and each training data pair contains a 2D human pose sequence and a 3D human pose sequence. The first training module is used to train a pre-constructed synchronous network based on the training dataset, so that the synchronous network learns the potential relationship between 2D human pose sequences and 3D human pose sequences. The second training module is used to construct a 3D human pose reconstruction network and train it based on the 3D human pose sequences in the training dataset. The 3D human pose reconstruction network includes an encoder, a discretized latent space, and a decoder. The third training module is used to construct a 3D human pose estimation network based on a sampler network, the discretized latent space in the trained 3D human pose reconstruction network, and the decoder. The sampler network in the 3D human pose estimation network is trained according to the 2D human pose sequence in the training dataset. During the training process, the auxiliary loss is calculated through the synchronization network for weak supervision, so that the 3D human pose estimation network can estimate the corresponding 3D human pose based on the input 2D human pose sequence. The processing module is used to perform three-dimensional human pose estimation based on the pre-trained 2D human pose estimation network and 3D human pose estimation network.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video 3D human pose estimation method based on discretized latent space as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the video three-dimensional human pose estimation method based on discretized latent space as described in any one of claims 1 to 6.

Images