A real-time three-dimensional human pose estimation method and device oriented to space-time topology modeling

Abstract

The present application belongs to the field of computer vision, and relates to a real-time three-dimensional human pose estimation method and device oriented to space-time topology modeling. The method comprises: using a contrast learning paradigm for unlabeled pre-training to extract human topology structure prior information; using a light-weight space-time human topology extraction network as an encoder, using human topology structure prior information, and using a double-flow structure to respectively and parallelly extract human kinematic topology relations in a time domain and human geometric topology relations in a space domain; performing accumulation operation on the human kinematic topology relations in the time domain and the human geometric topology relations in the space domain, and obtaining positions of human three-dimensional poses through linear regression and outputting corresponding three-dimensional skeleton coordinates. The present application can accurately capture human structure and kinematic topology features, and can realize efficient and accurate three-dimensional human pose estimation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a real-time 3D human pose estimation method and device for spatiotemporal topology modeling, which is used for efficient and accurate 3D human pose estimation on human-computer interaction edge devices such as Jetson Nano. Background Technology

[0002] 3D human pose estimation, a challenging task in computer vision, has gradually become a core technology in applications such as human-computer interaction, intelligent monitoring, and virtual reality. This technology aims to accurately infer the 3D pose of the human body from 2D images or videos captured from various perspectives. Because it can provide detailed information on human movement and behavior, 3D human pose estimation plays a crucial role in augmented reality experiences, improving human-computer interaction efficiency, and promoting health monitoring. Accurate 3D human pose estimation not only greatly enriches the dimensions of human-computer interaction but also provides new avenues for realizing human-computer interaction, automated monitoring, and motion-sensing games. Especially in applications on edge devices, 3D human pose estimation technology can accurately analyze human pose in real-time in resource-constrained environments, which is crucial for real-time performance, energy efficiency, and user privacy protection. Therefore, developing lightweight 3D human pose estimation systems that can overcome hardware limitations and run in real-time on resource-constrained edge devices has become a key focus for researchers in this field.

[0003] 3D human pose estimation aims to accurately extract the position estimates of the human body's 3D pose (key points, such as the head, left hand, right foot, etc.) from image or video data and output the corresponding 3D skeleton coordinates. Thanks to current robust and efficient 2D human pose estimation algorithms, the two-stage method has been widely used since its emergence. It first uses an existing efficient 2D pose estimator to estimate 2D coordinates from the image, and then uses a 2D-to-3D boosting network to output the 3D coordinates. Extensive experiments have demonstrated that the two-stage method can obtain more accurate joint coordinate estimates using a lighter network. These estimates can then be used as input for other human-computer interaction tasks such as behavior understanding and human body analysis, thereby effectively improving the accuracy and effectiveness of human-computer interaction. The human body, as a dynamic and complex biomechanical system, has a topological structure that encompasses not only the spatial arrangement of joints and limbs but also their interconnections and motion dependencies. Specifically, human topological understanding involves deconstructing the human body as a whole and the geometric relationships and motion constraints between its components. In 3D space, these topological properties allow us to create mathematical models that can predict the accurate position of joints at a given time point and the changes in joint position across different action sequences. In machine learning, and especially deep learning, this understanding enables researchers to design complex neural network architectures, such as Graph Convolutional Networks (GCNs), which learn the structured information of the human body and use it as the basis for estimating 3D pose. Furthermore, accurate human topological understanding is crucial for handling occlusion and blurring in visual data. When certain body parts are not visible in an image, a deep understanding of the human topology can infer the location of the invisible parts from information about other visible parts, thus achieving an accurate estimation of the complete human pose. This method is particularly useful in areas such as multi-person interaction, complex action recognition, and real-time motion capture. In summary, human topological understanding not only improves the accuracy and robustness of 3D human pose estimation but is also a key technology for realizing deeper human-computer interaction and human monitoring applications. With the improvement of computing power and the continuous optimization of algorithms, 3D human pose estimation methods based on human topological understanding are expected to play an increasingly important role in future intelligent systems. Summary of the Invention

[0004] This invention addresses the challenges of deploying human pose estimation algorithms and insufficient human topology extraction in resource-constrained edge devices such as the Jetson Nano. It proposes a lightweight 3D human pose estimation method oriented towards spatiotemporal topology modeling.

[0005] The technical solution adopted in this invention is as follows:

[0006] A real-time 3D human pose estimation method based on spatiotemporal topology modeling includes the following steps:

[0007] 1) Use contrastive learning paradigm for unlabeled pre-training to extract prior information on human topology and provide the encoder with initialization parameters containing high-dimensional human features;

[0008] 2) Using a lightweight spatiotemporal human topology extraction network as an encoder, and utilizing the prior information of human topology extracted in step 1), a two-stream structure is used to extract the kinematic topology of the human body in the time domain and the geometric topology of the human body in the spatial domain in parallel.

[0009] 3) The human kinematic topology in the time domain and the human geometric topology in the spatial domain obtained in step 2) are accumulated, and the position of the human three-dimensional posture is obtained by linear regression and the corresponding three-dimensional skeleton coordinates are output.

[0010] Furthermore, step 1) involves a contrastive learning paradigm for 3D human pose estimation based on topological consistency across multiple viewpoints. This paradigm uses 2D poses of the same individual captured from different viewpoints at the same time as positive samples, and poses from different times or individuals as negative samples, to perform self-supervised learning, thereby capturing and optimizing the high-dimensional topological features of the human body structure. This method improves model performance and generalization ability by utilizing an enhanced memory architecture during the pre-training phase and applying a contrastive learning loss function to minimize the distance between entities with the same topological features and maximize the distance between entities with different topological features.

[0011] Furthermore, step 2) involves an efficient data mapping paradigm that significantly reduces computational complexity by efficiently encoding the temporal and spatial dimensions separately. This method optimizes the capture of human dynamics and structural features by processing temporal and spatial information independently, thereby improving data processing efficiency and the accuracy of pose estimation.

[0012] Further, step 2) includes:

[0013] 2.1) In the spatial domain, using the human body multi-layer topology extraction module, based on the two-dimensional human body posture, the structural information of the human body from joints to limbs to the whole body is obtained in a multi-channel progressive manner.

[0014] 2.2) In the time domain, using low-pass filtering in the frequency domain, based on the two-dimensional human posture, the long-term motion trend of the human body (global) is obtained and the relevant features are aggregated.

[0015] Furthermore, step 2.1) involves a sub-channel progressive residual framework, specifically divided into: a) a local joint link module; b) an intra-limb constraint module; and c) an inter-limb interaction module. These three modules work together on the input to capture the human body structure at different semantic levels, from joints to the entire body. The use of the sub-channel progressive residual framework enhances the model's understanding of the coordinated movements of different parts of the human body and helps capture complex topological relationships between joints. This spatial domain design method significantly improves the accuracy of human pose estimation, especially the estimation error of end-joint points with large degrees of freedom and complex movements.

[0016] Furthermore, step 2.2) involves a spectral feature extraction network that introduces a Discrete Cosine Transform (DCT) and a Low-pass Filter (LF) to effectively reduce redundancy and compress data in the video input in the frequency domain while preserving key motion trends and main poses.

[0017] A real-time 3D human pose estimation device for spatiotemporal topology modeling, comprising:

[0018] The prior information extraction module is used to extract prior information about the human body's topological structure by performing unlabeled pre-training using a contrastive learning paradigm.

[0019] The dual-stream information processing module uses a lightweight spatiotemporal human topology extraction network as an encoder. It utilizes prior information about human topology and employs a dual-stream structure to extract human kinematic topology in the time domain and human geometric topology in the spatial domain in parallel.

[0020] The 3D human pose estimation module is used to accumulate the human kinematic topology in the time domain and the human geometric topology in the spatial domain, and obtain the position of the human 3D pose through linear regression and output the corresponding 3D skeleton coordinates.

[0021] The beneficial effects of this invention are as follows:

[0022] This invention accurately captures human anatomy and kinematic topology features, achieving efficient and accurate 3D human pose estimation. In the spatial domain, the function and position of each joint are meticulously analyzed, and a hierarchical structure from local to global enhances the understanding of human pose. In the temporal domain, discrete cosine transform and low-pass filtering techniques are employed to reduce input data redundancy and improve the ability to capture dynamic pose changes. An efficient data mapping paradigm and a multi-channel progressive design optimize data processing in both spatial and temporal domains, significantly improving computational efficiency. Furthermore, this invention integrates a contrastive learning method, capturing consistent human features from different perspectives through self-supervised learning, effectively improving the model's generalization ability and accuracy. Attached Figure Description

[0023] Figure 1 The overall flowchart of the real-time 3D human pose estimation method based on spatiotemporal topology modeling of the present invention;

[0024] Figure 2 A schematic diagram illustrating the comparative learning pre-training method of this invention for capturing structured prior information about the human body. Detailed Implementation

[0025] To describe the method of the present invention more clearly and completely, the present invention will be further explained below with reference to specific embodiments and accompanying drawings.

[0026] refer to Figure 1 This invention provides a real-time 3D human pose estimation method based on spatiotemporal topology modeling. The method first utilizes a contrastive learning paradigm for label-free pre-training to extract prior information about the human body's topological structure. Then, a lightweight spatiotemporal human topology extraction network is used as the encoder. Utilizing the prior information about the human body's topology, a two-stream structure is employed to extract the kinematic topology in the time domain and the geometric topology in the spatial domain in parallel. Finally, the kinematic topology in the time domain and the geometric topology in the spatial domain are accumulated, and the position of the human body's 3D pose is obtained through linear regression, outputting the corresponding 3D skeleton coordinates. The main steps of this invention are described in detail below.

[0027] Step 1: Comparative learning pre-training to capture structured prior information about the human body

[0028] First, a two-dimensional pose is randomly selected from multiple perspectives as the input to encoder f, which encodes the input into a general high-dimensional human topological representation h. For example... Figure 2 As shown, different encoders f, Processing 2D poses from different cameras, we obtain high-dimensional human topological representations h, At this point, the individual poses from all viewpoints corresponding to the current pose are considered positive sample pairs and input into the homostructure encoder for momentum updates. Subsequently, the contrastive loss function (InfoNCE loss) is used to minimize the distance between high-dimensional representations of the same topology, while maximizing the distance between features of different topologies:

[0029]

[0030] Where q represents the feature vector, i.e., the high-dimensional human topological representation h, k + Indicates a positive sample. Let S represent negative samples, τ be the similarity function, τ be the temperature parameter (an adjustable hyperparameter) controlling the scale of the similarity scores, and K be the number of negative samples. It's important to note that positive samples are subsequently stored in a first-in, first-out memory database and will be used as negative samples in future iterations.

[0031] Step 2: Time Domain Information Processing

[0032] First, the time-domain signal is transformed to the frequency domain using discrete cosine transform, and then a low-pass filter is used to extract the low-frequency features that characterize the main motion trend (global):

[0033]

[0034] Where f represents the index of the frequency domain vector, c(f) is the normalization coefficient used to ensure orthogonality and conservation of variational energy, T represents the number of samples in the time domain, and X t This represents the time-domain feature input.

[0035] This step introduces discrete cosine transform and low-pass filter to extract low-frequency coefficients that are of primary concern with global motion trends and main poses, thereby reducing the number of model parameters and computational complexity.

[0036] Step 3: Spatial Domain Information Processing

[0037] This invention designs a multi-layer topology extraction module for the human body, employing a progressive channel segmentation design to learn the topological dependencies of the human body at the joint, limb, and body levels. The spatial domain input X is divided into three equal parts, each input to a corresponding level's processing module. Processing is then performed hierarchically, from joint to limb to body. The output of each level's processing module is accumulated with the input of the next level, enabling information interaction from local to global levels.

[0038] Step 3.1: Joint-level feature processing

[0039] This invention designs a local joint linking module that fits the connection information between human joints using an adjacency matrix and employs a graph convolutional neural network framework to capture information between joints.

[0040]

[0041]

[0042] Wherein, GCN represents Graph Convolutional Network. This represents two-dimensional human posture input. Represents the degree matrix of the diagonal nodes. Let represent the symmetric normalized adjacency matrix, and W represent the weight matrix of the science departments. σ represents the output of the three-dimensional human posture, and σ represents the Gaussian error linear unit.

[0043] Step 3.2: Processing of Limb Layer Features

[0044] This invention designs a limb-internal constraint module that utilizes parent joints with the same motion trend within the same limb as references to construct motion constraints on the distal joints of the limb, thereby reducing motion uncertainty. The core principle is to use a convolutional network to capture the motion features of the entire limb.

[0045]

[0046] Where σ represents the Gaussian error linear unit, Represents two-dimensional human pose input, Conv j This represents a one-dimensional convolutional layer. LN(·) represents the aggregated limb representation, LN(·) represents layer normalization, and MLP is a channel multilayer perceptron. The aggregated limb representation will be accumulated with the features of the end joints to achieve motion constraints.

[0047] Step 3.3: Processing of whole-body hierarchical features

[0048] This invention designs a limb interaction module that explores long-distance dependencies between human joints through a self-attention mechanism:

[0049]

[0050] Among them, H i C represents the index of the attention head, C′ represents the channel dimension, and MSA represents multi-head self-attention. This represents a two-dimensional attitude input, Concat represents the concatenation operation, and W... out This represents the output weight matrix. This represents the 3D pose output, where h represents the number of attention heads, and Q... s K s V s These are the query matrix, key matrix, and value matrix, respectively. Their main purpose is to obtain the association scores between the query vector and the key vector. These scores determine the degree of attention the attention mechanism needs to focus on at the current position. The score matrix is ​​then normalized to make the self-attention layer easier to train. Finally, the scores are converted into weights, and a channel multilayer perceptron is used to extract the global interaction relationships between limbs.

[0051] In real-world algorithm deployment experiments based on Jetson Nano, the method of this invention can achieve an inference speed of 112.76 FPS (frames per second), which fully meets the requirements for deployment and real-time inference (20-30 FPS). Moreover, the algorithm's performance can achieve an average joint position error of 43.5 mm, which is at a leading level among current algorithms.

[0052] Another embodiment of the present invention provides a real-time three-dimensional human pose estimation device for spatiotemporal topology modeling, comprising:

[0053] The prior information extraction module is used to extract prior information about the human body's topological structure by performing unlabeled pre-training using a contrastive learning paradigm.

[0054] The dual-stream information processing module uses a lightweight spatiotemporal human topology extraction network as an encoder. It utilizes prior information about human topology and employs a dual-stream structure to extract human kinematic topology in the time domain and human geometric topology in the spatial domain in parallel.

[0055] The 3D human pose estimation module is used to accumulate the human kinematic topology in the time domain and the human geometric topology in the spatial domain, and obtain the position of the human 3D pose through linear regression and output the corresponding 3D skeleton coordinates.

[0056] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0057] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0058] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0059] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A real-time 3D human pose estimation method oriented to spatio-temporal topology modeling, characterized in that, The method comprises the following steps: Pre-training without labels using a contrastive learning paradigm to extract human topological structure prior information; Taking a lightweight spatiotemporal human topological extraction network as an encoder, using human topological structure prior information, and adopting a double-flow structure to extract human kinematic topological relationships in the time domain and human geometric topological relationships in the space domain in parallel; Performing accumulation operation on the human kinematic topological relationships in the time domain and the human geometric topological relationships in the space domain, and obtaining the position of the human three-dimensional pose through linear regression and outputting the corresponding three-dimensional skeleton coordinates; The double-flow structure comprises: In the space domain, a human multi-layer topological extraction module is used to obtain the structural information of the human from the joint node to the limb to the whole body in a channel-by-channel progressive manner; In the time domain, the discrete cosine transform and the low-pass filter are used to extract the low-frequency features of the long-term motion trend of the human; The human multi-layer topological extraction module comprises a local joint linking module, a limb internal constraint module and a limb interconnection module, which respectively learn the human topological dependency at the joint, limb and body levels.

2. The method of claim 1, wherein, The contrastive learning paradigm takes the two-dimensional poses of the same individual captured by different perspectives at the same time as positive samples, and the poses of different times or individuals as negative samples, performs self-supervised learning to capture and optimize the high-dimensional topological features of the human structure, adopts a contrastive learning loss function to minimize the distance of the topological features with the same topology and maximize the distance of the topological features of different topologies, thereby improving the performance and generalization ability of the model.

3. The method of claim 1, wherein, The local joint linking module fits the connection information between the human joint nodes through an adjacency matrix and adopts a graph convolutional neural network to capture the information between the joint nodes: wherein, denotes a graph convolutional network, denotes a two-dimensional human pose input, denotes a diagonal node degree matrix, denotes a symmetric normalized adjacency matrix, denotes a weight matrix of a scientific department, denotes a three-dimensional human pose output, denotes a Gaussian error linear unit.

4. The method of claim 1, wherein, The limb internal constraint module uses the parent joint nodes with the same motion trend in the same limb as a reference to construct the motion constraint for the end joint nodes of the limb to reduce the uncertainty of the motion, and uses a convolutional network to capture the motion features of the whole limb: wherein, represents a Gaussian error linear unit, represents a two-dimensional human pose input, represents a one-dimensional convolutional layer, represents an aggregated limb representation, represents layer normalization, MLP is a channel multi-layer perceptron, and the aggregated limb representation is accumulated with the features of the end joint nodes to realize motion constraints.

5. The method of claim 1, wherein, The limb interconnection module explores the long-distance dependency relationship between the human joint nodes through a self-attention mechanism: wherein, denotes an index of attention head, denotes a channel dimension, denotes multi-head self-attention, denotes a two-dimensional pose input, denotes a concatenation operation, denotes an output weight matrix, denotes a three-dimensional pose output, denotes a number of attention heads, , , are a query matrix, a key matrix, and a value matrix, respectively, denotes layer normalization.

6. A real-time 3D human pose estimation apparatus for spatio-temporal topology modeling using the method of any one of claims 1-5. The method comprises the following steps: An prior information extraction module is configured to pre-train without labels using a contrastive learning paradigm to extract human topological structure prior information; A double-flow information processing module is configured to take a lightweight spatiotemporal human topological extraction network as an encoder, use human topological structure prior information, and adopt a double-flow structure to extract human kinematic topological relationships in the time domain and human geometric topological relationships in the space domain in parallel; A three-dimensional human pose estimation module is configured to perform accumulation operation on the human kinematic topological relationships in the time domain and the human geometric topological relationships in the space domain, and obtain the position of the human three-dimensional pose through linear regression and output the corresponding three-dimensional skeleton coordinates.

7. A computer device, comprising: The computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-5. The computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a computer to implement the method in any one of claims 1-5.

Images