Real-time three-dimensional human body posture estimation method and device for spatio-temporal topology modeling

Abstract

The invention belongs to the field of computer vision, and relates to a time-space topology modeling-oriented real-time three-dimensional human body posture estimation method and device. The method comprises the following steps: carrying out label-free pre-training by utilizing a comparative learning normal form, and extracting prior information of a human body topological structure; a lightweight space-time human body topology extraction network is used as an encoder, human body topology structure prior information is utilized, and a double-flow structure is adopted to extract a human body kinematics topological relation in a time domain and a human body geometry topological relation in a space domain in parallel; and carrying out accumulation operation on the human body kinematics topological relation in the time domain and the human body geometry topological relation in the space domain, obtaining the position of the human body three-dimensional posture through linear regression, and outputting corresponding three-dimensional skeleton coordinates. According to the method, the human body structure and kinematics topological characteristics can be accurately captured, and efficient and accurate three-dimensional human body posture estimation can be realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a real-time three-dimensional human posture estimation method and device for spatiotemporal topological modeling, which is used for efficient and accurate three-dimensional human posture estimation on human-computer interaction edge devices such as Jetson Nano. Background Art

[0002] 3D human pose estimation, as a challenging task in the field of computer vision, has gradually become a core technology in application fields such as human-computer interaction, intelligent monitoring, and virtual reality. This technology aims to accurately infer the 3D pose of the human body from 2D images or videos captured from various perspectives. Because it can provide detailed information about human motion and behavior, 3D human pose estimation plays a vital role in enhancing the reality experience, improving the efficiency of human-computer interaction, and promoting health monitoring. Accurate 3D human pose can not only greatly enrich the dimension of human-computer interaction, but also provide a new way to achieve human-computer interaction, automated monitoring, somatosensory games, etc. Especially in the application on edge devices, 3D human pose estimation technology can accurately analyze human pose in real time in resource-limited environments, which is crucial for real-time, energy efficiency, and protecting user privacy. Therefore, developing a lightweight 3D human pose estimation system that can overcome hardware limitations and run in real time on resource-limited edge devices has become the focus of researchers in this field.

[0003] 3D human pose estimation aims to accurately extract the position estimate of the 3D pose of the human body (key points, such as head, left hand, right foot, etc.) from images or video data, and output the corresponding 3D skeleton coordinates. Thanks to the current robust and efficient 2D human pose estimation algorithm, the two-stage method has been widely used since its emergence. It first uses the existing efficient 2D pose estimator to estimate the 2D coordinates from the image, and then uses a 2D to 3D lifting network to achieve the output of 3D coordinates. A large number of experiments have shown that the two-stage method can use a lighter network to obtain more accurate joint point coordinate estimates. These estimation results can be used as input for other human-computer interaction tasks such as behavior understanding and human body analysis, thereby effectively improving the accuracy and effectiveness of human-computer interaction. As a dynamic and complex biomechanical system, the topological structure of the human body not only includes the spatial arrangement of each joint and limb, but also includes their interconnection and motion dependencies. Specifically, human topological understanding involves deconstructing the geometric relationships and motion constraints between the human body as a whole and its components. In 3D space, these topological properties allow us to create mathematical models that can predict the exact position of the joints at a given time point and the changes in the joint positions in different action sequences. In the field of machine learning and deep learning in particular, this understanding has enabled researchers to design complex neural network architectures, such as graph convolutional networks (GCNs), which can learn the structured information of the human body and use it as the basis for estimating 3D pose. In addition, accurate human topology understanding is also important for dealing with occlusion and blur in visual data. When some parts of the body are not visible in the image, a deep understanding of human topology can infer the position of the invisible parts through the information of other visible parts, thereby achieving accurate estimation of the complete human pose. This approach is particularly useful in fields such as multi-person interaction, complex action recognition, and real-time motion capture. In short, human topology understanding not only improves the accuracy and robustness of 3D human pose estimation, but is also a key technology for achieving deeper human-computer interaction and human monitoring applications. With the improvement of computing power and the continuous optimization of algorithms, 3D human pose estimation methods based on human topology understanding are expected to play a more important role in future intelligent systems. Summary of the invention

[0004] In response to the problems of difficulty in deploying human posture estimation algorithms and insufficient human topology extraction in edge devices with limited computing resources represented by Jetson Nano, the present invention invents a lightweight three-dimensional human posture estimation method for spatiotemporal topological modeling.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A real-time 3D human posture estimation method for spatiotemporal topological modeling includes the following steps:

[0007] 1) Use contrastive learning to perform unlabeled pre-training to extract prior information about the human topology and provide the encoder with initialization parameters with high-dimensional feature information of the human body;

[0008] 2) Using a lightweight spatiotemporal human topology extraction network as an encoder, using the human topology structure prior information extracted in step 1), a dual-stream structure is used to extract the human kinematic topological relationship in the time domain and the human geometric topological relationship in the spatial domain in parallel;

[0009] 3) The human body kinematic topological relationship in the time domain and the human body geometric topological relationship in the space domain obtained in step 2) are accumulated, and the position of the human body's three-dimensional posture is obtained through linear regression and the corresponding three-dimensional skeleton coordinates are output.

[0010] Furthermore, step 1) involves a contrastive learning paradigm for 3D human pose estimation based on human topological consistency under multiple views. The contrastive learning paradigm uses the 2D poses of the same individual captured from different views at the same time as positive samples, and the poses of different times or individuals as negative samples, to perform self-supervised learning to capture and optimize the high-dimensional topological features of the human body structure. This method improves the performance and generalization ability of the model by utilizing an enhanced memory architecture in the pre-training stage and applying a contrastive learning loss function to minimize the distance of features with the same topological features and maximize the distance of features with different topological features.

[0011] Furthermore, step 2) involves an efficient data mapping paradigm that significantly reduces computational complexity by efficiently encoding the time and space dimensions separately. This approach optimizes the capture of human body dynamic and structural features by independently processing time and space information, thereby improving data processing efficiency and the accuracy of posture estimation.

[0012] Further, step 2) comprises:

[0013] 2.1) In the spatial domain, the human body multi-layer topology extraction module is used to obtain the structural information of the human body from joints to limbs to the whole body in a channel-by-channel progressive manner based on the two-dimensional human body posture;

[0014] 2.2) In the time domain, frequency domain low-pass filtering is used to obtain the long-term motion trend (global) of the human body based on the two-dimensional human posture and aggregate related features.

[0015] Furthermore, step 2.1) involves a split-channel progressive residual framework, which is specifically divided into: a) local joint connection module; b) intra-limb constraint module; c) inter-limb interaction module. The three modules work together on the input to capture the human body structure at different semantic levels from joints to the whole body. The use of the split-channel progressive residual framework enhances the model's understanding of the coordinated motion of various parts of the human body and helps capture the complex topological relationship between joints. This spatial domain design method significantly improves the accuracy of human posture estimation, especially the estimation error of end joints with large degrees of freedom and complex motion.

[0016] Furthermore, step 2.2) involves a spectral feature extraction network that introduces Discrete Cosine Transform (DCT) and Low-pass Filter (LF), which effectively reduces the redundancy in the video input and compresses the data in the frequency domain while retaining key motion trends and main postures.

[0017] A real-time three-dimensional human posture estimation device for spatiotemporal topological modeling, comprising:

[0018] A prior information extraction module is used to perform unlabeled pre-training using a contrastive learning paradigm to extract prior information about the human topological structure;

[0019] A dual-stream information processing module is used to use a lightweight spatiotemporal human topology extraction network as an encoder, utilize the prior information of human topology structure, and adopt a dual-stream structure to extract the human kinematic topological relationship in the time domain and the human geometric topological relationship in the space domain in parallel;

[0020] The 3D human posture estimation module is used to accumulate the human body kinematic topological relationship in the time domain and the human body geometric topological relationship in the space domain, obtain the position of the human body's 3D posture through linear regression and output the corresponding 3D skeleton coordinates.

[0021] The beneficial effects of the present invention are as follows:

[0022] The present invention accurately captures the human body structure and kinematic topological features, and realizes efficient and accurate three-dimensional human body posture estimation. In the spatial domain, the role and position of each joint are carefully analyzed, and the understanding of human body posture is enhanced through the local to global hierarchical structure. In the time domain, discrete cosine transform and low-pass filtering technology are used to reduce the redundancy of input data and improve the ability to capture dynamic posture changes. The efficient data mapping paradigm and channel progressive design optimize the data processing in the space and time domains and significantly improve the computational efficiency. In addition, the present invention integrates the contrastive learning method, and captures consistent human features from different perspectives through self-supervised learning, which effectively improves the generalization ability and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 . Overall flow chart of the real-time three-dimensional human posture estimation method for spatiotemporal topological modeling of the present invention;

[0024] Figure 2 .Schematic diagram of the contrastive learning pre-training of the present invention capturing structural prior information of the human body. DETAILED DESCRIPTION

[0025] In order to describe the method of the present invention more clearly and completely, the present invention will be further described below in conjunction with specific embodiments of the present invention and the accompanying drawings.

[0026] refer to Figure 1 The present invention provides a real-time 3D human posture estimation method for spatiotemporal topological modeling. The method first uses the contrastive learning paradigm to perform unlabeled pre-training to extract the prior information of the human topological structure; then, a lightweight spatiotemporal human topology extraction network is used as an encoder, and the prior information of the human topological structure is used to adopt a dual-stream structure to extract the human kinematic topological relationship in the time domain and the human geometric topological relationship in the space domain in parallel; then, the human kinematic topological relationship in the time domain and the human geometric topological relationship in the space domain are accumulated, and the position of the human 3D posture is obtained by linear regression and the corresponding 3D skeleton coordinates are output. The main steps of the method of the present invention are described in detail below.

[0027] Step 1: Contrastive learning pre-training captures human structural prior information

[0028] First, a 2D pose is randomly selected from multiple views as the input of the encoder f, which encodes the input into a general high-dimensional human topology representation h. Figure 2 As shown, different encoders f, Process the two-dimensional postures from different cameras to obtain high-dimensional human topological representations h, At this point, the individual poses of all views corresponding to the current pose moment are regarded as positive sample pairs and input into the momentum-updated encoder of the same structure. Subsequently, the contrast loss function (InfoNCE loss) is used to minimize the distance between high-dimensional representations representing the same topology, while maximizing the distance between different topological features:

[0029]

[0030] Among them, q represents the feature vector, that is, the high-dimensional human topological representation h, k + represents a positive sample, represents negative samples, S is the similarity function, τ is the temperature parameter (an adjustable hyperparameter) used to control the scale of the similarity score, and K represents the number of negative samples. It is worth noting that positive samples are then stored in a first-in-first-out memory bank and will be used as negative samples in future iterations.

[0031] Step 2: Time Domain Information Processing

[0032] First, the discrete cosine transform is used to transform the time domain signal into the frequency domain, and a low-pass filter is used to extract the low-frequency features that characterize the main motion trend (global):

[0033]

[0034] Where f represents the index of the frequency domain vector, c(f) is the normalization coefficient used to ensure orthogonality and conservation of varying energy, T represents the number of samples in the time domain, and X t Represents the time domain feature input.

[0035] This step introduces discrete cosine transform and low-pass filter to extract low-frequency coefficients that mainly focus on global motion trends and main postures, thereby reducing the number of model parameters and computational complexity.

[0036] Step 3: Spatial domain information processing

[0037] The present invention designs a human body multi-layer topology extraction module, which adopts a channel segmentation progressive design to learn the human body topology dependencies at the joint, limb and body levels. The spatial domain input X is divided into three equal parts of the same size, and input into the processing modules of the corresponding levels respectively, and then processed according to the joint-limb-body level. The output of each level processing module is accumulated with the input of the next level to realize the interaction of information from local to global.

[0038] Step 3.1: Joint point hierarchical feature processing

[0039] The present invention designs a local joint connection module, which fits the connection information between human joints through the adjacency matrix and uses the framework of graph convolutional neural network to capture the information between joints:

[0040]

[0041]

[0042] Among them, GCN represents graph convolutional network, represents a 2D human pose input, represents the diagonal node degree matrix, represents the symmetric normalized adjacency matrix, W represents the weight matrix of the science department, represents the 3D human posture output, and σ represents the Gaussian error linear unit.

[0043] Step 3.2: Limb-level feature processing

[0044] The present invention designs an intra-limb constraint module, which uses the parent joints with the same motion trend in the same limb as a reference to construct motion constraints for the end joints of the limb to reduce the uncertainty of motion. The core is to use a convolutional network to capture the motion characteristics of the entire limb:

[0045]

[0046] Where σ represents the Gaussian error linear unit, Represents a two-dimensional human posture input, Conv j represents a one-dimensional convolutional layer, represents the aggregated limb representation, LN(·) represents layer normalization, MLP is a channel multi-layer perceptron, and the aggregated limb representation is accumulated with the features of the terminal joints to realize motion constraints.

[0047] Step 3.3: Full body level feature processing

[0048] This paper designs a limb interaction module to explore the long-distance dependencies between human joints through the self-attention mechanism:

[0049]

[0050] Among them, H i represents the index of the attention head, C′ represents the channel dimension, MSA represents multi-head self-attention, represents the 2D posture input, Concat represents the concatenation operation, and W out represents the output weight matrix, represents the 3D pose output, h represents the number of attention heads, Q s , K s 、V s They are the query matrix, key matrix and value matrix. Their main purpose is to obtain the correlation scores between the query vector and the key vector. These scores determine the degree to which the attention mechanism needs to pay attention to the current position. Then the score matrix is ​​normalized to make the self-attention layer easier to train. Finally, the scores are converted into weights and the global interaction relationship between limbs is extracted through the channel multi-layer perceptron.

[0051] In the algorithm deployment experiment in a real-world scenario based on Jetson Nano, the method of the present invention can achieve an inference speed of 112.76 FPS (frames per second), which fully meets the requirements of deployment and real-time inference (20-30 FPS), and the performance of the algorithm can reach 43.5 mm in terms of the average joint position error index, reaching a leading level among current algorithms.

[0052] Another embodiment of the present invention provides a real-time three-dimensional human posture estimation device for spatiotemporal topological modeling, which includes:

[0053] A prior information extraction module is used to perform unlabeled pre-training using a contrastive learning paradigm to extract prior information about the human topological structure;

[0054] A dual-stream information processing module is used to use a lightweight spatiotemporal human topology extraction network as an encoder, utilize the prior information of human topology structure, and adopt a dual-stream structure to extract the human kinematic topological relationship in the time domain and the human geometric topological relationship in the space domain in parallel;

[0055] The 3D human posture estimation module is used to accumulate the human body kinematic topological relationship in the time domain and the human body geometric topological relationship in the space domain, obtain the position of the human body's 3D posture through linear regression and output the corresponding 3D skeleton coordinates.

[0056] The specific implementation process of each module refers to the above description of the method of the present invention.

[0057] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0058] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0059] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A real-time 3D human posture estimation method for spatiotemporal topological modeling, characterized in that: The following steps are involved: Use contrastive learning paradigm to perform unlabeled pre-training to extract prior information of human topological structure; A lightweight spatiotemporal human topology extraction network is used as the encoder. The human topology structure prior information is used to extract the human kinematic topology relationship in the time domain and the human geometric topology relationship in the space domain in parallel using a dual-stream structure. The human body kinematic topological relationship in the time domain and the human body geometric topological relationship in the space domain are accumulated, and the position of the human body's three-dimensional posture is obtained through linear regression and the corresponding three-dimensional skeleton coordinates are output.

2. The method according to claim 1, characterized in that The contrastive learning paradigm uses the two-dimensional posture of the same individual captured from different perspectives at the same time as positive samples, and the postures of different times or individuals as negative samples, and performs self-supervised learning to capture and optimize the high-dimensional topological features of the human body structure; the contrastive learning paradigm uses a contrastive learning loss function to minimize the distance of the same topological features and maximize the distance of different topological features, thereby improving the performance and generalization ability of the model.

3. The method according to claim 1, characterized in that The dual-stream structure is used to extract the human body kinematic topological relationship in the time domain and the human body geometric topological relationship in the space domain in parallel, including: In the spatial domain, the human body multi-layer topology extraction module is used to obtain the structural information of the human body from joints to limbs to the whole body in a channel-by-channel progressive manner. In the time domain, discrete cosine transform and low-pass filter are used to extract the low-frequency features of the long-term motion trend of the human body.

4. The method according to claim 3, characterized in that The human body multi-layer topology extraction module includes a local joint connection module, an intra-limb constraint module and an inter-limb interaction module, which respectively learn the human body topology dependencies at the joint, limb and body levels.

5. The method according to claim 4, characterized in that The local joint link module fits the connection information between human joints through the adjacency matrix and uses a graph convolutional neural network to capture the information between joints: Among them, GCN represents graph convolutional network, represents a 2D human pose input, represents the diagonal node degree matrix, represents the symmetric normalized adjacency matrix, W represents the weight matrix of the science department, represents the 3D human body posture output, σ represents the Gaussian error linear unit.

6. The method according to claim 4, characterized in that The intra-limb constraint module uses the parent joints with the same motion trend in the same limb as a reference to construct motion constraints for the end joints of the limb to reduce the uncertainty of motion, and uses a convolutional network to capture the motion characteristics of the entire limb: Where σ represents the Gaussian error linear unit, Represents a two-dimensional human posture input, Conv j represents a one-dimensional convolutional layer, represents the aggregated limb representation, LN(·) represents layer normalization, MLP is a channel multi-layer perceptron, and the aggregated limb representation is accumulated with the features of the terminal joints to realize motion constraints.

7. The method according to claim 4, characterized in that The limb interaction module explores the long-distance dependencies between human joints through the self-attention mechanism: Among them, H i represents the index of the attention head, C′ represents the channel dimension, MSA represents multi-head self-attention, represents the 2D posture input, Concat represents the concatenation operation, and W out represents the output weight matrix, represents the 3D pose output, h represents the number of attention heads, Q s , K s 、V s are the query matrix, key matrix and value matrix respectively, and LN(·) represents layer normalization.

8. A real-time three-dimensional human posture estimation device for spatiotemporal topological modeling, characterized in that: include: A prior information extraction module is used to perform unlabeled pre-training using a contrastive learning paradigm to extract prior information about the human topological structure; A dual-stream information processing module is used to use a lightweight spatiotemporal human topology extraction network as an encoder, utilize the prior information of human topology structure, and adopt a dual-stream structure to extract the human kinematic topological relationship in the time domain and the human geometric topological relationship in the space domain in parallel; The 3D human posture estimation module is used to accumulate the human body kinematic topological relationship in the time domain and the human body geometric topological relationship in the space domain, obtain the position of the human body's 3D posture through linear regression and output the corresponding 3D skeleton coordinates.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images