A three-dimensional human pose estimation method fusing color image and depth image

Abstract

The present application belongs to the technical field of three-dimensional human posture estimation, and in particular to a three-dimensional human posture estimation method fusing color images and depth images. The method comprises the following steps: step one, splitting an RGB-D dataset into a color image dataset and a depth image dataset, which are respectively taken as two inputs of a depth image processing module and a color image processing module; step two, performing image preprocessing and depth image voxelization processing on the input depth image; step three, sending the input color image into a two-dimensional human key point detection module, and extracting low-level features from the color image through a convolutional neural network. The present method detects human key points in the color image based on a Transformer network and outputs a key point heat map, and the network is improved by introducing relative position coding, thereby improving the performance of the posture estimation network and providing accurate data input for three-dimensional human posture estimation.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional human pose estimation technology, specifically a three-dimensional human pose estimation method that integrates color images and depth images. Background Technology

[0002] As a subfield of machine learning research, human pose estimation has developed rapidly in recent years and gradually entered the public eye. Unlike the manual labeling methods in traditional detection methods, human pose estimation based on deep learning models is a self-interpretive method that automatically extracts features from input information. Furthermore, the expressive power of the network model increases exponentially with the number of network stacks. Because deep learning models are relatively simple to operate and have a powerful ability to represent features, they have been widely applied in many research fields with particularly good results.

[0003] 3D human pose estimation involves extracting features from an input image and predicting the 3D spatial coordinates of human joints. It incorporates depth information estimation into color image-based human pose estimation, allowing for the localization of human joints in fields such as human behavior understanding, medical physiotherapy, motion analysis, intelligent monitoring, and human-computer interaction. Current models for predicting 3D human pose can be categorized based on the input data. A popular approach is to train models using video data for 3D human pose estimation, incorporating temporal image sequences as network input. This aims to address the inaccuracy caused by human occlusion in single monocular images. The method primarily involves adding temporal features during the prediction process. However, it's evident that this approach not only adds an extra dimension of information to the input but also leads to an exponential increase in data during network training. This single-minded pursuit of accuracy neglects the issue of training costs.

[0004] Another type of method mainly uses images as input for network training and 3D human pose estimation. There have been many successful studies on human pose estimation methods that only use color images as input. However, since color images reflect color and texture information, it is difficult to directly obtain human depth information from a single color image without the aid of labeling tools. Furthermore, color images are greatly affected by environmental changes such as lighting and shadows; in environments with strong or weak lighting, the estimation effect is significantly weakened, and pose estimation may even fail. Moreover, due to the large degrees of freedom of human joints, the problem of human occlusion in the image is also a major challenge for accurately estimating 3D human pose.

[0005] Similarly, many scholars are conducting algorithmic research on 3D human pose estimation using single depth images. This is because, compared to color images, depth images record the spatial distance information between the target object and the camera, are color-independent, and are unaffected by environmental factors such as changes in lighting, thus addressing some of the challenges encountered with color images. However, these methods generally use outdated models and rely on limited data, lacking human information, and cannot solve most existing problems, generally failing to achieve the accuracy of existing models.

[0006] Current mainstream methods based on the fusion of single color and depth images, such as Template-based Human Pose and Shape Estimation from a Single RGB-D Image, achieve parametric models of 2D and 3D key points by minimizing an objective function. This can be understood as using the color image to calculate 2D key points of the human body, and then using the depth image to formulaically adjust the coordinates to calculate the position of these key points in 3D space. However, this method has significant drawbacks: the calculation of 3D coordinates using formulas is highly limited, its generalization ability is weak when recognizing humans in different scenes, and training with traditional networks results in excessively large training data and errors due to the large number of convolutional layers. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a three-dimensional human pose estimation method that integrates color images and depth images, thus solving the problems mentioned in the background section.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0011] A method for 3D human pose estimation that integrates color and depth images, comprising the following steps:

[0012] Step 1: Split the RGB-D dataset into a color image dataset and a depth image dataset, which will serve as the two inputs for the depth image processing module and the color image processing module, respectively.

[0013] Step 2: Perform image preprocessing and depth image voxelization on the input depth image;

[0014] Step 3: The input color image is fed into the 2D human key point detection module, and the color image is processed by a convolutional neural network to extract low-level features.

[0015] Step 4: Fuse the voxel mesh information from Step 2 with the 2D keypoint heatmap information from Step 3 as the input to the 3D convolutional network;

[0016] Step 5: Input the information from Step 4 into the 3D keypoint prediction network. During the network processing, a series of 3D convolutional layers are used to extract features from the voxel grid and the 2D score map.

[0017] Furthermore, in step one, the RGB-D dataset is split into a color image dataset and a depth image dataset using the OpenCV library for training.

[0018] Furthermore, in step two, bilateral filtering is used to denoise the depth image, determine the voxel resolution and create a voxel space, and after traversing the entire depth image, the voxel grid of the entire depth image voxel block is obtained.

[0019] Furthermore, in step three, the two-dimensional human keypoint detection module consists of three parts: a CNN backbone network for extracting low-level image features; a Transformer encoder for capturing long-range spatial interactions between feature vectors across locations; and a head for outputting the predicted keypoint heatmap. Relative position encoding is added to the Transformer encoder, and after being added to the input, a new feature vector is obtained, which forces the shallow layers in the network to pay more attention to local information and helps the network to better handle positional relationships in sequence data.

[0020] Furthermore, in step four, the two-dimensional human body key point heatmap is tiled along the Z-axis, and the heatmap information is used as a scoring map. This is combined with the voxel grid to obtain a voxel grid containing the two-dimensional human body key point scoring map.

[0021] Furthermore, in step five, the 3D keypoint prediction network utilizes a 3D CNN network architecture. The network learns the spatial relationships and probability distributions of keypoints in the voxel grid and fractional maps, and finally outputs the 3D coordinates of human keypoints. Here, the idea of ​​dense connection is used to improve the 3D convolutional network, connecting all feature maps of each layer to the feature maps of the next layer, and the feature maps are added together using cascade operations.

[0022] (III) Beneficial Effects

[0023] Compared with existing technologies, this invention provides a three-dimensional human pose estimation method that integrates color images and depth images, which has the following beneficial effects:

[0024] This method uses a Transformer network to detect human key points in color images and outputs a key point heatmap. It introduces relative position encoding to improve the network, thereby enhancing the performance of the pose estimation network and providing accurate data input for 3D human pose estimation.

[0025] This method improves 3D CNN networks by reusing dense network feature information, which helps to alleviate the problem of inaccurate joint position prediction caused by the loss of information from the three-dimensional voxel blocks of the human body. Attached Figure Description

[0026] Figure 1 This is a flowchart of the present invention for three-dimensional human pose estimation by fusing depth images and color images;

[0027] Figure 2 This is a diagram of the Transformer encoder network structure in the two-dimensional key point heat map detection module of the present invention;

[0028] Figure 3 This is a diagram of the relative position encoding structure added to the attention layer of the Transformer encoder network in this invention;

[0029] Figure 4 This is a diagram of the three-dimensional convolutional network and the dense network structure introduced therein in this invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example

[0032] like Figure 1-4 As shown in the figure, an embodiment of the present invention proposes a three-dimensional human pose estimation method that fuses color images and depth images. The method includes the following steps:

[0033] Specifically, such as Figure 1 As shown, the steps are as follows, and should be performed sequentially:

[0034] Step 1: Split the RGB-D dataset into a color image dataset and a depth image dataset, which will serve as the two inputs for the depth image processing module and the color image processing module, respectively.

[0035] Specifically, the OpenCV library is used to read RGB-D image pairs from the dataset, and the color image and depth image are separated to obtain corresponding color and depth images;

[0036] Step 2: Perform image preprocessing and depth image voxelization on the input depth image.

[0037] Perform preliminary filtering on the depth image to remove noise;

[0038] Specifically, the depth image is read, and high-frequency and low-frequency noise in the depth image is processed by bilateral filtering. A weighted average is then performed by combining spatial distance and pixel value similarity to preserve edge information and achieve image smoothing.

[0039] Determine the voxel resolution and create a voxel space. After traversing the entire depth image, obtain a voxel mesh containing voxel blocks of the entire depth image.

[0040] Specifically, when voxelizing the processed depth image, considering computational resources and algorithm efficiency, a voxel grid size of 32×32×32 and a resolution of 3cm were chosen. Based on camera intrinsic parameters and depth values, each pixel in the depth image was converted into its corresponding 3D coordinates. This step used the following formula:

[0041] w = d·K -1 ·p

[0042] Where w is the 3D coordinate, d is the depth value, K is the camera intrinsic parameter matrix, and p is the pixel coordinate.

[0043] Based on the size and resolution of the voxel grid, points in the point cloud are mapped to corresponding voxels; if at least one point in a voxel falls within it, the value of that voxel is set to 1; otherwise, it is set to 0.

[0044] Step 3: The input color image is fed into the 2D human key point detection module, and the color image is processed by a convolutional neural network to extract low-level features.

[0045] Specifically, such as Figure 2 The system reads RGB images and inputs them into a 2D keypoint detection network. It uses the ResNet convolutional neural network architecture as the backbone network. During the forward propagation of ResNet, the outputs of the first few layers are selected as low-level features, which contain basic information about the image.

[0046] The feature vectors are fed into the Transformer encoder. By utilizing the characteristics of the attention mechanism, the model is allowed to assign different weights based on the relevance of the input features, so that more relevant position pairs have higher similarity, thereby increasing their attention; finally, a heatmap of each key point of the human body is output.

[0047] Specifically, the feature map is flattened into a feature sequence. That is, L d-dimensional vectors; the feature sequence is used as the input of the Transformer encoder, and passes through N attention layers and feedforward network in the encoder; in the attention layer, the feature sequence is multiplied by the weight matrix to obtain the query matrix Q, the key matrix K and the value matrix V.

[0048] Furthermore, relative position encoding is added to the Transformer network. The relative encoding is independent of the output and is added to the input to obtain a new encoded feature vector. The re-encoded feature vector is fed into the Transformer network, forcing the shallow layers in the network to pay more attention to local information, helping the network to better handle positional relationships in sequence data, thereby improving the performance of the two-dimensional keypoint detection network.

[0049] Specifically, such as Figure 3 Here, the relative positional encoding of the context pattern is used to adjust the relative positional relationship between Q and K, and a similarity matrix is ​​calculated to help the model better understand the relative positional relationship between key points. The similarity score is calculated through this step, and then a soft-max operation is performed to obtain the attention weight matrix. Finally, this matrix is ​​multiplied by the value matrix V to obtain the final weighted sum, which is the output of the attention layer. The specific formula is as follows:

[0050]

[0051] Where B is the relative position encoding matrix.

[0052] After attention calculation, the output feature vector undergoes nonlinear transformation and mapping through a feedforward network, using the ReLU function as the activation function to extract more complex feature representations.

[0053] To mitigate information loss and gradient vanishing issues, residual connections and layer normalization operations are applied after each self-attention layer and feedforward network. The input feature vector is added to the transformed feature vector and then normalized.

[0054] The entire process involves multi-level feature encoding and representation learning through the superposition of multiple self-attention layers and feedforward networks. In particular, the last attention layer is used as an aggregator to form the maximum activation in the predicted key point heatmap. Finally, the network uses a head to output the human key point heatmap.

[0055] Step 4: Fuse the voxel mesh information from Step 2 with the 2D keypoint heatmap information from Step 3 as the input to the 3D convolutional network:

[0056] Specifically, the human body keypoint heatmap output from step three is tiled along the Z-axis to make it have the same spatial dimension as the voxel mesh. After tiling, each voxel has a corresponding two-dimensional score map in the Z-axis direction. Backprojection is performed based on the predicted two-dimensional keypoints and depth image to obtain the positions of the keypoints in three-dimensional space. A human body voxel mesh V containing the human body is extracted, and this voxel mesh V is compared with the two-dimensional score map S. 2D It also serves as input to the 3D keypoint prediction network.

[0057] Step 5: Input the information from Step 4 into the 3D keypoint prediction network. During the network processing, a series of 3D convolutional layers are used to extract features from the voxel grid and the 2D score map. Specifically, the 3D keypoint prediction network uses a 3D CNN architecture, which is an encoder-decoder architecture, to convert the voxel grid V from Step 4 into the tiled 2D score map S. 2D As input to a 3D human pose estimation network, it is processed through a series of 3D convolutions to estimate the score volume. With the two-dimensional score map S 2D Similarly, by representing the probability of corresponding key points in three-dimensional space, the network can learn the spatial relationship and probability distribution of key points in the voxel grid and fractional map through convolution operations, and finally output the three-dimensional coordinates of human body key points.

[0058] Furthermore, such as Figure 4 In the encoding process, dense connections are used to improve the 3D convolutional network. All feature maps in each layer are connected to the feature maps in the next layer. The feature maps are added together using concatenation operations to achieve feature sharing and make up for the information loss of feature vectors in the 3D convolutional network.

[0059] This invention is a 3D human pose estimation method that integrates color and depth images. It combines color and depth images by using the idea of ​​combining heatmaps and voxel meshes. In the 2D human pose estimation part, relative position encoding is used to effectively capture the spatial relationship between joints. In the 3D human pose estimation part, a dense network is used to enhance pose details, making full use of the rich texture information of color images and the accurate spatial information of depth images, thereby improving performance. This invention has higher application potential in various application scenarios and can provide new ideas and methods for widespread application in reality.

[0060] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for estimating three-dimensional human pose by fusing color images and depth images, characterized in that: The method includes the following steps: Step 1: Split the RGB-D dataset into a color image dataset and a depth image dataset, which will serve as the two inputs for the depth image processing module and the color image processing module, respectively. Step 2: Perform image preprocessing and depth image voxelization on the input depth image; Step 3: The input color image is fed into the 2D human key point detection module, and the color image is processed by a convolutional neural network to extract low-level features. In step three, the two-dimensional human keypoint detection module consists of three parts: a CNN backbone network for extracting low-level image features; a Transformer encoder for capturing long-range spatial interactions between feature vectors across locations; and a head for outputting the predicted keypoint heatmap. Relative position encoding is added to the Transformer encoder, and after being added to the input, a new feature vector is obtained, which forces the shallow layers in the network to pay more attention to local information and helps the network to better handle positional relationships in sequence data. Step 4: Fuse the voxel mesh information from Step 2 with the 2D keypoint heatmap information from Step 3 as the input to the 3D convolutional network; In step four, the two-dimensional human body key point heat map is tiled along the Z-axis, and the heat map information is used as a scoring map. It is combined with the voxel grid to obtain a voxel grid containing the two-dimensional human body key point scoring map. Step 5: Input the information from Step 4 into the 3D keypoint prediction network. During the network processing, a series of 3D convolutional layers are used to extract features from the voxel grid and the 2D score map. In step five, the 3D keypoint prediction network uses a 3D CNN network architecture. The network learns the spatial relationships and probability distributions of keypoints in the voxel grid and fractional map, and finally outputs the 3D coordinates of human keypoints. Here, the idea of ​​dense connection is used to improve the 3D convolutional network, connecting all feature maps of each layer to the feature maps of the next layer. The feature maps are added together using cascade operations.

2. The three-dimensional human pose estimation method based on the fusion of color images and depth images according to claim 1, characterized in that: In step one, the RGB-D dataset is split into a color image dataset and a depth image dataset using the OpenCV library for training.

3. The method for 3D human pose estimation by fusing color images and depth images according to claim 1, characterized in that: In step two, bilateral filtering is used to denoise the depth image, determine the voxel resolution and create a voxel space. After traversing the entire depth image, the voxel grid of the entire depth image voxel block is obtained.

Images