A method for extracting human skeleton based on point cloud data

Abstract

This invention belongs to the field of computer vision technology, specifically to a method for extracting a human skeleton from point cloud data. To make the human skeleton extracted in complex environments more stable and robust, this method uses a trained deep learning network to extract features from the original point cloud. This method then aggregates these features using a feature pooling operation, deriving joint points from the aggregated features. Finally, these joint points are connected in a specific order to form a three-dimensional human skeleton, enabling applications such as behavior recognition and other human-computer interaction applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for extracting a human skeleton based on point cloud data. Background Art

[0002] 3D human skeletons are widely used in fields such as human-computer interaction, motion capture, and healthcare. Obtaining a 3D human skeleton in complex environments is a prerequisite for applying skeleton data. Using traditional skeleton extraction algorithms to process 3D point cloud information alone is likely to result in problems such as skeleton disconnection and missing parts, and it is impossible to obtain a semantically meaningful 3D skeleton. Deep learning methods can better process 3D point clouds, demonstrating greater robustness to complex human poses and missing point clouds. 3D human pose estimation methods using depth images and multi-views have been a popular approach for 3D skeleton extraction in recent years.

[0003] Currently, human skeleton extraction based on point cloud data mostly uses traditional curve skeleton extraction methods. The global shrinkage method adopted by the algorithm easily leads to asynchrony and is not robust enough to missing point cloud data. Due to the rotation invariance and sparsity of point cloud data, how to improve the stability and robustness of human skeletons extracted in complex environments is an important problem that needs to be solved.

[0004] Traditional point cloud skeleton extraction methods generally assume a complete point cloud. Incomplete or incomplete point clouds significantly impact the skeleton's characterization and the quality of skeleton extraction. Deep learning point cloud skeleton extraction technology, leveraging global contextual information, effectively addresses these issues and enhances the reliability and accuracy of skeleton extraction.

[0005] Furthermore, the human point cloud skeleton extracted by deep learning not only outputs structural information about the bones, but also provides a visualization method for the skeleton detection results, allowing people to better understand the skeleton detection process and identify and improve existing problems. Furthermore, deep learning models can automatically learn features through training on large amounts of data, thereby achieving better generalization performance and good robustness in different scenarios. Furthermore, deep learning models can also utilize an end-to-end training approach to extract the skeleton directly from the raw point cloud data, eliminating the need for complex manual processing of the raw data. Summary of the Invention

[0006] The purpose of the present invention is to provide a human skeleton extraction method based on point cloud data, which combines point cloud data converted from depth images to realize joint point extraction of point cloud data, thereby completing the skeleton extraction method.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for extracting a human skeleton based on point cloud data comprises the following steps:

[0009] Step 1: The original depth image is converted into point cloud data through the internal parameters and coordinate transformation of the depth camera;

[0010] Step 2: Visualize the point cloud data, check the converted point cloud data, and remove meaningless data and data with conversion errors;

[0011] Step 3: De-noise the point cloud data and use filtering to downsample the point cloud data.

[0012] Step 4: Generate the processed point cloud data into the processing file required by the deep learning network for training the network;

[0013] Step 5: Construct local areas in the overall point cloud data:

[0014] Step 6: Set up a feature extraction network to extract features from local areas;

[0015] Step 7: Use the feature extraction network to extract the features of the point cloud data at multiple scales and save the features;

[0016] Step 8: Perform pooling operation to gather the extracted features;

[0017] Step 9: The gathered features are passed through the fully connected layer to obtain the joint point data, and the joint points are saved;

[0018] Step 10: Visualize the skeleton data by connecting the extracted joint point data using a 3D visualization program to further obtain the human skeleton data.

[0019] Furthermore, the original depth image in step 1 is converted into point cloud data through the internal parameters and coordinate transformation of the depth camera, and the specific steps are as follows:

[0020] Step 1.1, obtain the internal parameter matrix K of the depth camera:

[0021]

[0022] Among them, f x , f y , C x , C y The unit is pixel, C x ,C y is the translation size from the camera coordinate system to the origin of the pixel coordinate system, f is the image distance, Respectively represent the physical length of a pixel in the X and Y directions on the camera's photosensitive plate;

[0023] Step 1.2: According to the internal parameter matrix of the depth camera and the coordinate system conversion formula, the corresponding points after the depth image is converted into point cloud data are obtained. The formula is as follows:

[0024]

[0025] Among them, D is the depth information of the point, f is the image distance, Respectively represent the physical length of a pixel in the X and Y directions on the camera's photosensitive plate.

[0026] Furthermore, in step 5, a local area is constructed in the overall point cloud data, and the specific steps are as follows:

[0027] Step 5.1: Sample layer samples the input points. The sampling layer selects a set of points in the input point cloud data, which is defined as the center of each local area. It uses iterative farthest point sampling to iteratively select M points from the N point cloud data. Each time, it selects a point that is consistent with the currently selected point set S = {P0, P1, ..., P i All points with the largest distance from each other are added to the set S;

[0028] Step 5.2: Construct local regions. Generate multiple local regions with the points sampled in step 5.1 as the center, and set the number of regions and the radius of the regional sphere.

[0029] Furthermore, in step 6, a feature extraction network is set, and the specific steps are as follows:

[0030] Step 6.1, define a spatial transformation matrix to transform the original point cloud data into a spatial position or shape that is easier to identify. Spatial transformation is represented by matrices and vectors:

[0031]

[0032] Right now:

[0033]

[0034] Where T represents the rotation matrix, b x , b y , b z Indicates displacement in the x, y, and z directions;

[0035] In step 6.2, a convolution module and a maximum pooling operation are added to make the point cloud data conform to the disorder feature. The final D-dimensional feature selects the maximum eigenvalue or the sum of eigenvalues ​​corresponding to N points in each dimension.

[0036] Step 6.3, the final feature extraction formula is:

[0037] f({x1, x2, ..., x n})≈γ o g(h(x1),h(x2),...,h(x n ))

[0038] Among them, the h function is equivalent to the feature extraction layer of the MLP perceptron, the g function is a symmetric method, and γ represents higher-dimensional extraction.

[0039] Furthermore, in step 7, the features of the point cloud data are extracted using a feature extraction network at multiple scales. The specific steps are as follows:

[0040] Step 7.1, use different scales to obtain features around each local area point, with a radius R A Take N as the radius of the sphere A points, sample to obtain feature F A , with radius R B Take N as the radius of the sphere B points, sample to obtain feature F B , with radius R C Take N as the radius of the sphere C points, sample to obtain feature F C , the formula is as follows:

[0041] F A =P(R A ,N A )

[0042] F B =P(R B ,N B )

[0043] F C =P(R C ,N C ) Where P represents the feature extraction method;

[0044] In step 7.2, the features obtained at different scales from the midpoint of each local area in step 7.1 are connected to obtain features of different receptive fields as the features of the local area. The formula is as follows:

[0045] F=Concat(F A ,F B ,F C )

[0046] Among them, Concat represents the concatenation function.

[0047] Furthermore, in step 8, the features extracted in step 7 are clustered, and the formula is as follows:

[0048] S=SA(F)

[0049] Among them, F is the extracted feature and SA represents the feature aggregation method.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] The features of the original point cloud are extracted through a trained deep learning network, and then the features are aggregated through feature pooling. The joint points are obtained based on the aggregated features, and finally the joint points are connected in a specific order to form a three-dimensional human skeleton, thereby realizing applications such as behavior recognition or other human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is the depth camera data graph;

[0053] Figure 2 is a point cloud data graph;

[0054] Figure 3 Extract the rendering of the 3D human skeleton. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] A method for extracting a human skeleton based on point cloud data comprises the following steps:

[0057] Step 1, the original depth image (such as Figure 1 As shown) is converted into point cloud data (as shown) through the internal parameters and coordinate transformation of the depth camera Figure 2 The specific steps are as follows:

[0058] Step 1.1, obtain the internal parameter matrix K of the depth camera:

[0059]

[0060] Among them, f x , f y , C x , C y The unit is pixel, C x ,C y is the translation size from the camera coordinate system to the origin of the pixel coordinate system, f is the image distance, Respectively represent the physical length of a pixel in the X and Y directions on the camera's photosensitive plate;

[0061] Step 1.2: According to the internal parameter matrix of the depth camera and the coordinate system conversion formula, the corresponding points after the depth image is converted into point cloud data are obtained. The formula is as follows:

[0062]

[0063] Among them, D is the depth information of the point, f is the image distance, Respectively represent the physical length of a pixel in the X and Y directions on the camera's photosensitive plate.

[0064] Step 2: Visualize the point cloud data, check the converted point cloud data, and remove meaningless data and data with conversion errors;

[0065] Step 3: De-noise the point cloud data and use filtering to downsample the point cloud data.

[0066] Step 4: Generate the processed point cloud data into the processing file required by the deep learning network for training the network;

[0067] Step 5: Construct a local area in the overall point cloud data. The specific steps are as follows:

[0068] Step 5.1: Sample layer samples the input points. The sampling layer selects a set of points in the input point cloud data, which is defined as the center of each local area. It uses iterative farthest point sampling to iteratively select M points from the N point cloud data. Each time, it selects a point that is consistent with the currently selected point set S = {P0, P1, ..., P i All points with the largest distance from each other are added to the set S;

[0069] Step 5.2: Construct local regions. Generate 125 local regions with the points sampled in step 5.1 as the center. The radii of the regional spheres are 0.1, 0.2, and 0.4 respectively.

[0070] Step 6: Set up a feature extraction network to extract features from local areas. The specific steps are as follows:

[0071] Step 6.1, define a spatial transformation matrix to transform the original point cloud data into a spatial position or shape that is easier to identify. Spatial transformation is represented by matrices and vectors:

[0072]

[0073] Right now:

[0074]

[0075] Where T represents the rotation matrix, b x , b y , b z Indicates displacement in the x, y, and z directions;

[0076] In step 6.2, a convolution module and a maximum pooling operation are added to make the point cloud data conform to the disorder feature. The final D-dimensional feature selects the maximum eigenvalue or the sum of eigenvalues ​​corresponding to N points in each dimension.

[0077] Step 6.3, the final feature extraction formula is:

[0078] f({x1, x2, ..., x n})≈γ o g(h(x1),h(x2),...,h(x n ))

[0079] Among them, the h function is equivalent to the feature extraction layer of the MLP perceptron, the g function is a symmetric method, and γ represents higher-dimensional extraction.

[0080] Step 7, use the feature extraction network to extract the features of the point cloud data at multiple scales and save the features. Step 7.1, use different scales to obtain features around each local area point with a radius R A Take N as the radius of the sphere A points, sample to obtain feature F A , with radius R B Take N as the radius of the sphere B points, sample to obtain feature F B , with radius R C Take N as the radius of the sphere C points, sample to obtain feature F C , the formula is as follows:

[0081] F A =P(R A ,N A )

[0082] F B =P(R B ,N B )

[0083] F C =P(R C ,N C ) Where P represents the feature extraction method;

[0084] In step 7.2, the features obtained at different scales from the midpoint of each local area in step 7.1 are connected to obtain features of different receptive fields as the features of the local area. The formula is as follows:

[0085] F=Concat(F A ,F B ,F C )

[0086] Among them, Concat represents the concatenation function.

[0087] Step 8: Perform pooling operation to aggregate the extracted features. The formula is as follows:

[0088] S=SA(F)

[0089] Among them, F is the extracted feature and SA represents the feature aggregation method.

[0090] Step 9: The gathered features are passed through the fully connected layer to obtain the joint point data and save the joint points;

[0091] Step 10: Visualize the skeleton data and use a 3D visualization program to connect the extracted joint point data to further obtain the human skeleton data (such as Figure 3 shown).

Claims

1. A method for extracting a human skeleton based on point cloud data, characterized in that: The following steps are involved: Step 1: The original depth image is converted into point cloud data through the internal parameters and coordinate transformation of the depth camera; Step 2: Visualize the point cloud data, check the converted point cloud data, and remove meaningless data and data with conversion errors; Step 3: De-noise the point cloud data and use filtering to downsample the point cloud data. Step 4: Generate the processed point cloud data into the processing file required by the deep learning network for training the network; Step 5: Construct local areas in the overall point cloud data: Step 6: Set up a feature extraction network to extract features from local areas; Step 7: Use the feature extraction network to extract the features of the point cloud data at multiple scales and save the features; Step 8: Perform pooling operation to gather the extracted features; Step 9: The gathered features are passed through the fully connected layer to obtain the joint point data and save the joint points; Step 10: Visualize the skeleton data by connecting the extracted joint point data using a 3D visualization program to further obtain the human skeleton data.

2. The method for extracting a human skeleton based on point cloud data according to claim 1, wherein: In step 1, the original depth image is converted into point cloud data through the internal parameters and coordinate transformation of the depth camera. The specific steps are as follows: Step 1.1, obtain the internal parameter matrix K of the depth camera: Among them, f x , f y , C x , C y The unit is pixel, C x ,C y is the translation size from the camera coordinate system to the origin of the pixel coordinate system, f is the image distance, Respectively represent the physical length of a pixel in the X and Y directions on the camera's photosensitive plate; Step 1.2: According to the internal parameter matrix of the depth camera and the coordinate system conversion formula, the corresponding points after the depth image is converted into point cloud data are obtained. The formula is as follows: Among them, D is the depth information of the point, f is the image distance, Respectively represent the physical length of a pixel in the X and Y directions on the camera's photosensitive plate.

3. The method for extracting a human skeleton based on point cloud data according to claim 1, wherein: In step 5, a local area is constructed in the overall point cloud data. The specific steps are: Step 5.1: Sample layer samples the input points. The sampling layer selects a set of points in the input point cloud data, which is defined as the center of each local area. It uses iterative farthest point sampling to iteratively select M points from the N point cloud data. Each time, it selects a point that is consistent with the currently selected point set S = {P0, P1, ..., P i All points with the largest distance from each other are added to the set S; Step 5.2: Construct local regions. Generate multiple local regions with the points sampled in step 5.1 as the center, and set the number of regions and the radius of the regional sphere.

4. The method for extracting a human skeleton based on point cloud data according to claim 1, wherein: In step 6, a feature extraction network is set, and the specific steps are as follows: Step 6.1, define a spatial transformation matrix to transform the original point cloud data into a spatial position or shape that is easier to identify. Spatial transformation is represented by matrices and vectors: Right now: Where T represents the rotation matrix, b x , b y , b z Indicates displacement in the x, y, and z directions; In step 6.2, a convolution module and a maximum pooling operation are added to make the point cloud data conform to the disorder feature. The final D-dimensional feature selects the maximum eigenvalue or the sum of eigenvalues ​​corresponding to N points in each dimension. Step 6.3, the final feature extraction formula is: Among them, the h function is equivalent to the feature extraction layer of the MLP perceptron, the g function is a symmetric method, and γ represents higher-dimensional extraction.

5. The method for extracting a human skeleton based on point cloud data according to claim 3, wherein: In step 7, the feature extraction network is used to extract the features of the point cloud data at multiple scales. The specific steps are as follows: Step 7.1, use different scales to obtain features around each local area point, with a radius R A Take N as the radius of the sphere A points, sample to obtain feature F A , with radius R B Take N as the radius of the sphere B points, sample to obtain feature F B , with radius R C Take N as the radius of the sphere C points, sample to obtain feature F C , the formula is as follows: F A =P(R A ,N A ) F B =P(R B ,N B ) F C =P(R C ,N C ) Where P represents the feature extraction method; In step 7.2, the features obtained at different scales from the midpoint of each local area in step 7.1 are connected to obtain features of different receptive fields as the features of the local area. The formula is as follows: F=Concat(F A ,F B ,F C ) Among them, Concat represents the concatenation function.

6. The method for extracting a human skeleton based on point cloud data according to claim 1, wherein: In step 8, the features extracted in step 7 are aggregated, and the formula is as follows: S=SA(F) Among them, F is the extracted feature and SA represents the feature aggregation method.

Images