Three-dimensional human pose estimation method based on multi-view images and sparse point cloud

Abstract

The application provides a three-dimensional human body posture estimation method based on multi-view images and sparse point clouds, relates to the field of 3D vision, and comprises the following steps: voxelizing multi-view RGB information and initial point cloud data to determine first modal voxel data and second modal voxel data; performing preliminary feature extraction on the first modal voxel data and the second modal voxel data according to three-dimensional convolution to determine first modal initial feature information and second modal initial feature information; fusing the first modal initial feature information and the second modal initial feature information, performing feature extraction on the fused information, and determining a three-dimensional posture heat map result; and determining three-dimensional human body posture joint point coordinates according to the three-dimensional posture heat map result. According to the application, the multi-view two-dimensional posture information and the point cloud information are unified to the same voxel space for feature fusion by using the representation method of the voxel space, and then the three-dimensional posture representation is determined, so that the precision and the accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of 3D vision, and in particular to a method for estimating three-dimensional human pose based on multi-view images and sparse point clouds. Background Technology

[0002] Scene analysis has always been a key issue in the field of vision. It utilizes sensor information within a scene to extract scene attributes or important foreground features, such as foreground target detection and tracking, and scene semantic segmentation. 3D human pose estimation uses sensor information to locate the joints of a pedestrian's skeleton in a scene and reconstruct the complete pedestrian skeleton information. This task is more complex and difficult than localization, tracking, or segmentation. In the field of 3D human pose estimation, some methods exist that can achieve an error within 2cm per joint, but their implementation scenarios are all indoors with small scene areas. Furthermore, most current low-error 3D pose estimation methods rely on multi-view 2D pose detection. When the application scenario is shifted to a large scene, the performance of the front-end 2D pose detection deteriorates, resulting in these methods not performing well in large-scale human pose detection applications.

[0003] With the maturity of LiDAR technology, it has become possible to collect scene information using LiDAR and cameras. However, due to the limitation of the laser beam emitted by LiDAR, the point cloud data obtained in large scenes is usually sparse. On the other hand, under the condition of large scene acquisition, human targets in the camera's field of view are often relatively small, and current advanced 2D pose estimation algorithms may still have missed or false pose point detections. Summary of the Invention

[0004] To address the aforementioned issues, this application proposes a method and apparatus for estimating three-dimensional human pose based on multi-view images and sparse point clouds. By utilizing voxel space representation methods, it unifies multi-view two-dimensional pose information and point cloud information into the same voxel space for feature fusion, thereby determining the three-dimensional pose representation and improving accuracy.

[0005] The first aspect of this application proposes a method for 3D human pose estimation based on multi-view images and sparse point clouds, including:

[0006] Voxelization preprocessing is performed on the multi-view 2D pose RGB information and the initial point cloud data to determine the first modal voxel data and the second modal voxel data;

[0007] Preliminary feature extraction is performed on the first modality voxel data and the second modality voxel data based on 3D convolution to determine the initial feature information of the first modality and the initial feature information of the second modality;

[0008] The initial feature information of the first mode is fused with the initial feature information of the second mode, and features are extracted from the fused information to determine the three-dimensional pose heatmap result.

[0009] Based on the three-dimensional posture heatmap results, the coordinates of the three-dimensional human body posture joints are determined.

[0010] Optionally, the step of performing voxelization preprocessing on the multi-view RGB information and the initial point cloud data to determine the first modality voxel data and the second modality voxel data includes:

[0011] The voxel space is divided, and the actual spatial coordinates corresponding to each voxel in the voxel space are determined;

[0012] A two-dimensional attitude heatmap is extracted from the multi-view RGB information using a two-dimensional attitude detection algorithm, and the two-dimensional attitude heatmap is back-projected onto the voxel space to determine the first modal voxel data.

[0013] The initial point cloud data is mapped to the voxel space using the actual spatial coordinates to determine the second modal voxel data.

[0014] Optionally, the partitioning of the voxel space and the determination of the actual spatial coordinates corresponding to each voxel in the voxel space include:

[0015] The method for partitioning the voxel space is determined, wherein, for the voxel space to be partitioned with physical dimensions of X×Y×Z and resolution of K×W×H, the physical dimensions of the voxels after quantization at the three-dimensional scale are determined based on the physical dimensions and the resolution.

[0016] The pedestrian's position and center coordinates c are determined based on the initial point cloud data.

[0017] Based on the pedestrian's position, the center coordinate c, and the voxel space partitioning method, determine the actual spatial coordinates O corresponding to each voxel in the voxel space. i =(x i ,y i ,z i ).

[0018] Optionally, determining the first modality voxel data includes:

[0019] The physical coordinates O represented by the voxel are determined based on the camera parameters. i =(x i ,y i ,z i );

[0020] For the two-dimensional attitude heatmap P vIf the two-dimensional coordinates of the voxel projected into the two-dimensional pixel space are Q i =π(O i Then the two-dimensional coordinates Q i The two-dimensional attitude heatmap P v The heatmap feature index, where P v ∈R C×W×H ;

[0021] Combined with the two-dimensional coordinates Q i The first modality voxel data A is determined by the V viewpoint heatmap features indexed, where A∈R C×K×W×H .

[0022] Optionally, the combined two-dimensional coordinates Q i The V viewpoint heatmap features of the index are used to determine the first modality voxel data A, including:

[0023] If the two-dimensional coordinates of the same voxel after projection from V viewpoints If the data does not exceed the field of view of camera v, then the first modal voxel data A is the average value of the V viewpoint heatmap features, which can be formulated as follows:

[0024]

[0025] Where i is the i-th voxel, V is the number of viewpoints, and [] represents the index.

[0026] Optionally, the second modality voxel data is determined according to one of the following processes:

[0027] The voxel block is assigned a value based on whether there are scan data points in the voxel block, and the voxel block value is determined.

[0028] The voxel block value is determined by assigning a value to the voxel block based on the number of scan data points in the voxel block.

[0029] The second modal voxel data B is determined based on the voxel block value, where B∈R 1×K×W×H .

[0030] Optionally, the initial feature information of the first mode and the initial feature information of the second mode satisfy:

[0031] If the initial feature information of the first modality determined based on the first modal voxel data A is F A The initial feature information of the first mode determined based on the second modal voxel data B is F. B ,but and

[0032] Optionally, the step of fusing the initial feature information of the first mode with the initial feature information of the second mode, and extracting features from the fused information to determine the three-dimensional pose heatmap result includes:

[0033] The initial feature information of the first modality and the initial feature information of the second modality are fused by dimensional concatenation to determine the preliminary fused feature F, where F = Concatenate(F A ,F B ),

[0034] Based on the preliminary fusion feature F parsed by the 3D convolutional and pose task network, the 3D pose heatmap result D is determined, where D∈R J×K×W×H J represents the number of joints in the output.

[0035] Optionally, determining the coordinates of the three-dimensional human posture joints based on the three-dimensional posture heatmap results includes:

[0036] The 3D pose heatmap results are processed using softargmax operations to determine the coordinates of the 3D human pose joints, which are then formulated as follows:

[0037] M j =∑ i D i ×O i ,i=1,2,3,4...,(K×W×H).

[0038] Where j is the j-th joint.

[0039] The second aspect of this application proposes a 3D human pose estimation device based on multi-view images and sparse point clouds, comprising:

[0040] The preprocessing module is used to perform voxelization preprocessing on multi-view RGB information and initial point cloud data to determine the first modality voxel data and the second modality voxel data;

[0041] The feature extraction module is used to perform preliminary feature extraction on the first modality voxel data and the second modality voxel data based on three-dimensional convolution, and determine the initial feature information of the first modality and the initial feature information of the second modality;

[0042] The fusion module is used to fuse the initial feature information of the first mode with the initial feature information of the second mode, and to extract features from the fused information to determine the three-dimensional pose heatmap result.

[0043] The coordinate output module is used to determine the coordinates of the joints of the three-dimensional human posture based on the three-dimensional posture heatmap results.

[0044] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0045] By utilizing voxel space representation methods, multi-view 2D pose information and point cloud information are unified into the same voxel space for feature fusion, thereby determining the 3D pose representation and improving accuracy and precision.

[0046] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0047] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0048] Figure 1 This is a flowchart illustrating a three-dimensional human pose estimation method based on multi-view images and sparse point clouds according to an exemplary embodiment of this application;

[0049] Figure 2 This is a flowchart illustrating a three-dimensional human pose estimation method based on multi-view images and sparse point clouds according to an exemplary embodiment of this application;

[0050] Figure 3 This is a block diagram illustrating a three-dimensional human pose estimation device based on multi-view images and sparse point clouds according to an exemplary embodiment of this application. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0052] LiDAR (LiDAR) is characterized by its long range, high accuracy, and strong stability. With advancements in LiDAR technology, low-cost, low-point-cloud-density LiDARs have emerged on the market. These LiDARs produce very sparse point clouds, and the point cloud density decreases quadratically with increasing target distance. Therefore, the point cloud of a person's surface obtained from a single frame of LiDAR data is often incomplete and cannot accurately reflect all the information about the human posture. However, the 3D position information of the scanned surface points is relatively accurate. RGB information obtained from a camera can generally provide complete human information, but due to the large scene size, the estimation results at the 2D level will have some errors. Consequently, the 3D information obtained using epipolar geometry will also introduce some bias. By fusing the information from both modes, their respective strengths can be combined to achieve a better estimation result.

[0053] By employing local spatial voxelization, point cloud data can be directly represented in voxel space, while two-dimensional data information can be presented in voxel space through projection. This voxel-space representation allows for the natural fusion of data from two different modalities. Voxelization makes the data more regular and also allows for better results when using traditional convolution.

[0054] Figure 1 This is a flowchart illustrating a 3D human pose estimation method based on multi-view images and sparse point clouds according to an exemplary embodiment of this application, such as... Figure 1 As shown, it includes:

[0055] Step 101: Perform voxelization preprocessing on the multi-view two-dimensional pose RGB information and the initial point cloud data to determine the first modal voxel data and the second modal voxel data.

[0056] In this embodiment of the application, under a fixed large scene, initial point cloud data is obtained by LiDAR, RGB information is obtained by multi-view camera, and the information obtained by the sensor is used to estimate the three-dimensional human pose of pedestrians in the scene, and finally the three-dimensional skeleton information of pedestrians in the scene is obtained. The input of two-dimensional information is a two-dimensional human joint semantic segmentation map, instance segmentation map, etc., which have human joints or human contours.

[0057] This application employs existing two-dimensional pose detection algorithms to obtain a two-dimensional heatmap representation of the human pose from each viewpoint and projects it into a three-dimensional voxel space. Then, the human point cloud data is mapped to the three-dimensional voxel space according to its actual three-dimensional position. Specifically, step 101 also includes:

[0058] Step 201: Divide the voxel space and determine the actual spatial coordinates corresponding to each voxel in the voxel space.

[0059] First, the voxel space needs to be divided to determine the actual spatial coordinates of each voxel.

[0060] The specific process involves determining the voxel space partitioning method. For a voxel space with physical dimensions of X×Y×Z and a resolution of K×W×H, the physical dimensions of the voxels after quantization at the three-dimensional scale are determined based on the physical dimensions and resolution. Next, the pedestrian's position and center coordinates are determined based on the initial point cloud data of the target pedestrian. Finally, by combining the pedestrian's position, center coordinate c, and the voxel space partitioning method, the actual spatial coordinates O corresponding to each voxel in the voxel space are determined. i =(x i ,y i ,z i ).

[0061] Among these requirements, it is necessary to ensure that the voxel space can cover pedestrians.

[0062] Step 202: Extract a two-dimensional attitude heatmap from the multi-view RGB information using a two-dimensional attitude detection algorithm, and back-project the two-dimensional attitude heatmap into the voxel space to determine the first modality voxel data.

[0063] In this embodiment of the application, a two-dimensional pose detection algorithm is used to extract the first modal voxel data of the target pedestrian, wherein the first modal voxel data is two-dimensional pose information or contour information.

[0064] Optionally, the physical coordinates O represented by the voxel can be determined based on the camera parameters. i =(x i ,y i ,z i );

[0065] For the two-dimensional attitude heatmap P v If the two-dimensional coordinates of a voxel projected into a two-dimensional pixel space are Q i =π(O i If the two-dimensional coordinates Q are given, then... i For two-dimensional attitude heatmap P v The heatmap feature index, where P v ∈R C×W×H ;

[0066] Joint two-dimensional coordinates Q i The V viewpoint heatmap features of the index are used to determine the first modality voxel data A, where A∈R C ×K×W×H .

[0067] Wherein, the two-dimensional coordinates of the same voxel after projection from V viewpoints If the data does not exceed the field of view of camera v, then the first modal voxel data A is the average value of the heatmap features of V viewpoints, which can be formulated as follows:

[0068]

[0069] Where i is the i-th voxel, V is the number of viewpoints, and [] represents the index.

[0070] Step 203: Map the initial point cloud data to the voxel space using actual spatial coordinates to determine the second modal voxel data.

[0071] In this embodiment of the application, the initial point cloud data are all represented by their physical coordinates in the actual space. Therefore, the point cloud data can be mapped to the voxel space based on the obtained voxel space representation. Specifically, the mapping method is to assign a value to the voxel block based on whether there are scan data points in the voxel block to determine the voxel block value, or to assign a value to the voxel block based on the number of scan data points in the voxel block to determine the voxel block value.

[0072] In this embodiment of the application, if there are scan data points in the voxel block, the voxel block is assigned the value 1.

[0073] After determining the voxel block values, the second modal voxel data B obtained from the initial point cloud data can be obtained, where B∈R. 1×K×W×H .

[0074] Step 102: Perform preliminary feature extraction on the voxel data of the first modality and the voxel data of the second modality based on 3D convolution to determine the initial feature information of the first modality and the initial feature information of the second modality.

[0075] In this embodiment, the first modal voxel data A generated from multi-view RGB information and the second modal voxel data B generated from initial point cloud data need to be represented in the same voxel space.

[0076] When using a basic convolutional framework to extract features from different modalities, it is necessary to ensure that the spatial resolution remains consistent when obtaining the initial feature information for the final task. That is, if the initial feature information of the first modality determined based on the voxel data A of the first modality is F... A The initial feature information of the first mode determined based on the voxel data B of the second mode is F. B It needs to meet the following requirements. and The initial feature information of the two modalities obtained in this way can be fused along the feature channel dimension.

[0077] Step 103: Fuse the initial feature information of the first mode with the initial feature information of the second mode, and extract features from the fused information to determine the three-dimensional pose heatmap result.

[0078] In this embodiment of the application, after obtaining the aligned first modality initial feature information and the second modality initial feature information in step 102, they need to be spliced ​​to obtain the spliced ​​preliminary fusion feature F, and then feature fusion is performed through subsequent three-dimensional convolution operations. The aim is to allow the two initial feature information to complement each other and promote each other, and finally parse the three-dimensional human pose heat map result D through the final task layer.

[0079] Specifically, firstly, the initial feature information of the first modality and the initial feature information of the second modality are fused by dimensional concatenation to determine the preliminary fused feature F, where F = Concatenate(F A ,F B ), Then, by analyzing the preliminary fused features F through a 3D convolutional and pose task network, the 3D pose heatmap result D is determined, where D∈R J×K×W×H J represents the number of joints in the output.

[0080] Step 104: Determine the coordinates of the joints of the three-dimensional human body posture based on the three-dimensional posture heatmap results.

[0081] In this embodiment of the application, it is necessary to process the three-dimensional pose heatmap results of each joint channel. The activation layer uses softmax operation to make the sum of all values ​​of each channel equal to 1.

[0082] In addition, in order to enable backpropagation of the error gradient between the prediction and the true value during loss function supervision, the position calculation of the 3D human pose joints uses the softargmax operation to process the 3D pose heatmap results and determine the coordinates of the 3D human pose joints.

[0083] In one possible embodiment, the joint coordinates M of the j-th joint are... j The output is:

[0084] M j =∑ i D i ×O i ,i=1,2,3,4...,(K×W×H).

[0085] This application embodiment utilizes a voxel space representation method to unify multi-view two-dimensional pose information and point cloud information into the same voxel space for feature fusion, thereby determining a three-dimensional pose representation and improving accuracy and precision.

[0086] Figure 3 This is a block diagram of a three-dimensional human pose estimation device 300 based on multi-view images and sparse point clouds, according to an exemplary embodiment of this application, including: a preprocessing module 310, a feature extraction module 320, a fusion module 330, and a coordinate output module 340.

[0087] The preprocessing module 310 is used to perform voxelization preprocessing on the multi-view RGB information and the initial point cloud data to determine the first modality voxel data and the second modality voxel data;

[0088] The feature extraction module 320 is used to perform preliminary feature extraction on the voxel data of the first modality and the voxel data of the second modality based on the three-dimensional convolution, and to determine the initial feature information of the first modality and the initial feature information of the second modality.

[0089] The fusion module 330 is used to fuse the initial feature information of the first mode with the initial feature information of the second mode, and to extract features from the fused information to determine the three-dimensional pose heatmap result.

[0090] The coordinate output module 340 is used to determine the coordinates of the joints of the three-dimensional human body posture based on the three-dimensional posture heatmap results.

[0091] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0092] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for estimating 3D human pose based on multi-view images and sparse point clouds, characterized in that, include: Voxelization preprocessing is performed on the multi-view 2D pose RGB information and the initial point cloud data to determine the first modality voxel data and the second modality voxel data; Preliminary feature extraction is performed on the first modality voxel data and the second modality voxel data based on 3D convolution to determine the initial feature information of the first modality and the initial feature information of the second modality; The initial feature information of the first mode is fused with the initial feature information of the second mode, and features are extracted from the fused information to determine the three-dimensional pose heatmap result. Based on the three-dimensional posture heatmap results, determine the coordinates of the three-dimensional human body posture joints; The step of performing voxelization preprocessing on multi-view 2D pose RGB information and initial point cloud data to determine first-modal RGB voxel data and second-modal point cloud voxel data includes: The voxel space is divided, and the actual spatial coordinates corresponding to each voxel in the voxel space are determined; A two-dimensional attitude heatmap is extracted from the multi-view two-dimensional attitude RGB information using a two-dimensional attitude detection algorithm, and the two-dimensional attitude heatmap is back-projected onto the voxel space to determine the first modal voxel data. The initial point cloud data is mapped to the voxel space using the actual spatial coordinates to determine the second modal voxel data.

2. The method according to claim 1, characterized in that, The process of dividing the voxel space and determining the actual spatial coordinates corresponding to each voxel in the voxel space includes: The method for partitioning the voxel space is determined, wherein, for the voxel space to be partitioned with physical dimensions of X×Y×Z and resolution of K×W×H, the physical dimensions of the voxels after quantization at the three-dimensional scale are determined based on the physical dimensions and the resolution. ; The pedestrian's location and center coordinates are determined based on the initial point cloud data. ; Based on the pedestrian's location and the center coordinates The method of dividing the voxel space determines the actual spatial coordinates corresponding to each voxel in the voxel space. .

3. The method according to claim 2, characterized in that, Determining the first modality voxel data includes: The physical coordinates represented by the voxels are determined based on the camera parameters. ; For the two-dimensional attitude heatmap If the two-dimensional coordinates of the voxel projected into the two-dimensional pixel space are Then the two-dimensional coordinates The two-dimensional attitude heatmap The heatmap feature index, where... ; Combined with the two-dimensional coordinates The V-view heatmap features of the index are used to determine the first modality voxel data. ,in, .

4. The method according to claim 3, characterized in that, The combined two-dimensional coordinates The V-view heatmap features of the index are used to determine the first modality voxel data. ,include: If the two-dimensional coordinates of the same voxel after projection from V viewpoints Not beyond the camera The field of view is then the voxel data of the first modality. The average value of the V viewpoint heatmap features is formulated as follows: ， in, Let be the i-th voxel, V be the number of viewpoints, and [ ] represent the index.

5. The method according to claim 4, characterized in that, The second modality voxel data is determined according to one of the following processes: The voxel block is assigned a value based on whether there are scan data points in the voxel block, and the voxel block value is determined. The voxel block value is determined by assigning a value to the voxel block based on the number of scan data points in the voxel block. The second modal voxel data is determined based on the voxel block value. ,in, .

6. The method according to claim 5, characterized in that, The initial feature information of the first mode and the initial feature information of the second mode satisfy: Based on the first modal voxel data The determined initial feature information of the first mode is According to the second modal voxel data The determined initial feature information of the first mode is ,but and .

7. The method according to claim 6, characterized in that, The step of fusing the initial feature information of the first mode with the initial feature information of the second mode, and extracting features from the fused information to determine the three-dimensional pose heatmap result includes: The initial feature information of the first modality and the initial feature information of the second modality are fused by dimensional concatenation to determine the preliminary fused feature F, wherein, , ; The 3D pose heatmap result is determined based on the preliminary fusion feature F obtained by parsing the 3D convolutional and pose task network. ,in, J represents the number of joints in the output.

8. The method according to claim 7, characterized in that, The step of determining the coordinates of the three-dimensional human body posture joints based on the three-dimensional posture heatmap results includes: pass The 3D posture heatmap results are processed to determine the coordinates of the 3D human posture joints, which are then formulated as follows: in, Let j be the j-th joint.

9. A three-dimensional human pose estimation device based on multi-view images and sparse point clouds, characterized in that, include: The preprocessing module is used to perform voxelization preprocessing on the multi-view two-dimensional pose RGB information and the initial point cloud data to determine the first modal voxel data and the second modal voxel data. The feature extraction module is used to perform preliminary feature extraction on the first modality voxel data and the second modality voxel data based on three-dimensional convolution, and determine the initial feature information of the first modality and the initial feature information of the second modality; The fusion module is used to fuse the initial feature information of the first mode with the initial feature information of the second mode, and to extract features from the fused information to determine the three-dimensional pose heatmap result. The coordinate output module is used to determine the coordinates of the joints of the three-dimensional human posture based on the three-dimensional posture heatmap results. The step of performing voxelization preprocessing on multi-view 2D pose RGB information and initial point cloud data to determine first-modal RGB voxel data and second-modal point cloud voxel data includes: The voxel space is divided, and the actual spatial coordinates corresponding to each voxel in the voxel space are determined; A two-dimensional attitude heatmap is extracted from the multi-view two-dimensional attitude RGB information using a two-dimensional attitude detection algorithm, and the two-dimensional attitude heatmap is back-projected onto the voxel space to determine the first modal voxel data. The initial point cloud data is mapped to the voxel space using the actual spatial coordinates to determine the second modal voxel data.

Images