Three-dimensional human pose estimation method and device based on RGBD multi-modal information

Abstract

The application relates to the technical field of computer vision, in particular to a three-dimensional human body pose estimation method and device based on RGBD multi-modal information, wherein the method comprises the following steps: processing RGBD multi-modal information to obtain two-dimensional human body joint point coordinates and three-dimensional human body point clouds, obtaining a fusion voxel grid based on different voxel depth convolution network requirements, inputting the fusion voxel grid into a corresponding voxel depth convolution network, solving the voxel depth convolution network, obtaining the output of a feature voxel grid, estimating three-dimensional key points based on the output of the feature voxel grid, and obtaining a three-dimensional human body pose estimation result. According to the embodiment of the application, the three-dimensional human body pose estimation can be completed by using two three-dimensional depth convolution networks based on the embedding of RGB information and depth information, the function utilization rate of existing hardware facilities is expanded, the application range of the three-dimensional human body pose estimation is expanded, the estimation efficiency is improved, and the obtained result is more reliable and accurate.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for estimating three-dimensional human pose based on RGBD (Red, Green, Blue-Depth map) multimodal information. Background Technology

[0002] With the joint development of deep learning technology and parallel computing hardware, large datasets and general-purpose deep convolutional network algorithms are increasingly being used in the field of computer vision. In computer vision, 3D tasks are typically characterized by high computational cost, complex methods, and difficulty in achieving stable solution spaces.

[0003] Among related technologies, the deep learning-based approach to 3D human pose estimation includes: parameterizing missing information through depth estimation methods, aggregating multi-view information by collecting multi-camera multi-view data, or using depth information for human pose estimation.

[0004] However, in related technologies, training models with missing parameter information requires a large amount of matching datasets, and the reliability of the generated results is insufficient. Furthermore, the process of collecting multi-view data is subject to strict site restrictions, consumes a lot of computation, and has limited applications. Using depth information for estimation reduces the accuracy of the results and cannot achieve efficient and accurate pose estimation, which urgently needs to be addressed. Summary of the Invention

[0005] This application provides a three-dimensional human pose estimation method and apparatus based on RGBD multimodal information to solve the problems in related technologies, such as the need for a large amount of matching datasets for model training with missing parameter information, insufficient reliability of generated results, strict site restrictions and high computational consumption in the process of collecting multi-view data, limited application, and reduced accuracy of results when using depth information estimation, thus failing to achieve efficient and accurate pose estimation.

[0006] The first aspect of this application provides a method for estimating three-dimensional human pose based on RGBD multimodal information, comprising the following steps: processing RGBD multimodal information to obtain two-dimensional human joint coordinates and three-dimensional human point cloud; fusing the two-dimensional human joint coordinates and the three-dimensional human point cloud based on the requirements of different voxel depth convolutional networks to obtain a fused voxel mesh; inputting the fused voxel mesh into the corresponding voxel depth convolutional network, solving the voxel depth convolutional network to obtain the output of the feature voxel mesh, and estimating three-dimensional key points based on the output of the feature voxel mesh to obtain the three-dimensional human pose estimation result.

[0007] In addition, in one embodiment of this application, before estimating the three-dimensional key points based on the output of the feature voxel grid to obtain the three-dimensional human pose estimation result, the method further includes: using an N x [p,p,p] dimension generated feature voxel grid and the output of the feature voxel grid to perform L2-loss training.

[0008] In one embodiment of this application, the processing of the multimodal information of RGBD to obtain two-dimensional human joint coordinates and three-dimensional human point cloud includes: a two-dimensional human pose estimation method based on a deep convolutional network, extracting a two-dimensional feature heatmap or the XY coordinates of two-dimensional key points in the image coordinate system from the RGB (Red, Green, Blue) image; projecting the region of four points in the two-dimensional human pose estimation onto the depth map, cropping out the region where the human image is located, so as to convert the depth map into a three-dimensional point cloud, and obtaining the two-dimensional human joint coordinates and the three-dimensional human point cloud.

[0009] In another embodiment of this application, the voxel deep convolutional network includes a global high-precision network and a local fast network.

[0010] In one embodiment of this application, the step of estimating the three-dimensional key points based on the output of the feature voxel grid includes: finding the maximum value for each feature voxel grid; converting the three-dimensional coordinates of the maximum value under the feature voxel grid into the world coordinate system through an affine transformation to obtain the real world coordinates of any key point, until the real world coordinates of all the required key points are obtained.

[0011] A second aspect of this application provides a three-dimensional human pose estimation device based on RGBD multimodal information, comprising: an acquisition module for processing RGBD multimodal information to obtain two-dimensional human joint coordinates and a three-dimensional human point cloud; a fusion module for fusing the two-dimensional human joint coordinates and the three-dimensional human point cloud based on different voxel depth convolutional network requirements to obtain a fused voxel mesh; and an estimation module for inputting the fused voxel mesh into a corresponding voxel depth convolutional network, solving the voxel depth convolutional network to obtain the output of a feature voxel mesh, and estimating three-dimensional key points based on the output of the feature voxel mesh to obtain a three-dimensional human pose estimation result.

[0012] In addition, in one embodiment of this application, the apparatus further includes: a training module, used to perform L2-loss training using an N x [p,p,p] dimensional generated feature voxel grid and the output of the feature voxel grid before estimating the three-dimensional key points based on the output of the feature voxel grid to obtain the three-dimensional human pose estimation result.

[0013] In one embodiment of this application, the acquisition module includes: an extraction unit, used to extract a two-dimensional feature heatmap or the XY coordinates of two-dimensional key points in the image coordinate system from the RGB image based on a two-dimensional human pose estimation method using a deep convolutional network; and an acquisition unit, used to project the region of four points in the two-dimensional human pose estimation onto the depth map, crop out the region where the human image is located, so as to convert the depth map into a three-dimensional point cloud, and obtain the two-dimensional human joint coordinates and the three-dimensional human point cloud.

[0014] In another embodiment of this application, the voxel deep convolutional network includes a global high-precision network and a local fast network.

[0015] In one embodiment of this application, the estimation module includes: a search unit for finding the maximum value for each feature voxel grid; and a transformation unit for converting the three-dimensional coordinates of the maximum value under the feature voxel grid into the world coordinate system through an affine transformation to obtain the real world coordinates of any joint point, until the real world coordinates of all required joint points are obtained.

[0016] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the three-dimensional human pose estimation method based on RGBD multimodal information as described in the above embodiments.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional human pose estimation method based on RGBD multimodal information.

[0018] This application's embodiments can process RGBD multimodal information to obtain two-dimensional human joint coordinates and three-dimensional human point clouds. Based on the requirements of different voxel depth convolutional networks, the two-dimensional human joint coordinates and three-dimensional human point clouds are fused to obtain a fused voxel mesh, which is then input into the corresponding voxel depth convolutional network. The voxel depth convolutional network is solved to obtain the output of the feature voxel mesh. Then, based on the output of the feature voxel mesh, three-dimensional key points are estimated to obtain the three-dimensional human pose estimation result. This expands the functional utilization of existing hardware facilities, broadens the application scope of three-dimensional human pose estimation, improves estimation efficiency, and makes the obtained results more reliable and accurate. Therefore, it solves the problems in related technologies, such as the need for a large amount of matching datasets for model training with missing parameter information, insufficient reliability of generated results, strict site restrictions and high computational consumption in the process of collecting multi-view data, limited application, and reduced accuracy of results when using depth information estimation, making it impossible to achieve efficient and accurate pose estimation.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 This is a flowchart illustrating a three-dimensional human pose estimation method based on RGBD multimodal information provided in an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of the structure of a three-dimensional human pose estimation device based on RGBD multimodal information according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0025] The following describes a method and apparatus for 3D human pose estimation based on RGBD multimodal information, according to embodiments of this application, with reference to the accompanying drawings. Addressing the issues mentioned in the background section, such as the need for large datasets for training models with missing parameter information, insufficient reliability of generated results, strict site limitations and high computational consumption in the process of collecting multi-view data, and the reduced accuracy of results when using depth information for estimation, this application provides a method for 3D human pose estimation based on RGBD multimodal information. By processing RGBD multimodal information, two-dimensional human joint coordinates and a three-dimensional human point cloud are obtained. Based on the requirements of different voxel depth convolutional networks, the two-dimensional human joint coordinates and the three-dimensional human point cloud are fused to obtain a fused voxel mesh. This fused voxel mesh is then input into the corresponding voxel depth convolutional network, and the voxel depth convolutional network is solved to obtain the output of the feature voxel mesh. Based on the output of the feature voxel mesh, the three-dimensional key points are estimated to obtain the 3D human pose estimation result. This method expands the functional utilization of existing hardware facilities, broadens the application scope of 3D human pose estimation, improves estimation efficiency, and makes the obtained results more reliable and accurate. This solves the problems in related technologies, such as the need for a large amount of matching datasets for model training due to missing parameter information, insufficient reliability of generated results, strict site restrictions and high computational consumption in the process of collecting multi-view data, limited application, and reduced accuracy of results when using depth information estimation, making it impossible to achieve efficient and accurate pose estimation.

[0026] Specifically, Figure 1 This is a flowchart illustrating a three-dimensional human pose estimation method based on RGBD multimodal information provided in an embodiment of this application.

[0027] like Figure 1 As shown, the three-dimensional human pose estimation method based on RGBD multimodal information includes the following steps:

[0028] In step S101, the RGBD multimodal information is processed to obtain two-dimensional human joint coordinates and three-dimensional human point cloud.

[0029] It is understood that the multimodal information processing in this application embodiment can be the combination of information from the RGB image and depth map captured by the target estimation object. The coordinates of the two-dimensional human joints can be the coordinate values ​​of the joints in the coordinate system of the obtained two-dimensional image. The two-dimensional image can be completed by uniform scale and the three-dimensional human point cloud can be obtained by normalizing each dimension. In a physical sense, it is the target human skin surface captured by the depth camera.

[0030] The embodiments of this application can process RGBD multimodal information to obtain two-dimensional human joint coordinates and three-dimensional human point clouds, thereby realizing data matching for RGBD multimodal information and improving the comprehensiveness of the data.

[0031] In one embodiment of this application, the multimodal information of RGBD is processed to obtain two-dimensional human joint coordinates and three-dimensional human point cloud, including: a two-dimensional human pose estimation method based on a deep convolutional network, extracting two-dimensional feature heatmaps or the XY coordinates of two-dimensional key points in the image coordinate system from the RGB image; projecting the region of four points in the two-dimensional human pose estimation onto the depth map, cropping out the region where the human image is located, so as to convert the depth map into a three-dimensional point cloud, and obtain two-dimensional human joint coordinates and three-dimensional human point cloud.

[0032] It is understood that the two-dimensional human pose estimation method using deep convolutional networks in this application embodiment can be a two-dimensional human pose estimation technique based on deep convolutional networks. The region of the four points in the two-dimensional human pose estimation can be the four bounding-box points generated in the two-dimensional human pose estimation result. The obtained four points are used as feature points, and the camera extrinsic parameters are calculated using the camera intrinsic parameters and methods built into the RGBD camera. The four bounding-box points in the two-dimensional human pose estimation are projected onto the depth map through affine transformation, and the region of the projected four points is the area where the human image is located.

[0033] In actual implementation, because the RGBD cameras are very close, the resulting RGB images have a high similarity to the depth map. Therefore, the four bounding-box points generated in the 2D human pose estimation results can be used as feature points and projected onto the depth map through affine transformation to obtain the region where the human figure is located in the depth map. This region can then be cropped to obtain a 2D human image region with dimensions [w, h], where the value of each pixel is the true depth value. The XYZ axis ranges of the single person's 3D space are set to [-1.0, 1.0], [-1.0, 1.0], [-1.7, 0.4], respectively, in meters. After completing the 2D human region image with a uniform scale and normalizing each dimension, a 3D point cloud can be obtained.

[0034] This application embodiment can use a two-dimensional human pose estimation method based on deep convolutional networks to extract two-dimensional feature heatmaps or the XY coordinates of two-dimensional key points in the image coordinate system from the RGB image, and use the region of four points in the two-dimensional human pose estimation to project onto the depth map, crop out the area where the human image is located, so as to transform the depth map into a three-dimensional point cloud, obtain two-dimensional human joint coordinates and three-dimensional human point cloud, and expand the functional utilization of existing hardware facilities and improve the accuracy of the obtained estimation results by completing the fusion of multimodal information.

[0035] In step S102, based on the requirements of different voxel depth convolutional networks, the coordinates of two-dimensional human joints and the three-dimensional human point cloud are fused to obtain a fused voxel mesh.

[0036] It is understood that the different voxel depth convolutional network requirements in the embodiments of this application can correspond to different fused voxel mesh types. The two-dimensional human joint coordinates and three-dimensional human point clouds obtained in the above steps can be fused to obtain fused voxel meshes that are suitable for different networks.

[0037] In actual execution, the mesh within the range of [-1.0, 1.0], [-1.0, 1.0], [-1.7, 0.4] meters obtained from the above steps generates voxel meshes of sizes [64, 64, 64] and [96, 96, 96], with each mesh having an accuracy of approximately 3 cm or 2 cm, respectively. Each 2D human body can be set to have N joints, with the real range of [[-1.0, 1.0], [-1.0, 1.0], [-1.7, 0.4]] meters representing world space. For each joint, a ray is projected in world space with the camera plane as the starting point, and a Gaussian column is generated centered on the ray. This yields a Gaussian column projected in world coordinates centered on that joint. The resulting Gaussian column is then voxelized to obtain voxel meshes of sizes [64, 64, 64] or [96, 96, 96]. This process is repeated to obtain N voxel meshes. Voxel meshes contain only two-dimensional XY position information and lack true depth information. However, the joints of the human body have dual correlations in terms of image and physical meaning. Therefore, depth information and global correlation information can be added to the mesh information according to the needs of convolutional networks of different depths to obtain the corresponding fused voxel meshes, and finally obtain global high-precision meshes or fast meshes.

[0038] Specifically, the global high-precision mesh can directly transform the full-body 3D point cloud generated in the above steps into a single global human information voxel mesh, which simultaneously possesses the true 3D position information of the human body surface and global human body association information. The N Gaussian cylindrical voxel meshes and the single global human information voxel mesh are combined to form the global high-precision mesh, which is then input into the corresponding network as a fused voxel mesh.

[0039] A fast mesh can be obtained by discarding global information, coexisting N Gaussian cylinders with global human voxels, and taking their intersection to form N voxel meshes. These are then used as another type of fused voxel mesh input to the corresponding network. The physical meaning of this mesh can be the effective point cloud information of each human joint within the Gaussian surface range.

[0040] The embodiments of this application can fuse two-dimensional human joint coordinates and three-dimensional human point clouds based on the requirements of different voxel depth convolutional networks to obtain a fused voxel mesh, thereby meeting the mesh output under different requirements and improving the applicability of the estimation process.

[0041] In another embodiment of this application, the voxel depthwise convolutional network includes a global high-precision network and a local fast network.

[0042] It is understood that the global high-precision network in this application embodiment is suitable for application scenarios with high precision requirements, while the local fast network is suitable for application scenarios with high solution speed requirements.

[0043] Specifically, both networks have three basic operating groups: a regular group, a residual group, and a fast group. The regular group consists of two 3D convolutional networks and one ReLU layer or one batch normalization layer connected in series. The residual group adopts the design from ResNet, and the fast group adopts the design from MoibleNet. The global high-precision network uses the encoder-decoder structure of U-Net, so it also has a decoder group, which consists of one 3D transposed convolutional network and one ReLU layer or one batch normalization layer connected in series.

[0044] The global high-precision network comprises a backbone structure and a cross-layer connection structure. The backbone structure consists of two regular groups or residual groups (collectively called the encoding group), two decoding groups, and a 3D convolutional network, used to encode and decode the main features of the output. The cross-layer connection structure directly accumulates the outputs of the first and second regular groups into the inputs of the penultimate and penultimate decoding groups. This global high-precision network can be used to aggregate multi-scale image information and global information.

[0045] Local fast networks, in conjunction with fast grids, are used as input for computation. The decoder and encoder groups in the aforementioned global high-precision network setup are replaced with fast groups to improve computational speed and efficiency. Local fast networks, composed of cascaded fast groups, exhibit strong linearity.

[0046] For example, the global high-precision network was validated on the Multi-Kinect-Temple dataset, achieving an average computational accuracy of 5.71cm. Meanwhile, the local fast network, also validated on the Multi-Kinect-Temple dataset, achieved a computation speed of 32ms (approximately 30 frames) while maintaining an accuracy of around 6.4cm and requiring only 2GB of GPU memory.

[0047] In the embodiments of this application, the voxel deep convolutional network includes a global high-precision network and a local fast network, thereby meeting different solution requirements, diversifying the data processing process, and improving the estimation speed and accuracy.

[0048] In step S103, the fused voxel mesh is input into the corresponding voxel depth convolutional network, the voxel depth convolutional network is solved, the output of the feature voxel mesh is obtained, and the three-dimensional key points are estimated based on the output of the feature voxel mesh to obtain the three-dimensional human pose estimation result.

[0049] It is understood that in this embodiment, the fused voxel mesh is input into the corresponding voxel depth convolutional network. Specifically, the global high-precision network uses the global high-precision mesh obtained in the above steps as input, and the local fast network uses the fast mesh obtained in the above steps as input, thereby obtaining the feature voxel mesh. The 3D keypoints can be the coordinates of human joints in the 3D data state.

[0050] In some embodiments, three-dimensional human pose estimation can be applied to multi-person scenarios. By arranging human figures sequentially during two-dimensional human pose estimation and then performing calculations sequentially, the three-dimensional multi-person human pose estimation results can be obtained.

[0051] In this embodiment, a fused voxel mesh can be input into a corresponding voxel depth convolutional network, the voxel depth convolutional network can be solved to obtain the output of the feature voxel mesh, and the three-dimensional key points can be estimated based on the output of the feature voxel mesh to obtain the three-dimensional human pose estimation result, thereby improving the efficiency of the estimation process and enhancing the reliability of the obtained result.

[0052] In one embodiment of this application, estimating 3D key points based on the output of a feature voxel grid includes: finding the maximum value for each feature voxel grid; converting the 3D coordinates of the maximum value under the feature voxel grid into the world coordinate system through an affine transformation to obtain the real world coordinates of any key point, until the real world coordinates of all required key points are obtained.

[0053] It is understood that in the embodiments of this application, the feature voxel grid can be the N x [p,p,p] dimension output obtained after the voxel depth convolutional network is solved, which is the score of the feature voxel grid under N key points. The size of p can be adjusted and set without being greater than the network input. p determines the accuracy of the network.

[0054] In actual execution, the maximum value can be found for each feature voxel grid, and the three-dimensional coordinates of the value under the feature voxel grid can be transformed into the world coordinate system through affine transformation to obtain the real world coordinates of the joint. The operation is repeated for N feature voxel grids to finally obtain the real world coordinates of all the required joints of the human body, and thus obtain the three-dimensional human pose estimation result.

[0055] In addition, in one embodiment of this application, before estimating the three-dimensional key points based on the output of the feature voxel grid to obtain the three-dimensional human pose estimation result, the method further includes: using an N x [p,p,p] dimension generated feature voxel grid and the output of the feature voxel grid to perform L2-loss training.

[0056] It is understood that the embodiments of this application can be trained for both two stages: 2D human pose estimation and 3D human pose estimation. Specifically, the 2D human pose estimation stage is trained using 2D feature heatmaps and ground truth values ​​for L2-loss training, while the 3D human pose estimation stage is trained using an N x [p,p,p] dimensional generated feature voxel grid and the output of a voxel depthwise convolutional network for L2-loss training. The N x [p,p,p] dimensional generated feature voxel grid can be generated from ground truth 3D human joints.

[0057] For example, for each voxel mesh generated at a size of [p,p,p], the true 3D world coordinates of the human joints are taken and transformed into voxel squares of size [p,p,p] using an affine transformation. The coordinates of this square relative to the voxel mesh are set to [px,py,pz], and its value is set to 1.0. All other values ​​in the voxel mesh are 0. Then, a 3D Gaussian sphere is generated centered on this voxel square. The size of the Gaussian sphere is adjustable; for example, σ = 6 and the variance is 0. The voxel values ​​in the Gaussian sphere gradually decrease to 0 as they are distributed. This process is repeated for N joints to obtain N x [p,p,p] dimension generated feature voxel meshes, which serve as training data in the 3D human pose estimation stage.

[0058] The embodiments of this application can use an N x [p,p,p] dimension generated feature voxel grid and the output of the feature voxel grid for L2-loss training. By using the obtained data for multi-dimensional training and learning, the comprehensiveness of the deep learning process is further realized, and the comprehensiveness and accuracy of the neural network operation are improved.

[0059] The 3D human pose estimation method based on RGBD multimodal information proposed in this application can process RGBD multimodal information to obtain 2D human joint coordinates and 3D human point clouds. Based on the requirements of different voxel depth convolutional networks, the 2D human joint coordinates and 3D human point clouds are fused to obtain a fused voxel mesh, which is then input into the corresponding voxel depth convolutional network. The voxel depth convolutional network is solved to obtain the output of the feature voxel mesh. Then, based on the output of the feature voxel mesh, 3D key points are estimated to obtain the 3D human pose estimation result. This expands the functional utilization of existing hardware facilities, broadens the application scope of 3D human pose estimation, improves estimation efficiency, and makes the obtained results more reliable and accurate. Therefore, it solves the problems in related technologies, such as the need for a large amount of matching datasets for model training with missing parameter information, insufficient reliability of generated results, strict site restrictions and high computational consumption in the process of collecting multi-view data, limited application, and reduced accuracy of results when using depth information estimation, making it impossible to achieve efficient and accurate pose estimation.

[0060] Next, referring to the accompanying drawings, a three-dimensional human pose estimation device based on RGBD multimodal information proposed according to an embodiment of this application is described.

[0061] Figure 2 This is a block diagram of a three-dimensional human pose estimation device based on RGBD multimodal information according to an embodiment of this application.

[0062] like Figure 2 As shown, the three-dimensional human pose estimation device 10 based on RGBD multimodal information includes: an acquisition module 100, a fusion module 200, and an estimation module 300.

[0063] The acquisition module 100 is used to process RGBD multimodal information to obtain two-dimensional human joint coordinates and three-dimensional human point cloud.

[0064] The fusion module 200 is used to fuse two-dimensional human joint coordinates and three-dimensional human point clouds to obtain a fused voxel mesh based on the requirements of convolutional networks with different voxel depths.

[0065] The estimation module 300 is used to input the fused voxel mesh into the corresponding voxel depth convolutional network, solve the voxel depth convolutional network, obtain the output of the feature voxel mesh, and estimate the three-dimensional key points based on the output of the feature voxel mesh to obtain the three-dimensional human pose estimation result.

[0066] In addition, in one embodiment of this application, the device 10 further includes a training module.

[0067] The training module is used to train L2 loss using an N x [p,p,p] dimension generated feature voxel grid and the output of the feature voxel grid before estimating the 3D key points based on the output of the feature voxel grid to obtain the 3D human pose estimation result.

[0068] In one embodiment of this application, the acquisition module 100 includes an extraction unit and an acquisition unit.

[0069] The extraction unit is used for two-dimensional human pose estimation based on deep convolutional networks to extract two-dimensional feature heatmaps or the XY coordinates of two-dimensional key points in the image coordinate system from the RGB image.

[0070] The acquisition unit is used to project the region of four points in the two-dimensional human pose estimation onto the depth map, crop out the region where the human figure is located, and convert the depth map into a three-dimensional point cloud to obtain the coordinates of the two-dimensional human joint points and the three-dimensional human point cloud.

[0071] In another embodiment of this application, the voxel depthwise convolutional network includes a global high-precision network and a local fast network.

[0072] In one embodiment of this application, the estimation module 300 includes a lookup unit and a conversion unit.

[0073] The search unit is used to find the maximum value for each feature voxel grid.

[0074] The transformation unit is used to convert the 3D coordinates of the maximum value in the feature voxel grid into the world coordinate system through affine transformation, so as to obtain the real world coordinates of any joint point, until the real world coordinates of all the required joint points are obtained.

[0075] It should be noted that the foregoing explanation of the embodiment of the three-dimensional human pose estimation method based on RGBD multimodal information also applies to the three-dimensional human pose estimation device based on RGBD multimodal information in this embodiment, and will not be repeated here.

[0076] The 3D human pose estimation device based on RGBD multimodal information proposed in this application can process RGBD multimodal information to obtain 2D human joint coordinates and 3D human point clouds. Based on the requirements of different voxel depth convolutional networks, the 2D human joint coordinates and 3D human point clouds are fused to obtain a fused voxel mesh, which is then input into the corresponding voxel depth convolutional network. The voxel depth convolutional network is solved to obtain the output of the feature voxel mesh. Then, based on the output of the feature voxel mesh, 3D key points are estimated to obtain the 3D human pose estimation result. This expands the functional utilization of existing hardware facilities, broadens the application scope of 3D human pose estimation, improves estimation efficiency, and makes the obtained results more reliable and accurate. Therefore, it solves the problems in related technologies, such as the need for a large amount of matching datasets for model training with missing parameter information, insufficient reliability of generated results, strict site restrictions and high computational consumption in the process of collecting multi-view data, limited application, and reduced accuracy of results when using depth information estimation, making efficient and accurate pose estimation impossible.

[0077] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0078] The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.

[0079] When the processor 302 executes the program, it implements the three-dimensional human pose estimation method based on RGBD multimodal information provided in the above embodiments.

[0080] Furthermore, electronic devices also include:

[0081] Communication interface 303 is used for communication between memory 301 and processor 302.

[0082] The memory 301 is used to store computer programs that can run on the processor 302.

[0083] The memory 301 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0084] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0085] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.

[0086] Processor 302 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0087] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional human pose estimation method based on RGBD multimodal information.

[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0090] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0091] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0092] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0093] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0095] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for estimating three-dimensional human pose based on RGBD multimodal information, characterized in that, Includes the following steps: The RGBD multimodal information is processed to obtain the coordinates of two-dimensional human joints and the three-dimensional human point cloud. Based on the requirements of different voxel depth convolutional networks, the coordinates of the two-dimensional human joints and the three-dimensional human point cloud are fused to obtain a fused voxel mesh. This fused voxel mesh includes a global high-precision mesh and a fast mesh. The global high-precision mesh transforms the three-dimensional human point cloud into a single global human information voxel mesh, which includes the true three-dimensional position information of the human surface and global human association information. N Gaussian cylindrical voxel meshes and the single global human information voxel mesh are combined to form the global high-precision mesh. The intersection of the N Gaussian cylindrical voxel meshes and the single global human information voxel mesh is then divided into N voxel meshes to obtain the fast mesh. The fused voxel grid is input into the corresponding voxel deep convolutional network, and the voxel deep convolutional network is solved to obtain the output of the feature voxel grid. Based on the output of the feature voxel grid, 3D key points are estimated to obtain the 3D human pose estimation result. The voxel deep convolutional network includes a global high-precision network and a local fast network. The global high-precision network uses the global high-precision grid as input, and the local fast network uses the fast grid as input to obtain the feature voxel grid. The global high-precision network has a backbone structure and a cross-layer connection structure. The backbone structure contains two regular groups or residual groups, two decoding groups, and one 3D convolutional network. The encoding group includes the regular group and the residual group to encode the main features of the decoded output. The cross-layer connection structure directly accumulates the outputs of the first and second regular groups into the inputs of the penultimate and penultimate decoding groups. The local fast network works with the fast grid as input to solve the problem, replacing the decoding group and the encoding group in the global high-precision network with a fast group. The local fast network is formed by concatenating the fast groups.

2. The method according to claim 1, characterized in that, Before estimating the 3D key points based on the output of the feature voxel mesh to obtain the 3D human pose estimation result, the method further includes: L2-loss is trained using an N x [p,p,p] dimension generated feature voxel grid and the output of the feature voxel grid.

3. The method according to claim 1, characterized in that, The process of processing the multimodal information of RGBD to obtain two-dimensional human joint coordinates and three-dimensional human point cloud includes: Two-dimensional human pose estimation based on deep convolutional networks extracts two-dimensional feature heatmaps or the XY coordinates of two-dimensional key points in the image coordinate system from RGB images. The region of four points in the two-dimensional human pose estimation is projected onto the depth map, and the region where the human figure is located is cropped to convert the depth map into a three-dimensional point cloud, thereby obtaining the coordinates of the two-dimensional human joint points and the three-dimensional human point cloud.

4. The method according to claim 1, characterized in that, The estimation of 3D key points based on the output of the feature voxel mesh includes: Find the maximum value for each feature voxel grid; The three-dimensional coordinates of the maximum value under the feature voxel grid are transformed into the world coordinate system through affine transformation to obtain the real world coordinates of any joint point, until the real world coordinates of all the required joint points are obtained.

5. A three-dimensional human pose estimation device based on RGBD multimodal information, characterized in that, Includes the following steps: The acquisition module is used to process RGBD multimodal information to obtain two-dimensional human joint coordinates and three-dimensional human point cloud. A fusion module is used to fuse the coordinates of the two-dimensional human joints and the three-dimensional human point cloud based on the requirements of different voxel depth convolutional networks to obtain a fused voxel mesh. The fused voxel mesh includes a global high-precision mesh and a fast mesh. The global high-precision mesh transforms the three-dimensional human point cloud into a single global human information voxel mesh, which includes the true three-dimensional position information of the human surface and global human association information. N Gaussian cylinder voxel meshes and the single global human information voxel mesh are combined to form the global high-precision mesh. The intersection of the N Gaussian cylinder voxel meshes and the single global human information voxel mesh is then divided into N voxel meshes to obtain the fast mesh. The estimation module is used to input the fused voxel mesh into the corresponding voxel depth convolutional network, solve the voxel depth convolutional network to obtain the output of the feature voxel mesh, and estimate 3D key points based on the output of the feature voxel mesh to obtain the 3D human pose estimation result. The voxel depth convolutional network includes a global high-precision network and a local fast network. The global high-precision network uses the global high-precision mesh as input, and the local fast network uses the fast mesh as input to obtain the feature voxel mesh. The global high-precision network has a backbone structure and... The cross-layer connection structure comprises a backbone structure with two regular groups or residual groups, two decoding groups, and a three-dimensional convolutional network connected in series. The encoding group includes the regular group and the residual group, encoding the main features of the decoded output. The cross-layer connection structure directly accumulates the outputs of the first and second regular groups into the inputs of the penultimate and penultimate decoding groups. The local fast network works with the fast grid as input for computation, replacing the decoding group and the encoding group in the global high-precision network setting with fast groups. The local fast network is formed by connecting the fast groups in series.

6. The apparatus according to claim 5, characterized in that, Also includes: The training module is used to perform L2-loss training using an N x [p,p,p] dimensional generated feature voxel grid and the output of the feature voxel grid before estimating the three-dimensional key points based on the output of the feature voxel grid to obtain the three-dimensional human pose estimation result.

7. The apparatus according to claim 5, characterized in that, The voxel-based deep convolutional network includes a global high-precision network and a local fast network.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the three-dimensional human pose estimation method based on RGBD multimodal information as described in any one of claims 1-4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the three-dimensional human pose estimation method based on RGBD multimodal information as described in any one of claims 1-4.

Images