A method and device for three-dimensional reconstruction of human body

Abstract

The present application relates to the technical fields of computer vision and computer graphics, and provides a method and device for 3D reconstruction of a human body, which realizes 3D reconstruction of a human body by combining global reconstruction and local reconstruction. Firstly, point cloud data corresponding to N consecutive frames of target RGBD images are fused and segmented to obtain human body point cloud data, and an initial human body model is generated based on the human body point cloud data. Since each frame of the target RGBD image contains complete data of the human body under various viewing angles, the integrity of the initial human body model is ensured. Then, according to the target RGB image corresponding to the target point cloud data, a 2D feature point set of the target area is extracted and converted into a 3D feature point set. According to the 3D feature point set and the point cloud data of the target area, a local model is generated, and the local model is aligned with the initial human body model. The detailed information of the target area in the initial human body model is enriched by the local model, so as to obtain a high-precision target human body model.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and computer graphics, and in particular to a method and device for three-dimensional reconstruction of a human body. Background Art

[0002] As virtual reality (VR) and augmented reality (AR) technologies gradually expand from military and industrial fields to entertainment and life fields, people's social methods have changed. Realistic virtual avatars socialize in virtual space, recreating the face-to-face immersion of the first generation of social methods. Virtual / augmented reality social methods may become the fifth generation of social media after the fourth generation of mobile Internet. As a key technology for virtual social interaction, human body 3D reconstruction has important research significance.

[0003] Generally, 3D reconstruction of the human body involves morphology, posture and texture data. During the reconstruction process, the collected information is first obtained from various sensors, and then the collected information is processed using a 3D reconstruction method to reconstruct a 3D model of the human body.

[0004] In recent years, with the continuous development of imaging technology, visual 3D reconstruction technology based on RGB cameras has gradually become a research hotspot. With the emergence of RGBD cameras and the proposal and optimization of dense SLAM algorithms, the quality and efficiency of 3D reconstruction have been further improved.

[0005] At present, there are three main types of reconstruction systems for human 3D reconstruction, namely the "birdcage" reconstruction system based on multi-array cameras, the reconstruction system based on a single camera, and the reconstruction system based on a sparse multi-view camera. Since the reconstruction system based on a sparse multi-view camera is easy to build and easy to use, it is widely used in real life. However, the reconstruction system based on a sparse multi-view camera requires complete single-view human body data in the field of view of each camera. Therefore, a certain distance is required between the person and each camera, resulting in a limited pixel area occupied by the face and hands of the person in the camera. The reconstructed human body model lacks details of the face and hands and has low accuracy. Summary of the invention

[0006] The embodiments of the present application provide a method and device for three-dimensional reconstruction of a human body, so as to improve the reconstruction accuracy of a three-dimensional model of a human body.

[0007] In a first aspect, an embodiment of the present application provides a method for three-dimensional reconstruction of a human body, comprising:

[0008] Obtain N consecutive frames of target RGBD images of the target object maintaining a set posture. Each frame of the target RGBD image is obtained by stitching together one RGBD image synchronously acquired by each calibrated RGBD camera;

[0009] Convert each frame of target RGBD image into point cloud data, and fuse N consecutive frames of point cloud data into one frame of target point cloud data;

[0010] Segmenting the target point cloud data to obtain human body point cloud data of the target object, and generating an initial human body model based on the human body point cloud data;

[0011] Extracting a 2D feature point set of a target area from a target RGB image corresponding to the target point cloud data, and converting the 2D feature point set into a 3D feature point set according to a conversion relationship between an RGB image and an RGBD image;

[0012] Acquire point cloud data of the target area from the target point cloud data according to the extreme value pixel point set in the target RGB image;

[0013] Generate a local model of the target object according to the 3D feature point set and the point cloud data of the target area;

[0014] The local model is aligned with the initial human body model to generate a target human body model.

[0015] In a second aspect, an embodiment of the present application provides a reconstruction terminal, including a display, a memory, a processor, and at least one external communication interface, wherein the external communication interface, the display, and the memory are connected to the processor via a bus:

[0016] The memory stores a computer program, and the processor implements the following operations by executing the computer program:

[0017] Acquire, through the at least one external communication interface, N consecutive frames of target RGBD images of the target object maintaining a set posture, each frame of the target RGBD image being obtained by stitching together one RGBD image synchronously acquired by each calibrated RGBD camera;

[0018] Convert each frame of target RGBD image into point cloud data, and fuse N consecutive frames of point cloud data into one frame of target point cloud data;

[0019] Segmenting the target point cloud data to obtain human body point cloud data of the target object, and generating an initial human body model based on the human body point cloud data;

[0020] Extracting a 2D feature point set of the target area from the target RGB image corresponding to the target point cloud data, and converting the 2D feature point set into a 3D feature point set according to a conversion relationship between the RGB image and the RGBD image;

[0021] Acquire point cloud data of the target area from the target point cloud data according to the extreme value pixel point set in the target RGB image;

[0022] A local model of the target object is generated according to the 3D feature point set and the point cloud data of the target area; the local model is aligned with the initial human body model to generate a target human body model, and the target human body model is displayed on a display.

[0023] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a method for three-dimensional reconstruction of a human body.

[0024] In the above-mentioned embodiment of the present application, a combination of global reconstruction and local reconstruction is adopted to realize three-dimensional reconstruction of the human body. In the global reconstruction process, N consecutive frames of target RGBD images of the target object are obtained, and each frame of the RGBD image is converted into point cloud data and then fused and segmented to obtain human body point cloud data of the target object. Based on the human body point cloud data, an initial human body model is generated. Since each frame of the target RGBD image is obtained by splicing one RGBD image synchronously acquired by each calibrated RGBD camera, each frame of the target RGBD image contains human body data from various perspectives, thereby ensuring the integrity of the initial human body model. In the local reconstruction process, according to the target RGB image corresponding to the fused target point cloud data, a 2D feature point set of the target area is extracted, and a 3D feature point set is obtained after conversion. According to the 3D feature point set and the obtained point cloud data of the target area, a local model is generated, and the local model is aligned with the initial human body model. The detailed information of the target area in the initial human body model is enriched by the local model, thereby obtaining a high-precision target human body model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 The architecture diagram of the reconstruction system of the sparse multi-view camera provided in the embodiment of the present application is exemplarily shown;

[0027] Figure 2 The flowchart of the method for three-dimensional reconstruction of a human body provided by an embodiment of the present application is exemplarily shown;

[0028] Figure 3 The flowchart of the method for fusing N consecutive frames of point-by-point data provided by the embodiment of the present application is exemplarily shown;

[0029] Figure 4 The structure diagram of the Function4D network model provided in the embodiment of the present application is exemplarily shown;

[0030] Figure 5 The flowchart of the method for extracting point transport data of a target area provided by an embodiment of the present application is exemplarily shown;

[0031] Figure 6 The flowchart of the method for aligning a local model with an initial human body model provided in an embodiment of the present application is exemplarily shown;

[0032] Figure 7 The schematic diagram of updating the geometric vertices in the initial human body model provided by the embodiment of the present application is exemplarily shown;

[0033] Figure 8 The structural diagram of the reconstruction terminal provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0034] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0035] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0036] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0037] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to orders other than those given in the diagrams or descriptions of the embodiments of this application.

[0038] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0039] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0040] At present, there are three main types of human body 3D reconstruction, depending on the configuration of the reconstruction system:

[0041] 1) Reconstruction is performed based on a "bird cage" reconstruction system with multiple array cameras. During the reconstruction process, the person to be reconstructed needs to remain still in the "bird cage" for several seconds. After the camera has collected the human body data, the point cloud data of the person to be reconstructed is obtained based on the calibration relationship between the cameras in the reconstruction system and the triangulation method. After denoising, meshing, triangulation and other operations, a three-dimensional model of the human body is generated. A high-precision three-dimensional model of the human body can be obtained through the "bird cage" reconstruction system with multiple array cameras, but the collected object needs to be still, and the time required for model reconstruction varies depending on the application scenario. For example, the modeling time in a movie or animation scene can be as long as several hours, and personnel with professional knowledge are required to scan the collected object. Since the scanner and the camera used are relatively expensive, it is difficult to use them in people's daily lives.

[0042] 2) Reconstruction based on a single camera reconstruction system. A human 3D reconstruction system built with a single camera or TOF camera is used to collect human data. During the collection process, the person to be reconstructed needs to remain stationary for a period of time, and the collection personnel hold the camera around the person to be reconstructed to try to cover the human body in all angles to obtain complete human data; or, the person to be reconstructed rotates in front of the camera and does actions such as leaning over, raising legs, and raising arms to obtain complete human data. Based on the obtained human data, point cloud data is generated, the point cloud is denoised, and feature point extraction, iterated closest points (ICP), truncated signed distance function (TSDF), meshing, triangulation and other operations are performed to generate a human 3D model. Through a single camera reconstruction system, if the data collected by an ordinary RGB camera is used to directly generate a human 3D model, the fidelity of the model is low, and the scale and shape of the model are difficult to align with the collected image. If the data collected by RGBD is used, whether the camera is stationary and the person is rotating, or the person is stationary and the camera is rotating, it is difficult to obtain complete human data from all angles, and the reconstructed model is often incomplete. In addition, the accuracy of the reconstruction model of the single-camera reconstruction system is unstable and requires a certain amount of learning cost. Currently, it is not suitable for use in daily life.

[0043] 3) Reconstruction based on a reconstruction system of sparse multi-view cameras. This system consists of 3-8 structured light cameras or TOF cameras. These cameras can usually be arranged in a circle so that each camera can obtain human body data from different perspectives. During the acquisition process, the person to be reconstructed does not need to remain still. After obtaining the human body data from multiple perspectives, the point cloud data of the person to be reconstructed is obtained based on the calibration relationship between the cameras and the triangulation method. After denoising, segmentation, meshing, triangulation and other operations, a three-dimensional human body model is generated. Compared with the first two methods, the reconstruction system based on sparse multi-view cameras is easy to build and easy to use, and can be widely used in real life. However, in the reconstruction system of sparse multi-view cameras, because the position of each camera is fixed, the time for collecting effective human body data is short, and it is difficult to obtain full-view data of the complete human body, resulting in poor integrity of the reconstructed model.

[0044] At present, in order to obtain a complete three-dimensional human body model, most reconstruction systems for sparse multi-view cameras use the Poisson reconstruction method. However, this reconstruction method makes the model too smooth and loses the outer contour information of the human body, thereby reducing the fidelity of the model. In addition, during the reconstruction process, due to the certain distance between the person and each camera, the pixel area occupied by the person's face and hands in the camera is limited, and the reconstructed human body model lacks facial and hand details.

[0045] In order to improve the integrity of the reconstructed three-dimensional human model in the reconstruction system of sparse multi-view cameras, and solve the problem of missing details in local areas such as the head and hands of the three-dimensional human model, and ensure the reconstruction accuracy of the model, the embodiment of the present application provides a three-dimensional human body reconstruction method and device. A reconstruction system based on 3 to 6 civilian-grade RGBD cameras can reconstruct a high-fidelity three-dimensional human body model with outer contours such as clothing, and the model has high-precision facial details and hand details. During the entire reconstruction process, the reconstruction personnel have no learning cost. Compared with a system that uses a single RGBD camera for three-dimensional reconstruction of the human body, a more accurate and high-integrity three-dimensional human body model can be restored. There is no need for the reconstruction personnel to hold the camera and turn around the person to be reconstructed, or for the person to be reconstructed to stand on a rotating device such as a roulette wheel, and the reconstruction process is simple.

[0046] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0047] Figure 1 The architecture diagram of the reconstruction system of the sparse multi-view camera provided in the embodiment of the present application is exemplarily shown; Figure 1 As shown in the figure, the system includes 4 RGBD cameras, which are placed at 4 viewing angles to form a square. The interval between adjacent RGBD cameras is 2-3 meters, and all RGBD cameras face the center of the square. Each RGBD camera is connected by an audio cable to synchronously collect data.

[0048] It should be noted that Figure 1 This is just an example. The embodiment of the present application does not impose any restrictive requirements on the number of RGBD cameras and the enclosed pattern. For example, the number of RGBD cameras can also be 3, 6, etc., and the enclosed pattern can also be a regular shape such as a rectangle or a circle. Furthermore, in addition to using an audio cable to connect, each RGBD camera can also be connected using a network cable, Bluetooth, etc. for data synchronization.

[0049] In the above reconstruction system, the resolution requirement for the RGBD camera is relatively low. Generally, 3-6 civilian-grade RGBD cameras can meet the reconstruction requirements, and the cost is relatively low.

[0050] After the reconstruction system is built, the RGBD cameras in the system are calibrated to unify the camera coordinates. In an optional implementation, a main camera is selected from the RGBD cameras, and the coordinate systems of the other cameras are aligned to the coordinates of the main camera.

[0051] For example, Figure 1The first camera in is the main camera, and the pose relationship of the second camera to the fourth camera relative to the main camera is determined respectively to obtain the calibration matrix. Based on the calibration matrix, the coordinate systems of the second camera to the fourth camera are converted to the coordinate system of the main camera to complete the unified calibration of the coordinate system.

[0052] The embodiments of the present application do not impose restrictive requirements on the algorithm for calibrating multiple cameras. For example, a dual-target calibration method can be used to calibrate using a chessboard. By identifying the corner points of the chessboard, a matrix equation is established and solved to obtain a calibration matrix between the two cameras (including a rotation matrix r and a translation vector t). The point cloud data obtained by the RGBD camera can also be used. A cubic calibration object can be used to extract the normal vector of the calibration object plane, align the two RGBD cameras to obtain the normal vector of the plane point cloud, obtain the rotation matrix r, and then determine the translation vector t based on the intersecting vertices on the calibration object plane to obtain the calibration matrix.

[0053] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the type of RGBD camera. For example, it can be a TOF camera (such as a Kinect camera), or a structured light camera or a binocular camera.

[0054] based on Figure 1 The reconstruction system shown, Figure 2 The flowchart of the human body three-dimensional reconstruction method provided in the embodiment of the present application is exemplarily shown. The process is executed by the reconstruction terminal and mainly includes the following steps:

[0055] S201: Acquire N consecutive frames of target RGBD images of a target object maintaining a set posture, where each frame of the target RGBD image is obtained by stitching together one RGBD image synchronously acquired by each calibrated RGBD camera.

[0056] In an optional embodiment, the target object enters Figure 1 In the center of the reconstruction system shown, the arms are opened at a certain angle, in a "T pose" or "A pose", with the palms open, and each RGBD camera continuously collects N frames of RGBD images. Since each RGBD camera in the reconstruction system has been calibrated, the 1 RGBD image synchronously collected by each RGBD camera can be spliced ​​based on the calibration matrix to obtain a complete target RGBD image. After continuously collecting N frames, N consecutive frames of target RGBD images can be obtained.

[0057] S202: Convert each frame of the target RGBD image into point cloud data, and fuse N consecutive frames of point cloud data into one frame of target point cloud data.

[0058] When executing S202, each frame of the target RGBD image is converted into point cloud data to obtain N consecutive frames of point cloud data. An iterative closest point (ICP) algorithm is used to determine the non-rigid conversion relationship between the N consecutive frames of point cloud data. Based on the determined conversion relationship, the N consecutive frames of point cloud data are fused into one frame of target point cloud data.

[0059] For specific implementation, see Figure 3 :

[0060] S2021: Select a reference frame from N consecutive frames of point cloud data.

[0061] Taking N equals 3 as an example, the second frame of point cloud data is selected as the reference frame.

[0062] S2022: For each non-reference frame, perform the following operations: sample the point cloud data of a non-reference frame to obtain a first sampling point set, use the first sampling point set as an initial sampling position, sample the point cloud data of the reference frame to obtain a second sampling point set, and determine a conversion relationship between a non-reference frame and a reference frame based on the first sampling point set and the second sampling point set.

[0063] Still taking N equal to 3 as an example, the 1st frame point cloud data and the 3rd frame point cloud data are non-reference frame point cloud data. First, uniform sampling is performed in the 1st frame point cloud data to obtain the first sampling point set corresponding to the 1st frame point cloud data. Each acquisition point in the first sampling point set is used as the initial sampling position of the 2nd frame point cloud data, and the 2nd frame point cloud data is sampled to obtain the second sampling point set. Then, based on the first sampling point set and the second sampling point set, the ICP algorithm is used to determine the conversion relationship between the 1st frame point cloud data and the 2nd frame point cloud data, which is recorded as T. 1-2 Similarly, we can get the conversion relationship between the point cloud data of the third frame and the point cloud data of the second frame, which is recorded as T 3-2 .

[0064] S2023: According to the conversion relationship between each non-reference frame and the reference frame, the point cloud data of each non-reference frame is fused into the point cloud data of the reference frame to obtain the target point cloud data.

[0065] Still taking N equal to 3 as an example, when executing S3023, the conversion relationship T is determined respectively. 1-2 And the conversion relationship T 3-2 The inverse matrix of T 1-2 The inverse matrix of T is used to fuse the point cloud data of the first frame into the point cloud data of the second frame. 3-2 The inverse matrix of is used to fuse the third frame of point cloud data into the second frame of point cloud data to obtain a frame of target point cloud data.

[0066] S203: Segment the target point cloud data to obtain human body point cloud data of the target object, and generate an initial human body model based on the human body point cloud data.

[0067] When executing S203, the target point cloud is segmented using point cloud segmentation technology (such as RanSaC algorithm, KDTree algorithm, OCTree algorithm, etc.) to obtain human body point cloud data of the target object, and a complete initial human body model is reconstructed based on the human body point cloud data.

[0068] In an optional implementation, the global reconstruction process of S201 to S203 may adopt a Function4D deep network model based on implicit functions, the structure of which is as follows: Figure 4 The Function4D model is used to obtain the Truncated Signed Distance Function (TSDF) values ​​of all voxels of the target object in the voxel space, and then the vertices and faces of the human body model are extracted according to the Marching Cube algorithm to obtain a complete initial human body model.

[0069] S204: extracting a 2D feature point set of the target area from the target RGB image corresponding to the target point cloud data, and converting the 2D feature point set into a 3D feature point set according to a conversion relationship between the RGB image and the RGBD image.

[0070] In some embodiments, after executing S202 to obtain the target point cloud data, re-rendering is performed to obtain an RGBD image and an RGB image under the front perspective of the target object, and the obtained RGBD image and RGB image are aligned so that the resolution of the RGB image and the RGBD image are the same and the perspective is consistent. In this way, the pixels in the RGB image correspond one-to-one to the pixels in the RGBD image. Since the pixels in the RGBD image correspond one-to-one to the data points in the target point cloud data, point cloud data with one-to-one correspondence between the pixels in the RGB image can be obtained.

[0071] When executing S204, first, the target RGB image (i.e., the aligned RGB image) corresponding to the target point cloud data is identified to obtain a target area image, a 2D feature point set is extracted from the target area image, and then, the 2D feature point set is converted into a 3D feature point set using the conversion relationship between the RGB image and the RGBD image.

[0072] Optionally, the target region is a region where details are missing, such as a head, a hand, etc. of a human model. Taking the target region as a face region as an example, when extracting a set of 2D feature points, a 51 or 68 feature point detection algorithm may be used.

[0073] S205: Acquire point cloud data of the target area from the target point cloud data according to the extreme value pixel point set in the target RGB image.

[0074] In an optional implementation, when executing S205, a detection frame of the target area is determined according to the extreme pixel set in the target RGB image, and the point cloud data of the target area is obtained from the target point cloud data based on the determined detection frame. Figure 5 :

[0075] S2051: Determine an extreme pixel point set in the target RGB image according to the pixel coordinates of each 2D feature point in the 2D feature point set of the target area.

[0076] In an optional implementation, if the pixel coordinates of a target 2D feature point are greater than the pixel coordinates of the 2D feature points in the preset neighborhood, the target 2D feature point is determined to be an extreme pixel point. After traversing the 2D feature point set, an extreme pixel point set is obtained.

[0077] S2052: Determine the maximum and minimum values ​​of the U coordinate, and the maximum and minimum values ​​of the V coordinate from the extreme pixel point set.

[0078] When executing S2052, the U coordinates and V coordinates of each extreme pixel point in the extreme pixel point set are compared to determine the maximum value (denoted as max_u) and minimum value (denoted as min_u) of the U coordinate, as well as the maximum value (denoted as max_v) and minimum value (denoted as min_v) of the V coordinate.

[0079] S2053: Increase the maximum value of the U coordinate and the V coordinate by the first pixel value, and decrease the maximum value of the U coordinate and the V coordinate by the second pixel value, to obtain a detection frame corresponding to the target area.

[0080] In an optional implementation, when executing S2053, max_u and max_v are increased by 5-10 pixel values, and min_u and min_v are decreased by 5-10 pixel values, respectively, to obtain a detection frame corresponding to the target area.

[0081] It should be noted that the sizes of the first pixel value and the second pixel value can be set according to actual needs and can be different.

[0082] S2054: Obtaining point cloud data of the target area from the target point cloud data according to the detection frame.

[0083] Since the RGBD image and the RGB image under the frontal perspective corresponding to the target point cloud data are aligned, there are point clouds in the target point cloud data that correspond one-to-one to the pixel points in the RGB image. Therefore, according to the determined detection frame, the point cloud data of the target area can be obtained from the target point cloud data.

[0084] S206: Generate a local model of the target object according to the 3D feature point set and the point cloud data of the target area.

[0085] When executing S206, each 3D feature point in the 3D feature point set is rigidly fitted with the 3D feature points in the preset parameterized model corresponding to the target area to obtain an updated parameterized model, point cloud data of the updated parameterized model is extracted, and the extracted point cloud data and the point cloud data of the target area are non-rigidly fitted to obtain a local model of the target object.

[0086] Optionally, when the target area is a face area, the preset parameterized model may be a FLAM model; and when the target area is a hand area, the preset parameterized model may be a MANO model.

[0087] S207: Align the local model with the initial human body model to generate a target human body model.

[0088] When executing S207, the vertices in the local model are aligned with the vertices in the initial human body model, the vertices of the target area in the initial human body model are extracted, and then these vertices are deformed toward the local model, and the target area is reconstructed to obtain a high-precision target human body model. Figure 6 :

[0089] S2071: Extracting each first geometric vertex p corresponding to the target area from the initial human body model.

[0090] S2072: Select the second geometric vertices q corresponding to each first geometric vertex in the local model.

[0091] In an optional implementation, the second geometric vertex q is a point in the local model that is closest to the first geometric vertex.

[0092] S2073: For each first geometric vertex, the first geometric vertex is directed toward the corresponding second geometric vertex q and the normal vector of the corresponding second geometric vertex Project the composed plane R to obtain the updated first geometric vertex q new .

[0093] The update process of the first geometric vertex can be found in Figure 7 .

[0094] S2074: Based on each updated first geometric vertex, deform the initial human body model to obtain a target human body model.

[0095] In S2074, after the initial human body model is deformed, the organizational form of the face patch is kept unchanged, thereby obtaining a high-precision target human body model with rich details of the face and hands.

[0096] In the above-mentioned embodiment of the present application, the three-dimensional reconstruction of the human body based on sparse viewpoints is divided into two parts: global reconstruction and local reconstruction. First, global reconstruction is performed: the target object enters the reconstruction system constructed by each RGBD camera under different viewing angles, maintains a fixed posture, and after collecting three consecutive frames of RGBD images, the target object can leave the reconstruction system without having to remain still; the point cloud data of the three consecutive frames of RGBD images are fused to obtain a frame of target point cloud data, and then reconstruction is performed based on the target point cloud data to obtain the initial human body model. Since each frame of RGBD image is obtained by splicing RGBD images synchronously collected by RGBD cameras under various viewing angles, each frame of RGBD image contains complete human body data. After the point cloud data extracted from the RGBD image is fused, the target point cloud data contains complete human body data, thereby ensuring the integrity of the initial human body model. Then, local reconstruction is performed: based on the fused target point cloud data, the RGB image and RGBD image of the target object under the frontal view are obtained, and the RGB image and RGBD image are aligned. The 2D feature points of the target area are extracted from the aligned RGB image to obtain a 2D feature point set. According to the conversion relationship between the RGB image and the RGBD image, the 2D feature point set is converted into a 3D feature point set. Further, the point cloud data of the target area is selected from the target point cloud data, and the local model is reconstructed based on the 3D feature point set and the selected point cloud data. Finally, the geometric vertices in the initial human body model and the local model are aligned, the geometric vertices in the target area of ​​the initial human body model are updated, and the initial human body model is deformed according to the updated geometric vertices to generate a high-precision target human body model with rich details of the face and hands, thereby improving the authenticity of the reconstructed model.

[0097] The human body three-dimensional reconstruction method provided in the above-mentioned embodiments of the present application can not only reconstruct a human body model with clear outer contours such as clothing and local details without sacrificing authenticity, but also obtain local models such as the head and hands with high-precision geometric details under the collection distance of a civilian-grade RGBD camera, thereby obtaining a complete high-precision three-dimensional human body model.

[0098] It should be noted that the embodiments of the present application can not only perform three-dimensional reconstruction for a single target object, but also perform three-dimensional reconstruction for multiple target objects in the acquisition scene. It only requires that there is no occlusion relationship between the heads and hands of the multiple target objects, and each RGBD camera can collect complete data of each target object.

[0099] Based on the same technical concept, an embodiment of the present application provides a reconstruction terminal, which can execute the method flow of three-dimensional reconstruction of the human body provided in the embodiment of the present application and achieve the same technical effect, which will not be repeated here.

[0100] See also Figure 8 The reconstruction terminal includes a processor 801, a memory 802, a display 803 and at least one external communication interface 804. The at least one external communication interface 804, the display 803 and the memory 802 are connected to the processor 801 through a bus 805; the at least one external communication interface 804 is configured to obtain N consecutive frames of target RGBD images of the target object maintaining a set posture, the display 803 is configured to display a target human body model, and a computer program is stored in the memory 802. The processor 801 implements the human body three-dimensional reconstruction method in the aforementioned embodiment by executing the computer program.

[0101] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0102] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0104] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A method for three-dimensional reconstruction of a human body, characterized in that: include: Obtain N consecutive frames of target RGBD images of the target object maintaining a set posture. Each frame of the target RGBD image is obtained by stitching together one RGBD image synchronously acquired by each calibrated RGBD camera; Convert each frame of target RGBD image into point cloud data, and fuse N consecutive frames of point cloud data into one frame of target point cloud data; Segmenting the target point cloud data to obtain human body point cloud data of the target object, and generating an initial human body model according to the human body point cloud data; Extracting a 2D feature point set of the target area from the target RGB image corresponding to the target point cloud data, and converting the 2D feature point set into a 3D feature point set according to a conversion relationship between the RGB image and the RGBD image; Acquire point cloud data of the target area from the target point cloud data according to the extreme value pixel point set in the target RGB image; Generate a local model of the target object according to the 3D feature point set and the point cloud data of the target area; align the local model with the initial human body model to generate a target human body model; According to the extreme pixel point set in the target RGB image, the point cloud data of the target area is obtained from the target point cloud data: If the pixel coordinates of a target 2D feature point are greater than the pixel coordinates of the 2D feature points in the preset neighborhood, the target 2D feature point is determined to be an extreme pixel point; after traversing the 2D feature point set, an extreme pixel point set is obtained; from the extreme pixel point set, the maximum and minimum values ​​of the U coordinate, and the maximum and minimum values ​​of the V coordinate are determined; the U coordinates and the V coordinates of each extreme pixel point in the extreme pixel point set are compared, and the maximum and minimum values ​​of the U coordinate, and the maximum and minimum values ​​of the V coordinate are determined respectively; the maximum value of the U coordinate and the V coordinate is increased by a first pixel value, and the maximum value of the U coordinate and the V coordinate is reduced by a second pixel value, to obtain a detection frame corresponding to the target area; and the point cloud data of the target area is obtained from the point cloud data according to the detection frame; The RGBD cameras are spaced at a preset distance from each other, and the RGBD cameras form a regular shape and face the center of the shape.

2. The method according to claim 1, characterized in that The step of fusing N consecutive frames of point cloud data into one frame of target point cloud data includes: Select a reference frame from N consecutive frames of point cloud data; For each non-reference frame, the following operations are performed: sampling the point cloud data of the non-reference frame to obtain a first sampling point set, using the first sampling point set as an initial sampling position, sampling the point cloud data of the reference frame to obtain a second sampling point set, and determining a conversion relationship between the non-reference frame and the reference frame according to the first sampling point set and the second sampling point set; According to the conversion relationship between each non-reference frame and the reference frame, the point cloud data of each non-reference frame is fused into the point cloud data of the reference frame to obtain the target point cloud data.

3. The method according to claim 1, characterized in that The generating a local model of the target object according to the 3D feature point set and the point cloud data of the target area comprises: Performing rigid fitting on each 3D feature point in the 3D feature point set and a 3D feature point in a preset parameterized model corresponding to the target area to obtain an updated parameterized model; The point cloud data of the updated parameterized model is extracted, and the extracted point cloud data and the point cloud data of the target area are non-rigidly fitted to obtain a local model of the target object.

4. The method according to claim 1, characterized in that The step of aligning the local model with the initial human body model to generate a target human body model comprises: Extracting each first geometric vertex corresponding to the target area from the initial human body model; Selecting second geometric vertices in the local model corresponding to each of the first geometric vertices; For each first geometric vertex, project the first geometric vertex onto a plane consisting of a corresponding second geometric vertex and a normal vector of the corresponding second geometric vertex to obtain an updated first geometric vertex; Based on each updated first geometric vertex, the initial human body model is deformed to obtain a target human body model.

5. A reconstruction terminal, characterized in that: The device comprises a display, a memory, a processor and at least one external communication interface, wherein the external communication interface, the display and the memory are connected to the processor via a bus: The memory stores a computer program, and the processor implements the following operations by executing the computer program: Acquire, through the at least one external communication interface, N consecutive frames of target RGBD images of the target object maintaining a set posture, each frame of the target RGBD image being obtained by stitching together one RGBD image synchronously acquired by each calibrated RGBD camera; Convert each frame of target RGBD image into point cloud data, and fuse N consecutive frames of point cloud data into one frame of target point cloud data; Segmenting the target point cloud data to obtain human body point cloud data of the target object, and generating an initial human body model according to the human body point cloud data; Extracting a 2D feature point set of the target area from the target RGB image corresponding to the target point cloud data, and converting the 2D feature point set into a 3D feature point set according to a conversion relationship between the RGB image and the RGBD image; Acquire point cloud data of the target area from the target point cloud data according to the extreme value pixel point set in the target RGB image; Generate a local model of the target object according to the 3D feature point set and the point cloud data of the target area; align the local model with the initial human body model to generate a target human body model, and display it on a display; the processor obtains the point cloud data of the target area from the target point cloud data according to the extreme pixel points in the target RGB image, and is specifically configured as follows: If the pixel coordinates of a target 2D feature point are greater than the pixel coordinates of the 2D feature points in the preset neighborhood, the target 2D feature point is determined to be an extreme pixel point; after traversing the 2D feature point set, an extreme pixel point set is obtained; from the extreme pixel point set, the maximum and minimum values ​​of the U coordinate, and the maximum and minimum values ​​of the V coordinate are determined; the U coordinates and the V coordinates of each extreme pixel point in the extreme pixel point set are compared, and the maximum and minimum values ​​of the U coordinate, and the maximum and minimum values ​​of the V coordinate are determined respectively; the maximum value of the U coordinate and the V coordinate is increased by a first pixel value, and the maximum value of the U coordinate and the V coordinate is reduced by a second pixel value, to obtain a detection frame corresponding to the target area; and the point cloud data of the target area is obtained from the point cloud data according to the detection frame; The RGBD cameras are spaced at a preset distance from each other, and the RGBD cameras form a regular shape and face the center of the shape.

6. The reconstruction terminal according to claim 5, characterized in that: The processor fuses N consecutive frames of point cloud data into one frame of target point cloud data, and is specifically configured as follows: Select a reference frame from N consecutive frames of point cloud data; For each non-reference frame, the following operations are performed: sampling the point cloud data of the non-reference frame to obtain a first sampling point set, using the first sampling point set as an initial sampling position, sampling the point cloud data of the reference frame to obtain a second sampling point set, and determining a conversion relationship between the non-reference frame and the reference frame according to the first sampling point set and the second sampling point set; According to the conversion relationship between each non-reference frame and the reference frame, the point cloud data of each non-reference frame is fused into the point cloud data of the reference frame to obtain the target point cloud data.

7. The reconstruction terminal according to claim 5, characterized in that: The processor aligns the local model with the initial human body model to generate a target human body model, and is specifically configured as follows: Extracting each first geometric vertex corresponding to the target area from the initial human body model; Selecting second geometric vertices in the local model corresponding to each of the first geometric vertices; For each first geometric vertex, project the first geometric vertex onto a plane consisting of a corresponding second geometric vertex and a normal vector of the corresponding second geometric vertex to obtain an updated first geometric vertex; Based on each updated first geometric vertex, the initial human body model is deformed to obtain a target human body model.

Images