Face reconstruction method and system

Abstract

This application discloses a face reconstruction method and system. Based on the execution of this face reconstruction method, in response to a face reconstruction request, multiple depth maps corresponding to the object to be reconstructed are acquired, and the depth maps corresponding to the object to be reconstructed are converted into point cloud data. Then, point cloud registration and point cloud fusion processing are performed on the point cloud data to obtain panoramic point cloud data. Face reconstruction is then performed based on this panoramic point cloud data. Specifically, the real-time acquisition and conversion of the depth maps of the object to be reconstructed into point cloud data for processing achieves proportional face reconstruction. The point cloud registration and point cloud fusion processing overcomes the problem of difficulty in registering point clouds from different perspectives due to their different structures. Furthermore, it preserves point clouds in non-overlapping areas, eliminating the layering phenomenon during the fusion of point cloud data corresponding to camera markers at different angles, making the overall point cloud smoother and flatter. Thus, real-time acquisition, processing, and proportional reconstruction of face reconstruction are achieved, improving the precision of face reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer vision processing technology, and more specifically, to a face reconstruction method and system. Background Technology

[0002] Face reconstruction has important applications in film, games, animation, AR, and VR. Currently, there are several methods for face reconstruction: model-based face reconstruction and deep learning-based end-to-end face reconstruction. Model-based face reconstruction, such as the 3D face standard model CANDIDE and the 3D face statistical model 3DMM, requires building a face database. The reconstructed faces often closely resemble those in the database, making true proportional and detailed reconstruction difficult. Deep learning-based end-to-end face reconstruction generally uses deep learning algorithms, such as the regression model 3DMM CNN, the 3D dense face alignment algorithm 3DDFA, and the face alignment algorithm PRNet. However, the reconstructed faces also tend to closely resemble the training data, making true proportional and detailed reconstruction difficult. Therefore, existing model-based and deep learning-based end-to-end face reconstruction methods suffer from the problem of reconstructed faces closely resembling the training data and having low detail. Summary of the Invention

[0003] The purpose of this application is to provide a face reconstruction method and system. Based on the execution of this face reconstruction method, face reconstruction can be carried out in real time based on the object to be reconstructed, processed, and reconstructed proportionally, thereby improving the accuracy of face reconstruction.

[0004] To achieve the above objectives, this application provides a face reconstruction method, the method comprising:

[0005] In response to a face reconstruction request, multiple depth maps corresponding to the object to be reconstructed are obtained; wherein, the multiple depth maps are obtained through a preset number of cameras, and each depth map includes a camera identifier;

[0006] The multiple depth maps are classified to determine a target depth map set; the target depth map set includes a preset number of target depth maps, and each target depth map in the target depth map set corresponds to a different camera identifier;

[0007] Obtain preset camera parameters, and generate a target point cloud data set based on the preset camera parameters and the target depth map set;

[0008] Point cloud registration and point cloud fusion are performed on multiple point cloud data included in the target point cloud dataset to obtain the panoramic point cloud data of the object to be reconstructed.

[0009] Based on a preset algorithm, the panoramic cloud data of the object to be reconstructed is processed to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

[0010] Optionally, each of the multiple depth maps further includes a timestamp; classifying the multiple depth maps to determine the target depth map set includes:

[0011] Based on the timestamp and camera identifier of each depth map, the multiple depth maps are classified to determine the target depth map set.

[0012] Optionally, classifying the multiple depth maps based on the timestamp and camera identifier of each depth map to determine the target depth map set includes:

[0013] Based on the camera identifier, the multiple depth maps are classified to determine multiple initial depth map sets; each initial depth map set corresponds to the camera identifier.

[0014] Obtain the first timestamp corresponding to the first depth map in the first initial depth map set; the first initial depth map set is any one of the plurality of initial depth map sets; the first depth map is any one of the plurality of depth maps included in the first initial depth map set;

[0015] Based on the first timestamp and the preset time period, a target depth map set is determined from the plurality of initial depth map sets.

[0016] Optionally, the preset camera parameters include camera intrinsic parameters corresponding to the camera identifier and camera extrinsic parameters corresponding to the camera identifier. The step of obtaining the preset camera parameters and generating a target point cloud data set based on the preset camera parameters and the target depth map set includes:

[0017] Obtain the intrinsic parameters of the reference camera corresponding to the reference camera identifier and the intrinsic parameters of each camera identifier; the reference camera identifier is the identifier corresponding to a preset reference camera.

[0018] Based on the reference camera intrinsic parameters and the camera intrinsic parameters corresponding to each camera identifier, the multiple target depth maps included in the target depth map set are transformed to generate a target point cloud data set; each point cloud data included in the target point cloud data set corresponds to a camera identifier.

[0019] Optionally, the step of performing point cloud registration and point cloud fusion processing on multiple point cloud data included in the target point cloud dataset to obtain panoramic point cloud data of the object to be reconstructed includes:

[0020] Based on the camera extrinsic parameters corresponding to each camera identifier, coarse registration of multiple point cloud data included in the target point cloud data set is performed to obtain a coarsely registered target point cloud data set.

[0021] The coarse registration target point cloud data set includes multiple coarse registration point cloud data, which are then subjected to fine registration and point cloud fusion processing to obtain the panoramic point cloud data of the object to be reconstructed.

[0022] Optionally, the step of performing coarse point cloud registration on multiple point cloud data included in the target point cloud data set based on the camera extrinsic parameters corresponding to each camera identifier to obtain a coarsely registered target point cloud data set includes:

[0023] Obtain the reference camera extrinsic parameters corresponding to the reference camera identifier and the camera extrinsic parameters corresponding to each camera identifier; the reference camera identifier is the identifier corresponding to a preset reference camera.

[0024] Based on the reference camera extrinsic parameters and the camera extrinsic parameters corresponding to each camera identifier, determine the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier, respectively;

[0025] Based on the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier, the multiple point cloud data included in the target point cloud data set are transformed to obtain the coarse registration target point cloud data set; the coarse registration target point cloud data set includes the preset number of coarse registration point cloud data.

[0026] Optionally, the step of performing fine point cloud registration and point cloud fusion processing on multiple coarsely registered point cloud datasets included in the coarsely registered target point cloud dataset to obtain panoramic point cloud data of the object to be reconstructed includes:

[0027] Obtain the reference point cloud data and the first point cloud data from the coarse registration target point cloud data set, wherein the reference point cloud data and the first point cloud data are two adjacent coarse registration point cloud data in the coarse registration target point cloud data set, and the reference point cloud data is the point cloud data corresponding to the reference camera identifier;

[0028] A first transformation matrix is ​​determined based on the reference point cloud data and the first point cloud data;

[0029] The first point cloud data is finely registered according to the transformation matrix to obtain the finely registered first point cloud data.

[0030] The reference point cloud data and the finely registered first point cloud data are subjected to point cloud fusion processing to obtain the updated reference point cloud data.

[0031] Based on the updated reference point cloud data and the remaining coarse registration point cloud data in the coarse registration target point cloud data set, a temporary fused point cloud data set is generated; the remaining coarse registration point cloud data is the coarse registration point cloud data in the coarse registration target point cloud data set that has not undergone fine registration and point cloud fusion processing.

[0032] The updated baseline point cloud data and the temporary fused point cloud data included in the temporary fused point cloud data set are sequentially subjected to fine point cloud registration and point cloud fusion processing. When the temporary fused point cloud data included in the temporary fused point cloud data set is 1, the panoramic point cloud data of the object to be reconstructed is obtained.

[0033] Optionally, the step of processing the panoramic cloud data of the object to be reconstructed based on a preset algorithm to obtain the reconstructed face mesh data corresponding to the object to be reconstructed includes:

[0034] The panoramic cloud data of the object to be reconstructed is processed based on the Poisson reconstruction algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

[0035] Optionally, the preset quantity is 12.

[0036] This application also provides a face reconstruction system, the system comprising:

[0037] The acquisition module is used to acquire multiple depth maps corresponding to the object to be reconstructed in response to a face reconstruction request; wherein, the multiple depth maps are acquired by a preset number of cameras, and each depth map includes a camera identifier;

[0038] A frame synchronization module is used to classify the multiple depth maps and determine a target depth map set; the target depth map set includes a preset number of target depth maps, and each target depth map in the target depth map set corresponds to a different camera identifier;

[0039] The point cloud data acquisition module is used to acquire preset camera parameters and generate a target point cloud data set based on the preset camera parameters and the target depth map set.

[0040] The point cloud registration and fusion module is used to perform point cloud registration and point cloud fusion processing on multiple point cloud data included in the target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed.

[0041] The face reconstruction module is used to process the panoramic cloud data of the object to be reconstructed based on a preset algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

[0042] This application provides a face reconstruction method and system. Based on the execution of this face reconstruction method, in response to a face reconstruction request, multiple depth maps corresponding to the object to be reconstructed are acquired. These multiple depth maps are acquired through a preset number of cameras, classified to determine a target depth map set, and a target point cloud data set is generated based on the preset camera parameters and the target depth map set. Then, point cloud registration and point cloud fusion processing are performed on the multiple point cloud data included in the target point cloud data set to obtain panoramic point cloud data of the object to be reconstructed. Finally, based on a preset algorithm, the panoramic point cloud data of the object to be reconstructed is processed to obtain the reconstructed face mesh data corresponding to the object to be reconstructed. As can be seen, the embodiments of this application acquire depth maps of the object to be reconstructed in real time, convert them into point cloud data, and process them to achieve proportional face processing. Based on point cloud registration and point cloud fusion processing, the problem of difficulty in registration caused by different structures of point clouds from different perspectives is overcome. Furthermore, the non-overlapping area of ​​the point cloud is preserved, eliminating the layering phenomenon when fusing point cloud data corresponding to camera identifiers at different angles, making the overall point cloud smoother. Thus, real-time acquisition, processing, and proportional reconstruction of face reconstruction are achieved, improving the accuracy of face reconstruction. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 A flowchart of a face reconstruction method provided in this application embodiment;

[0045] Figure 2 A panoramic view of the depth image acquisition scene provided in the embodiments of this application;

[0046] Figure 3 This is a top view of a depth image acquisition scene provided in an embodiment of this application;

[0047] Figure 4 A schematic diagram of a face image corresponding to point cloud data that has not undergone point cloud registration, provided in an embodiment of this application.

[0048] Figure 5 A schematic diagram of a face image corresponding to point cloud data that has been registered with point cloud, provided in an embodiment of this application;

[0049] Figure 6 A schematic diagram of face images corresponding to point cloud data before and after point cloud fusion, provided in an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of a face reconstruction system provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] Existing face reconstruction methods include model-based face reconstruction and deep learning-based end-to-end face reconstruction. Model-based face reconstruction reconstructs faces by building a face database, but the reconstructed faces often closely resemble those in the database, making it difficult to achieve truly proportional and detailed reconstruction. Deep learning-based end-to-end face reconstruction typically uses deep learning algorithms, but the reconstructed faces also tend to resemble the training data. Therefore, existing face reconstruction methods suffer from low detail.

[0053] This application provides a face reconstruction method and system. It collects the depth map of the object to be reconstructed in real time, converts it into point cloud data, and processes it to achieve proportional face reconstruction. Based on point cloud registration and point cloud fusion processing, it overcomes the problem of difficulty in registration of point clouds from different perspectives due to their different structures. It also retains point clouds in non-overlapping areas and eliminates the layering phenomenon when fusing point cloud data corresponding to camera markers at different angles, making the overall point cloud smoother. Thus, it realizes real-time acquisition, processing, and proportional reconstruction of face reconstruction, improving the accuracy of face reconstruction.

[0054] The following is a detailed description of a face reconstruction method in this application:

[0055] Figure 1 This is a flowchart illustrating a face reconstruction method provided in an embodiment of this application. Figure 1 As shown, the face reconstruction method in this embodiment includes:

[0056] S101: In response to a face reconstruction request, obtain multiple depth maps corresponding to the object to be reconstructed; wherein, the multiple depth maps are obtained through a preset number of cameras, and each depth map in the multiple depth maps includes a camera identifier.

[0057] It should be noted that after acquiring a depth map, the depth camera also records the timestamp of the depth map acquisition and stores the acquired depth map on the hard drive of the computer connected to the camera. The depth map can be named and stored using the camera serial number and the timestamp.

[0058] Specifically, in this embodiment of the application, after receiving a face reconstruction request, the face reconstruction system has the subject make different facial expressions in front of the camera array, and acquire multiple depth maps corresponding to the reconstructed object through multiple depth cameras in the camera array. The preset number is 12, that is, depth maps are acquired through 12 depth cameras.

[0059] For details, see attached. Figure 2 , 3 The diagram shows a panoramic view of the depth image acquisition scene, where the dashed lines represent the object to be reconstructed. In this embodiment, 12 depth cameras are used to acquire depth maps. The positions of each camera are different, and the angles corresponding to the facial depth maps acquired by each camera are also different, ensuring that the face can be captured from various angles, including the front, side, chin, ears, and forehead. Specifically, camera 1 is fixed to the left vertical bar of the subject (i.e., the object to be reconstructed) to acquire images of the left ear area. Cameras 2 to 4 are fixed to the left front vertical bar of the subject to acquire images of the left side of the head, forehead, and face. Camera 5 is fixed above the horizontal bar in front of the subject, with its body forward, to acquire images of the left side of the face, especially the nasal area. Camera 6 is fixed below the horizontal bar in front of the subject, with its body backward, to acquire images of the left chin area of ​​the face. Cameras 7 to 12 are located on the right side, and their acquisition areas correspond to those of cameras 1 to 6 on the left.

[0060] It should be noted that the specific type of depth camera can be a RealSense D415 depth camera, a RealSense D455 depth camera, etc. The above are just examples and do not limit the specific type of depth camera. Those skilled in the art can determine the type based on the actual situation, and all of them are within the scope of protection of this application.

[0061] In this embodiment of the application, after setting up the acquisition scene and before acquiring the depth map, camera calibration is performed on 12 cameras. Specifically, the intrinsic parameters of the depth cameras and the extrinsic parameters for coordinate transformation between each pair of cameras are calibrated. The calibration process is as follows: a checkerboard pattern is printed and pasted onto a plane as a calibration object; multiple sets of photos are taken of the calibration object by adjusting its orientation, with each set consisting of 12 photos; then, the corner points of the checkerboard pattern are extracted from the photos; based on the corner points, the intrinsic and extrinsic parameters of the cameras under ideal distortion-free conditions are estimated, and the least squares method is applied to estimate the distortion coefficient under actual radial distortion; the maximum likelihood method is applied for optimization estimation to improve the estimation accuracy. After camera calibration, the depth map can be processed based on the calibrated camera extrinsic and intrinsic parameters.

[0062] S102: Classify the multiple depth maps to determine a target depth map set; the target depth map set includes a preset number of target depth maps, and each target depth map in the target depth map set corresponds to a different camera identifier.

[0063] It should be noted that, in the embodiments of this application, at least one set of target depth maps can be generated, wherein each set of target depth maps includes 12 target depth maps, that is, each set of target depth maps includes one depth map acquired by each depth camera.

[0064] Each of the multiple depth maps further includes a timestamp; classifying the multiple depth maps to determine the target depth map set includes: classifying the multiple depth maps according to the timestamp and camera identifier of each depth map to determine the target depth map set.

[0065] Specifically, in this embodiment of the application, classifying multiple depth maps is equivalent to the frame synchronization process for multiple depth maps. For each depth camera, a depth map with a similar timestamp is acquired, generating a target depth map set. The depth maps included in this target depth map set are the target depth maps for the synchronization frame. Specifically, the step of classifying the multiple depth maps according to the timestamp and camera identifier of each depth map to determine the target depth map set includes: classifying the multiple depth maps according to the camera identifier to determine multiple initial depth map sets; each of the multiple initial depth map sets corresponds to the camera identifier; acquiring the first timestamp corresponding to the first depth map in the first initial depth map set; the first initial depth map set is any one of the multiple initial depth map sets; the first depth map is any one of the multiple depth maps included in the first initial depth map set; and determining the target depth map set from the multiple initial depth map sets according to the first timestamp and a preset time period.

[0066] In this embodiment, the preset time period is 10 milliseconds. First, all depth maps are classified according to camera identifiers. Twelve cameras acquire data, generating twelve initial depth map sets. Then, the first depth map from the first initial depth map set is selected. Taking the first initial depth map set as an example (e.g., the depth map of the first frame generated by camera 1), the search continues among the depth maps generated by the other eleven cameras to find the eleven depth maps with the closest timestamps. These twelve depth maps are considered a group. If the timestamp difference in this group is greater than 10 milliseconds, the group is considered not a synchronized frame and is discarded. If the timestamp difference is less than 10 milliseconds, it is retained. Subsequent depth maps from camera 1 are then retrieved sequentially, and the above steps are repeated to obtain M groups of depth map data. Each group contains twelve depth maps from different perspectives acquired at the same time (time interval less than 10 milliseconds).

[0067] S103: Obtain preset camera parameters, and generate a target point cloud data set based on the preset camera parameters and the target depth map set;

[0068] Specifically, the preset camera parameters include camera intrinsic parameters corresponding to the camera identifier and camera extrinsic parameters corresponding to the camera identifier. The step of obtaining the preset camera parameters and generating a target point cloud data set based on the preset camera parameters and the target depth map set includes: obtaining the baseline camera intrinsic parameters corresponding to the baseline camera identifier and the camera intrinsic parameters corresponding to each camera identifier; the baseline camera identifier is the identifier corresponding to a preset baseline camera; and converting multiple target depth maps included in the target depth map set according to the baseline camera intrinsic parameters and the camera intrinsic parameters corresponding to each camera identifier to generate a target point cloud data set; each point cloud data set included in the target point cloud data set corresponds to a camera identifier.

[0069] Specifically, based on the reference camera intrinsic parameters and the camera intrinsic parameters corresponding to each camera identifier, the multiple target depth maps included in the target depth map set are transformed to generate the target point cloud data set. The transformation is performed using a preset formula, as follows:

[0070] x w =Z c ·(u-u0)·d x / f

[0071] y w =Z c ·(v-v0)·d y / f

[0072] Z w =Z c

[0073] Where f is the length of the camera's focal length, dx and d y This represents the actual physical length represented by one pixel in the x and y directions. u0 represents the difference in the horizontal direction between the origin of the image coordinate system and the origin of the pixel coordinate system, and v0 represents the difference in the vertical direction between the origin of the image coordinate system and the origin of the pixel coordinate system. The origin of the image coordinate system is the intersection of the camera's optical axis and the imaging plane, measured in mm. The origin of the pixel coordinate system is the top-left corner of the depth map, measured in pixels. (u, v) represents the positional information of a pixel in the depth map, where u represents the horizontal coordinate data of the pixel in the depth map, and v represents the vertical coordinate data of the pixel in the depth map. Z c Represents the depth value, (x w y w , z w () represents the coordinates of the generated 3D point cloud, x w This represents the horizontal axis coordinate data in the point cloud coordinate system, y w This represents the vertical axis coordinate data in the point cloud coordinate system, z. w This represents the vertical axis coordinate data in the point cloud coordinate system. After the transformation, the M sets of target depth maps correspond to the M sets of target point cloud data, each set containing point cloud data from 12 different viewpoints.

[0074] S104: Perform point cloud registration and point cloud fusion processing on multiple point cloud data included in the target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed;

[0075] In this embodiment of the application, point cloud registration includes two registration processes: coarse point cloud registration and fine point cloud registration. Specifically, the step of performing point cloud registration and point cloud fusion processing on multiple point cloud data included in the target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed includes: performing coarse point cloud registration on multiple point cloud data included in the target point cloud data set according to the camera extrinsic parameters corresponding to each camera identifier to obtain a coarsely registered target point cloud data set; and performing fine point cloud registration and point cloud fusion processing on multiple coarsely registered point cloud data included in the coarsely registered target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed.

[0076] Specifically, the step of performing coarse point cloud registration on multiple point cloud data included in the target point cloud data set based on the camera extrinsic parameters corresponding to each camera identifier to obtain a coarsely registered target point cloud data set includes: obtaining the reference camera extrinsic parameters corresponding to the reference camera identifier and the camera extrinsic parameters corresponding to each camera identifier; the reference camera identifier is the identifier corresponding to a preset reference camera; determining the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier based on the reference camera extrinsic parameters and the camera extrinsic parameters corresponding to each camera identifier; and transforming the multiple point cloud data included in the target point cloud data set based on the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier to obtain the coarsely registered target point cloud data set; the coarsely registered target point cloud data set includes the preset number of coarsely registered point cloud data.

[0077] In this embodiment of the application, coarse registration of the point cloud involves acquiring a reference camera and performing registration based on the reference camera. It should be noted that the preset reference camera can be any one of the 12 cameras. Taking camera 1 as an example, the coarse registration of the point cloud using the extrinsic parameters of the depth camera is explained as follows: Using camera 1 as the reference, the point clouds in the coordinate systems of cameras 2 to 12 are sequentially transformed to the coordinate system of camera 1. After the transformation, all point clouds are in the coordinate system of camera 1. Specifically, the transformation formula is as follows:

[0078]

[0079] Where (x,y,z) represents the coordinates of any point in the point cloud to be registered in the current camera coordinate system. R represents the coordinates of the point transformed into the target camera coordinate system, and R and T represent the rotation matrix and offset matrix from the current camera coordinate system to the preset reference camera coordinate system, respectively.

[0080] In this embodiment, after coarse registration of the point cloud data, the overlapping region of the coarsely registered point cloud is further extracted. The ICP (Iterative Closest Point) algorithm is used to register the overlapping region, and the transformation matrix obtained from the registration is applied to the complete point cloud to complete fine registration. Fine registration is based on the overlapping region between the point clouds of two cameras; therefore, point clouds generated by two adjacent cameras are selected for fine registration each time to ensure that the overlapping region is included. In this example, the point clouds generated by two adjacent cameras are finely registered sequentially. Taking camera 1 as a preset reference camera as an example, fine registration can be performed from point cloud data of camera 12 to point cloud data of camera 11, from point cloud data of camera 11 to point cloud data of camera 10, and so on, until finally registration is achieved with point cloud data of camera 1. When performing fine registration of the point cloud data from camera 12 to camera 11, the first step is to calculate the nearest distance from all points in the point cloud data from camera 12 to the point cloud data from camera 11. If the nearest distance of a point is less than 1 mm, it is considered to belong to the overlapping area of ​​the two point clouds; otherwise, it belongs to the non-overlapping area. The set of all points in the overlapping area of ​​the point cloud data from camera 12 is denoted as point cloud C. Similarly, the set of all points in the overlapping area of ​​the point cloud data from camera 11 is denoted as point cloud D. The Iterative Closest Point Algorithm (ICP) is used to calculate the transformation matrix from point cloud C to point cloud D. Then, the transformation matrix is ​​used to perform coordinate transformation on the point cloud data from camera 12, completing the fine registration of the point cloud data from camera 12 to camera 11.

[0081] In this embodiment of the application, after fine registration of the point cloud data, point cloud fusion is further performed. A viewpoint is selected as the main viewpoint, and point cloud data from other viewpoints are fused to this main viewpoint. During the fusion process, overlapping point cloud data is removed, and non-overlapping point cloud data is retained. Then, the MLS (Moving Lest Squares) algorithm is used to smooth the point cloud data, eliminating the layering phenomenon generated during point cloud fusion, thus achieving point cloud fusion. Specifically, taking camera 1 as the preset reference camera as an example, point clouds from cameras 2 to 12 are fused to this main viewpoint in ascending order. When the point cloud data from camera 2 is fused to the point cloud data from camera 1, the nearest distance from all points in the point cloud data from camera 2 to the point cloud data from camera 1 is first calculated. If the nearest distance of a point is less than 1 mm, the point is considered to belong to the overlapping area of ​​the point cloud data from camera 1 and camera 2; otherwise, it belongs to the non-overlapping area. The set of all points in the non-overlapping area of ​​the point cloud data from camera 2 is denoted as point cloud E. Then, using the same method, all points in the overlapping area between the point cloud data from camera 2 and point cloud E are obtained and denoted as point cloud F. Point cloud F is then fused with the point cloud data from camera 1, that is, the set of all points in point cloud F and point cloud data from camera 1 is denoted as the new point cloud Z. Finally, moving least squares (MLS) is used to smooth point cloud Z.

[0082] It should be noted that, since the non-overlapping areas are filtered by a preset distance in this embodiment, the filtered point cloud E is smaller than the point cloud area in the point cloud data of camera 2 that does not overlap with the point cloud data of camera 1. The set of all points in the non-overlapping area of ​​the point cloud data of camera 2 is denoted as point cloud E. Then, all points in the overlapping area between the point cloud data of camera 2 and point cloud E are obtained by the same method and denoted as point cloud F. That is, point cloud F is the point cloud in the point cloud data of camera 2 that does not overlap with the point cloud data of camera 1.

[0083] In this embodiment of the application, to improve the processing efficiency of point cloud data, fine registration and point cloud fusion of point cloud data are combined. That is, point cloud fusion is performed directly after each fine registration. Specifically, the step of performing fine registration and point cloud fusion on multiple coarse registration point cloud data included in the coarse registration target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed includes: obtaining reference point cloud data and first point cloud data in the coarse registration target point cloud data set, wherein the reference point cloud data and the first point cloud data are two adjacent coarse registration point cloud data in the coarse registration target point cloud data set, and the reference point cloud data is the point cloud data corresponding to the reference camera identifier; determining a first transformation matrix based on the reference point cloud data and the first point cloud data; and performing point cloud fusion on the object to be reconstructed based on the transformation matrix. The first point cloud data is finely registered to obtain finely registered first point cloud data; the reference point cloud data and the finely registered first point cloud data are fused to obtain updated reference point cloud data; a temporary fused point cloud data set is generated based on the updated reference point cloud data and the remaining coarsely registered point cloud data in the coarsely registered target point cloud data set; the remaining coarsely registered point cloud data are the coarsely registered point cloud data in the coarsely registered target point cloud data set that have not undergone fine registration and point cloud fusion processing; fine point cloud registration and point cloud fusion processing are sequentially performed on the updated reference point cloud data and the temporary fused point cloud data included in the temporary fused point cloud data set; when the temporary fused point cloud data included in the temporary fused point cloud data set is 1, the panoramic point cloud data of the object to be reconstructed is obtained.

[0084] The following is in conjunction with the appendix Figure 4 , 5 Section 6 describes the changes in point cloud data before and after point cloud registration and point cloud fusion in the embodiments of this application. (See Appendix 6.) Figure 4 , 5 6, Appendix Figure 4 This is an illustration of point cloud data that has not been registered. It contains point clouds from 12 different viewpoints, and the image obtained before registration is very chaotic. (Attached) Figure 5 This is a set of point cloud data after registration. At this point, all 12 point clouds are in the same camera coordinate system. The image shows a complete face point cloud, but some layering still exists, such as obvious color differences on the nose, cheeks, and sides of the neck. It is evident that the face reconstruction image data obtained after coarse registration of the point clouds... Figure 6 (a) and Figure 6 (b) is a comparison image before fusion, as shown in the appendix. Figure 6 (a) A magnified view of the nose area's point cloud before point cloud fusion shows obvious layering, with two layers of nose outlines, and a clear color difference between the two layers, with the outer layer being darker and the inner layer brighter; as shown in the image. Figure 6(b) After point cloud fusion, the layering issue disappears. It is evident that converting the depth map into point cloud data, followed by point cloud registration and fusion, yields better facial image data.

[0085] It should be noted that in the actual experimental processing, since the point cloud converted from the depth map has no color, it is difficult to find some registration and fusion problems. To improve the accuracy of point cloud registration and fusion processing, the initial experimental process can be fine-tuned. During the experiment, in addition to acquiring depth maps, RGB images were also acquired for display clarity. During processing, each point cloud was colored using the acquired RGB images, which facilitates image observation and finding registration and fusion problems. The point cloud registration and fusion methods were adjusted to improve the accuracy of point cloud data processing. Subsequently, in the face reconstruction process, since the registration and fusion methods have been determined, only the depth map needs to be acquired for point cloud registration and fusion processing, thus improving the efficiency of face reconstruction.

[0086] S105: Process the panoramic cloud data of the object to be reconstructed based on a preset algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

[0087] In this embodiment of the application, after sequentially performing point cloud registration and point cloud fusion processing on M groups of data (12 point cloud data within each group), M frames of continuous and complete 3D face point cloud data are obtained. Based on these face point cloud data, a 3D point cloud reconstruction algorithm is used to reconstruct faces in consecutive frames. In this example, Poisson reconstruction is used, and the reconstruction process is as follows: input point cloud, construct octree; calculate point cloud normal vectors; establish and solve the Poisson equation; set a threshold to extract isosurfaces, and output mesh.

[0088] Specifically, the step of processing the panoramic cloud data of the object to be reconstructed based on a preset algorithm to obtain the reconstructed face mesh data corresponding to the object to be reconstructed includes: processing the panoramic cloud data of the object to be reconstructed based on the Poisson reconstruction algorithm to obtain the reconstructed face mesh data corresponding to the object to be reconstructed.

[0089] It should be noted that the reconstructed face mesh data includes the connection relationships between mesh patches and the normal vectors of the mesh patches, and the mesh patches can be triangular patches. In this embodiment of the application, the reconstructed face mesh data can also be stored in a mesh database.

[0090] This application provides a face reconstruction method that acquires depth maps of the object to be reconstructed in real time, converts them into point cloud data, and processes them to achieve proportional face reconstruction. By performing point cloud registration and fusion processing on the point cloud data, it overcomes the problem of difficulty in registering point clouds from different viewpoints due to their different structures. Furthermore, it preserves point clouds in non-overlapping areas, eliminating the layering phenomenon during the fusion of point cloud data corresponding to camera identifiers from different angles, making the overall point cloud smoother. Thus, it achieves real-time acquisition, processing, and proportionally accurate fine 3D face reconstruction, improving the precision of face reconstruction. This solves the problem in existing solutions where face reconstruction is based on a face database, and the reconstructed face often closely resembles faces in the database, making true proportional fine reconstruction difficult and resulting in low precision.

[0091] The following describes a face reconstruction system according to an embodiment of this application. Please refer to [link / reference]. Figure 7 Based on the face reconstruction method in the above embodiments, this application implements the face reconstruction method through a face reconstruction system. The face reconstruction system in this application includes:

[0092] The acquisition module 10 is used to acquire multiple depth maps corresponding to the object to be reconstructed in response to a face reconstruction request; wherein, the multiple depth maps are acquired by a preset number of cameras, and each depth map includes a camera identifier;

[0093] The frame synchronization module 20 is used to classify the multiple depth maps and determine a target depth map set; the target depth map set includes a preset number of target depth maps, and each target depth map in the target depth map set corresponds to a different camera identifier;

[0094] The point cloud data acquisition module 30 is used to acquire preset camera parameters and generate a target point cloud data set based on the preset camera parameters and the target depth map set.

[0095] The point cloud registration and fusion module 40 is used to perform point cloud registration and point cloud fusion processing on multiple point cloud data included in the target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed.

[0096] The face reconstruction module 50 is used to process the panoramic cloud data of the object to be reconstructed based on a preset algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

[0097] Each of the multiple depth maps also includes a timestamp; the frame synchronization module 20 is specifically used to classify the multiple depth maps according to the timestamp and camera identifier of each depth map to determine the target depth map set.

[0098] Frame synchronization module 20 is specifically used for:

[0099] Based on the camera identifier, the multiple depth maps are classified to determine multiple initial depth map sets; each initial depth map set corresponds to the camera identifier; a first timestamp corresponding to a first depth map in the first initial depth map set is obtained; the first initial depth map set is any one of the multiple initial depth map sets; the first depth map is any one of the multiple depth maps included in the first initial depth map set; a target depth map set is determined from the multiple initial depth map sets based on the first timestamp and a preset time period.

[0100] The preset camera parameters include camera intrinsic parameters corresponding to the camera identifier and camera extrinsic parameters corresponding to the camera identifier. The point cloud data acquisition module 30 is specifically used for:

[0101] Obtain the intrinsic parameters of the reference camera corresponding to the reference camera identifier and the intrinsic parameters of the camera corresponding to each camera identifier; the reference camera identifier is the identifier corresponding to a preset reference camera; based on the intrinsic parameters of the reference camera and the intrinsic parameters of the camera corresponding to each camera identifier, transform the multiple target depth maps included in the target depth map set to generate a target point cloud data set; each point cloud data included in the target point cloud data set corresponds to a camera identifier.

[0102] Point cloud registration and fusion module 40, specifically used for:

[0103] Based on the camera extrinsic parameters corresponding to each camera identifier, coarse point cloud registration is performed on multiple point cloud data included in the target point cloud data set to obtain a coarsely registered target point cloud data set; fine point cloud registration and point cloud fusion processing are performed on multiple coarsely registered point cloud data included in the coarsely registered target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed.

[0104] Point cloud registration and fusion module 40, specifically used for:

[0105] Obtain the extrinsic parameters of the reference camera corresponding to the reference camera identifier and the extrinsic parameters of each camera identifier; the reference camera identifier is the identifier corresponding to a preset reference camera; based on the reference camera extrinsic parameters and the extrinsic parameters of each camera identifier, determine the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier respectively; based on the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier, transform the multiple point cloud data included in the target point cloud data set respectively to obtain the coarse registration target point cloud data set; the coarse registration target point cloud data set includes the preset number of coarse registration point cloud data.

[0106] Point cloud registration and fusion module 40, specifically used for:

[0107] Obtain reference point cloud data and first point cloud data from the coarse registration target point cloud data set, wherein the reference point cloud data and the first point cloud data are two adjacent coarse registration point cloud data in the coarse registration target point cloud data set, and the reference point cloud data is the point cloud data corresponding to the reference camera identifier; determine a first transformation matrix based on the reference point cloud data and the first point cloud data; perform fine registration on the first point cloud data based on the transformation matrix to obtain finely registered first point cloud data; perform point cloud fusion processing on the reference point cloud data and the finely registered first point cloud data to obtain an updated... Reference point cloud data; based on the updated reference point cloud data and the remaining coarse registration point cloud data in the coarse registration target point cloud data set, a temporary fused point cloud data set is generated; the remaining coarse registration point cloud data are the coarse registration point cloud data in the coarse registration target point cloud data set that have not undergone fine registration and point cloud fusion processing; fine registration and point cloud fusion processing are sequentially performed on the updated reference point cloud data and the temporary fused point cloud data included in the temporary fused point cloud data set; when the temporary fused point cloud data included in the temporary fused point cloud data set is 1, the panoramic point cloud data of the object to be reconstructed is obtained.

[0108] The face reconstruction module 50 is specifically used to: process the panoramic cloud data of the object to be reconstructed based on the Poisson reconstruction algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

[0109] The preset quantity is 12.

[0110] This application provides a face reconstruction system that collects depth maps of the object to be reconstructed in real time, converts them into point cloud data, and processes them to achieve proportional face reconstruction. By performing point cloud registration and point cloud fusion processing on the point cloud data, it overcomes the problem of difficulty in registering point clouds from different perspectives due to their different structures. Furthermore, it retains point clouds in non-overlapping areas and eliminates the layering phenomenon when fusing point cloud data corresponding to camera identifiers from different angles, making the overall point cloud smoother. Thus, it achieves real-time acquisition, processing, and proportional reconstruction of face reconstruction, improving the accuracy of face reconstruction.

[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A face reconstruction method, characterized in that, The method includes: In response to a face reconstruction request, multiple depth maps corresponding to the object to be reconstructed are obtained; wherein, the multiple depth maps are obtained through a preset number of cameras, and each depth map includes a camera identifier; The multiple depth maps are classified to determine a target depth map set; the target depth map set includes a preset number of target depth maps, and each target depth map in the target depth map set corresponds to a different camera identifier; Obtain preset camera parameters, and generate a target point cloud data set based on the preset camera parameters and the target depth map set; Point cloud registration and point cloud fusion processing are performed on multiple point cloud datasets included in the target point cloud dataset to obtain panoramic point cloud data of the object to be reconstructed. The point cloud fusion processing includes: selecting a primary viewpoint and sequentially fusing point cloud data from other viewpoints to the primary viewpoint; filtering non-overlapping regions by a preset distance to obtain point cloud data from the camera point cloud data to be fused that does not overlap with a preset reference camera point cloud data; filtering overlapping regions by the preset distance to obtain all points in the overlapping region between the camera point cloud data to be fused and the non-overlapping point cloud data; and fusing all points in the overlapping region with the preset reference camera point cloud data to obtain updated point cloud data. Based on a preset algorithm, the panoramic cloud data of the object to be reconstructed is processed to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

2. The method according to claim 1, characterized in that, Each of the multiple depth maps also includes a timestamp; classifying the multiple depth maps to determine the target depth map set includes: Based on the timestamp and camera identifier of each depth map, the multiple depth maps are classified to determine the target depth map set.

3. The method according to claim 2, characterized in that, The step of classifying the multiple depth maps and determining the target depth map set based on the timestamp and camera identifier of each depth map includes: Based on the camera identifier, the multiple depth maps are classified to determine multiple initial depth map sets; each initial depth map set corresponds to the camera identifier. Obtain the first timestamp corresponding to the first depth map in the first initial depth map set; the first initial depth map set is any one of the plurality of initial depth map sets; the first depth map is any one of the plurality of depth maps included in the first initial depth map set; Based on the first timestamp and the preset time period, a target depth map set is determined from the plurality of initial depth map sets.

4. The method according to claim 1, characterized in that, The preset camera parameters include camera intrinsic parameters corresponding to the camera identifier and camera extrinsic parameters corresponding to the camera identifier. The step of obtaining the preset camera parameters and generating a target point cloud data set based on the preset camera parameters and the target depth map set includes: Obtain the intrinsic parameters of the reference camera corresponding to the reference camera identifier and the intrinsic parameters of each camera identifier; the reference camera identifier is the identifier corresponding to a preset reference camera. Based on the reference camera intrinsic parameters and the camera intrinsic parameters corresponding to each camera identifier, the multiple target depth maps included in the target depth map set are transformed to generate a target point cloud data set; each point cloud data included in the target point cloud data set corresponds to a camera identifier.

5. The method according to claim 4, characterized in that, The step of performing point cloud registration and point cloud fusion processing on multiple point cloud data included in the target point cloud dataset to obtain panoramic point cloud data of the object to be reconstructed includes: Based on the camera extrinsic parameters corresponding to each camera identifier, coarse registration of multiple point cloud data included in the target point cloud data set is performed to obtain a coarsely registered target point cloud data set. The coarse registration target point cloud data set includes multiple coarse registration point cloud data, which are then subjected to fine registration and point cloud fusion processing to obtain the panoramic point cloud data of the object to be reconstructed.

6. The method according to claim 5, characterized in that, The step of performing coarse point cloud registration on multiple point cloud data included in the target point cloud data set based on the camera extrinsic parameters corresponding to each camera identifier to obtain a coarsely registered target point cloud data set includes: Obtain the reference camera extrinsic parameters corresponding to the reference camera identifier and the camera extrinsic parameters corresponding to each camera identifier; the reference camera identifier is the identifier corresponding to a preset reference camera. Based on the reference camera extrinsic parameters and the camera extrinsic parameters corresponding to each camera identifier, determine the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier, respectively; Based on the rotation matrix and offset matrix corresponding to the reference camera identifier and each camera identifier, the multiple point cloud data included in the target point cloud data set are transformed to obtain the coarse registration target point cloud data set; the coarse registration target point cloud data set includes the preset number of coarse registration point cloud data.

7. The method according to claim 5, characterized in that, The step of performing fine point cloud registration and point cloud fusion processing on multiple coarsely registered point cloud datasets included in the coarsely registered target point cloud dataset to obtain panoramic point cloud data of the object to be reconstructed includes: Obtain the reference point cloud data and the first point cloud data from the coarse registration target point cloud data set, wherein the reference point cloud data and the first point cloud data are two adjacent coarse registration point cloud data in the coarse registration target point cloud data set, and the reference point cloud data is the point cloud data corresponding to the reference camera identifier; A first transformation matrix is ​​determined based on the reference point cloud data and the first point cloud data; The first point cloud data is finely registered according to the transformation matrix to obtain the finely registered first point cloud data. The reference point cloud data and the finely registered first point cloud data are subjected to point cloud fusion processing to obtain the updated reference point cloud data. Based on the updated reference point cloud data and the remaining coarse registration point cloud data in the coarse registration target point cloud data set, a temporary fused point cloud data set is generated; the remaining coarse registration point cloud data is the coarse registration point cloud data in the coarse registration target point cloud data set that has not undergone fine registration and point cloud fusion processing. The updated baseline point cloud data and the temporary fused point cloud data included in the temporary fused point cloud data set are sequentially subjected to fine point cloud registration and point cloud fusion processing. When the temporary fused point cloud data included in the temporary fused point cloud data set is 1, the panoramic point cloud data of the object to be reconstructed is obtained.

8. The method according to claim 1, characterized in that, The process of processing the panoramic cloud data of the object to be reconstructed based on a preset algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed includes: The panoramic cloud data of the object to be reconstructed is processed based on the Poisson reconstruction algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

9. The method according to claim 1, characterized in that, The preset quantity is 12.

10. A face reconstruction system, characterized in that, The system includes: The acquisition module is used to acquire multiple depth maps corresponding to the object to be reconstructed in response to a face reconstruction request; wherein, the multiple depth maps are acquired by a preset number of cameras, and each depth map includes a camera identifier; A frame synchronization module is used to classify the multiple depth maps and determine a target depth map set; the target depth map set includes a preset number of target depth maps, and each target depth map in the target depth map set corresponds to a different camera identifier; The point cloud data acquisition module is used to acquire preset camera parameters and generate a target point cloud data set based on the preset camera parameters and the target depth map set. The point cloud registration and fusion module is used to perform point cloud registration and point cloud fusion processing on multiple point cloud data included in the target point cloud data set to obtain the panoramic point cloud data of the object to be reconstructed. The point cloud registration and fusion module is specifically used to select a viewpoint as the main viewpoint, and sequentially fuse the point cloud data from other viewpoints into the point cloud data of the main viewpoint; filter non-overlapping regions by a preset distance to obtain point cloud data of non-overlapping regions between the point cloud data of the camera to be fused and the preset reference camera point cloud data; filter overlapping regions by the preset distance to obtain all points in the overlapping region between the point cloud data of the camera to be fused and the point cloud data of the non-overlapping regions; and fuse all points in the overlapping region with the preset reference camera point cloud data to obtain updated point cloud data. The face reconstruction module is used to process the panoramic cloud data of the object to be reconstructed based on a preset algorithm to obtain the reconstructed face grid data corresponding to the object to be reconstructed.

Images