Three-dimensional face reconstruction method, device, equipment and medium

Abstract

The present application relates to a three-dimensional face reconstruction method, device, equipment and medium, and relates to the field of computer technology. The method includes: using a binocular camera device to collect binocular image data of a target object in at least two viewing directions; converting according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data; performing face point cloud processing according to the image pair data to obtain a face point cloud data set corresponding to the target object; performing registration processing according to the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data; based on the preset face base point cloud data, performing fusion processing on the posture reference point cloud data and the face conversion point cloud data to obtain the target face point cloud data; performing surface reconstruction according to the binocular face point cloud data to obtain a three-dimensional face model corresponding to the target object. The present application reduces costs while meeting the model accuracy required for the three-dimensional face model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a three-dimensional face reconstruction method, device, equipment and medium. Background Art

[0002] Face information is important for identifying others, and three-dimensional information has important applications in many fields and directions of machine vision. Therefore, it is of great significance to transform common flat face images into three-dimensional ones.

[0003] At present, there are two main methods for three-dimensional reconstruction of real human faces: the first method is to build a monocular camera array and use the principle of parallax to achieve three-dimensional face model reconstruction, but the monocular camera array is expensive, and all cameras in the monocular camera array need to be synchronously controlled to shoot at the same time, and a fixed shooting location is required to calibrate the camera times and improve the calibration accuracy. That is, the various requirements for achieving three-dimensional face model reconstruction by building a monocular camera array are relatively strict, which increases the cost of three-dimensional face model reconstruction; the second method is to use laser scanner equipment to achieve three-dimensional face model reconstruction using the principle of laser ranging, but the laser scanner used in this method is also expensive, which increases the equipment investment cost of three-dimensional face model reconstruction. Summary of the invention

[0004] In view of this, the present application provides a three-dimensional face reconstruction method, device, equipment and medium to complete three-dimensional face reconstruction using consumer-grade binocular camera equipment, thereby reducing costs while meeting the model accuracy required for the three-dimensional face model.

[0005] In a first aspect, an embodiment of the present application provides a three-dimensional face reconstruction method, comprising: collecting binocular image data of a target object in at least two viewing directions through a binocular camera device; for the binocular image data in any of the viewing directions, converting according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data; performing face point cloud processing based on the image pair data to obtain a face point cloud data set corresponding to the target object; performing registration processing based on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data; based on preset face base point cloud data, fusing the posture reference point cloud data and the face conversion point cloud data to obtain target face point cloud data; performing surface reconstruction based on the face point cloud data to obtain a three-dimensional face model corresponding to the target object.

[0006] Optionally, the binocular image data of the target object in at least two viewing directions is collected by a binocular camera device, including: controlling the binocular camera device to rotate within a preset angle range with the front face of the target object as the central axis, and the binocular camera device and the target object are located in the same horizontal plane; when it is detected that the binocular camera device is rotated to any preset viewing direction, controlling the binocular camera device to collect the binocular image data of the target object in the current viewing direction; or controlling the binocular camera devices located in N viewing directions to collect the target part of the target object to obtain the binocular image data of the target object in N viewing directions, where N is greater than or equal to 2.

[0007] Optionally, the converting is performed according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data, including: determining the disparity map information of the binocular image data, the disparity map information including disparity values ​​of at least two pixel points; for the disparity value of any pixel point in the disparity map information, converting the disparity value into a corresponding depth of field value using preset conversion parameters; and generating the image pair data according to the depth of field values ​​corresponding to each pixel point in the disparity map information.

[0008] Optionally, the image pair data includes color image information and depth image information, and performing face point cloud processing based on the image pair data to obtain a face point cloud dataset corresponding to the target object includes: determining a target color image aligned with the depth image information based on the color image information and depth image information in the image pair data; and performing point cloud processing based on facial area information in the target color image to obtain the face point cloud dataset.

[0009] Optionally, determining a target color image aligned with the depth image information based on the color image information and the depth image information in the image pair data includes: for any depth pixel point of the depth image information, converting the pixel coordinates and the depth of field value of the depth pixel point into three-dimensional coordinates in a world coordinate system; converting the three-dimensional coordinates into target pixel coordinates of the depth pixel point, the target pixel coordinates of the depth pixel point corresponding to the color pixel point of the color image information in the image pair data; determining a target color pixel point corresponding to the target pixel coordinates of the depth pixel point from the color pixel points of the color image information; obtaining a color value corresponding to the target color pixel point, and assigning the color value to the depth pixel point; and using the color values ​​corresponding to each of the depth pixel points of the depth image information to recompose a target color image aligned with the depth image information.

[0010] Optionally, point cloud processing is performed based on the face area information in the target color image to obtain the face cloud data set, including: determining the face area information in the target color image; determining the face mask matrix corresponding to the target color image based on the face area information; using the face mask matrix, processing the depth image information to obtain a target depth image; converting the target depth image into face point cloud data of a target object, and forming the face cloud data set based on the face point cloud data of the target object.

[0011] Optionally, converting the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system includes: inputting the pixel coordinates and the depth value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula; wherein the first homogeneous transformation formula includes:

[0012]

[0013]

[0014] The K d The internal parameter matrix of the depth sensing module in the binocular camera device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

[0015] Optionally, converting the three-dimensional coordinates into the target pixel coordinates of the depth pixel point includes: inputting the three-dimensional coordinates into a second homogeneous transformation formula, and obtaining the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula; wherein the second homogeneous transformation formula includes:

[0016]

[0017]

[0018] The K c Including the internal parameter matrix of the color photosensitive module in the binocular camera device, the T w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2cand the offset vector t w2c The three-dimensional coordinates include (the x w , the y w , the z w ), the target pixel coordinates of the depth pixel point include (the u c , the v c ).

[0019] Optionally, converting the target depth image into facial point cloud data of the target object includes: determining each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtaining the pixel coordinates and depth value of the pixel point with a non-zero depth value; converting the pixel coordinates and depth value of the pixel point with a non-zero depth value into three-dimensional coordinates in a world coordinate system by a first homogeneous transformation formula; determining that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value; and forming the facial point cloud data of the target object by the three-dimensional points corresponding to each pixel point with a non-zero depth value.

[0020] Optionally, the facial point cloud data in the facial point cloud data set are registered to obtain posture reference point cloud data and face conversion point cloud data, including: determining the center of mass of each facial point cloud data in the facial point cloud data set; performing unified processing according to the center of mass of each facial point cloud data set to obtain an intermediate facial point cloud data set corresponding to the target object; selecting posture reference point cloud data from the intermediate facial point cloud data set; and using the posture reference point cloud data to register non-posture reference point cloud data in the intermediate facial point cloud data set to obtain face conversion point cloud data.

[0021] Optionally, the centroids of the various facial point cloud data are uniformly processed to obtain an intermediate facial point cloud data set corresponding to the target object, including: for each facial point cloud data, subtracting the coordinates of the mass point from the spatial coordinates of each point in the facial point cloud data to obtain intermediate facial point cloud data corresponding to the facial point cloud data; based on the intermediate facial point cloud data corresponding to each facial point cloud data, an intermediate facial point cloud data set corresponding to the target object is formed.

[0022] Optionally, the use of the posture reference point cloud data to align the non-posture reference point cloud data in the intermediate face point cloud data set to obtain face conversion point cloud data includes: extracting any of the non-posture reference point cloud data from the intermediate face point cloud data set as target intermediate face point cloud data; selecting M target point data from the intermediate face point cloud data; searching for the nearest neighbor point data corresponding to each of the M target point data from the posture reference point cloud data; generating the nearest point pair data corresponding to each target point data according to the nearest neighbor point data corresponding to each target point data; and The nearest point pair data corresponding to the M target point data constitute a nearest point pair set; according to the nearest point pair set, determine the target rotation matrix and the target offset vector; use the target rotation matrix and the target offset vector to transform the target intermediate facial point cloud data to obtain the initial conversion point cloud data corresponding to the target intermediate facial point cloud data; determine the conversion error between the posture reference point cloud data and the initial conversion point cloud data; if the conversion error is less than a preset first threshold, determine the initial conversion point cloud data as the facial conversion point cloud data corresponding to the target intermediate facial point cloud data.

[0023] Optionally, the use of the posture reference point cloud data to align the non-posture reference point cloud data in the intermediate facial point cloud data set also includes: judging whether the conversion error is less than a preset first threshold; if the conversion error is not less than the preset first threshold, determining the number of registration iterations corresponding to the target intermediate facial point cloud data; if the number of registration iterations is less than the preset iteration number threshold, jumping to the step of selecting M target point data from the intermediate facial point cloud data for re-alignment until the number of registration iterations corresponding to the target intermediate facial point cloud data reaches the preset number threshold.

[0024] Optionally, determining the center of mass of each facial point cloud data in the facial point cloud data set includes: obtaining the spatial coordinates of all points in each facial point cloud data in the facial point cloud data set; and determining the average value of the spatial coordinates of all points in each facial point cloud data as the center of mass of each facial point cloud data.

[0025] In a second aspect, an embodiment of the present application provides a three-dimensional face reconstruction device, comprising:

[0026] A binocular image data acquisition module is used to acquire binocular image data of a target object in at least two viewing directions through a binocular camera device;

[0027] A binocular image data conversion module, used for converting the binocular image data under any of the viewing angle directions according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data;

[0028] A face point cloud data set module is used to perform face point cloud processing on the data according to the image to obtain a face point cloud data set corresponding to the target object;

[0029] A point cloud registration processing module is used to perform registration processing based on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data;

[0030] A fusion processing module is used to fuse the posture reference point cloud data and the face conversion point cloud data based on the preset face base point cloud data to obtain the target face point cloud data;

[0031] The surface reconstruction module is used to perform surface reconstruction based on the target face point cloud data to obtain a three-dimensional face model corresponding to the target object.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the three-dimensional face reconstruction method as described in any one of the first aspects when executing the program stored in the memory.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the three-dimensional face reconstruction method as described in any one of the first aspects are implemented.

[0034] After obtaining binocular image data in at least two viewing directions by photographing the target object through a camera device, the embodiment of the present application can convert the disparity map information of the binocular image data in each viewing direction to obtain image pair data corresponding to the binocular image data, and perform face point cloud processing based on the image pair data to obtain a face point cloud data set corresponding to the target object, and then perform registration processing based on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data, and perform fusion processing on the posture reference point cloud data and the face conversion point cloud data based on the preset face base point cloud data to utilize the face point cloud data to obtain the face point cloud data corresponding to the target object. The face-based point cloud data is used for shape correction, that is, the face reference point cloud data is added as a shape reference during the point cloud registration process, which greatly reduces the shape deviation of the result obtained when the point cloud fusion is finally performed, such as irregular protrusions or depressions, abnormal nose shape, etc., and the target face point cloud data is obtained. Subsequently, the surface is reconstructed based on the target face point cloud data to obtain a three-dimensional face model, so that the final three-dimensional face model meets the model accuracy required for real-person expression modeling, thereby achieving the purpose of completing the three-dimensional reconstruction task of real-person faces using consumer-grade binocular camera equipment, while meeting the model accuracy required for three-dimensional face models and reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 is a flowchart of the steps of a three-dimensional face reconstruction method provided in an embodiment of the present application;

[0038] Figure 2 It is a schematic diagram of obtaining binocular image data of a model's face at different viewing angles shown in an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram of another method of obtaining binocular image data of a model's face at different viewing angles shown in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of visual representation of a facial region of a human face shown in an embodiment of the present invention;

[0041] Figure 5A schematic diagram of a surface reconstruction algorithm shown in an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of an implementation process of a method for determining target face point cloud data provided by an embodiment of the present invention;

[0043] Figure 7 A schematic diagram of an implementation process of registering the remaining intermediate face point cloud data of the target face in N-1 viewing directions provided by an embodiment of the present invention;

[0044] Figure 8 A schematic diagram of an implementation flow of fusing posture reference point cloud data and face conversion point cloud data of a target object in N-1 viewing directions provided by an embodiment of the present invention;

[0045] Fig. 9 A structural block diagram of a three-dimensional face reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] Reference Figure 1 , shows a flowchart of the steps of a 3D face reconstruction method provided by an embodiment of the present application. Specifically, the 3D face reconstruction method provided by the present application may include the following steps:

[0048] Step 110: Collect binocular image data of the target object in at least two viewing directions by using a binocular camera device.

[0049] In the actual processing example, since the human face is three-dimensional, the raised part will cause occlusion. In order to eliminate the influence of occlusion, the embodiment of the present application can use one or more binocular camera devices to shoot the target object from different directions to collect the face of the target object from different viewing angles and obtain binocular image data of the target object at different viewing angles. For example, one or more binocular camera devices can be used to shoot the head of the target object from N different viewing angles to collect binocular image data of the target object from N viewing angles. Wherein, N is a positive integer and ≥2; the binocular camera device refers to a device including a binocular camera, and the binocular camera can be used to collect binocular image data.

[0050] Furthermore, the embodiments of the present application collect binocular image data of a target object in at least two viewing directions through a binocular camera device, which may specifically include: controlling the binocular camera device to rotate within a preset angle range with the front face of the target object as the central axis, and the binocular camera device and the target object are located in the same horizontal plane; when it is detected that the binocular camera device is rotated to any preset viewing direction, controlling the binocular camera device to collect binocular image data of the target object in the current viewing direction; or controlling the binocular camera devices located in N viewing directions to collect the target part of the target object to obtain the binocular image data of the target object in N viewing directions, where N is greater than or equal to 2.

[0051] Specifically, the modeling object in the embodiment of the present application is a face. The binocular camera device as an image acquisition device must be in the same horizontal plane as the photographed object. The more corresponding shooting and acquisition times, that is, the larger N is, the more conducive to the accuracy of subsequent operations. Based on this, in an optional implementation of the present application, the binocular image data of the target object's face in N viewing directions can be obtained by adjusting the position of the binocular camera device. Specifically, the front face of the target object (that is, the face of the target object) can be used as the central axis, and the binocular camera device can be controlled to rotate according to a preset rotation angle within a preset angle range, and after each control of the binocular camera device to rotate to a preset rotation angle, the binocular camera device is controlled to shoot the target part of the target object (that is, the face) to obtain the binocular image data of the target object in the viewing direction corresponding to the current rotation angle, that is, the binocular image data of the target object's face is collected by the binocular camera device. In this way, by adjusting the position and posture of the binocular camera device (that is, the orientation of the binocular camera device in the shooting space), the binocular image data of the target object's face in N viewing directions can be obtained. For example, taking a single consumer-grade binocular camera as an example, such as Figure 2 As shown, with the model's face (i.e., the front face of the target object, which can be the model's head) as the central axis, the consumer-grade binocular camera is controlled to rotate at a rotation angle of 15° (along the guide rail) within a maximum range of 120°, which means that the consumer-grade binocular camera rotates once every 15°, and after each control of the consumer-grade binocular camera to rotate at a rotation angle of 15° (along the guide rail), the consumer-grade binocular camera is called to collect binocular image data of the model's face. In this way, binocular image data of the model's face in 120 / 15=8 viewing angles can be obtained.

[0052] Of course, the present application can also obtain binocular image data of the target object in N different viewing directions by other means. Furthermore, in another optional implementation, the face of the target object can be used as the central axis, and consumer-grade binocular camera devices can be respectively set in N viewing directions within a preset angle range, so that the head of the target object can be photographed by controlling the binocular camera devices in these N viewing directions to obtain binocular image data of the target face in N viewing directions. For example, taking 3 consumer-grade binocular cameras as an example, Figure 3 As shown in the figure, with the model's face (i.e., the front face of the model's head as the target object) as the central axis, consumer-grade binocular cameras are respectively set in three viewing directions within a maximum range of 120°, and these three consumer-grade binocular cameras are synchronously called to obtain binocular image data of the model's face in three viewing directions. In this way, the binocular image data of the target object in three viewing directions can still be obtained through the three binocular cameras.

[0053] Step 120 : For the binocular image data in any viewing angle direction, conversion is performed according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data.

[0054] Specifically, after obtaining the binocular image data under at least two viewing directions, the embodiment of the present application can obtain the disparity map information under each viewing direction from the binocular image data under each viewing direction, and can convert the disparity map information into depth image data, such as converting the disparity map information into RGB-D data as image pair data corresponding to the binocular image data.

[0055] Furthermore, the embodiments of the present application perform conversion according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data, which may specifically include: determining the disparity map information of the binocular image data, the disparity map information including disparity values ​​of at least two pixel points; for the disparity value of any pixel point in the disparity map information, converting the disparity value into a corresponding depth of field value using preset conversion parameters; and generating the image pair data according to the depth of field values ​​corresponding to each pixel point in the disparity map information.

[0056] The disparity map information obtained from the binocular image data can be completed by a visual algorithm library such as OpenCV, OpenGL, etc. For example, the binocular image data can be input into a pre-trained visual algorithm library model to use the visual algorithm library model to determine and output the disparity map information of the binocular image data. The disparity map information of the binocular image data can represent the disparity map of the binocular image data.

[0057] For any pixel point of the disparity map, the calculation formula can be used Convert the disparity value d of the pixel point to the depth value zc , thus the disparity map information can be converted into Depth image data. It should be noted that b in the calculation formula is the distance between the optical centers of the left and right cameras in the camera, which can be obtained by measurement; f x is the focal length of the camera, which can be read through the module interface or obtained by manual calibration, and the embodiments of the present application are not limited to this.

[0058] Step 130 , performing face point cloud processing on the data according to the image to obtain a face point cloud data set corresponding to the target object.

[0059] Specifically, after the binocular image data under each viewing direction is converted into corresponding image pair data, the embodiment of the present application determines the target color image aligned with the depth image information for the image pair data under any viewing direction according to the color image (RGB image) information and the depth image information in the image pair data, and then performs point cloud processing according to the facial information of the face in the target color image, such as converting the facial information of the face in the target color image into a point cloud, and removing the three-dimensional point cloud data that is not related to the face, that is, removing the three-dimensional points that are not related to the face, to obtain the face point cloud data set.

[0060] Furthermore, the image pair data in the embodiment of the present application includes color image information and depth image information. The above-mentioned face point cloud processing based on the image pair data to obtain the face point cloud data set corresponding to the target object may specifically include: determining a target color image aligned with the depth image information based on the color image information and depth image information in the image pair data; performing point cloud processing based on the facial area information in the target color image to obtain the face point cloud data set.

[0061] Specifically, for the image pair data of the target object under N viewing directions, in an embodiment of the present invention, for the image pair data of the target object under any viewing direction, according to the color image information and the depth image information in the image pair data, a target color image aligned with the depth image information can be determined, and the facial region information of the face in the target color image can be determined, and then the target color image can be converted into a facial mask matrix according to the facial region information, and the depth image information is processed using the facial mask matrix to obtain a target depth image, and the target depth image is converted into facial point cloud data of the target face under the viewing direction, so that a facial point cloud data set of the target object can be formed based on the facial point cloud data of the target face under N viewing directions. Furthermore, the embodiment of the present application performs point cloud processing based on the face area information in the target color image to obtain the face cloud data set, which may specifically include: determining the face area information in the target color image; determining the face mask matrix corresponding to the target color image based on the face area information; using the face mask matrix, processing the depth image information to obtain a target depth image; converting the target depth image into face point cloud data of the target object, and forming the face cloud data set based on the face point cloud data of the target object. Among them, the face area information can be used to represent the face area in the target color image.

[0062] For example, with the image pair data of the target face under the i-th (i=1, 2, 3..., N) viewing direction, according to the RGB image information and the Depth image information in the image pair data, determine the target color image aligned with the Depth image information, that is, the RGB′ image, whose image size is consistent with the Depth image information, determine the face area in the RGB′ image, that is, determine the face area information in the target color image, such as the face area in the target color image can be manually outlined by the user, or the face area in the target color image can be identified by a pre-trained recognition model, so as to convert the RGB′ image into a face mask matrix according to the face area, and then use the face mask matrix to process the Depth image information to obtain the target depth image, that is, the Depth′ image, and convert the Depth′ image into the face point cloud data of the target face under the i-th viewing direction. In this way, the face point cloud data of the target face under N viewing directions can be obtained, and then the face point cloud data set of the target object can be generated based on the face point cloud data of the target face under N viewing directions.

[0063] Among them, in an embodiment of the present invention, according to the color image information and depth image information in the image pair data, a target color image aligned with the depth image information is determined specifically in the following manner: for any depth pixel point of the depth image information, the pixel coordinates and the depth of field value of the depth pixel point are converted into three-dimensional coordinates in a world coordinate system; the three-dimensional coordinates are converted into the target pixel coordinates of the depth pixel point, and the target pixel coordinates of the depth pixel point correspond to the color pixel point of the color image information in the image pair data; from the color pixel points of the color image information, a target color pixel point corresponding to the target pixel coordinate of the depth pixel point is determined; the color value corresponding to the target color pixel point is obtained, and the color value is assigned to the depth pixel point; and the color values ​​corresponding to each of the depth pixel points of the depth image information are used to reconstruct the target color image aligned with the depth image information.

[0064] Specifically, based on the image pair data, for any depth pixel point of the depth image information in the image pair data, the pixel coordinates and depth of field value of the depth pixel point can be input into the first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula, thereby completing the conversion of the pixel coordinates and depth of field value of the depth pixel point into three-dimensional coordinates in the world coordinate system.

[0065] For example, for any depth pixel point of the Depth image information in the image pair data, the pixel coordinates (u d ,v d ) and depth of field value z c Input into the first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula, wherein the first homogeneous transformation formula includes:

[0066]

[0067]

[0068] The K d The internal parameter matrix of the depth sensing module in the binocular camera device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

[0069] The K d The internal parameter matrix of the depth sensing module in the camera device can be 3x3 in size and is obtained by reading the camera device interface or manually calibrating. w2d The external parameter matrix of the depth sensing module is 4x4 in size, which is the Euclidean transformation matrix from the world coordinate system to the depth sensing module, and is composed of a 3x3 rotation matrix R w2d and the offset vector t w2d Both can be obtained by manual calibration. The pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

[0070] For the three-dimensional coordinates in the world coordinate system, the three-dimensional coordinates in the world coordinate system are input into the second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula, so as to convert the three-dimensional coordinates in the world coordinate system into the target pixel coordinates of the depth pixel point. The target pixel coordinates of the depth pixel point are actually the pixel coordinates of the color pixel point in the color image information, that is, the target pixel coordinates of the depth pixel point correspond to the color pixel point of the color image information in the image pair data.

[0071] For example, for the three-dimensional coordinates in the world coordinate system, the three-dimensional coordinates in the world coordinate system are input into the second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel output by the second homogeneous transformation formula. The target pixel coordinates of the depth pixel correspond to the color pixel of the color image information in the image pair data. The target pixel coordinates of the depth pixel are the pixel coordinates of the color pixel in the RGB image (u c ,v c ), where the second homogeneous transformation formula includes:

[0072]

[0073]

[0074] The K c The internal parameter matrix of the color photosensitive module in the camera device can be 3x3 in size and can be read through the camera device interface or manually calibrated. w2c The external parameter matrix of the color photosensitive module is 4x4 in size, which is the Euclidean transformation matrix from the world coordinate system to the color photosensitive module, and is composed of a 3x3 rotation matrix R w2c and the offset vector tw2c Both can be obtained by manual calibration. The three-dimensional coordinates include (the x w , the y w , the z w ), the target pixel coordinates of the depth pixel point include (the u c , the v c ).

[0075] In an embodiment of the present invention, the target pixel coordinates of the depth pixel are the pixel coordinates of the color pixel in the color image, that is, the color pixel corresponding to the color image information in the image pair data. In this way, the target color pixel corresponding to the target pixel coordinates of the depth pixel can be determined from the color pixel in the color image information, the color value corresponding to the target color pixel is obtained, and the color value is assigned to the depth pixel. In this way, the color value corresponding to each depth pixel in the depth image information is found in the color image information, so that each depth pixel in the depth image information has a corresponding color value and depth of field value. Subsequently, the color values ​​corresponding to each depth pixel in the depth image information are used to recompose the target color image aligned with the depth image information, which is called an RGB′ image, and its image size is consistent with the Depth image. The RGB′ image and the Depth image aligned at the pixel level are called RGB'-D data.

[0076] It should be noted that for the human face area, the range of the human face area is usually defined as "the area surrounded by the left ear root (excluding the left ear), the hairline, the right ear root (excluding the right ear), and the chin extension", and its visual expression is as follows: Figure 4 shown.

[0077] For the depth image information in the image pair data, the depth image information can be read in matrix form to obtain the depth matrix corresponding to the depth image information, and the depth matrix and the face mask matrix are processed to obtain the target depth image. In this way, the depth image is processed using the face mask matrix to obtain the target depth image. Among them, after the depth image information is read in matrix form as a depth matrix, the depth values ​​of the depth pixels in the non-face area of ​​the depth image information can be reset to zero by calculating the Hamada product of the matrix to obtain the target depth matrix, and the image corresponding to the target depth matrix is ​​determined as the target depth image, so that the depth values ​​of the pixels in the non-face area of ​​the target depth image are reset to zero.

[0078] For example, the depth image information is read in matrix form as the depth matrix M Depth Then, M Depth and M face Input the preset matrix operation formula to obtain the target depth matrix M output by the preset matrix operation formula Depth′ , M Depth′The corresponding image is the target depth image, that is, the Depth' image. The Depth' image and the RGB' image together constitute RGB'-D' data, wherein the preset matrix operation formula includes:

[0079] M Depth′ =M face *M Depth ;

[0080] The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

[0081] It should be noted that after the above steps, in the subsequent process of converting the target depth image into the face point cloud data of the target object in the viewing direction, the z w Whether the coordinates are 0 is used to remove the background, hair, ears, neck, torso and other areas in the original point cloud data that are irrelevant to the real-person expression modeling task that the present invention is concerned with, leaving a clean face area that is strongly related to the real-person expression, which is helpful to reduce the difficulty of subsequent point cloud alignment.

[0082] Furthermore, the embodiment of the present application converts the target depth image into facial point cloud data of the target object, which may specifically include: determining each pixel point with a non-zero depth of field value in the target depth image, and for any pixel point with a non-zero depth of field value, obtaining the pixel coordinates and depth value of the pixel point with a non-zero depth of field value; converting the pixel coordinates and depth value of the pixel point with a non-zero depth of field value into three-dimensional coordinates in a world coordinate system through a first homogeneous transformation formula; determining that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth of field value are the three-dimensional points corresponding to the pixel point with a non-zero depth of field value; and forming the facial point cloud data of the target object by the three-dimensional points corresponding to each pixel point with a non-zero depth of field value. Specifically, after determining each pixel point with a non-zero depth value in the target depth image, the embodiment of the present application can obtain the pixel coordinates and depth value of the pixel point with a non-zero depth value for any pixel point with a non-zero depth value; then the pixel coordinates and depth value of the pixel point with a non-zero depth value can be converted into the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value through the first homogeneous transformation formula, and then the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value can be determined as the three-dimensional point corresponding to the pixel point with a non-zero depth value, so that the face point cloud data of the target object in the viewing direction is composed of the three-dimensional points corresponding to each pixel point with a non-zero depth value. For example, for the Depth' image, determine each pixel point with a non-zero depth value in the Depth' image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates (u d ,vd ) and depth of field value z c , the pixel coordinates (u d ,v d ) and depth of field value z c , is converted into the three-dimensional coordinates (x w ,y w ,z w ), thereby converting the pixel points on the plane into three-dimensional points in space, so that the three-dimensional points corresponding to each pixel point with a non-zero depth value can be obtained, and the three-dimensional points corresponding to each pixel point with a non-zero depth value form a face point cloud of the target face in the viewing direction, and the color value of each three-dimensional point can be obtained from the corresponding pixel point in the above RGB′ image.

[0083] Step 140, performing registration processing on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data.

[0084] After the above steps, the facial point cloud data of the target object in each viewing direction can be obtained, and the facial point cloud data of the target object in each viewing direction can be stored in a set to form a facial point cloud data set of the target object. Subsequently, point cloud registration can be performed based on the facial point cloud data in the facial point cloud data set. For example, the facial voltage data of the target object in the frontal face posture can be selected from the facial point cloud data set as the posture reference point cloud data. Then, the posture reference point cloud data can be used to perform registration on the remaining facial point cloud data (i.e., non-posture reference point cloud data) in the facial point cloud data set, so that their spatial postures are basically consistent, and the face conversion point cloud data is obtained. Furthermore, the embodiment of the present application performs registration processing on the facial point cloud data in the facial point cloud data set to obtain posture reference point cloud data and face conversion point cloud data, which may specifically include: determining the center of mass of each facial point cloud data in the facial point cloud data set; performing unified processing based on the center of mass of each facial point cloud data set to obtain an intermediate facial point cloud data set corresponding to the target object; selecting posture reference point cloud data from the intermediate facial point cloud data set; and using the posture reference point cloud data to register the non-posture reference point cloud data in the intermediate facial point cloud data set to obtain face conversion point cloud data.

[0085] Step 150: Based on the preset face base point cloud data, the posture reference point cloud data and the face conversion point cloud data are fused to obtain target face point cloud data.

[0086] Specifically, after obtaining the posture reference point cloud data and the face conversion point cloud data, the embodiment of the present application can use the preset face base point cloud data as a shape reference to perform point cloud fusion on the posture reference point cloud data and all the face conversion point cloud data, so as to merge the face point cloud data under multiple different visual directions into a complete face point cloud, obtain the desired complete face point cloud data, and determine the obtained complete face point cloud data as the target face point cloud data.

[0087] Step 160: Perform surface reconstruction based on the target face point cloud data to obtain a three-dimensional face model corresponding to the target object.

[0088] Specifically, after obtaining the target facial point cloud data, the embodiment of the present application can obtain a three-dimensional facial model by performing surface reconstruction on the target facial point cloud data, and then add texture and other information to the three-dimensional facial model to obtain the final three-dimensional facial model.

[0089] It can be seen that after the embodiment of the present application obtains binocular image data under at least two viewing directions by photographing the target object through the camera device, the disparity map information of the binocular image data under each viewing direction can be converted to obtain image pair data corresponding to the binocular image data, and face point cloud processing can be performed based on the image pair data to obtain a face point cloud data set corresponding to the target object, and then registration processing can be performed based on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data, and the posture reference point cloud data and the face conversion point cloud data can be fused based on the preset face base point cloud data to facilitate Using face base point cloud data for shape correction, that is, adding face reference point cloud data as shape reference in the point cloud registration process, greatly reduces the shape deviation of the result obtained when the point cloud fusion is finally performed, such as irregular protrusions or depressions, abnormal nose shape, etc., to obtain the target face point cloud data, and then perform surface reconstruction based on the target face point cloud data to obtain a three-dimensional face model, so that the final three-dimensional face model meets the model accuracy required for real-person expression modeling, thereby achieving the purpose of using consumer-grade binocular camera equipment to complete the three-dimensional reconstruction task of real people's faces, while meeting the model accuracy required for three-dimensional face models and reducing costs.

[0090] It should be noted that after obtaining the desired point cloud of the complete face, that is, the target face point cloud data, it is necessary to perform surface reconstruction to restore its geometric shape, thereby turning it into a three-dimensional face model. Taking Delaunay triangulation reconstruction as an example, a feasible surface reconstruction algorithm is as follows Figure 5 Of course, the triangular reconstruction described is only the most common reconstruction method. In addition to triangles, quadrilaterals and pentagons can also be used to represent the surface. In the face reconstruction scenario targeted by the present invention, triangular reconstruction is the most preferred.

[0091] Among them, in the embodiment of the present invention, if Figure 6 FIG. 1 is a schematic diagram of an implementation flow of a method for determining target face point cloud data provided by an embodiment of the present invention. The method can be applied to a processor and may specifically include the following steps:

[0092] Step 610, determining the centroid of each face point cloud data in the face point cloud data set;

[0093] Step 620, performing unified processing according to the centroids of the various facial point cloud data to obtain an intermediate facial point cloud data set corresponding to the target object;

[0094] Step 630, selecting posture reference point cloud data from the intermediate face point cloud data set;

[0095] Step 640, using the posture reference point cloud data, registering the non-posture reference point cloud data in the intermediate face point cloud data set to obtain face conversion point cloud data;

[0096] Step 650: Based on the preset face base point cloud data, the posture reference point cloud data and the face conversion point cloud data are fused to obtain target face point cloud data.

[0097] Specifically, after determining the centroids corresponding to the facial point cloud data of the target face under the N viewing directions, the intermediate facial point cloud data of the target face under the N viewing directions can be determined according to the centroids, and then based on the intermediate facial point cloud data of the target face under the N viewing directions, an intermediate facial point cloud data set corresponding to the target object can be formed.

[0098] In an embodiment of the present invention, for the facial point cloud data of the target face under N viewing directions, the center of mass (i.e., the center of mass coordinates) corresponding to each of the facial point clouds of the target face under N viewing directions is determined. Among them, for the facial point cloud data of the target face under any viewing direction, the average value of the spatial coordinates of all the points in the facial point cloud data is obtained, and the average value is determined as the center of mass of the facial point cloud. Further, the embodiment of the present application determines the center of mass of each facial point cloud data in the facial point cloud data set, which may specifically include: obtaining the spatial coordinates of all points in each facial point cloud data in the facial point cloud data set; and determining the average value of the spatial coordinates of all points in each facial point cloud data as the center of mass of each facial point cloud data.

[0099] For example, the face point cloud data of the target face under N viewing directions constitute a face point cloud data set C. For the face point cloud data c in the face point cloud data set C, i , face point cloud data c i The center of mass μ is the face point cloud of the target face in any of the face point clouds of the target face in N viewing directions.i Point cloud face point cloud data c i The average value of all point space coordinates is used to obtain the face point cloud data c i The average value of all the spatial coordinates of the points is determined to be the face point cloud data c i The calculation formula is as follows, where N i It is the face point cloud data c i The number of points contained, p j It is the face point cloud data c i The spatial coordinates of the jth point in .

[0100]

[0101] In addition, after determining the centroids corresponding to the facial point cloud data of the target face under N viewing directions, the intermediate facial point cloud of the target face under N viewing directions can be determined based on the centroids. Among them, for the facial point cloud of the target face under any viewing direction, the spatial coordinates of each point in the facial point cloud are subtracted from the centroid (corresponding to the facial point cloud) to obtain the intermediate facial point cloud data of the target face under the viewing direction. In this way, all point cloud data can be unified into a centroid coordinate system with the centroid as the origin of its coordinate system, which is convenient for the subsequent calculation of the target rotation matrix R and the target offset vector t by the SVD algorithm. Thus, an intermediate facial point cloud data set corresponding to the target object can be formed based on the intermediate facial point cloud data of the target face under N viewing directions. Furthermore, the embodiments of the present application perform unified processing according to the centroids of the various facial point cloud data to obtain an intermediate facial point cloud data set corresponding to the target object, which may specifically include: for each facial point cloud data, subtracting the coordinates of the mass point from the spatial coordinates of each point in the facial point cloud data to obtain the intermediate facial point cloud data corresponding to the facial point cloud data; based on the intermediate facial point cloud data corresponding to each facial point cloud data, forming an intermediate facial point cloud data set corresponding to the target object.

[0102] For example, face point cloud data c i Corresponding to the face point cloud of the target face under any viewing direction in the face point cloud of the target face under N viewing directions, the face point cloud data c i Every point p in j Subtract the face point cloud data c i The corresponding center of mass μ i , the new point cloud c′ i , new point cloud c′ i That is, the middle face point cloud data of the target face under the viewing direction, from which a new point cloud set C can be formed μ , as the intermediate face point cloud dataset. The calculation formula is as follows:

[0103] c′i ={p j -μ i |p j ∈c i}={p′ j}.

[0104] In actual processing, the posture reference point cloud data can be selected from the middle face point cloud data of the target face under the N viewing directions, that is, the posture reference point cloud data can be selected from the middle face point cloud data set, and the posture reference point cloud data can include the middle face point cloud data of the target face under the frontal viewing direction. For example, for the middle face point cloud data of the target face under N viewing directions, the point cloud set C μ , from the point cloud set C μ The middle face point cloud data of the target face under the frontal viewing direction is selected as the posture reference point cloud data, that is, the posture reference point cloud P; then, the posture reference point cloud P can be used to align the remaining middle face point cloud data of the target object's face under the N-1 viewing directions to obtain the face conversion point cloud data of the target object under the N-1 viewing directions.

[0105] In the embodiment of the present invention, the posture reference point cloud data is the middle face point cloud of the target face in the frontal viewing direction, and thus there are the remaining middle face point cloud data of the target face in N-1 viewing directions (i.e. the non-posture reference point cloud data in the middle face point cloud data set), so that the posture reference point cloud data can be used to align the remaining middle face point cloud data of the target face in N-1 viewing directions to obtain the conversion point cloud data of the target face in N-1 viewing directions. Further, the embodiment of the present application uses the posture reference point cloud data to align the non-posture reference point cloud data in the middle face point cloud data set to obtain face conversion point cloud data, including: extracting any of the non-posture reference point cloud data from the middle face point cloud data set as the target middle face point cloud data; selecting M target point data from the middle face point cloud data; searching for the nearest neighbor point data corresponding to each of the M target point data from the posture reference point cloud data; generating the nearest point pair data corresponding to each target point data according to the nearest neighbor point data corresponding to each target point data; According to the nearest point pair data corresponding to the M target point data, a nearest point pair set is formed; according to the nearest point pair set, a target rotation matrix and a target offset vector are determined; using the target rotation matrix and the target offset vector, the target intermediate face point cloud data is transformed to obtain the initial conversion point cloud data corresponding to the target intermediate face point cloud data; the conversion error between the posture reference point cloud data and the initial conversion point cloud data is determined; if the conversion error is less than a preset first threshold, the initial conversion point cloud data is determined as the face conversion point cloud data corresponding to the target intermediate face point cloud data. Wherein, M is a positive integer, and M is greater than 1.

[0106] It should be noted that the remaining intermediate facial point cloud data of the target face in N-1 viewing directions are registered using the posture reference point cloud data to obtain the transformed point cloud data of the target face in N-1 viewing directions. The purpose is to make their (the remaining intermediate facial point clouds of the target face in N-1 viewing directions) spatial postures basically consistent.

[0107] Among them, Figure 7 As shown, it is a schematic diagram of an implementation process of registering the remaining intermediate face point cloud data of the target face in N-1 viewing directions provided by an embodiment of the present invention. The method can be specifically applied to a processor and may include the following steps:

[0108] Step 701 : for the remaining intermediate facial point cloud data of the target face in any of the N-1 viewing directions, select M target point data from the remaining intermediate facial point cloud data.

[0109] In the embodiment of the present invention, for the remaining intermediate face point cloud data of the target face under N-1 viewing directions, for the remaining intermediate face point cloud data of the target face under any viewing direction, M point data are (randomly) selected from the remaining intermediate face point cloud as target point data, that is, any non-pose reference point cloud data is extracted from the intermediate face point cloud data set as target intermediate face point cloud data, and M target point data are selected from the target intermediate face point cloud data. For example, for the remaining intermediate face point cloud of the target face under N-1 viewing directions, for the remaining intermediate face point cloud Q of the target face under any viewing direction (Q belongs to the point cloud set C μ ), randomly select M point data from the remaining intermediate face point cloud Q as M target point data.

[0110] Step 702: Using a preset KD-tree algorithm, search for the nearest neighbor point data corresponding to each of the M points in the posture reference point cloud data.

[0111] In the embodiment of the present invention, for the pose reference point cloud data, a KD-tree data structure is established on it to store the spatial coordinates of all its points, so as to use the preset KD-tree algorithm, that is, with the help of the characteristics of KD-tree, to find the nearest neighbor point data corresponding to each of the M points from the pose reference point cloud data, that is, to find the nearest neighbor point data corresponding to each of the M target point data from the pose reference point cloud data, so as to find the nearest neighbor point corresponding to each point (among the M points) from the pose reference point cloud. For example, a KD-tree data structure is established on the pose reference point cloud P to store the spatial coordinates of all its points. For the remaining intermediate face point cloud Q (Q belongs to the point cloud set C μ ), and use the preset KD-tree algorithm, that is, with the help of the characteristics of KD-tree, to find the nearest neighbor points corresponding to each of the M points from the posture reference point cloud.

[0112] Step 703: M pairs of nearest point pair data are formed from the M target point data and the nearest neighbor point data corresponding to each of the M target point data, and the M pairs of nearest point pair data form a nearest point pair set.

[0113] For the above M target point data, and the nearest neighbor point data corresponding to each of the M target point data, M pairs of nearest point pair data can be formed, and the M pairs of nearest point pair data form a nearest point pair set. For example, for the above M points in the above example, and the nearest neighbor points corresponding to each of the M points, M pairs of nearest point pairs can be formed, and the M pairs of nearest point pairs form a nearest point pair set H, as shown below.

[0114] H={(p i ,q i )|p i∈P,q i ∈Q,1≤i≤m}.

[0115] Step 704: Determine a target rotation matrix and a target offset vector according to the set of closest point pairs.

[0116] In the embodiment of the present invention, for the closest point pair set obtained above, a target rotation matrix and a target offset vector can be determined according to the closest point pair set.

[0117] Among them, the M pairs of nearest point data in the nearest point pair set are input into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula.

[0118] Perform SVD decomposition on the construction matrix to obtain the first eigenvalue and the second eigenvalue, input the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtain the target rotation matrix output by the preset rotation matrix operation formula.

[0119] The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud data are input into the preset offset vector operation formula to obtain the target offset vector output by the preset offset vector operation formula.

[0120] For example, a construction matrix W of size 3x3 is constructed for SVD decomposition, and M pairs of closest point pairs in the closest point pair set are input into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula, wherein the preset matrix construction formula includes:

[0121]

[0122] The W includes the construction matrix, the N H Including the number of the closest point pair data in the closest point pair set, the q i and the p i Includes the closest point pair data in the closest point pair set.

[0123] Perform SVD decomposition on the construction matrix W to obtain the first eigenvalue U and the second eigenvalue V T , σ1, σ2, σ3 are the eigenvalues, and the calculation formula is as follows:

[0124]

[0125] The first eigenvalue U and the second eigenvalue V T Input the preset rotation matrix operation formula to obtain the target rotation matrix R output by the preset rotation matrix operation formula, wherein the preset rotation matrix operation formula includes:

[0126] R=UV T ;

[0127] The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T Including the second eigenvalue.

[0128] The target rotation matrix R, the centroid of the posture reference point cloud data P, and the centroid of the remaining intermediate face point cloud data Q are input into the preset offset vector operation formula to obtain the target offset vector t output by the preset offset vector operation formula, wherein the preset offset vector operation formula includes:

[0129] t=μ p -Rμ q ;

[0130] The t includes the target offset vector, the μ p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the μ q Include the centroid of the remaining intermediate face point cloud.

[0131] After the above steps, for the remaining intermediate facial point cloud data of the target face in any of the N-1 viewing directions, the corresponding target rotation matrix and target offset vector can be obtained.

[0132] Step 705: transform the remaining intermediate face point cloud data using the target rotation matrix and the target offset vector to obtain initial transformed point cloud data of the target face in the viewing direction.

[0133] In an embodiment of the present invention, for the remaining intermediate facial point cloud data of the target face in any one of the N-1 viewing directions, the target rotation matrix and the target offset vector corresponding to the remaining intermediate facial point cloud data can be used to transform the remaining intermediate facial point cloud data to obtain the initial transformed point cloud data of the target face in the viewing direction, that is, the target intermediate facial point cloud data is transformed to obtain the initial transformed point cloud data corresponding to the target intermediate facial point cloud data. For example, for the remaining intermediate facial point cloud data Q of the target face in any one of the N-1 viewing directions, the target rotation matrix R and the target offset vector t corresponding to the remaining intermediate facial point cloud data Q can be used to transform the remaining intermediate facial point cloud data Q to obtain the initial transformed point cloud data of the target face in the viewing direction. Among them, for the remaining intermediate face point cloud data of the target face in any of the N-1 viewing directions, the target rotation matrix and the target offset vector corresponding to the remaining intermediate face point cloud data are input into the point cloud conversion formula to obtain the initial conversion point cloud data of the target face in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes:

[0134] Q′={Rq+t|q∈Q};

[0135] The Q′ includes the initial conversion point cloud data, the Q includes the remaining intermediate face point cloud data, the t includes the target offset vector, and the R includes the target rotation matrix.

[0136] For example, for the remaining intermediate facial point cloud data Q of the target face in any of the N-1 viewing directions, the target rotation matrix R and the target offset vector t corresponding to the remaining intermediate facial point cloud data Q can be input into the above formula to obtain the initial conversion point cloud data Q′ of the target face in the viewing direction output by the point cloud conversion formula. In this way, the remaining intermediate facial point cloud data Q is converted by the above formula, and the spatial posture of the obtained initial conversion point cloud data Q′ approaches the posture reference point cloud P, that is, approaches the frontal viewing direction.

[0137] Step 706: determine a conversion error between the posture reference point cloud data and the initial conversion point cloud data, and determine the face conversion point cloud data of the target object in the viewing direction according to the conversion error.

[0138] Specifically, for the initial conversion point cloud data of the target face in any of the N-1 viewing directions, in an embodiment of the present invention, after determining the (point cloud) conversion error between the posture reference point cloud data and the initial conversion point cloud data, the conversion point cloud of the target face in the viewing direction can be determined based on the conversion error, that is, the face conversion point cloud data of the target object in the viewing direction can be determined.

[0139] Among them, for the initial conversion point cloud data of the target face in any of the N-1 viewing directions, the posture reference point cloud data and the initial conversion point cloud data can be input into a preset posture difference calculation formula to obtain the posture difference between the posture reference point cloud data and the initial conversion point cloud data output by the preset posture difference calculation formula, and determine the posture difference as the conversion error.

[0140] For example, for the initial conversion point cloud data Q' of the target face in any one of the N-1 viewing directions, the initial conversion point cloud data Q' and the posture reference point cloud data P can be input into a preset posture difference calculation formula to obtain the posture difference between the initial conversion point cloud data Q' and the posture reference point cloud data P output by the preset posture difference calculation formula as the (point cloud) conversion error L, wherein the preset posture difference calculation formula includes:

[0141]

[0142] The Q′ includes the initial conversion point cloud data, and the P includes the posture reference point cloud data.

[0143] In addition, for the initial conversion point cloud data of the target face in any of the N-1 viewing directions, in the embodiment of the present invention, the (point cloud) conversion error between the posture reference point cloud data and the initial conversion point cloud data is determined. If the conversion error is less than the preset first threshold, the initial conversion point cloud data is determined to be the conversion point cloud data of the target face in the viewing direction, and is output as the face conversion point cloud data of the target object in the viewing direction. If the conversion error is not less than the preset first threshold, jump to the step of selecting M target point data from the remaining intermediate face point cloud data for re-registration until the number of registration iterations reaches the preset number threshold, and output the conversion point cloud data of the target face in the viewing direction.

[0144] Furthermore, the embodiment of the present application utilizes the posture reference point cloud data to align the non-posture reference point cloud data in the intermediate facial point cloud data set, and may also include: determining whether the conversion error is less than a preset first threshold; if the conversion error is not less than the preset first threshold, determining the number of registration iterations corresponding to the target intermediate facial point cloud data; if the number of registration iterations is less than the preset iteration number threshold, jumping to the step of selecting M target point data from the target intermediate facial point cloud data for re-alignment until the number of registration iterations corresponding to the target intermediate facial point cloud data reaches the preset number threshold. Specifically, if the conversion error is not less than the preset first threshold, first determine whether the number of registration iterations reaches the preset number threshold, if the number of registration iterations reaches the preset number threshold, determine the initial conversion point cloud data as the conversion point cloud data of the target face in the viewing direction; if the number of registration iterations does not reach the preset number threshold, jump to the step of selecting M points from the remaining intermediate face point cloud to re-perform the registration until the number of registration iterations reaches the preset number, and output the conversion point cloud data of the target face in the viewing direction as the face conversion point cloud data of the target object in the viewing direction. Subsequently, the preset face reference point cloud data can be used to fuse the posture reference point cloud data and the face conversion point cloud data of the target object in the viewing direction of N-1 to obtain the target face point cloud data.

[0145] It should be noted that the facial reference point cloud data may be a male or female facial reference point cloud. Specifically, an open source reference three-dimensional model may be obtained, and the facial part thereof may be captured as the facial reference point cloud. The embodiments of the present invention are not limited to this.

[0146] It can be seen that the embodiment of the present invention introduces preset facial reference point cloud data. During the point cloud registration process, the facial reference point cloud data is used to fuse the posture reference point cloud data and the facial conversion point cloud data of the target object in N-1 viewing directions to obtain the target facial point cloud data.

[0147] Among them, Figure 8As shown, it is a schematic diagram of an implementation process of fusing posture reference point cloud data and face conversion point cloud data of a target object in N-1 viewing directions provided by an embodiment of the present invention. The method is specifically used for a processor and may include the following steps:

[0148] Step 801: Merge the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions to obtain complete face point cloud data.

[0149] In the embodiment of the present invention, the posture reference point cloud data and the face conversion point cloud data of the target object in the N-1 viewing direction are merged to obtain the complete face point cloud data. For example, the posture reference point cloud data P and the face conversion point cloud data Q' of the target object in the N-1 viewing direction are merged to obtain the complete face point cloud data M. It should be noted that the complete face point cloud data may be a point cloud data containing a complete face, which is not limited in the embodiment of the present invention.

[0150] Step 802 , selecting a plurality of point data from the complete face point cloud data to form verification point cloud data, and determining a shape difference between the verification point cloud data and preset face reference point cloud data.

[0151] For the complete face point cloud data, multiple point data are selected from the complete face point cloud data to form verification point cloud data, and the shape difference between the verification point cloud data and the preset face reference point cloud data is determined.

[0152] For example, for the complete face point cloud data M, a plurality of points are randomly selected from the complete face point cloud data M to form the verification point cloud data m, and the shape difference between the verification point cloud data m and the preset face reference point cloud data B is determined.

[0153] Among them, in an embodiment of the present invention, the shape difference between the verification point cloud data and the preset face reference point cloud data is determined specifically in the following manner: the chamfer distance between the verification point cloud data and the preset face reference point cloud data is obtained, and the chamfer distance is determined as the shape difference between the verification point cloud data and the preset face reference point cloud data.

[0154] It should be noted that, in an embodiment of the present invention, one of the following can also be obtained between the verification point cloud data and the preset face reference point cloud data: L1, L2, deformation of L1, deformation of L2, which is used to describe the shape difference between the verification point cloud data and the preset face reference point cloud data. The embodiment of the present invention is not limited to this.

[0155] For example, in an embodiment of the present invention, the Chamfer Distance is used as a specific measurement of the shape difference E, the chamfer distance between the verification point cloud data m and the face reference point cloud data B is obtained, and the chamfer distance is determined to be the shape difference E between the verification point cloud data m and the face reference point cloud data B.

[0156] The calculation formula is as follows, where N m and N B are the sizes of the downsampled verification point cloud data m and the face reference point cloud data B (i.e. the number of elements therein), respectively. The first term is the average of the sum of the minimum distances from any point in the verification point cloud data m to the reference point cloud B, and the second term is the average of the sum of the minimum distances from any point in the reference point cloud B to the verification point cloud data m. The two are combined as the chamfer distance in 3D space. The smaller its value is, the smaller the shape difference between the two point clouds is.

[0157]

[0158] Step 803: According to the shape difference, the posture reference point cloud data and the face conversion point cloud data of the target object in the N-1 viewing directions are fused to obtain the target face point cloud data.

[0159] For the shape difference between the verification point cloud data and the preset face reference point cloud data, the posture reference point cloud data and the transformed point cloud data of the target face in N-1 viewing directions are fused according to the shape difference to obtain the target face point cloud data.

[0160] Among them, if the shape difference is less than the preset second threshold, the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions are fused to obtain the target face point cloud data; if the shape difference is not less than the preset second threshold, jump to the step of selecting multiple point data from the complete face point cloud data to form the verification point cloud data.

[0161] Specifically, if the shape difference is less than a preset second threshold, the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions are initially fused to obtain the target face point cloud data for display so that the user can confirm whether the shape is normal.

[0162] If the user confirms that the shape is normal, the shape confirmation instruction is triggered. When the user's shape confirmation instruction is received, the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions are fused to obtain the target face point cloud data.

[0163] If the user confirms that the shape is abnormal, a shape error instruction is triggered. After receiving the user's shape error instruction, the preset second threshold is adjusted, and the process jumps to the step of selecting multiple point data from the complete face point cloud data to form the verification point cloud data.

[0164] For example, if the shape difference E is less than the preset second threshold c, the posture reference point cloud data P and the conversion point cloud data Q' of the target face in N-1 viewing directions are initially fused to obtain the target face point cloud data for display so that the user can confirm whether the shape is normal.

[0165] If the user confirms that the shape is normal, the shape confirmation instruction is triggered. When the user's shape confirmation instruction is received, the posture reference point cloud data P and the conversion point cloud data Q' of the target face in N-1 viewing directions are fused to obtain the target face point cloud data A.

[0166] If the user confirms that the shape is abnormal, a shape error instruction is triggered. After receiving the user's shape error instruction, the preset second threshold c is adjusted, and the process jumps to the step of selecting multiple point data from the complete face point cloud data to form the verification point cloud data m.

[0167] Through the above description of the technical solution provided by the embodiment of the present invention, after the target object is photographed by a camera device to obtain binocular image data in at least two viewing directions, the disparity map information of the binocular image data in each viewing direction is converted to obtain image pair data corresponding to the binocular image data, and according to the color image information and depth image information in the image pair data, a target color image aligned with the depth image information is determined, and then the target color image is converted into a face mask matrix according to the face area information in the target color image, the depth image information is processed using the face mask matrix to obtain a target depth image, and then the target depth image is converted into face point cloud data of the target object to obtain a face point cloud data set corresponding to the target object, and then the face point cloud data set corresponding to the target object can be obtained. The face point cloud data in the face point cloud data set is registered to obtain posture reference point cloud data and face conversion point cloud data, and the posture reference point cloud data and the face conversion point cloud data are fused based on the preset face base point cloud data, that is, the face reference point cloud is added as a shape reference in the point cloud registration process, which greatly reduces the shape deviation of the result obtained when the point cloud fusion is finally performed, such as irregular protrusions or depressions, abnormal nose shape, etc., to obtain the target face point cloud, and perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model, so that the final three-dimensional face model meets the model accuracy required for real-person expression modeling, and achieves reliable three-dimensional reconstruction accuracy from low-precision point cloud data, so as to achieve the purpose of completing the three-dimensional reconstruction task of real-person faces using consumer-grade camera equipment.

[0168] In addition, in the embodiment of the present invention, only one or more consumer-grade binocular camera devices are required, thereby obtaining binocular image data in multiple different viewing directions to participate in the subsequent three-dimensional face model reconstruction, with low hardware investment cost and simple equipment maintenance.

[0169] Given that only one or more binocular camera devices are required in the embodiments of the present invention, the scale of point cloud data subsequently collected and processed is generally in the thousands. Compared with three-dimensional reconstruction involving tens of thousands, hundreds of thousands, or millions of points, a large amount of computing resources can be saved, thereby improving the efficiency of three-dimensional face modeling.

[0170] The present application also provides a three-dimensional face reconstruction device. Fig. 9 As shown, the 3D face reconstruction device provided in the embodiment of the present application includes the following modules:

[0171] A binocular image data acquisition module 910 is used to acquire binocular image data of a target object in at least two viewing directions through a binocular camera device;

[0172] A binocular image data conversion module 920 is used to convert the binocular image data under any of the viewing angles according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data;

[0173] A face point cloud data set module 930 is used to perform face point cloud processing on the data according to the image to obtain a face point cloud data set corresponding to the target object;

[0174] A point cloud registration processing module 940 is used to perform registration processing based on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data;

[0175] A fusion processing module 950 is used to fuse the posture reference point cloud data and the face conversion point cloud data based on the preset face base point cloud data to obtain target face point cloud data;

[0176] The surface reconstruction module 960 is used to perform surface reconstruction based on the target face point cloud data to obtain a three-dimensional face model corresponding to the target object.

[0177] Optionally, the binocular image data acquisition module 910 includes: a rotation control submodule and a shooting submodule. The rotation control submodule is used to control the binocular camera device to rotate within a preset angle range with the front face of the target object as the central axis, and the binocular camera device and the target object are located in the same horizontal plane; the shooting submodule is used to control the binocular camera device to collect binocular image data of the target object in the current viewing direction when it is detected that the binocular camera device rotates to any preset viewing direction.

[0178] Optionally, the binocular image data acquisition module 910 includes: a shooting control submodule. The shooting control submodule is used to control the binocular camera devices located in N viewing angles to capture the target part of the target object, and obtain the binocular image data of the target object in N viewing angles, where N is greater than or equal to 2.

[0179] Optionally, the binocular image data conversion module 920 includes:

[0180] A disparity determination submodule, used to determine disparity map information of the binocular image data, wherein the disparity map information includes disparity values ​​of at least two pixel points;

[0181] A conversion determination submodule, configured to convert a disparity value of any pixel point in the disparity map information into a corresponding depth of field value using a preset conversion parameter;

[0182] The image pair data is generated according to the depth value corresponding to each pixel point in the disparity map information.

[0183] Optionally, the image pair data includes color image information and depth image information, and the face point cloud data set module 930 includes:

[0184] A target color image determination submodule, configured to determine a target color image aligned with the depth image information based on the color image information and the depth image information in the image pair data;

[0185] The point cloud processing submodule is used to perform point cloud processing based on the facial area information of the human face in the target color image to obtain the human face point cloud data set.

[0186] Optionally, the target color image determination submodule includes:

[0187] A first conversion unit, configured to convert, for any depth pixel point of the depth image information, a pixel coordinate and a depth value of the depth pixel point into a three-dimensional coordinate in a world coordinate system;

[0188] A second conversion unit, configured to convert the three-dimensional coordinates into target pixel coordinates of the depth pixel points, where the target pixel coordinates of the depth pixel points correspond to color pixel points of the color image information in the image pair data;

[0189] A target color pixel point unit, used to determine a target color pixel point corresponding to a target pixel coordinate of the depth pixel point from the color pixel points of the color image information;

[0190] A color value unit, used to obtain a color value corresponding to the target color pixel point, and assign the color value to the depth pixel point;

[0191] The target color image unit is used to recompose a target color image aligned with the depth image information by using the color value corresponding to each depth pixel point of the depth image information.

[0192] Optionally, the point cloud processing submodule includes:

[0193] A facial region determination unit, used to determine facial region information of a human face in the target color image;

[0194] A face mask matrix unit, used to determine a face mask matrix corresponding to the target color image according to the face area information of the face;

[0195] A target depth image unit, used to process the depth image information using the face mask matrix to obtain a target depth image;

[0196] The face cloud data unit is used to convert the target depth image into face point cloud data of the target object, and form the face cloud data set based on the face point cloud data of the target object.

[0197] Optionally, the first conversion unit is specifically used to: input the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous conversion formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous conversion formula;

[0198] Wherein, the first homogeneous transformation formula includes:

[0199]

[0200]

[0201] The K d The internal parameter matrix of the depth sensing module in the binocular camera device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2dThe pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

[0202] Optionally, the second conversion unit is specifically used to: input the three-dimensional coordinates into a second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula;

[0203] Wherein, the second homogeneous transformation formula includes:

[0204]

[0205]

[0206] The K c Including the internal parameter matrix of the color photosensitive module in the binocular camera device, the T w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2c and the offset vector t w2c The three-dimensional coordinates include (the x w , the y w , the z w ), the target pixel coordinates of the depth pixel point include (the u c , the v c ).

[0207] Optionally, the face cloud data unit is specifically used to: determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and depth value of the pixel point with a non-zero depth value; convert the pixel coordinates and depth value of the pixel point with a non-zero depth value into three-dimensional coordinates in a world coordinate system by a first homogeneous transformation formula; determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value; and form the face point cloud data of the target object by the three-dimensional points corresponding to each pixel point with a non-zero depth value.

[0208] Optionally, the point cloud registration processing module 940 includes:

[0209] A centroid determination submodule, used to determine the centroid of each face point cloud data in the face point cloud data set;

[0210] A centroid processing submodule, used for performing unified processing according to the centroids of the respective face point cloud data to obtain an intermediate face point cloud data set corresponding to the target object;

[0211] A posture reference point cloud selection submodule is used to select posture reference point cloud data from the intermediate face point cloud data set;

[0212] The registration submodule is used to register the non-posture reference point cloud data in the intermediate face point cloud data set using the posture reference point cloud data to obtain face conversion point cloud data.

[0213] Optionally, the centroid processing submodule includes:

[0214] An intermediate face point cloud data unit is used for obtaining intermediate face point cloud data corresponding to each face point cloud data by subtracting the coordinates of the mass point from the spatial coordinates of each point in the face point cloud data;

[0215] The intermediate facial point cloud data set unit is used to form an intermediate facial point cloud data set corresponding to the target object based on the intermediate facial point cloud data corresponding to each facial point cloud data.

[0216] Optionally, the registration submodule includes:

[0217] A target intermediate face point cloud data extraction unit, used to extract any of the non-gesture reference point cloud data from the intermediate face point cloud data set to serve as the target intermediate face point cloud data;

[0218] A target point data selection unit, used to select M target point data from the intermediate face point cloud data;

[0219] A nearest neighbor point data unit, used to find the nearest neighbor point data corresponding to each of the M target point data from the posture reference point cloud data;

[0220] A nearest point pair data unit is used to generate the nearest point pair data corresponding to each target point data according to the nearest neighbor point data corresponding to each target point data;

[0221] A closest point pair collection unit, used to form a closest point pair collection according to the closest point pair data corresponding to the M target point data;

[0222] A target determination unit, used to determine a target rotation matrix and a target offset vector according to the set of closest point pairs;

[0223] A facial point cloud data conversion unit, configured to convert the target intermediate facial point cloud data using the target rotation matrix and the target offset vector to obtain initial converted point cloud data corresponding to the target intermediate facial point cloud data;

[0224] a conversion error determination unit, configured to determine a conversion error between the posture reference point cloud data and the initial conversion point cloud data;

[0225] The face conversion point cloud data unit is used to determine the initial conversion point cloud data as the face conversion point cloud data corresponding to the target intermediate face point cloud data if the conversion error is less than a preset first threshold.

[0226] Optionally, the registration submodule further includes:

[0227] A conversion error judgment unit, used to judge whether the conversion error is less than a preset first threshold;

[0228] A registration iteration number determination unit is used to determine the number of registration iterations corresponding to the target intermediate face point cloud data when the conversion error is not less than a preset first threshold; when the number of registration iterations is less than a preset iteration number threshold, trigger the registration iteration processing unit to jump to the step of selecting M target point data from the intermediate face point cloud data for re-registration;

[0229] The registration iteration processing unit is used to jump to the step of selecting M target point data from the target intermediate face point cloud data for re-registration until the number of registration iterations corresponding to the target intermediate face point cloud data reaches a preset number threshold.

[0230] Optionally, the centroid determination submodule includes:

[0231] A point space coordinate unit, used to obtain all point space coordinates of each face point cloud data in the face point cloud data set;

[0232] The centroid determination unit is used to determine the average value of the spatial coordinates of all points in each face point cloud data as the centroid of each face point cloud data.

[0233] It should be noted that the three-dimensional face reconstruction device provided above can execute the three-dimensional face reconstruction method provided by any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0234] Furthermore, an embodiment of the present application also provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the three-dimensional face reconstruction method described in any one of the above method embodiments when executing the program stored in the memory.

[0235] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the three-dimensional face reconstruction method described in any one of the above method embodiments are implemented.

[0236] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device, equipment, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0237] In this article, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0238] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A three-dimensional face reconstruction method, characterized in that: include: Collect binocular image data of the target object in at least two viewing directions through a binocular camera device; For the binocular image data under any of the viewing angle directions, converting the binocular image data according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data; Performing face point cloud processing on the data according to the image to obtain a face point cloud data set corresponding to the target object; Performing registration processing on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data; Based on the preset human face reference point cloud data, the posture reference point cloud data and the human face conversion point cloud data are fused to obtain the target human face point cloud data; Performing surface reconstruction based on the target face point cloud data to obtain a three-dimensional face model corresponding to the target object; The step of performing registration processing on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data includes: Determine the centroid of each face point cloud data in the face point cloud data set; Performing unified processing according to the centroids of the various face point cloud data to obtain an intermediate face point cloud data set corresponding to the target object; Selecting posture reference point cloud data from the intermediate face point cloud data set, wherein the posture reference point cloud data refers to face point cloud data of a target object in a frontal face posture selected from the face point cloud data set; Using the posture reference point cloud data, registering the non-posture reference point cloud data in the intermediate face point cloud data set to obtain face conversion point cloud data, wherein the face conversion point cloud data refers to point cloud data in which the posture reference point cloud data and the non-posture reference point cloud data have the same spatial posture after registration; The step of uniformly processing the centroids of the face point cloud data to obtain an intermediate face point cloud data set corresponding to the target object includes: For each facial point cloud data, subtract the coordinates of the mass point from the spatial coordinates of each point in the facial point cloud data to obtain intermediate facial point cloud data corresponding to the facial point cloud data; Based on the intermediate face point cloud data corresponding to each face point cloud data, an intermediate face point cloud data set corresponding to the target object is formed, wherein the intermediate face point cloud data set refers to a point cloud set formed by the intermediate face point cloud data corresponding to each face point cloud data under N viewing directions, where N is greater than or equal to 2; The step of fusing the posture reference point cloud data and the face conversion point cloud data to obtain the target face point cloud data includes: Merging the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions to obtain complete face point cloud data; Selecting a plurality of point data from the complete face point cloud data to form verification point cloud data, and determining a shape difference between the verification point cloud data and preset face reference point cloud data; According to the shape difference, the posture reference point cloud data and the face conversion point cloud data of the target object in the N-1 viewing directions are fused to obtain the target face point cloud data; If the shape difference is less than a preset second threshold, the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions are initially fused to obtain the target face point cloud data; If the user confirms that the shape is normal, the shape confirmation instruction is triggered. When the user's shape confirmation instruction is received, the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions are fused to obtain the target face point cloud data; If the user confirms that the shape is abnormal, a shape error instruction is triggered. After receiving the shape error instruction from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of point data from the complete face point cloud data to form the verification point cloud data; The image pair data includes color image information and depth image information, and the face point cloud processing is performed based on the image pair data to obtain a face point cloud data set corresponding to the target object, including: Determining, according to the color image information and the depth image information in the image pair data, a target color image aligned with the depth image information; Point cloud processing is performed based on the facial area information of the human face in the target color image to obtain the human face point cloud data set.

2. The three-dimensional face reconstruction method according to claim 1, characterized in that: The method of collecting binocular image data of a target object in at least two viewing directions by using a binocular camera device includes: Taking the front face of the target object as the central axis, control the binocular camera device to rotate within a preset angle range, and the binocular camera device and the target object are located in the same horizontal plane; when it is detected that the binocular camera device rotates to any preset viewing direction, control the binocular camera device to collect binocular image data of the target object in the current viewing direction; or, The binocular camera devices located in N viewing angles are controlled to capture the target part of the target object to obtain binocular image data of the target object in the N viewing angles.

3. The three-dimensional face reconstruction method according to claim 1, characterized in that: The converting according to the disparity map information of the binocular image data to obtain the image pair data corresponding to the binocular image data includes: Determine disparity map information of the binocular image data, where the disparity map information includes disparity values ​​of at least two pixels; For a disparity value of any pixel point in the disparity map information, convert the disparity value into a corresponding depth of field value using a preset conversion parameter; The image pair data is generated according to the depth value corresponding to each pixel point in the disparity map information.

4. The three-dimensional face reconstruction method according to claim 1, characterized in that: The step of determining, according to the color image information and the depth image information in the image pair data, a target color image aligned with the depth image information comprises: For any depth pixel point of the depth image information, convert the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system; Converting the three-dimensional coordinates into target pixel coordinates of the depth pixel points, where the target pixel coordinates of the depth pixel points correspond to color pixel points of the color image information in the image pair data; Determining, from the color pixels of the color image information, a target color pixel corresponding to the target pixel coordinates of the depth pixel; Obtaining a color value corresponding to the target color pixel, and assigning the color value to the depth pixel; The target color image aligned with the depth image information is recomposed by using the color value corresponding to each depth pixel point of the depth image information.

5. The three-dimensional face reconstruction method according to claim 1, characterized in that: Performing point cloud processing based on the facial region information in the target color image to obtain the facial point cloud data set includes: Determining facial region information of a human face in the target color image; Determining a face mask matrix corresponding to the target color image according to the face area information; Processing the depth image information using the face mask matrix to obtain a target depth image; The target depth image is converted into face point cloud data of a target object, and the face point cloud data set is formed based on the face point cloud data of the target object.

6. The three-dimensional face reconstruction method according to claim 4, characterized in that: The step of converting the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system includes: Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula; Wherein, the first homogeneous transformation formula includes: ； ； Including the internal parameter matrix of the depth sensing module in the binocular camera device, The external parameter matrix of the depth sensing module is composed of the rotation matrix and the offset vector The pixel coordinates of the depth pixel point include ( , ), Including the depth of field value, the three-dimensional coordinates include ( , , ).

7. The method according to claim 4, characterized in that The converting the three-dimensional coordinates into the target pixel coordinates of the depth pixel point comprises: Inputting the three-dimensional coordinates into a second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula; Wherein, the second homogeneous transformation formula includes: ； ； Including the internal parameter matrix of the color photosensitive module in the binocular camera device, The external parameter matrix of the color photosensitive module is composed of the rotation matrix and the offset vector The three-dimensional coordinates include ( , , ), the target pixel coordinates of the depth pixel point include ( , ).

8. The three-dimensional face reconstruction method according to claim 5, characterized in that: The step of converting the target depth image into face point cloud data of the target object comprises: Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value; The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system by a first homogeneous transformation formula; Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value; The three-dimensional points corresponding to the pixels with non-zero depth values ​​respectively constitute the face point cloud data of the target object.

9. The three-dimensional face reconstruction method according to claim 1, characterized in that: The method of using the posture reference point cloud data to register the non-posture reference point cloud data in the intermediate face point cloud data set to obtain face conversion point cloud data includes: Extracting any of the non-pose reference point cloud data from the intermediate face point cloud data set as target intermediate face point cloud data; Select M target point data from the intermediate face point cloud data; From the posture reference point cloud data, searching for the nearest neighbor point data corresponding to each of the M target point data; Generate the nearest point pair data corresponding to each target point data according to the nearest neighbor point data corresponding to each target point data; According to the closest point pair data corresponding to the M target point data, a closest point pair set is formed; Determine a target rotation matrix and a target offset vector according to the set of closest point pairs; The target rotation matrix and the target offset vector are used to transform the target intermediate face point cloud data to obtain initial transformed point cloud data corresponding to the target intermediate face point cloud data; Determining a conversion error between the posture reference point cloud data and the initial conversion point cloud data; If the conversion error is less than a preset first threshold, the initial conversion point cloud data is determined as the face conversion point cloud data corresponding to the target intermediate face point cloud data.

10. The 3D face reconstruction method according to claim 9, wherein the step of using the posture reference point cloud data to register the non-posture reference point cloud data in the intermediate face point cloud data set further comprises: Determining whether the conversion error is less than a preset first threshold; If the conversion error is not less than a preset first threshold, determining the number of registration iterations corresponding to the target intermediate face point cloud data; If the number of registration iterations is less than the preset iteration threshold, jump to the step of selecting M target point data from the intermediate facial point cloud data for re-registration until the number of registration iterations corresponding to the target intermediate facial point cloud data reaches the preset threshold.

11. The three-dimensional face reconstruction method according to claim 1, characterized in that: The determining the centroid of each face point cloud data in the face point cloud data set includes: Obtaining the spatial coordinates of all points in each face point cloud data set in the face point cloud data set; The average value of the spatial coordinates of all points in each face point cloud data is determined as the centroid of each face point cloud data.

12. A three-dimensional face reconstruction device, characterized in that: include: A binocular image data acquisition module is used to acquire binocular image data of a target object in at least two viewing directions through a binocular camera device; A binocular image data conversion module, used for converting the binocular image data under any of the viewing angle directions according to the disparity map information of the binocular image data to obtain image pair data corresponding to the binocular image data; A face point cloud data set module is used to perform face point cloud processing on the data according to the image to obtain a face point cloud data set corresponding to the target object; A point cloud registration processing module is used to perform registration processing based on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data; A fusion processing module is used to fuse the posture reference point cloud data and the face conversion point cloud data based on the preset face reference point cloud data to obtain the target face point cloud data; A surface reconstruction module, used to perform surface reconstruction based on the target face point cloud data to obtain a three-dimensional face model corresponding to the target object; The step of performing registration processing on the face point cloud data in the face point cloud data set to obtain posture reference point cloud data and face conversion point cloud data includes: Determine the centroid of each face point cloud data in the face point cloud data set; Performing unified processing according to the centroids of the various face point cloud data to obtain an intermediate face point cloud data set corresponding to the target object; Selecting posture reference point cloud data from the intermediate face point cloud data set, wherein the posture reference point cloud data refers to face point cloud data of a target object in a frontal face posture selected from the face point cloud data set; Using the posture reference point cloud data, registering the non-posture reference point cloud data in the intermediate face point cloud data set to obtain face conversion point cloud data, wherein the face conversion point cloud data refers to point cloud data in which the posture reference point cloud data and the non-posture reference point cloud data have the same spatial posture after registration; The step of uniformly processing the centroids of the face point cloud data to obtain an intermediate face point cloud data set corresponding to the target object includes: For each facial point cloud data, subtract the coordinates of the mass point from the spatial coordinates of each point in the facial point cloud data to obtain intermediate facial point cloud data corresponding to the facial point cloud data; Based on the intermediate face point cloud data corresponding to each face point cloud data, an intermediate face point cloud data set corresponding to the target object is formed, wherein the intermediate face point cloud data set refers to a point cloud set formed by the intermediate face point cloud data corresponding to each face point cloud data under N viewing directions, where N is greater than or equal to 2; The step of fusing the posture reference point cloud data and the face conversion point cloud data to obtain the target face point cloud data includes: Merging the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions to obtain complete face point cloud data; Selecting a plurality of point data from the complete face point cloud data to form verification point cloud data, and determining a shape difference between the verification point cloud data and preset face reference point cloud data; According to the shape difference, the posture reference point cloud data and the face conversion point cloud data of the target object in the N-1 viewing directions are fused to obtain the target face point cloud data; If the shape difference is less than a preset second threshold, the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions are initially fused to obtain the target face point cloud data; If the user confirms that the shape is normal, the shape confirmation instruction is triggered. When the user's shape confirmation instruction is received, the posture reference point cloud data and the face conversion point cloud data of the target object in N-1 viewing directions are fused to obtain the target face point cloud data; If the user confirms that the shape is abnormal, a shape error instruction is triggered. After receiving the shape error instruction from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of point data from the complete face point cloud data to form the verification point cloud data; The image pair data includes color image information and depth image information, and the face point cloud processing is performed based on the image pair data to obtain a face point cloud data set corresponding to the target object, including: Determining, according to the color image information and the depth image information in the image pair data, a target color image aligned with the depth image information; Point cloud processing is performed based on the facial area information of the human face in the target color image to obtain the human face point cloud data set.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement the steps of the three-dimensional face reconstruction method described in any one of claims 1-11 when executing the program stored in the memory.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional face reconstruction method according to any one of claims 1 to 11 are implemented.

Images