Three-dimensional face model reconstruction method, device, electronic device and storage medium

Abstract

The embodiment of the present invention relates to a method, device, electronic device and storage medium for reconstructing a three-dimensional face model, including: obtaining a face image pair of a target object under N viewing directions; for a face image pair under any viewing direction, determining a target color image according to a color image and a depth image in the face image pair; inputting the target color image into a preset image algorithm plug-in to obtain a face area in the target color image; converting the target color image into a face mask matrix according to the face area; using the face mask matrix, processing the depth image to obtain a target depth image, and converting the target depth image into a face point cloud of the target object under the viewing direction; determining a target face point cloud according to the face point cloud of the target object under N viewing directions, and performing surface reconstruction on the target face point cloud to obtain a three-dimensional face model. Thus, it is possible to obtain a face point cloud more conveniently and efficiently.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for reconstructing a three-dimensional face model. Background Art

[0002] Currently, the reconstruction of three-dimensional face models based on real human faces has broad application scenarios in many fields such as face recognition systems, medicine, movie advertising, computer animation, games, video conferencing and videophones, human-computer interaction, etc., especially in face recognition.

[0003] At present, there are two main ways to reconstruct a three-dimensional face model: Method 1, build a monocular camera array and use the parallax principle to achieve three-dimensional face model reconstruction; Method 2, use laser scanning equipment and the laser ranging principle to achieve three-dimensional face model reconstruction.

[0004] Since the original point cloud can be obtained through a monocular camera array or a laser scanning device during the reconstruction of a 3D face model, manually editing the original point cloud to remove redundant points to obtain the face point cloud is time-consuming and laborious. Summary of the invention

[0005] In view of this, in order to solve the above technical problems or part of the technical problems, the embodiments of the present invention provide a three-dimensional face model reconstruction method, device, electronic device and storage medium.

[0006] In a first aspect, an embodiment of the present invention provides a method for reconstructing a three-dimensional face model, the method comprising:

[0007] Acquire a face image pair of a target object in N viewing directions, wherein the face image pair includes a color image and a depth image;

[0008] For a face image pair under any of the viewing angle directions, determining a target color image aligned with the depth image according to the color image and the depth image in the face image pair;

[0009] Inputting the target color image into a preset image algorithm plug-in to obtain a human face area in the target color image;

[0010] According to the human face facial region, converting the target color image into a human face facial mask matrix;

[0011] Using the face mask matrix, the depth image is processed to obtain a target depth image, and the target depth image is converted into a face point cloud of the target object in the viewing direction;

[0012] According to the face point clouds of the target object under the N viewing directions, a target face point cloud is determined, and a surface reconstruction is performed on the target face point cloud to obtain a three-dimensional face model.

[0013] In a possible implementation, acquiring a face image pair of the target object in N viewing directions includes:

[0014] Controlling the image acquisition device to rotate within a preset angle range, wherein the preset angle range takes the face of the target object as a central axis, and the image acquisition device and the face of the target object are located in the same horizontal plane; and

[0015] When it is detected that the image acquisition device is rotated to any preset viewing angle direction, the image acquisition device is controlled to acquire a face image pair of the target object in the current viewing angle direction;

[0016] or,

[0017] Image acquisition devices located in N viewing angles are controlled to synchronously acquire a pair of facial images of the target object.

[0018] In a possible implementation, determining, based on the color image and the depth image in the face image pair, a target color image aligned with the depth image includes:

[0019] For any depth pixel point of the depth image in the face image pair, convert the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system;

[0020] Converting the three-dimensional coordinates into target pixel coordinates, where the target pixel coordinates correspond to color pixels of the color image in the image pair;

[0021] Determining a target color pixel point corresponding to the target pixel coordinate from the color pixel points of the color image;

[0022] Assigning the color value corresponding to the target color pixel to the depth pixel;

[0023] A target color image aligned with the depth image is generated by using the color value corresponding to each of the depth pixels in the depth image.

[0024] In a possible implementation, converting the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system includes:

[0025] Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula;

[0026] Wherein, the first homogeneous transformation formula includes:

[0027]

[0028]

[0029] The K d The internal parameter matrix of the depth sensing module in the image acquisition device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (u d , v d ), the z c Including the depth of field value, the three-dimensional coordinates include (x w ,y w , z w ).

[0030] In a possible implementation, converting the three-dimensional coordinates into target pixel coordinates includes:

[0031] Input the three-dimensional coordinates into a second homogeneous transformation formula to obtain the target pixel coordinates output by the second homogeneous transformation formula;

[0032] Wherein, the second homogeneous transformation formula includes:

[0033]

[0034]

[0035] The K c The internal parameter matrix of the color photosensitive module in the image acquisition device is included. w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2c and the offset vector t w2c The three-dimensional coordinates include (x w ,y w , z w ), the target pixel coordinates of the depth pixel point include (u c , v c ).

[0036] In a possible implementation, converting the target color image into a face mask matrix according to the face area includes:

[0037] The pixel values ​​of the pixel points in the target color image that belong to the face area are set to non-zero, and the pixel values ​​of the pixel points in the target color image that do not belong to the face area are set to zero, so as to convert the target color image into a face mask matrix.

[0038] In a possible implementation, the using the face mask matrix to process the depth image to obtain a target depth image includes:

[0039] Reading the depth image in matrix form to obtain a depth matrix corresponding to the depth image;

[0040] The depth matrix and the face mask matrix are processed to obtain a target depth image.

[0041] In a possible implementation, the processing the depth matrix and the face mask matrix to obtain a target depth image includes:

[0042] Input the depth matrix and the face mask matrix into a preset matrix operation formula to obtain a target depth matrix output by the preset matrix operation formula;

[0043] Determine that an image corresponding to the target depth matrix is ​​a target depth image, wherein depth values ​​of pixels in a non-face area of ​​the target depth image are reset to zero;

[0044] Wherein, the preset matrix operation formula includes:

[0045] M Depth′ =M face *M Depth ;

[0046] The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

[0047] In a possible implementation, converting the target depth image into a face point cloud of the target object in the viewing direction includes:

[0048] Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value;

[0049] The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system by a first homogeneous transformation formula;

[0050] Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value;

[0051] The three-dimensional points corresponding to the pixels with non-zero depth values ​​respectively form a face point cloud of the target object in the viewing direction.

[0052] In a possible implementation, determining a target face point cloud according to the face point clouds of the target object in the N viewing directions includes:

[0053] Determine the centroids corresponding to the face point clouds of the target object in the N viewing directions, and determine the middle face point clouds of the target object in the N viewing directions according to the centroids;

[0054] Selecting a posture reference point cloud from the middle face point clouds of the target object in the N viewing directions, wherein the posture reference point cloud includes the middle face point cloud of the target object in the frontal viewing direction;

[0055] Using the posture reference point cloud, register the remaining intermediate face point clouds of the target object in N-1 viewing directions to obtain conversion point clouds of the target object in N-1 viewing directions;

[0056] The preset human face reference point cloud is used to fuse the posture reference point cloud and the conversion point clouds of the target object under the N-1 viewing directions to obtain a target human face point cloud.

[0057] In a possible implementation, determining the centroids corresponding to the face point clouds of the target object in the N viewing directions includes:

[0058] For the face point cloud of the target object under any of the viewing angles, an average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

[0059] In a possible implementation, determining the intermediate face point clouds of the target object in the N viewing directions according to the centroid includes:

[0060] For the face point cloud of the target object in any of the viewing directions, the center of mass is subtracted from the spatial coordinates of each point in the face point cloud to obtain an intermediate face point cloud of the target object in the viewing direction.

[0061] In a possible implementation, the using the posture reference point cloud to register the remaining intermediate face point clouds of the target object in N-1 viewing directions to obtain the conversion point clouds of the target object in N-1 viewing directions includes:

[0062] For the remaining middle face point cloud of the target object in any of the N-1 viewing directions, select M points from the remaining middle face point cloud;

[0063] Using a preset KD-tree algorithm, searching for the nearest neighbor point corresponding to each of the M points from the posture reference point cloud;

[0064] M points and their corresponding nearest neighbor points form M pairs of nearest point pairs, and the M pairs of nearest point pairs form a nearest point pair set;

[0065] Determine a target rotation matrix and a target offset vector according to the set of closest point pairs;

[0066] The remaining intermediate face point cloud is transformed using the target rotation matrix and the target offset vector to obtain an initial transformed point cloud of the target object in the viewing direction;

[0067] A conversion error between the posture reference point cloud and the initial conversion point cloud is determined, and a conversion point cloud of the target object in the viewing direction is determined according to the conversion error.

[0068] In a possible implementation, determining a target rotation matrix and a target offset vector according to the set of closest point pairs includes:

[0069] Inputting the M pairs of closest point pairs in the set of closest point pairs into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula;

[0070] Performing SVD decomposition on the construction matrix to obtain a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtaining a target rotation matrix output by the preset rotation matrix operation formula;

[0071] The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

[0072] In a possible implementation manner, the preset matrix construction formula includes:

[0073]

[0074] The W includes the construction matrix, the N H Including the number of closest point pairs in the closest point pair set, the q i and the p iIncluding the closest point pair in the set of closest point pairs;

[0075] The preset rotation matrix operation formula includes:

[0076] R=UV T ;

[0077] The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T including the second eigenvalue;

[0078] The preset offset vector calculation formula includes:

[0079] t=u p -Ru q ;

[0080] The t includes the target offset vector, the u p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the u q Include the centroid of the remaining intermediate face point cloud.

[0081] In a possible implementation, the using the target rotation matrix and the target offset vector to transform the remaining intermediate face point cloud to obtain the initial transformed point cloud of the target object in the viewing direction includes:

[0082] Inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into a point cloud conversion formula;

[0083] Obtaining an initial conversion point cloud of the target object in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes:

[0084] Q′={Rq+t|q∈Q};

[0085] The Q′ includes the initial conversion point cloud, the Q includes the remaining intermediate face point cloud, the t includes the target offset vector, and the R includes the target rotation matrix.

[0086] In a possible implementation, the determining a conversion error between the posture reference point cloud and the initial conversion point cloud includes:

[0087] Inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula;

[0088] Obtaining a posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as a conversion error;

[0089] Wherein, the preset posture difference calculation formula includes:

[0090]

[0091] The Q′ includes the initial conversion point cloud, and the P includes the posture reference point cloud.

[0092] In a possible implementation, determining, according to the conversion error, a conversion point cloud of the target object in the viewing direction includes:

[0093] If the conversion error is less than a preset first threshold, determining that the initial conversion point cloud is a conversion point cloud of the target face in the viewing direction;

[0094] If the conversion error is not less than the preset first threshold, jump to the step of selecting M points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target object in the viewing direction.

[0095] In a possible implementation, the method of using a preset human face reference point cloud to fuse the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain a target human face point cloud includes:

[0096] Merging the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain a complete face point cloud;

[0097] Selecting a plurality of points from the complete face point cloud to form a verification point cloud, and determining a shape difference between the verification point cloud and a preset face reference point cloud;

[0098] According to the shape difference, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud.

[0099] In a possible implementation manner, determining the shape difference between the verification point cloud and a preset face reference point cloud includes:

[0100] A chamfer distance between the verification point cloud and a preset human face reference point cloud is obtained, and the chamfer distance is determined as a shape difference between the verification point cloud and the preset human face reference point cloud.

[0101] In a possible implementation, the step of fusing the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions according to the shape difference to obtain the target face point cloud includes:

[0102] If the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud;

[0103] If the shape difference is not less than the preset second threshold, jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0104] In a possible implementation, fusing the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions to obtain the target face point cloud includes:

[0105] If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud;

[0106] If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

[0107] In a second aspect, an embodiment of the present invention provides a 3D face model reconstruction device, the device comprising:

[0108] An image pair acquisition module, used to acquire a face image pair of a target object in N viewing directions, wherein the face image pair includes a color image and a depth image;

[0109] An image determination module, configured to determine, for a face image pair under any of the viewing angle directions, a target color image aligned with the depth image based on the color image and the depth image in the face image pair;

[0110] A region determination module, used for inputting the target color image into a preset image algorithm plug-in to obtain a human face region in the target color image;

[0111] A matrix acquisition module, used for converting the target color image into a human face mask matrix according to the human face facial region;

[0112] An image processing module, used to process the depth image using the face mask matrix to obtain a target depth image;

[0113] A point cloud acquisition module, used for converting the target depth image into a face point cloud of the target object in the viewing direction;

[0114] A point cloud determination module, used to determine a target face point cloud according to the face point clouds of the target object in the N viewing directions;

[0115] The point cloud reconstruction module is used to perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

[0116] In a possible implementation manner, the image pair acquisition module is specifically used to:

[0117] Controlling the image acquisition device to rotate within a preset angle range, wherein the preset angle range takes the face of the target object as a central axis, and the image acquisition device and the face of the target object are located in the same horizontal plane; and

[0118] When it is detected that the image acquisition device is rotated to any preset viewing angle direction, the image acquisition device is controlled to acquire a face image pair of the target object in the current viewing angle direction;

[0119] or,

[0120] Image acquisition devices located in N viewing angles are controlled to synchronously acquire a pair of facial images of the target object.

[0121] In a possible implementation, the image determination module includes:

[0122] A coordinate and depth value conversion submodule, for converting the pixel coordinates and the depth value of any depth pixel of the depth image in the face image pair into three-dimensional coordinates in a world coordinate system;

[0123] A coordinate conversion submodule, used for converting the three-dimensional coordinates into target pixel coordinates, wherein the target pixel coordinates correspond to color pixels of the color image in the image pair;

[0124] A pixel point determination submodule, used to determine a target color pixel point corresponding to the target pixel coordinate from the color pixel points of the color image;

[0125] A color value assigning submodule, used to assign the color value corresponding to the target color pixel to the depth pixel;

[0126] The image generation submodule is used to generate a target color image aligned with the depth image by using the color value corresponding to each of the depth pixels in the depth image.

[0127] In a possible implementation manner, the coordinate and depth of field value conversion submodule is specifically used for:

[0128] Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula;

[0129] Wherein, the first homogeneous transformation formula includes:

[0130]

[0131]

[0132] The K d The internal parameter matrix of the depth sensing module in the image acquisition device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (u d , v d ), the z c Including the depth of field value, the three-dimensional coordinates include (x w ,y w , z w ).

[0133] In a possible implementation, the coordinate conversion submodule is specifically used for:

[0134] Input the three-dimensional coordinates into a second homogeneous transformation formula to obtain the target pixel coordinates output by the second homogeneous transformation formula;

[0135] Wherein, the second homogeneous transformation formula includes:

[0136]

[0137]

[0138] The K c The internal parameter matrix of the color photosensitive module in the image acquisition device is included. w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2c and the offset vector t w2c The three-dimensional coordinates include (x w ,y w , z w ), the target pixel coordinates of the depth pixel point include (u c , v c ).

[0139] In a possible implementation manner, the matrix acquisition module is specifically used to:

[0140] The pixel values ​​of the pixel points in the target color image that belong to the face area are set to non-zero, and the pixel values ​​of the pixel points in the target color image that do not belong to the face area are set to zero, so as to convert the target color image into a face mask matrix.

[0141] In a possible implementation, the image processing module specifically includes:

[0142] An image reading submodule, configured to read the depth image in a matrix form to obtain a depth matrix corresponding to the depth image;

[0143] The matrix processing submodule is used to process the depth matrix and the face mask matrix to obtain a target depth image.

[0144] In a possible implementation manner, the matrix processing submodule is specifically used for:

[0145] Input the depth matrix and the face mask matrix into a preset matrix operation formula to obtain a target depth matrix output by the preset matrix operation formula;

[0146] Determine that an image corresponding to the target depth matrix is ​​a target depth image, wherein depth values ​​of pixels in a non-face area of ​​the target depth image are reset to zero;

[0147] Wherein, the preset matrix operation formula includes:

[0148] M Depth′ =M face *M Depth ;

[0149] The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

[0150] In a possible implementation, the point cloud acquisition module is specifically used to:

[0151] Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value;

[0152] The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system by a first homogeneous transformation formula;

[0153] Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value;

[0154] The three-dimensional points corresponding to the pixels with non-zero depth values ​​respectively form a face point cloud of the target object in the viewing direction.

[0155] In a possible implementation, the point cloud determination module specifically includes:

[0156] A centroid determination submodule, used to determine the centroids corresponding to the face point clouds of the target object in the N viewing directions;

[0157] A point cloud determination submodule, used to determine the middle face point cloud of the target object in the N viewing directions according to the centroid;

[0158] A point cloud selection submodule, used to select a posture reference point cloud from the middle face point clouds of the target object under the N viewing angles, wherein the posture reference point cloud includes the middle face point cloud of the target object under the frontal viewing angle;

[0159] A point cloud registration submodule, used to register the remaining intermediate face point clouds of the target object in N-1 viewing directions using the posture reference point cloud, to obtain the conversion point clouds of the target object in N-1 viewing directions;

[0160] The point cloud fusion submodule is used to use the preset human face reference point cloud to fuse the posture reference point cloud and the conversion point cloud of the target object under N-1 viewing directions to obtain the target human face point cloud.

[0161] In a possible implementation manner, the centroid determination submodule is specifically used for:

[0162] For the face point cloud of the target object under any of the viewing angles, an average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

[0163] In a possible implementation manner, the point cloud determination submodule is specifically used for:

[0164] For the face point cloud of the target object in any of the viewing directions, the center of mass is subtracted from the spatial coordinates of each point in the face point cloud to obtain an intermediate face point cloud of the target object in the viewing direction.

[0165] In a possible implementation, the point cloud registration submodule specifically includes:

[0166] A point selection submodule, for selecting M points from the remaining middle face point cloud of the target object in any one of the N-1 viewing directions;

[0167] A point search submodule, used to use a preset KD-tree algorithm to search for the nearest neighbor point corresponding to each of the M points from the posture reference point cloud;

[0168] A set forming submodule, used for forming M pairs of nearest point pairs from the M points and the nearest neighbor points corresponding to the M points, and forming a nearest point pair set from the M pairs of nearest point pairs;

[0169] A matrix and vector determination submodule, used to determine a target rotation matrix and a target offset vector according to the set of closest point pairs;

[0170] A point cloud conversion submodule, used to transform the remaining intermediate face point cloud using the target rotation matrix and the target offset vector to obtain an initial conversion point cloud of the target object in the viewing direction;

[0171] A conversion error determination submodule, used to determine a conversion error between the posture reference point cloud and the initial conversion point cloud;

[0172] The point cloud determination submodule is used to determine the conversion point cloud of the target object in the viewing direction according to the conversion error.

[0173] In a possible implementation manner, the matrix and vector determination submodule is specifically used for:

[0174] Inputting the M pairs of closest point pairs in the set of closest point pairs into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula;

[0175] Performing SVD decomposition on the construction matrix to obtain a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtaining a target rotation matrix output by the preset rotation matrix operation formula;

[0176] The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

[0177] In a possible implementation manner, the preset matrix construction formula includes:

[0178]

[0179] The W includes the construction matrix, the N H Including the number of closest point pairs in the closest point pair set, the q i and the p i Including the closest point pair in the set of closest point pairs;

[0180] The preset rotation matrix operation formula includes:

[0181] R=UV T ;

[0182] The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T including the second eigenvalue;

[0183] The preset offset vector calculation formula includes:

[0184] t=μ p -Rμ q ;

[0185] The t includes the target offset vector, the μ p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the μ q Include the centroid of the remaining intermediate face point cloud.

[0186] In a possible implementation, the point cloud conversion submodule is specifically used for:

[0187] Inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into a point cloud conversion formula;

[0188] Obtaining an initial conversion point cloud of the target object in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes:

[0189] Q′={Rq+t|q∈Q};

[0190] The Q′ includes the initial conversion point cloud, the Q includes the remaining intermediate face point cloud, the t includes the target offset vector, and the R includes the target rotation matrix.

[0191] In a possible implementation manner, the conversion error determination submodule is specifically used to:

[0192] Inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula;

[0193] Obtaining a posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as a conversion error;

[0194] Wherein, the preset posture difference calculation formula includes:

[0195]

[0196] The Q′ includes the initial conversion point cloud, and the P includes the posture reference point cloud.

[0197] In a possible implementation manner, the point cloud determination submodule is specifically used for:

[0198] If the conversion error is less than a preset first threshold, determining that the initial conversion point cloud is a conversion point cloud of the target face in the viewing direction;

[0199] If the conversion error is not less than the preset first threshold, jump to the step of selecting M points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target object in the viewing direction.

[0200] In a possible implementation, the point cloud fusion submodule specifically includes:

[0201] A point cloud merging submodule, used to merge the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain a complete face point cloud;

[0202] A point cloud composition submodule, used for selecting a plurality of points from the complete face point cloud to form a verification point cloud;

[0203] A shape difference determination submodule, used to determine the shape difference between the verification point cloud and a preset face reference point cloud;

[0204] The point cloud fusion submodule is used to fuse the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions according to the shape difference to obtain the target face point cloud.

[0205] In a possible implementation manner, the shape difference determination submodule is specifically used to:

[0206] A chamfer distance between the verification point cloud and a preset human face reference point cloud is obtained, and the chamfer distance is determined as a shape difference between the verification point cloud and the preset human face reference point cloud.

[0207] In a possible implementation, the point cloud fusion submodule is specifically used for:

[0208] If the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud;

[0209] If the shape difference is not less than the preset second threshold, jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0210] In a possible implementation manner, the point cloud fusion submodule is specifically used to include:

[0211] If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud;

[0212] If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

[0213] In a third aspect, the present invention provides an electronic device, characterized in that it includes: a processor and a memory, wherein the processor is used to execute a three-dimensional face model reconstruction program stored in the memory to implement the three-dimensional face model reconstruction method described in any one of the first aspects.

[0214] In a fourth aspect, the present invention provides a storage medium, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the three-dimensional face model reconstruction method described in any one of the first aspects.

[0215] The technical solution provided by the embodiment of the present invention obtains a face image pair of the target object under N viewing directions, the face image pair includes a color image and a depth image, and for the face image pair under any viewing direction, determines a target color image aligned with the depth image according to the color image and the depth image in the face image pair, inputs the target color image into a preset image algorithm plug-in, obtains a face area in the target color image, converts the target color image into a face mask matrix according to the face area, processes the depth image using the face mask matrix to obtain a target depth image, and converts the target depth image into a target depth image. The face point cloud of the target object under the viewing direction is converted into the face point cloud of the target object under the viewing direction, the target face point cloud is determined according to the face point clouds of the target object under N viewing directions, and the target face point cloud is surface reconstructed to obtain a three-dimensional face model. Compared with first obtaining the original point cloud, editing the original point cloud in a manual way to remove redundant points to obtain the face point cloud, by determining the face area in the target color image, thereby generating a face mask matrix, using the face mask matrix to process the depth image to obtain the target depth image, and converting the target depth image into a face point cloud, the face point cloud can be obtained more conveniently and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0216] Figure 1 A schematic diagram of an implementation flow of a method for reconstructing a three-dimensional face model provided by an embodiment of the present invention;

[0217] Figure 2 A schematic diagram of an application scenario of obtaining an image pair of a target object in N viewing directions according to an embodiment of the present invention;

[0218] Figure 3 A schematic diagram of another application scenario of acquiring an image pair of a target object in N viewing directions according to an embodiment of the present invention;

[0219] Figure 4 is a schematic diagram of the facial region of a human face;

[0220] Figure 5 A schematic diagram of the contour line of the facial area of ​​a human face;

[0221] Figure 6 This is a diagram of the visual effect of the facial matrix of a human face;

[0222] Figure 7 It is a schematic diagram of the surface reconstruction algorithm;

[0223] Figure 8 A schematic diagram of an implementation process of determining a target face point cloud provided by an embodiment of the present invention;

[0224] Fig. 9 A schematic diagram of an implementation process of registering the remaining intermediate face point clouds of a target object in N-1 viewing directions provided by an embodiment of the present invention;

[0225] Fig.10 A schematic diagram of an implementation flow of fusing a posture reference point cloud and a conversion point cloud of a target object in N-1 viewing directions provided by an embodiment of the present invention;

[0226] Fig.11 A block diagram of an embodiment of a 3D face model reconstruction device provided by an embodiment of the present invention;

[0227] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0228] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0229] The following is a further explanation of the three-dimensional face model reconstruction method provided by the present invention with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation of the embodiments of the present invention.

[0230] See also Figure 1 , is a schematic diagram of an implementation flow of a method for reconstructing a three-dimensional face model provided by an embodiment of the present invention. Figure 1As shown, the method may include the following steps:

[0231] S101, obtaining an image pair of a target object in N viewing directions, where the face image pair includes a color image and a depth image.

[0232] In the embodiment of the present invention, considering that the human face is three-dimensional, if the facial image of the target object is collected from a single viewing angle, it is easy for the raised part of the face in the viewing angle to block other parts. Therefore, in order to avoid the above phenomenon, the facial image of the target object can be collected from multiple (represented as N) different viewing angles. Here, N is a natural number greater than 1.

[0233] Furthermore, when capturing the facial image of the target object, the image capture device (such as a consumer-grade camera) can be on the same horizontal plane as the target object's face, and the larger the above-mentioned N is, that is, the more viewing directions there are, the more times the image capture device captures images, which is more conducive to improving the accuracy of the final reconstructed three-dimensional face model.

[0234] Based on this, in an embodiment of the present invention, a face image pair of the target object under N viewing angles is obtained, wherein the face image pair includes a color image (RGB image) and a depth image (Depth image). That is to say, a color image and a depth image of the face of the target object under N viewing angles are obtained.

[0235] Specifically, as an optional implementation, obtaining a pair of face images of the target object in N viewing directions includes: controlling the image acquisition device to rotate within a preset angle range, where the preset angle range is centered on the face (i.e., frontal face) of the target object, and when detecting that the image acquisition device rotates to any preset viewing direction, controlling the image acquisition device to acquire a pair of face images of the target object in the current viewing direction. In this way, by adjusting the posture of the image acquisition device (i.e., the orientation of the image acquisition device in the shooting space), a pair of face images of the target object in N viewing directions can be obtained.

[0236] A specific example, such as Figure 2 As shown, with the face of the target object as the central axis, a preset angle range of 120° is set. Within the preset angle range, the image acquisition device is controlled to rotate at a rotation angle of 15°. This means that the image acquisition device is controlled to acquire a face image pair of the target object every time it rotates 15°. In this way, the face image pairs of the target object under 8 (120 / 15=8) viewing angle directions can be obtained.

[0237] As an optional implementation method, obtaining a face image pair of the target object in N viewing directions includes: taking the face of the target object as the central axis, setting an image acquisition device in each of the N viewing directions within a preset angle range, and controlling the image acquisition devices in the N viewing directions to synchronously acquire the face image pair of the target object.

[0238] A specific example, such as Figure 3 As shown, a preset angle range of 120° is set with the face of the target object as the central axis. An image acquisition device is respectively set in the three viewing angle directions within the preset angle range. The three image acquisition devices are controlled to synchronously acquire the face image pairs of the target object, so that the face image pairs of the target object in the three viewing angle directions can be obtained.

[0239] It should be noted that the above-mentioned image acquisition device includes two photosensitive modules, namely a color photosensitive module and a depth photosensitive module, so that a face image pair including a color image and a depth image can be acquired.

[0240] S102: for a face image pair in any viewing direction, determine a target color image aligned with the depth image according to the color image and the depth image in the face image pair.

[0241] In one embodiment, determining a target color image (denoted as RGB' image) aligned with the Depth image based on the RGB image and the Depth image in the face image pair includes: for any depth pixel point of the Depth image in the face image pair, converting the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in the world coordinate system, converting the three-dimensional coordinates into target pixel coordinates, the target pixel coordinates corresponding to the color pixel point of the RGB image in the image pair, determining the target color pixel point corresponding to the target pixel coordinate from the color pixel points of the RGB image, assigning the color value corresponding to the target color pixel point to the depth pixel point, and using the color values ​​corresponding to each depth pixel point of the Depth image to generate RGB' aligned with the Depth image.

[0242] Among them, converting the pixel coordinates and depth of field values ​​of the depth pixel points into three-dimensional coordinates in the world coordinate system includes: inputting the pixel coordinates and depth of field values ​​of the depth pixel points into the first homogeneous transformation formula as shown in the following formula (1), and obtaining the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula.

[0243]

[0244] in, is the external parameter matrix of the depth sensor module, which is the Euclidean transformation matrix from the world coordinate system to the depth sensor module, and is composed of the rotation matrix R w2d and the offset vector t w2dThe size of the external parameter matrix of the depth sensor module can be 4x4, and the rotation matrix R w2d and the offset vector t w2d The size of can be 3*3, both of which can be obtained through manual calibration. The pixel coordinates of the depth pixel point include (u d , v d ), z c Including depth of field value, three-dimensional coordinates include (x w ,y w , z w ).

[0245] Converting the three-dimensional coordinates in the world coordinate system into target pixel coordinates includes: inputting the three-dimensional coordinates in the world coordinate system into the second homogeneous transformation formula shown in the following formula (ii) to obtain the target pixel coordinates output by the second homogeneous transformation formula.

[0246]

[0247] Among them, K c It includes the internal parameter matrix of the color photosensitive module in the image acquisition device, and the matrix size can be 3x3, which can be read through the image acquisition device interface or obtained by manual calibration.

[0248] in, It is the external parameter matrix of the color photosensitive module in the image acquisition device. Its matrix size is 4x4. It is the Euclidean transformation matrix from the world coordinate system to the color photosensitive module. It is composed of a 3x3 rotation matrix R w2c and the offset vector t w2c Both can be obtained through manual calibration.

[0249] The target pixel coordinates of the depth pixel point include (u c , v c ).

[0250] In the embodiment of the present invention, the target pixel coordinates of the depth pixel point are the pixel coordinates of the color pixel point in the color image, that is, the color pixel point corresponding to the color image in the face image pair. In this way, the target color pixel point corresponding to the target pixel coordinates of the depth pixel point can be determined from the color pixel points in the color image, the color value corresponding to the target color pixel point is obtained, and the color value is assigned to the depth pixel point. In this way, the color value corresponding to each depth pixel point in the depth image is found in the color image, so that each depth pixel point in the depth image has a corresponding color value and depth of field value. Subsequently, the color value corresponding to each depth pixel point in the depth image is used to generate a target color image aligned with the depth image, which is called an RGB' image, and its image size is consistent with the Depth image.

[0251] In addition, in the embodiment of the present invention, the RGB' image and the Depth image aligned at the pixel level are referred to as RGB'-D data.

[0252] S103, inputting the target color image into a preset image algorithm plug-in to obtain a human face region in the target color image.

[0253] First, the range of the facial area of ​​a human face in the embodiment of the present invention is defined as "the area surrounded by the left ear root (excluding the left ear), the hairline, the right ear root (excluding the right ear), and the chin extension", and its visual representation is as follows: Figure 4 shown.

[0254] In the embodiment of the present invention, the target color image is input into a preset image algorithm plug-in to obtain the face area of ​​the target color image, wherein the output result of the image algorithm plug-in is saved in the form of a point set consisting of pixel coordinates of pixel points on the contour line of the face area, and the visual representation of the contour line of the face area is as follows: Figure 5 shown.

[0255] The above-mentioned image algorithm plug-in is an automatic processing script that is secondary developed based on an image algorithm library (such as OpenCV, etc.). It can use the loaded three-dimensional model as a reference to search for similar shape contours in the input image to obtain the facial area of ​​the human face.

[0256] S104, converting the target color image into a face mask matrix according to the face area.

[0257] For the face area in the target color image outlined by the preset image algorithm plug-in, the pixel values ​​of the pixels belonging to the face area in the target color image are set to non-zero, and the pixel values ​​of the pixels not belonging to the face area in the target color image are set to zero, so as to convert the target color image into a face mask matrix (denoted as M face ). This means that the face matrix M face It is composed of 0 and 1, where 0 represents a non-face area and 1 represents a face area. For details, see Figure 6 shown.

[0258] S105, using the face mask matrix, processing the depth image to obtain a target depth image, and converting the target depth image into a face point cloud of the target object in the viewing direction.

[0259] First of all, it can be seen from the description of S102 to S105 above that in the embodiment of the present invention, by processing the face image pair of the target object in any viewing direction, a group of face point clouds of the target object can be obtained, that is, in the embodiment of the present invention, face point clouds of the target face in N viewing directions can be obtained.

[0260] The depth image is processed using the face mask matrix to obtain a target depth image, including: reading the depth image in matrix form, obtaining a depth matrix corresponding to the depth image, processing the depth matrix and the face mask matrix to obtain a target depth image (denoted as Depth' image).

[0261] Specifically, after the depth image is read as a depth matrix in matrix form, the depth values ​​of the depth pixels in the non-face area of ​​the depth image can be reset to zero by calculating the Hammarby product of the matrix to obtain a target depth matrix.

[0262] The depth matrix M read in matrix form Depth and M face Input the preset matrix operation formula as shown in the following formula (III) to obtain the target depth matrix M output by the preset matrix operation formula Depth‘ , M Depth‘ The corresponding image is the target depth image. In addition, in the embodiment of the present invention, the Depth' image and the RGB' image are combined to form RGB'-D' data.

[0263] M Depth′ =M face *M Depth Formula (III)

[0264] It should be noted that after the above steps, in the subsequent process of converting the target depth image into the face point cloud of the target object in the viewing direction, the z w Whether the coordinates are 0 is used to remove the background, hair, ears, neck, torso and other areas in the original point cloud data that are irrelevant to the real-person expression modeling task that the present invention is concerned with, leaving a clean face area that is strongly related to the real-person expression, which is helpful to reduce the difficulty of subsequent point cloud alignment.

[0265] Converting a target depth image into a face point cloud of a target object in a viewing direction includes: determining each pixel point with a non-zero depth value in the target depth image, obtaining the pixel coordinates and the depth value of the pixel point with a non-zero depth value for any pixel point with a non-zero depth value, converting the pixel coordinates and the depth value of the pixel point with a non-zero depth value into three-dimensional coordinates in a world coordinate system corresponding to the pixel point with a non-zero depth value through a first homogeneous transformation formula; determining that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value; and forming a face point cloud of the target face in the viewing direction from the three-dimensional points corresponding to each pixel point with a non-zero depth value.

[0266] S106, determining a target face point cloud according to the face point cloud of the target object in N viewing directions, and performing surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

[0267] After obtaining the desired point cloud of the complete face, i.e. the target face point cloud, surface reconstruction is performed to restore the geometric shape, thus turning it into a 3D face model. Taking Delaunay triangulation reconstruction as an example, a feasible surface reconstruction algorithm is as follows: Figure 7 Of course, the triangular reconstruction described is only the most common reconstruction method. In addition to triangles, quadrilaterals and pentagons can also be used to represent the surface. In the face reconstruction scenario targeted by the present invention, triangular reconstruction is the most preferred.

[0268] It can be seen from the description of S106 that in the embodiment of the present invention, the target face point cloud can be determined according to the face point clouds of the target object in N viewing directions, and the target face point cloud can be reconstructed to obtain a three-dimensional face model.

[0269] Specifically, Figure 8 As shown in FIG. 1 , a schematic diagram of an implementation process of determining a target face point cloud provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the process may include the following steps:

[0270] S801, determining the centroids corresponding to the facial point clouds of the target object in N viewing directions, and determining the middle facial point clouds of the target object in the N viewing directions according to the centroids.

[0271] In an embodiment of the present invention, the centroids corresponding to the facial point clouds of the target object under N viewing directions are determined, including: for the facial point cloud of the target object under any viewing direction, obtaining the average value of the spatial coordinates of all points in the facial point cloud, and determining the average value as the centroid of the facial point cloud.

[0272] For example, the face point clouds of the target object under N viewing directions form a point cloud set C. For point cloud C in point cloud set C i , its center of mass u i Point cloud C i The average value of the spatial coordinates of all points in . The specific calculation formula is as follows (IV):

[0273]

[0274] Among them, i It is point cloud C i The number of points included, P j It is point cloud C i The spatial coordinates of the jth point in .

[0275] Furthermore, the intermediate face point cloud of the target object under N viewing directions is determined according to the centroid, including: for the face point cloud of the target object under any viewing direction, the spatial coordinates of each point in the face point cloud are subtracted from the centroid to obtain the intermediate face point cloud of the target object under the viewing direction.

[0276] For example, point cloud C i Every point P in j Subtract point cloud C i The corresponding center of mass u i , the new point cloud C′ i , new point cloud C′ i That is, the middle face point cloud of the target object under the viewing direction, from which a new point cloud set C can be formed u .

[0277] This operation unifies all point clouds into the centroid coordinate system with the centroid as the origin of the coordinate system, making it easier to calculate the target rotation matrix R and the target offset vector t using the SVD algorithm.

[0278] S802, selecting a posture reference point cloud from the middle face point cloud of the target object under N viewing directions, the posture reference point cloud including the middle face point cloud of the target object under the frontal viewing direction and the middle face point cloud of the target face under N viewing directions.

[0279] For example, for the middle face point cloud of the target object under N viewing directions, the point cloud set C u , from the point cloud set C u The middle face point cloud of the target object in the frontal view direction is selected as the posture reference point cloud (denoted as P).

[0280] S803, using the posture reference point cloud, register the remaining intermediate face point clouds of the target object in N-1 viewing directions to obtain the conversion point clouds of the target object in N-1 viewing directions.

[0281] The remaining intermediate face point clouds refer to the face point clouds of the target object in N viewing directions, except for the posture reference point cloud, that is, the intermediate face point cloud of the target object in the frontal viewing direction. That is, the remaining intermediate face point clouds of the target object in N-1 viewing directions.

[0282] In the embodiment of the present invention, the remaining middle face point clouds of the target face in N-1 viewing directions are registered using the posture reference point cloud to obtain the conversion point cloud of the target object in N-1 viewing directions. The purpose here is to make the spatial posture of the remaining middle face point clouds of the target object in N-1 viewing directions basically consistent.

[0283] Specifically, Fig. 9As shown, it is a schematic diagram of an implementation process of registering the remaining intermediate face point clouds of the target object in N-1 viewing directions provided by an embodiment of the present invention. The process may include the following steps:

[0284] S901, for the remaining middle face point cloud of the target object in any one of the N-1 viewing directions, select M points from the remaining middle face point cloud.

[0285] In one embodiment, for the remaining middle face point cloud of the target object in any viewing direction, M points are randomly selected from the remaining middle face point cloud.

[0286] S902, using a preset KD-tree algorithm, searching for the nearest neighbor points corresponding to each of the M points in the posture reference point cloud.

[0287] In an embodiment of the present invention, for a posture reference point cloud, a KD-tree data structure is established on it to store the spatial coordinates of all its points, so that the preset KD-tree algorithm, that is, with the help of the characteristics of the KD-tree, is used to find the nearest neighbor points corresponding to each of the M points in the posture reference point cloud, that is, to find the nearest neighbor point corresponding to each point (among the M points) from the posture reference point cloud.

[0288] S903, forming M pairs of nearest point pairs from the M points and their corresponding nearest neighbor points, and forming a nearest point pair set from the M pairs of nearest point pairs.

[0289] In the embodiment of the present invention, for the above-mentioned M points and the nearest neighboring points corresponding to each of the M points, M pairs of nearest point pairs can be formed, and the M pairs of nearest point pairs form a nearest point pair set H, as shown below:

[0290] H={(p i ,q i )|p i ∈P,q i ∈Q,1≤i≤m}.

[0291] S904: Determine a target rotation matrix and a target offset vector according to the set of closest point pairs.

[0292] In an embodiment of the present invention, a target rotation matrix and a target offset vector are determined based on a set of nearest point pairs, including: inputting M pairs of nearest point pairs in the set of nearest point pairs into a preset matrix construction formula as exemplified in the following formula (five), obtaining a construction matrix output by the preset matrix construction formula, performing SVD decomposition on the construction matrix according to the following formula (six), obtaining a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula as exemplified in the following formula (seven), obtaining a target rotation matrix output by the preset rotation matrix operation formula. Finally, inputting the target rotation matrix, the centroid of the posture reference point cloud, and the centroid of the remaining intermediate face point cloud into a preset offset vector operation formula as exemplified in the following formula (eight), obtaining a target offset vector output by the preset offset vector operation formula.

[0293]

[0294] In the above formula (V), W includes the construction matrix, N H Includes the number of closest point pairs in the closest point pair set, q i and p i Include the closest point pair in the set of closest point pairs.

[0295]

[0296] In the above formula (VI), U is the first eigenvalue, V T is the second eigenvalue, σ1, σ2, σ3 are eigenvalues.

[0297] R=UV T Formula (VII)

[0298] In the above formula (VII), R includes the target rotation matrix, U includes the first eigenvalue, V T Includes the second eigenvalue.

[0299] t=u p -Ru q Formula (8)

[0300] In the above formula (eight), t includes the target offset vector, u p Includes the center of mass of the pose reference point cloud, R includes the target rotation matrix, u q Include the centroid of the remaining intermediate face point cloud.

[0301] S905, using the target rotation matrix and the target offset vector, transform the remaining intermediate face point cloud to obtain the initial transformed point cloud of the target face in the viewing direction.

[0302] In an embodiment of the present invention, for the remaining intermediate facial point cloud of the target object in any of the N-1 viewing directions, the remaining intermediate facial point cloud can be transformed using the target rotation matrix and target offset vector corresponding to the remaining intermediate facial point cloud to obtain the initial transformed point cloud of the target face in that viewing direction.

[0303] Specifically, the target rotation matrix and the target offset vector are used to transform the remaining intermediate face point cloud to obtain the initial transformed point cloud of the target object in the viewing direction, including: inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into the point cloud transformation formula shown in the following formula (9), and obtaining the initial transformed point cloud of the target object in the viewing direction output by the point cloud transformation formula.

[0304] Q′={Rq+t|q∈Q} Formula (9)

[0305] In the above formula (9), Q′ includes the initial conversion point cloud, Q includes the remaining intermediate face point cloud, t includes the target offset vector, and R includes the target rotation matrix.

[0306] In this way, the spatial posture of the initial conversion point cloud Q′ can be brought closer to the posture reference point cloud P, that is, closer to the frontal face viewing direction.

[0307] S906, determining a conversion error between the posture reference point cloud and the initial conversion point cloud, and determining a conversion point cloud of the target object in the viewing direction according to the conversion error.

[0308] In an embodiment of the present invention, for the initial conversion point cloud of the target object in any of the N-1 viewing directions, the (point cloud) conversion error between the posture reference point cloud and the initial conversion point cloud is determined, and based on the conversion error, the conversion point cloud of the target object in the viewing direction can be determined.

[0309] Specifically, in an embodiment of the present invention, determining the conversion error between the posture reference point cloud and the initial conversion point cloud includes: inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula as exemplified by the following formula (10), obtaining the posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as the conversion error.

[0310]

[0311] In the above formula (10), Q′ includes the initial conversion point cloud, and P includes the posture reference point cloud.

[0312] In addition, in an embodiment of the present invention, after determining the (point cloud) conversion error between the posture reference point cloud and the initial conversion point cloud, if the conversion error is less than a preset first threshold, the initial conversion point cloud is determined to be the conversion point cloud of the target face in the viewing direction.

[0313] If the conversion error is not less than the preset first threshold, jump to the step of selecting M points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target face in the viewing direction.

[0314] S804, using the preset human face reference point cloud, fuses the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions to obtain the target human face point cloud.

[0315] In the embodiment of the present invention, a preset face reference point cloud is introduced, and the face reference point cloud and the conversion point cloud of the target face in N-1 viewing directions are fused through the face reference point cloud to obtain the target face point cloud. Here, the face reference point cloud can be a male or female face reference point cloud. Specifically, an open source reference 3D model can be obtained, and the face part thereof can be intercepted as the face reference point cloud, which is not limited in the embodiment of the present invention.

[0316] Specifically, Fig.10 As shown, it is a schematic diagram of an implementation process of fusing a posture reference point cloud and a conversion point cloud of a target object under N-1 viewing directions provided by an embodiment of the present invention. The process may include the following steps:

[0317] S1001, merging the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions to obtain a complete face point cloud.

[0318] In the embodiment of the present invention, the posture reference point cloud and the conversion point cloud of the target object in the N-1 viewing angle direction are merged to obtain a complete face point cloud.

[0319] It should be noted that the complete face point cloud may be a point cloud including a complete face, which is not limited in the embodiment of the present invention.

[0320] S1002, selecting a plurality of points from the complete face point cloud to form a verification point cloud, and determining a shape difference between the verification point cloud and a preset face reference point cloud.

[0321] In an embodiment of the present invention, the shape difference between the verification point cloud and the preset face reference point cloud can be determined by the following method: obtaining the chamfer distance between the verification point cloud and the preset face reference point cloud, and determining the chamfer distance as the shape difference between the verification point cloud and the preset face reference point cloud. The calculation formula for obtaining the chamfer distance between the verification point cloud and the preset face reference point cloud is shown in the following formula (XI):

[0322]

[0323] In the above formula (XI), N mand N B are the sizes of the downsampled verification point cloud m and the face reference point cloud B (i.e. the number of elements therein), respectively. The left side of the plus sign is the average of the sum of the minimum distances from any point in the verification point cloud m to the reference point cloud B, and the right side of the plus sign is the average of the sum of the minimum distances from any point in the reference point cloud B to the verification point cloud m. The two are combined as the chamfer distance in 3D space.

[0324] It should be noted that the smaller the chamfer distance between two point clouds is, the smaller the shape difference between the two point clouds is.

[0325] It should also be noted that in an embodiment of the present invention, one of the following can be obtained between the verification point cloud and the preset face reference point cloud: L1, L2, deformation of L1, deformation of L2, which is used to describe the shape difference between the verification point cloud and the preset face reference point cloud. The embodiment of the present invention is not limited to this.

[0326] S1003, based on the shape difference, the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions are fused to obtain the target face point cloud.

[0327] For the shape difference between the verification point cloud and the preset face reference point cloud, the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions are fused according to the shape difference to obtain the target face point cloud.

[0328] Among them, if the shape difference is less than the preset second threshold, the posture reference point cloud and the conversion point cloud of the target object under N-1 viewing directions are fused to obtain the target face point cloud; if the shape difference is not less than the preset second threshold, jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0329] Specifically, if the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions are fused to obtain a target face point cloud, and the target face point cloud is displayed for the user to confirm whether the shape is normal.

[0330] If the user confirms that the shape is normal, the shape confirmation instruction is triggered. When the user's shape confirmation instruction is received, the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions are fused to obtain the target face point cloud.

[0331] If the user confirms that the shape is abnormal, a shape error instruction is triggered. After receiving the user's shape error instruction, the preset second threshold is adjusted, and the process jumps to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0332] If the shape difference is not less than the preset second threshold, jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0333] The technical solution provided by the embodiment of the present invention obtains a face image pair of the target object under N viewing directions, the face image pair includes a color image and a depth image, and for the face image pair under any viewing direction, determines a target color image aligned with the depth image according to the color image and the depth image in the face image pair, inputs the target color image into a preset image algorithm plug-in, obtains a face area in the target color image, converts the target color image into a face mask matrix according to the face area, processes the depth image using the face mask matrix to obtain a target depth image, and converts the target depth image into a target depth image. The face point cloud of the target object under the viewing direction is converted into the face point cloud of the target object under the viewing direction, the target face point cloud is determined according to the face point clouds of the target object under N viewing directions, and the target face point cloud is surface reconstructed to obtain a three-dimensional face model. Compared with first obtaining the original point cloud, editing the original point cloud in a manual way to remove redundant points to obtain the face point cloud, by determining the face area in the target color image, thereby generating a face mask matrix, using the face mask matrix to process the depth image to obtain the target depth image, and converting the target depth image into a face point cloud, the face point cloud can be obtained more conveniently and efficiently.

[0334] In addition, in the embodiment of the present invention, only one or more image acquisition devices (such as consumer-grade cameras) are required, thereby obtaining facial image pairs of the target object in N viewing directions to participate in the subsequent three-dimensional face model reconstruction. The hardware investment cost is low and the equipment maintenance is simple.

[0335] Given that only one or more image acquisition devices (such as consumer-grade cameras) are required in the embodiments of the present invention, the scale of point cloud data subsequently collected and processed is generally in the thousands, which saves a large amount of computing resources compared to three-dimensional reconstruction involving tens of thousands, hundreds of thousands, or millions of points.

[0336] In the process of point cloud registration, adding the facial reference point cloud as a shape reference greatly reduces the shape deviation of the result obtained during the final point cloud fusion, such as irregular protrusions or depressions, abnormal nose shape, etc., thereby achieving reliable 3D reconstruction accuracy from low-precision point cloud data.

[0337] Corresponding to the above method embodiment, the embodiment of the present invention also provides a 3D face model reconstruction device, such as Fig.11 As shown, the device may include: an image pair acquisition module 1110, an image determination module 1120, a region determination module 1130, a matrix acquisition module 1140, an image processing module 1150, a point cloud acquisition module 1160, a point cloud determination module 1170, and a point cloud reconstruction module 1180.

[0338] An image pair acquisition module 1110 is used to acquire a face image pair of a target object in N viewing directions, wherein the face image pair includes a color image and a depth image;

[0339] An image determination module 1120 is configured to determine, for a face image pair under any of the viewing angle directions, a target color image aligned with the depth image based on the color image and the depth image in the face image pair;

[0340] The region determination module 1130 is used to input the target color image into a preset image algorithm plug-in to obtain a human face region in the target color image;

[0341] A matrix acquisition module 1140, configured to convert the target color image into a human face mask matrix according to the human face region;

[0342] An image processing module 1150 is used to process the depth image using the face mask matrix to obtain a target depth image;

[0343] A point cloud acquisition module 1160, configured to convert the target depth image into a face point cloud of the target object in the viewing direction;

[0344] A point cloud determination module 1170 is used to determine a target face point cloud according to the face point clouds of the target object in the N viewing directions;

[0345] The point cloud reconstruction module 1180 is used to perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

[0346] In a possible implementation manner, the image pair acquisition module 1110 is specifically used to:

[0347] Controlling the image acquisition device to rotate within a preset angle range, wherein the preset angle range takes the face of the target object as a central axis, and the image acquisition device and the face of the target object are located in the same horizontal plane; and

[0348] When it is detected that the image acquisition device is rotated to any preset viewing angle direction, the image acquisition device is controlled to acquire a face image pair of the target object in the current viewing angle direction;

[0349] or,

[0350] Image acquisition devices located in N viewing angles are controlled to synchronously acquire a pair of facial images of the target object.

[0351] In a possible implementation, the image determination module 1120 includes (not shown in the figure):

[0352] A coordinate and depth value conversion submodule, for converting the pixel coordinates and the depth value of any depth pixel of the depth image in the face image pair into three-dimensional coordinates in a world coordinate system;

[0353] A coordinate conversion submodule, used for converting the three-dimensional coordinates into target pixel coordinates, wherein the target pixel coordinates correspond to color pixels of the color image in the image pair;

[0354] A pixel point determination submodule, used to determine a target color pixel point corresponding to the target pixel coordinate from the color pixel points of the color image;

[0355] A color value assigning submodule, used to assign the color value corresponding to the target color pixel to the depth pixel;

[0356] The image generation submodule is used to generate a target color image aligned with the depth image by using the color value corresponding to each of the depth pixels in the depth image.

[0357] In a possible implementation manner, the coordinate and depth of field value conversion submodule is specifically used for:

[0358] Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula;

[0359] Wherein, the first homogeneous transformation formula includes:

[0360]

[0361]

[0362] The K d The internal parameter matrix of the depth sensing module in the image acquisition device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (u d , v d ), the z c Including the depth of field value, the three-dimensional coordinates include (x w ,y w , z w ).

[0363] In a possible implementation, the coordinate conversion submodule is specifically used for:

[0364] Inputting the three-dimensional coordinates into a second homogeneous transformation formula to obtain target pixel coordinates output by the second homogeneous transformation formula;

[0365] Wherein, the second homogeneous transformation formula includes:

[0366]

[0367]

[0368] The K c The internal parameter matrix of the color photosensitive module in the image acquisition device is included. w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2c and the offset vector t w2c The three-dimensional coordinates include (x w ,y w , z w ), the target pixel coordinates of the depth pixel point include (u c , v c ).

[0369] In a possible implementation, the matrix acquisition module 1140 is specifically used to:

[0370] The pixel values ​​of the pixel points in the target color image that belong to the face area are set to non-zero, and the pixel values ​​of the pixel points in the target color image that do not belong to the face area are set to zero, so as to convert the target color image into a face mask matrix.

[0371] In a possible implementation, the image processing module 1150 specifically includes:

[0372] An image reading submodule, configured to read the depth image in a matrix form to obtain a depth matrix corresponding to the depth image;

[0373] The matrix processing submodule is used to process the depth matrix and the face mask matrix to obtain a target depth image.

[0374] In a possible implementation manner, the matrix processing submodule is specifically used for:

[0375] Input the depth matrix and the face mask matrix into a preset matrix operation formula to obtain a target depth matrix output by the preset matrix operation formula;

[0376] Determine that an image corresponding to the target depth matrix is ​​a target depth image, wherein depth values ​​of pixel points in a non-face area in the target depth image are reset to zero;

[0377] Wherein, the preset matrix operation formula includes:

[0378] M Depth′ =M face *M Depth ;

[0379] The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

[0380] In a possible implementation, the point cloud acquisition module 1160 is specifically used to:

[0381] Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value;

[0382] The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system by a first homogeneous transformation formula;

[0383] Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value;

[0384] The three-dimensional points corresponding to the pixels with non-zero depth values ​​respectively form a face point cloud of the target object in the viewing direction.

[0385] In a possible implementation, the point cloud determination module 1170 specifically includes:

[0386] A centroid determination submodule, used to determine the centroids corresponding to the face point clouds of the target object in the N viewing directions;

[0387] A point cloud determination submodule, used to determine the middle face point cloud of the target object in the N viewing directions according to the centroid;

[0388] A point cloud selection submodule, used to select a posture reference point cloud from the middle face point clouds of the target object under the N viewing angles, wherein the posture reference point cloud includes the middle face point cloud of the target object under the frontal viewing angle;

[0389] A point cloud registration submodule, used to register the remaining intermediate face point clouds of the target object in N-1 viewing directions using the posture reference point cloud, to obtain the conversion point clouds of the target object in N-1 viewing directions;

[0390] The point cloud fusion submodule is used to use the preset human face reference point cloud to fuse the posture reference point cloud and the conversion point cloud of the target object under N-1 viewing directions to obtain the target human face point cloud.

[0391] In a possible implementation manner, the centroid determination submodule is specifically used for:

[0392] For the face point cloud of the target object under any of the viewing angles, an average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

[0393] In a possible implementation manner, the point cloud determination submodule is specifically used for:

[0394] For the face point cloud of the target object in any of the viewing directions, the center of mass is subtracted from the spatial coordinates of each point in the face point cloud to obtain an intermediate face point cloud of the target object in the viewing direction.

[0395] In a possible implementation, the point cloud registration submodule specifically includes:

[0396] A point selection submodule, for selecting M points from the remaining middle face point cloud of the target object in any one of the N-1 viewing directions;

[0397] A point search submodule, used to use a preset KD-tree algorithm to search for the nearest neighbor point corresponding to each of the M points from the posture reference point cloud;

[0398] A set forming submodule, used for forming M pairs of nearest point pairs from the M points and the nearest neighbor points corresponding to the M points, and forming a nearest point pair set from the M pairs of nearest point pairs;

[0399] A matrix and vector determination submodule, used to determine a target rotation matrix and a target offset vector according to the set of closest point pairs;

[0400] A point cloud conversion submodule, used to transform the remaining intermediate face point cloud using the target rotation matrix and the target offset vector to obtain an initial conversion point cloud of the target object in the viewing direction;

[0401] A conversion error determination submodule, used to determine a conversion error between the posture reference point cloud and the initial conversion point cloud;

[0402] The point cloud determination submodule is used to determine the conversion point cloud of the target object in the viewing direction according to the conversion error.

[0403] In a possible implementation manner, the matrix and vector determination submodule is specifically used for:

[0404] Inputting the M pairs of closest point pairs in the set of closest point pairs into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula;

[0405] Performing SVD decomposition on the construction matrix to obtain a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtaining a target rotation matrix output by the preset rotation matrix operation formula;

[0406] The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

[0407] In a possible implementation manner, the preset matrix construction formula includes:

[0408]

[0409] The W includes the construction matrix, the N H Including the number of closest point pairs in the closest point pair set, the q i and the p i Including the closest point pair in the set of closest point pairs;

[0410] The preset rotation matrix operation formula includes:

[0411] R=UV T ;

[0412] The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T including the second eigenvalue;

[0413] The preset offset vector calculation formula includes:

[0414] t=u p -Ru q ;

[0415] The t includes the target offset vector, the u p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the u q Include the centroid of the remaining intermediate face point cloud.

[0416] In a possible implementation, the point cloud conversion submodule is specifically used for:

[0417] Inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into a point cloud conversion formula;

[0418] Obtaining an initial conversion point cloud of the target object in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes:

[0419] Q′={Rq+t|q∈Q};

[0420] The Q′ includes the initial conversion point cloud, the Q includes the remaining intermediate face point cloud, the t includes the target offset vector, and the R includes the target rotation matrix.

[0421] In a possible implementation manner, the conversion error determination submodule is specifically used to:

[0422] Inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula;

[0423] Obtaining a posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as a conversion error;

[0424] Wherein, the preset posture difference calculation formula includes:

[0425]

[0426] The Q′ includes the initial conversion point cloud, and the P includes the posture reference point cloud.

[0427] In a possible implementation manner, the point cloud determination submodule is specifically used for:

[0428] If the conversion error is less than a preset first threshold, determining that the initial conversion point cloud is a conversion point cloud of the target face in the viewing direction;

[0429] If the conversion error is not less than the preset first threshold, jump to the step of selecting M points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target object in the viewing direction.

[0430] In a possible implementation, the point cloud fusion submodule specifically includes:

[0431] A point cloud merging submodule, used to merge the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain a complete face point cloud;

[0432] A point cloud composition submodule, used for selecting a plurality of points from the complete face point cloud to form a verification point cloud;

[0433] A shape difference determination submodule, used to determine the shape difference between the verification point cloud and a preset face reference point cloud;

[0434] The point cloud fusion submodule is used to fuse the posture reference point cloud and the conversion point cloud of the target object in N-1 viewing directions according to the shape difference to obtain the target face point cloud.

[0435] In a possible implementation manner, the shape difference determination submodule is specifically used to:

[0436] A chamfer distance between the verification point cloud and a preset human face reference point cloud is obtained, and the chamfer distance is determined as a shape difference between the verification point cloud and the preset human face reference point cloud.

[0437] In a possible implementation, the point cloud fusion submodule is specifically used for:

[0438] If the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud;

[0439] If the shape difference is not less than the preset second threshold, jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0440] In a possible implementation manner, the point cloud fusion submodule is specifically used to include:

[0441] If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud;

[0442] If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

[0443] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Fig.12 The electronic device 1200 shown includes: at least one processor 1201, a memory 1202, at least one network interface 1204 and other user interfaces 1203. The various components in the electronic device 1200 are coupled together via a bus system 1205. It is understood that the bus system 1205 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 1205 is not described in detail. Fig.12 Various buses are labeled as bus system 1205.

[0444] The user interface 1203 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0445] It can be understood that the memory 1202 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The memory 1202 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0446] In some embodiments, the memory 1202 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 12021 and an application program 12022 .

[0447] The operating system 12021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 12022 includes various application programs, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 12022.

[0448] In the embodiment of the present invention, by calling the program or instruction stored in the memory 1202, specifically, the program or instruction stored in the application 12022, the processor 1201 is used to execute the method steps provided by each method embodiment, for example, including:

[0449] Acquire a face image pair of a target object in N viewing directions, wherein the face image pair includes a color image and a depth image;

[0450] For a face image pair under any of the viewing angle directions, determining a target color image aligned with the depth image according to the color image and the depth image in the face image pair;

[0451] Inputting the target color image into a preset image algorithm plug-in to obtain a human face area in the target color image;

[0452] According to the human face facial region, converting the target color image into a human face facial mask matrix;

[0453] Using the face mask matrix, the depth image is processed to obtain a target depth image, and the target depth image is converted into a face point cloud of the target object in the viewing direction;

[0454] According to the face point clouds of the target object under the N viewing directions, a target face point cloud is determined, and a surface reconstruction is performed on the target face point cloud to obtain a three-dimensional face model.

[0455] The method disclosed in the above embodiment of the present invention can be applied to the processor 1201, or implemented by the processor 1201. The processor 1201 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1201 or the instruction in the form of software. The above processor 1201 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1202, and the processor 1201 reads the information in the memory 1202 and completes the steps of the above method in combination with its hardware.

[0456] It is understood that the embodiments described herein can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0457] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0458] The electronic device provided in this embodiment may be Fig.12 The electronic device shown in can execute all the steps of the three-dimensional face model reconstruction method provided in this embodiment, thereby achieving the technical effect of the three-dimensional face model reconstruction method shown in this embodiment. For details, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0459] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0460] When one or more programs in the storage medium can be executed by one or more processors, the three-dimensional face model reconstruction method executed on the electronic device side can be implemented.

[0461] The processor is used to execute the three-dimensional face model reconstruction program stored in the memory to implement the following steps of the three-dimensional face model reconstruction method performed on the electronic device side:

[0462] Acquire a face image pair of a target object in N viewing directions, wherein the face image pair includes a color image and a depth image;

[0463] For a face image pair under any of the viewing angle directions, determining a target color image aligned with the depth image according to the color image and the depth image in the face image pair;

[0464] Inputting the target color image into a preset image algorithm plug-in to obtain a human face area in the target color image;

[0465] According to the human face facial region, converting the target color image into a human face facial mask matrix;

[0466] Using the face mask matrix, the depth image is processed to obtain a target depth image, and the target depth image is converted into a face point cloud of the target object in the viewing direction;

[0467] According to the face point clouds of the target object under the N viewing directions, a target face point cloud is determined, and a surface reconstruction is performed on the target face point cloud to obtain a three-dimensional face model.

[0468] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0469] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0470] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for reconstructing a three-dimensional face model, characterized in that: The method comprises: Acquire a face image pair of a target object in N viewing directions, wherein the face image pair includes a color image and a depth image; For a face image pair under any of the viewing angle directions, determining a target color image aligned with the depth image according to the color image and the depth image in the face image pair; Inputting the target color image into a preset image algorithm plug-in to obtain a human face area in the target color image; According to the human face facial region, converting the target color image into a human face facial mask matrix; Using the face mask matrix, the depth image is processed to obtain a target depth image, and the target depth image is converted into a face point cloud of the target object in the viewing direction; Determine a target face point cloud according to the face point clouds of the target object in the N viewing directions, and perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model; Wherein, determining the target face point cloud according to the face point cloud of the target object in the N viewing directions includes: Determine the centroids corresponding to the face point clouds of the target object in the N viewing directions, and determine the middle face point clouds of the target object in the N viewing directions according to the centroids; Selecting a posture reference point cloud from the middle face point clouds of the target object in the N viewing directions, wherein the posture reference point cloud includes the middle face point cloud of the target object in the frontal viewing direction; Using the posture reference point cloud, register the remaining intermediate face point clouds of the target object in N-1 viewing directions to obtain conversion point clouds of the target object in N-1 viewing directions; Using a preset human face reference point cloud, the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions are fused to obtain a target human face point cloud; The method of using the preset human face reference point cloud to fuse the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain the target human face point cloud includes: Merging the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain a complete face point cloud; Selecting a plurality of points from the complete face point cloud to form a verification point cloud, and determining a shape difference between the verification point cloud and a preset face reference point cloud; According to the shape difference, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud; The step of fusing the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions according to the shape difference to obtain a target face point cloud includes: If the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud; If the shape difference is not less than the preset second threshold, jump to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud; The step of fusing the posture reference point cloud and the conversion point clouds of the target object under N-1 viewing directions to obtain a target face point cloud includes: If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud; If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

2. The method according to claim 1, characterized in that The step of obtaining a face image pair of a target object in N viewing directions includes: Controlling the image acquisition device to rotate within a preset angle range, wherein the preset angle range takes the face of the target object as a central axis; and When it is detected that the image acquisition device is rotated to any preset viewing angle direction, the image acquisition device is controlled to acquire a face image pair of the target object in the current viewing angle direction; or, Image acquisition devices located in N viewing angles are controlled to synchronously acquire a pair of facial images of the target object.

3. The method according to claim 1, characterized in that The step of determining a target color image aligned with the depth image according to the color image and the depth image in the face image pair comprises: For any depth pixel point of the depth image in the face image pair, convert the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system; Converting the three-dimensional coordinates into target pixel coordinates, where the target pixel coordinates correspond to color pixels of the color image in the image pair; Determining a target color pixel point corresponding to the target pixel coordinate from the color pixel points of the color image; Assigning the color value corresponding to the target color pixel to the depth pixel; A target color image aligned with the depth image is generated by using the color value corresponding to each of the depth pixels in the depth image.

4. The method according to claim 3, characterized in that The step of converting the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system includes: Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula; Wherein, the first homogeneous transformation formula includes: ； ； Said Including the internal parameter matrix of the depth sensing module in the image acquisition device, the The external parameter matrix of the depth sensing module is composed of the rotation matrix and the offset vector The pixel coordinates of the depth pixel point include ( , ), Including the depth of field value, the three-dimensional coordinates include ( , , ).

5. The method according to claim 3, characterized in that: The converting the three-dimensional coordinates into target pixel coordinates comprises: Inputting the three-dimensional coordinates into a second homogeneous transformation formula to obtain target pixel coordinates output by the second homogeneous transformation formula; Wherein, the second homogeneous transformation formula includes: ； ； Said Including the internal parameter matrix of the color photosensitive module in the image acquisition device, the The external parameter matrix of the color photosensitive module is composed of the rotation matrix and the offset vector The three-dimensional coordinates include ( , , ), the target pixel coordinates of the depth pixel point include ( , ).

6. The method according to claim 1, characterized in that The step of converting the target color image into a face mask matrix according to the face area comprises: The pixel values ​​of the pixel points in the target color image that belong to the face area are set to non-zero, and the pixel values ​​of the pixel points in the target color image that do not belong to the face area are set to zero, so as to convert the target color image into a face mask matrix.

7. The method according to claim 1, characterized in that The process of processing the depth image using the face mask matrix to obtain a target depth image includes: Reading the depth image in matrix form to obtain a depth matrix corresponding to the depth image; The depth matrix and the face mask matrix are processed to obtain a target depth image.

8. The method according to claim 7, characterized in that The processing of the depth matrix and the face mask matrix to obtain a target depth image includes: Input the depth matrix and the face mask matrix into a preset matrix operation formula to obtain a target depth matrix output by the preset matrix operation formula; Determine that an image corresponding to the target depth matrix is ​​a target depth image, wherein depth values ​​of pixel points in a non-face area in the target depth image are reset to zero; Wherein, the preset matrix operation formula includes: ； Said including the target depth matrix, the Including a face mask matrix, the Including the depth matrix.

9. The method according to claim 1, characterized in that: The step of converting the target depth image into a face point cloud of the target object in the viewing direction comprises: Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value; The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system by a first homogeneous transformation formula; Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value; The three-dimensional points corresponding to the pixels with non-zero depth values ​​respectively form a face point cloud of the target object in the viewing direction.

10. The method according to claim 1, characterized in that The determining of the centroids corresponding to the face point clouds of the target object in the N viewing directions includes: For the face point cloud of the target object under any of the viewing angles, an average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

11. The method according to claim 1, characterized in that: The determining, according to the centroid, the intermediate face point clouds of the target object in the N viewing directions comprises: For the face point cloud of the target object in any of the viewing directions, the center of mass is subtracted from the spatial coordinates of each point in the face point cloud to obtain an intermediate face point cloud of the target object in the viewing direction.

12. The method according to claim 1, characterized in that The method of using the posture reference point cloud to register the remaining intermediate face point clouds of the target object in N-1 viewing directions to obtain the conversion point clouds of the target object in N-1 viewing directions includes: For the remaining middle face point cloud of the target object in any of the N-1 viewing directions, select M points from the remaining middle face point cloud; Using a preset KD-tree algorithm, searching for the nearest neighbor point corresponding to each of the M points from the posture reference point cloud; M points and their corresponding nearest neighbor points form M pairs of nearest point pairs, and the M pairs of nearest point pairs form a nearest point pair set; Determine a target rotation matrix and a target offset vector according to the set of closest point pairs; The remaining intermediate face point cloud is transformed using the target rotation matrix and the target offset vector to obtain an initial transformed point cloud of the target object in the viewing direction; A conversion error between the posture reference point cloud and the initial conversion point cloud is determined, and a conversion point cloud of the target object in the viewing direction is determined according to the conversion error.

13. The method according to claim 12, characterized in that Determining a target rotation matrix and a target offset vector according to the set of closest point pairs includes: Inputting the M pairs of closest point pairs in the set of closest point pairs into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula; Performing SVD decomposition on the construction matrix to obtain a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtaining a target rotation matrix output by the preset rotation matrix operation formula; The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

14. The method according to claim 13, characterized in that The preset matrix construction formula includes: ； Said Including the construction matrix, the Including the number of closest point pairs in the closest point pair set, the and stated Including the closest point pair in the set of closest point pairs; The preset rotation matrix operation formula includes: ； Said Including the target rotation matrix, the Including the first eigenvalue, the including the second eigenvalue; The preset offset vector calculation formula includes: ； Said including the target offset vector, the Including the centroid of the posture reference point cloud, the Including the target rotation matrix, the Include the centroid of the remaining intermediate face point cloud.

15. The method according to claim 12, characterized in that The step of transforming the remaining intermediate face point cloud using the target rotation matrix and the target offset vector to obtain an initial transformed point cloud of the target object in the viewing direction includes: Inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into a point cloud conversion formula; Obtaining an initial conversion point cloud of the target object in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes: ； Said Including the initial conversion point cloud, the Including the remaining intermediate face point cloud, the including the target offset vector, the Contains the target rotation matrix.

16. The method according to claim 12, characterized in that The determining of the conversion error between the posture reference point cloud and the initial conversion point cloud comprises: Inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula; Obtaining a posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as a conversion error; Wherein, the preset posture difference calculation formula includes: ； Said Including the initial conversion point cloud, the Including the posture reference point cloud.

17. The method according to claim 12, characterized in that Determining the conversion point cloud of the target object in the viewing direction according to the conversion error includes: If the conversion error is less than a preset first threshold, determining that the initial conversion point cloud is a conversion point cloud of the target face in the viewing direction; If the conversion error is not less than the preset first threshold, jump to the step of selecting M points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target object in the viewing direction.

18. The method according to claim 1, characterized in that The determining of the shape difference between the verification point cloud and a preset face reference point cloud includes: A chamfer distance between the verification point cloud and a preset human face reference point cloud is obtained, and the chamfer distance is determined as a shape difference between the verification point cloud and the preset human face reference point cloud.

19. A 3D face model reconstruction device, characterized in that: The device comprises: An image pair acquisition module, used to acquire a face image pair of a target object in N viewing directions, wherein the face image pair includes a color image and a depth image; An image determination module, configured to determine, for a face image pair under any of the viewing angle directions, a target color image aligned with the depth image based on the color image and the depth image in the face image pair; A region determination module, used for inputting the target color image into a preset image algorithm plug-in to obtain a human face region in the target color image; A matrix acquisition module, used for converting the target color image into a human face mask matrix according to the human face facial region; An image processing module, used to process the depth image using the face mask matrix to obtain a target depth image; A point cloud acquisition module, used for converting the target depth image into a face point cloud of the target object in the viewing direction; A point cloud determination module, used to determine a target face point cloud according to the face point clouds of the target object in the N viewing directions; A point cloud reconstruction module is used to perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model; Wherein, determining the target face point cloud according to the face point cloud of the target object in the N viewing directions includes: Determine the centroids corresponding to the face point clouds of the target object in the N viewing directions, and determine the middle face point clouds of the target object in the N viewing directions according to the centroids; Selecting a posture reference point cloud from the middle face point clouds of the target object in the N viewing directions, wherein the posture reference point cloud includes the middle face point cloud of the target object in the frontal viewing direction; Using the posture reference point cloud, register the remaining intermediate face point clouds of the target object in N-1 viewing directions to obtain conversion point clouds of the target object in N-1 viewing directions; Using a preset human face reference point cloud, the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions are fused to obtain a target human face point cloud; The method of using the preset human face reference point cloud to fuse the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain the target human face point cloud includes: Merging the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions to obtain a complete face point cloud; Selecting a plurality of points from the complete face point cloud to form a verification point cloud, and determining a shape difference between the verification point cloud and a preset face reference point cloud; According to the shape difference, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud; The step of fusing the posture reference point cloud and the conversion point clouds of the target object in N-1 viewing directions according to the shape difference to obtain a target face point cloud includes: If the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud; If the shape difference is not less than the preset second threshold, jump to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud; The step of fusing the posture reference point cloud and the conversion point clouds of the target object under N-1 viewing directions to obtain a target face point cloud includes: If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target object in the N-1 viewing directions are fused to obtain a target face point cloud; If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

20. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a three-dimensional face model reconstruction program stored in the memory to implement the three-dimensional face model reconstruction method according to any one of claims 1 to 18.

21. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the three-dimensional face model reconstruction method described in any one of claims 1-18.

Images