Three-dimensional face model reconstruction method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a method, device, electronic device and storage medium for reconstructing a three-dimensional face model. The method includes: obtaining an image pair of a target face under n viewing directions, wherein the image pair includes a color image and a depth image; for the image pair of the target face under any viewing direction, determining a target color image aligned with the depth image according to the color image and the depth image in the image pair; determining a face area in the target color image, and converting the target color image into a face mask matrix according to the face area; using the face mask matrix, processing the depth image to obtain a target depth image, and converting the target depth image into a face point cloud of the target face under the viewing direction; determining a target face point cloud according to the face point cloud of the target face under n viewing directions, and performing surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision technology of artificial intelligence, and in particular to a method, device, electronic device and storage medium for reconstructing a three-dimensional face model. Background Art

[0002] Currently, the reconstruction of three-dimensional face models based on real human faces has broad application scenarios in many fields such as face recognition systems, medicine, movie advertising, computer animation, games, video conferencing and videophones, human-computer interaction, etc., especially in face recognition.

[0003] At present, there are two main ways to reconstruct a three-dimensional face model: Method 1, build a monocular camera array and use the parallax principle to achieve three-dimensional face model reconstruction; Method 2, use laser scanning equipment and the laser ranging principle to achieve three-dimensional face model reconstruction.

[0004] Since the original point cloud can be obtained through a monocular camera array or a laser scanning device during the reconstruction of a 3D face model, manually editing the original point cloud to remove redundant points to obtain the face point cloud is time-consuming and laborious. Summary of the invention

[0005] In order to solve the above-mentioned time-consuming and labor-intensive technical problem of manually editing the original point cloud to remove redundant points to obtain the face point cloud, an embodiment of the present invention provides a three-dimensional face model reconstruction method, device, electronic device and storage medium.

[0006] In a first aspect of an embodiment of the present invention, a method for reconstructing a three-dimensional face model is first provided, the method comprising:

[0007] Acquire an image pair of the target face in n viewing directions, wherein the image pair includes a color image and a depth image;

[0008] For the image pair of the target face in any viewing direction, determine a target color image aligned with the depth image according to the color image and the depth image in the image pair;

[0009] Determine a human face region in the target color image, and convert the target color image into a human face mask matrix according to the human face region;

[0010] Using the face mask matrix, the depth image is processed to obtain a target depth image, and the target depth image is converted into a face point cloud of the target face in the viewing direction;

[0011] According to the facial point clouds of the target face under the n viewing directions, a target facial point cloud is determined, and a surface reconstruction is performed on the target facial point cloud to obtain a three-dimensional face model.

[0012] In an optional implementation, acquiring the image pairs of the target face in n viewing directions includes:

[0013] Taking the target face as the central axis, controlling the image acquisition device to rotate according to a preset rotation angle within a preset angle range; and

[0014] After controlling the image acquisition device to rotate the preset rotation angle each time, calling the image acquisition device to acquire an image pair of a target face;

[0015] or,

[0016] Taking the target face as the central axis, image acquisition devices are respectively set in n viewing angle directions within a preset angle range;

[0017] The image acquisition devices in n viewing angles are synchronously called to obtain image pairs of the target face in n viewing angles.

[0018] In an optional implementation, determining, according to the color image and the depth image in the image pair, a target color image aligned with the depth image includes:

[0019] For any depth pixel point of the depth image in the image pair, convert the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system;

[0020] Converting the three-dimensional coordinates into target pixel coordinates of the depth pixel points, where the target pixel coordinates of the depth pixel points correspond to color pixel points of the color image in the image pair;

[0021] Determining, from the color pixels of the color image, a target color pixel corresponding to the target pixel coordinates of the depth pixel;

[0022] Obtaining a color value corresponding to the target color pixel, and assigning the color value to the depth pixel;

[0023] The target color image aligned with the depth image is recomposed by using the color values ​​corresponding to the depth pixels of the depth image.

[0024] In an optional implementation, converting the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system includes:

[0025] Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula;

[0026] Wherein, the first homogeneous transformation formula includes:

[0027]

[0028]

[0029] The K d The internal parameter matrix of the depth sensing module in the image acquisition device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

[0030] In an optional implementation, converting the three-dimensional coordinates into target pixel coordinates of the depth pixel point includes:

[0031] Inputting the three-dimensional coordinates into a second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula;

[0032] Wherein, the second homogeneous transformation formula includes:

[0033]

[0034]

[0035] The K c The internal parameter matrix of the color photosensitive module in the image acquisition device is included. w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2c and the offset vector t w2c The three-dimensional coordinates include (the x w , the y w , the z w ), the target pixel coordinates of the depth pixel point include (the u c , the v c ).

[0036] In an optional implementation, the determining the facial region of a human face in the target color image includes:

[0037] Determine a contour line drawn by a user using a line drawing tool in the target color image, and determine that the area within the contour line is a human face area.

[0038] In an optional implementation, the converting the target color image into a face mask matrix according to the face area includes:

[0039] The face area in the target color image is set to non-zero, and the non-face area in the target color image is set to zero, so as to convert the target color image into a face mask matrix.

[0040] In an optional implementation, the using the face mask matrix to process the depth image to obtain a target depth image includes:

[0041] Reading the depth image in matrix form to obtain a depth matrix corresponding to the depth image;

[0042] The depth matrix and the face mask matrix are processed to obtain a target depth image.

[0043] In an optional implementation, the processing of the depth matrix and the face mask matrix to obtain a target depth image includes:

[0044] Input the depth matrix and the face mask matrix into a preset matrix operation formula to obtain a target depth matrix output by the preset matrix operation formula;

[0045] Determine that an image corresponding to the target depth matrix is ​​a target depth image, wherein depth values ​​of pixel points in a non-face area in the target depth image are reset to zero;

[0046] Wherein, the preset matrix operation formula includes:

[0047] M Depth′ =M face *M Depth ;

[0048] The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

[0049] In an optional implementation, converting the target depth image into a face point cloud of the target face in the viewing direction includes:

[0050] Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value;

[0051] The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system corresponding to the pixel point with a non-zero depth value by using a first homogeneous conversion formula;

[0052] Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value;

[0053] The three-dimensional points corresponding to the pixels with non-zero depth of field values ​​form a facial point cloud of the target face in the viewing direction.

[0054] In an optional implementation, determining the target face point cloud according to the face point clouds of the target face in the n viewing directions includes:

[0055] Determine the centroids corresponding to the facial point clouds of the target face in the n viewing directions, and determine the middle facial point clouds of the target face in the n viewing directions according to the centroids;

[0056] Selecting a posture reference point cloud from the middle face point clouds of the target face under the n viewing angles, wherein the posture reference point cloud includes the middle face point cloud of the target face under the frontal viewing angle;

[0057] Using the posture reference point cloud, register the remaining intermediate face point clouds of the target face in the n-1 viewing directions to obtain n-1 conversion point clouds of the target face in the n-1 viewing directions;

[0058] The preset human face reference point cloud is used to fuse the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain the target face point cloud.

[0059] In an optional implementation, the determining the centroids corresponding to the facial point clouds of the target faces in the n viewing directions respectively includes:

[0060] For the face point cloud of the target face under any of the viewing angles, an average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

[0061] In an optional implementation, determining the middle face point clouds of the target face in the n viewing directions according to the centroid includes:

[0062] For the facial point cloud of the target face in any of the viewing directions, the center of mass is subtracted from the spatial coordinates of each point in the facial point cloud to obtain an intermediate facial point cloud of the target face in the viewing direction.

[0063] In an optional implementation, the using the posture reference point cloud to register the remaining intermediate face point clouds of the target face in the n-1 viewing directions to obtain the n-1 conversion point clouds of the target face in the viewing directions includes:

[0064] For the remaining middle face point cloud of the target face in any of the n-1 viewing directions, select m points from the remaining middle face point cloud;

[0065] Using a preset KD-tree algorithm, find the nearest neighbor points corresponding to each of the m points from the posture reference point cloud;

[0066] The m points and the nearest neighbor points corresponding to the m points form m pairs of nearest point pairs, and the m pairs of nearest point pairs form a nearest point pair set;

[0067] Determine a target rotation matrix and a target offset vector according to the set of closest point pairs;

[0068] The remaining intermediate face point cloud is transformed using the target rotation matrix and the target offset vector to obtain an initial transformed point cloud of the target face in the viewing direction;

[0069] A conversion error between the posture reference point cloud and the initial conversion point cloud is determined, and a conversion point cloud of the target face in the viewing direction is determined based on the conversion error.

[0070] In an optional implementation, determining a target rotation matrix and a target offset vector according to the set of closest point pairs includes:

[0071] Inputting m pairs of closest point pairs in the set of closest point pairs into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula;

[0072] Performing SVD decomposition on the construction matrix to obtain a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtaining a target rotation matrix output by the preset rotation matrix operation formula;

[0073] The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

[0074] In an optional implementation, the preset matrix construction formula includes:

[0075]

[0076] The W includes the construction matrix, the N H Including the number of closest point pairs in the closest point pair set, the q i and the p i Including the closest point pair in the set of closest point pairs;

[0077] The preset rotation matrix operation formula includes:

[0078] R=UV T :

[0079] The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T including the second eigenvalue;

[0080] The preset offset vector calculation formula includes:

[0081] t=μ p -Rμ q ;

[0082] The t includes the target offset vector, the μ p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the μ q Include the centroid of the remaining intermediate face point cloud.

[0083] In an optional implementation, the using the target rotation matrix and the target offset vector to transform the remaining intermediate face point cloud to obtain an initial transformed point cloud of the target face in the viewing direction includes:

[0084] Inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into a point cloud conversion formula;

[0085] Obtaining an initial conversion point cloud of the target face in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes:

[0086] Q′={Rq+t|q∈Q};

[0087] The Q′ includes the initial conversion point cloud, the Q includes the remaining intermediate face point cloud, the t includes the target offset vector, and the R includes the target rotation matrix.

[0088] In an optional implementation, the determining a conversion error between the posture reference point cloud and the initial conversion point cloud comprises:

[0089] Inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula;

[0090] Obtaining a posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as a conversion error;

[0091] Wherein, the preset posture difference calculation formula includes:

[0092]

[0093] The Q′ includes the initial conversion point cloud, and the P includes the posture reference point cloud.

[0094] In an optional implementation, determining the conversion point cloud of the target face in the viewing direction according to the conversion error includes:

[0095] If the conversion error is less than a preset first threshold, determining that the initial conversion point cloud is a conversion point cloud of the target face in the viewing direction;

[0096] If the conversion error is not less than the preset first threshold, jump to the step of selecting m points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target face in the viewing direction.

[0097] In an optional embodiment, the preset human face reference point cloud is used to fuse the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain the target human face point cloud, including:

[0098] Merging the posture reference point cloud and the conversion point clouds of the target face in n-1 viewing directions to obtain a complete face point cloud;

[0099] Selecting a plurality of points from the complete face point cloud to form a verification point cloud, and determining a shape difference between the verification point cloud and a preset face reference point cloud;

[0100] According to the shape difference, the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions are fused to obtain the target face point cloud.

[0101] In an optional implementation, the determining the shape difference between the verification point cloud and a preset face reference point cloud includes:

[0102] A chamfer distance between the verification point cloud and a preset human face reference point cloud is obtained, and the chamfer distance is determined as a shape difference between the verification point cloud and the preset human face reference point cloud.

[0103] In an optional implementation, the step of fusing the posture reference point cloud and the conversion point clouds of the target face in n-1 viewing directions according to the shape difference to obtain the target face point cloud includes:

[0104] If the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions are fused to obtain a target face point cloud;

[0105] If the shape difference is not less than the preset second threshold, jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0106] In an optional implementation, fusing the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain the target face point cloud includes:

[0107] If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions are fused to obtain a target face point cloud;

[0108] If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

[0109] In a second aspect of an embodiment of the present invention, a 3D face model reconstruction device is provided, the device comprising:

[0110] An image pair acquisition module, used to acquire an image pair of a target face in n viewing directions, wherein the image pair includes a color image and a depth image;

[0111] An image determination module, configured to determine, for the image pair of the target face in any viewing angle, a target color image aligned with the depth image according to the color image and the depth image in the image pair;

[0112] A region determination module, used to determine a facial region of a human face in the target color image;

[0113] A matrix acquisition module, used for converting the target color image into a human face mask matrix according to the human face facial region;

[0114] An image processing module, used to process the depth image using the face mask matrix to obtain a target depth image;

[0115] A point cloud acquisition module, used for converting the target depth image into a face point cloud of the target face in the viewing direction;

[0116] A point cloud determination module, used to determine a target face point cloud according to the face point clouds of the target face in the n viewing directions;

[0117] The point cloud reconstruction module is used to perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

[0118] In an optional implementation, the image pair acquisition module is specifically used to:

[0119] Taking the target face as the central axis, controlling the image acquisition device to rotate according to a preset rotation angle within a preset angle range; and

[0120] After controlling the image acquisition device to rotate the preset rotation angle each time, calling the image acquisition device to acquire an image pair of a target face;

[0121] or,

[0122] Taking the target face as the central axis, image acquisition devices are respectively set in n viewing angle directions within a preset angle range;

[0123] The image acquisition devices in n viewing angles are synchronously called to obtain image pairs of the target face in n viewing angles.

[0124] In an optional implementation, the image determination module specifically includes:

[0125] A coordinate and depth value conversion submodule, for converting the pixel coordinates and the depth value of any depth pixel of the depth image in the image pair into three-dimensional coordinates in a world coordinate system;

[0126] a coordinate conversion submodule, configured to convert the three-dimensional coordinates into target pixel coordinates of the depth pixel points, wherein the target pixel coordinates of the depth pixel points correspond to color pixel points of the color image in the image pair;

[0127] A pixel point determination submodule, used to determine a target color pixel point corresponding to the target pixel coordinate of the depth pixel point from the color pixel points of the color image;

[0128] A color value assignment submodule, used to obtain the color value corresponding to the target color pixel point, and assign the color value to the depth pixel point;

[0129] The image composition submodule is used to recompose a target color image aligned with the depth image by using the color values ​​corresponding to the depth pixels of the depth image.

[0130] In an optional implementation, the coordinate and depth of field value conversion submodule is specifically used for:

[0131] Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula;

[0132] Wherein, the first homogeneous transformation formula includes:

[0133]

[0134]

[0135] The K d The internal parameter matrix of the depth sensing module in the image acquisition device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

[0136] In an optional implementation, the coordinate conversion submodule is specifically used for:

[0137] Inputting the three-dimensional coordinates into a second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula;

[0138] Wherein, the second homogeneous transformation formula includes:

[0139]

[0140]

[0141] The K c The internal parameter matrix of the color photosensitive module in the image acquisition device is included. w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2c and the offset vector t w2c The three-dimensional coordinates include (the x w , the y w , the z w ), the target pixel coordinates of the depth pixel point include (the u c , the v c ).

[0142] In an optional implementation, the region determination module is specifically configured to:

[0143] Determine a contour line drawn by a user using a line drawing tool in the target color image, and determine that the area within the contour line is a human face area.

[0144] In an optional implementation, the matrix acquisition module is specifically used to:

[0145] The face area in the target color image is set to non-zero, and the non-face area in the target color image is set to zero, so as to convert the target color image into a face mask matrix.

[0146] In an optional implementation, the image processing module specifically includes:

[0147] An image reading submodule, configured to read the depth image in a matrix form to obtain a depth matrix corresponding to the depth image;

[0148] The matrix processing submodule is used to process the depth matrix and the face mask matrix to obtain a target depth image.

[0149] In an optional implementation, the matrix processing submodule is specifically used for:

[0150] Input the depth matrix and the face mask matrix into a preset matrix operation formula to obtain a target depth matrix output by the preset matrix operation formula;

[0151] Determine that an image corresponding to the target depth matrix is ​​a target depth image, wherein depth values ​​of pixel points in a non-face area in the target depth image are reset to zero;

[0152] Wherein, the preset matrix operation formula includes:

[0153] M Depth′ =M face *M Depth ;

[0154] The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

[0155] In an optional implementation, the point cloud acquisition module is specifically used to:

[0156] Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value;

[0157] The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system corresponding to the pixel point with a non-zero depth value by using a first homogeneous conversion formula;

[0158] Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value;

[0159] The three-dimensional points corresponding to the pixels with non-zero depth of field values ​​form a facial point cloud of the target face in the viewing direction.

[0160] In an optional implementation, the point cloud determination module specifically includes:

[0161] A centroid determination submodule, used to determine the centroids corresponding to the facial point clouds of the target faces in the n viewing directions;

[0162] A point cloud determination submodule, used to determine the middle face point cloud of the target face in the n viewing directions according to the centroid;

[0163] A point cloud selection submodule, used to select a posture reference point cloud from the middle face point clouds of the target face under the n viewing directions, wherein the posture reference point cloud includes the middle face point cloud of the target face under the frontal viewing direction;

[0164] A point cloud registration submodule, for registering the remaining intermediate face point clouds of the target face in the n-1 viewing directions using the posture reference point cloud, to obtain n-1 conversion point clouds of the target face in the viewing directions;

[0165] The point cloud fusion submodule is used to use the preset face reference point cloud to fuse the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain the target face point cloud.

[0166] In an optional implementation, the centroid determination submodule is specifically used for:

[0167] For the face point cloud of the target face under any of the viewing angles, an average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

[0168] In an optional implementation, the point cloud determination submodule is specifically used to:

[0169] For the facial point cloud of the target face in any of the viewing directions, the center of mass is subtracted from the spatial coordinates of each point in the facial point cloud to obtain an intermediate facial point cloud of the target face in the viewing direction.

[0170] In an optional implementation, the point cloud registration submodule specifically includes:

[0171] A point selection unit, for selecting m points from the remaining middle face point cloud of the target face in any one of the n-1 viewing directions;

[0172] A point search unit, used to search for the nearest neighbor points corresponding to each of the m points from the posture reference point cloud using a preset KD-tree algorithm;

[0173] A set forming unit, used for forming m pairs of nearest point pairs from the m points and the nearest neighbor points corresponding to the m points, and forming a nearest point pair set from the m pairs of nearest point pairs;

[0174] A matrix and vector determination unit, used to determine a target rotation matrix and a target offset vector according to the set of closest point pairs;

[0175] A point cloud conversion unit, configured to convert the remaining intermediate face point cloud using the target rotation matrix and the target offset vector to obtain an initial conversion point cloud of the target face in the viewing direction;

[0176] a conversion error determination unit, configured to determine a conversion error between the posture reference point cloud and the initial conversion point cloud;

[0177] The point cloud determination unit is used to determine the conversion point cloud of the target face in the viewing direction according to the conversion error.

[0178] In an optional implementation, the matrix and vector determination unit is specifically used for:

[0179] Inputting m pairs of closest point pairs in the set of closest point pairs into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula;

[0180] Performing SVD decomposition on the construction matrix to obtain a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtaining a target rotation matrix output by the preset rotation matrix operation formula;

[0181] The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

[0182] In an optional implementation, the preset matrix construction formula includes:

[0183]

[0184] The W includes the construction matrix, the N H Including the number of closest point pairs in the closest point pair set, the q i and the p i Including the closest point pair in the set of closest point pairs;

[0185] The preset rotation matrix operation formula includes:

[0186] R=UV T ;

[0187] The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T including the second eigenvalue;

[0188] The preset offset vector calculation formula includes:

[0189] t=μ p -Rμ q ;

[0190] The t includes the target offset vector, the μ p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the μ q Include the centroid of the remaining intermediate face point cloud.

[0191] In an optional implementation, the point cloud conversion unit is specifically used to:

[0192] Inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into a point cloud conversion formula;

[0193] Obtaining an initial conversion point cloud of the target face in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes:

[0194] Q′={Rq+t|q∈Q};

[0195] The Q′ includes the initial conversion point cloud, the Q includes the remaining intermediate face point cloud, the t includes the target offset vector, and the R includes the target rotation matrix.

[0196] In an optional implementation, the conversion error determination unit is specifically configured to:

[0197] Inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula;

[0198] Obtaining a posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as a conversion error;

[0199] Wherein, the preset posture difference calculation formula includes:

[0200]

[0201] The Q′ includes the initial conversion point cloud, and the P includes the posture reference point cloud.

[0202] In an optional implementation, the point cloud determination unit is specifically used to:

[0203] If the conversion error is less than a preset first threshold, determining that the initial conversion point cloud is a conversion point cloud of the target face in the viewing direction;

[0204] If the conversion error is not less than the preset first threshold, jump to the step of selecting m points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target face in the viewing direction.

[0205] In an optional implementation, the point cloud fusion submodule specifically includes:

[0206] A point cloud merging unit, used for merging the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain a complete face point cloud;

[0207] A point cloud composition unit, used for selecting a plurality of points from the complete face point cloud to form a verification point cloud;

[0208] A shape difference determination unit, used to determine the shape difference between the verification point cloud and a preset face reference point cloud;

[0209] The point cloud fusion unit is used to fuse the posture reference point cloud and the conversion point clouds of the target face in n-1 viewing directions according to the shape difference to obtain the target face point cloud.

[0210] In an optional implementation, the shape difference determining unit is specifically configured to:

[0211] A chamfer distance between the verification point cloud and a preset human face reference point cloud is obtained, and the chamfer distance is determined as a shape difference between the verification point cloud and the preset human face reference point cloud.

[0212] In an optional implementation, the point cloud fusion unit specifically includes:

[0213] a point cloud fusion subunit, configured to fuse the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain a target face point cloud if the shape difference is less than a preset second threshold;

[0214] A jump subunit is used to jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud if the shape difference is not less than the preset second threshold.

[0215] In an optional implementation, the point cloud fusion subunit is specifically used for:

[0216] If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions are fused to obtain a target face point cloud;

[0217] If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

[0218] In a third aspect of an embodiment of the present invention, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0219] Memory, used to store computer programs;

[0220] The processor is used to implement the three-dimensional face model reconstruction method described in the first aspect when executing the program stored in the memory.

[0221] In a fourth aspect of the embodiments of the present invention, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the instructions are executed on a computer, the computer executes the three-dimensional face model reconstruction method described in the first aspect above.

[0222] In a fifth aspect of the embodiments of the present invention, a computer program product comprising instructions is also provided. When the computer program product is run on a computer, the computer executes the three-dimensional face model reconstruction method described in the first aspect.

[0223] The technical solution provided by the embodiment of the present invention obtains an image pair of a target face under n viewing directions, wherein the image pair includes a color image and a depth image. For the image pair of the target face under any viewing direction, a target color image aligned with the depth image is determined according to the color image and the depth image in the image pair, a face facial region in the target color image is determined, and according to the face facial region, the target color image is converted into a face facial mask matrix, the depth image is processed using the face facial mask matrix to obtain a target depth image, the target depth image is converted into a face point cloud of the target face under the viewing direction, a target face point cloud is determined according to the face point cloud of the target face under n viewing directions, and the target face point cloud is surface reconstructed to obtain a three-dimensional face model. Compared with first obtaining the original point cloud, manually editing the original point cloud to remove redundant points to obtain the face point cloud, by determining the face area in the target color image, thereby generating a face mask matrix, using the face mask matrix to process the depth image to obtain the target depth image, and converting the target depth image into a face point cloud, the face point cloud can be obtained more conveniently and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0224] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0225] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0226] Figure 1 A schematic diagram of an implementation flow of a method for reconstructing a three-dimensional face model shown in an embodiment of the present invention;

[0227] Figure 2 It is a schematic diagram of obtaining an image pair of a model's face at different viewing angles shown in an embodiment of the present invention;

[0228] Figure 3 It is a schematic diagram of another method of obtaining an image pair of a model's face at different viewing angles according to an embodiment of the present invention;

[0229] Figure 4 A schematic diagram of visual representation of a facial region of a human face shown in an embodiment of the present invention;

[0230] Figure 5 A schematic diagram of manually outlining a facial region of a human face shown in an embodiment of the present invention;

[0231] Figure 6 A schematic diagram of a face mask matrix of a human face shown in an embodiment of the present invention;

[0232] Figure 7 A schematic diagram of a surface reconstruction algorithm shown in an embodiment of the present invention;

[0233] Figure 8 Schematic diagram of an implementation flow of a method for determining a target face point cloud in an embodiment of the present invention;

[0234] Fig. 9 It is a schematic diagram of an implementation process of registering the remaining intermediate face point clouds of the target face in n-1 viewing directions shown in an embodiment of the present invention;

[0235] Fig.10 It is a schematic diagram of an implementation process of fusing a posture reference point cloud and a conversion point cloud of a target face in n-1 viewing directions shown in an embodiment of the present invention;

[0236] Fig.11 It is a structural schematic diagram of a three-dimensional face model reconstruction device shown in an embodiment of the present invention;

[0237] Fig.12 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0238] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0239] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0240] like Figure 1FIG. 1 is a schematic diagram of an implementation flow of a method for reconstructing a three-dimensional face model provided by an embodiment of the present invention. The method can be applied to a processor and can specifically include the following steps:

[0241] S101, obtaining an image pair of a target face in n viewing directions, wherein the image pair includes a color image and a depth image.

[0242] In the embodiment of the present invention, since the human face is three-dimensional, the raised part will cause occlusion. In order to eliminate the influence of occlusion, it is necessary to capture the human face from different viewing directions, that is, to capture the human face from n (n is a positive integer, and ≥1) viewing directions.

[0243] The corresponding modeling object is the human face. The image acquisition device (such as a consumer-grade camera) must be at the same horizontal plane as the object. The more acquisition times, that is, the larger n is, the more conducive it is to the accuracy of subsequent operations.

[0244] Based on this, in an embodiment of the present invention, image pairs of the target face in n viewing directions are obtained, wherein the image pairs include a color image and a depth image, which means that the image pairs of the target face in each viewing direction include a color image and a depth image.

[0245] Among them, in the embodiment of the present invention, the image pairs of the target face under n viewing angles can be obtained in the following manner: taking the target face as the central axis, controlling the image acquisition device to rotate according to a preset rotation angle within a preset angle range, and calling the image acquisition device to acquire the image pairs of the target face after each control of the image acquisition device to rotate the preset rotation angle. In this way, by adjusting the posture of the image acquisition device (i.e., the orientation of the image acquisition device in the shooting space), the image pairs of the target face under n viewing angles can be obtained.

[0246] For example, taking a single consumer camera as an example, Figure 2 As shown, with the model's face (i.e., front face) as the central axis, the consumer-grade camera is controlled to rotate at a rotation angle of 15° (along the guide rail) within a maximum range of 120°, which means that the consumer-grade camera rotates once every 15°, and after each control of the consumer-grade camera to rotate at a rotation angle of 15° (along the guide rail), the consumer-grade camera is called to capture an image pair of the model's face. In this way, image pairs of the model's face in 120 / 15=8 viewing angles can be obtained.

[0247] Among them, in an embodiment of the present invention, the image pairs of the target face in n viewing directions can be obtained in the following manner: taking the target face as the central axis, respectively setting image acquisition devices in n viewing directions within a preset angle range; synchronously calling the image acquisition devices in n viewing directions to obtain the image pairs of the target face in n viewing directions.

[0248] For example, take three consumer cameras as an example. Figure 3 As shown in the figure, with the model's face (i.e., the front face) as the central axis, consumer-grade cameras are set up in three viewing directions within a maximum range of 120°, and these three consumer-grade cameras are called synchronously to obtain image pairs of the model's face in three viewing directions. In this way, the image pairs of the target face in three viewing directions can still be obtained through the three cameras.

[0249] It should be noted that the image acquisition device usually refers to a consumer-grade camera, which includes two photosensitive modules, namely a color photosensitive module and a depth photosensitive module, so that image pairs can be captured, namely a color image (RGB image) and a depth image (Depth image). The embodiment of the present invention is not limited to this.

[0250] S102: for the image pair of the target face in any viewing angle, determine a target color image aligned with the depth image according to the color image and the depth image in the image pair.

[0251] S103, determining a human face region in the target color image, and converting the target color image into a human face mask matrix according to the human face region.

[0252] S104, using the face mask matrix, processing the depth image to obtain a target depth image, and converting the target depth image into a face point cloud of the target face in the viewing direction.

[0253] For an image pair of a target face under n viewing directions, in an embodiment of the present invention, for an image pair of a target face under any viewing direction, a target color image aligned with the depth image is determined based on the color image and the depth image in the image pair, a face facial region in the target color image is determined, and based on the face facial region, the target color image is converted into a face facial mask matrix, the depth image is processed using the face facial mask matrix to obtain a target depth image, and the target depth image is converted into a face point cloud of the target face under the viewing direction.

[0254] For example, with an image pair of the target face under the i-th (i=1, 2, 3..., n) viewing direction, a target color image aligned with the Depth image is determined based on the RGB image and the Depth image in the image pair, i.e., an RGB′ image, whose image size is consistent with the Depth image, and the face area in the RGB′ image is determined. Based on the face area, the RGB′ image is converted into a face mask matrix, i.e., the manual outline result is converted into a face mask matrix, and the Depth image is processed using the face mask matrix to obtain a target depth image, i.e., a Depth′ image, and the Depth′ image is converted into a face point cloud of the target face under the i-th viewing direction. In this way, face point clouds of the target face under n viewing directions can be obtained.

[0255] Among them, in an embodiment of the present invention, based on the color image and the depth image in the image pair, a target color image aligned with the depth image is determined specifically in the following manner: for any depth pixel point of the depth image in the image pair, the pixel coordinates and the depth of field value of the depth pixel point are converted into three-dimensional coordinates in the world coordinate system; the three-dimensional coordinates are converted into the target pixel coordinates of the depth pixel point, and the target pixel coordinates of the depth pixel point correspond to the color pixel point of the color image in the image pair; from the color pixels of the color image, the target color pixel point corresponding to the target pixel coordinates of the depth pixel point is determined; the color value corresponding to the target color pixel point is obtained, and the color value is assigned to the depth pixel point; and the color values ​​corresponding to each depth pixel point of the depth image are used to reconstruct the target color image aligned with the depth image.

[0256] For an image pair, for any depth pixel point of the depth image in the image pair, the pixel coordinates and depth of field value of the depth pixel point are input into the first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula. In this way, the conversion of the pixel coordinates and depth of field value of the depth pixel point is completed and converted into three-dimensional coordinates in the world coordinate system.

[0257] For example, for any depth pixel point of the Depth image in the image pair, the pixel coordinates (u d , v d ) and depth of field value z c Input into the first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula, wherein the first homogeneous transformation formula includes:

[0258]

[0259]

[0260] The K dThe internal parameter matrix of the depth sensing module in the image acquisition device may be 3x3 in size and is obtained by reading the image acquisition device interface or manually calibrating. w2d The external parameter matrix of the depth sensing module is 4x4 in size, which is the Euclidean transformation matrix from the world coordinate system to the depth sensing module, and is composed of a 3x3 rotation matrix R w2d and the offset vector t w2d Both can be obtained by manual calibration. The pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

[0261] For the three-dimensional coordinates in the world coordinate system, the three-dimensional coordinates in the world coordinate system are input into the second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula, so as to convert the three-dimensional coordinates in the world coordinate system into the target pixel coordinates of the depth pixel point. The target pixel coordinates of the depth pixel point are actually the pixel coordinates of the color pixel point in the color image, that is, the target pixel coordinates of the depth pixel point correspond to the color pixel point of the color image in the image pair.

[0262] For example, for the three-dimensional coordinates in the world coordinate system, the three-dimensional coordinates in the world coordinate system are input into the second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel output by the second homogeneous transformation formula. The target pixel coordinates of the depth pixel correspond to the color pixel of the color image in the image pair. The target pixel coordinates of the depth pixel are the pixel coordinates of the color pixel in the RGB image (u c , v c ), where the second homogeneous transformation formula includes:

[0263]

[0264]

[0265] The K c The internal parameter matrix of the color photosensitive module in the image acquisition device can be 3x3 in size and can be read through the image acquisition device interface or manually calibrated. w2c The external parameter matrix of the color photosensitive module is 4x4 in size, which is the Euclidean transformation matrix from the world coordinate system to the color photosensitive module, and is composed of a 3x3 rotation matrix R w2c and the offset vector t w2c Both can be obtained by manual calibration. The three-dimensional coordinates include (the xw , the y w , the z w ), the target pixel coordinates of the depth pixel point include (the u c , the v c ).

[0266] In an embodiment of the present invention, the target pixel coordinates of the depth pixel point are the pixel coordinates of the color pixel point in the color image, that is, the color pixel point corresponding to the color image in the image pair. In this way, the target color pixel point corresponding to the target pixel coordinates of the depth pixel point can be determined from the color pixel points of the color image, and the color value corresponding to the target color pixel point is obtained, and the color value is assigned to the depth pixel point. In this way, the color value corresponding to each depth pixel point in the depth image is found in the color image, so that each depth pixel point in the depth image has a corresponding color value and depth of field value. Subsequently, the color values ​​corresponding to each depth pixel point of the depth image are used to recompose the target color image aligned with the depth image, which is called an RGB′ image, and its image size is consistent with the Depth image. The RGB′ image and the Depth image aligned at the pixel level are called RGB'-D data.

[0267] For the target color image, the user can use a line drawing tool to manually outline the face area of ​​the human face on the target color image, so that the embodiment of the present invention can determine the contour line drawn by the user using the line drawing tool in the target color image, and determine that the area within the contour line is the face area of ​​the human face. The pixel coordinates of the pixel points on the manually drawn contour line can be formed into a set for storage.

[0268] It should be noted that the facial area of ​​a human face is usually defined as "the area surrounded by the left ear root (excluding the left ear), the hairline, the right ear root (excluding the right ear), and the chin extension", and its visual expression is as follows: Figure 4 As shown, users can refer to Figure 4 The target color image, i.e., the facial region in the RGB′ image, is outlined using a line drawing tool, which will produce contour lines, such as Figure 5 shown.

[0269] For the artificial outline result, that is, the face area in the target color image that is artificially outlined, the face area in the target color image is set to non-zero, and the non-face area in the target color image is set to zero, so as to convert the target color image into a face mask matrix.

[0270] For example, for the face area in the manually outlined RGB′ image, the face area in the RGB′ image is set to 1, and the non-face area in the RGB′ image is set to 0, so as to convert the RGB′ image into a face mask matrix, which means converting the manually outlined result into a face mask matrix M composed of 0 and 1. face , where 0 represents a non-face area and 1 represents a face area. face The shape of is the same as the RGB'-D Depth image, such as Figure 6 shown.

[0271] For the depth image in the image pair, the depth image is read in matrix form to obtain a depth matrix corresponding to the depth image, and the depth matrix and the face mask matrix are processed to obtain a target depth image. In this way, the depth image is processed using the face mask matrix to obtain a target depth image.

[0272] Among them, after the depth image is read as a depth matrix in matrix form, the depth of field values ​​of the depth pixels in the non-face area of ​​the depth image can be reset to zero by calculating the Hamada product of the matrix to obtain a target depth matrix, and the image corresponding to the target depth matrix is ​​determined as the target depth image, so that the depth of field values ​​of the pixels in the non-face area of ​​the target depth image are reset to zero.

[0273] For example, the depth image is read in matrix form as the depth matrix M Depth Then, M Depth and M face Input the preset matrix operation formula to obtain the target depth matrix M output by the preset matrix operation formula Depth′ , M Depth′ The corresponding image is the target depth image, that is, the Depth' image. The Depth' image and the RGB' image together constitute RGB'-D' data, wherein the preset matrix operation formula includes:

[0274] M Depth′ =M face *M Depth ;

[0275] The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

[0276] It should be noted that after the above steps, in the subsequent process of converting the target depth image into the face point cloud of the target face in the viewing direction, the target face is directly converted into the face point cloud according to the z wWhether the coordinates are 0 is used to remove the background, hair, ears, neck, torso and other areas in the original point cloud data that are irrelevant to the real-person expression modeling task that the present invention is concerned with, leaving a clean face area that is strongly related to the real-person expression, which is helpful to reduce the difficulty of subsequent point cloud alignment.

[0277] For the target depth image, determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and depth value of the pixel point with a non-zero depth value; convert the pixel coordinates and depth value of the pixel point with a non-zero depth value into three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value through a first homogeneous transformation formula; determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value; and form a facial point cloud of the target face in the viewing direction from the three-dimensional points corresponding to each pixel point with a non-zero depth value.

[0278] For example, for the Depth' image, determine the pixels with non-zero depth values ​​in the Depth' image, and for any pixel with non-zero depth value, obtain the pixel coordinates (u d , v d ) and depth of field value z c , the pixel coordinates (u d , v d ) and depth of field value z c , is converted into the three-dimensional coordinates (x w ,y w , z w ), thereby converting the pixel points on the plane into three-dimensional points in space, so that the three-dimensional points corresponding to each pixel point with a non-zero depth value can be obtained, and the three-dimensional points corresponding to each pixel point with a non-zero depth value form a face point cloud of the target face in the viewing direction, and the color value of each three-dimensional point can be obtained from the corresponding pixel point in the above RGB′ image.

[0279] S105, determining a target face point cloud according to the face point clouds of the target face in the n viewing directions, and performing surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

[0280] After the above steps, the facial point cloud of the target face in each viewing direction can be obtained, so that the target facial point cloud can be determined according to the facial point cloud of the target face in n viewing directions, and the target facial point cloud can be reconstructed to obtain a three-dimensional face model.

[0281] It should be noted that after obtaining the desired point cloud of the complete face, that is, the target face point cloud, it is necessary to perform surface reconstruction to restore its geometric shape, thereby turning it into a three-dimensional face model. Taking Delaunay triangulation reconstruction as an example, a feasible surface reconstruction algorithm is as follows Figure 7 Of course, the triangular reconstruction described is only the most common reconstruction method. In addition to triangles, quadrilaterals and pentagons can also be used to represent the surface. In the face reconstruction scenario targeted by the present invention, triangular reconstruction is the most preferred.

[0282] Among them, in the embodiment of the present invention, if Figure 8 FIG. 1 is a schematic diagram of an implementation flow of a method for determining a target face point cloud provided by an embodiment of the present invention. The method is hard-applied to a processor and may specifically include the following steps:

[0283] S801, determining the centroids corresponding to the facial point clouds of the target face in the n viewing directions, and determining the middle facial point clouds of the target face in the n viewing directions according to the centroids.

[0284] In the embodiment of the present invention, for the face point cloud of the target face under n viewing directions, the centroid (i.e., centroid coordinate) corresponding to each of the face point clouds of the target face under n viewing directions is determined. Among them, for the face point cloud of the target face under any viewing direction, the average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

[0285] For example, the face point clouds of the target face under n viewing directions form a point cloud set C. For point cloud c in point cloud set C i , point cloud c i The center of mass u is the face point cloud of the target face in any of the face point clouds of the target face in n viewing directions. i Point cloud c i The average value of all the spatial coordinates of the points is used to obtain the point cloud c i The average value of all the spatial coordinates of the points is determined as the point cloud c i The calculation formula is as follows, where N i It is the point cloud c i The number of points contained, p j It is the point cloud c i The spatial coordinates of the jth point in .

[0286]

[0287] In addition, after determining the centroids corresponding to the facial point clouds of the target face in n viewing directions, the middle facial point cloud of the target face in n viewing directions can be determined based on the centroids. Among them, for the facial point cloud of the target face in any viewing direction, the spatial coordinates of each point in the facial point cloud are subtracted from the centroid (corresponding to the facial point cloud) to obtain the middle facial point cloud of the target face in the viewing direction. This operation unifies all point clouds into a centroid coordinate system with the centroid as the origin of its coordinate system, which facilitates the subsequent calculation of the target rotation matrix R and the target offset vector t using the SVD algorithm.

[0288] For example, the point cloud c i Corresponding to the face point cloud of the target face under any viewing direction in the face point cloud of the target face under n viewing directions, the point cloud c i Every point p in j Subtract point cloud c i The corresponding center of mass u i , the new point cloud c′ i , new point cloud c′ i That is, the middle face point cloud of the target face under the viewing direction, from which a new point cloud set C can be formed μ The calculation formula is as follows:

[0289] c′ i ={p j -u i |p j ∈c i}={p′ j}.

[0290] S802, selecting a posture reference point cloud from the middle face point clouds of the target face in the n viewing directions, wherein the posture reference point cloud includes the middle face point cloud of the target face in the frontal viewing direction.

[0291] For the middle face point cloud of the target face under n viewing directions, a posture reference point cloud is selected from the middle face point cloud of the target face under n viewing directions, wherein the posture reference point cloud includes the middle face point cloud of the target face under the frontal viewing direction.

[0292] For example, for the middle face point cloud of the target face under n viewing directions, the point cloud set C μ , from the point cloud set C μ The middle face point cloud of the target face in the frontal view direction is selected as the posture reference point cloud, that is, the posture reference point cloud P.

[0293] S803, using the posture reference point cloud, align the remaining intermediate face point clouds of the target face in the n-1 viewing directions to obtain n-1 conversion point clouds of the target face in the n-1 viewing directions.

[0294] In an embodiment of the present invention, the posture reference point cloud is the middle facial point cloud of the target face in the frontal viewing direction, so there are remaining middle facial point clouds of the target face in n-1 viewing directions. Therefore, the posture reference point cloud can be used to align the remaining middle facial point clouds of the target face in n-1 viewing directions to obtain the conversion point cloud of the target face in n-1 viewing directions.

[0295] It should be noted that the remaining intermediate facial point clouds of the target face in n-1 viewing directions are registered using the posture reference point cloud to obtain the conversion point clouds of the target face in n-1 viewing directions. The purpose is to make their (the remaining intermediate facial point clouds of the target face in n-1 viewing directions) spatial postures basically consistent.

[0296] Among them, Fig. 9 As shown, it is a schematic diagram of an implementation process of registering the remaining intermediate face point clouds of the target face in n-1 viewing directions provided by an embodiment of the present invention. The method can be specifically applied to a processor and may include the following steps:

[0297] S901, for the remaining intermediate face point cloud of the target face in any of the n-1 viewing directions, select m points from the remaining intermediate face point cloud.

[0298] In an embodiment of the present invention, for the remaining middle facial point clouds of the target face under n-1 viewing directions, for the remaining middle facial point clouds of the target face under any viewing direction, m points are (randomly) selected from the remaining middle facial point clouds.

[0299] For example, for the remaining middle face point clouds of the target face under n-1 viewing directions, for the remaining middle face point clouds Q of the target face under any viewing direction (Q belongs to the point cloud set C μ ), randomly select m points from the remaining intermediate face point cloud Q.

[0300] S902, using a preset KD-tree algorithm, searching for the nearest neighbor point corresponding to each of the m points in the posture reference point cloud.

[0301] In an embodiment of the present invention, for a posture reference point cloud, a KD-tree data structure is established on it to store the spatial coordinates of all its points, so that the preset KD-tree algorithm, that is, with the help of the characteristics of the KD-tree, is used to find the nearest neighbor points corresponding to each of the m points in the posture reference point cloud, that is, to find the nearest neighbor point corresponding to each point (among the m points) from the posture reference point cloud.

[0302] For example, a KD-tree data structure is established on the pose reference point cloud P to store the spatial coordinates of all its points. μ ), and use the preset KD-tree algorithm, that is, with the help of the characteristics of KD-tree, to find the nearest neighbor points corresponding to each of the m points from the posture reference point cloud.

[0303] S903, forming m pairs of nearest point pairs from the m points and the nearest neighboring points corresponding to the m points, and forming a nearest point pair set from the m pairs of nearest point pairs.

[0304] For the above m points and the nearest neighbor points corresponding to each of the m points, m pairs of nearest point pairs can be formed, and the m pairs of nearest point pairs constitute a nearest point pair set.

[0305] For example, for the above m points and the nearest neighbor points corresponding to each of the m points, m pairs of nearest point pairs can be formed, and the m pairs of nearest point pairs constitute a nearest point pair set H, as shown below.

[0306] H={(p i ,q i )|p i ∈P,q i ∈Q, 1≤i≤m}.

[0307] S904: Determine a target rotation matrix and a target offset vector according to the set of closest point pairs.

[0308] In the embodiment of the present invention, for the closest point pair set obtained above, a target rotation matrix and a target offset vector can be determined according to the closest point pair set.

[0309] Among them, the m pairs of closest point pairs in the closest point pair set are input into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula.

[0310] Perform SVD decomposition on the construction matrix to obtain the first eigenvalue and the second eigenvalue, input the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtain the target rotation matrix output by the preset rotation matrix operation formula.

[0311] The target rotation matrix, the centroid of the posture reference point cloud, and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

[0312] For example, a construction matrix W of size 3x3 is constructed for SVD decomposition, and m pairs of nearest point pairs in the nearest point pair set are input into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula, wherein the preset matrix construction formula includes:

[0313]

[0314] The W includes the construction matrix, the N H Including the number of closest point pairs in the closest point pair set, the q i and the p i Include the closest point pair in the set of closest point pairs.

[0315] Perform SVD decomposition on the construction matrix W to obtain the first eigenvalue U and the second eigenvalue V T , σ1, σ2, σ3 are the eigenvalues, and the calculation formula is as follows:

[0316]

[0317] The first eigenvalue U and the second eigenvalue V T Input the preset rotation matrix operation formula to obtain the target rotation matrix R output by the preset rotation matrix operation formula, wherein the preset rotation matrix operation formula includes:

[0318] R=UV T ;

[0319] The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T Including the second eigenvalue.

[0320] The target rotation matrix R, the centroid of the posture reference point cloud P, and the centroid of the remaining intermediate face point cloud Q are input into the preset offset vector operation formula to obtain the target offset vector t output by the preset offset vector operation formula, wherein the preset offset vector operation formula includes:

[0321] t=μ p -Rμ q ;

[0322] The t includes the target offset vector, the μ p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the μ q Include the centroid of the remaining intermediate face point cloud.

[0323] After the above steps, for the remaining intermediate face point clouds of the target face in any of the n-1 viewing directions, the corresponding target rotation matrix and target offset vector can be obtained.

[0324] S905: Using the target rotation matrix and the target offset vector, transform the remaining intermediate face point cloud to obtain an initial transformed point cloud of the target face in the viewing direction.

[0325] In an embodiment of the present invention, for the remaining intermediate facial point cloud of the target face in any of the n-1 viewing directions, the remaining intermediate facial point cloud can be transformed using the target rotation matrix and target offset vector corresponding to the remaining intermediate facial point cloud to obtain the initial transformed point cloud of the target face in that viewing direction.

[0326] For example, for the remaining intermediate facial point cloud Q of the target face in any of the n-1 viewing directions, the remaining intermediate facial point cloud Q can be transformed using the target rotation matrix R and target offset vector t corresponding to the remaining intermediate facial point cloud Q to obtain the initial transformed point cloud of the target face in that viewing direction.

[0327] Among them, for the remaining intermediate face point cloud of the target face in any of the n-1 viewing directions, the target rotation matrix and the target offset vector corresponding to the remaining intermediate face point cloud are input into the point cloud conversion formula to obtain the initial conversion point cloud of the target face in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes:

[0328] Q′={Rq+t|q∈Q};

[0329] The Q′ includes the initial conversion point cloud, the Q includes the remaining intermediate face point cloud, the t includes the target offset vector, and the R includes the target rotation matrix.

[0330] For example, for the remaining intermediate facial point cloud Q of the target face in any of the n-1 viewing directions, the target rotation matrix R and the target offset vector t corresponding to the remaining intermediate facial point cloud Q can be input into the above formula to obtain the initial conversion point cloud Q′ of the target face in the viewing direction output by the point cloud conversion formula. In this way, the remaining intermediate facial point cloud Q is transformed by the above formula, and the spatial posture of the obtained initial conversion point cloud Q′ approaches the posture reference point cloud P, that is, approaches the frontal face viewing direction.

[0331] S906, determining a conversion error between the posture reference point cloud and the initial conversion point cloud, and determining a conversion point cloud of the target face in the viewing direction according to the conversion error.

[0332] For the initial conversion point cloud of the target face in any of the n-1 viewing directions, in an embodiment of the present invention, the (point cloud) conversion error between the posture reference point cloud and the initial conversion point cloud is determined, and based on the conversion error, the conversion point cloud of the target face in the viewing direction can be determined.

[0333] Among them, for the initial conversion point cloud of the target face in any of the n-1 viewing directions, the posture reference point cloud and the initial conversion point cloud can be input into a preset posture difference calculation formula to obtain the posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and the posture difference is determined as the conversion error.

[0334] For example, for the initial conversion point cloud Q' of the target face in any of the n-1 viewing directions, the initial conversion point cloud Q' and the posture reference point cloud P can be input into a preset posture difference calculation formula to obtain the posture difference between the initial conversion point cloud Q' and the posture reference point cloud P output by the preset posture difference calculation formula as the (point cloud) conversion error L, wherein the preset posture difference calculation formula includes:

[0335]

[0336] The Q′ includes the initial conversion point cloud, and the P includes the posture reference point cloud.

[0337] In addition, for the initial conversion point cloud of the target face in any of the n-1 viewing directions, in an embodiment of the present invention, the (point cloud) conversion error between the posture reference point cloud and the initial conversion point cloud is determined. If the conversion error is less than a preset first threshold, the initial conversion point cloud is determined to be the conversion point cloud of the target face in the viewing direction.

[0338] If the conversion error is not less than the preset first threshold, jump to the step of selecting m points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target face in the viewing direction. Specifically, if the conversion error is not less than the preset first threshold, first determine whether the number of iterations reaches the preset number, if so, determine the initial conversion point cloud as the conversion point cloud of the target face in the viewing direction, if not, jump to the step of selecting m points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target face in the viewing direction.

[0339] S804, using a preset facial reference point cloud, the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions are fused to obtain a target face point cloud.

[0340] In the embodiment of the present invention, a preset face reference point cloud is introduced, and the face reference point cloud and the conversion point cloud of the target face in n-1 viewing directions are fused through the face reference point cloud to obtain the target face point cloud.

[0341] It should be noted that the facial reference point cloud may be a male or female facial reference point cloud. Specifically, an open source reference three-dimensional model may be obtained, and the facial part thereof may be captured as the facial reference point cloud. The embodiment of the present invention is not limited to this.

[0342] Among them, Fig.10 As shown, it is a schematic diagram of an implementation process of fusing a posture reference point cloud and a conversion point cloud of a target face under n-1 viewing directions provided by an embodiment of the present invention. The method is specifically used for a processor and may include the following steps:

[0343] S1001, merging the posture reference point cloud and the conversion point clouds of the target face in n-1 viewing directions to obtain a complete face point cloud.

[0344] In the embodiment of the present invention, the posture reference point cloud and the conversion point cloud of the target face in the n-1 viewing direction are merged to obtain a complete face point cloud.

[0345] For example, the posture reference point cloud P and the transformation point cloud Q' of the target face in the n-1 viewing direction are merged to obtain the complete face point cloud M.

[0346] It should be noted that the complete face point cloud may be a point cloud including a complete face, which is not limited in the embodiment of the present invention.

[0347] S1002, selecting a plurality of points from the complete face point cloud to form a verification point cloud, and determining a shape difference between the verification point cloud and a preset face reference point cloud.

[0348] For a complete face point cloud, multiple points are selected from the complete face point cloud to form a verification point cloud, and the shape difference between the verification point cloud and a preset face reference point cloud is determined.

[0349] For example, for a complete face point cloud M, a plurality of points are randomly selected from the complete face point cloud M to form a verification point cloud m, and a shape difference between the verification point cloud m and a preset face reference point cloud B is determined.

[0350] Among them, in an embodiment of the present invention, the shape difference between the verification point cloud and the preset face reference point cloud is determined specifically in the following manner: the chamfer distance between the verification point cloud and the preset face reference point cloud is obtained, and the chamfer distance is determined as the shape difference between the verification point cloud and the preset face reference point cloud.

[0351] It should be noted that in an embodiment of the present invention, one of the following can also be obtained between the verification point cloud and the preset face reference point cloud: L1, L2, deformation of L1, deformation of L2, which is used to describe the shape difference between the verification point cloud and the preset face reference point cloud. The embodiment of the present invention is not limited to this.

[0352] For example, in an embodiment of the present invention, the Chamfer Distance is used as a specific measure of the shape difference E, the chamfer distance between the verification point cloud m and the face reference point cloud B is obtained, and the chamfer distance is determined to be the shape difference E between the verification point cloud m and the face reference point cloud B.

[0353] The calculation formula is as follows, where N m and N B are the sizes of the downsampled verification point cloud m and the face reference point cloud B (i.e. the number of elements therein), respectively. The first term is the average of the sum of the minimum distances from any point in the verification point cloud m to the reference point cloud B, and the second term is the average of the sum of the minimum distances from any point in the reference point cloud B to the verification point cloud m. The two are combined as the chamfer distance in 3D space. The smaller its value is, the smaller the shape difference between the two point clouds is.

[0354]

[0355] S1003: According to the shape difference, the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions are fused to obtain a target face point cloud.

[0356] For the shape difference between the verification point cloud and the preset face reference point cloud, the posture reference point cloud and the conversion point cloud of the target face in n-1 viewing directions are fused according to the shape difference to obtain the target face point cloud.

[0357] Among them, if the shape difference is less than the preset second threshold, the posture reference point cloud and the conversion point cloud of the target face under n-1 viewing directions are fused to obtain the target face point cloud; if the shape difference is not less than the preset second threshold, jump to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0358] Specifically, if the shape difference is less than a preset second threshold, the posture reference point cloud and the conversion point cloud of the target face under n-1 viewing directions are initially fused to obtain the target face point cloud for display so that the user can confirm whether the shape is normal.

[0359] If the user confirms that the shape is normal, the shape confirmation instruction is triggered. When the user's shape confirmation instruction is received, the posture reference point cloud and the conversion point cloud of the target face in n-1 viewing directions are fused to obtain the target face point cloud.

[0360] If the user confirms that the shape is abnormal, a shape error instruction is triggered. After receiving the user's shape error instruction, the preset second threshold is adjusted, and the process jumps to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

[0361] For example, if the shape difference E is less than the preset second threshold c, the posture reference point cloud P and the conversion point cloud Q' of the target face in n-1 viewing directions are initially fused to obtain the target face point cloud for display so that the user can confirm whether the shape is normal.

[0362] If the user confirms that the shape is normal, the shape confirmation instruction is triggered. When the user's shape confirmation instruction is received, the posture reference point cloud P and the conversion point cloud Q' of the target face in n-1 viewing directions are fused to obtain the target face point cloud A.

[0363] If the user confirms that the shape is abnormal, the shape error instruction is triggered. After receiving the user's shape error instruction, the preset second threshold c is adjusted, and the process jumps to the step of selecting multiple points from the complete face point cloud to form a verification point cloud m.

[0364] Through the above description of the technical solution provided by the embodiment of the present invention, an image pair of the target face in n viewing directions is obtained, wherein the image pair includes a color image and a depth image, and for the image pair of the target face in any viewing direction, a target color image aligned with the depth image is determined according to the color image and the depth image in the image pair, a face facial region in the target color image is determined, and according to the face facial region, the target color image is converted into a face facial mask matrix, the depth image is processed using the face facial mask matrix to obtain a target depth image, the target depth image is converted into a facial point cloud of the target face in the viewing direction, a target face point cloud is determined according to the face point cloud of the target face in n viewing directions, and a surface reconstruction is performed on the target face point cloud to obtain a three-dimensional face model.

[0365] Compared with first obtaining the original point cloud, manually editing the original point cloud to remove redundant points to obtain the face point cloud, by determining the face area in the target color image, thereby generating a face mask matrix, using the face mask matrix to process the depth image to obtain the target depth image, and converting the target depth image into a face point cloud, the face point cloud can be obtained more conveniently and efficiently.

[0366] In addition, in the embodiment of the present invention, only one or more image acquisition devices (such as consumer-grade cameras) are required, thereby obtaining image pairs of the target face in n viewing directions to participate in the subsequent three-dimensional face model reconstruction, with low hardware investment cost and simple equipment maintenance.

[0367] Given that only one or more image acquisition devices (such as consumer-grade cameras) are required in the embodiments of the present invention, the scale of point cloud data subsequently collected and processed is generally in the thousands, which saves a large amount of computing resources compared to three-dimensional reconstruction involving tens of thousands, hundreds of thousands, or millions of points.

[0368] In the process of point cloud registration, adding the facial reference point cloud as a shape reference greatly reduces the shape deviation of the result obtained during the final point cloud fusion, such as irregular protrusions or depressions, abnormal nose shape, etc., thereby achieving reliable 3D reconstruction accuracy from low-precision point cloud data.

[0369] Corresponding to the above method embodiment, the embodiment of the present invention also provides a 3D face model reconstruction device, such as Fig.11 As shown, the device may include: an image pair acquisition module 1110, an image determination module 1120, a region determination module 1130, a matrix acquisition module 1140, an image processing module 1150, a point cloud acquisition module 1160, a point cloud determination module 1170, and a point cloud reconstruction module 1180.

[0370] An image pair acquisition module 1110 is used to acquire an image pair of a target face in n viewing directions, wherein the image pair includes a color image and a depth image;

[0371] An image determination module 1120 is configured to determine, for the image pair of the target face in any viewing angle, a target color image aligned with the depth image according to the color image and the depth image in the image pair;

[0372] A region determination module 1130, configured to determine a facial region of a human face in the target color image;

[0373] A matrix acquisition module 1140, configured to convert the target color image into a human face mask matrix according to the human face region;

[0374] An image processing module 1150 is used to process the depth image using the face mask matrix to obtain a target depth image;

[0375] A point cloud acquisition module 1160, configured to convert the target depth image into a face point cloud of the target face in the viewing direction;

[0376] A point cloud determination module 1170, configured to determine a target face point cloud according to the face point clouds of the target face in the n viewing directions;

[0377] The point cloud reconstruction module 1180 is used to perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

[0378] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it includes a processor 121, a communication interface 122, a memory 123 and a communication bus 124, wherein the processor 121, the communication interface 122, and the memory 123 communicate with each other through the communication bus 124.

[0379] Memory 123, used for storing computer programs;

[0380] The processor 121 is used to execute the program stored in the memory 123 to implement the following steps:

[0381] Acquire an image pair of a target face under n viewing directions, wherein the image pair includes a color image and a depth image; for the image pair of the target face under any viewing direction, determine a target color image aligned with the depth image according to the color image and the depth image in the image pair; determine a face region in the target color image, and convert the target color image into a face mask matrix according to the face region; use the face mask matrix to process the depth image to obtain a target depth image, and convert the target depth image into a face point cloud of the target face under the viewing direction; determine a target face point cloud according to the face point clouds of the target face under n viewing directions, and perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model.

[0382] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0383] The communication interface is used for communication between the above electronic device and other devices.

[0384] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0385] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0386] In another embodiment of the present invention, a storage medium is provided, in which instructions are stored. When the storage medium is run on a computer, the computer executes the three-dimensional face model reconstruction method described in any of the above embodiments.

[0387] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute the three-dimensional face model reconstruction method described in any one of the above embodiments.

[0388] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium, or transmitted from one storage medium to another storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0389] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0390] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0391] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for reconstructing a three-dimensional face model, characterized in that: The method comprises: Acquire an image pair of the target face in n viewing directions, wherein the image pair includes a color image and a depth image; For the image pair of the target face in any viewing direction, determine a target color image aligned with the depth image according to the color image and the depth image in the image pair; Determine a human face region in the target color image, and convert the target color image into a human face mask matrix according to the human face region; Using the face mask matrix, the depth image is processed to obtain a target depth image, and the target depth image is converted into a face point cloud of the target face in the viewing direction; Determine a target face point cloud according to the face point clouds of the target face under the n viewing directions, and perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model; Wherein, determining the target face point cloud according to the face point clouds of the target face under the n viewing directions comprises: determining the center of mass corresponding to each of the face point clouds of the target face under the n viewing directions, and determining the middle face point cloud of the target face under the n viewing directions according to the center of mass; selecting a posture reference point cloud from the middle face point clouds of the target face under the n viewing directions, the posture reference point cloud comprising the middle face point cloud of the target face under the frontal viewing direction; using the posture reference point cloud, registering the remaining middle face point clouds of the target face under the n-1 viewing directions, to obtain the conversion point clouds of the target face under the n-1 viewing directions; using the preset face reference point cloud, fusing the posture reference point cloud and the conversion point clouds of the target face under the n-1 viewing directions, to obtain the target face point cloud; The method of using a preset face reference point cloud to fuse the posture reference point cloud and the conversion point clouds of the target face under n-1 viewing directions to obtain a target face point cloud includes: merging the posture reference point cloud and the conversion point clouds of the target face under n-1 viewing directions to obtain a complete face point cloud; selecting a plurality of points from the complete face point cloud to form a verification point cloud, and determining a shape difference between the verification point cloud and the preset face reference point cloud; and fusing the posture reference point cloud and the conversion point clouds of the target face under n-1 viewing directions according to the shape difference to obtain a target face point cloud. Wherein, the step of fusing the posture reference point cloud and the conversion point clouds of the target face under n-1 viewing directions according to the shape difference to obtain the target face point cloud comprises: if the shape difference is less than a preset second threshold, fusing the posture reference point cloud and the conversion point clouds of the target face under n-1 viewing directions to obtain the target face point cloud; if the shape difference is not less than the preset second threshold, jumping to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud; Among them, the step of fusing the posture reference point cloud and the conversion point clouds of the target face under n-1 viewing directions to obtain the target face point cloud includes: if a shape confirmation instruction from the user is received, fusing the posture reference point cloud and the conversion point clouds of the target face under n-1 viewing directions to obtain the target face point cloud; if a shape error instruction from the user is received, adjusting the preset second threshold and jumping to the step of selecting multiple points from the complete face point cloud to form a verification point cloud.

2. The method according to claim 1, characterized in that The step of obtaining the image pairs of the target face in n viewing directions includes: Taking the target face as the central axis, controlling the image acquisition device to rotate according to a preset rotation angle within a preset angle range; and After controlling the image acquisition device to rotate the preset rotation angle each time, calling the image acquisition device to acquire an image pair of a target face; or, Taking the target face as the central axis, image acquisition devices are respectively set in n viewing angle directions within a preset angle range; The image acquisition devices in n viewing angles are synchronously called to obtain image pairs of the target face in n viewing angles.

3. The method according to claim 1, characterized in that: The step of determining, according to the color image and the depth image in the image pair, a target color image aligned with the depth image comprises: For any depth pixel point of the depth image in the image pair, convert the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system; Converting the three-dimensional coordinates into target pixel coordinates of the depth pixel points, where the target pixel coordinates of the depth pixel points correspond to color pixel points of the color image in the image pair; Determining, from the color pixels of the color image, a target color pixel corresponding to the target pixel coordinates of the depth pixel; Obtaining a color value corresponding to the target color pixel, and assigning the color value to the depth pixel; The target color image aligned with the depth image is recomposed by using the color values ​​corresponding to the depth pixels of the depth image.

4. The method according to claim 3, characterized in that The step of converting the pixel coordinates and the depth value of the depth pixel point into three-dimensional coordinates in a world coordinate system includes: Inputting the pixel coordinates and the depth of field value of the depth pixel point into a first homogeneous transformation formula to obtain the three-dimensional coordinates in the world coordinate system output by the first homogeneous transformation formula; Wherein, the first homogeneous transformation formula includes: The K d The internal parameter matrix of the depth sensing module in the image acquisition device is included. w2d The external parameter matrix of the depth sensing module is composed of the rotation matrix R w2d and the offset vector t w2d The pixel coordinates of the depth pixel point include (the u d , the v d ), the z c Including the depth of field value, the three-dimensional coordinates include (the x w , the y w , the z w ).

5. The method according to claim 3, characterized in that: The converting the three-dimensional coordinates into the target pixel coordinates of the depth pixel point comprises: Inputting the three-dimensional coordinates into a second homogeneous transformation formula to obtain the target pixel coordinates of the depth pixel point output by the second homogeneous transformation formula; Wherein, the second homogeneous transformation formula includes: The K c The internal parameter matrix of the color photosensitive module in the image acquisition device is included. w2c The external parameter matrix of the color photosensitive module is composed of the rotation matrix R w2c and the offset vector t w2c The three-dimensional coordinates include (the x w , the y w , the z w ), the target pixel coordinates of the depth pixel point include (the u c , the v c ).

6. The method according to claim 1, characterized in that The step of determining a facial region of a human face in the target color image comprises: Determine a contour line drawn by a user using a line drawing tool in the target color image, and determine that the area within the contour line is a human face area.

7. The method according to claim 1, characterized in that The step of converting the target color image into a face mask matrix according to the face area comprises: The face area in the target color image is set to non-zero, and the non-face area in the target color image is set to zero, so as to convert the target color image into a face mask matrix.

8. The method according to claim 1, characterized in that The process of processing the depth image using the face mask matrix to obtain a target depth image includes: Reading the depth image in matrix form to obtain a depth matrix corresponding to the depth image; The depth matrix and the face mask matrix are processed to obtain a target depth image.

9. The method according to claim 8, characterized in that The processing of the depth matrix and the face mask matrix to obtain a target depth image includes: Input the depth matrix and the face mask matrix into a preset matrix operation formula to obtain a target depth matrix output by the preset matrix operation formula; Determine that an image corresponding to the target depth matrix is ​​a target depth image, wherein depth values ​​of pixel points in a non-face area in the target depth image are reset to zero; Wherein, the preset matrix operation formula includes: M Depth′ =M face *M Depth ; The M Depth′ Including the target depth matrix, the M face Including the face mask matrix, the M Depth Including the depth matrix.

10. The method according to claim 1, characterized in that The step of converting the target depth image into a face point cloud of the target face in the viewing direction comprises: Determine each pixel point with a non-zero depth value in the target depth image, and for any pixel point with a non-zero depth value, obtain the pixel coordinates and the depth value of the pixel point with a non-zero depth value; The pixel coordinates and the depth value of the pixel point with a non-zero depth value are converted into three-dimensional coordinates in a world coordinate system corresponding to the pixel point with a non-zero depth value by using a first homogeneous conversion formula; Determine that the three-dimensional coordinates in the world coordinate system corresponding to the pixel point with a non-zero depth value are the three-dimensional points corresponding to the pixel point with a non-zero depth value; The three-dimensional points corresponding to the pixels with non-zero depth of field values ​​form a facial point cloud of the target face in the viewing direction.

11. The method according to claim 1, characterized in that: The determining of the centroids corresponding to the facial point clouds of the target faces in the n viewing directions includes: For the face point cloud of the target face under any of the viewing angles, an average value of the spatial coordinates of all points in the face point cloud is obtained, and the average value is determined as the centroid of the face point cloud.

12. The method according to claim 1, characterized in that The determining, according to the centroid, the middle face point clouds of the target face in the n viewing directions comprises: For the facial point cloud of the target face in any of the viewing directions, the center of mass is subtracted from the spatial coordinates of each point in the facial point cloud to obtain an intermediate facial point cloud of the target face in the viewing direction.

13. The method according to claim 1, characterized in that The method of using the posture reference point cloud to register the remaining intermediate face point clouds of the target face in the n-1 viewing directions to obtain the conversion point clouds of the target face in the n-1 viewing directions includes: For the remaining middle face point cloud of the target face in any of the n-1 viewing directions, select m points from the remaining middle face point cloud; Using a preset KD-tree algorithm, find the nearest neighbor points corresponding to each of the m points from the posture reference point cloud; The m points and the nearest neighbor points corresponding to the m points form m pairs of nearest point pairs, and the m pairs of nearest point pairs form a nearest point pair set; Determine a target rotation matrix and a target offset vector according to the set of closest point pairs; The remaining intermediate face point cloud is transformed using the target rotation matrix and the target offset vector to obtain an initial transformed point cloud of the target face in the viewing direction; A conversion error between the posture reference point cloud and the initial conversion point cloud is determined, and a conversion point cloud of the target face in the viewing direction is determined based on the conversion error.

14. The method according to claim 13, characterized in that Determining a target rotation matrix and a target offset vector according to the set of closest point pairs includes: Inputting m pairs of closest point pairs in the set of closest point pairs into a preset matrix construction formula to obtain a construction matrix output by the preset matrix construction formula; Performing SVD decomposition on the construction matrix to obtain a first eigenvalue and a second eigenvalue, inputting the first eigenvalue and the second eigenvalue into a preset rotation matrix operation formula, and obtaining a target rotation matrix output by the preset rotation matrix operation formula; The target rotation matrix, the centroid of the posture reference point cloud and the centroid of the remaining intermediate face point cloud are input into a preset offset vector operation formula to obtain a target offset vector output by the preset offset vector operation formula.

15. The method according to claim 14, characterized in that The preset matrix construction formula includes: The W includes the construction matrix, the N H Including the number of closest point pairs in the closest point pair set, the q i and the p i Including the closest point pair in the set of closest point pairs; The preset rotation matrix operation formula includes: R=UV T ; The R includes the target rotation matrix, the U includes the first eigenvalue, and the V T including the second eigenvalue; The preset offset vector calculation formula includes: t=μ p -Rμ q ; The t includes the target offset vector, the μ p includes the center of mass of the posture reference point cloud, the R includes the target rotation matrix, the μ q Include the centroid of the remaining intermediate face point cloud.

16. The method according to claim 13, characterized in that The step of transforming the remaining intermediate face point cloud using the target rotation matrix and the target offset vector to obtain an initial transformed point cloud of the target face in the viewing direction includes: Inputting the target rotation matrix, the target offset vector and the remaining intermediate face point cloud into a point cloud conversion formula; Obtaining an initial conversion point cloud of the target face in the viewing direction output by the point cloud conversion formula, wherein the point cloud conversion formula includes: Q′={Rq+t|q∈Q}; The Q′ includes the initial conversion point cloud, the Q includes the remaining intermediate face point cloud, the t includes the target offset vector, and the R includes the target rotation matrix.

17. The method according to claim 13, characterized in that The determining of the conversion error between the posture reference point cloud and the initial conversion point cloud comprises: Inputting the posture reference point cloud and the initial conversion point cloud into a preset posture difference calculation formula; Obtaining a posture difference between the posture reference point cloud output by the preset posture difference calculation formula and the initial conversion point cloud, and determining the posture difference as a conversion error; Wherein, the preset posture difference calculation formula includes: The Q′ includes the initial conversion point cloud, and the P includes the posture reference point cloud.

18. The method according to claim 13, characterized in that Determining the conversion point cloud of the target face in the viewing direction according to the conversion error includes: If the conversion error is less than a preset first threshold, determining that the initial conversion point cloud is a conversion point cloud of the target face in the viewing direction; If the conversion error is not less than the preset first threshold, jump to the step of selecting m points from the remaining intermediate face point cloud until the number of iterations reaches the preset number, and output the conversion point cloud of the target face in the viewing direction.

19. The method according to claim 1, characterized in that The determining of the shape difference between the verification point cloud and a preset face reference point cloud includes: A chamfer distance between the verification point cloud and a preset human face reference point cloud is obtained, and the chamfer distance is determined as a shape difference between the verification point cloud and the preset human face reference point cloud.

20. A 3D face model reconstruction device, characterized in that: The device comprises: An image pair acquisition module, used to acquire an image pair of a target face in n viewing directions, wherein the image pair includes a color image and a depth image; An image determination module, configured to determine, for the image pair of the target face in any viewing angle, a target color image aligned with the depth image according to the color image and the depth image in the image pair; A region determination module, used to determine a facial region of a human face in the target color image; A matrix acquisition module, used for converting the target color image into a human face mask matrix according to the human face facial region; An image processing module, used to process the depth image using the face mask matrix to obtain a target depth image; A point cloud acquisition module, used for converting the target depth image into a face point cloud of the target face in the viewing direction; A point cloud determination module, used to determine a target face point cloud according to the face point clouds of the target face in the n viewing directions; A point cloud reconstruction module is used to perform surface reconstruction on the target face point cloud to obtain a three-dimensional face model; Wherein, the point cloud determination module specifically includes: A centroid determination submodule, used to determine the centroids corresponding to the facial point clouds of the target faces in the n viewing directions; A point cloud determination submodule, used to determine the middle face point cloud of the target face in the n viewing directions according to the centroid; A point cloud selection submodule, used to select a posture reference point cloud from the middle face point clouds of the target face under the n viewing directions, wherein the posture reference point cloud includes the middle face point cloud of the target face under the frontal viewing direction; A point cloud registration submodule, for registering the remaining intermediate face point clouds of the target face in the n-1 viewing directions using the posture reference point cloud, to obtain n-1 conversion point clouds of the target face in the viewing directions; A point cloud fusion submodule is used to fuse the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions using a preset face reference point cloud to obtain a target face point cloud; The point cloud fusion submodule specifically includes: A point cloud merging unit, used for merging the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain a complete face point cloud; A point cloud composition unit, used for selecting a plurality of points from the complete face point cloud to form a verification point cloud; A shape difference determination unit, used to determine the shape difference between the verification point cloud and a preset face reference point cloud; A point cloud fusion unit, configured to fuse the posture reference point cloud and the conversion point clouds of the target face in n-1 viewing directions according to the shape difference to obtain a target face point cloud; Wherein, the point cloud fusion unit specifically includes: a point cloud fusion subunit, configured to fuse the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions to obtain a target face point cloud if the shape difference is less than a preset second threshold; A jump subunit, configured to jump to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud if the shape difference is not less than the preset second threshold; Wherein, the point cloud fusion subunit is specifically used for: If a shape confirmation instruction is received from the user, the posture reference point cloud and the conversion point clouds of the target face in the n-1 viewing directions are fused to obtain a target face point cloud; If a shape error instruction is received from the user, the preset second threshold is adjusted, and the process jumps to the step of selecting a plurality of points from the complete face point cloud to form a verification point cloud.

21. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 19 when executing a program stored in a memory.

22. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

Images