Methods, apparatus, equipment, and storage media for three-dimensional reconstruction of two-dimensional faces

Abstract

This invention relates to the field of 3D reconstruction, and discloses a method, apparatus, device, and storage medium for 3D reconstruction of a 2D face. The method includes: receiving a 2D face image; performing mouth keypoint detection processing on the 2D face image based on a mediapipe neural network to obtain a 2D keypoint set; obtaining a 3D face image based on a pix2pix neural network; performing mouth keypoint detection processing on the 3D face image based on a mediapipe neural network to obtain a 3D keypoint set; extracting N pairs of 2D keypoints from the 2D keypoint set and N pairs of 3D keypoints from the 3D keypoint set, where N is a positive integer; obtaining the absolute value of the deviation according to a preset deviation algorithm; determining whether the absolute value of the deviation is less than an error threshold; if it is less than the error threshold, adding the 2D face image to the reconstructed 2D image training set; and inputting the images in the reconstructed 2D image training set into the pix2pix neural network for training processing to generate a new pix2pix neural network.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction, and more particularly to a method, apparatus, device, and storage medium for three-dimensional reconstruction of a two-dimensional face. Background Technology

[0002] Many existing algorithms exist for 3D reconstruction of 2D faces, but they all suffer from inaccurate lip measurements. For example, the mouth is closed in a 2D face, but the lips are open in the 3D reconstruction. This method utilizes strong pixel-to-pixel correspondences (transformations) to accurately convert 2D face photos into 3D faces.

[0003] This paper utilizes strong pix-to-pixel correspondences (transformations) to address the almost inevitable issue of inaccurate mouth closure in 3D face reconstruction from 2D images. This inaccuracy stems from a characteristic of neural networks during training: the loss function of neural networks considers the global loss, taking into account the overall quality of the reconstructed 3D image rather than the precision of a single mouth closure. Therefore, despite the existence of numerous existing 3D reconstruction techniques, all suffer from insufficient accuracy in mouth closure. Thus, a new technique is needed to address the problem of inaccurate mouth reconstruction in 3D face image reconstruction from 2D images. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that the reconstruction of the mouth in the current two-dimensional face image is not accurate enough when reconstructing it into a three-dimensional image.

[0005] The first aspect of this invention provides a method for three-dimensional reconstruction of a two-dimensional face, comprising the following steps:

[0006] A two-dimensional face image is received, and based on a preset mediapipe neural network, the mouth key point detection processing is performed on the two-dimensional face image to obtain a two-dimensional key point set;

[0007] Based on a pre-set pix2pix neural network, the two-dimensional face image is reconstructed into a three-dimensional face image.

[0008] Based on a pre-set mediapipe neural network, the mouth key point detection process is performed on the three-dimensional face image to obtain a three-dimensional key point set;

[0009] N pairs of two-dimensional key points are extracted from the two-dimensional key point set, and N pairs of three-dimensional key points are extracted from the three-dimensional key point set, where N is a positive integer;

[0010] According to the preset deviation algorithm, the deviation values ​​of the N pairs of two-dimensional key points and the N pairs of three-dimensional key points are calculated to obtain the absolute value of the deviation.

[0011] Determine whether the absolute value of the deviation is less than a preset error threshold;

[0012] If the error is less than a preset error threshold, the two-dimensional face image is added to a preset reconstructed two-dimensional image training set.

[0013] The images in the reconstructed 2D image training set are input into a preset pix2pix neural network for training processing to generate a new pix2pix neural network.

[0014] Optionally, in a first implementation of the first aspect of the present invention, the step of performing deviation value calculation processing on the N pairs of two-dimensional key points and the N pairs of three-dimensional key points according to a preset deviation algorithm to obtain the absolute value of the deviation includes:

[0015] Calculate the average number of pixels between each pair of two-dimensional keypoints in the N pairs of two-dimensional keypoints to obtain the two-dimensional closure value;

[0016] Calculate the average number of pixels between each pair of 3D keypoints in the N pairs of 3D keypoints to obtain the 3D closure value;

[0017] Calculate the absolute value of the difference between the two-dimensional closure value and the three-dimensional closure value to obtain the absolute value of the deviation.

[0018] Optionally, in a second implementation of the first aspect of the present invention, the step of extracting N pairs of two-dimensional key points from the two-dimensional key point set and extracting N pairs of three-dimensional key points from the three-dimensional key point set includes:

[0019] Based on a pre-set mediapipe neural network, N pairs of two-dimensional key points are extracted from the set of two-dimensional key points;

[0020] Based on a pre-set mediapipe neural network, N pairs of 3D key points are extracted from the 3D key point set according to the correspondence between the N pairs of 2D key points and the 3D face image.

[0021] Optionally, in a third implementation of the first aspect of the present invention, after inputting the images in the reconstructed two-dimensional image training set into a preset pix2pix neural network for training processing to generate a new pix2pix neural network, the method further includes:

[0022] Replace the preset pix2pix neural network with the new pix2pix neural network.

[0023] Optionally, in the fourth implementation of the first aspect of the present invention, after determining whether the absolute value of the deviation is less than a preset error threshold, and before inputting the images in the reconstructed 2D image training set into a preset pix2pix neural network for training processing to generate a new pix2pix neural network, the method further includes:

[0024] If the error is not less than a preset error threshold, the two-dimensional face image is added to a preset verification image set.

[0025] Optionally, in a fifth implementation of the first aspect of the present invention, after inputting the images in the reconstructed two-dimensional image training set into a preset pix2pix neural network for training processing to generate a new pix2pix neural network, the method further includes:

[0026] Based on the new pix2pix neural network, each image in the verification image set is subjected to 3D reconstruction processing to obtain a verification 3D image set;

[0027] According to the preset verification algorithm, the verification 3D image set is subjected to deviation analysis processing to obtain the analysis results;

[0028] When the analysis result is qualified, the new pix2pix neural network replaces the preset pix2pix neural network.

[0029] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing deviation analysis processing on the verification three-dimensional image set according to a preset verification algorithm to obtain the analysis result includes:

[0030] Based on a pre-set mediapipe neural network, M pairs of 2D key points and M pairs of 3D key points corresponding to the verification 3D image set are extracted, where M is a positive integer;

[0031] According to the preset deviation algorithm, the deviation values ​​of the M pairs of two-dimensional key points and the M pairs of three-dimensional key points are calculated to obtain the absolute value of the deviation.

[0032] If the absolute value of the deviation is less than the preset verification threshold, the analysis result is confirmed as a qualified result.

[0033] If the absolute value of the deviation is not less than the preset verification threshold, the analysis result will be confirmed as an unqualified result.

[0034] A second aspect of the present invention provides a three-dimensional reconstruction device for a two-dimensional face, the device comprising:

[0035] A two-dimensional detection module is used to receive a two-dimensional face image and perform mouth key point detection processing on the two-dimensional face image based on a preset mediapipe neural network to obtain a two-dimensional key point set.

[0036] The 3D reconstruction module is used to perform 3D reconstruction processing on the 2D face image based on a preset pix2pix neural network to obtain a 3D face image.

[0037] The three-dimensional detection module is used to perform mouth key point detection processing on the three-dimensional face image based on a preset mediapipe neural network to obtain a three-dimensional key point set.

[0038] An extraction module is used to extract N pairs of two-dimensional key points from the two-dimensional key point set and N pairs of three-dimensional key points from the three-dimensional key point set, where N is a positive integer;

[0039] The deviation calculation module is used to perform deviation value calculation on the N pairs of two-dimensional key points and the N pairs of three-dimensional key points according to the preset deviation algorithm to obtain the absolute value of the deviation.

[0040] The judgment module is used to determine whether the absolute value of the deviation is less than a preset error threshold;

[0041] The training set addition module is used to add the two-dimensional face image to a preset reconstructed two-dimensional image training set if the error is less than a preset error threshold.

[0042] The training module is used to input the images in the reconstructed 2D image training set into a preset pix2pix neural network for training processing to generate a new pix2pix neural network.

[0043] A third aspect of the present invention provides a three-dimensional reconstruction device for a two-dimensional face, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the three-dimensional reconstruction device for the two-dimensional face to perform the above-described three-dimensional reconstruction method for the two-dimensional face.

[0044] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described method for three-dimensional reconstruction of a two-dimensional face.

[0045] In this embodiment of the invention, the generated 3D reconstructed face image has a more accurate mouth closure degree and is extremely fast, achieving the 3D reconstruction process of a 2D face image with very light weight. Existing 3D reconstruction technologies require 0.05 seconds to generate a 3D face image, but pix2pix only requires 0.01 seconds. 3D face reconstruction has a wide range of applications, such as single-image speech, digital human representation, and video motion synchronization. Using our method, more accurate 3D faces can be generated, resulting in better effects for subsequent applications and solving the technical problem of insufficient accuracy in mouth reconstruction when reconstructing 3D images from 2D face images. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of an embodiment of the three-dimensional reconstruction method for a two-dimensional face in this invention.

[0047] Figure 2 A schematic diagram of the two-dimensional keypoint set of a mediepipe neural network;

[0048] Figure 3 A schematic diagram of constructing a 3D face image from a 2D face image;

[0049] Figure 4 A schematic diagram for extracting three pairs of two-dimensional key points from a two-dimensional key point set;

[0050] Figure 5 This is a schematic diagram of one embodiment of the three-dimensional reconstruction device for a two-dimensional face in this invention.

[0051] Figure 6 This is a schematic diagram of another embodiment of the three-dimensional reconstruction device for a two-dimensional face in this invention;

[0052] Figure 7 This is a schematic diagram of one embodiment of a three-dimensional reconstruction device for a two-dimensional face according to an embodiment of the present invention. Detailed Implementation

[0053] This invention provides a method, apparatus, device, and storage medium for three-dimensional reconstruction of a two-dimensional face.

[0054] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0055] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the three-dimensional reconstruction method for a two-dimensional face in this invention includes:

[0056] 101. Receive a two-dimensional face image, and perform mouth key point detection processing on the two-dimensional face image based on a preset mediapipe neural network to obtain a two-dimensional key point set;

[0057] In this embodiment, a mediapipe neural network model for facial landmark detection is used to detect the distance between the upper and lower lips of 3000 2D and 3D faces. The mediapipe neural network model is characterized by its speed, dense and accurate face detection points. To calculate the degree of lip closure, accurate and dense facial landmarks are needed; therefore, the mediapipe neural network was chosen.

[0058] You can refer to this. Figure 2 , Figure 2 This is a schematic diagram of the two-dimensional keypoint set of a mediepipe neural network. Figure 2 The key element is concentrated around the mouth of the face.

[0059] 102. Based on a pre-set pix2pix neural network, perform three-dimensional reconstruction processing on two-dimensional face images to obtain three-dimensional face images;

[0060] 103. Based on a pre-set mediapipe neural network, perform mouth key point detection processing on a 3D face image to obtain a 3D key point set;

[0061] In steps 102-103, you can refer to... Figure 3 , Figure 3 A schematic diagram illustrating the construction of a 3D face image from a 2D face image. Figure 3The image on the left is a selected 2D face image, while the image on the right is a reconstructed 3D face image. The 3D reconstruction processing scheme using the pix2pix neural network is similar to step 101, which also uses the mediapipe neural network to detect and process the key points of the 3D mouth to obtain a set of 3D key points.

[0062] 104. Extract N pairs of 2D keypoints from a 2D keypoint set and N pairs of 3D keypoints from a 3D keypoint set, where N is a positive integer;

[0063] In this embodiment, reference can be made to Figure 4 , Figure 4 This is a schematic diagram illustrating the extraction of three pairs of two-dimensional keypoints from a two-dimensional keypoint set. Figure 4 The three pairs of two-dimensional key points are from Figure 2 The two-dimensional keypoints are selected from the set. Similarly, three pairs of three-dimensional keypoints are extracted from the three-dimensional keypoint set to facilitate the next step of deviation analysis.

[0064] Further, perform the following steps at 104:

[0065] 1041. Based on a pre-built mediapipe neural network, extract N pairs of two-dimensional key points from a two-dimensional key point set;

[0066] 1042. Based on a pre-built mediapipe neural network, extract N pairs of 3D key points from the 3D key point set according to the correspondence between N pairs of 2D key points and 3D face images.

[0067] In steps 1041-1042, three pairs of two-dimensional keypoints and three pairs of three-dimensional keypoints are collected. Based on the reconstructed relational mapping content, the mediapipe neural network is used for extraction and processing. That is, the three pairs of two-dimensional keypoints and three pairs of three-dimensional keypoints are nodes that correspond to each other in the two-dimensional to three-dimensional transformation.

[0068] 105. Based on the preset deviation algorithm, perform deviation value calculation on N pairs of two-dimensional key points and N pairs of three-dimensional key points to obtain the absolute value of the deviation;

[0069] In this embodiment, deviation analysis is used to obtain the absolute value of the deviation by analyzing the distance difference between the open and closed states of the three pairs of two-dimensional key points and the three pairs of three-dimensional key points.

[0070] Furthermore, the following steps can be performed at 105:

[0071] 1051. Calculate the average number of pixels between each pair of two-dimensional keypoints in N pairs of two-dimensional keypoints to obtain the two-dimensional closure value;

[0072] 1052. Calculate the average number of pixels between each pair of 3D keypoints in N pairs of 3D keypoints to obtain the 3D closure value;

[0073] 1053. Calculate the absolute value of the difference between the two-dimensional closure value and the three-dimensional closure value to obtain the absolute value of the deviation.

[0074] In steps 1051-1053, the degree of closure is defined as follows: using the facial landmark detection in step 105, locate 3 points on the upper lip and 3 points on the lower lip. For example, if the coordinates of the first point on the upper lip are (50, 80) and the coordinates of the first point on the lower lip are (50, 89), then the vertical distance between these two points is 89 - 80 = 9 pixels. Figure 4 As shown in the key points, the average vertical distance between three pairs of upper and lower lip detection points is selected to measure the degree of mouth closure. For example, if the vertical distance of the first set of detection points is 10, the vertical distance of the second set is 15, and the vertical distance of the third set is 15, then the degree of mouth closure is: (10+15+15) / 3 = 10 pixels. The Mediepipe network is used to perform keypoint detection on 2D face photos, calculating the degree of mouth closure B for each 2D face. The Mediepipe network is used to perform keypoint detection on 3D face photos, calculating the degree of mouth closure C for each 2D face.

[0075] Definition of absolute deviation: Absolute deviation = |degree of closure B - degree of closure C|, where |*| is the absolute value.

[0076] 106. Determine whether the absolute value of the deviation is less than the preset error threshold;

[0077] In this embodiment, by calculating the error value, paired photos with similar degrees of mouth closure between the 3D and 2D faces are selected during the 3D reconstruction process. The error values ​​for the closure degrees of 3000 pairs of 2D and 3D face photos are calculated, and paired 2D and 3D photos with an error value less than 3 pixels are selected. If there are enough images, the threshold for the difference can be lowered to 1-2 pixels, resulting in a more accurate final result. However, the lower the threshold, the fewer paired images are obtained.

[0078] Furthermore, after 106 and before 108, the following steps can also be performed:

[0079] 1061. If the error is not less than the preset error threshold, the two-dimensional face image is added to the preset verification image set.

[0080] In this embodiment, the verification image set uses 2D images with inaccurate mouth closure in the 3D reconstruction. If the difference between the 3D mouth and the 2D mouth closure is greater than 3 pixels, the image is not in the training set of the pix2pix model.

[0081] 107. If the error is less than the preset error threshold, the two-dimensional face image is added to the preset reconstructed two-dimensional image training set.

[0082] 108. Input the images from the reconstructed 2D image training set into a pre-set pix2pix neural network for training processing to generate a new pix2pix neural network.

[0083] In steps 107-108, paired 2D and 3D images with a preset error threshold are used as the training set to train the pix2pix neural network model. Here's an introduction to the pix2pix neural network: when image A and image B have a strong correlation, the pix2pix neural network can transform image A into image B. Since our 3D faces are obtained by reconstructing 3D faces from 2D faces, they have a very strong correlation and pixel-level correspondence. Therefore, after training on the paired dataset, the pix2pix neural network can accurately transform 2D face photos into 3D face photos.

[0084] Furthermore, after step 108, perform the following steps:

[0085] 1081. Replace the preset pix2pix neural network with the new pix2pix neural network.

[0086] In this embodiment, the newly trained pix2pix neural network replaces the original pix2pix neural network, and the new pix2pix neural network is more accurate in constructing the mouth state of the 3D image.

[0087] Furthermore, under scheme 1061, after 108, the following steps can be performed:

[0088] 1082. Based on a new pix2pix neural network, perform 3D reconstruction processing on each image in the validation image set to obtain a validation 3D image set;

[0089] 1083. Based on the preset verification algorithm, perform deviation analysis on the verification 3D image set to obtain the analysis results;

[0090] 1084. When the analysis result is qualified, the new pix2pix neural network replaces the preset pix2pix neural network.

[0091] In steps 1082-1084, the original unsuccessfully reconstructed images are reconstructed into 3D images. Then, the deviation of each 3D image is analyzed. If the required value is met, the pre-trained pix2pix neural network replaces the original pix2pix neural network.

[0092] Furthermore, the following steps can be performed on 1083:

[0093] 10831. Based on a pre-built mediapipe neural network, extract M pairs of 2D key points and M pairs of 3D key points corresponding to the verification 3D image set and the verification image set, where M is a positive integer;

[0094] 10832. Based on the preset deviation algorithm, perform deviation value calculation on M pairs of two-dimensional key points and M pairs of three-dimensional key points to obtain the absolute value of the deviation;

[0095] 10833. When the absolute value of the deviation is less than the preset verification threshold, the analysis result is confirmed as a qualified result.

[0096] 10834. If the absolute value of the deviation is not less than the preset verification threshold, the analysis result will be confirmed as an unqualified result.

[0097] In steps 10831-10834, to avoid random errors from single images, we selected all 2D images from the 3000 images whose mouth closure was inaccurate during 3D reconstruction and used them as input to the pix2pix neural network. We found that the error value of the mouth closure in the 3D face optimized by pix2pix was generally smaller than that obtained by the 3D reconstruction method. We summed the error values ​​of each of the 3000 images to obtain the result.

[0098] The total error value obtained by the 3D reconstruction method is 5371.2, while the total error value of the pix2pix neural network optimization is 4156.4. The verification threshold is 4500, so the analysis result is confirmed as a qualified result.

[0099] In this embodiment of the invention, the generated 3D reconstructed face image has a more accurate mouth closure degree and is extremely fast, achieving the 3D reconstruction process of a 2D face image with very light weight. Existing 3D reconstruction technologies require 0.05 seconds to generate a 3D face image, but pix2pix only requires 0.01 seconds. 3D face reconstruction has a wide range of applications, such as single-image speech, digital human representation, and video motion synchronization. Using our method, more accurate 3D faces can be generated, resulting in better effects for subsequent applications and solving the technical problem of insufficient accuracy in mouth reconstruction when reconstructing 3D images from 2D face images.

[0100] The above describes the method for three-dimensional reconstruction of a two-dimensional face in an embodiment of the present invention. The following describes the apparatus for three-dimensional reconstruction of a two-dimensional face in an embodiment of the present invention. Please refer to [link / reference]. Figure 5 One embodiment of the three-dimensional reconstruction device for a two-dimensional face in this invention includes:

[0101] The two-dimensional detection module 501 is used to receive a two-dimensional face image and perform mouth key point detection processing on the two-dimensional face image based on a preset mediapipe neural network to obtain a two-dimensional key point set.

[0102] The 3D reconstruction module 502 is used to perform 3D reconstruction processing on the 2D face image based on a preset pix2pix neural network to obtain a 3D face image.

[0103] The three-dimensional detection module 503 is used to perform mouth key point detection processing on the three-dimensional face image based on a preset mediapipe neural network to obtain a three-dimensional key point set.

[0104] Extraction module 504 is used to extract N pairs of two-dimensional key points from the two-dimensional key point set and N pairs of three-dimensional key points from the three-dimensional key point set, where N is a positive integer;

[0105] The deviation calculation module 505 is used to perform deviation value calculation on the N pairs of two-dimensional key points and the N pairs of three-dimensional key points according to a preset deviation algorithm to obtain the absolute value of the deviation.

[0106] The judgment module 506 is used to determine whether the absolute value of the deviation is less than a preset error threshold.

[0107] The training set addition module 507 is used to add the two-dimensional face image to the preset reconstructed two-dimensional image training set if the error is less than a preset error threshold.

[0108] The training module 508 is used to input the images in the reconstructed 2D image training set into a preset pix2pix neural network for training processing to generate a new pix2pix neural network.

[0109] In this embodiment of the invention, the generated 3D reconstructed face image has a more accurate mouth closure degree and is extremely fast, achieving the 3D reconstruction process of a 2D face image with very light weight. Existing 3D reconstruction technologies require 0.05 seconds to generate a 3D face image, but pix2pix only requires 0.01 seconds. 3D face reconstruction has a wide range of applications, such as single-image speech, digital human representation, and video motion synchronization. Using our method, more accurate 3D faces can be generated, resulting in better effects for subsequent applications and solving the technical problem of insufficient accuracy in mouth reconstruction when reconstructing 3D images from 2D face images.

[0110] Please see Figure 6 Another embodiment of the three-dimensional reconstruction device for a two-dimensional face in this invention includes:

[0111] The two-dimensional detection module 501 is used to receive a two-dimensional face image and perform mouth key point detection processing on the two-dimensional face image based on a preset mediapipe neural network to obtain a two-dimensional key point set.

[0112] The 3D reconstruction module 502 is used to perform 3D reconstruction processing on the 2D face image based on a preset pix2pix neural network to obtain a 3D face image.

[0113] The three-dimensional detection module 503 is used to perform mouth key point detection processing on the three-dimensional face image based on a preset mediapipe neural network to obtain a three-dimensional key point set.

[0114] Extraction module 504 is used to extract N pairs of two-dimensional key points from the two-dimensional key point set and N pairs of three-dimensional key points from the three-dimensional key point set, where N is a positive integer;

[0115] The deviation calculation module 505 is used to perform deviation value calculation on the N pairs of two-dimensional key points and the N pairs of three-dimensional key points according to a preset deviation algorithm to obtain the absolute value of the deviation.

[0116] The judgment module 506 is used to determine whether the absolute value of the deviation is less than a preset error threshold.

[0117] The training set addition module 507 is used to add the two-dimensional face image to the preset reconstructed two-dimensional image training set if the error is less than a preset error threshold.

[0118] The training module 508 is used to input the images in the reconstructed 2D image training set into a preset pix2pix neural network for training processing to generate a new pix2pix neural network.

[0119] Specifically, the deviation calculation module 505 is used for:

[0120] Calculate the average number of pixels between each pair of two-dimensional keypoints in the N pairs of two-dimensional keypoints to obtain the two-dimensional closure value;

[0121] Calculate the average number of pixels between each pair of 3D keypoints in the N pairs of 3D keypoints to obtain the 3D closure value;

[0122] Calculate the absolute value of the difference between the two-dimensional closure value and the three-dimensional closure value to obtain the absolute value of the deviation.

[0123] Specifically, the extraction module 504 is used for:

[0124] Based on a pre-set mediapipe neural network, N pairs of two-dimensional key points are extracted from the set of two-dimensional key points;

[0125] Based on a pre-set mediapipe neural network, N pairs of 3D key points are extracted from the 3D key point set according to the correspondence between the N pairs of 2D key points and the 3D face image.

[0126] The three-dimensional reconstruction device for the two-dimensional face further includes a replacement module 509, which is specifically used for:

[0127] Replace the preset pix2pix neural network with the new pix2pix neural network.

[0128] The three-dimensional reconstruction device for the two-dimensional face further includes a verification set addition module 510, which is specifically used for:

[0129] If the error is not less than a preset error threshold, the two-dimensional face image is added to a preset verification image set.

[0130] The three-dimensional reconstruction device for the two-dimensional face further includes a verification module 511, which is specifically used for:

[0131] Based on the new pix2pix neural network, each image in the verification image set is subjected to 3D reconstruction processing to obtain a verification 3D image set;

[0132] According to the preset verification algorithm, the verification 3D image set is subjected to deviation analysis processing to obtain the analysis results;

[0133] When the analysis result is qualified, the new pix2pix neural network replaces the preset pix2pix neural network.

[0134] The verification module 511 can also be specifically used for:

[0135] Based on a pre-set mediapipe neural network, M pairs of 2D key points and M pairs of 3D key points corresponding to the verification 3D image set are extracted, where M is a positive integer;

[0136] According to the preset deviation algorithm, the deviation values ​​of the M pairs of two-dimensional key points and the M pairs of three-dimensional key points are calculated to obtain the absolute value of the deviation.

[0137] If the absolute value of the deviation is less than the preset verification threshold, the analysis result is confirmed as a qualified result.

[0138] If the absolute value of the deviation is not less than the preset verification threshold, the analysis result will be confirmed as an unqualified result.

[0139] In this embodiment of the invention, the generated 3D reconstructed face image has a more accurate mouth closure degree and is extremely fast, achieving the 3D reconstruction process of a 2D face image with very light weight. Existing 3D reconstruction technologies require 0.05 seconds to generate a 3D face image, but pix2pix only requires 0.01 seconds. 3D face reconstruction has a wide range of applications, such as single-image speech, digital human representation, and video motion synchronization. Using our method, more accurate 3D faces can be generated, resulting in better effects for subsequent applications and solving the technical problem of insufficient accuracy in mouth reconstruction when reconstructing 3D images from 2D face images.

[0140] above Figure 5 and Figure 6 The three-dimensional reconstruction device for a two-dimensional face in this embodiment of the invention is described in detail from the perspective of modular functional entities. The three-dimensional reconstruction device for a two-dimensional face in this embodiment of the invention is described in detail below from the perspective of hardware processing.

[0141] Figure 7 This is a schematic diagram of a three-dimensional reconstruction device for a two-dimensional face provided in an embodiment of the present invention. The three-dimensional reconstruction device 700 for a two-dimensional face can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 710 (e.g., one or more processors) and a memory 720, and one or more storage media 730 (e.g., one or more mass storage devices) storing application programs 733 or data 732. The memory 720 and storage media 730 can be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the three-dimensional reconstruction device 700 for the two-dimensional face. Furthermore, the processor 710 may be configured to communicate with the storage media 730 and execute the series of instruction operations in the storage media 730 on the three-dimensional reconstruction device 700 for the two-dimensional face.

[0142] The 3D reconstruction device 700 based on 2D facial features may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input / output interfaces 760, and / or one or more operating systems 731, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 7 The illustrated structure of a 3D reconstruction device for a 2D face does not constitute a limitation on 3D reconstruction devices based on 2D faces. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0143] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the three-dimensional reconstruction method of the two-dimensional face.

[0144] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for three-dimensional reconstruction of a two-dimensional face, characterized in that, Including the following steps: A two-dimensional face image is received, and based on a preset mediapipe neural network, the mouth key point detection processing is performed on the two-dimensional face image to obtain a two-dimensional key point set; Based on a pre-set pix2pix neural network, the two-dimensional face image is reconstructed into a three-dimensional face image. Based on a pre-set mediapipe neural network, the mouth key point detection process is performed on the three-dimensional face image to obtain a three-dimensional key point set; N pairs of two-dimensional key points are extracted from the two-dimensional key point set, and N pairs of three-dimensional key points are extracted from the three-dimensional key point set, where N is a positive integer; According to the preset deviation algorithm, the deviation values ​​of the N pairs of two-dimensional key points and the N pairs of three-dimensional key points are calculated to obtain the absolute value of the deviation. Determine whether the absolute value of the deviation is less than a preset error threshold; If the error is less than a preset error threshold, the two-dimensional face image is added to a preset reconstructed two-dimensional image training set. The images in the reconstructed 2D image training set are input into a preset pix2pix neural network for training processing to generate a new pix2pix neural network; The step of performing deviation value calculation on the N pairs of two-dimensional key points and the N pairs of three-dimensional key points according to the preset deviation algorithm to obtain the absolute value of the deviation includes: Calculate the average number of vertically spaced pixels corresponding to each pair of two-dimensional key points in the N pairs of two-dimensional key points to obtain the two-dimensional closure value. The N pairs of two-dimensional key points include: N pairs of two-dimensional upper and lower lip detection points. The average number of vertically spaced pixels corresponding to each pair of three-dimensional key points in the N pairs of three-dimensional key points is calculated to obtain the three-dimensional closure value. The N pairs of three-dimensional key points include: N pairs of three-dimensional upper and lower lip detection points. Calculate the absolute value of the difference between the two-dimensional closure value and the three-dimensional closure value to obtain the absolute value of the deviation.

2. The method for three-dimensional reconstruction of a two-dimensional face according to claim 1, characterized in that, The step of extracting N pairs of two-dimensional key points from the two-dimensional key point set and N pairs of three-dimensional key points from the three-dimensional key point set includes: Based on a pre-set mediapipe neural network, N pairs of two-dimensional key points are extracted from the set of two-dimensional key points; Based on a pre-set mediapipe neural network, N pairs of 3D key points are extracted from the 3D key point set according to the correspondence between the N pairs of 2D key points and the 3D face image.

3. The method for three-dimensional reconstruction of a two-dimensional face according to claim 1, characterized in that, After inputting the images from the reconstructed 2D image training set into a preset pix2pix neural network for training to generate a new pix2pix neural network, the process further includes: Replace the preset pix2pix neural network with the new pix2pix neural network.

4. The method for three-dimensional reconstruction of a two-dimensional face according to claim 1, characterized in that, After determining whether the absolute value of the deviation is less than a preset error threshold, and before inputting the images from the reconstructed 2D image training set into a preset pix2pix neural network for training to generate a new pix2pix neural network, the method further includes: If the error is not less than a preset error threshold, the two-dimensional face image is added to a preset verification image set.

5. The method for three-dimensional reconstruction of a two-dimensional face according to claim 4, characterized in that, After inputting the images from the reconstructed 2D image training set into a preset pix2pix neural network for training to generate a new pix2pix neural network, the process further includes: Based on the new pix2pix neural network, each image in the verification image set is subjected to 3D reconstruction processing to obtain a verification 3D image set; According to the preset verification algorithm, the verification 3D image set is subjected to deviation analysis processing to obtain the analysis results; When the analysis result is qualified, the new pix2pix neural network replaces the preset pix2pix neural network.

6. The method for three-dimensional reconstruction of a two-dimensional face according to claim 5, characterized in that, The step of performing deviation analysis on the verification 3D image set according to the preset verification algorithm, and obtaining the analysis results include: Based on a pre-set mediapipe neural network, M pairs of 2D key points and M pairs of 3D key points corresponding to the verification 3D image set are extracted, where M is a positive integer; According to the preset deviation algorithm, the deviation values ​​of the M pairs of two-dimensional key points and the M pairs of three-dimensional key points are calculated to obtain the absolute value of the deviation. If the absolute value of the deviation is less than the preset verification threshold, the analysis result is confirmed as a qualified result. If the absolute value of the deviation is not less than the preset verification threshold, the analysis result will be confirmed as an unqualified result.

7. A three-dimensional reconstruction device for a two-dimensional face, characterized in that, The three-dimensional reconstruction device for the two-dimensional face includes: A two-dimensional detection module is used to receive a two-dimensional face image and perform mouth key point detection processing on the two-dimensional face image based on a preset mediapipe neural network to obtain a two-dimensional key point set. The 3D reconstruction module is used to perform 3D reconstruction processing on the 2D face image based on a preset pix2pix neural network to obtain a 3D face image. The three-dimensional detection module is used to perform mouth key point detection processing on the three-dimensional face image based on a preset mediapipe neural network to obtain a three-dimensional key point set. An extraction module is used to extract N pairs of two-dimensional key points from the two-dimensional key point set and N pairs of three-dimensional key points from the three-dimensional key point set, where N is a positive integer; The deviation calculation module is used to perform deviation value calculation on the N pairs of two-dimensional key points and the N pairs of three-dimensional key points according to the preset deviation algorithm to obtain the absolute value of the deviation. The judgment module is used to determine whether the absolute value of the deviation is less than a preset error threshold; The training set addition module is used to add the two-dimensional face image to a preset reconstructed two-dimensional image training set if the error is less than a preset error threshold. The training module is used to input the images in the reconstructed 2D image training set into a preset pix2pix neural network for training processing to generate a new pix2pix neural network. Specifically, the deviation calculation module is used for: Calculate the average number of vertically spaced pixels corresponding to each pair of two-dimensional key points in the N pairs of two-dimensional key points to obtain the two-dimensional closure value. The N pairs of two-dimensional key points include: N pairs of two-dimensional upper and lower lip detection points. The average number of vertically spaced pixels corresponding to each pair of three-dimensional key points in the N pairs of three-dimensional key points is calculated to obtain the three-dimensional closure value. The N pairs of three-dimensional key points include: N pairs of three-dimensional upper and lower lip detection points. Calculate the absolute value of the difference between the two-dimensional closure value and the three-dimensional closure value to obtain the absolute value of the deviation.

8. A three-dimensional reconstruction device for a two-dimensional face, characterized in that, The 3D reconstruction device for the 2D face includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; The at least one processor invokes the instructions in the memory to cause the three-dimensional reconstruction device of the two-dimensional face to perform the three-dimensional reconstruction method of the two-dimensional face as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional reconstruction method for a two-dimensional face as described in any one of claims 1-6.

Images