Three-dimensional reconstruction method, device, electronic device and readable storage medium
The method enhances monocular face 3D reconstruction by predicting 3DMM parameters and generating target color and normal maps using neural networks, addressing output discrepancies and reducing labor costs without true texture mapping.
Patent Information
- Application Number
- CN202310336781.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-03-28
AI Technical Summary
Existing monocular face 3D reconstruction technologies often produce outputs that differ significantly from the visual effect of the given 2D image, and require manual true texture mapping, leading to increased labor costs and burdens.
A method involving a first neural network to predict 3DMM parameters, rearrange pixel colors based on these parameters, and a second neural network to generate a target color and normal map, utilizing complementary information without relying on true texture maps.
Improves the 3D reconstruction effect and similarity with the input image while reducing manpower costs by eliminating the need for manual true texture mapping.
Smart Images

Figure CN116894911B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional reconstruction method, device, electronic equipment and readable storage medium. Background Art
[0002] At present, given an unconstrained single 2D image containing a face, the task of monocular 3D reconstruction of a face is to generate a 3D model and texture map of the face. In recent years, this technology has developed rapidly and has been widely used in many fields such as virtual human production and game modeling, with broad application prospects and huge market value.
[0003] Among the existing monocular face 3D reconstruction technologies, the output of a single network for monocular face 3D reconstruction is quite different from the visual effect of a given two-dimensional image; the output of multiple networks for monocular face 3D reconstruction requires supervision by true textures, and the real texture mapping relies on manual processing by staff, which greatly increases the labor cost and burden. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a three-dimensional reconstruction method, device, electronic device and readable storage medium, which can effectively improve the three-dimensional reconstruction effect without relying on true texture mapping, and significantly reduce the manpower cost and burden required for single three-dimensional reconstruction.
[0005] In a first aspect, an embodiment of the present invention provides a three-dimensional reconstruction method, comprising: obtaining a facial image to be reconstructed; predicting first 3DMM parameters corresponding to the facial image through a first target neural network obtained in advance; rearranging the color value of each pixel in the facial image based on the first 3DMM parameters to obtain a first color rearrangement map; and generating a first initial color map corresponding to the facial image based on the first 3DMM parameters; predicting a first target color map and a first target normal map corresponding to the facial image based on the first color rearrangement map and the first initial color map through a second target neural network obtained in advance, so as to perform three-dimensional reconstruction of the facial image based on the first target color map and the first target normal map; wherein the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage.
[0006] In a second aspect, an embodiment of the present invention further provides a three-dimensional reconstruction device, comprising: an image acquisition module, used to acquire a facial image to be reconstructed; a first prediction module, used to predict a first 3DMM parameter corresponding to the facial image through a first target neural network obtained in advance by training; a map generation module, used to rearrange the color value of each pixel in the facial image based on the first 3DMM parameter to obtain a first color rearrangement map; and, based on the first 3DMM parameter, generate a first initial color map corresponding to the facial image; a second prediction module, used to predict a first target color map and a first target normal map corresponding to the facial image based on the first color rearrangement map and the first initial color map through a second target neural network obtained in advance by training, so as to perform three-dimensional reconstruction of the facial image based on the first target color map and the first target normal map; wherein the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage.
[0007] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.
[0008] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.
[0009] A three-dimensional reconstruction method, device, electronic device and readable storage medium provided by an embodiment of the present invention first obtain a face image to be reconstructed, and then predict a first 3DMM parameter corresponding to the face image through a first target neural network obtained by pre-training, so as to rearrange the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map, and generate a first initial color map corresponding to the face image based on the first 3DMM parameter. Finally, a first target color map and a first target normal map corresponding to the face image are predicted based on the first color rearrangement map and the first initial color map through a second target neural network obtained by pre-training, so as to perform three-dimensional reconstruction of the face image based on the first target color map and the first target normal map; wherein the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage. The above method first uses the first target neural network to predict the first 3DMM parameters, thereby generating a first initial color screenshot based on the first 3DMM parameters, and rearranging the color value of each pixel in the face image based on the first 3DMM parameters to obtain a first color rearrangement map, and further uses the second target neural network to use the complementary information between the first initial color map and the first color rearrangement map to predict the first target color map and the first target normal map required for three-dimensional reconstruction. By combining the first target color map and the first target normal map, the visual effect of the three-dimensional reconstruction result can be significantly improved, and the similarity between the three-dimensional reconstruction result and the face image can also be effectively improved. In addition, in the embodiment of the present invention, the second target neural network is obtained by training based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage. There is no need to obtain the corresponding true value texture map in the training stage. Therefore, the embodiment of the present invention can also effectively improve the three-dimensional reconstruction effect without relying on the true value texture map, and significantly reduce the manpower cost and burden required for three-dimensional reconstruction.
[0010] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.
[0011] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0013] Figure 1 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention;
[0014] Figure 2 A schematic flow chart of another three-dimensional reconstruction method provided by an embodiment of the present invention;
[0015] Figure 3 A schematic diagram of a flow chart of a training method for a first target neural network provided by an embodiment of the present invention;
[0016] Figure 4 A schematic diagram of input and output of a second initial neural network provided by an embodiment of the present invention;
[0017] Figure 5 A schematic flow chart of a training method for a second target neural network provided by an embodiment of the present invention;
[0018] Figure 6 A schematic diagram of the structure of a three-dimensional reconstruction device provided by an embodiment of the present invention;
[0019] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0021] At present, the output result of the monocular face 3D reconstruction scheme using a single network is quite different from the visual effect of the given 2D image; the monocular face 3D reconstruction scheme using multiple networks requires true value texture supervision, so the accuracy of the true texture map will affect the effect of 3D reconstruction. If the true texture map relies on manual processing by staff, it will greatly increase the labor cost and burden, and if it relies on machine processing, it lacks accuracy. Based on this, the present invention implements a 3D reconstruction method, device, electronic device and readable storage medium, which can effectively improve the 3D reconstruction effect without relying on true value texture mapping, and significantly reduce the labor cost and burden required for 3D reconstruction.
[0022] To facilitate understanding of this embodiment, a three-dimensional reconstruction method disclosed in an embodiment of the present invention is first described in detail. All steps of the method are executed by a terminal or a server. Figure 1 The flowchart of a three-dimensional reconstruction method shown in FIG. 1 mainly includes the following steps S102 to S108:
[0023] Step S102, obtaining a facial image to be reconstructed. The facial image may be a two-dimensional image containing a face, which may also be referred to as a monocular facial image. In one embodiment, the facial image may be stored in a designated area, so that the facial image to be reconstructed is read in the designated area; or an upload channel may be provided for the user, so that the facial image uploaded by the user through the upload channel is obtained.
[0024] Step S104, predicting the first 3DMM parameters corresponding to the face image through the first target neural network obtained by pre-training. Among them, the input of the first target neural network is the face image, and the output is the first 3DMM parameters. The first 3DMM (3DMorphable Model, three-dimensional deformable model) parameters are also the 3DMM parameters predicted by the first target neural network in the model application stage. The 3DMM parameters may include one or more of the 3DMM shape coefficient, 3DMM expression coefficient, 3DMM texture coefficient, three-dimensional model posture, and illumination coefficient. In one embodiment, the face image is input into the first target neural network to obtain the corresponding first 3DMM parameters.
[0025] Step S106, based on the first 3DMM parameter, the color value of each pixel in the face image is rearranged to obtain a first color rearrangement map; and, based on the first 3DMM parameter, a first initial color map corresponding to the face image is generated. In one embodiment, a first face three-dimensional model can be generated based on the first 3DMM parameter, and then after aligning the first face three-dimensional model with the face image, the color value of each pixel in the face image is rearranged using the map coordinates provided by the first face three-dimensional model to obtain the first color rearrangement map. In addition, the texture basis provided by the 3DMM (that is, the 3DMM texture basis) can be linearly added based on the first 3DMM parameter to obtain the first initial color map. The first color rearrangement map, the first initial color map, and the first face three-dimensional model are the color rearrangement map, the initial color map, and the face three-dimensional model in the model application stage, respectively.
[0026] Step S108, using the pre-trained second target neural network, based on the first color rearrangement map and the first initial color map, predict the first target color map and the first target normal map corresponding to the face image, so as to perform three-dimensional reconstruction of the face image based on the first target color map and the first target normal map; wherein the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage, the input of the second target neural network is the first spliced map obtained by channel splicing of the first color rearrangement map and the first initial color map, and the output is the first target color map and the first target normal map, the first spliced map, the first target color map, and the first target normal map are respectively the spliced map, the target color map, and the target normal map in the model application stage. In one embodiment, the first spliced map is input into the second target neural network to obtain the first target color map and the first target normal map.
[0027] The three-dimensional reconstruction method provided in the embodiment of the present invention first uses a first target neural network to predict a first 3DMM parameter, thereby generating a first initial color screenshot based on the first 3DMM parameter, and rearranging the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map, and further predicts the first target color map and the first target normal map required for three-dimensional reconstruction by using the complementary information between the first initial color map and the first color rearrangement map through a second target neural network. By combining the first target color map and the first target normal map, the visual effect of the three-dimensional reconstruction result can be significantly improved, and the similarity between the three-dimensional reconstruction result and the face image can also be effectively improved. In addition, the second target neural network in the embodiment of the present invention is obtained by training based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage. There is no need to obtain the corresponding true value texture map in the training stage. Therefore, the embodiment of the present invention can also effectively improve the three-dimensional reconstruction effect without relying on the true value texture map, and significantly reduce the manpower cost and burden required for three-dimensional reconstruction.
[0028] To facilitate understanding of the aforementioned step S106, an embodiment of the present invention provides an implementation method for rearranging the color value of each pixel in a face image based on a first 3DMM parameter to obtain a first color rearrangement map, including: (1) generating a first face three-dimensional model corresponding to the face image according to the 3DMM shape coefficient and the 3DMM expression coefficient in the first 3DMM parameter; (2) aligning the face image and the first face three-dimensional model according to the three-dimensional model posture in the first 3DMM parameter; (3) when the face image is aligned with the first face three-dimensional model, rearranging the color value of each pixel in the face image according to the mapping coordinates of the first face three-dimensional model to obtain a first color rearrangement map. The mapping coordinates are a coordinate form used to align the face image and the first face three-dimensional model, which provides a two-dimensional coordinate corresponding to each triangular face of the first face three-dimensional model on the face image, and the mapping coordinates are fixed and can be provided by 3DMM. In one embodiment, a three-dimensional face model S can be generated based on the 3DMM shape coefficient and the 3DMM expression coefficient, and the three-dimensional face model S and the input face image can be aligned using the three-dimensional model posture, and the face pixel colors in the face image are arranged according to the map coordinates provided by the three-dimensional face model S to obtain a generated first color rearrangement map.
[0029] To facilitate understanding of the aforementioned step S106, an embodiment of the present invention further provides an implementation method for generating a first initial color map corresponding to a face image based on a first 3DMM parameter, and a preset 3DMM texture base can be linearly added according to the 3DMM texture coefficient in the first 3DMM parameter to obtain the first initial color map corresponding to the face image. In one implementation, the 3DMM texture base includes multiple base textures, each base texture is a map with color, and the weighted sum of all base textures is calculated according to the corresponding 3DMM texture coefficients to obtain the first initial color map.
[0030] To facilitate understanding of the aforementioned step S108, an embodiment of the present invention further provides an implementation method of predicting a first target color map and a first target normal map corresponding to a facial image based on a first color rearrangement map and a first initial color map through a second target neural network obtained through pre-training. Specifically, the first color rearrangement map and the first initial color map are channel-spliced to obtain a first spliced map, and then the first target color map and the first target normal map corresponding to the facial image are predicted based on the first spliced map through a second target neural network obtained through pre-training. In one embodiment, the first color rearrangement map and the first initial color map are both three-channel images, including three channels of RGB (Red, Green, Blue). A six-channel image can be obtained by channel stitching the first color rearrangement map and the first initial color map. The six-channel image is the first stitched image. Optionally, the first three channels of the first stitched image are the RGB channels of the first color rearrangement map, and the last three channels are the RGB channels of the first initial color map. The first barren image is input into the second target neural network to obtain the first target color map and the first target normal map.
[0031] Based on the above embodiments, the present invention provides another 3D reconstruction method. Figure 2 FIG. 1 is a flow chart of another 3D reconstruction method, the method comprising the following steps S202 to S218:
[0032] Step S202, loading the first neural network model and the second neural network model. In one embodiment, before performing 3D reconstruction, the parameters of the first neural network model and the second neural network model need to be preloaded to facilitate subsequent 3D reconstruction using the first neural network model and the second neural network model.
[0033] Step S204, input any face image.
[0034] Step S206: predicting the first 3DMM parameters of the face image by the first target neural network.
[0035] Step S208, generating a first human face three-dimensional model according to the first 3DMM parameters.
[0036] Step S210, aligning the first three-dimensional face model with the face image.
[0037] Step S212, arranging pixels in the face image according to the mapping coordinates provided by the first three-dimensional face model to obtain a first color rearranged map.
[0038] Step S124, generating a first initial color map according to the first 3DMM parameters.
[0039] Step S216, splicing the first color rearrangement map and the first initial color map by channel, and predicting the first target color map and the first target normal map by the second target neural network.
[0040] Step S218, determine whether to end. If yes, end; if not, execute step S204.
[0041] For the first target neural network provided in the above embodiment, the embodiment of the present invention further provides a training method for the first target neural network, see the following steps 1.1 to 1.4:
[0042] Step 1.1, obtaining a training image set, wherein the training image set includes a plurality of two-dimensional face images.
[0043] Step 1.2, predicting the second 3DMM parameters corresponding to the two-dimensional face image through the first initial neural network. In one embodiment, for each two-dimensional face image I t , the two-dimensional face image I t By inputting it into the first initial neural network Net1, the second 3DMM parameters such as 3DMM shape coefficient, 3DMM expression coefficient, 3DMM texture coefficient, 3D model posture, and illumination coefficient output by the first initial neural network Net1 for the two-dimensional face image can be obtained. Among them, the second 3DMM parameters are also the 3DMM parameters predicted by the first initial neural network in the model training stage.
[0044] Step 1.3, based on the second 3DMM parameters, generate a second three-dimensional face model and a second initial color map corresponding to the two-dimensional face image. The second three-dimensional face model and the second initial color map are respectively the three-dimensional face model and the initial color map in the model training stage. In one embodiment, the three-dimensional face model S can be generated according to the 3DMM shape coefficient and the 3DMM expression coefficient, and the face color map T1 can be generated according to the 3DMM texture coefficient.
[0045] Step 1.4, determining a first target loss value based on the second face 3D model and the second initial color map, and using the first target loss value to train the first initial neural network to obtain a first target neural network. In a specific implementation, see steps 1.4.1 to 1.4.3 below:
[0046] Step 1.4.1, performing differentiable rendering on the second face 3D model and the second initial color map to obtain a first rendered image. In one embodiment, differentiable rendering can be used to obtain the first rendered image based on the face 3D model S and the second initial color map T1.
[0047] Step 1.4.2, calculating a first error value between the two-dimensional face image and the first rendered image, and calculating a second error value between the key point coordinates in the two-dimensional face image and the key point coordinates in the second three-dimensional face model. In one embodiment, a facial key point detection method (such as Mediapipe) can be used to extract the key point coordinates LM in the two-dimensional face image. t , and use the projection matrix to calculate the coordinates of the key points in the second face 3D model Calculate the two-dimensional face image I t With the first rendered image The first error value between the key point coordinates LM t With key point coordinates The second error value between .
[0048] Step 1.4.3, weighted sum the first error value and the second error value to obtain a first target loss value. In one embodiment, the weighted sum of the first error value and the second error value can be used as the final first target loss value, and the first target loss value is used to train the first initial neural network Net1.
[0049] Based on the above embodiments, the present invention provides another method for training a first target neural network. Figure 3 The flowchart of a training method of a first target neural network is shown, and the method includes the following steps S302 to S316:
[0050] Step S302: input any two-dimensional face image.
[0051] Step S304, predicting the second 3DMM parameters by the first initial neural network Net1.
[0052] Step S306: Generate a second rendered image through differentiable rendering.
[0053] Step S308, calculating the key point coordinates through the projection matrix.
[0054] Step S310, calculating a first target loss value.
[0055] Step S312, optimizing the network parameters of the first initial neural network Net1.
[0056] Step S314, determine whether the maximum number of iterations has been reached. If yes, execute step S316; if no, execute step S302.
[0057] Step S316, save network parameters.
[0058] For the second target neural network provided in the above embodiment, the embodiment of the present invention further provides a training method for the second target neural network, see the following steps 2.1 to 1.5:
[0059] Step 2.1, according to the second 3DMM parameters and the second human face three-dimensional model, rearrange the color value of each pixel in the human face two-dimensional image to obtain a second color rearrangement map. In one embodiment, the three-dimensional model posture in the second 3DMM parameters can be used to align the second human face three-dimensional model S and the human face two-dimensional image I t , the two-dimensional face image I t The face pixel colors in are arranged according to the map coordinates to obtain a second color rearranged map T2.
[0060] Step 2.2, channel stitching the second color rearrangement map and the second initial color map to obtain a second stitched map. In one embodiment, the 3DMM texture coefficients can be used to linearly add the 3DMM texture base to generate the second initial color map T1, and then the second initial color map T1 and the second color rearrangement map T2 are channel stitched to obtain the second stitched map.
[0061] Step 2.3, predicting a second target color map and a second target normal map corresponding to the two-dimensional face image based on the second stitched map through a second initial neural network. Figure 4 Schematic diagram of input and output of a second initial neural network shown in FIG. 1 , wherein the input of the second initial neural network is an image obtained by channel-wise splicing of the second color rearrangement map T2 and the second initial color map T1 (i.e., the second spliced map), and the output is a second target color map T F-D and the second target normal map T F-N .
[0062] Step 2.4, based on the second target color map and the second target normal map, determine the rendering loss value, the structure loss value, the symmetry loss value and the smoothness loss value. For ease of understanding, the embodiments of the present invention respectively propose implementation methods for determining the rendering loss value, the structure loss value, the symmetry loss value and the smoothness loss value, see the following methods 1 to 4 for details:
[0063] Method 1: Rendering loss value:
[0064] Step a1, performing differentiable rendering on the second target color map, the second target normal map and the second face 3D model to obtain a second rendered image. In one embodiment, differentiable rendering technology is used to transform the second target color map T F-D , second target normal map T F-N The second face 3D model S is rendered into a 2D image, which is also the second rendered image.
[0065] Step a2, using a preset absolute error loss function, to calculate a rendering loss value based on the two-dimensional face image and the second rendered image. With the input face 2D image I t Calculate the mean absolute error loss function The calculation result of the absolute error loss function is the rendering loss value L1.
[0066] Method 2: structural loss value, which includes color structural loss value and / or normal structural loss value:
[0067] Step b1, using a preset absolute error loss function, calculates a color structure loss value based on the second target color map and the second initial color map. The color structure loss value is also the structure loss value of the second target color map. In one embodiment, the second target color map T F-D Calculate the mean absolute error loss function L2 = MAE(T F-D ,T1), the calculation result of the absolute error loss function is the color structure loss value L2.
[0068] Step b2, using a preset absolute error loss function, calculates a normal structure loss value based on the second target normal map and the preset standard normal map. The normal structure loss value is also the structure loss value of the second target normal map. In one embodiment, the second target normal map T F-N With the standard normal map T N Calculate the mean absolute error loss function L3 = MAE(T F-N ,T N), the calculation result of the absolute error loss function is the normal structure loss value L3.
[0069] Method 3: Symmetry loss value. The symmetry loss value includes color symmetry loss value and / or normal symmetry loss value:
[0070] Step c1, smoothing filter processing is performed on the second target color map to obtain a filtered color map, and then the filtered color map is symmetrically inverted to obtain a flipped color map. In one embodiment, a Gaussian filter kernel can be used to filter the second target color map T F-D Perform smoothing filtering to obtain the filtered color map T smooth-F-D , and then the filtered color map T smooth-F-D Perform a left-right symmetrical inversion to obtain the flipped color map T smooth-F-D '.
[0071] Step c2, using a preset absolute error loss function, calculates a color symmetry loss value based on the filtered color map and the flipped color map. The color symmetry loss value is also the symmetry loss value of the second target color map. In one embodiment, the filtered color map T smooth-F-D And the flipped color map T smooth-F-D 'The mean absolute error loss function L4 = MAE(T smooth-F-D ,T smooth-F-D '), the calculation result of the absolute error loss function is the color symmetry loss value L4.
[0072] Step c3, smoothing and filtering the second target normal map to obtain a filtered normal map, and then symmetrically inverting the filtered normal map to obtain a flipped normal map. In one embodiment, a Gaussian filter can be used to filter the second target normal map T F-N Perform smoothing filtering to obtain the filtered normal map T smooth-F-N , and then the filtered normal map T smooth-F-N Perform a left-right symmetrical inversion to obtain the flipped normal map T smooth-F-N '.
[0073] Step c4, using a preset absolute error loss function, calculates a normal symmetry loss value based on the filtered normal map and the flipped normal map. The normal symmetry loss value is also the symmetry loss value of the second target normal map. In one embodiment, the filtered normal map T smooth-F-N And the flipped normal map T smooth-F-N 'The mean absolute error loss function L5 = MAE(T smooth-F-N ,T smooth-F-N '), the calculation result of the absolute error loss function is the normal symmetric loss value L5.
[0074] Method 4: Smoothing loss value. The smoothing loss value includes the total color smoothing loss value and / or the total normal smoothing loss value:
[0075] Step d1, shifting the second target color map according to at least one specified direction to obtain a shifted color map corresponding to each specified direction. The specified directions may include up, down, left, and right. In one embodiment, the second target color map T F-D Offset one pixel in each of the four directions, up, down, left, and right, to get the offset color map T F-D1 、T F-D2 、T F-D3 、T F-D4 .
[0076] Step d2, using a preset absolute error loss function, based on the offset color map corresponding to each specified direction and the second target color map, determine the color smoothing loss value corresponding to each specified direction, and determine the sum of the color smoothing loss values corresponding to each specified direction as the total color smoothing loss value. The total color smoothing loss value is also the total smoothing loss value of the second target color map. In one embodiment, the offset color map T F-D1 、T F-D2 、T F-D3 、T F-D4 The second target color map T before offset F-D Calculate the sum of the mean absolute error loss function L6 = MAE(T F-D1 ,T F-D )+MAE(T F-D2 ,T F-D )+MAE(T F-D3 ,T F-D )+MAE(T F-D4 ,T F-D ), the result is the total color smoothing loss value L6.
[0077] Step d3, performing an offset process on the second target normal map according to at least one specified direction to obtain an offset normal map corresponding to each specified direction. F-N Offset one pixel in each of the four directions, up, down, left, and right, to get the offset normal map T F-N1 、T F-N2 、T F-N3 、T F-N4 .
[0078] Step d4, using a preset absolute error loss function, based on the offset normal map corresponding to each specified direction and the second target normal map, determine the normal smoothing loss value corresponding to each specified direction, and determine the sum of the normal smoothing loss values corresponding to each specified direction as the normal smoothing total loss value. The normal smoothing total loss value is also the smoothing total loss value of the second target normal map. In one embodiment, the offset normal map T F-N1 、T F-N2 、T F-N3 、T F-N4 The second target normal map T before offset F-N Calculate the sum of the mean absolute error loss function L7 = MAE(T F-N1 ,T F-N )+MAE(T F-N2 ,T F-N )+MAE(T F-N3 ,T F-D N)+MAE(T F-N4 ,T F-N ), the result is the total loss value of normal smoothing L7.
[0079] Step 2.5, weighted sum one or more of the rendering loss value, structure loss value, symmetry loss value and smoothness loss value to obtain a second target loss value, and use the second target loss value to train the second initial neural network to obtain a second target neural network. In one embodiment, the rendering loss value L1, the color structure loss value L2, the normal structure loss value L3, the color symmetry loss value L4, the normal symmetry loss value L5, the color smoothness total loss value L6, and the normal smoothness total loss value L7 can be weighted summed to obtain the final second target loss value L F :
[0080] L F =w1*L1+w2*L2+w3*L3+w4*L4+w5*L5+w6*L6+w7*L7.
[0081] Wherein, w1, w2, w3, w4, w5, w6, and w7 are weights. In one embodiment,
[0082] Calculate the second target loss value L for each input 2D face image F And use the gradient descent algorithm to train the second initial neural network Net2.
[0083] Based on the above-mentioned embodiment, the present invention provides another training method for the second target neural network. Figure 5 The flowchart of a training method for a second target neural network is shown, and the method includes the following steps S502 to S524:
[0084] Step S502: input any two-dimensional face image.
[0085] Step S504, predicting the second 3DMM parameters by the first initial neural network Net1.
[0086] Step S506, generating a second human face three-dimensional model according to the second 3DMM parameters.
[0087] Step S508: align the second three-dimensional face model with the two-dimensional face image.
[0088] Step S510, arranging pixels in the two-dimensional face image according to the mapping coordinates provided by the second three-dimensional face model to obtain a second color rearranged map.
[0089] Step S512, generating a second initial color map according to the second 3DMM parameters.
[0090] Step S514, splicing the second color rearrangement map and the second initial color map by channel, and predicting the second target color map and the second target normal map by the second initial neural network Net2.
[0091] Step S516, calculating the rendering loss value.
[0092] Step S518, calculating the structure loss value / symmetry loss value / smoothness loss value.
[0093] Step S520, optimizing the network parameters of the second initial neural network Net2.
[0094] Step S522, determine whether the maximum number of iterations has been reached. If yes, execute step S526; if no, execute step S502.
[0095] Step S524, save the network parameters.
[0096] The three-dimensional reconstruction method provided by the embodiment of the present invention fully and adaptively explores and utilizes the complementary information between the first initial color map generated by linear addition of 3DMM and the first color rearrangement map of the input face image rearranged according to texture coordinates, designs an additional prior loss function, and significantly improves the similarity between the reconstructed three-dimensional face model and the texture map and the face image without relying on the true value texture map.
[0097] The core technical points of the 3D reconstruction method provided by the embodiment of the present invention are as follows:
[0098] (1) For each 2D face image I t First, the facial key point detection method is used to extract the facial two-dimensional key point coordinates LM t, use the first initial neural network Net1 to predict the 3DMM shape coefficient, 3DMM expression coefficient, 3DMM texture coefficient, 3D model posture, and illumination coefficient, calculate the generated face 3D model S, and use differentiable rendering to get the first rendered image of S Use the projection matrix to calculate the facial key point coordinates of the 3D face model S By calculating the key point coordinates With the key point coordinates LM t The pixel-by-pixel MSE error and the two-dimensional face image I t With the first rendered image The error, and the global sum are used as the first target loss value to train the first initial neural network Net1.
[0099] (2) After the first initial neural network Net1 is trained, the second initial neural network Net2 is trained: for each two-dimensional face image I t , use the trained first target neural network to predict the 3DMM shape coefficient, 3DMM expression coefficient, 3DMM texture coefficient and 3D model pose. Use the 3DMM texture coefficient to linearly add the texture basis provided by the 3DMM to generate the second initial color map T1, and use the 3D model pose to align the second face 3D model S and the face 2D image I t , the two-dimensional face image I t The face pixel colors in are arranged according to the map coordinates to obtain a second color rearranged map T2.
[0100] (3) The second initial color map T1 and the second color rearrangement map T2 are channel-concatenated as input, and the second target color map T2 is obtained through the second initial neural network Net2. F-D and the second target normal map T F-N Using differentiable rendering technology, the second target color map T F-D , second target normal map T F-N and the second human face 3D model S as a second rendered image And calculate the second rendered image and the 2D face image I t The error of the second target color map T F-D , second target normal map T F-N Perform smoothing and flipping operations, calculate the error between the maps before and after flipping, and then constrain the symmetry of the face map; calculate the second target color map T F-D The error between the second initial color map T1 and the second target color map T F-D Structural properties of; Calculate the second target normal map T F-N With the standard normal map T NThe error between them constrains the second target normal map T F-N Structural properties of; Calculate the second target color map T F-D , second target normal map T F-N The error between each pixel and its neighboring pixels constrains the second target color map T F-D , second target normal map T F-N The smoothness property of the above four error loss functions is taken as the weighted summation as the final loss function, and the second initial neural network Net2 is trained by the gradient descent algorithm.
[0101] In summary, the embodiments of the present invention can effectively improve the visual effect of the reconstruction result and the degree of similarity with the input image. In addition, the embodiments of the present invention can also avoid the reliance on the true value texture map in the neural network training process, thereby reducing the labor cost consumed in the true value texture map data preparation process.
[0102] For the 3D reconstruction method provided in the above embodiment, the present invention provides a 3D reconstruction device, see Figure 6 The schematic diagram of the structure of a 3D reconstruction device shown in FIG. 1 mainly includes the following parts:
[0103] An image acquisition module 602 is used to acquire a face image to be reconstructed;
[0104] A first prediction module 604, configured to predict first 3DMM parameters corresponding to the face image by using a first target neural network obtained through pre-training;
[0105] A map generation module 606 is used to rearrange the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map; and to generate a first initial color map corresponding to the face image based on the first 3DMM parameter;
[0106] The second prediction module 608 is used to predict the first target color map and the first target normal map corresponding to the face image based on the first color permutation map and the first initial color map through the second target neural network obtained in advance through training, so as to perform three-dimensional reconstruction of the face image based on the first target color map and the first target normal map; wherein, the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage.
[0107] The 3D reconstruction device provided by the embodiment of the present invention first uses the first target neural network to predict the first 3DMM parameter, thereby generating a first initial color screenshot based on the first 3DMM parameter, and rearranging the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map, and further predicts the first target color map and the first target normal map required for the monocular face 3D reconstruction by using the complementary information between the first initial color map and the first color rearrangement map through the second target neural network. By combining the first target color map and the first target normal map, the visual effect of the 3D reconstruction result can be significantly improved, and the similarity between the 3D reconstruction result and the face image can also be effectively improved. In addition, the second target neural network in the present invention is obtained by training based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage. There is no need to obtain the corresponding true value texture map in the training stage. Therefore, the embodiment of the present invention can also effectively improve the monocular face 3D reconstruction effect without relying on the true value texture map, and significantly reduce the manpower cost and burden required for the monocular face 3D reconstruction.
[0108] In one embodiment, the texture generation module 606 is also used to: generate a first three-dimensional face model corresponding to the face image based on the 3DMM shape coefficient and the 3DMM expression coefficient in the first 3DMM parameters; align the face image and the first three-dimensional face model according to the three-dimensional model posture in the first 3DMM parameters; when the face image is aligned with the first three-dimensional face model, rearrange the color value of each pixel in the face image according to the texture coordinates of the first three-dimensional face model to obtain a first color rearrangement texture.
[0109] In one implementation, the texture generation module 606 is further configured to: perform linear addition on a preset 3DMM texture basis according to the 3DMM texture coefficient in the first 3DMM parameter to obtain a first initial color texture corresponding to the face image.
[0110] In one embodiment, the second prediction module 608 is further used to: perform channel stitching on the first color rearrangement map and the first initial color map to obtain a first stitched map; and predict a first target color map and a first target normal map corresponding to the face image based on the first stitched map using a pre-trained second target neural network.
[0111] An embodiment of the present invention also provides a training device, which also includes a first training module, used to: obtain a training image set; wherein the training image set includes multiple two-dimensional face images; predict second 3DMM parameters corresponding to the two-dimensional face images through a first initial neural network; based on the second 3DMM parameters, generate a second three-dimensional face model and a second initial color map corresponding to the two-dimensional face image; determine a first target loss value based on the second three-dimensional face model and the second initial color map, and use the first target loss value to train the first initial neural network to obtain a first target neural network.
[0112] In one embodiment, the first training module is also used to: perform differentiable rendering processing on the second three-dimensional face model and the second initial color map to obtain a first rendered image; calculate a first error value between the two-dimensional face image and the first rendered image, and calculate a second error value between the key point coordinates in the two-dimensional face image and the key point coordinates in the second three-dimensional face model; and perform weighted summation of the first error value and the second error value to obtain a first target loss value.
[0113] In one embodiment, the training device further includes a second training module, which is used to: rearrange the color value of each pixel in the two-dimensional face image according to the second 3DMM parameter and the second three-dimensional face model to obtain a second color rearrangement map; perform channel splicing on the second color rearrangement map and the second initial color map to obtain a second spliced map; predict the second target color map and the second target normal map corresponding to the two-dimensional face image based on the second spliced map through the second initial neural network; determine the rendering loss value, the structural loss value, the symmetry loss value and the smoothness loss value based on the second target color map and the second target normal map; perform weighted summation on one or more of the rendering loss value, the structural loss value, the symmetry loss value and the smoothness loss value to obtain a second target loss value, and use the second target loss value to train the second initial neural network to obtain a second target neural network.
[0114] In one embodiment, the second training module is also used to: perform differentiable rendering processing on the second target color map, the second target normal map and the second face three-dimensional model to obtain a second rendered image; and use a preset absolute error loss function to calculate a rendering loss value based on the face two-dimensional image and the second rendered image.
[0115] In one embodiment, the second training module is also used to: calculate a color structure loss value based on the second target color map and the second initial color map using a preset absolute error loss function; and calculate a normal structure loss value based on the second target normal map and a preset standard normal map using a preset absolute error loss function; wherein the structure loss value includes a color structure loss value and / or a normal structure loss value.
[0116] In one embodiment, the second training module is also used to: perform smoothing filtering on the second target color map to obtain a filtered color map, and then perform symmetrical inversion processing on the filtered color map to obtain a flipped color map; use a preset absolute error loss function to calculate a color symmetry loss value based on the filtered color map and the flipped color map; and perform smoothing filtering on the second target normal map to obtain a filtered normal map, and then perform symmetrical inversion processing on the filtered normal map to obtain a flipped normal map; use a preset absolute error loss function to calculate a normal symmetry loss value based on the filtered normal map and the flipped normal map; wherein the symmetry loss value includes a color symmetry loss value and / or a normal symmetry loss value.
[0117] In one embodiment, the second training module is also used to: offset the second target color map according to at least one specified direction to obtain an offset color map corresponding to each specified direction; determine the color smoothing loss value corresponding to each specified direction based on the offset color map corresponding to each specified direction and the second target color map using a preset absolute error loss function, and determine the sum of the color smoothing loss values corresponding to each specified direction as the total color smoothing loss value; and, offset the second target normal map according to at least one specified direction to obtain an offset normal map corresponding to each specified direction; determine the normal smoothing loss value corresponding to each specified direction based on the offset normal map corresponding to each specified direction and the second target normal map using a preset absolute error loss function, and determine the sum of the normal smoothing loss values corresponding to each specified direction as the total normal smoothing loss value; wherein the smoothing loss value includes the total color smoothing loss value and / or the total normal smoothing loss value.
[0118] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0119] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program. When the computer program is run by the processor, it executes:
[0120] A three-dimensional reconstruction method includes: obtaining a face image to be reconstructed; predicting a first 3DMM parameter corresponding to the face image through a first target neural network obtained by pre-training; rearranging the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map; and generating a first initial color map corresponding to the face image based on the first 3DMM parameter; predicting a first target color map and a first target normal map corresponding to the face image based on the first color rearrangement map and the first initial color map through a second target neural network obtained by pre-training, so as to perform three-dimensional reconstruction of the face image based on the first target color map and the first target normal map; wherein the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage.
[0121] In one embodiment, based on the first 3DMM parameters, the color value of each pixel in the face image is rearranged to obtain a first color rearrangement map, including: generating a first three-dimensional face model corresponding to the face image according to the 3DMM shape coefficient and the 3DMM expression coefficient in the first 3DMM parameters; aligning the face image and the first three-dimensional face model according to the three-dimensional model posture in the first 3DMM parameters; when the face image is aligned with the first three-dimensional face model, rearranging the color value of each pixel in the face image according to the mapping coordinates of the first three-dimensional face model to obtain the first color rearrangement map.
[0122] In one embodiment, generating a first initial color map corresponding to a face image based on a first 3DMM parameter includes: linearly adding a preset 3DMM texture basis according to a 3DMM texture coefficient in the first 3DMM parameter to obtain a first initial color map corresponding to the face image.
[0123] In one embodiment, a first target color map and a first target normal map corresponding to a facial image are predicted based on a first color permutation map and a first initial color map by a pre-trained second target neural network, including: channel-splicing the first color permutation map and the first initial color map to obtain a first spliced map; and predicting the first target color map and the first target normal map corresponding to the facial image based on the first spliced map by a pre-trained second target neural network.
[0124] In one embodiment, a pre-training method for a first target neural network includes: obtaining a training image set; wherein the training image set includes a plurality of two-dimensional face images; predicting a second 3DMM parameter corresponding to the two-dimensional face image through a first initial neural network; generating a second three-dimensional face model and a second initial color map corresponding to the two-dimensional face image based on the second 3DMM parameter; determining a first target loss value based on the second three-dimensional face model and the second initial color map, and using the first target loss value to train the first initial neural network to obtain a first target neural network.
[0125] In one embodiment, determining a first target loss value based on a second three-dimensional face model and a second initial color map includes: performing differentiable rendering processing on the second three-dimensional face model and the second initial color map to obtain a first rendered image; calculating a first error value between the two-dimensional face image and the first rendered image, and calculating a second error value between the coordinates of key points in the two-dimensional face image and the coordinates of key points in the second three-dimensional face model; and performing weighted summation of the first error value and the second error value to obtain a first target loss value.
[0126] In one embodiment, the pre-training method of the second target neural network includes: rearranging the color value of each pixel in the two-dimensional face image according to the second 3DMM parameter and the second three-dimensional face model to obtain a second color rearrangement map; performing channel stitching on the second color rearrangement map and the second initial color map to obtain a second stitched map; predicting the second target color map and the second target normal map corresponding to the two-dimensional face image based on the second stitched map through the second initial neural network; determining the rendering loss value, the structural loss value, the symmetry loss value and the smoothness loss value based on the second target color map and the second target normal map; performing weighted summation on one or more of the rendering loss value, the structural loss value, the symmetry loss value and the smoothness loss value to obtain the second target loss value, and using the second target loss value to train the second initial neural network to obtain the second target neural network.
[0127] In one embodiment, a rendering loss value is determined based on a second target color map and a second target normal map, including: performing differentiable rendering processing on the second target color map, the second target normal map, and the second three-dimensional face model to obtain a second rendered image; and using a preset absolute error loss function to calculate the rendering loss value based on the two-dimensional face image and the second rendered image.
[0128] In one embodiment, a structural loss value is determined based on a second target color map and a second target normal map, including: using a preset absolute error loss function to calculate a color structure loss value based on the second target color map and the second initial color map; and, using a preset absolute error loss function to calculate a normal structure loss value based on the second target normal map and a preset standard normal map; wherein the structural loss value includes a color structure loss value and / or a normal structure loss value.
[0129] In one embodiment, a symmetry loss value is determined based on a second target color map and a second target normal map, including: performing smoothing filtering on the second target color map to obtain a filtered color map, and then performing symmetrical inversion processing on the filtered color map to obtain a flipped color map; using a preset absolute error loss function to calculate a color symmetry loss value based on the filtered color map and the flipped color map; and, performing smoothing filtering on the second target normal map to obtain a filtered normal map, and then performing symmetrical inversion processing on the filtered normal map to obtain a flipped normal map; using a preset absolute error loss function to calculate a normal symmetry loss value based on the filtered normal map and the flipped normal map; wherein the symmetry loss value includes a color symmetry loss value and / or a normal symmetry loss value.
[0130] In one embodiment, a smoothing loss value is determined based on a second target color map and a second target normal map, including: performing an offset processing on the second target color map according to at least one specified direction to obtain an offset color map corresponding to each specified direction; using a preset absolute error loss function, based on the offset color map corresponding to each specified direction and the second target color map, determining a color smoothing loss value corresponding to each specified direction, and determining the sum of the color smoothing loss values corresponding to each specified direction as a total color smoothing loss value; and performing an offset processing on the second target normal map according to at least one specified direction to obtain an offset normal map corresponding to each specified direction; using a preset absolute error loss function, based on the offset normal map corresponding to each specified direction and the second target normal map, determining a normal smoothing loss value corresponding to each specified direction, and determining the sum of the normal smoothing loss values corresponding to each specified direction as a total normal smoothing loss value; wherein the smoothing loss value includes a total color smoothing loss value and / or a total normal smoothing loss value.
[0131] The electronic device provided by the embodiment of the present invention first uses the first target neural network to predict the first 3DMM parameter, thereby generating a first initial color screenshot based on the first 3DMM parameter, and rearranging the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map, and further predicts the first target color map and the first target normal map required for monocular face 3D reconstruction by using the complementary information between the first initial color map and the first color rearrangement map through the second target neural network. By combining the first target color map and the first target normal map, the visual effect of the 3D reconstruction result can be significantly improved, and the similarity between the 3D reconstruction result and the face image can also be effectively improved. In addition, the second target neural network in the present invention is obtained by training based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage. There is no need to obtain the corresponding true value texture map in the training stage. Therefore, the embodiment of the present invention can also effectively improve the monocular face 3D reconstruction effect without relying on the true value texture map, and significantly reduce the manpower cost and burden required for monocular face 3D reconstruction.
[0132] Figure 7 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 70, a memory 71, a bus 72 and a communication interface 73, wherein the processor 70, the communication interface 73 and the memory 71 are connected via the bus 72; the processor 70 is used to execute an executable module stored in the memory 71, such as a computer program.
[0133] The memory 71 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 73 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0134] The bus 72 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0135] Among them, the memory 71 is used to store programs, and the processor 70 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 70 or implemented by the processor 70.
[0136] The processor 70 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 70. The above processor 70 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 71, and the processor 70 reads the information in the memory 71 and completes the steps of the above method in combination with its hardware.
[0137] The computer program product of a readable storage medium provided in an embodiment of the present invention includes a computer readable storage medium storing program code, and the instructions included in the program code can be used to execute:
[0138] A three-dimensional reconstruction method includes: obtaining a face image to be reconstructed; predicting a first 3DMM parameter corresponding to the face image through a first target neural network obtained by pre-training; rearranging the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map; and generating a first initial color map corresponding to the face image based on the first 3DMM parameter; predicting a first target color map and a first target normal map corresponding to the face image based on the first color rearrangement map and the first initial color map through a second target neural network obtained by pre-training, so as to perform three-dimensional reconstruction of the face image based on the first target color map and the first target normal map; wherein the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage.
[0139] In one embodiment, based on the first 3DMM parameters, the color value of each pixel in the face image is rearranged to obtain a first color rearrangement map, including: generating a first three-dimensional face model corresponding to the face image according to the 3DMM shape coefficient and the 3DMM expression coefficient in the first 3DMM parameters; aligning the face image and the first three-dimensional face model according to the three-dimensional model posture in the first 3DMM parameters; when the face image is aligned with the first three-dimensional face model, rearranging the color value of each pixel in the face image according to the mapping coordinates of the first three-dimensional face model to obtain the first color rearrangement map.
[0140] In one embodiment, generating a first initial color map corresponding to a face image based on a first 3DMM parameter includes: linearly adding a preset 3DMM texture basis according to a 3DMM texture coefficient in the first 3DMM parameter to obtain a first initial color map corresponding to the face image.
[0141] In one embodiment, a first target color map and a first target normal map corresponding to a facial image are predicted based on a first color permutation map and a first initial color map by a pre-trained second target neural network, including: channel-splicing the first color permutation map and the first initial color map to obtain a first spliced map; and predicting the first target color map and the first target normal map corresponding to the facial image based on the first spliced map by a pre-trained second target neural network.
[0142] In one embodiment, a pre-training method for a first target neural network includes: obtaining a training image set; wherein the training image set includes a plurality of two-dimensional face images; predicting a second 3DMM parameter corresponding to the two-dimensional face image through a first initial neural network; generating a second three-dimensional face model and a second initial color map corresponding to the two-dimensional face image based on the second 3DMM parameter; determining a first target loss value based on the second three-dimensional face model and the second initial color map, and using the first target loss value to train the first initial neural network to obtain a first target neural network.
[0143] In one embodiment, determining a first target loss value based on a second three-dimensional face model and a second initial color map includes: performing differentiable rendering processing on the second three-dimensional face model and the second initial color map to obtain a first rendered image; calculating a first error value between the two-dimensional face image and the first rendered image, and calculating a second error value between the coordinates of key points in the two-dimensional face image and the coordinates of key points in the second three-dimensional face model; and performing weighted summation of the first error value and the second error value to obtain a first target loss value.
[0144] In one embodiment, the pre-training method of the second target neural network includes: rearranging the color value of each pixel in the two-dimensional face image according to the second 3DMM parameter and the second three-dimensional face model to obtain a second color rearrangement map; performing channel stitching on the second color rearrangement map and the second initial color map to obtain a second stitched map; predicting the second target color map and the second target normal map corresponding to the two-dimensional face image based on the second stitched map through the second initial neural network; determining the rendering loss value, the structural loss value, the symmetry loss value and the smoothness loss value based on the second target color map and the second target normal map; performing weighted summation on one or more of the rendering loss value, the structural loss value, the symmetry loss value and the smoothness loss value to obtain the second target loss value, and using the second target loss value to train the second initial neural network to obtain the second target neural network.
[0145] In one embodiment, a rendering loss value is determined based on a second target color map and a second target normal map, including: performing differentiable rendering processing on the second target color map, the second target normal map, and the second three-dimensional face model to obtain a second rendered image; and using a preset absolute error loss function to calculate the rendering loss value based on the two-dimensional face image and the second rendered image.
[0146] In one embodiment, a structural loss value is determined based on a second target color map and a second target normal map, including: using a preset absolute error loss function to calculate a color structure loss value based on the second target color map and the second initial color map; and, using a preset absolute error loss function to calculate a normal structure loss value based on the second target normal map and a preset standard normal map; wherein the structural loss value includes a color structure loss value and / or a normal structure loss value.
[0147] In one embodiment, a symmetry loss value is determined based on a second target color map and a second target normal map, including: performing smoothing filtering on the second target color map to obtain a filtered color map, and then performing symmetrical inversion processing on the filtered color map to obtain a flipped color map; using a preset absolute error loss function to calculate a color symmetry loss value based on the filtered color map and the flipped color map; and, performing smoothing filtering on the second target normal map to obtain a filtered normal map, and then performing symmetrical inversion processing on the filtered normal map to obtain a flipped normal map; using a preset absolute error loss function to calculate a normal symmetry loss value based on the filtered normal map and the flipped normal map; wherein the symmetry loss value includes a color symmetry loss value and / or a normal symmetry loss value.
[0148] In one embodiment, a smoothing loss value is determined based on a second target color map and a second target normal map, including: performing an offset processing on the second target color map according to at least one specified direction to obtain an offset color map corresponding to each specified direction; using a preset absolute error loss function, based on the offset color map corresponding to each specified direction and the second target color map, determining a color smoothing loss value corresponding to each specified direction, and determining the sum of the color smoothing loss values corresponding to each specified direction as a total color smoothing loss value; and performing an offset processing on the second target normal map according to at least one specified direction to obtain an offset normal map corresponding to each specified direction; using a preset absolute error loss function, based on the offset normal map corresponding to each specified direction and the second target normal map, determining a normal smoothing loss value corresponding to each specified direction, and determining the sum of the normal smoothing loss values corresponding to each specified direction as a total normal smoothing loss value; wherein the smoothing loss value includes a total color smoothing loss value and / or a total normal smoothing loss value.
[0149] The readable storage medium provided by the embodiment of the present invention first uses the first target neural network to predict the first 3DMM parameter, thereby generating a first initial color screenshot based on the first 3DMM parameter, and rearranging the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement map, and further predicts the first target color map and the first target normal map required for monocular face 3D reconstruction by using the complementary information between the first initial color map and the first color rearrangement map through the second target neural network. By combining the first target color map and the first target normal map, the visual effect of the 3D reconstruction result can be significantly improved, and the similarity between the 3D reconstruction result and the face image can also be effectively improved. In addition, the second target neural network in the present invention is obtained by training based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage. There is no need to obtain the corresponding true value texture map in the training stage. Therefore, the embodiment of the present invention can also effectively improve the monocular face 3D reconstruction effect without relying on the true value texture map, and significantly reduce the manpower cost and burden required for monocular face 3D reconstruction.
[0150] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0151] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A three-dimensional reconstruction method, characterized in that: include: Obtaining a face image to be reconstructed; Predicting first 3DMM parameters corresponding to the face image through a pre-trained first target neural network; Based on the first 3DMM parameters, rearrange the color value of each pixel in the face image to obtain a first color rearrangement map; and, generating a first initial color map corresponding to the face image based on the first 3DMM parameters; Predicting a first target color map and a first target normal map corresponding to the face image based on the first color rearrangement map and the first initial color map by using a pre-trained second target neural network, so as to perform three-dimensional reconstruction of the face image based on the first target color map and the first target normal map; wherein the second target neural network is trained based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage; Using a pre-trained second target neural network, based on the first color rearrangement map and the first initial color map, predicting a first target color map and a first target normal map corresponding to the face image, including: channel-splicing the first color rearrangement map and the first initial color map to obtain a first spliced map; using a pre-trained second target neural network, based on the first spliced map, predicting a first target color map and a first target normal map corresponding to the face image.
2. The three-dimensional reconstruction method according to claim 1, characterized in that: Based on the first 3DMM parameter, the color value of each pixel in the face image is rearranged to obtain a first color rearrangement map, including: Generate a first three-dimensional face model corresponding to the face image according to the 3DMM shape coefficient and the 3DMM expression coefficient in the first 3DMM parameters; Performing alignment processing on the face image and the first face 3D model according to the 3D model posture in the first 3DMM parameters; When the face image is aligned with the first three-dimensional face model, the color value of each pixel in the face image is rearranged according to the mapping coordinates of the first three-dimensional face model to obtain a first color rearrangement map.
3. The three-dimensional reconstruction method according to claim 1, characterized in that: Generating a first initial color map corresponding to the face image based on the first 3DMM parameters includes: According to the 3DMM texture coefficients in the first 3DMM parameters, a preset 3DMM texture basis is linearly added to obtain a first initial color map corresponding to the face image.
4. The three-dimensional reconstruction method according to claim 1, characterized in that: The pre-training method of the first target neural network includes: Acquire a training image set; wherein the training image set includes a plurality of two-dimensional face images; Predicting second 3DMM parameters corresponding to the two-dimensional face image through a first initial neural network; Based on the second 3DMM parameters, generating a second three-dimensional face model and a second initial color map corresponding to the two-dimensional face image; A first target loss value is determined based on the second three-dimensional face model and the second initial color map, and the first target loss value is used to train the first initial neural network to obtain a first target neural network.
5. The three-dimensional reconstruction method according to claim 4, characterized in that: Determining a first target loss value based on the second human face 3D model and the second initial color map includes: Performing differentiable rendering processing on the second human face three-dimensional model and the second initial color map to obtain a first rendered image; Calculating a first error value between the two-dimensional face image and the first rendered image, and calculating a second error value between the coordinates of key points in the two-dimensional face image and the coordinates of key points in the second three-dimensional face model; A weighted sum is performed on the first error value and the second error value to obtain a first target loss value.
6. The three-dimensional reconstruction method according to claim 4, characterized in that: The pre-training method of the second target neural network includes: According to the second 3DMM parameters and the second three-dimensional face model, the color value of each pixel in the two-dimensional face image is rearranged to obtain a second color rearrangement map; Perform channel stitching on the second color rearrangement map and the second initial color map to obtain a second stitched map; Predicting, by a second initial neural network, a second target color map and a second target normal map corresponding to the two-dimensional face image based on the second stitched map; Determine a rendering loss value, a structure loss value, a symmetry loss value, and a smoothness loss value based on the second target color map and the second target normal map; A second target loss value is obtained by weighted summing one or more of the rendering loss value, the structure loss value, the symmetry loss value, and the smoothness loss value, and the second initial neural network is trained using the second target loss value to obtain a second target neural network.
7. The three-dimensional reconstruction method according to claim 6, characterized in that: Determining a rendering loss value based on the second target color map and the second target normal map includes: Performing differentiable rendering processing on the second target color map, the second target normal map, and the second three-dimensional face model to obtain a second rendered image; A rendering loss value is calculated based on the two-dimensional face image and the second rendered image using a preset absolute error loss function.
8. The three-dimensional reconstruction method according to claim 6, characterized in that: Determining a structure loss value based on the second target color map and the second target normal map includes: Calculating a color structure loss value based on the second target color map and the second initial color map using a preset absolute error loss function; And, using a preset absolute error loss function, calculating a normal structure loss value based on the second target normal map and a preset standard normal map; The structure loss value includes the color structure loss value and / or the normal structure loss value.
9. The three-dimensional reconstruction method according to claim 6, characterized in that: Determining a symmetry loss value based on the second target color map and the second target normal map includes: Performing smoothing filtering on the second target color map to obtain a filtered color map, and then performing symmetric inversion processing on the filtered color map to obtain a flipped color map; Calculating a color symmetry loss value based on the filtered color map and the flipped color map using a preset absolute error loss function; and performing smoothing filtering on the second target normal map to obtain a filtered normal map, and then performing symmetric inversion processing on the filtered normal map to obtain a flipped normal map; Calculating a normal symmetry loss value based on the filtered normal map and the flipped normal map using a preset absolute error loss function; The symmetry loss value includes the color symmetry loss value and / or the normal symmetry loss value.
10. The three-dimensional reconstruction method according to claim 6, characterized in that: Determining a smoothing loss value based on the second target color map and the second target normal map includes: Performing an offset process on the second target color map according to at least one specified direction to obtain an offset color map corresponding to each of the specified directions; Using a preset absolute error loss function, based on the offset color map corresponding to each of the specified directions and the second target color map, a color smoothing loss value corresponding to each of the specified directions is determined, and a sum of the color smoothing loss values corresponding to each of the specified directions is determined as a total color smoothing loss value; And, performing an offset process on the second target normal map according to at least one specified direction to obtain an offset normal map corresponding to each of the specified directions; Using a preset absolute error loss function, based on the offset normal map corresponding to each of the specified directions and the second target normal map, determine the normal smoothing loss value corresponding to each of the specified directions, and determine the sum of the normal smoothing loss values corresponding to each of the specified directions as the total normal smoothing loss value; The smoothing loss value includes the total color smoothing loss value and / or the total normal smoothing loss value.
11. A three-dimensional reconstruction device, characterized in that: include: An image acquisition module, used to acquire a face image to be reconstructed; A first prediction module, used to predict a first 3DMM parameter corresponding to the face image through a first target neural network obtained by pre-training; A texture generation module, configured to rearrange the color value of each pixel in the face image based on the first 3DMM parameter to obtain a first color rearrangement texture; and, generating a first initial color map corresponding to the face image based on the first 3DMM parameters; A second prediction module is used to predict a first target color map and a first target normal map corresponding to the face image based on the first color rearrangement map and the first initial color map by using a second target neural network obtained by pre-training, so as to perform three-dimensional reconstruction of the face image based on the first target color map and the first target normal map; wherein the second target neural network is obtained by training based on the second initial color map, the second target color map, the second target normal map and the preset standard normal map in the training stage; The second prediction module is specifically used to: perform channel stitching on the first color rearrangement map and the first initial color map to obtain a first stitched map; and predict a first target color map and a first target normal map corresponding to the face image based on the first stitched map using a pre-trained second target neural network.
12. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Image Style Transfer for Three-Dimensional Models
CN110084874A
Face image processing method and device
CN111951373A
Model head portrait creation method and device, electronic equipment and storage medium
CN112669447A
Three-dimensional face reconstruction method and device, electronic equipment and storage medium
CN112819947A
PBR three-dimensional reconstruction method and system based on deep learning, and computer storage medium
CN113345063A
