Human face replacement method based on three-dimensional reconstruction
By using 3D reconstruction and optimized facial parameters, combined with affine transformation and Poisson fusion techniques, the problem of high data volume and computational resource requirements of existing face-swapping methods is solved. This achieves high-quality replacement of any face with any face, maintaining consistency in expression and posture, and improving the clarity and stability of the replaced face.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LANZHOU FUTURE NEW FILM CULTURE & TECH GRP CO LTD
- Filing Date
- 2022-10-26
- Publication Date
- 2026-07-14
Smart Images
Figure CN115619937B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for replacing human faces based on three-dimensional reconstruction, belonging to the field of image processing technology. Background Technology
[0002] Face swapping is a long-standing and important topic in the field of computer vision. Face swapping refers to replacing a target face image with a desired source face image through image processing techniques. The replaced face has the expression and posture of the target face while retaining the texture features of the source face.
[0003] With the rapid development of deep learning theory, face-swapping methods based on generative adversarial networks (GANs) have become a relatively common face-swapping technology. By collecting face photos of the source and target faces with different poses, expressions, and lighting conditions, a training dataset is constructed to train a face replacement model from the source face to the target face.
[0004] Face-swapping methods based on deep learning generative adversarial networks still have certain limitations. A large amount of facial image data is required to train a relatively stable generative model, which is time-consuming and places certain demands on computer graphics cards. The trained model can only replace a specified source face with a specified target face; a single model cannot replace any face with any face. Due to the limitations of the input resolution of deep learning networks, the replaced face is relatively blurry, with significant loss of detail. Summary of the Invention
[0005] This invention addresses the shortcomings of existing technologies by providing a method for human face replacement based on three-dimensional reconstruction. The specific technical solution is as follows:
[0006] The method for human face replacement based on 3D reconstruction includes the following steps:
[0007] Step 1: 3D Reconstruction of Source Face and Target Face in Video
[0008] Input two-dimensional images of the source face and the target face, and use the 3DMM face 3D reconstruction method to solve the 3DMM parameters of the source face and the target face. The 3DMM parameters include shape parameters, expression parameters, and pose parameters.
[0009] Step 2: Calculate the unified solution for the 3DMM shape parameters of the face in the video.
[0010] Filter video frames containing the target face at a frontal angle, then filter the shape parameters of the video frames containing the target face at a frontal angle, and calculate an average shape as a unified solution for the video face shape parameters;
[0011] Step 3: Optimization of 3DMM expression and pose parameters
[0012] The expression and pose parameters solved by 3DMM are optimized by calculating the offset between the projected positions of key points selected in the source face 3D points and the detection positions of 68 key points of the source face.
[0013] Step 4: Combine and generate new 3DMM shapes
[0014] Obtain the shape parameters of the source face, obtain the expression parameters and pose parameters of the target face, and combine the shape parameters of the source face, the expression parameters and pose parameters of the target face to generate a 3DMM shape for the replacement face;
[0015] Step 5: Generate a new face image that incorporates the texture features of the source face image and the expression and pose of the target face image.
[0016] The 3D point cloud reconstructed from the source face is transformed to the camera coordinate system according to the pose parameters, and then projected onto the source image. By aligning the facial key points and translating them, a mapping relationship between the 3D points and the pixels of the source face image is established. Then, a mapping relationship between the 3D points of the combined model and the pixels of the target face image is established. For the pixel value of a certain position projected by the combined model onto the target face image, the corresponding face coordinates of the source image are calculated through affine transformation, and the pixel value of that position is calculated through bilinear interpolation.
[0017] Step Six: Face Image Fusion
[0018] For the generated new face image, the boundary of the face is obtained by using facial key points, the face region image inside the boundary is extracted and aligned to the target face image in the video, and Poisson fusion is used to fuse the face images.
[0019] As an improvement to the above technical solution, in step one, a video of the target face is input, the face detection extracts the face region image, and the 3DMM parameters of the current frame are calculated.
[0020] As an improvement to the above technical solution, in step one, the reconstruction representation formula used in the 3DMM face 3D reconstruction method is as follows:
[0021]
[0022] in, S represents the average face model. i α represents the principal component analysis part corresponding to the shape. i The coefficient representing the current shape, e i This represents the principal component analysis component corresponding to the facial expression, β. iThis represents the coefficient corresponding to the current expression; the three-dimensional shape of the current face image is generated by a linear combination of the average shape and m basic shapes, and the three-dimensional expression of the current face image is generated by a linear combination of the average expression and n basic expressions.
[0023] As an improvement to the above technical solution, step three involves optimizing the facial expression and pose parameters calculated by 3DMM. The error calculated by the objective function is the positional offset between the 68 key projection points selected in the 3D shape projection points and the 68 key facial points detected in the actual face. The optimization method uses the Levenberg-Marquardt method or the Gauss-Newton method.
[0024] As an improvement to the above technical solution, in step four, during the combination process, a combination model is generated. The generation of the combination model refers to obtaining the optimized shape parameter vector of the source face and the expression parameter vector of the target face respectively and combining them to generate a new set of face 3DMM feature vector coefficients.
[0025] As an improvement to the above technical solution, in step five, the 3D point cloud is transformed to the camera coordinate system, and the transformation formula is as follows:
[0026]
[0027] Among them, X C Y C Z C X represents the 3D point cloud coordinates of the face in the camera coordinate system. W Y W Z W R represents the 3D point cloud coordinates of the face in the world coordinate system, T represents the rotation matrix, and R and T are calculated from the 3DMM pose parameters.
[0028] As an improvement to the above technical solution, in step five, projecting the camera coordinate system onto the image coordinate system means that the camera projects the three-dimensional point cloud of the face onto the two-dimensional image through the projection matrix to obtain the coordinates of the image projection points. By using the projection point coordinates of the three-dimensional point cloud of the same model on two different face images, and using affine transformation, a pixel mapping relationship is established to generate a new face image with the texture of the source image, the expression of the target image, and the pose.
[0029] As an improvement to the above technical solution, in step six, face image fusion includes pixel weight mixing of face regions. Using the face boundary obtained from facial key points, pixel fusion weights are set for the face image within the boundary. The fusion formula is as follows:
[0030] I=αA n +(1-α)B 0≤α≤1
[0031]
[0032] Among them, A n Let B represent the new face image generated in step five, B represent the target face region, α represent the fusion weight, and s represent the fused image region.
[0033] When s belongs to the five sensory organs, α = 1; when s belongs to other regions of the five sensory organs, α is calculated from the gradient difference of the fused regions. Indicates the fusion region A n Gradient value of a face, G B This represents the gradient value of the face in the fused region B.
[0034] The beneficial effects of this invention are:
[0035] The proposed 3D reconstruction-based human face replacement method can replace any single frontal face image with any video face without requiring a large amount of face data and computing resources. At the same time, it ensures that the replaced face has rich texture details, and the expression and posture are highly consistent with the target face in the video, and the replaced face can guarantee high clarity.
[0036] This invention can achieve face swapping with arbitrary rotation angles and complex expressions using only one or two source face photos, quickly fulfilling the need for face replacement; at the same time, it preserves the target person's actions and expressions. The face replacement process does not require a large number of source face images for input, nor does it require additional model training costs for the target face. Attached Figure Description
[0037] Figure 1 This is a flowchart of the human face replacement method based on three-dimensional reconstruction as described in this invention. Detailed Implementation
[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0039] Example 1
[0040] like Figure 1 As shown, the human face replacement method based on three-dimensional reconstruction includes the following steps:
[0041] Step 1: 3D Reconstruction of Source Face and Target Face in Video
[0042] Input two-dimensional images of the source face (usually a frontal image with a natural expression) and the target face. Use the 3DMM face 3D reconstruction method to solve for the 3DMM parameters of the source face and the target face. The 3DMM parameters include shape parameters, expression parameters, and pose parameters. The 3DMM face 3D reconstruction method selected is a single-image 3DMM face 3D reconstruction method based on deep learning.
[0043] exist Figure 1 In this context, face A refers to the source face, face B video refers to the target face video, and face B frontal face refers to a natural expression image of face B.
[0044] Given a face image A, calculate the 3DMM parameters of A.
[0045] In step one, a video of the target face is input, face detection extracts the face region image, and the 3DMM parameters of the current frame are calculated. For example, if a B-face video is input, face detection extracts the face region image, and the 3DMM parameters of the current frame are calculated.
[0046] Input a frontal image of B, and calculate the unified solution of 3DMM parameters and shape parameters of the frontal image of B.
[0047] Additionally, the 3DMM face 3D reconstruction method mentioned in step one, short for 3D Morphable Model, uses the following reconstruction representation formula:
[0048]
[0049] in, S represents the average face model. i α represents the principal component analysis part corresponding to the shape. i The coefficient representing the current shape, e i This represents the principal component analysis component corresponding to the facial expression, β. i This represents the coefficient corresponding to the current expression; the 3D shape of the current face image is generated by a linear combination of the average shape and m basic shapes, and the 3D expression of the current face image is generated by a linear combination of the average expression and n basic expressions. `i` is an integer symbol used in the accumulation symbol, for example...
[0050] Step 2: Calculate the unified solution for the 3DMM shape parameters of the face in the video.
[0051] Filter video frames containing the target face at a frontal angle, then filter the shape parameters of the video frames containing the target face at a frontal angle, and calculate an average shape as a unified solution for the video face shape parameters.
[0052] In step two, because the shape parameters calculated for the same face in different frames of the video are inconsistent, directly using different shape parameters will cause the 3D face model in the video to exhibit irregular jitter in space and time, and the actual face shape will not change over time. Therefore, the shape parameters of the face images in the video frames at a frontal angle are selected, and an average shape is calculated as a unified solution for the video face shape parameters.
[0053] The unified solution for calculating the face shape parameters in the video mentioned in step two aims to ensure the shape invariance of the same face and prevent the three-dimensional shape of the same face from repeatedly jumping in time due to slight differences in the calculated shape, which would directly affect the stability of the texture mapping established in step five.
[0054] Step 3: Optimization of 3DMM expression and pose parameters
[0055] The expression and pose parameters solved by 3DMM are optimized by calculating the offset between the projected positions of key points selected in the source face 3D points and the detection positions of 68 key points of the source face.
[0056] Among them, the facial expression parameters and pose parameters calculated by 3DMM are optimized. The error calculated by the objective function is the position offset (projection error) between the 68 key projection points selected in the 3D shape projection points and the 68 key facial points detected in the actual face. The optimization method uses the Levenberg-Marquardt method or the Gauss-Newton method.
[0057] Step 4: Combine and generate new 3DMM shapes.
[0058] Obtain the shape parameters of the source face, obtain the expression parameters and pose parameters of the target face, and combine the shape parameters of the source face, the expression parameters and pose parameters of the target face to generate a 3DMM shape for the replacement face;
[0059] In the process of combination, a combination model is generated. The generation of the combination model refers to the combination of the optimized shape parameter vector of the source face and the expression parameter vector of the target face to generate a new set of face 3DMM feature vector coefficients.
[0060] Step 5: Generate a new face image that incorporates the texture features of the source face image and the expression and pose of the target face image.
[0061] The 3D point cloud reconstructed from the source face is transformed to the camera coordinate system according to the pose parameters, and then projected onto the source image. By aligning the facial key points and translating them, a mapping relationship between the 3D points and the pixels of the source face image is established. Then, a mapping relationship between the 3D points of the combined model and the pixels of the target face image is established. For the pixel value of a certain position projected by the combined model onto the target face image, the corresponding face coordinates of the source image are calculated through affine transformation, and the pixel value of that position is calculated through bilinear interpolation.
[0062] The formula for transforming a 3D point cloud to the camera coordinate system is as follows:
[0063]
[0064] Among them, X C Y C Z C X represents the 3D point cloud coordinates of the face in the camera coordinate system. W Y W Z W R represents the 3D point cloud coordinates of the face in the world coordinate system, T represents the rotation matrix, and R and T are calculated from the 3DMM pose parameters.
[0065] In step five, projecting the camera coordinate system onto the image coordinate system means that the camera projects the 3D point cloud of the face onto the 2D image through the projection matrix to obtain the coordinates of the image projection points. By using the projection point coordinates of the 3D point cloud of the same model on two different face images, and employing affine transformation, a pixel mapping relationship is established to generate a new face image with the texture of the source image, the expression of the target image, and the pose.
[0066] Step Six: Face Image Fusion
[0067] For the generated new face image, the boundary of the face is obtained by using facial key points, the face region image inside the boundary is extracted and aligned to the target face image in the video, and Poisson fusion is used to fuse the face images.
[0068] In step six, face image fusion includes pixel weight mixing of face regions. Face boundaries are obtained using facial key points, and pixel fusion weights are set for the face images within the boundaries. The fusion formula is as follows:
[0069] I=αA n +(1-α)B 0≤α≤1
[0070]
[0071] Among them, A nThe new face image generated in step five is the new face region A; B represents the target face region, which is the face region B; α represents the fusion weight, and the operation is performed using the pixel values in its face image.
[0072] Let s represent the image region to be fused. When s belongs to the facial features, α = 1 to ensure that the replaced and fused face image retains the facial feature texture characteristics of A. When s belongs to other areas of the facial features (such as the skin area), α is calculated from the gradient difference of the fused regions. Indicates the fusion region A n Gradient value of a face, G B This represents the gradient value of the face in the fused region B.
[0073] After pixel fusion, Poisson fusion is performed on the face region to generate a fused face image. Face swapping complete.
[0074] By dividing the facial features and skin regions and using an adaptive facial region fusion method, the differences in face replacement caused by different skin tones and lighting conditions can be reduced, making the final face replacement result more natural and realistic.
[0075] If the facial features and skin areas are not separated and the faces are directly merged, the final face replacement result will be even more stiff and distorted.
[0076] In the above embodiments, a unified solution for the shape parameters of the 3D reconstruction of the video face is calculated, which improves the stability of the video face replacement.
[0077] The facial expression and pose parameters of the 3DMM face model are optimized. Based on the CNN (deep neural network)-based 3D face reconstruction results, the facial expression and pose parameters of the 3D face model are further optimized to make the reconstructed 3D face model more realistic.
[0078] Face combination model generation can achieve facial expression transfer by combining the 3DMM parameters of the source face and the target face.
[0079] It generates high-resolution face texture maps with texture features of source face images and expression and pose of target face images. By utilizing the projection relationship of the 3D face model onto the source and target face images, a texture pixel mapping relationship is established. High-resolution combined face texture maps can be generated through image affine transformation and bilinear interpolation methods.
[0080] An adaptive face image fusion method reduces the differences in face replacement caused by different skin tones and lighting conditions through pixel mixing. On this basis, Poisson fusion of facial features is then performed, which also preserves the facial features of the source face, making the final face replacement result more natural and realistic.
[0081] This invention can replace any two faces without requiring a large amount of facial data and computing resources; the replaced face can maintain high clarity.
[0082] This invention can achieve face swapping with arbitrary rotation angles and complex expressions using only one or two source face photos, quickly fulfilling the need for face replacement; at the same time, it preserves the target person's actions and expressions. The face replacement process does not require a large number of source face images for input, nor does it require additional model training costs for the target face.
[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for human face replacement based on three-dimensional reconstruction, characterized in that... Includes the following steps: Step 1: 3D Reconstruction of Source Face and Target Face in Video Input two-dimensional images of the source face and the target face, and use the 3DMM face 3D reconstruction method to solve the 3DMM parameters of the source face and the target face. The 3DMM parameters include shape parameters, expression parameters, and pose parameters. Step 2: Calculate the unified solution for the 3DMM shape parameters of the face in the video. Filter video frames containing the target face at a frontal angle, then filter the shape parameters of the video frames containing the target face at a frontal angle, and calculate an average shape as a unified solution for the video face shape parameters; Step 3: Optimization of 3DMM expression and pose parameters The expression and pose parameters solved by 3DMM are optimized by calculating the offset between the projected positions of key points selected in the source face 3D points and the detection positions of 68 key points of the source face. Step 4: Combine and generate new 3DMM shapes Obtain the shape parameters of the source face, obtain the expression parameters and pose parameters of the target face, and combine the shape parameters of the source face, the expression parameters and pose parameters of the target face to generate a 3DMM shape for the replacement face; Step 5: Generate a new face image that incorporates the texture features of the source face image and the expression and pose of the target face image. The 3D point cloud reconstructed from the source face is transformed into the camera coordinate system according to the pose parameters, and then projected onto the source image. By aligning the facial key points, scaling and translation are performed to establish the mapping relationship between the 3D points and the pixels of the source face image. Then, establish the mapping relationship between the 3D points of the combined model and the pixels of the target face image; for the pixel value of a certain position projected by the combined model onto the target face image, calculate the source image face coordinates corresponding to the coordinates of that position through affine transformation, and calculate the pixel value of that position through bilinear interpolation. Step Six: Face Image Fusion For the generated new face image, the face boundary is obtained by using facial key points, the face region image inside the boundary is extracted and aligned to the target face image in the video, and Poisson fusion is used to fuse the face images; In step six, face image fusion includes pixel weight mixing of face regions. Using facial landmarks to obtain the face boundary, pixel fusion weights are set for the face image within the boundary. The fusion formula is as follows: ； ； in, B represents the new face image generated in step five, and B represents the target face region. represents the fusion weight, and s represents the fusion image region; When 's' refers to the five facial features =1; when s belongs to other areas of the five senses Calculated from the gradient difference in the fused region. Indicates the fusion area Gradient values of the face, This represents the gradient value of the face in the fused region B.
2. The method for human face replacement based on three-dimensional reconstruction according to claim 1, characterized in that: In step one, the target face is input as a video, the face detection extracts the face region image, and the 3DMM parameters of the current frame are calculated.
3. The method for human face replacement based on three-dimensional reconstruction according to claim 1, characterized in that: In step one, the reconstruction representation formula used in the 3DMM face 3D reconstruction method is as follows: ； in, This represents the average face model. This represents the principal component analysis portion corresponding to the shape. The coefficients representing the current shape, This represents the principal component analysis component corresponding to the facial expression. This represents the coefficient corresponding to the current expression; the three-dimensional shape of the current face image is generated by a linear combination of the average shape and m basic shapes, and the three-dimensional expression of the current face image is generated by a linear combination of the average expression and n basic expressions.
4. The method for human face replacement based on three-dimensional reconstruction according to claim 1, characterized in that: Step 3 involves optimizing the facial expression and pose parameters calculated by 3DMM. The error calculated by the objective function is the positional offset between the 68 key projection points selected in the 3D shape projection points and the 68 facial key points detected in the actual face. The optimization method uses the Levenberg-Marquardt method or the Gauss-Newton method.
5. The method for human face replacement based on three-dimensional reconstruction according to claim 1, characterized in that: In step four, during the combination process, a combined model is generated. The generation of the combined model refers to obtaining the optimized shape parameter vector of the source face and the expression parameter vector of the target face respectively and combining them to generate a new set of face 3DMM feature vector coefficients.
6. The method for human face replacement based on three-dimensional reconstruction according to claim 1, characterized in that: In step five, the 3D point cloud is transformed to the camera coordinate system. The transformation formula is as follows: ； in, This represents the 3D point cloud coordinates of the face in the camera coordinate system. R represents the 3D point cloud coordinates of the face in the world coordinate system, T represents the rotation matrix, and R and T are calculated from the 3DMM pose parameters.
7. The method for human face replacement based on three-dimensional reconstruction according to claim 1, characterized in that: In step five, projecting the camera coordinate system onto the image coordinate system means that the camera projects the 3D point cloud of the face onto the 2D image through the projection matrix to obtain the coordinates of the image projection points. By using the projection point coordinates of the 3D point cloud of the same model on two different face images, and employing affine transformation, a pixel mapping relationship is established to generate a new face image with the texture of the source image, the expression of the target image, and the pose.