A three-dimensional face model automatic mapping generation system and method
By acquiring and transforming key points of facial images, and combining 3D models and deep learning, the problem of high complexity in existing 3D face-swapping technologies has been solved. This enables efficient automatic mapping generation of 3D facial models, which is applicable to various topologies and has low hardware requirements.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LANZHOU FUTURE NEW FILM CULTURE & TECH GRP CO LTD
- Filing Date
- 2022-10-18
- Publication Date
- 2026-05-19
AI Technical Summary
Existing 3D face-swapping technologies are complex and require a large amount of training data, resulting in low efficiency and difficulty in achieving efficient automatic texturing of 3D face models.
By acquiring key points of a face image, dividing it into sub-regions, transforming and fusing the vertices and normal vectors of a 3D model, and combining this with a deep learning model to extract detailed features, automatic texture generation of a 3D face model is achieved.
It achieves efficient and rapid 3D face texture generation, is applicable to any topological model, has low hardware performance requirements, and produces realistic and natural results.
Smart Images
Figure CN115512421B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D face model technology, specifically to a 3D face model automatic texture generation system and method. Background Technology
[0002] Face-swapping technology is an application that has emerged in recent years, primarily used for face swapping on 2D images. Examples include the face-swapping techniques used in popular online short videos. Replacing one person's face with the face of another in a video can be applied in the film and television industry; for instance, if an actor is unable to complete post-production filming for some reason, another actor's face can be used to replace them. However, face-swapping in the 3D field is still relatively rare. With the recent rise of the digital human universe, the demand for 3D face-swapping is gradually emerging. Currently, most face-swapping involves 3D models and lacks simultaneous face mapping.
[0003] For example, Chinese patent application number CN202110882708.4 provides a face-swapping and re-enactment method and apparatus based on 3D face decomposition, which features the ability to simultaneously perform face swapping and expression re-enactment on any person using the same process. For example, Chinese patent application number CN201910996426.X discloses a 3D face fusion method and system based on a graphics processor, which obtains a natural and richly styled fused portrait by controlling the mixing ratio of the general outline components and contour details of the user's face and a specific image at various scales.
[0004] There are few existing face-swapping patents, and the methods are complex. Some require training models, which is time-consuming and labor-intensive to prepare training data. Summary of the Invention
[0005] The purpose of this invention is to provide an automatic texture generation system and method for three-dimensional human face models to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for automatically generating textures for a 3D face model, wherein the specific steps of the method are as follows:
[0007] Obtain the first face image, the second face image, and the second face normal vector image;
[0008] Obtain key points from the first and second face images;
[0009] Group the key points that meet the criteria into a sub-region, and divide all key points in turn.
[0010] Obtain the target region of the first face image, which will be used as the region to be replaced in the second face image;
[0011] The key point sub-regions in the target region of the first face image are transformed to the corresponding key point sub-regions in the second face image to obtain the third face image;
[0012] The second face image is fused with the target area of the third face image to obtain a fourth face image;
[0013] Obtain the normal vector image of the detailed features of the first face image;
[0014] The key point sub-regions within the target region of the first face image detail feature normal vector image are transformed into the second face normal vector image to obtain the detail normal vector image corresponding to the fourth face image.
[0015] Furthermore, the acquisition of key points in the first and second face images includes key points detected through face recognition.
[0016] Furthermore, the acquisition of key points in the first and second face images includes obtaining key points by corresponding the coordinates of the vertices of the 3D model with those of the first and second face images, and the mapping relationship between the key points of the first and second face images.
[0017] Furthermore, the key point segmentation is obtained by triangulation using key points detected by face recognition.
[0018] Furthermore, the key point segmentation is obtained using the surface information of the three-dimensional model, and the three-dimensional vertices that make up a surface correspond to a set of image coordinates on the second face image.
[0019] Furthermore, the target region extraction process includes:
[0020] Remove hair from the first face image;
[0021] Remove non-positive visible occlusions from the first face image.
[0022] Furthermore, the key point sub-region transformation process includes:
[0023] Obtain key point sub-regions from the first and second face images within the target area;
[0024] Calculate the transformation from the key point sub-region of the first face image to the key point region of the second face image;
[0025] The key point region of the first face image is transformed to the corresponding key point region of the second face image according to the transformation, to obtain the third face image.
[0026] Furthermore, the target region fusion includes image processing of the target regions of the second face image and the target regions of the fourth face image.
[0027] Furthermore, the transformation process of the key point sub-region of the detailed feature normal vector image includes:
[0028] Obtain the key point sub-regions of the first face detail normal vector image and the second face normal vector image within the target area;
[0029] Calculate the transformation from the key point sub-region of the first face detail normal vector image to the key point region of the second face normal vector image;
[0030] The key point region of the first face detail normal vector image is transformed to the corresponding key point region of the second face normal vector image according to the transformation, to obtain the fourth face detail normal vector face image.
[0031] A system for automatically generating textures for 3D human face models, the system comprising:
[0032] The image acquisition module is used to acquire a first face image, a second face image, and a second face normal vector image;
[0033] The key point acquisition module is used to acquire key points in the first face image and the second face image;
[0034] The key point segmentation module is used to group several key points that meet certain conditions into a sub-region.
[0035] The target region acquisition module calculates the target region in the first face image;
[0036] The key point sub-region transformation module is used to transform the key point sub-region in the target region of the first face image to the key point region in the second face image;
[0037] An image processing module is used to fuse the target area of the second face image with the third face image to obtain a fourth face image;
[0038] The detail feature extraction module is used to obtain the detail feature normal vector image in the first face image;
[0039] The normal vector image transformation module is used to transform the key point sub-region of the target area in the first face detail feature normal vector image to the second face normal vector image to obtain the fourth face normal vector image.
[0040] Processor and memory storing computer program instructions;
[0041] When the processor executes the computer program instructions, it implements the above-described method for automatically generating textures for three-dimensional human face models.
[0042] A computer storage medium storing a computer and program instructions, wherein the computer program instructions are processed to generate the method for automatically generating textures of a 3D human face model.
[0043] Compared with the prior art, the beneficial effects of the present invention are:
[0044] This invention proposes an automatic texture generation method for 3D face models, enabling face swapping using 3D face textures, applicable to models with arbitrary topological structures. It can generate face textures for any 3D model; it is highly efficient, has a fast execution speed, and low hardware performance requirements. Attached Figure Description
[0045] Figure 1 This is a framework diagram of the automatic texture generation system for 3D human face models according to the present invention. Detailed Implementation
[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0047] Example:
[0048] Please see Figure 1 The present invention provides a technical solution, specifically including:
[0049] Image acquisition module 11 acquires a first face image 12, a second face image 16, and a second normal vector image 25.
[0050] The key point acquisition module 15 extracts key points 17 from the first and second face images. These key points can be feature points in the face images, referring to feature points that characterize facial features; or they can be the image coordinates of all face vertices on the first face image 12 obtained through the mapping relationship between the vertices of the 3D model and the first face image 12. The correspondence 29 between the key points of the first and second face images can be obtained through the relationship between the vertices of the two 3D models, or through the unique order between facial feature points.
[0051] The keypoint segmentation module 19 divides the keypoints 17 of the first and second face images into regions, resulting in keypoint sub-regions 21 and 22 of the first and second face images, which serve as input to the keypoint sub-region transformation module 13. Keypoints are the key points in the face image, divided into groups of three using triangulation. Keypoints are the image coordinates corresponding to vertices of the 3D model on the image, divided according to the 3D model's patch information; the image coordinates of vertices contained in a patch are grouped together. The mapping relationship between 3D model vertices and face images can be directly read from the model file.
[0052] The target region acquisition module 18 needs to remove occluded areas and retain the target regions that need to be replaced during face image generation. Specifically, the final generated face image does not include the hair portion, as the hair has a separate model and texture. To make the final generated texture more realistic, the hair occlusion needs to be removed; at the same time, the first face image may not be a frontal photo and may have a slight angle, so it is also necessary to remove some invisible points from the first face image 12. Invisible points can be obtained by combining the corresponding 3D model of the first face image.
[0053] The key point sub-region transformation module 13 transforms the image in the target region of the first face image 12 into the second face image 16, transforming the face in the second face image into the first face image. It calculates the transformation relationship between the key point regions of the first face image and the key point regions of the second face image, and transforms the key point sub-region 21 of the first face image into the corresponding key point sub-region 22 of the second face image according to the transformation relationship, thus obtaining the fourth face image 23.
[0054] Image processing module 24 performs fusion processing on the target region 20 of the fourth face image 23 and the second face image 16. Specifically, the fusion processing requires additional image preprocessing operations, including but not limited to image equalization and weighting. Without image preprocessing, direct Poisson fusion may result in the face region of the fused image being too bright (white), too dark (black), or having fusion boundaries. I = αI1 + (1-α)I2 0 ≤ α ≤ 1, where I represents the weighted image, and I1 and I2 represent the two face images. G1 and G2 represent the gradient images corresponding to two face images. The preprocessed images can be fused using Poisson fusion. To ensure the fusion effect, fusion is performed again on specific regions, which include, but are not limited to, facial features. I = P(I1, I2, R), where I1 and I2 represent face images and R represents specific regions.
[0055] The detail feature extraction module 26 extracts facial details from the first face image using the formula D = F(I), where F represents the model trained by deep learning, I represents the face image, and D represents the detail features obtained from the face image, thus obtaining the first face detail feature normal vector image 27.
[0056] The normal vector image transformation module 28 transforms the first face detail feature normal vector image 27 into the target area of the second face normal vector image 30 according to the transformation relationship between key point sub-regions, to obtain the fourth face detail feature normal vector image.
[0057] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or basic characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the invention is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.
[0058] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for automatically generating textures for a 3D human face model, characterized in that, Includes the following steps: Obtain the first face image, the second face image, and the second face normal vector image; Obtain key points from the first and second face images; Group the key points that meet the criteria into a sub-region, and divide all key points in turn. Obtain the target region of the first face image, which will be used as the region to be replaced in the second face image; The key point sub-regions in the target region of the first face image are transformed to the corresponding key point sub-regions in the second face image to obtain the third face image; The second face image is fused with the target area of the third face image to obtain a fourth face image; Obtain the normal vector image of the detailed features of the first face image; The key point sub-regions within the target region of the first face image detail feature normal vector image are transformed into the second face normal vector image to obtain the detail normal vector image corresponding to the fourth face image.
2. The method for automatically generating textures for a 3D human face model according to claim 1, characterized in that, The acquisition of key points in the first and second face images includes key points detected through face recognition.
3. The method for automatically generating textures for a 3D human face model according to claim 1, characterized in that, The acquisition of key points in the first and second face images includes: acquiring key points based on the correspondence between the vertices of the 3D model and the coordinates of the first and second face images, and determining the mapping relationship between the key points of the first and second face images based on the vertices of the 3D model.
4. The method for automatically generating textures for a 3D human face model according to claim 1, characterized in that, The key point segmentation is obtained by triangulation of key points detected by face recognition.
5. The method for automatically generating textures for a three-dimensional human face model according to claim 1, characterized in that, The key point division is obtained using the surface information of the three-dimensional model. The three-dimensional vertices that make up a surface correspond to a set of image coordinates on the second face image.
6. The method for automatically generating textures for a three-dimensional human face model according to claim 1, characterized in that, The target region extraction process includes: Remove hair from the first face image; Remove non-positive visible occlusions from the first face image.
7. The method for automatically generating textures for a three-dimensional human face model according to claim 1, characterized in that, The key point sub-region transformation process includes: Obtain key point sub-regions from the first and second face images within the target area; Calculate the transformation from the key point sub-region of the first face image to the key point region of the second face image; The key point region of the first face image is transformed to the corresponding key point region of the second face image according to the transformation, to obtain the third face image.
8. The method for automatically generating textures for a three-dimensional human face model according to claim 1, characterized in that, The target region fusion includes image processing of the target regions of the second face image and the fourth face image.
9. The method for automatically generating textures for a three-dimensional human face model according to claim 1, characterized in that, The detailed feature normal vector image key point sub-region transformation process includes: Obtain the key point sub-regions of the first face detail normal vector image and the second face normal vector image within the target area; Calculate the transformation from the key point sub-region of the first face detail normal vector image to the key point region of the second face normal vector image; The key point region of the first face detail normal vector image is transformed to the corresponding key point region of the second face normal vector image according to the transformation, to obtain the fourth face detail normal vector face image.
10. A system for automatically generating textures for 3D human face models, characterized in that, The system includes: The image acquisition module is used to acquire a first face image, a second face image, and a second face normal vector image; The key point acquisition module is used to acquire key points in the first face image and the second face image; The key point segmentation module is used to group several key points that meet certain conditions into a sub-region. The target region acquisition module calculates the target region in the first face image; The key point sub-region transformation module is used to transform the key point sub-region in the target region of the first face image to the key point region in the second face image; An image processing module is used to fuse the target area of the second face image with the third face image to obtain a fourth face image; The detail feature extraction module is used to obtain the detail feature normal vector image in the first face image; The normal vector image transformation module is used to transform the key point sub-region of the target area in the first face detail feature normal vector image to the second face normal vector image to obtain the fourth face normal vector image. Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the automatic texture generation method for three-dimensional human face models as described in any one of claims 1-9.