3D face model generation method based on image fusion

Abstract

A 3D face model generation method based on image fusion comprises the following steps: S1, making UV mapping and projection chartlets for the face of a figure model, ensuring that face grids are suitable for front projection, and forming a face chartlet of the figure model; s2, performing face marking on the face photo and the character model face map; s3, defining a point selection set of the five sense organs, and generating a mask of the five sense organs; s4, transforming the face picture through rotation to enable the face picture to adapt to the face map of the character model; s5, adjusting the color balance of the face photo to enable the face photo to be matched with the face map of the character model; and S6, pasting the adjusted face picture on the face map of the character model by using a five-sense-organ mask. According to the image fusion-based 3D face model generation method provided by the invention, key features in different images can be automatically identified and fused, so that a highly realistic 3D face is generated, the labor cost is reduced, and the generated face model is more realistic.

Description

Technical Field

[0001] The invention belongs to the fields of computer graphics and artificial intelligence, and in particular relates to a method for generating a 3D human face model based on image fusion. Background Art

[0002] In today's digital age, 3D modeling and rendering technology plays an important role in many fields such as movies, games, and virtual reality. However, the generation process of traditional 3D face models is not only time-consuming and labor-intensive, but also requires a high degree of professional skills. The generation of traditional 3D face models relies on artists to manually draw maps, and traditional image transformation processing is to manually use image editing tools to cut out and place the face. This process not only requires a lot of time for fine-tuning, but also requires extremely high processing of details. Manually using image editing tools to cut out and place the face, the processing of facial contours and skin color adaptation is relatively rough, which leads to low accuracy of the fusion result and the fusion process is difficult to automate; in addition, the differences in styles of different artists may also lead to visual inconsistencies in the final product. With the advancement of technology, automation and intelligence have become the key to improving efficiency and unifying quality.

[0003] With the rapid development of artificial intelligence technology, especially the breakthrough of deep learning in the field of image processing, 3D face model generation technology has emerged, bringing revolutionary changes to the 3D modeling industry. Image fusion technology combines the advantages of machine learning and computer vision, which is helpful for the generation of 3D face models. Therefore, in order to reduce manual operations and to improve the realism of the model, a method for generating a 3D face model based on image fusion is needed. The present invention solves this technical problem. Summary of the invention

[0004] The present invention provides a 3D face model generation method based on image fusion, which can automatically identify and fuse key features in different images to generate a 3D face with high realism, thereby reducing labor costs and making the generated face model more realistic.

[0005] A method for generating a 3D face model based on image fusion comprises the following steps: S1. Make UV mapping and projection map for the face of the character model to ensure that the face mesh is suitable for frontal projection and form the face map of the character model; S2, performing facial marking on the face photo and the face map of the character model; S3, defining a set of selected points of facial features and generating facial features masks; S4, rotating and scaling the face photo to make it fit the face map of the character model; S5. Adjust the color balance of the face photo to match the face map of the character model; S6. Use facial features mask to paste the adjusted face photo onto the character model face texture.

[0006] Furthermore, the step S1 includes the following steps: S11. Perform topology on the face of the character model; S12. Adjust the UV of the obtained facial mesh body to the front projection state.

[0007] Furthermore, the step S2 includes the following steps: S21, using a face recognition algorithm to mark facial features of the face photo and the face map of the character model respectively; S22, using the pre-trained facial feature point detection model as a feature extractor to locate and identify n key points in the face image; S23, reading the data of the face photo and the face map of the character model into the memory through OPENCV, and using the data as the algorithm input of the feature extractor; S24. The feature extractor returns an n×2 element matrix, where x and y represent the coordinates of each point on the plane, and each matrix corresponds to a facial feature marker of a face.

[0008] Furthermore, the step S21 includes the following steps: S211, use Dlib as the algorithm for face recognition; S212, importing two photos, namely, the face photo and the face map of the character model, into Dlib; S213, record the image data of the obtained character model face map as im1, and record the image number of the face photo as im2.

[0009] Furthermore, the step S3 includes the following steps: S31, selecting sets of key points corresponding to the five sense organs from the obtained n key points respectively; S32. Using the set of key points corresponding to the selected facial features as a mask set.

[0010] Further, the step S4 includes the following steps: S41, according to the obtained mask set, set the mask set generated by im1 as L1, and set the mask set generated by im2 as L2; S42, calculating the transformation matrix M from L2 to L1; S43, using the transformation matrix M, rotating, moving, and scaling im2 so that it corresponds to the position of im1; S44, transform the coordinate value in L2 according to M.

[0011] Furthermore, the step S42 includes the following steps: S421. Using conventional Pullman analysis: ,

[0012] Where n represents the number of key points, and are the rows of L1 and L2 respectively, R is a 2×2 orthogonal matrix, s is a scalar, T is a two-dimensional vector, and R, s, and T are found by singular value decomposition to minimize the expression in the above figure; S422. Obtain the final transformation matrix M using affine transformation.

[0013] Furthermore, the step S5 includes the following steps: S51, use the face photo to match the color of the model map; S52. Divide im2 by the Gaussian blur of im2, and then multiply it by the Gaussian blur of im1 to obtain a new im2.

[0014] Further, the step S6 includes the following steps: S61, use im1 to subtract the adjusted facial features mask, then add im2 and multiply the facial features mask, and finally get the synthesized map; S62, pasting the obtained texture onto the character model, thereby obtaining the required three-dimensional model.

[0015] The technical effects of the present invention are as follows:

[0016] (1) This solution can automatically perform facial feature recognition and feature extraction on face photos and model maps through face recognition algorithms and feature extractors, and then define facial feature masks. After that, the mask positions of the face photos and model maps are made to correspond to each other through transformation matrices, and the model maps are adapted to the model maps. In this way, an automated process is realized, which greatly reduces manual operations. Compared with manual mapping operations, this solution can improve the realism of the face model, thereby ensuring the quality of the face model. (2) This solution can solve the problems of long manual labor and imprecise facial model adjustment in traditional methods through automated processes, providing a powerful tool for 3D artists and designers, allowing staff to focus more on creativity and design and avoid the impact of tedious technical details on their work. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the present invention.

[0018] Figure 2 This is a schematic diagram of adjusting the UV of the facial mesh body to the front projection state in the present invention.

[0019] Figure 3 Schematic diagram of facial feature marking of a human face in the present invention.

[0020] Figure 4 Schematic diagram of the facial features mask before and after transformation in the present invention.

[0021] Figure 5 It is a schematic diagram of the face photo after matching the model map color in the present invention.

[0022] Figure 6 This is the final rendering of the texture and the three-dimensional model obtained in the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and drawings.

[0024] See also Figure 1 , a 3D face model generation method based on image fusion, comprising the following steps: S1. Make UV mapping and projection map for the face of the character model to ensure that the face mesh is suitable for frontal projection and form the face map of the character model; S2, performing facial marking on the face photo and the face map of the character model; S3, defining a set of selected points of facial features and generating facial features masks; S4, rotating and scaling the face photo to make it fit the face map of the character model; S5. Adjust the color balance of the face photo to match the face map of the character model; S6. Use facial features mask to paste the adjusted face photo onto the character model face texture.

[0025] See also Figure 2 , the step S1 comprises the following steps: S11. Perform topology on the face of the character model; S12. Adjust the UV of the obtained facial mesh body to the front projection state.

[0026] Furthermore, the step S2 includes the following steps: S21, using a face recognition algorithm to mark facial features of the face photo and the face map of the character model respectively; S22, using the pre-trained facial feature point detection model as a feature extractor to locate and identify n key points in the face image; S23, reading the data of the face photo and the face map of the character model into the memory through OPENCV, and using the data as the algorithm input of the feature extractor; S24. The feature extractor returns an n×2 element matrix, where x and y represent the coordinates of each point on the plane, and each matrix corresponds to a facial feature marker of a face.

[0027] Specifically, to build the feature extractor, we used a pre-trained model (shape_predictor_68_face_landmarks.dat). This is a pre-trained face landmark detection model that is used to locate and identify 68 key points in a face image. These key points cover various parts of the face, such as eyes, nose, mouth, cheeks, etc. The model is based on a machine learning technique, which is to train many face images to learn the characteristics of face key points. Using this model, face key points can be automatically located and identified in computer vision applications, thereby realizing functions such as face recognition, face alignment, and expression recognition. The model is provided by the Dlib library and can be used in multiple languages ​​such as Python and C++. In this embodiment, 68 key points are identified.

[0028] Furthermore, the step S21 includes the following steps: S211, use Dlib as the algorithm for face recognition; S212, importing two photos, namely, the face photo and the face map of the character model, into Dlib; S213, record the image data of the obtained character model face map as im1, and record the image number of the face photo as im2.

[0029] See also Figure 3 , the step S3 comprises the following steps: S31, selecting sets of key points corresponding to the five sense organs from the obtained n key points respectively; S32. Using the set of key points corresponding to the selected facial features as a mask set.

[0030] First, it is necessary to define a set of selected points for the facial features. The set defined in this embodiment is as follows: Left eyebrow collection: point 22 to point 27; Right eyebrow collection: point 17 to point 22; Left eye collection: point 42 to point 48; Right eye collection: point 36 to point 42; Nose collection: point 27 to point 35; Mouth collection: Point 48 to Point 61;

[0031] Then the mask set we select is: left eyebrow set + right eyebrow set + left eye set + right eye set + nose set + mouth set.

[0032] See also Figure 4 , the step S4 comprises the following steps: S41, according to the obtained mask set, set the mask set generated by im1 as L1, and set the mask set generated by im2 as L2; S42, calculating the transformation matrix M from L2 to L1; S43, using the transformation matrix M, rotating, moving, and scaling im2 so that it corresponds to the position of im1; S44, transform the coordinate value in L2 according to M.

[0033] Furthermore, the step S42 includes the following steps: S421. Using conventional Pullman analysis: ,

[0034] Where n represents the number of key points, and are the rows of L1 and L2 respectively, R is a 2×2 orthogonal matrix, s is a scalar, T is a two-dimensional vector, and R, s, and T are found by singular value decomposition to minimize the expression in the above figure;

[0035] S422. Obtain the final transformation matrix M using affine transformation.

[0036] See also Figure 5 , the step S5 comprises the following steps: S51, use the face photo to match the color of the model map; S52. Divide im2 by the Gaussian blur of im2, and then multiply it by the Gaussian blur of im1 to obtain a new im2.

[0037] Due to the different skin colors and lighting between the face photo and the model face map, the color of the covered area is discontinuous. The face photo is needed to match the color of the model map, so the color matching is achieved through the above method.

[0038] See also Figure 6 , the step S6 comprises the following steps: S61, use im1 to subtract the adjusted facial features mask, then add im2 and multiply the facial features mask, and finally get the synthesized map; S62, pasting the obtained texture onto the character model, thereby obtaining the required three-dimensional model.

[0039] The working process of the present invention is as follows:

[0040] First, UV mapping and topology adjustment are performed on the character model to ensure that the facial mesh is suitable for frontal projection; then, Dlib's face recognition algorithm is used to mark facial features on the model map and face photo, and data is extracted through the feature extractor to obtain an element matrix, where one matrix corresponds to the facial feature markings of a face; then, based on several key points previously marked for facial features, a set of selected points for the facial features is defined to generate a facial feature mask; by calculating the transformation matrix, the rotation, transformation, and scaling of the face photo are adjusted to adapt to the model map, and the color balance is adjusted to match the model map; finally, the facial feature mask is used to merge the adjusted face photo into the model map to complete the generation of the 3D model face map.

[0041] The above embodiments are only preferred embodiments of the present invention. The above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention and are not intended to limit the scope of the present invention. For those skilled in the art, various modifications and improvements can be made on the basis of the present invention without departing from the spirit and essence of the present invention. These modifications and improvements are also regarded as the protection scope of the present invention. Therefore, this application protects not only the above embodiments, but also the scope consistent with the principles and features of the present invention.

Claims

1. A method for generating a 3D face model based on image fusion, characterized in that: The following steps are involved: S1. Make UV mapping and projection map for the face of the character model to ensure that the face mesh is suitable for frontal projection and form the face map of the character model; S2, performing facial marking on the face photo and the face map of the character model; S3, defining a set of selected points of facial features and generating facial features masks; S4, rotating, transforming, and scaling the face photo to make it fit the face map of the character model; S5. Adjust the color balance of the face photo to match the face map of the character model; S6. Use facial features mask to paste the adjusted face photo onto the character model face texture.

2. The method for generating a 3D face model based on image fusion according to claim 1, characterized in that: The step S1 comprises the following steps: S11. Perform topology on the face of the character model; S12. Adjust the UV of the obtained facial mesh body to the front projection state.

3. The method for generating a 3D face model based on image fusion according to claim 1, characterized in that: The step S2 comprises the following steps: S21, using a face recognition algorithm to mark facial features of the face photo and the face map of the character model respectively; S22, using the pre-trained facial feature point detection model as a feature extractor to locate and identify n key points in the face image; S23, reading the data of the face photo and the face map of the character model into the memory through OPENCV, and using the data as the algorithm input of the feature extractor; S24. The feature extractor returns an n×2 element matrix, where x and y represent the coordinates of each point on the plane, and each matrix corresponds to a facial feature marker of a face.

4. The method for generating a 3D face model based on image fusion according to claim 3, characterized in that: The step S21 comprises the following steps: S211, use Dlib as the algorithm for face recognition; S212, importing two photos, namely, the face photo and the face map of the character model, into Dlib; S213, record the image data of the obtained character model face map as im1, and record the image number of the face photo as im2.

5. The method for generating a 3D face model based on image fusion according to claim 3, characterized in that: The step S3 comprises the following steps: S31, selecting sets of key points corresponding to the five sense organs from the obtained n key points respectively; S32. Using the set of key points corresponding to the selected facial features as a mask set.

6. The method for generating a 3D face model based on image fusion according to claim 5, characterized in that: The step S4 comprises the following steps: S41, according to the obtained mask set, set the mask set generated by im1 as L1, and set the mask set generated by im2 as L2; S42, calculating the transformation matrix M from L2 to L1; S43, using the transformation matrix M, rotating, moving, and scaling im2 so that it corresponds to the position of im1; S44, transform the coordinate value in L2 according to M.

7. The method for generating a 3D face model based on image fusion according to claim 6, characterized in that: The step S42 comprises the following steps: S421. Using conventional Pullman analysis: ， Where n represents the number of key points, and are the rows of L1 and L2 respectively, R is a 2×2 orthogonal matrix, s is a scalar, T is a two-dimensional vector, and R, s, and T are found by singular value decomposition to minimize the expression in the above figure; S422. Obtain the final transformation matrix M using affine transformation.

8. The method for generating a 3D face model based on image fusion according to claim 4, characterized in that: The step S5 comprises the following steps: S51, use the face photo to match the color of the model map; S52. Divide im2 by the Gaussian blur of im2, and then multiply it by the Gaussian blur of im1 to obtain a new im2.

9. The method for generating a 3D face model based on image fusion according to claim 4, characterized in that: The step S6 comprises the following steps: S61, use im1 to subtract the adjusted facial features mask, then add im2 and multiply the facial features mask, and finally get the synthesized map; S62, pasting the obtained texture onto the character model, thereby obtaining the required three-dimensional model.

Images