A general three-dimensional face model UV mapping generation method, system and readable storage medium based on a two-dimensional face image

Abstract

The application discloses a kind of general three-dimensional face model UV map generation method, system and readable storage medium based on two-dimensional face image, belong to image processing technical field, by obtaining two-dimensional face image and standard skin color basic UV map, the face key points of both are extracted or selected respectively;Two kinds of key points are aligned using thin sample board strip transformation, and the preliminary deformation result of face is obtained;Then the preliminary deformation result is smoothed by spline interpolation;Finally, the smoothed face contour is pasted to the basic UV map, and the boundary transition processing is carried out using Gaussian blur method, and the three-dimensional face model UV map is generated.The application combines different types of algorithms of deep learning and traditional image processing, improves the generation accuracy, naturalness and complex region processing capability of UV map, enhances the map fusion effect, has good universality and practicality, and can be widely applied in virtual reality, film and television production and other fields.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method, system, and storage medium for generating a universal three-dimensional face model UV map based on two-dimensional facial images. Background Technology

[0002] The UV mapping technology for generating 3D face models based on 2D facial images is mainly used in fields such as computer graphics, film and television special effects, and virtual reality (VR). UV mapping is a technique that accurately maps 2D images (often called texture images) onto the surface of 3D models. In 3D face modeling, the core task of UV mapping is to extract information such as skin texture and color from 2D images and apply it to 3D models to generate more realistic facial effects.

[0003] With the increasing demand for virtual characters, game characters, and film and television special effects, accurately applying realistic facial textures to 3D face models has become a crucial issue. Traditional 3D modeling techniques can provide realistic geometric structures, but visual effects rely on high-quality texture mapping. UV mapping is one of the core technologies for achieving this effect. It unfolds the surface of a 3D model into a 2D space, mapping the 2D image to the individual vertices of the 3D model, thus enabling the 2D image to be correctly displayed on the 3D model.

[0004] Generating UV maps from 2D images typically involves geometric unfolding, which unfolds a complex 3D surface into a flat 2D image, ensuring minimal stretching or distortion. This is particularly challenging in facial modeling, where the complex geometry of the face (such as eyes, nose, and mouth) presents significant technical difficulties in generating high-quality UV maps.

[0005] In existing technologies, methods for generating 3D face model UV maps based on 2D facial images are mainly divided into three categories: manual generation, automated generation, and deep learning-based generation. However, all of them have obvious drawbacks. In the early days, UV mapping was largely generated manually. Modelers manually flattened the model surface and adjusted UV coordinates using 3D software (such as Maya and Blender) to ensure the texture image seamlessly matched the model surface. While this method could produce high-precision results, it was extremely time-consuming and technically demanding, especially as the complexity of the model increased, leading to a significant increase in the workload of generating UV maps. Furthermore, the manual method carried the risk of stretching or misalignment, particularly when dealing with complex geometries.

[0006] Existing automated methods can automatically generate UV coordinates based on the geometric characteristics of a 3D model, unfolding complex 3D structures onto a 2D plane. Common automated unfolding methods include geometry-segmentation-based unfolding algorithms and projection-based unfolding algorithms. Geometry-segmentation algorithms typically divide the model into multiple parts, each unfolded independently; while projection algorithms generate UV coordinates by projecting the model along a certain direction. In facial modeling, automated UV unfolding algorithms usually automatically generate UV maps suitable for specific facial regions (such as eyes and mouth) based on their geometric features. However, due to the complexity of facial geometry, automatically generated UV maps can sometimes be stretched or distorted, thus often requiring manual adjustment.

[0007] In recent years, deep learning technology has been increasingly applied to UV mapping generation for 3D face models. By training a large number of face images and 3D models, the model can automatically learn the mapping relationship between 3D geometry and 2D images. For example, methods based on convolutional neural networks (CNNs) can extract key feature points from 2D face images and then use these feature points to generate accurate UV coordinate mappings. Compared with traditional geometric unfolding techniques, this method can handle complex facial structures more intelligently and generates UV maps with higher accuracy. However, most deep learning-based solutions can only achieve good generation results for a fixed set of UVs. When the UV format changes, the model weights trained by the current deep learning algorithm cannot adaptively adapt, resulting in significant limitations.

[0008] In summary, existing technologies for generating 3D face model UV maps based on 2D facial images still suffer from low efficiency, insufficient accuracy, and poor versatility, failing to meet the growing application demands. Therefore, providing an efficient, accurate, and universal method, system, and storage medium for generating 3D face model UV maps based on 2D facial images is a problem urgently needing to be solved by those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides a method, system and storage medium for generating UV maps of a general three-dimensional face model based on two-dimensional facial images, which can automatically generate accurate UV coordinates based on two-dimensional facial images, thereby seamlessly mapping facial skin, color, light and shadow information onto a three-dimensional model.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: On one hand, this invention provides a general method for generating UV maps of 3D human face models based on 2D facial images, including: Obtain a 2D human face image; Obtain the base UV map for standard skin tone; Select the key UV points for the face from the base UV map; Two-dimensional facial key points are extracted from the two-dimensional face image based on deep learning algorithms; Thin spline transformation is used to align UV facial key points with 2D facial key points to obtain preliminary facial deformation results; Spline interpolation is used to smooth the initial facial deformation results, resulting in a smoothed facial contour. The smoothed facial contours are applied to the base UV map, and boundary transition processing is performed to obtain the 3D face model UV map.

[0011] Preferably, thin spline transformation is used to align UV facial key points with 2D facial key points to obtain preliminary facial deformation results, including: Establish a one-to-one mapping relationship between UV key points and 2D face key points; Constructing the transformation function for thin spline transformation: ; in, These are the coefficients of the affine transformation; It is the contribution weight of the i-th control point to the deformation in the x-direction; It is a radial basis function; Key points of two-dimensional face and corresponding UV key points The distance between them; The coefficients and weights of the affine transformation are solved by solving a system of linear equations, which are: ; Where K is the radial basis function matrix, P is the affine partial matrix; It is a weight vector. y is the affine coefficient vector; y is the coordinate value containing all target control points. Substitute the obtained affine coefficients and weights into the transformation function, apply the transformation function to the two-dimensional facial key points, and obtain the corresponding positions of each two-dimensional facial key point in the UV facial region.

[0012] Preferably, spline interpolation is used to smooth the initial facial deformation results to obtain a smoothed facial contour, including: Extract aligned key points from the initial facial deformation results and obtain the corresponding two-dimensional coordinates and pixel color values; The facial region is divided into multiple continuous sub-regions; For each sub-region, a cubic spline function satisfying the continuity of the first and second derivatives is constructed using the coordinates of key points as nodes; The solution is obtained by satisfying the conditions that the function values ​​at the endpoints of adjacent intervals are equal, the first derivative is equal, the second derivative is equal, and the boundary conditions are met. Based on the cubic spline function, the coordinates of the transition points are calculated between adjacent key points in each sub-region at a preset step size, so that a continuous smooth curve is formed between adjacent key points. The acquired key points and transition points are binarized according to sub-regions to obtain a face segmentation binary image. The extracted binary image of the face segmentation is subjected to the same thin spline transformation to obtain a new binary image of the face. Finally, the new binary image of the face is multiplied with the result of the initial facial deformation to obtain the facial region.

[0013] Preferably, the smoothing steps for the forehead area include: Obtain the coordinates of all 2D facial key points in the upper half of the eyebrow region; Using each coordinate as a reference, calculate the color upwards from the forehead. Stop when a point with a color exceeding the threshold is encountered. Repeat the operation until all points and coordinates on the top of the forehead are found. The point on the top of the forehead is smoothly transitioned using cubic spline interpolation.

[0014] Preferably, Gaussian blurring is used to process the boundary transition.

[0015] On the other hand, the present invention also provides a general three-dimensional face model UV mapping generation system based on two-dimensional facial images, comprising: The acquisition module is used to acquire two-dimensional face images; The base texture acquisition module is used to acquire the base UV map of a standard skin tone; The first extraction module is used to select the key facial UV points of the base UV map; The second extraction module is used to extract the two-dimensional facial key points of the two-dimensional face image based on a deep learning algorithm; The alignment module is used to align UV facial key points with 2D facial key points using thin spline transformation to obtain preliminary facial deformation results. The smoothing module is used to smooth the initial facial deformation results using spline interpolation to obtain a smoothed facial contour. The texture module is used to apply the smoothed facial contours onto the base UV map and perform boundary transition processing to obtain the 3D face model UV map.

[0016] Thirdly, the present invention also provides a readable storage medium, wherein the computer program, when executed, can implement the above-described method for generating a general three-dimensional face model UV map based on a two-dimensional facial image.

[0017] As can be seen from the above technical solutions, compared with the prior art, this invention discloses a method, system, and storage medium for generating a general 3D face model UV map based on 2D facial images. It extracts 2D facial key points using deep learning algorithms, ensuring the accuracy of key point localization; employs thin spline transformation for key point alignment, better preserving the topological structure of facial features, making the initial deformation result more closely resemble the shape of a real face; and combines spline interpolation for smoothing, ensuring the continuous smoothness of the facial contour, effectively avoiding harsh edges, and improving the visual effect of the UV map. Furthermore, for areas that are difficult to process, such as the forehead, this invention ensures the natural integrity of the forehead area through precise color threshold judgment and spline interpolation transition, solving the problem of poor processing results for some facial areas in traditional methods. Further, this invention employs Gaussian blur for boundary transition processing, allowing the smoothed facial contour to blend naturally with the basic UV map, reducing stitching marks, and further ensuring the consistency and realism of the generated 3D face model texture. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 The overall flowchart provided for this invention.

[0020] Figure 2 The UV structure of the metahuman face.

[0021] Figure 3 This represents the distribution of key facial features in a two-dimensional human face.

[0022] Figure 4 Artificially pre-mixed facial skin tone.

[0023] Figure 5 Obtain the facial region for graphic transformation.

[0024] Figure 6 This is a diagram showing the effect without boundary transition.

[0025] Figure 7 This is a rendering of the result after the boundary transition.

[0026] Figure 8 This is a diagram showing the actual test results. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The core of this patent lies in the combination of deep learning computer vision algorithms and traditional image processing, with manual design serving as an auxiliary method. Taking the MetaHuman model as an example, as follows: Figure 2 The image shows the UV structure of the head, from which the inner and outer corners of the eyes, the tip of the nose, and the corners of the mouth can be clearly identified. Figure 3 This involves defining the regions of key facial features in a 2D human face and assigning an index number to each feature. The overall process is as follows: Figure 1 First, you need to manually select the current UV, that is... Figure 2 First, the positions of the inner and outer corners of the eyes, the tip of the nose, and the corners of the mouth are identified and their coordinates are recorded. Second, a deep learning-based facial landmark detection algorithm is used to extract all facial landmarks of the current 2D face and record their coordinates. Third, based on traditional image processing algorithms, a thin spline transformation is used to align the selected landmark positions and complete the approximate deformation of the face. Fourth, based on the positions of all the deformed landmarks, the facial area is extracted. To avoid sharp facial angles, a smooth transition method is used to add smoothing points to achieve an approximate elliptical face shape. Finally, since directly merging the face with the base facial texture will produce obvious edges, resulting in an appearance similar to a "monkey face," the edges need to be blurred and faded to obtain a 3D facial UV map that meets the requirements.

[0029] This invention discloses a general method for generating UV maps of 3D face models based on 2D facial images, comprising: Two-dimensional face images are acquired through photography, and the photography rules are as follows: The camera and the head are on the same horizontal line, and the camera is at a 90° angle to the ground.

[0030] The subject should not have any facial expressions (such as smiling), should close their mouth, relax naturally, open their eyes, and complete the photo shoot.

[0031] The subject should not have any face obstruction, such as glasses, masks, or hats, and their forehead should be exposed.

[0032] Obtain the base UV map for standard skin tone; Select the key points of the face from the base UV map.

[0033] Because skin color varies under different lighting conditions, this scheme uses an artificially set base color tone, namely the yellow skin tone of Asians, to standardize skin tone, as follows: Figure 4 As shown, due to the unavailability of ear features from a frontal facial photograph, the shape, color, and other characteristics of the ears were pre-created manually. Facial information was removed using image editing software, while preserving the overall color and style of the head.

[0034] Since the distribution of facial features in most existing or custom UV formats does not match the distribution of facial features in real facial images, it is necessary to align the facial features of a human face image to the positions of facial features on a standard UV. Actual verification has shown that it is necessary to manually select the approximate positions of the facial features and record the corresponding coordinate values.

[0035] Two-dimensional facial key points are extracted from the two-dimensional face image based on deep learning algorithms.

[0036] Typically, based on existing publicly available 2D facial landmark detection algorithms, CNN deep learning is used to test the accuracy of key point extraction on the actual subject's face. If the actual effect is poor, it is necessary to manually collect and label facial landmarks in batches to train the algorithm. Finally, the trained model is used to infer the facial images, obtain the key points of the subject's face, and record the coordinates.

[0037] Thin spline transformation is used to align UV facial key points with 2D facial key points to obtain preliminary facial deformation results; Spline interpolation is used to smooth the initial facial deformation results, resulting in a smoothed facial contour. The smoothed facial contours are applied to the base UV map, and boundary transition processing is performed to obtain the 3D face model UV map.

[0038] Thin Plate Spline (TPS) is a commonly used interpolation transformation technique, particularly suitable for the distortion and deformation of geometric shapes. TPS is very common in computer vision, image processing, and 3D deformation because it can perform a globally smooth transformation of a point set while preserving local smoothness.

[0039] The name Thin Template Transform (TPS) comes from a classic physics problem: minimizing the bending energy of a thin metal plate. Geometrically, TPS is used to transform one set of points into another while preserving smooth deformation properties. TPS can pass precisely through known points (control points) while performing smooth interpolation elsewhere.

[0040] Thin spline transformation is used to align UV facial key points with 2D facial key points, resulting in preliminary facial deformation results, including: Establish a one-to-one mapping relationship between UV key points and 2D face key points; Constructing the transformation function for thin spline transformation: ; in, These are the coefficients of the affine transformation; ; It is a radial basis function (RBF), where It is Euclidean distance; Key points of two-dimensional face and corresponding UV key points The distance between them.

[0041] The basic idea of ​​this formula is: for each point An effect is exerted on the entire plane, and this effect is transmitted through the radial basis functions. It spreads. Radial basis functions The form ensures that the deformation is stronger when the points are close together and gradually weakens when the points are far apart. Affine section It provides a global linear transformation, while the weights provide nonlinear distortions in detail.

[0042] A key characteristic of the thin strip transformation is that it selects the optimal solution by minimizing the bending energy. The specific bending energy is given by the following formula:

[0043] By minimizing this energy function, TPS ensures that the difference results are "smooth".

[0044] To determine the specific form of the thin template transformation, the affine coefficients need to be solved. and weight This is achieved by solving a system of linear equations. The dimension of this system of equations is related to the number of control points, and is typically of size 1. This includes: Location of control points; Boundary conditions ensure the optimal combination of affine transformation and nonlinear deformation.

[0045] The general form of the system of equations is the formula:

[0046] in: It is a radial basis function matrix. ; It is an affine partial matrix containing the coordinates of points; It is a weight vector. y is the affine coefficient vector, and y is the coordinate value of all target control points.

[0047] Through the above transformation process, the coordinate points manually selected on the UV standard format are mapped to the key points of the face to be tested by the algorithm, the face is adjusted to the approximate area, and all key points of the face are subjected to matrix operation according to the same transformation relationship to obtain the new coordinates of the key points after the face adjustment, and recorded.

[0048] Spline interpolation is a mathematical method for data fitting that uses a smooth curve formed by concatenating multiple low-order polynomials to interpolate between functions or data points. Spline functions exhibit high smoothness at each node (typically with continuous first or second derivatives). Cubic spline interpolation is the most commonly used. It uses a cubic polynomial to fit the data within each interval and ensures the continuity of the function and its first and second derivatives at each node.

[0049] In step four above, the adjusted face image and corresponding facial key point coordinates were obtained. Cubic spline interpolation was used to add multiple points between each key point, making the facial area smooth and without sharp angles. Since there were no corresponding key points for the forehead area, forehead points were added based on the existing key points. The specific process is as follows: Obtain the coordinates of all points in the upper half of the eyebrow region of the original facial key points; Using each coordinate as a reference, color calculation is performed upwards towards the forehead. When a point with a darker color is encountered, the calculation stops, and this position is considered to be the hair area. Repeat the above steps until the coordinates of all the tops of the foreheads are found; Using cubic spline interpolation, these points are smoothly transitioned, and all points are recorded.

[0050] At this point, all facial landmarks have been acquired and are smoothed. Using traditional image processing methods for irregular region segmentation, all these landmarks are binarized as irregular regions. Then, the extracted binary facial segmentation image is subjected to the same TPS transform (using the same transform matrix as the original face) to obtain a new binary facial image. Finally, this binary image is multiplied by the TPS-transformed ground truth facial image to obtain the facial region. The flowchart above is as follows. Figure 5 As shown.

[0051] Through the above steps, a base UV map of a standard skin tone can be obtained, along with a cropped image of the face area obtained after transformation. The face image needs to be pasted onto the base UV map. If it's pasted directly without any processing, a "monkey face" effect will occur, meaning the transition between the two images is unnatural, visually resembling a monkey's face. To avoid this, a Gaussian transition processing is needed at the boundary, specifically applying Gaussian blur to the seam to make the transition between the two images more natural. This is done using Gaussian blur, a traditional image processing technique, with the Gaussian kernel size modified to fit the current base UV map size.

[0052] After setting the Gaussian blur, the connection points between the base UV map and the face are processed to make the blur more natural. The actual effect is as follows. Figure 6 and Figure 7 As shown, where Figure 6 This is the effect of not performing a boundary transition. Figure 7 This is the effect after boundary transition. The image shows the chin area of ​​a face. The bottom left side of both images is the base UV map, and the top right side is the face area to be tested. It can be clearly compared that the boundary transition plays an important role in the generation of face maps.

[0053] as follows Figure 8 As shown, this is the effect of converting the face under test into the UV standard mode after the above processing. To protect the privacy of the test subjects, the key parts of the test results have been blurred. This solution can automatically generate face textures in batches, and at the same time, visualize and intuitively feel the test effect, and directly apply them to the standard 3D human head, so that the head texture is close to the actual face under test.

[0054] On the other hand, the present invention also provides a general three-dimensional face model UV mapping generation system based on two-dimensional facial images, comprising: The acquisition module is used to acquire two-dimensional face images; The base texture acquisition module is used to acquire the base UV map of a standard skin tone; The first extraction module is used to select the key facial UV points of the base UV map; The second extraction module is used to extract the two-dimensional facial key points of the two-dimensional face image based on a deep learning algorithm; The alignment module is used to align UV facial key points with 2D facial key points using thin spline transformation to obtain preliminary facial deformation results. The smoothing module is used to smooth the initial facial deformation results using spline interpolation to obtain a smoothed facial contour. The texture module is used to apply the smoothed facial contours onto the base UV map and perform boundary transition processing to obtain the 3D face model UV map.

[0055] Thirdly, the present invention also provides a readable storage medium, wherein the computer program, when executed, can implement the above-described method for generating a general three-dimensional face model UV map based on a two-dimensional facial image.

[0056] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0057] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A general method for generating UV maps of 3D human face models based on 2D facial images, characterized in that, include: Obtain a 2D face image; Obtain the base UV map for standard skin tone; Select the key UV points for the face from the base UV map; Two-dimensional facial key points are extracted from the two-dimensional face image based on deep learning algorithms; Thin spline transformation is used to align UV facial key points with 2D facial key points to obtain preliminary facial deformation results; Spline interpolation is used to smooth the initial facial deformation results, resulting in a smoothed facial contour. The smoothed facial contours are applied to the base UV map, and boundary transition processing is performed to obtain the 3D face model UV map.

2. The method for generating a general 3D face model UV map based on a 2D facial image according to claim 1, characterized in that, Thin spline transformation is used to align UV facial key points with 2D facial key points, resulting in preliminary facial deformation results, including: Establish a one-to-one mapping relationship between UV key points and 2D face key points; Constructing the transformation function for thin spline transformation: ； in, These are the coefficients of the affine transformation; It is the contribution weight of the i-th control point to the deformation in the x-direction; It is a radial basis function; Key points of two-dimensional face and corresponding UV key points The distance between them; The coefficients and weights of the affine transformation are solved by solving a system of linear equations, which are: ； Where K is the radial basis function matrix, P is the affine partial matrix; It is a weight vector. y is the affine coefficient vector; y is the coordinate value containing all target control points. Substitute the obtained affine coefficients and weights into the transformation function, apply the transformation function to the two-dimensional facial key points, and obtain the corresponding positions of each two-dimensional facial key point in the UV facial region.

3. The method for generating a general 3D face model UV map based on a 2D facial image according to claim 1, characterized in that, Spline interpolation is used to smooth the initial facial deformation results, resulting in a smoothed facial contour, including: Extract aligned key points from the initial facial deformation results and obtain the corresponding two-dimensional coordinates and pixel color values; The facial region is divided into multiple continuous sub-regions; For each sub-region, a cubic spline function satisfying the continuity of the first and second derivatives is constructed using the coordinates of key points as nodes; The solution is obtained by satisfying the conditions that the function values ​​at the endpoints of adjacent intervals are equal, the first derivative is equal, the second derivative is equal, and the boundary conditions are met. Based on the cubic spline function, the coordinates of the transition points are calculated between adjacent key points in each sub-region at a preset step size, so that a continuous smooth curve is formed between adjacent key points. The acquired key points and transition points are binarized according to sub-regions to obtain a face segmentation binary image. The extracted binary image of the face segmentation is subjected to the same thin spline transformation to obtain a new binary image of the face. Finally, the new binary image of the face is multiplied with the result of the initial facial deformation to obtain the facial region.

4. The method for generating a general 3D face model UV map based on a 2D facial image according to claim 3, characterized in that, The smoothing steps for the forehead area include: Obtain the coordinates of all 2D facial key points in the upper half of the eyebrow region; Using each coordinate as a reference, calculate the color upwards from the forehead. Stop when a point with a color exceeding the threshold is encountered. Repeat the operation until all points and coordinates on the top of the forehead are found. The point on the top of the forehead is smoothly transitioned using cubic spline interpolation.

5. The method for generating a general three-dimensional face model UV map based on a two-dimensional facial image according to claim 1, characterized in that, Gaussian blurring is used to process the boundary transition.

6. A general-purpose 3D face model UV mapping generation system based on 2D facial images, characterized in that, include: The acquisition module is used to acquire two-dimensional face images; The base texture acquisition module is used to acquire the base UV map of a standard skin tone; The first extraction module is used to select the key facial UV points of the base UV map; The second extraction module is used to extract the two-dimensional facial key points of the two-dimensional face image based on a deep learning algorithm; The alignment module is used to align UV facial key points with 2D facial key points using thin spline transformation to obtain preliminary facial deformation results. The smoothing module is used to smooth the initial facial deformation results using spline interpolation to obtain a smoothed facial contour. The texture module is used to apply the smoothed facial contours onto the base UV map and perform boundary transition processing to obtain the 3D face model UV map.

7. A readable storage medium, characterized in that, When the computer program is executed, it can realize a general three-dimensional face model UV mapping generation method based on two-dimensional facial images according to any one of claims 1-5.

Images