Head three-dimensional model generation method based on 2D facial expression analysis

Abstract

The invention discloses a 3D head model generation method based on 2D facial expression analysis. The method comprises four steps of expression analysis, shielding object processing, multi-angle image generation and 3D model optimization. Firstly, expression parameters are extracted through an expression recognition model, and expressions are neutralized to generate expression-free images. Thirdly, automatically identifying and removing a shielding object by utilizing an image segmentation and deep learning inviting technology, and keeping the continuity of facial textures; then, a multi-angle view is generated from the non-shielded neutral expression image, and the consistency of the geometry and texture of the face is ensured; and finally, optimizing the shape and the chartlet of the 3D model by adopting a differentiable rendering technology to realize high consistency with the 2D image, thereby obtaining a high-precision and high-fidelity 3D head model. According to the multi-angle optimization method, the precision and consistency of 3D model generation are improved by combining differentiable rendering and deep learning. Geometric distortion is reduced through neutral expression generation, shielding objects are automatically removed, and the precision of face reconstruction is effectively improved. And the practical value is very high.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing, and in particular to a method for generating a three-dimensional head model based on 2D facial expression analysis. Background Art

[0002] With the increasing demand for 3D reconstruction of faces in virtual reality (VR), augmented reality (AR) and film and television entertainment, efficient and high-precision 3D head model generation technology has become increasingly important under limited computing resources. However, existing technologies are unable to efficiently generate 3D head models using only 2D facial images with expressions, especially when dealing with occluders (such as hair, glasses, and beards). Existing technologies are unable to accurately remove occluders while preserving the details of the facial features and generate neutral expression images suitable for different angles. This directly affects the quality of subsequent 3D model generation.

[0003] At present, with the rapid development of computer image processing technology, the generation and application of 3D models have become an important part of many fields. In particular, in the fields of virtual reality (VR), augmented reality (AR), film and television entertainment, game development, medical imaging and remote communication, the demand for 3D character modeling is increasing. In the face of these application scenarios, how to efficiently and accurately generate 3D character head models from 2D images has become a technical research focus in this field.

[0004] In existing 2D to 3D face modeling research, the common practice is to use deep learning technology to convert images to geometric shapes. These technologies map the input 2D image to depth information or 3D geometric structure by building a neural network. Although some success has been achieved in the rough reconstruction of textureless 3D geometric figures, traditional 2D to 3D conversion technology still faces many challenges in generating high-fidelity real face 3D models, especially in the face of complex situations such as different expression changes, occlusions (such as glasses, hair), and multi-angle matching. Specifically, the existing technology has shortcomings in dealing with the following problems:

[0005] 1. Problems in dealing with expression changes:

[0006] In most daily portraits, the characters often have some kind of expression (such as smiling, frowning, surprise, etc.), while in 3D character modeling and many practical applications, a 3D head model with a neutral expression is generated. Therefore, generating a 3D model with a neutral expression from a 2D image with an expression has become a technical challenge. Existing technologies usually rely on directly generating depth information, but do not effectively solve how to automatically neutralize complex and diverse facial expressions. Using images with expressions for 3D modeling may lead to deviations in facial geometry, especially when the character's expression is drastic, this error may even affect the overall geometric accuracy of the model.

[0007] 2. The influence of obstructions:

[0008] In actual shooting scenes, people often wear glasses, hats, or have occluders such as hair and beards. These occluders interfere with the geometric information of different areas of the face, causing serious errors in extracting facial geometry directly from 2D images. For example, glasses may affect the geometric accuracy of the eyes, hair may obstruct the head contour, and beards may cover the mouth and chin areas. When dealing with occluders, existing technologies usually rely on manual removal or a one-size-fits-all approach to remove occluded areas, resulting in the generated 3D model being lacking in the restoration of facial details. How to automatically identify and remove these occluders, and still be able to restore high-fidelity, seamless facial textures after removing the occluded areas, has become a problem that needs to be solved urgently.

[0009] 3. Disadvantages of multi-view imaging:

[0010] The generation of 3D head models usually requires views from different angles to restore the true geometry of the head. In traditional reconstruction techniques such as photogrammetry and structured light, this problem can be solved by dedicated equipment, but most 2D images (such as ordinary portrait photos) only contain one view of the head. Existing 2D to 3D conversion techniques have limited recovery of geometric details in single-view images, and the generated models often appear rough and distorted in 3D structure. In addition, input images from a single angle may have ambiguous information (especially under complex lighting conditions), which will affect the accuracy of the model. Therefore, it is a challenging task to generate a 3D head model with high-precision geometric details based on a single-view image.

[0011] 4. Problems in combining model generation with texture generation:

[0012] In existing 2D image-based 3D modeling technologies, the generated geometry and texture maps are often separated. The model shape and map texture need to be optimized separately, which can easily lead to insufficient geometry and texture fit of the final generated model at different viewing angles, resulting in poor visual effects. Achieving joint optimization of geometry models and maps, as well as generating high-quality, high-fidelity 3D maps, remains a technical bottleneck in this field.

[0013] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0014] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a method for generating a three-dimensional head model based on 2D facial expression analysis.

[0015] To achieve the above object, the present invention adopts the following technical solutions:

[0016] A method for generating a three-dimensional head model based on 2D facial expression analysis comprises the following steps:

[0017] S1: Use the expression recognition model to analyze the expression of the input 2D face image and extract the expression parameters; according to the extracted expression parameters, use the expression editing model to neutralize the expression in the input image and generate a neutral expression image;

[0018] S2: Combine image segmentation algorithms to automatically identify occluded areas in neutral expression images; use deep learning-based inpainting technology to remove occluded areas and automatically fill in the removed areas to maintain consistency of facial texture;

[0019] S3: Based on the unobstructed neutral expression image, the face image generation model is used to generate multiple neutral expression images at different angles to ensure the continuity and consistency of the facial geometry and texture under different viewing angles;

[0020] S4: Combined with multi-view neutral expression images, the shape parameters and textures of the parameterized 3D head model are reversely optimized using differentiable rendering technology; by iteratively optimizing the projection error, the generated 3D head model is ensured to be consistent with the 2D images under different viewpoints, obtaining high-precision geometric shapes and high-fidelity textures.

[0021] Furthermore, step S1 specifically includes:

[0022] Use the expression parameter model to analyze the input 2D face image and output an expression vector that describes the activation intensity of each action unit on the face.

[0023] Based on the expression vector, the original expression image is edited using the expression editing model to generate a neutral expression image;

[0024] The generated neutral expression image is re-detected through the expression recognition model. If the difference between the weight of any action unit and the neutral expression vector exceeds the preset threshold, the image is re-input into the expression editing model for iterative optimization until the generated image is close to the neutral expression.

[0025] Furthermore, step S2 specifically includes:

[0026] The neutral expression image is processed using an image segmentation algorithm to generate a segmentation mask that marks each pixel to identify the occluded area;

[0027] According to the marking result of the segmentation mask, the color value of the pixel identified as the occluder in the image is set to zero, and the corresponding occluder mask is created;

[0028] The occluder mask and image are input into the deep learning image generation model, and the inpainting function of the model is used to remove the occluder and intelligently fill in the original occluded position to maintain the continuity and consistency of the facial texture.

[0029] Furthermore, step S3 specifically includes:

[0030] Using the human-image generation model, neutral expression images of multiple angles are generated from unobstructed neutral expression images;

[0031] Apply perspective transformation and geometric deformation techniques to generate facial images at multiple angles from a standard image at a single angle;

[0032] This ensures that the images generated at different viewing angles maintain neutral expression features and maintain the continuity and consistency of facial geometry and texture.

[0033] Furthermore, the person-image generation model is an image generation model based on a generative adversarial network (GAN) or a stable diffusion (StableDiffusion).

[0034] Furthermore, the facial images at multiple angles include viewing angles from left, right, top, bottom, and 15, 30, and 45 degrees.

[0035] Furthermore, step S4 specifically includes:

[0036] Using the multi-view, neutral, unobstructed 2D neutral expression image generated in step S3, using differentiable rendering technology to reversely iterate and optimize the 3D parameterized model to generate a 3D head model and texture;

[0037] Randomly initialize the shape parameters and texture of the 3D model, extract the vertex positions from the 3D head model, and adjust the vertex positions according to the shape parameters;

[0038] The 3D model is rendered to the viewing angle corresponding to the multi-angle 2D image, and iterative optimization is performed to minimize the difference between the rendered image and the actual 2D image.

[0039] Furthermore, the iterative optimization includes calculating a loss function, which is the sum of squares of differences between the rendered images of all viewing angles and the corresponding 2D images, wherein the difference between the rendering result of each viewing angle and the actual 2D image is measured by a Euclidean distance.

[0040] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a three-dimensional head model based on 2D facial expression analysis is implemented.

[0041] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for generating a three-dimensional head model based on 2D facial expression analysis is implemented.

[0042] The present invention has the following beneficial effects:

[0043] The present invention proposes a method for generating a 3D head model with full-process optimization, which is particularly suitable for 2D face images in complex scenes, including images with expressions and facial occlusions. This method generates a high-precision 3D head model from a single frame or a small number of 2D images through automated image processing techniques, such as expression recognition and neutralization, multi-view generation, automatic recognition and removal of occlusions, and differentiable rendering optimization, significantly improving modeling efficiency and accuracy. This process effectively addresses the facial geometric distortion caused by changes in expression, automatically removes occlusions to reduce interference with modeling, and generates high-precision image templates from multiple angles, providing multiple guarantees for 3D reconstruction.

[0044] The present invention first analyzes the input 2D portrait image with expression through the expression recognition model, extracts the expression parameters to obtain the neutral expression, and then uses the expression editing model to automatically neutralize the expression to generate an expressionless image. Next, the system automatically identifies and marks the occluded areas, such as glasses, hair and beard, in combination with the image segmentation algorithm, and uses the inpainting technology based on deep learning to intelligently remove these occluders while maintaining the consistency of facial texture. After processing the occluders, the system uses the face image generation model to generate multiple images at different angles from the unobstructed neutral expression image to ensure the continuity and consistency of the facial geometry and texture under different viewing angles. Finally, the system uses differentiable rendering technology to reversely optimize the shape parameters and mapping of the parameterized 3D head model, and continuously iterates and optimizes the projection error to ensure that the generated 3D head model is consistent with the 2D image under multiple viewing angles, achieving high-precision geometry and high-fidelity mapping.

[0045] The significant innovation of the present invention is that it provides a multi-angle optimization method, which combines differentiable rendering with deep learning to improve the accuracy and consistency of 3D model generation. This method reduces the geometric distortion caused by expression changes and occlusions, and improves the accuracy of facial reconstruction. The fully automated system does not rely on specific hardware and is suitable for multiple fields such as virtual reality, game development, film and television production, and medical imaging, and has strong practical application value. In addition, the efficiency and versatility of the method of the present invention make it suitable for a variety of scenarios. Users only need to provide conventional 2D photos to generate reliable 3D models. It is suitable for virtual characters, game development, medical imaging and other fields. It is a fully automated calculation process that improves generation efficiency and accuracy. It is suitable for a variety of application scenarios such as large-scale image processing and virtual portrait generation.

[0046] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a method for generating a three-dimensional head model based on 2D facial expression analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0049] Existing technologies often encounter two core problems when generating 3D head models from 2D portrait photos: First, in most daily photos, people usually have various expressions, but in the actual application of 3D head model generation, it is often necessary to generate 3D models with neutral expressions. This poses a challenge to traditional 3D model generation methods, because these methods usually have difficulty in handling complex and diverse expression changes, and cannot directly generate accurate neutral expression 3D models from 2D photos with expressions; second, in many images, people wear glasses or have facial occlusions such as beards, which will interfere with the accurate capture of facial geometric features, thereby affecting the generation of 3D models.

[0050] Therefore, the present invention aims to propose an innovative process, starting from overcoming these two major problems, and providing an optimized 3D head model generation method for 2D face images with expressions and occlusions. Through expression recognition and neutralization processing, automatic recognition and removal of occlusions (such as glasses, hair, beards, etc.), and generation of neutral unobstructed face images from multiple angles, and further generation of 3D head models, this method can solve the challenges of expression and occlusion to model generation while maintaining the facial details and geometric features of the person.

[0051] See also Figure 1 The embodiment of the present invention provides a method for generating a three-dimensional head model based on 2D facial expression analysis, comprising the following steps:

[0052] Step S1: Automatic expression recognition and automatic expression neutralization: Use the expression recognition model to perform expression analysis on the input 2D face image and extract expression parameters; based on the extracted expression parameters, use the expression editing model to neutralize the expression in the input image and generate a neutral expression image.

[0053] In a preferred embodiment, step S1 specifically includes: using an expression parameter model to perform expression analysis on an input 2D face image, and outputting an expression vector that describes the activation intensity of each action unit on the face; based on the expression vector, using an expression editing model to edit the original expression image to generate a neutral expression image; re-detecting the generated neutral expression image through an expression recognition model, and if the difference between the weight of any action unit and the neutral expression vector exceeds a preset threshold, the image is re-input into the expression editing model for iterative optimization until the generated image is close to a neutral expression.

[0054] Step S2: Automatic identification and removal of occluders: Combine the image segmentation algorithm to automatically identify the occluder area in the neutral expression image; use the deep learning-based inpainting technology to remove the occluder and automatically fill the removed area to maintain the consistency of facial texture.

[0055] In a preferred embodiment, step S2 specifically includes: using an image segmentation algorithm to process the neutral expression image to generate a segmentation mask, which marks each pixel to identify the occluder area; according to the marking result of the segmentation mask, setting the color value of the pixel identified as the occluder in the image to zero, and creating a corresponding occluder mask; inputting the occluder mask and the image into a deep learning image generation model (such as a deep learning algorithm such as a convolutional neural network (CNN)), using the inpainting function of the model to remove the occluder, and intelligently filling the original occluded position to maintain the continuity and consistency of the facial texture.

[0056] This step combines image segmentation technology and deep learning inpainting technology to achieve accurate identification and effective removal of occluders, while maintaining the natural transition and consistency of facial texture, improving the accuracy and realism of 3D model generation.

[0057] Step S3: Automatic generation of multi-angle neutral expression images: Based on the unobstructed neutral expression image, a face image generation model is used to generate multiple neutral expression images at different angles to ensure the continuity and consistency of the facial geometry and texture under different viewing angles.

[0058] In a preferred embodiment, step S3 specifically includes: using a human-image generation model to generate neutral expression images at multiple angles from an unobstructed neutral expression image, which may include but is not limited to viewing angles from the left, right, top, bottom, and 15, 30, 45 degrees and the like; applying perspective transformation and geometric deformation technology to generate facial images at multiple angles from a standard image at a single angle; thereby ensuring that the images generated at different perspectives maintain neutral expression features and maintain the continuity and consistency of the facial geometry and texture.

[0059] This step maintains the consistency of lighting and appearance features by expanding single-angle images to multiple-angle images, while ensuring the maintenance of neutral expressions, providing rich input data for generating high-precision 3D head models.

[0060] In different embodiments, the person-image generation model may be an image generation model based on a generative adversarial network (GAN) or stable diffusion.

[0061] Step S4: Automatic generation and optimization of 3D avatar model: Combined with multi-view neutral expression images, differentiable rendering technology is used to reversely optimize the shape parameters and textures of the parameterized 3D head model; by iteratively optimizing the projection error, the generated 3D head model is ensured to be consistent with the 2D images under different viewpoints, and high-precision geometric shapes and high-fidelity textures are obtained.

[0062] In a preferred embodiment, step S4 specifically includes: using the multi-perspective, neutral, unobstructed 2D neutral expression image generated in step S3, using differentiable rendering technology to reversely iteratively optimize the 3D parameterized model to generate a 3D head model and texture; randomly initializing the shape parameters and texture of the 3D model, extracting the vertex positions from the 3D head model, and adjusting the vertex positions according to the shape parameters; rendering the 3D model to the perspective corresponding to the multi-angle 2D image, and iteratively optimizing to minimize the difference between the rendered image and the actual 2D image. The iterative optimization includes calculating a loss function, which is the sum of the squares of the differences between the rendered images of all perspectives and the corresponding 2D images, wherein the difference between the rendering result of each perspective and the actual 2D image is measured by the Euclidean distance.

[0063] This step accurately matches the input multi-angle 2D image through an iterative process of multi-view optimization and differentiable rendering, achieving refined geometry and high-fidelity mapping, thereby providing higher accuracy and consistency for the generation of 3D models.

[0064] The present invention proposes a full-process optimization method for 2D facial images with expressions and occluders, which provides a more efficient, accurate and compatible solution from image expression analysis, occluder recognition and removal, neutral expression generation, to multi-angle image generation and final 3D model generation.

[0065] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.

[0066] A 3D head model generation method based on 2D face images and expression analysis optimizes and generates a high-precision 3D head model with textures through a series of processing steps such as expression analysis, image segmentation, occlusion processing, and multi-angle image generation. This method can automatically generate an accurate 3D head model from 2D images with different expressions and facial occlusions, including texture optimization, which effectively solves the problem that the existing methods cannot generate high-precision 3D models due to interference from images with expressions and occlusions.

[0067] First, the input 2D portrait image with expression is analyzed through the expression recognition model, and the expression parameters are extracted to obtain the neutral expression of the person. Then, the expression is automatically neutralized using the expression editing model to generate an image with a neutral expression. Secondly, in order to reduce the interference of occlusions on the generation of 3D models, the system combines the image segmentation algorithm to automatically identify and mark the areas of occlusions (such as glasses, hair, and beards, etc.), and uses the inpainting technology based on deep learning for intelligent removal and filling, ensuring the consistency of facial texture after removing the occlusions.

[0068] After generating unobstructed neutral expression images, the system generates multiple neutral images from different angles based on these images, and uses facial image generation models (such as GAN or Stable Diffusion) to expand the multi-view generated images to ensure the consistency of facial geometry and texture when observed from different directions. Finally, combined with these multi-view neutral expression images, the system uses differentiable rendering technology to reversely optimize the shape parameters and textures of the parameterized 3D head model. By continuously iteratively optimizing the projection error, the generated 3D head model is ensured to be consistent with the 2D images from different perspectives, thereby obtaining high-precision geometry and high-fidelity textures.

[0069] The present invention provides an innovative multi-angle optimization method, which improves the accuracy and consistency of 3D model generation by combining differentiable rendering with deep learning. Neutral expression generation reduces geometric distortion, and automatic removal of occluders effectively improves the accuracy of facial reconstruction. The system is fully automated and does not rely on specific hardware. It is widely applicable to multiple fields such as virtual reality, game development, film and television production, medical imaging, etc., and has strong practical application value.

[0070] A method for generating a high-precision 3D head model from a 2D face image, specifically comprising Figure 1 The 4 steps shown are:

[0071] Step 1: Automatic expression recognition and automatic expression neutralization

[0072] 1.1 The system first uses the expression recognition model to analyze the input 2D face image and output the expression parameters. Specifically, for the input image I∈R H×W×C Expression parameter model f AU Convert the input image into a set of k-dimensional expression vectors n∈R k , which is used to accurately describe the activation intensity of each action unit (AU) on the face.

[0073] 1.2 Use the expression editing model to edit the original expression image according to the expression parameters and convert the original image into a neutral expression image. Specifically, given an input image I∈R with expression H×W×C And the expression vector n∈R obtained in the previous step 1.1 k , the expression editing model makes it close to neutral, and generates a new image without expression. The generated image will be re-detected by the expression recognition model. If the weight of any action unit in the output expression parameter vector is equal to the neutral expression vector n 0 If the difference exceeds a preset threshold ∈, the generated image is re-entered into the expression editing model for further optimization until the generated image is close to a neutral expression, that is:

[0074] InitializationI0 =I, for t=0,1,2,.. loop iterations:

[0075] I t+1 =E(I t ,n)

[0076] n t+1 =f AU (I t+1 )

[0077] Termination conditions:

[0078]

[0079] In the present invention, expression neutralization does not rely directly on a single expression deletion operation, but combines captured expression parameters to generate neutral expressions through image quality optimization and high-precision positioning to protect facial details and texture information.

[0080] Step 2: Automatic identification and removal of occluders

[0081] 2.1 Use the image segmentation algorithm to generate the neutral expression image Ineutral∈R in step 1.2 above H ×W×C , generate the segmentation mask (M∈R H×W ), which assigns a label m to each pixel i,j ∈{0,1,2,…,N}, mark the occluded area (such as hair, glasses, beard, etc.) pixel by pixel.

[0082] 2.2 According to the pixel marking in 2.1 above, the color value of the pixel marked as an occluder in the image is set to 0, and an image mask is generated for the occluder mark. The mask and the image are input into Stable Diffusion or other powerful deep learning-based image generation models, and the occluder is removed using its inpainting method. The original occluded position is intelligently filled to ensure the consistency of the generated facial texture.

[0083] Unlike existing methods for dealing with occlusions that rely on a single specific model, this paper proposes a dual solution based on image segmentation + intelligent generation. First, the segmentation model is used to accurately identify different types of occlusions, and then the inpainting model is used to intelligently fill in facial textures as needed, thus achieving a general process that can handle various complex situations without training.

[0084] Step 3: Automatically generate neutral expression images from multiple angles

[0085] 3.1 In this method, based on the generated unobstructed neutral expression image, the system further uses a human-image generation model (such as an image generation model based on GAN or Stable Diffusion) to generate neutral expression images from different angles. First, through the early expression recognition and occlusion processing, a standard image with no obstruction and facial expression is obtained as the basic input. Next, the image generation model will generate facial images from multiple angles based on the input image, using perspective transformation and geometric deformation technology, such as images observed from the left, right, top, bottom, 15, 30, 45 degrees and other directions. The key to this step is to ensure that the generated image maintains neutral expression features while ensuring the continuity and consistency of facial geometry and texture under different perspectives. These expressionless images from different angles will provide more data support for the subsequent high-precision generation of 3D head models.

[0086] In the present invention, an image generation model is used to amplify an image from a single angle into images from multiple angles, while maintaining the consistency of lighting, appearance features and neutral expressions, providing richer input for subsequent high-precision 3D head model generation.

[0087] Step 4: Automatically generate and optimize 3D avatar models

[0088] 4.1 Multi-view, neutral, unobstructed 2D neutral expression image I generated according to step 3.1 (i) ∈R H×W×C , where i = 1, 2, ..., N, this method uses differentiable rendering to reversely iterate and optimize the 3D parameterized model to generate the 3D head model and texture. Specifically, first randomly initialize the shape parameter β and the texture I tex , extract vertex positions from the 3D head model, Then render the 3D model to the angle corresponding to each image in step 3.1, and then iterate and optimize to minimize the difference between the projected image and the multi-angle image generated in step 3.1. The optimization function is:

[0089]

[0090] Where T (i) is the perspective transformation matrix, R(V,T (i) ) is the rendering function.

[0091] In the present invention, through the above steps, a single image is expanded into a neutral expression image from multiple angles, so that there is enough information for differentiable rendering and reverse optimization. This iterative process based on multi-view optimization and differentiable rendering helps to more accurately match the input image, achieve delicate geometric shapes and high-fidelity mapping, and provide higher accuracy and consistency for the generation of 3D models.

[0092] In summary, the present invention provides a method for generating a three-dimensional head model based on 2D facial expression analysis, which can generate a high-precision 3D head model from a single frame or a small number of 2D facial images with expressions through a full-process automated method. The method effectively reduces the facial geometric distortion caused by expression changes through expression recognition and neutralization processing, and improves the accuracy of the shape and expression restoration of the 3D model. The automatic removal of occluders, such as glasses, hair and beards, eliminates the interference of these occluders on the facial geometric structure, reduces reconstruction errors, and improves the accuracy of facial area restoration. In addition, the multi-angle image generation technology is used, combined with the differentiable rendering technology, and the shape and mapping of the 3D model are reversely iterated to optimize, which significantly improves the authenticity of the geometric details and texture of the 3D model. This method not only improves the efficiency and accuracy of modeling, but also has a wide range of applicability. It can be applied to multiple fields such as virtual reality, game development, film and television production, medical imaging, etc. without relying on specific hardware, showing strong practical application value.

[0093] Compared with the traditional technology, the main advantages of the present invention are:

[0094] The present invention effectively improves the accuracy and consistency of 3D head model generation. First, the use of neutral expression to generate images greatly reduces the geometric distortion caused by expressions, making the reconstructed 3D model more accurate in shape and expression restoration. By automatically removing occluders (such as glasses, hair and beards) in the image, the interference of these occluders on the facial geometry is eliminated, the reconstruction error caused by hair and glasses occlusion is greatly reduced, and the accuracy of facial area restoration is improved. At the same time, the system is based on neutral images generated from multiple angles, combined with differentiable rendering technology, to reversely iterate and optimize the shape and mapping of the 3D model. By projecting the model from different perspectives onto a 2D image matrix, this method effectively reduces the information ambiguity and blur caused by a single perspective, thereby generating a higher quality, detail-preserving 3D head model. These can be extended to 3D image restoration in virtual reality, film and television production and medical fields.

[0095] The method of the present invention is both efficient and versatile, and is widely applicable to a variety of scenarios. Without relying on specific camera equipment, users only need to provide conventional 2D photos to generate reliable 3D models, which is applicable to the fields of virtual characters, game development, medical imaging, etc. The present invention is a fully automated calculation process, which improves the generation efficiency and accuracy, and is applicable to a variety of application scenarios such as large-scale image processing and virtual portrait generation.

[0096] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0097] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0098] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0099] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0100] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0101] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0102] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0103] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0104] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0105] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0106] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0107] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0108] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for generating a three-dimensional head model based on 2D facial expression analysis, characterized in that: The following steps are involved: S1: Use the expression recognition model to analyze the expression of the input 2D face image and extract the expression parameters; according to the extracted expression parameters, use the expression editing model to neutralize the expression in the input image and generate a neutral expression image; S2: Combine image segmentation algorithms to automatically identify occluded areas in neutral expression images; use deep learning-based inpainting technology to remove occluded areas and automatically fill in the removed areas to maintain consistency of facial texture; S3: Based on the unobstructed neutral expression image, the face image generation model is used to generate multiple neutral expression images at different angles to ensure the continuity and consistency of the facial geometry and texture under different viewing angles; S4: Combined with multi-view neutral expression images, the shape parameters and textures of the parameterized 3D head model are reversely optimized using differentiable rendering technology; by iteratively optimizing the projection error, the generated 3D head model is ensured to be consistent with the 2D images under different viewpoints, obtaining high-precision geometric shapes and high-fidelity textures.

2. The method for generating a three-dimensional head model based on 2D facial expression analysis according to claim 1, characterized in that: Step S1 specifically includes: Use the expression parameter model to analyze the input 2D face image and output an expression vector that describes the activation intensity of each action unit on the face. Based on the expression vector, the original expression image is edited using the expression editing model to generate a neutral expression image; The generated neutral expression image is re-detected through the expression recognition model. If the difference between the weight of any action unit and the neutral expression vector exceeds the preset threshold, the image is re-input into the expression editing model for iterative optimization until the generated image is close to the neutral expression.

3. The method for generating a three-dimensional head model based on 2D facial expression analysis as claimed in claim 1 or 2, characterized in that: Step S2 specifically includes: The neutral expression image is processed using an image segmentation algorithm to generate a segmentation mask that marks each pixel to identify the occluded area; According to the marking result of the segmentation mask, the color value of the pixel identified as the occluder in the image is set to zero, and the corresponding occluder mask is created; The occluder mask and image are input into the deep learning image generation model, and the inpainting function of the model is used to remove the occluder and intelligently fill in the original occluded position to maintain the continuity and consistency of the facial texture.

4. The method for generating a three-dimensional head model based on 2D facial expression analysis according to any one of claims 1 to 3, characterized in that: Step S3 specifically includes: Using the human-image generation model, neutral expression images of multiple angles are generated from unobstructed neutral expression images; Apply perspective transformation and geometric deformation techniques to generate facial images at multiple angles from a standard image at a single angle; This ensures that the images generated at different viewing angles maintain neutral expression features and maintain the continuity and consistency of facial geometry and texture.

5. The method for generating a three-dimensional head model based on 2D facial expression analysis as claimed in claim 4, characterized in that: The human-image generation model is an image generation model based on a generative adversarial network (GAN) or stable diffusion (Stable Diffusion).

6. The method for generating a three-dimensional head model based on 2D facial expression analysis as claimed in claim 4 or 5, characterized in that: The facial images at multiple angles include viewing angles from left, right, top, bottom, and 15, 30, and 45 degrees.

7. The method for generating a three-dimensional head model based on 2D facial expression analysis according to any one of claims 1 to 6, characterized in that: Step S4 specifically includes: Using the multi-view, neutral, unobstructed 2D neutral expression image generated in step S3, using differentiable rendering technology to reversely iterate and optimize the 3D parameterized model to generate a 3D head model and texture; Randomly initialize the shape parameters and texture of the 3D model, extract the vertex positions from the 3D head model, and adjust the vertex positions according to the shape parameters; The 3D model is rendered to the viewing angle corresponding to the multi-angle 2D image, and iterative optimization is performed to minimize the difference between the rendered image and the actual 2D image.

8. The method for generating a three-dimensional head model based on 2D facial expression analysis according to claim 7, characterized in that: The iterative optimization includes calculating a loss function, which is the sum of squares of differences between rendered images of all viewing angles and corresponding 2D images, wherein the difference between the rendering result of each viewing angle and the actual 2D image is measured by a Euclidean distance.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a three-dimensional head model based on 2D facial expression analysis according to any one of claims 1 to 8 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a three-dimensional head model based on 2D facial expression analysis as described in any one of claims 1 to 8 is implemented.

Images