Three-dimensional deformable face model generation method and system and electronic equipment

Abstract

The invention discloses a three-dimensional deformable face model generation method and system and electronic equipment, and belongs to the field of three-dimensional face modeling, and the method mainly comprises the following steps: S1, collecting a face video, and carrying out the preprocessing; s2, based on the preprocessed face video, generating a high-precision three-dimensional face model by using a neural implicit surface method; and S3, converting the high-precision three-dimensional face model into a parameterized statistical model to support real-time interaction. According to the precise model generation method based on the neural implicit surface (NeuS), a high-precision model with a large number of high-frequency details and a high sense of reality can be generated based on a video. Besides, the invention also innovatively provides a generation method of the parameterized statistical model based on physical rendering, the parameterized statistical model can be generated based on a high-precision model, and the model with a large number of high-frequency details and high reality can perform real-time interaction.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional face modeling, and in particular to a method, system and electronic device for generating three-dimensional deformable face models. Background Technology

[0002] The core goal of 3D face modeling is to transform real human faces into digital 3D models to support applications such as film and television special effects, virtual reality, and human-computer interaction. Early methods relied on manual modeling or specialized instruments, but these were costly and inefficient. With the development of multi-view geometry and deep learning, parametric statistical models (3DMM) and neural implicit surfaces (NeuS) have become two mainstream technologies, respectively solving the bottleneck problems of explicit control and realistic rendering.

[0003] However, both mainstream technologies have drawbacks in practical applications: 1. Current 3D deformable face model generation technologies are mainly divided into two categories: parametric statistical models and neural implicit surface methods, both of which have significant bottlenecks. Parametric statistical model methods (such as 3DMM) construct linear statistical models of faces through principal component analysis. Their core shortcomings lie in the linear combination mechanism and fixed topological constraints: on the one hand, the model can only represent low-frequency facial features and is insufficient in expressing high-dimensional details such as wrinkles and muscle micro-movements, resulting in low realism of the generated results; on the other hand, the mandatory requirement for all face meshes to maintain the same number of vertices and semantic alignment (such as a fixed 60,000 vertices) not only increases the cost of professional scanning data acquisition (accounting for more than 70% of the total time in the process) but also restricts the flexible presentation of local personalized features.

[0004] 2. Neural implicit surface methods (such as NeuS) model implicit surfaces using the signed distance function (SDF) and achieve pore-level hyper-realistic reconstruction by combining differentiable volume rendering. However, their inherent algorithmic flaws make industrial application difficult: First, volume rendering requires sampling hundreds of points along the light rays and calculating integrals, resulting in a single-frame rendering latency of >200ms (far exceeding the 33ms threshold required for 30fps real-time interaction); second, the model parameters lack physical interpretability, making it difficult to achieve precise control requirements such as expression-driven and local fine-tuning; third, dynamic scenes require the introduction of additional deformation fields, further amplifying the GPU computational burden (training time 2-8 hours), failing to meet the hard requirements of real-time interactive scenarios such as games and virtual live streaming.

[0005] In summary, parametric statistical modeling methods heavily rely on high-quality 3D scan data and primarily rely on low-frequency features for detail reconstruction. Neural implicit surface methods, on the other hand, only require multi-view 2D images and can reconstruct a large amount of high-frequency details, but they cannot achieve real-time interactive applications that generate face models. These limitations severely restrict the widespread application and development potential of these technologies, necessitating innovative solutions to overcome them.

[0006] In summary, existing technologies face a dual contradiction: the lack of high-frequency details and the incompatibility of real-time interaction. Parametric models, limited by linear representation and topological rigidity, sacrifice realism for real-time performance; while neural implicit methods overcome the realism bottleneck, they lose interactivity due to excessive computational load. This fundamental flaw severely hinders the large-scale application of high-fidelity 3D faces in consumer-grade scenarios. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems existing in the prior art and provide a method, system and electronic device for generating three-dimensional deformable face models. It combines parametric statistical modeling methods and neural implicit surface methods to construct a face model generation framework.

[0008] The objective of this invention is achieved through the following technical solution: Firstly, a method for generating a three-dimensional deformable human face model is provided, including the following steps: S1. Collect facial video and preprocess it; S2. Based on the preprocessed face video, a high-precision 3D face model is generated using the neural implicit surface method; S3. Convert the high-precision 3D face model into a parametric statistical model to support real-time interaction.

[0009] Preferably, a smartphone camera is used, a flashlight is turned on, and a video sequence is filmed around the subject in a dimly lit room. Filming one subject takes approximately 25 seconds. The acquisition of facial video includes: Shoot videos of the face from multiple angles, ensuring the face is unobstructed and the lighting is even.

[0010] Preferably, the preprocessing includes: Frames are filtered from the face video, and sparse reconstruction is performed on the filtered image frames; Facial feature points of each frame of the image are extracted using the scale-invariant feature transform algorithm, and the camera intrinsic and extrinsic parameters are optimized using the bundle adjustment method. Output camera pose parameters and 3D sparse point cloud in JSON format.

[0011] Preferably, the frame filtering includes: The Laplacian operator is used to extract the highest-resolution image from every N frames.

[0012] Preferably, the generation of a high-precision 3D face model using the neural implicit surface method includes: A two-branch multilayer perceptron architecture is used to predict the signed distance function value and the color value, respectively; the two-branch multilayer perceptron architecture includes: The geometric analytical network takes three-dimensional coordinates as input and outputs a signed distance function (SDF) value and a geometric feature vector. It consists of 6 fully connected layers, each with 128 neurons, and uses ReLU as the activation function. The texture parsing network outputs color based on input geometric features and viewpoint orientation. It consists of four fully connected layers, each with 64 neurons, and uses LeakyReLU as the activation function. Images are generated through volume rendering, and the network is trained by combining photometric loss, mask loss, and Eikonal loss. After training, the moving cube algorithm is used to extract the 3D mesh model from the trained signed distance function field.

[0013] Preferably, the process of generating an image through volume rendering includes: The signed distance function value is mapped to a volume density field through the Laplacian cumulative distribution function, and a volume rendering integral is performed along the ray direction to calculate the pixel color.

[0014] Preferably, the step of converting the high-precision 3D face model into a parametric statistical model includes: A deformable head model is constructed, and vertex displacement is predicted based on the vertices of the base head model and facial expression parameters. A learnable displacement field is constructed based on the vertices of the base head model, and details are learned through a multilayer perceptron. Construct a standard facial model, a fixed-topology facial base network, and provide high-frequency texture targets; A differentiable renderer was used to render both the deformable head model and the high-precision 3D face model, and the parameters were optimized using a joint loss function. Differentiable rasterization was used to render the models, and the SoftPhong shader was used to simulate subsurface scattering of the skin.

[0015] Preferably, the joint loss function includes: Pixel fidelity loss is used to constrain the consistency of rendered colors; Perceptual similarity loss is used to preserve texture details; Deformation canonical loss is used to suppress excessive deformation; Surface smoothing loss is used to improve mesh smoothness.

[0016] Secondly, a three-dimensional deformable human face model generation system is provided, including: The video acquisition and preprocessing module is used to acquire facial videos and perform preprocessing. A high-precision 3D face model generation module is used to generate high-precision 3D face models based on preprocessed face videos using the neural implicit surface method. The parametric statistical model generation module is used to convert the high-precision 3D face model into a parametric statistical model to support real-time interaction.

[0017] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and the processor executes the three-dimensional deformable human face model generation method described in the first aspect when executing the computer instructions.

[0018] It should be further noted that the technical features corresponding to the above options can be combined or substituted to form new technical solutions if there is no conflict.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention creatively integrates neural implicit surface reconstruction and parametric statistical modeling techniques. It overcomes the limitations of parametric statistical models, such as loss of high-frequency details, topological rigidity, and reliance on professional scanning data, generating high-precision models efficiently with just 25 seconds of video from a smartphone. It also solves the problems of insufficient real-time performance and poor parameter interpretability of neural implicit surface methods. By constructing a fine model containing skin microstructure texture and converting it into a lightweight parametric model, it achieves the unification of pore-level high-frequency details and real-time interaction. At the same time, with the help of a physically interpretable parameter mapping mechanism, it meets the precise control requirements such as expression-driven and local fine-tuning. It is fully adaptable to multiple application scenarios such as film and television production, game development, and virtual live streaming, significantly reducing costs and improving performance and flexibility. Attached Figure Description

[0020] Figure 1 This is a flowchart of a method for generating a three-dimensional deformable human face model according to the present invention; Figure 2 This invention provides a process for generating a fine model based on a neural implicit surface. Figure 3 This invention describes the process for generating a parametric statistical model based on physically based rendering. Detailed Implementation

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that the defects in the solutions in the prior art are all the results of the inventors' practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of this application in the following text should be the inventors' contributions to this application in the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

[0023] In view of the technical problems pointed out in the background art, the present invention provides the following embodiments: Reference Figure 1 In one exemplary embodiment, a method for generating a three-dimensional deformable human face model is provided, comprising the following steps: S1. Collect facial video and preprocess it; S2. Based on the preprocessed face video, a high-precision 3D face model is generated using the neural implicit surface method; S3. Convert the high-precision 3D face model into a parametric statistical model to support real-time interaction.

[0024] For example, step S1 specifically includes: Video capture: Using a smartphone equipped with a 12-megapixel camera (such as an iPhone 13), turn on the rear flashlight as the active light source, and slowly move around the subject (from 6 angles including front, 45° side profile, and side view) in a dimly lit room (ambient light intensity < 50 lux) to record for 25 seconds (30fps, 750 frames in total), ensuring that the face is unobstructed and the lighting is evenly distributed.

[0025] Frame Selection and Sparse Reconstruction: Frames were extracted from the recorded video, and the image with the highest sharpness was selected from every 10 frames (sharpness was evaluated using Laplacian variance), resulting in 75 valid images. Sparse reconstruction was then performed on these 75 images using COLMAP. Feature point matching: The SIFT algorithm is used to extract facial feature points in each frame of the image (an average of 1200 per frame), and the camera intrinsic and extrinsic parameters are optimized by bundle adjustment. Pose calculation: Output camera pose parameters (including rotation matrix R and translation vector t) and 3D sparse point cloud (approximately 5000 points), and save them as a JSON format file for subsequent network training.

[0026] For example, such as Figure 2 As shown, step S2 specifically includes: Network architecture configuration: Building a dual-branch multilayer perceptron (MLP) architecture: Geometric analytic network: 6 fully connected layers (128 neurons per layer, ReLU activation function), input 3D coordinates (x,y,z), output signed distance function (SDF) value and 256-dimensional geometric feature vector; Texture parsing network: 4 fully connected layers (64 neurons per layer, activation function is LeakyReLU), input geometric feature vector and view direction vector (dx,dy,dz), output RGB color values ​​(range [0,1]).

[0027] Rendering parameter settings: Map the SDF value to a volume density field using the Laplace CDF function; perform volume rendering integrals along the ray direction (sampling 64 points per ray) to calculate pixel color.

[0028] Loss function optimization: Train the network using a joint loss function, iterating 50,000 times (batch size 8). Luminous loss: Calculates the L2 pixel error between the rendered image and the input image, with a weight of 1.0, using the following formula: ; Mask loss: Facial masks are extracted using an image segmentation model (such as U-Net). The binary cross-entropy loss between the rendered result and the mask is calculated, with a weight of 0.5. The formula is as follows: .

[0029] Eikonal loss: The L2 norm of the SDF gradient is constrained to be close to 1 (to ensure surface continuity), with a weight set to 0.1. The formula is as follows: .

[0030] Fine model output: After training, the 3D mesh model is extracted from the SDF field using the Marching Cubes algorithm, which includes high-frequency geometric details such as skin wrinkles and pores, and saved in PLY format.

[0031] For example, such as Figure 3 As shown, step S3 specifically includes: Dual-model layered architecture construction Deformable head model: Based on the Basel Face Model (BFM) basic head model (50,000 vertices), a learnable displacement field is constructed: Input: Vertex coordinates of the BFM model and facial expression parameter α (100-dimensional principal components); Displacement prediction: The vertex displacement ΔV is learned through a 3-layer MLP (256 neurons per layer), and the deformed vertex is output. It is used to capture high-frequency geometric details (such as crow's feet and nasolabial folds) in precision models.

[0032] Standard facial model: using Figure 2 The fine model output from the middle.

[0033] Physical lighting rendering engine configuration Differentiable rasterization: Using PyTorch3D's differentiable renderer, rasterize the triangular mesh of the deformable head model and output screen-space pixel colors; SoftPhong shader: simulates the subsurface scattering effect of skin; the lighting model formula is: in, =0.2 (ambient light coefficient) =0.7 (diffuse reflectance coefficient) =0.1 (spectral coefficient), p=32 (spectral index), n is the normal vector, l is the direction of the light source, r is the reflection vector, and v is the viewing direction.

[0034] Render the two models separately to obtain the rendered images of the deformable head model. The image corresponding to the precision model .

[0035] Loss function optimization Joint optimization of dual-model parameters (30,000 iterations, batch size 4): Pixel fidelity loss: Ensures that the rendered colors are consistent with the rendered model. The weight is set to 1.0; Perceptual similarity loss: Texture similarity is calculated using features from the ReLU3_3 layer of the VGG16 network. The weight is set to 0.3 (φ is the feature extraction function); Deformation canonical loss: constrains the L2 norm of the displacement field ΔV to suppress excessive deformation. The weight is set to 0.01; Surface smoothing loss: Constrains the smoothness of the triangular face by the difference in vertex normal vectors. The weight is set to 0.05.

[0036] Parametric model output: After optimization, the parametric statistical model is output, including: Facial expression basis function: 100-dimensional principal component parameter α, which can control facial deformation through linear combination; Texture parameters: UV space texture mapping and SoftPhong lighting parameters, supporting real-time rendering.

[0037] The embodiments detail the complete process of the present invention from video acquisition to fine model generation and parametric model conversion, verify the unity of high-frequency geometric details and real-time interactive performance, and solve the core bottleneck of the prior art.

[0038] In another exemplary embodiment, a three-dimensional deformable human face model generation system is provided, comprising: The video acquisition and preprocessing module is used to acquire facial videos and perform preprocessing. A high-precision 3D face model generation module is used to generate high-precision 3D face models based on preprocessed face videos using the neural implicit surface method. The parametric statistical model generation module is used to convert the high-precision 3D face model into a parametric statistical model to support real-time interaction.

[0039] In another exemplary embodiment, an electronic device is provided, including a memory and a processor. The memory stores computer instructions that can be executed on the processor. When the processor executes the computer instructions, it performs a three-dimensional deformable face model generation method provided in this embodiment of the invention.

[0040] The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0041] The embodiments of the subject matter and functional operation described in this specification can be implemented in: tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing device.

[0042] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0043] Suitable processors for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0044] It should be understood that each block in a flowchart or block diagram can represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0045] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A method for generating a three-dimensional deformable human face model, characterized in that, Includes the following steps: S1. Collect facial video and preprocess it; S2. Based on the preprocessed face video, a high-precision 3D face model is generated using the neural implicit surface method; S3. Convert the high-precision 3D face model into a parametric statistical model to support real-time interaction.

2. The method for generating a three-dimensional deformable human face model according to claim 1, characterized in that, The collected facial video includes: Shoot videos of the face from multiple angles, ensuring the face is unobstructed and the lighting is even.

3. The method for generating a three-dimensional deformable human face model according to claim 1, characterized in that, The preprocessing includes: Frames are filtered from the face video, and sparse reconstruction is performed on the filtered image frames; Facial feature points of each frame of the image are extracted using the scale-invariant feature transform algorithm, and the camera intrinsic and extrinsic parameters are optimized using the bundle adjustment method. Output camera pose parameters and 3D sparse point cloud in JSON format.

4. The method for generating a three-dimensional deformable human face model according to claim 3, characterized in that, The frame filtering includes: The Laplacian operator is used to extract the highest-resolution image from every N frames.

5. The method for generating a three-dimensional deformable human face model according to claim 1, characterized in that, The method of generating a high-precision 3D face model using neural implicit surface methodology includes: A two-branch multilayer perceptron architecture is used to predict the signed distance function value and the color value, respectively; the two-branch multilayer perceptron architecture includes: The geometric analytic network consists of 6 fully connected layers, each with 128 neurons, and uses ReLU as the activation function. The texture parsing network consists of four fully connected layers, each with 64 neurons, and uses LeakyReLU as the activation function. Images are generated through volume rendering, and the network is trained by combining photometric loss, mask loss, and Eikonal loss. After training, the moving cube algorithm is used to extract the 3D mesh model from the trained signed distance function field.

6. The method for generating a three-dimensional deformable human face model according to claim 5, characterized in that, The process of generating an image through volume rendering includes: The signed distance function value is mapped to a volume density field through the Laplacian cumulative distribution function, and a volume rendering integral is performed along the ray direction to calculate the pixel color.

7. The method for generating a three-dimensional deformable human face model according to claim 1, characterized in that, The step of converting the high-precision 3D face model into a parametric statistical model includes: Construct a deformable head model and predict vertex displacement based on the vertex and facial expression parameters of the basic head model; Differentiable renderers were used to render deformable head models and high-precision 3D face models respectively, and parameters were optimized using a joint loss function.

8. The method for generating a three-dimensional deformable human face model according to claim 7, characterized in that, The joint loss function includes: Pixel fidelity loss is used to constrain the consistency of rendered colors; Perceptual similarity loss is used to preserve texture details; Deformation canonical loss is used to suppress excessive deformation; Surface smoothing loss is used to improve mesh smoothness.

9. A three-dimensional deformable human face model generation system, characterized in that, include: The video acquisition and preprocessing module is used to acquire facial videos and perform preprocessing. A high-precision 3D face model generation module is used to generate high-precision 3D face models based on preprocessed face videos using the neural implicit surface method. The parametric statistical model generation module is used to convert the high-precision 3D face model into a parametric statistical model to support real-time interaction.

10. An electronic device comprising a memory and a processor, wherein the memory stores computer instructions executable by the processor, characterized in that, The processor executes the three-dimensional deformable human face model generation method according to any one of claims 1-8 when running computer instructions.

Images