High-fidelity three-dimensional face reconstruction and generation method based on implicit neural function

Abstract

The application discloses a high-fidelity three-dimensional face reconstruction and generation method based on an implicit neural function, and proposes a high-fidelity parameterized face model which can be semantically decoupled and edited, and simultaneously introduces new identity, expression representation and semantic segmentation networks. With the aid of the implicit neural representation, the method can perform high-fidelity reconstruction and synthesis on face details, difficult expressions and large-angle camera poses, so that the generated face pictures have good reality and clarity.

Description

Technical Field

[0001] This invention relates to 3D face reconstruction and generation technology, and more particularly to a high-fidelity 3D face reconstruction and generation method based on implicit neural functions. Background Technology

[0002] With the advent of the digital human era, people yearn to have their own digital avatars on social media platforms and online virtual platforms. High-fidelity face reconstruction and generation have emerged as a result. People desire face reconstructions that are highly faithful, realistic, and clear, while allowing for decoupled editing of various facial attributes. Furthermore, the generated face needs to maintain consistent perspective across different camera poses and avoid hair fraying when the viewpoint changes.

[0003] In recent years, research organizations, represented by NVIDIA, have proposed face generation models with high-resolution output quality. Given a random number seed, these models generate faces with good realism and viewpoint consistency. However, these models have weak semantic editability, only allowing simple changes in camera pose without decoupling and editing facial attributes such as identity and lighting. Furthermore, when a single face image is input and these models are used for GAN inverse transformation to reconstruct the face, the reconstructed face fails to accurately learn identity and expression, resulting in insufficient clarity. Additionally, the generated face cannot maintain viewpoint consistency when changing camera pose; the face's identity and expression change with the viewpoint.

[0004] Recently, HeadNeRF has become a representative face reconstruction model, primarily composed of a volume rendering module and a 2D neural rendering upsampling module. While HeadNeRF offers some semantic editability, the reconstructed face images suffer from poor clarity and realism, exhibiting issues such as washed-out appearance and blurred hair, eyes, and teeth. Furthermore, its performance in reconstructing complex expressions and new perspectives with large-angle camera poses is less than ideal. This is because HeadNeRF's upsampling structure is poorly designed, containing significant aliasing interference and failing to maintain translation and rotation invariance. Additionally, because HeadNeRF directly uses the identity and expression vectors from a 3DMM as implicit inputs, and 3DMM's ability to represent identities and expressions is insufficient, it struggles to reconstruct facial details and complex expressions with high fidelity under training data with multiple identities and expressions.

[0005] Furthermore, faces generated by both face generation and face reconstruction models exhibit a problem of hair and teeth flickering when the camera pose changes, known as hair adhesion. This affects the realism of the generated faces and is a persistent issue in the field of face generation. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a high-fidelity 3D face reconstruction and generation method based on implicit neural functions. This invention designs a high-fidelity parametric face model that is semantically decoupled and editable. By introducing a novel face identity and expression representation, an upsampling module with rotation and translation invariance, and a semantic segmentation network, the face generated by this invention avoids the problem of hair adhesion, exhibiting excellent realism and clarity. Furthermore, this invention can perform high-fidelity reconstruction of facial details and complex expressions, while clearly synthesizing faces from large-angle new perspectives.

[0007] The objective of this invention is achieved through the following technical solution: a high-fidelity 3D face reconstruction and generation method based on implicit neural functions, the method comprising the following steps:

[0008] S1. Construct a training set based on the monocular face video dataset and the single face dataset, and preprocess the training data in the training set.

[0009] S2, for any face image in the training set, input it into a semantically editable parameterized face model to generate the corresponding rendered RGB image and rendered semantic segmentation map;

[0010] The parameterized face model consists of an RGB network and a semantic network; both the RGB network and the semantic network include a volume rendering module and an upsampling module.

[0011] In the volume rendering module, given a single face image or monocular video sequence with key point annotations, sampling is performed along the camera light rays, and the sampling points are position-encoded. The sampling point position codes and the input face semantic latent codes are used as inputs to an implicit neural function based on a multilayer perceptron, outputting a density value and a high-dimensional feature, and generating a feature map with the help of volume rendering; the semantic latent codes include identity latent codes, expression latent codes, texture latent codes, and illumination latent codes;

[0012] In the upsampling module Θ of the RGB network, the feature map is used as the input of the upsampling module Θ, and the semantic latent code is input into the mapping network based on the multilayer perceptron. The output of the mapping network is used as the style latent code input into each sub-network of the upsampling module Θ, and the RGB image is output.

[0013] In the upsampling module Ψ of the semantic network, the feature map is input into the upsampling module Ψ obtained by the 2D neural rendering module based on HeadNeRF, and the semantic segmentation map is output.

[0014] S3, a parameterized face model is trained using the training set, wherein the model parameters and semantic latent codes of the parameterized face model are both optimized;

[0015] S4 utilizes a trained parametric face model to perform new perspective synthesis, decoupled semantic editing, and expression transfer applications on test images.

[0016] Furthermore, in step S1, the InsightFace key point annotation algorithm is used to annotate the key points of the faces in the training data; and the one-euro filtering algorithm is used to reduce the annotation jitter between monocular video frames.

[0017] Furthermore, in the volume rendering module, a face tracking algorithm is used to reverse optimize the 3DMM information of the input face based on a 3D deformable face model (3DMM), and the texture vector and lighting vector are extracted as the initialization of the texture code and lighting code, respectively; a face recognition network is pre-trained based on MagFace to infer the identity features as the identity code.

[0018] Furthermore, in the volume rendering module, the corresponding face mesh is generated using the semantic latent code of the face and the labeled k key points. The k×3 3D key points of the reference space are extracted, the 3D key points are standardized, and they are used as a new expression representation.

[0019] Furthermore, in the volume rendering module, the identity code, expression code, and sampling point position code are input together into the implicit neural function. At the same time, the identity code is input again into the (n+1)th fully connected layer of the 2n-layer multilayer perceptron. The density value of the sampling point and an intermediate feature are predicted using the identity code, expression code, and sampling point position code. Based on this, the texture code and lighting code will predict a high-dimensional feature, and then a feature map will be generated with the help of volume rendering.

[0020] Furthermore, in the upsampling module Ψ of the semantic network, a noise reduction filter layer is added to the 2D neural rendering module of HeadNeRF. The noise reduction filter layer adopts an RDN network to effectively filter out noise on the human face.

[0021] Furthermore, the loss function of the parameterized face model includes pixel loss, decoupling loss, perceptual loss, semantic loss, and distribution loss;

[0022] The semantic loss L sem The formula is Among them, w * It is a pixel loss of a balanced semantic network and perceived loss Hyperparameter weights;

[0023] The distribution loss L dist The formula is in, It is a feature map generated by the volume rendering module of the RGB network. This is the correct input distribution for the upsampling module Θ.

[0024] Furthermore, during the training process of the parameterized face model, Ψ is first replaced with Θ, and a portion of face images randomly selected from the training set are used to pre-train the replaced parameterized face model. Then, Ψ in the RGB network is replaced with Θ.

[0025] The formal training is divided into three stages. In the first stage, random Fourier features are input into Θ, and mean squared error is used to constrain the training. and The distribution differences between them; in the second stage, using replace The former is then input into Θ. In the first two stages of training, the model parameters of the pre-trained Θ are fixed to avoid interference from relatively large noise during training. In the third stage, the model parameters of the upsampling module Θ are allowed to change freely, enabling the parameterized face model to effectively learn the details and difficult expressions of the face.

[0026] Furthermore, in the first stage of training, the weight of the distribution loss is set to be relatively large, and the distribution loss is used as the main body of the loss function. In the next two training stages, the weight of the distribution loss is reduced, and the attention of the parameterized face model is focused on learning the facial information of the input image.

[0027] As can be seen from the technical solution provided by the present invention above, the beneficial effects of the present invention are:

[0028] 1) By proposing a parameterized face model that can be semantically decoupled and edited, and introducing new identity and expression representations as well as a semantic segmentation network, it is possible to reconstruct face details and difficult expressions with high fidelity and clearly synthesize faces from new perspectives at large angles.

[0029] 2) The new editable parametric face model designed by this invention can decouple the generated face from its attributes, making the model widely applicable in face expression transfer and face stylization. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This invention provides a network structure diagram of an editable parametric face model.

[0032] Figure 2Example image showing the viewpoint consistency of face images generated by the model provided in this embodiment of the invention;

[0033] Figure 3 , Figure 4 , Figure 5 , Figure 6 Example diagram of semantically decoupled editable face images generated by the model provided in this embodiment of the invention;

[0034] Figure 7 A schematic diagram illustrating the qualitative comparison of large-angle camera pose synthesis for the model provided in the embodiments of the present invention;

[0035] Figure 8 This is a schematic diagram illustrating a qualitative comparison between the model and HeadNeRF reconstruction results provided in this embodiment of the invention.

[0036] Figure 9 A schematic diagram showing a qualitative comparison of the reconstruction results of the model provided in the embodiments of the present invention and the current best face generation model. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0038] This invention provides a high-fidelity 3D face reconstruction and generation method based on implicit neural functions, comprising the following steps:

[0039] Step 1: Construct a training set based on the collected high-definition monocular face video dataset and high-definition single-image face dataset, and preprocess the training data in the training set.

[0040] This invention employs the InsightFace keypoint annotation algorithm to annotate keypoints on faces in the training data. Furthermore, since existing algorithms exhibit significant inter-frame jitter when annotating monocular videos, this invention uses a one-euro filter algorithm to mitigate inter-frame annotation jitter and ensure training stability.

[0041] Step 2, as follows Figure 1 As shown, for any face image in the training set, it is input into a semantically editable parameterized face model to generate the corresponding rendered RGB image and rendered semantic segmentation map. The specific network structure and model representation for this step are as follows:

[0042] The semantically editable parametric face model designed in this invention is denoted as HeadNeRF V2. It consists of two parallel networks: the RGB network, denoted as RGBNet, and the semantic network, denoted as SemanticNet.

[0043] First, let's introduce the volume rendering module. Given a single face image or monocular video sequence with keypoint annotations, samples are taken along the camera's light rays, and these sampling points are denoted as... The position of x is encoded, denoted as γ(x). This is then input into an implicit neural function h based on a multilayer perceptron. θ The latter can be formalized as:

[0044]

[0045] Where θ represents the network parameters, z * Semantic code representing the input face: Identity code z id , emoji hidden code z lm Texture Hidden Code z alb and illumination hidden code z ill Let z = (z id ,z lm ,z alb ,z ill ), where parentheses indicate vector concatenation. In this embodiment,

[0046] Using a face tracking algorithm and based on a 3D deformable face model (3DMM), the 3DMM information of the input face is inversely optimized, and the texture and lighting vectors are extracted as z-vectors. alb and z ill Initialization. In the semantic implicit input initialization phase of the volume rendering module, to compensate for the insufficient ability of the 3DMM identity vector to represent facial identity (i.e., the vector cannot express details such as teeth and hair), this embodiment pre-trains a face recognition network based on MagFace to infer a 128-dimensional identity feature as z. id This enhances the ability of learnable hidden codes to represent a large number of identities.

[0047] As described in step 1, to utilize a large amount of data during the training phase and increase the model's generalization ability, this invention does not use manually annotated keypoint data as the training set. Instead, it uses the current best keypoint annotation algorithm, InspireFace, to annotate keypoints in monocular videos and single face images. Next, it generates a corresponding face mesh using the semantic latent code of the face and the annotated k keypoints, extracting k×3 3D keypoints from the reference space; in this embodiment, k is set to 68. Specifically, this invention extracts the 3D keypoints before they participate in camera parameter transformation, ensuring that they do not contain camera parameter information, thus naturally decoupling them from camera parameters. This invention standardizes the 3D keypoints and uses them as a new expression representation, i.e.

[0048] Furthermore, for the volume rendering module, this invention will z id and z lm Input h along with γ(x) θ At the same time, z id The (n+1)th fully connected layer of the 2n-layer multilayer perceptron is input again (for an eight-layer multilayer perceptron, the fifth fully connected layer is input), allowing the model to learn identity information more effectively. Then, z is used... id , z lm γ(x) predicts the density value σ of x and an intermediate feature. alb and z ill Based on this, a high-dimensional feature F(x) will be predicted. Then, a feature map will be generated using volume rendering. in, r(t) represents a ray of light emitted from the center of the camera. To distinguish the volume rendering modules in RGBNet and SemanticNet, Φ is used respectively. i Let i∈{1,2} represent them, that is, Φ1 represents the volume rendering module of RGBNet, Φ2 represents the volume rendering module of SemanticNet, and use Φ1 to represent the volume rendering module of SemanticNet. This represents the corresponding feature map.

[0049] Next, we will introduce the upsampling module Θ of RGBNet. This module will upsample the feature map... As input to the upsampling module Θ, z is simultaneously input to the multilayer perceptron-based mapping network M and mapped to 512 dimensions, denoted as w. Then, w is used as the style latent code input to each sub-network layer of the upsampling module Θ, guiding the parameterized face model HeadNeRF V2 to effectively learn facial detail information. After the image signal passes through the upsampling module Θ, the feature map... It was converted into an RGB image with a resolution of 1024, denoted as I1.

[0050] For SemanticNet, since semantic segmentation maps do not contain as much high-frequency information as RGB images, a more sophisticated structure like the upsampling module Θ in RGBNet is not needed. Instead, after obtaining the feature map... Then, it is input into the upsampling module Ψ obtained by the HeadNeRF-based 2D neural rendering module to obtain the corresponding semantic segmentation map I2 with a resolution of 1024. Preferably, a denoising filter layer is added to the HeadNeRF 2D neural rendering module. The denoising filter layer adopts an RDN network, which can effectively filter out noise on the face. The addition of SemanticNet not only enhances the learning of facial semantic information by the parameterized face model HeadNeRF V2, but also corrects the error between I1 and the real RGB image, accelerating the model's learning.

[0051] Furthermore, since the semantic latent code z of the input parameterized face model is learnable, it serves as a bridge between RGBNet and SemanticNet to transfer face information between the networks.

[0052] Step 3: In this embodiment of the invention, the model parameters and semantic hidden code z of the parameterized face model HeadNeRF V2 id ,z lm ,z alb ,z ill All are involved in the optimization. This invention has carefully designed a loss function that is beneficial to HeadNeRF V2 training, and it is described as follows:

[0053] (1) Pixel loss:

[0054] The pixel loss in this embodiment of the invention can be expressed by the formula:

[0055]

[0056] in, represents the image loss of RGBNet and SemanticNet, and the ground truth value of the training image, respectively.

[0057] (2) Decoupling loss:

[0058] The decoupling loss of this invention can help to effectively decouple semantic latent codes, i.e., identity, expression, texture, and lighting. The decoupling loss of this invention can be expressed by the following formula:

[0059]

[0060] Among them, z * This represents a learnable implicit code. This represents the initialization code of the input image, w *This represents the hyperparameter weights that balance the losses of various implicit codes.

[0061] (3) Perceived loss:

[0062] Perceptual loss LPIPS is an image perceptual loss that can guide the upsampling module of the parameterized face model HeadNeRF V2, especially Θ, to learn detailed facial information, thereby enhancing the realism and clarity of the generated images. The perceptual loss in this embodiment is denoted as...

[0063] (4) Semantic loss:

[0064] The semantic loss in this embodiment of the invention can optimize the learnable hidden code z, and the change in z will adjust the RGB image I1, thereby helping RGBNet learn facial semantic information and subtle expressions, accelerating model training. The semantic loss in this embodiment of the invention can be expressed by the formula:

[0065]

[0066] Among them, w * It balances the pixel loss of SemanticNet. and perceived loss Hyperparameter weights.

[0067] (5) Distributed loss:

[0068] The distribution loss in this embodiment of the invention is crucial for training the upsampling module Θ. Only by enabling the feature maps generated by the volume rendering module of RGBNet to function properly can the desired results be achieved. The distribution of the input distribution of the pre-trained upsampled module Θ is learned. Only by avoiding signal interference can the upsampling module Θ fully utilize its powerful generation capabilities. The distribution loss in this embodiment of the invention can be expressed by the formula:

[0069]

[0070] in, This is the correct input distribution of the upsampling module Θ. This invention divides the formal training into three stages. In stage one, the weights w of the distribution loss are... dist The weights are set relatively large, and the distribution loss is used as the main body of the loss function. In the next two training phases, the weights w are reduced. dist This allows the HeadNeRF V2 parameterized face model to focus its attention on learning the facial information of the input image. The three training phases will be described in detail in step 4.

[0071] (6) Total loss:

[0072] Experiments show that the optimal combination of loss functions in this embodiment of the invention is:

[0073]

[0074] Among them, w * This represents the weights that balance the various losses, while w sem Setting it to an appropriate small value allows the model to focus more on learning the high-frequency information of RGB images.

[0075] The loss function design of the parameterized face model HeadNeRF V2 effectively ensures its ability to learn difficult expressions, facial details, and editable semantic attributes.

[0076] Step 4: In this embodiment of the invention, the input distribution of the upsampling module Θ of RGBNet during pre-training. It follows a specific distribution, such as a Fourier distribution, thus guaranteeing the network's translation and rotation invariance. However, and The distribution of these components varies greatly, so directly using the former instead of the latter as input to the upsampling module can only generate images filled with noise blocks. Therefore, this invention carefully designs a training strategy to allow the upsampling module to leverage its advantages.

[0077] This invention first replaces Θ with Ψ, and pre-trains the replaced parameterized face model using randomly selected face images from the training set, thus Φ i For i∈{1,2}, the characteristic distribution of the neural radiation field can be learned. Next, Ψ in RGBNet is replaced with Θ.

[0078] This invention divides formal training into three stages. In the first stage, random Fourier features are input to Θ, instead of directly... Input it, and at the same time, use the mean square error to constrain it. and The distribution differences between them. In the second stage, using replace And the former is input into Θ. This invention uses a special substitution method to ensure training stability. In this stage, because... Guided by this, the parameterized face model HeadNeRF V2 can learn to control camera pose and facial semantic attributes.

[0079] In the first two stages of training, the pre-trained model parameters of Θ are fixed to avoid interference from relatively large noise during training. In the third stage, the model parameters of the upsampling module Θ are allowed to vary freely, enabling the parameterized face model HeadNeRF V2 to effectively learn facial details and complex expressions.

[0080] Thanks to the prior face information contained in the pre-trained Θ and the semantic latent code z of the input Θ, the face images generated by the parameterized face model HeadNeRF V2 have high realism and clarity.

[0081] Step 5: After the parameterized face model HeadNeRF V2 is trained, the test images can be used for new perspective synthesis, decoupled semantic editing, and expression transfer applications.

[0082] As attached Figure 2 As shown, we demonstrate the decoupled control of camera pose by HeadNeRF V2 and the viewpoint consistency of the synthesized new perspective images. Using pre-trained HeadNeRF V2, we reconstruct the input test image and generate images with new viewpoints by changing the camera pose. As can be seen from the figures, our new perspective images exhibit good viewpoint consistency across various camera poses. Furthermore, our camera pose editing does not affect facial attributes such as identity and expression, indicating that the former and facial attributes are well decoupled.

[0083] As attached Figure 3 As shown, we first reconstruct the faces on both sides, then interpolate their identity codes, and the middle result is the new identity face image obtained by continuously changing the identity codes and inputting them into HeadNeRF V2.

[0084] As attached Figure 4 , 5 As shown in Figure 6, the leftmost face is the result reconstructed from a randomly input test image of HeadNeRF V2. Given an arbitrary target texture, expression, and lighting code, we can similarly perform continuous texture, expression, and lighting editing on the reconstructed face using the pre-trained HeadNeRF V2. Our individual editing of attributes such as identity and expression does not change other attributes of the face, demonstrating the good decoupling of HeadNeRF V2's semantic editing.

[0085] As attached Figure 7 As shown, we compared the results of HeadNeRF V2 and the current best face generation model in synthesizing faces with large-angle camera poses. It can be seen that the faces generated by GAN-Control have unnatural facial details such as black lines on the facial contours and floating frames, while HeadNeRF V2 can naturally synthesize faces with large-angle new perspectives, which verifies the stability of the model.

[0086] As attached Figure 8As shown, we input any test image and compare the reconstruction results with HeadNeRF. We can see that the face reconstructed by HeadNeRF is blurry and has a whitening effect. At the same time, it cannot reproduce subtle expressions and glasses very well. On the other hand, the reconstruction result of HeadNeRF V2 is clear and high-fidelity.

[0087] As attached Figure 9 As shown, we compared the face reconstruction results with several of the best face generation models currently available. It can be seen that models such as EG3D are not ideal in terms of viewpoint consistency and reconstruction of difficult expressions; changing the camera pose affects the identity. In contrast, HeadNeRF V2 can reconstruct faces with high fidelity while maintaining viewpoint consistency.

[0088] We quantitatively compared the FID values ​​of images generated by HeadNeRF V2 with the current state-of-the-art face reconstruction and face generation models, as shown in Table 1. As can be seen, our results are second only to EF3D. However, the latter cannot perform semantic editing, can only adjust camera pose at small angles, and has a stable training data distribution. In contrast, we performed simple bilinear downsampling when calculating the FID values ​​of HeadNeRF V2 for fairness, which led to a loss of image information, resulting in FID values ​​that are higher than the true values.

[0089] Table 1. Quantitative Comparison of Reconstruction Results between HeadNeRF V2 and Other Face Generation and Reconstruction Models

[0090]

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0092] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A high-fidelity 3D face reconstruction and generation method based on implicit neural functions, characterized in that, Includes the following steps: S1. Construct a training set based on the monocular face video dataset and the single face dataset, and preprocess the training data in the training set. S2, for any face image in the training set, input it into a semantically editable parameterized face model to generate the corresponding rendered RGB image and rendered semantic segmentation map; The parameterized face model consists of an RGB network and a semantic network; both the RGB network and the semantic network include a volume rendering module and an upsampling module. In the volume rendering module, given a single face image or monocular video sequence with key point annotations, sampling is performed along the camera light rays, and the sampling points are position-encoded. The sampling point position codes and the input face semantic latent codes are used as inputs to an implicit neural function based on a multilayer perceptron, outputting a density value and a high-dimensional feature, and generating a feature map with the help of volume rendering; the semantic latent codes include identity latent codes, expression latent codes, texture latent codes, and illumination latent codes; In the upsampling module Θ of the RGB network, the feature map is used as the input of the upsampling module Θ, and the semantic latent code is input into the mapping network based on the multilayer perceptron. The output of the mapping network is used as the style latent code input into each sub-network of the upsampling module Θ, and the RGB image is output. In the upsampling module Ψ of the semantic network, the feature map is input into the upsampling module Ψ obtained by the 2D neural rendering module based on HeadNeRF, and the semantic segmentation map is output. S3, a parameterized face model is trained using the training set, wherein the model parameters and semantic latent codes of the parameterized face model are both optimized; During the training process of the parameterized face model, Ψ is first replaced with Θ, and a portion of face images randomly selected from the training set are used to pre-train the replaced parameterized face model. Then, Ψ in the RGB network is replaced with Θ. The formal training is divided into three stages. In the first stage, random Fourier features are input into Θ, and mean squared error is used to constrain the training. and The distribution differences between them, It is a feature map generated by the volume rendering module of the RGB network. This is the correct input distribution for the upsampling module Θ; in the second stage, using replace And input the former into Θ; in the first two stages of training, fix the model parameters of the pre-trained Θ to avoid it being interfered with by relatively large noise during training. In the third stage, let the model parameters of the upsampling module Θ start to change freely, so that the parameterized face model can effectively learn the details and difficult expressions of the face. S4 utilizes a trained parametric face model to perform new perspective synthesis, decoupled semantic editing, and expression transfer applications on test images.

2. The high-fidelity 3D face reconstruction and generation method based on implicit neural functions according to claim 1, characterized in that, In step S1, the InsightFace key point annotation algorithm is used to annotate the key points of the faces in the training data; the one-euro filtering algorithm is used to reduce the annotation jitter between monocular video frames.

3. The high-fidelity 3D face reconstruction and generation method based on implicit neural functions according to claim 1, characterized in that, In the volume rendering module, a face tracking algorithm is used to reverse-optimize the 3DMM information of the input face based on the 3D deformable face model 3DMM, and the texture vector and lighting vector are extracted as the initialization of the texture code and lighting code, respectively; a face recognition network is pre-trained based on MagFace to infer the identity features as the identity code.

4. The high-fidelity 3D face reconstruction and generation method based on implicit neural functions according to claim 1, characterized in that, In the volume rendering module, the semantic latent code of the face and the labeled k key points are used to generate the corresponding face mesh, extract the k×3 3D key points of the reference space, standardize the 3D key points, and use them as a new expression representation.

5. The high-fidelity 3D face reconstruction and generation method based on implicit neural functions according to claim 1, characterized in that, In the volume rendering module, the identity code, expression code, and sampling point position code are input together into the implicit neural function. At the same time, the identity code is input again into the (n+1)th fully connected layer of the 2n-layer multilayer perceptron. The density value of the sampling point and an intermediate feature are predicted using the identity code, expression code, and sampling point position code. Based on this, the texture code and lighting code will predict a high-dimensional feature, and then a feature map will be generated with the help of volume rendering.

6. The high-fidelity 3D face reconstruction and generation method based on implicit neural functions according to claim 1, characterized in that, In the upsampling module Ψ of the semantic network, a noise reduction filter layer is added to the 2D neural rendering module of HeadNeRF. The noise reduction filter layer adopts an RDN network to effectively filter out noise on the human face.

7. The high-fidelity 3D face reconstruction and generation method based on implicit neural functions according to claim 1, characterized in that, The loss function of the parameterized face model includes pixel loss, decoupling loss, perceptual loss, semantic loss, and distribution loss; The semantic loss L sem The formula is Among them, w * It is a pixel loss of a balanced semantic network and perceived loss Hyperparameter weights; The distribution loss L dist The formula is 8. The high-fidelity 3D face reconstruction and generation method based on implicit neural functions according to claim 1, characterized in that, In the first training phase, the weight of the distribution loss is increased, and the distribution loss is used as the main body of the loss function. In the next two training phases, the weight of the distribution loss is reduced, and the parameterized face model focuses its attention on learning the facial information of the input image.

Images