Three-dimensional face editing method and system based on neural radiance field decoupling
By using a neural radiation field decoupling method and leveraging three-plane representation and semantic mask editing, the problem of geometric control and material decoupling in 3D faces in existing technologies is solved, achieving efficient geometric editing and material style transfer effects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF COMPUTING TECH CHINESE ACAD OF SCI
- Filing Date
- 2022-10-28
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies struggle to explicitly control the geometry of 3D faces and decouple the control of materials, resulting in low generation quality, high training difficulty, and an inability to effectively transfer style.
We employ a neural radiation field-based approach to decouple geometry and materials through a three-plane representation, utilize semantic masks for editing, and achieve decoupled material replacement through style transfer. We also combine cross-entropy functions to optimize latent variables for efficient editing.
It enables geometric editing and material style transfer of 3D faces, maintaining consistency in editing effects and high-quality generation from different perspectives.
Smart Images
Figure CN115719413B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to computer graphics and computer vision, and more particularly to methods for geometric semantic editing and material control of 3D human face models. Background Technology
[0002] Efficiently generating consistent and high-quality 3D facial images is an area that is gradually gaining widespread attention and in-depth research. Current techniques combine neural radiation fields with adversarial neural networks to generate a facial volume space, followed by volume rendering to obtain a low-resolution, consistent 3D facial image, and then using a super-resolution module to obtain a high-resolution, consistent 3D facial image. However, these generation methods often lack explicit control over geometry, and some even lack decoupled control over materials. The ability to edit geometry and decouple material control are indispensable for applications such as 3D character design and educational training.
[0003] In 3D adversarial generative networks, existing methods (e.g., Nerface, FLAME-IN-NeRF, HeadNeRF, MoFaNeRF) borrow latent variables such as identity and expression components from explicit mesh models of 3D faces (e.g., FLAME, 3DMM) to provide the editing medium. Some methods (e.g., Nefies, HyperNeRF, Audio-NeRF) use learnable encodings to capture dynamic facial movements, but without a correspondence with additional information (e.g., audio, attributes), they cannot be used for interpretation and editing. FENeRF incorporates semantic masks into conditional neural radiation fields (e.g., GRAF, pi-GAN) to generate and provide the editing medium, but the quality of the generated data is limited. Summary of the Invention
[0004] Compared with the prior art, the present invention also uses semantic masks as the editing medium and is based on a pre-trained 3D adversarial generative network. However, while achieving geometry and material decoupling and providing semantic masks as the editing medium and using optimized methods for editing, the present invention meticulously preserves the original pre-trained network's generation capabilities and generation speed, and does not require retraining.
[0005] Specifically, this invention pre-trains a generator on a dataset of face images by inputting random noise, and outputs 3D faces capable of both local geometric and overall style editing. The aim is to address the problems of low generation quality, training difficulties, geometric control challenges, and difficulties in decoupling material control in existing technologies, as well as the inability to perform effective style transfer. It proposes a method and system based on three-plane representation for 3D face geometric editing using semantic masks and decoupling material replacement through style transfer. This method allows users to perform geometric editing of 3D faces using semantic masks and, given a reference image, perform 3D consistent style transfer of materials.
[0006] The three-dimensional face editing method based on neural radiation field decoupling proposed in this invention specifically includes:
[0007] Step 1: Random sampling is performed based on Gaussian distribution to obtain latent variable z, and the latent variable z is transformed into intermediate latent variable w through multilayer perceptron;
[0008] Step 2: Using the intermediate implicit variable w, synthesize the three planes p = (p xy ,p yz ,p xz );
[0009] Step 3: Decompose the three planes p into geometric features with local spatial variations. And the material characteristics that are globally invariant in space F app ;
[0010] Step 4: Based on this geometric feature A normalized three-plane structure is constructed, and light rays are emitted from it according to camera parameters to perform bilinear interpolation to sample features. The density σ and semantic label s are obtained through a geometric decoder.
[0011] Step 5: Based on this geometric feature and any material feature After obtaining three planes through inverse normalization, bilinear interpolation is performed to sample features from the light rays emitted from them based on camera parameters, and color c is obtained through a material decoder.
[0012] Step 6: Based on the density σ, the semantic label s, and the color c, generate a 3D face, and obtain the semantic mask and the 2D face image of the 3D face from a specified viewpoint through volume rendering.
[0013] The aforementioned three-dimensional face editing method based on neural radiation field decoupling further includes:
[0014] Step 7: Edit the semantic mask to obtain a new mask. Measure the distance between the new mask and the semantic mask using the cross-entropy function. Optimize the intermediate latent variable w in reverse. Based on the optimized intermediate latent variable w, obtain the edited 3D face image from a specified viewpoint through steps 1 to 6 as the editing result.
[0015] The aforementioned three-dimensional face editing method based on neural radiation field decoupling, wherein step 3 includes:
[0016] S31: Normalization Obtain the normalized three planes and material characteristics mean and variance
[0017] The aforementioned three-dimensional face editing method based on neural radiation field decoupling, wherein step 4 includes:
[0018] By decomposing and normalizing the three planes Divided into three parts, the three coordinate planes that make up the volume space are formed. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0019] Given camera parameters In this case, based on the rotation and translation, light rays emitted from the camera position are constructed in the volume space V. Points are sampled on each light ray, and the corresponding features are sent to the geometric decoder to obtain the density σ and the semantic label s.
[0020] The aforementioned three-dimensional face editing method based on neural radiation field decoupling, wherein step 5 includes:
[0021] Based on normalization of three planes and material characteristics mean and variance By anti-homogenization Obtain three planes
[0022] By decomposing and normalizing the three planes It consists of three parts, forming the three coordinate planes of the volume space, thus creating the volume space. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0023] Given camera parameters In this case, construct the volume space based on rotation and translation. The light rays emitted from the camera position are sampled, and the corresponding features are obtained by sending them to the material decoder to obtain the color c.
[0024] The aforementioned three-dimensional face editing method based on neural radiation field decoupling, wherein step 6 includes:
[0025] Based on the density σ and semantic label s, the final semantic label rendered on each ray is obtained through discretized volume rendering, resulting in the rendered semantic mask; based on the color c, the final color rendered on each ray is obtained through discretized volume rendering and density σ, ultimately yielding the two-dimensional face image.
[0026] This invention also proposes a three-dimensional face editing system based on neural radiation field decoupling, which includes:
[0027] The initial module is used to perform random sampling based on a Gaussian distribution to obtain the latent variable z, and then transforms the latent variable z into an intermediate latent variable w through a multilayer perceptron.
[0028] The three-plane synthesis module is used to synthesize three planes p = (p...) based on the intermediate implicit variable w. xy ,p yz ,p xz );
[0029] The decomposition module is used to decompose the three planes p into geometric features with local spatial variations. And the material characteristics that are globally invariant in space F app ;
[0030] Sampling module, used to sample based on this geometric feature A normalized three-plane structure is constructed, and light rays are emitted from it according to camera parameters to perform bilinear interpolation to sample features. The density σ and semantic label s are obtained through a geometric decoder.
[0031] The decoding module, based on this geometric feature and any material feature After obtaining three planes through inverse normalization, bilinear interpolation is performed to sample features from the light rays emitted from them based on camera parameters, and color c is obtained through a material decoder.
[0032] The rendering module is used to generate a 3D face based on the density σ, the semantic label s, and the color c, and obtain the semantic mask and the 2D face image of the 3D face from a specified viewpoint through volume rendering.
[0033] The aforementioned 3D face editing system based on neural radiation field decoupling further includes:
[0034] The editing module is used to edit the semantic mask to obtain a new mask, measure the distance between the new mask and the semantic mask through the cross-entropy function, optimize the intermediate latent variable w in reverse, and obtain the edited 3D face image of the face in a specified viewpoint based on the optimized intermediate latent variable w as the editing result.
[0035] This sampling module is used for normalization. Obtain the normalized three planes and material characteristics mean and variance
[0036] This sampling module is used to decompose and normalize the three planes. Divided into three parts, the three coordinate planes that make up the volume space are formed. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0037] Given camera parameters In the case of rotation and translation, the light rays emitted from the camera position in the volume space V are constructed, and the corresponding features are obtained by sampling points on each light ray and sending them into the geometric decoder to obtain the density σ and the semantic label s;
[0038] This decoding module is used for normalized three-plane decoding. and material characteristics mean and variance By anti-homogenization Obtain three planes
[0039] By decomposing and normalizing the three planes It consists of three parts, forming the three coordinate planes of the volume space, thus creating the volume space. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0040] Given camera parameters In this case, construct the volume space based on rotation and translation. The light rays emitted from the camera position are sampled at each ray, and the corresponding features are sent to the material decoder to obtain the color c;
[0041] The rendering module is used to obtain the final semantic label rendered on each ray through discretized volume rendering based on the density σ and semantic label s, and obtain the rendered semantic mask; based on the color c, the final color rendered on each ray is obtained through discretized volume rendering and density σ, and finally the two-dimensional face image is obtained.
[0042] The present invention also proposes a storage medium for storing a program that executes any of the three-dimensional face editing methods based on neural radiation field decoupling.
[0043] The present invention also proposes a client for any of the aforementioned three-dimensional face editing systems based on neural radiation field decoupling.
[0044] As can be seen from the above solutions, the advantages of the present invention are:
[0045] The system designed in this invention enables users to decouple the geometry and material of a 3D face. The geometry is edited through semantic masks, while the material can be transferred in 3D style by giving a reference object.
[0046] System flowchart as follows Figure 1 As shown, the system design incorporates several techniques, including three-plane feature decomposition and recombination, and geometric material decoupling generation.
[0047] like Figure 2 As shown, this invention illustrates a training strategy for achieving stylistic consistency in 3D-level two-dimensional face images rendered with the same material features but different geometric features. Based on the semantic mask corresponding to the face, each part (e.g., hair, skin, mouth, etc.) is stylistically constrained for similarity using a histogram color distribution loss function.
[0048] like Figure 3 As shown, this system demonstrates that it can achieve 3D style transfer effects for different 3D faces while maintaining geometry, based on different style references. It can be seen that the system of this invention can accurately achieve style transfer in 3D, exhibiting stable style transfer effects from different viewpoints.
[0049] like Figure 4 As shown, this system allows users to edit different 3D faces from different viewpoints using generated semantic masks. It then performs reverse optimization of latent variables to achieve geometric editing in 3D space. It can be seen that after the user completes geometric editing from different viewpoints, the system can achieve semantic editing while keeping other semantic masks unchanged, and maintain this editing effect from different viewpoints.
[0050] like Figure 5As shown, the significance of each design component of this system is illustrated. (a) Tests related to geometric decoupling editing are presented. Baseline 1 demonstrates the effect of geometric decoupling editing when geometry and material are not decoupled, i.e., without designing a three-plane decomposition and merging process, and without using a geometry decoder and a material decoder, but only using a single decoder to extract feature prediction density, semantic labels, and color from the original three planes. Baseline 2 utilizes a third-party network for predicting semantic masks for two-dimensional face images to directly optimize latent variables in reverse. Baseline 3 demonstrates the effect of geometric decoupling editing when the face volume space and semantic mask volume space are misaligned, i.e., by directly adding a new decoder to the original three-dimensional face generation model to extract feature prediction semantic labels from the original three planes. It can be seen that compared to the final design of this invention, these methods cannot achieve precise editing to some extent, such as hair length and eye size. (b) Tests related to material style transfer are presented. Baseline 1 demonstrates the effect of traditional style-mixing of latent variables. Baseline 2 demonstrates the style transfer effect of using the method of this invention to decouple geometry and materials, but without using a training strategy that promotes similar color distributions in different facial parts of face images rendered with volumetric rendering of different geometric features of the same style. Baseline 3 demonstrates the effect of using the method of this invention to decouple geometry and materials, but with a training strategy that constrains similar styles, by measuring and constraining the color distribution of the entire face rather than constraining each facial part separately. It can be seen that these methods cannot accurately transfer style to some extent, and even have some color diffusion problems (e.g., background color diffusion onto the hair). The method of this invention, however, can accurately achieve style transfer. Attached Figure Description
[0051] Figure 1 This is a network structure diagram of the system of the present invention;
[0052] Figure 2 This diagram illustrates the training strategy used by the system to improve the effectiveness of material style transfer.
[0053] Figure 3 This is a rendering of a 3D face style transfer.
[0054] Figure 4 An image showing the effect of geometric editing of a 3D face using semantic masks;
[0055] Figure 5 This demonstrates the effectiveness of each design component of the system. Detailed Implementation
[0056] This invention provides a method and system for generating and editing high-quality 3D faces with semantic mask interaction and decoupled control over materials, including the following key technical points:
[0057] Key point 1, Three-plane decomposition module
[0058] By employing standardization and destandardization, the three planes, which are essentially two-dimensional features, are transformed into two-dimensional geometric features with a mean of zero and a variance of one through standardization. The original mean and variance are then used as material features. When the material features and geometric features are combined through destandardization, the original three planes are restored.
[0059] Key point 2: Feature decoupling generation module for three-plane decomposition
[0060] To achieve decoupling between geometry and material properties, this invention designs a geometry decoder and a material decoder. The geometry decoder directly predicts density and semantic labels from geometric features, i.e., features obtained by bilinear interpolation sampling in a normalized three-plane configuration, thus determining the geometry of the face space. The material decoder, on the other hand, predicts color from features obtained by bilinear interpolation sampling in a non-normalized three-plane configuration, obtained by combining geometric and material features, thus determining the material of the face space. Therefore, when the material features change, the geometric features remain unchanged, achieving the goal of decoupling geometry and material properties.
[0061] Key point 3: Geometric editing mode based on semantic mask
[0062] To provide editable semantic masks, this invention designs a geometric encoder that predicts semantic labels simultaneously with density prediction, and obtains a 2D semantic mask for a given viewpoint through volume rendering, thus aligning the face volume space with the semantic mask volume space. After the user edits the semantic mask, this invention applies cross-entropy constraints to the modified and generated semantic masks to inversely optimize latent variables, thereby achieving effective editing of the 3D geometry. Furthermore, to ensure that unchanged regions are not disturbed by the ambiguity of the semantic mask, this invention also adds a loss function that preserves invariant regions on the semantic mask.
[0063] Key point 4: Training strategy based on fine-tuning
[0064] To address the training difficulties of traditional 3D face model generation, this invention is based on a pre-trained 3D face generation model. By extracting the pre-trained three-plane generation module and combining it with the geometry and material decoder of this invention, which has a small parameter scale, efficient training is performed under the decoupled generation process of this invention. This achieves the goal of geometry and material decoupling, geometry editing, and material style transfer in a short time.
[0065] Key Point 5: Training Strategies for Effective Material Style Transfer
[0066] To ensure that the same material feature has a similar style effect under different geometric features, this invention uses the color distribution of histograms. For each facial part (e.g., hair, skin, mouth, etc.) of a batch of facial images with different geometry but the same material, the color distribution is required to be similar, thereby achieving an effective style transfer effect. That is, for the same geometric feature, different material features or style features of the target reference object are applied to complete the control of the material.
[0067] The shortcomings of existing technologies stem from the failure to use more efficient and higher-quality 3D representations, the mixing of geometry and materials in generation, and the lack of consideration for the effectiveness of style transfer.
[0068] This invention, through research on more efficient and higher-quality 3D representation, specifically the three-plane model, discovers that this deficiency can be addressed by decomposing the three planes into spatially locally varying geometric features and spatially globally invariant material features. The material features are implemented using a statistical feature response method for the three planes. Therefore, after obtaining the decomposed geometric and material features, a decomposition decoder is used to create a geometric decoder and a material decoder, ensuring that the geometric properties in 3D space are entirely controlled by the geometric features, while the material properties are jointly controlled by both geometric and material features. Furthermore, to ensure that faces generated with the same material feature possess similar materiality under different combinations of geometric features, this invention designs a training strategy based on a histogram-based color loss function to explicitly constrain the similarity of color distribution across each facial region for faces with different geometric features but the same material feature.
[0069] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.
[0070] like Figure 1 As shown, a method and system for high-quality 3D face geometry editing and material decoupling control through semantic masking interaction includes:
[0071] S1: Latent variable z obtained by randomly sampling a Gaussian distribution;
[0072] S2: The StyleGAN generator first transforms the input latent variable z into an intermediate latent variable w through a multilayer perceptron, and then synthesizes a three-plane p = (p xy ,p yz ,p xz The three planes represent the entire volumetric space;
[0073] S3: Given three planes p, decompose the three planes into geometric features. and material characteristics F app ;
[0074] S4: Given geometric features A normalized three-plane structure is constructed, from which light rays are emitted according to pre-given camera parameters to perform bilinear interpolation sampling of features, and density σ and semantic label s are obtained through a geometric decoder;
[0075] S5: Given geometric features Material features sampled within any other material feature distribution After obtaining three planes through inverse normalization, bilinear interpolation is performed to sample features from the light rays emitted from them based on camera parameters, and color c is obtained through a geometric decoder.
[0076] S6: Using the density σ, semantic label s, and color c obtained from the above modules, volume rendering is used to generate a two-dimensional face image and semantic mask rendered from a given viewpoint.
[0077] S7: After the user edits the 2D semantic mask, the distance between the edited semantic mask and the generated semantic mask is measured using the cross-entropy function, and the latent variable w is optimized in reverse. Thus, after optimization, the generator, based on the optimized latent variable, generates a locally geometrically edited 3D face image rendered from any given viewpoint, following the processes described in S2-S6.
[0078] The method shown in S3 includes:
[0079] S31: This invention obtains three planes Then, through normalization The normalized three planes, i.e., the geometric features, are obtained. and material characteristics mean and variance
[0080] The method shown in S4 includes:
[0081] S41: In the normalized three-plane... Then, by decomposing it into three parts, forming the three coordinate planes of the volume space, a volume space is finally formed. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0082] S42: Given camera parameters In this case, based on the rotation and translation, light rays emitted from the camera position are constructed in the volume space V, and points are sampled on each ray to obtain the corresponding features.
[0083] S43: Feed the features of each point into the geometric decoder to obtain the density σ and semantic label s.
[0084] The method shown in S5 includes:
[0085] S51: In the normalized three planes and material characteristics mean and variance Then, through anti-homogenization Obtain three planes
[0086] S52: In obtaining the normalized three planes Then, by decomposing it into three parts, forming the three coordinate planes of the volume space, a volume space is finally formed. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0087] S53: Given camera parameters In this case, construct the volume space based on rotation and translation. The light rays emitted from the camera position are sampled at each ray to obtain the corresponding features.
[0088] S54: Input the features of each point into the material decoder to obtain the color c.
[0089] The method shown in S6 includes:
[0090] S61: Based on the density σ and semantic label s obtained in S4, the final semantic label is obtained by rendering each ray through discretized volume rendering, and finally the rendered semantic mask is obtained.
[0091] S62: Based on the color c obtained in S5, the final color of each ray is obtained by discretizing the volume rendering and combining it with the density σ obtained in S4, thus obtaining the rendered face image.
[0092] like Figure 2 As shown, this invention also employs a training strategy to apply histograms to different facial components on the rendered face image, using samples with the same material characteristics but different geometric features, to reflect color distribution and measure distance, constraining similarity, including:
[0093] S1: Sampling yields different geometric features and a specific material feature;
[0094] S2: Use Figure 1 The generation method shown generates a face image and semantic mask for each geometric feature under the material feature;
[0095] S3: Based on the semantic mask, the color distribution of each component (e.g., hair, skin, etc.) on the face image is represented by a histogram. Then, the distance between the histograms is constrained to approach zero, thereby constraining style similarity.
[0096] The method shown in S1 includes:
[0097] S11: A series of latent variables z are obtained through random sampling, and then... Figure 1 The method described herein yields geometric and material features.
[0098] S12: While preserving all geometric features, randomly select one of all material features as the reference material feature.
[0099] The method shown in S3 includes:
[0100] S31: Use the generated semantic mask to form a mask for each component of the face.
[0101] S32: For each component, obtain the corresponding pixels from the rendered face image based on the mask. After obtaining the pixels, calculate the corresponding histogram color distribution for each sample.
[0102] S33: Select the histogram color distribution of the original sample corresponding to the reference material feature, and constrain the distance between the histogram color distribution of all other examples and it to zero.
[0103] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0104] This invention also proposes a three-dimensional face editing system based on neural radiation field decoupling, which includes:
[0105] The initial module is used to perform random sampling based on a Gaussian distribution to obtain the latent variable z, and then transforms the latent variable z into an intermediate latent variable w through a multilayer perceptron.
[0106] The three-plane synthesis module is used to synthesize three planes p = (p...) based on the intermediate implicit variable w. xy ,p yz ,p xz );
[0107] The decomposition module is used to decompose the three planes p into geometric features with local spatial variations. And the material characteristics that are globally invariant in space F app ;
[0108] Sampling module, used to sample based on this geometric feature A normalized three-plane structure is constructed, and light rays are emitted from it according to camera parameters to perform bilinear interpolation to sample features. The density σ and semantic label s are obtained through a geometric decoder.
[0109] The decoding module, based on this geometric feature and any material feature After obtaining three planes through inverse normalization, bilinear interpolation is performed to sample features from the light rays emitted from them based on camera parameters, and color c is obtained through a material decoder.
[0110] The rendering module is used to generate a 3D face based on the density σ, the semantic label s, and the color c, and obtain the semantic mask and the 2D face image of the 3D face from a specified viewpoint through volume rendering.
[0111] The aforementioned 3D face editing system based on neural radiation field decoupling further includes:
[0112] The editing module is used to edit the semantic mask to obtain a new mask, measure the distance between the new mask and the semantic mask through the cross-entropy function, optimize the intermediate latent variable w in reverse, and obtain the edited 3D face image of the face in a specified viewpoint based on the optimized intermediate latent variable w as the editing result.
[0113] This sampling module is used for normalization. Obtain the normalized three planes and material characteristics mean and variance
[0114] This sampling module is used to decompose and normalize the three planes. Divided into three parts, the three coordinate planes that make up the volume space are formed. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0115] Given camera parameters In the case of rotation and translation, the light rays emitted from the camera position in the volume space V are constructed, and the corresponding features are obtained by sampling points on each light ray and sending them into the geometric decoder to obtain the density σ and the semantic label s;
[0116] This decoding module is used for normalized three-plane decoding. and material characteristics mean and variance By anti-homogenization Obtain three planes
[0117] By decomposing and normalizing the three planes It consists of three parts, forming the three coordinate planes of the volume space, thus creating the volume space. The features of each point are obtained by projecting the corresponding features onto each coordinate plane and then adding them together.
[0118] Given camera parameters In this case, construct the volume space based on rotation and translation. The light rays emitted from the camera position are sampled at each ray, and the corresponding features are sent to the material decoder to obtain the color c;
[0119] The rendering module is used to obtain the final semantic label rendered on each ray through discretized volume rendering based on the density σ and semantic label s, and obtain the rendered semantic mask; based on the color c, the final color rendered on each ray is obtained through discretized volume rendering and density σ, and finally the two-dimensional face image is obtained.
[0120] The present invention also proposes a storage medium for storing a program that executes any of the three-dimensional face editing methods based on neural radiation field decoupling.
[0121] The present invention also proposes a client for any of the aforementioned three-dimensional face editing systems based on neural radiation field decoupling.
Claims
1. A three-dimensional face editing method based on neural radiation field decoupling, characterized in that, include: Step 1: Perform random sampling based on Gaussian distribution to obtain latent variables. The latent variable is then processed using a multilayer perceptron. Transform into intermediate hidden variables ; Step 2: Utilize this intermediate hidden variable Synthetic three planes ; Step 3, the three planes Decomposed into geometric features with local variations in space. And spatially invariant material characteristics ;pass Obtain the normalized three planes and material characteristics mean and variance The normalized three planes are the geometric features. ; Step 4: Based on this geometric feature The features are sampled by bilinear interpolation based on the emitted light rays according to the camera parameters, and the density is obtained through a geometric decoder. and semantic tags ; Step 5: Based on this geometric feature and any material feature After obtaining three planes through inverse normalization, bilinear interpolation is performed on the emitted light rays based on camera parameters to sample features, and the color is obtained through a material decoder. ; The specific process of obtaining the three planes through inverse normalization includes: based on this geometric feature... and material characteristics mean and variance The three planes are obtained by inverse normalization. , ; Step 6, based on this density This semantic tag and that color A 3D face is generated, and a semantic mask and a 2D face image of the 3D face from a specified viewpoint are obtained through volume rendering.
2. The three-dimensional face editing method based on neural radiation field decoupling as described in claim 1, characterized in that, Also includes: Step 7: Edit the semantic mask to obtain a new mask, measure the distance between the new mask and the semantic mask using the cross-entropy function, and back-optimize the intermediate latent variable. Based on the optimized intermediate latent variables The edited 3D face image obtained through steps 1 to 6 is a 2D face image of the edited 3D face from a specified viewpoint, which is the result of the editing process.
3. The three-dimensional face editing method based on neural radiation field decoupling as described in claim 1, characterized in that, Step 4 includes: By decomposing and normalizing the three planes Divided into three parts, the three coordinate planes that make up the volume space are formed. The feature of each point is obtained by projecting the corresponding features onto each coordinate plane and then adding them together. Given camera parameters In this case, construct the volume space based on rotation and translation. The density is obtained by sampling points on each ray of light emitted from the camera position and sending the corresponding features to the geometric decoder. and the semantic tag .
4. The three-dimensional face editing method based on neural radiation field decoupling as described in claim 3, characterized in that, Step 5 includes: By decomposing and normalizing the three planes It consists of three parts, forming the three coordinate planes of the volume space, thus creating the volume space. The feature of each point is obtained by projecting the corresponding features onto each coordinate plane and then adding them together. Given camera parameters In this case, construct the volume space based on rotation and translation. The light rays emitted from the camera position are sampled, and the corresponding features of each ray are obtained and sent to the material decoder to obtain the color. .
5. The three-dimensional face editing method based on neural radiation field decoupling as described in claim 4, characterized in that, Step 6 includes: Based on this density and semantic tags The final semantic label is obtained by rendering each ray through discretized volume rendering, resulting in the rendered semantic mask; based on the color... Through discretized volume rendering and density The final color rendered on each ray is obtained, and the two-dimensional face image is finally obtained.
6. A three-dimensional face editing system based on neural radiation field decoupling, characterized in that, include: The initial module is used to perform random sampling based on a Gaussian distribution to obtain latent variables. The latent variable is then processed using a multilayer perceptron. Transform into intermediate hidden variables ; The three-plane synthesis module is used to synthesize the intermediate implicit variable. Synthetic three planes ; The decomposition module is used to decompose the three planes. Decomposed into geometric features with local variations in space. And spatially invariant material characteristics ;pass Obtain the normalized three planes and material characteristics mean and variance The normalized three planes are the geometric features. ; Sampling module, used to sample based on this geometric feature The features are sampled by bilinear interpolation based on the emitted light rays according to the camera parameters, and the density is obtained through a geometric decoder. and semantic tags ; The decoding module, based on this geometric feature and any material feature After obtaining three planes through inverse normalization, bilinear interpolation is performed on the emitted light rays based on camera parameters to sample features, and the color is obtained through a material decoder. ; The specific process of obtaining the three planes through inverse normalization includes: based on this geometric feature... and material characteristics mean and variance The three planes are obtained by inverse normalization. , ; The rendering module is used to render based on this density. This semantic tag and that color A 3D face is generated, and a semantic mask and a 2D face image of the 3D face from a specified viewpoint are obtained through volume rendering.
7. The three-dimensional face editing system based on neural radiation field decoupling as described in claim 6, characterized in that, Also includes: The editing module is used to edit the semantic mask to obtain a new mask, measure the distance between the new mask and the original semantic mask using the cross-entropy function, and back-optimize the intermediate latent variable. Based on the optimized intermediate latent variables The edited 3D face is obtained as a 2D face image from a specified viewpoint as the editing result; This sampling module is used to decompose and normalize the three planes. Divided into three parts, the three coordinate planes that make up the volume space are formed. The feature of each point is obtained by projecting the corresponding features onto each coordinate plane and then adding them together. Given camera parameters In this case, construct the volume space based on rotation and translation. The density is obtained by sampling points on each ray of light emitted from the camera position and sending the corresponding features to the geometric decoder. and the semantic tag ; This decoding module is used to decompose the normalized three planes. It consists of three parts, forming the three coordinate planes of the volume space, thus creating the volume space. The feature of each point is obtained by projecting the corresponding features onto each coordinate plane and then adding them together. Given camera parameters In this case, construct the volume space based on rotation and translation. The light rays emitted from the camera position are sampled, and the corresponding features of each ray are obtained and sent to the material decoder to obtain the color. ; This rendering module is used to determine the density. and semantic tags The final semantic label is obtained by rendering each ray through discretized volume rendering, resulting in the rendered semantic mask; based on the color... Through discretized volume rendering and density The final color rendered on each ray is obtained, and the two-dimensional face image is finally obtained.
8. A storage medium for storing a program that executes the three-dimensional face editing method based on neural radiation field decoupling as described in any one of claims 1 to 5.
9. A client for the three-dimensional face editing system based on neural radiation field decoupling as described in claim 6 or 7.