Explanatable and controllable three-dimensional reconstruction method and system based on neural radiation field

Abstract

The invention relates to an interpretable and controllable three-dimensional reconstruction method based on a neural radiation field, which effectively explores and identifies potential three-dimensional semantics by integrating decoupling representation learning (DRL) and multi-modal large language models (MLLMs). And meanwhile, a specially designed neural radiation field (NeRF) model is utilized to maintain high-quality three-dimensional reconstruction with consistent visual angles. Through the method, the accuracy of reconstruction is ensured, and the model is endowed with semantic perception and understanding ability similar to that of a human brain. Through the method, the human brain-like semantic perception and understanding ability of the model is enabled while the reconstruction accuracy is ensured, and a multi-scale, multi-level and explainable model autonomous decision-making mechanism from data and information to semantics, knowledge and rules is further constructed, so that first understanding, second decoupling and second control are realized on the semantic level, and the accuracy of reconstruction is improved. And finally, a new-generation explainable and controllable three-dimensional reconstruction framework is formed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an interpretable and controllable three-dimensional reconstruction method based on neural radiation fields. Background Art

[0002] 3D reconstruction aims to represent 3D objects by reconstructing a mathematical model suitable for computer processing using a certain amount of prior data. It is a key technology for digitally expressing the objective world. In recent years, the rapid development of artificial intelligence has promoted a major upgrade of traditional 3D reconstruction technology and provided new possibilities for innovation in this field. Compared with traditional explicit modeling forms (such as point clouds, meshes, and voxels), 3D implicit reconstruction methods represented by neural radiance fields (NeRF) have significant advantages in reconstruction quality, memory consumption, and dependence on manual annotation. However, due to the complexity of real 3D environments and the limitations of artificial intelligence, existing NeRF-based 3D reconstruction models still face problems in interpretability and controllability.

[0003] Specifically, the existing "black box" implicit reconstruction technology with deep networks as the core is a shortcut learning strategy. This method does not simulate the process of humans acquiring the physical laws of the three-dimensional world through decoupled semantic representations, but directly trains the network based on large-scale data to fit the mapping relationship between input and output. This type of black box strategy leads to the problems of unexplainability and difficulty in control of existing technologies, which restricts their application in complex tasks and sensitive fields. Decoupled representation learning (DRL) aims to address this challenge by decomposing complex data into independent and interpretable latent factors to achieve independent control of specific attributes (such as color, shape, size). However, most existing DRL methods are based on two-dimensional data, while the data volume of three-dimensional scenes is huge, the noise level is high, and the dimensions are complex, which makes it extremely challenging to directly apply the representation decoupling theory and semantic control methods originally designed for two-dimensional images to the three-dimensional field.

[0004] Chinese patent application publication number CN117611722A discloses a 3D reconstruction method based on generative neural radiance fields. By introducing generative neural radiance fields and decoupled representation learning into 3D reconstruction technology, this method addresses the interpretability and robustness issues of existing technologies for the reconstruction of irregular geometries and high-precision structures. This method achieves efficient and accurate 3D scene reconstruction and improves the diversity and robustness of the model. However, this method still suffers from poor interpretability and insufficient ability to discover and identify potential 3D semantics.

[0005] In summary, there is currently a lack of an interpretable and controllable three-dimensional reconstruction method and system to solve or partially solve the aforementioned problems. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an interpretable and controllable three-dimensional reconstruction method and system based on neural radiation fields, so as to solve or partially solve the problems of poor interpretability and insufficient ability to discover and identify potential three-dimensional semantics.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] One aspect of the present invention provides an interpretable and controllable three-dimensional reconstruction method based on neural radiation fields, comprising the following steps:

[0009] Obtain an input image, use a pre-trained disentangled representation learning network as a teacher network, a semantic encoder as a student network, and the input image as input to the semantic encoder, and obtain interpretable latent variables through implicit distillation;

[0010] A randomly sampled bias is applied to the interpretable latent variable, and the W+ space is sampled through a trainable affine transformation and a mapping transformation, respectively. A pre-trained generative neural radiation field is used to obtain an external coarse-grained decoupled image and an internal fine-grained decoupled image after 3D reconstruction, respectively. The bias is trained with the goal of minimizing a loss function including reconstruction loss, synergy loss, and consistency loss to obtain a decoupled representation vector.

[0011] A multimodal large model is used to identify the attributes with the highest discrimination in the input image and the built-in fine-grained decoupled image, and an association is established between the attribute and the decoupled representation vector to complete the three-dimensional reconstruction.

[0012] As a preferred technical solution, after obtaining the external coarse-grained decoupled image and the internal fine-grained decoupled image, the following is further included:

[0013] For the interpretable latent variables, an original reconstructed image is obtained by using a pre-trained generative neural radiance field;

[0014] The original reconstructed image, the external coarse-grained decoupled image, and the internal fine-grained decoupled image are respectively passed through a trainable decoder to obtain respective bias information.

[0015] As a preferred technical solution, a random sampling bias is applied to the interpretable latent variable, and the process of sampling W+ space by mapping transformation and obtaining a built-in fine-grained decoupled image using a pre-trained generative neural radiation field includes:

[0016] For the interpretable latent variables, the latent variables are converted into intermediate variables in the W+ space through mapping transformation;

[0017] The sampled bias is appended to the 9th to 18th dimensions of the intermediate variable, and the pre-trained generative neural radiance field is used to obtain the built-in fine-grained decoupled image.

[0018] As a preferred technical solution, the reconstruction loss is:

[0019] L R =MSE((Δβ', Δε'), (β, ε))

[0020] Among them, L R is the reconstruction loss, MSE() is the cross entropy function, Δβ′ and Δε′ are the bias direction increment and bias speed increment of the external coarse-grained decoupled image or the built-in fine-grained decoupled image predicted by the decoder, respectively, and β and ε are the bias direction and bias speed of random sampling, respectively.

[0021] As a preferred technical solution, the synergistic loss is:

[0022] L S =CrossEntropy((Δβ',Δε'),(Δβ",Δε"))

[0023] Among them, L s is the collaborative loss, CrossEntropy() is the cross entropy loss function, Δβ′ and Δε′ are the bias direction increment and bias speed increment of the external coarse-grained decoupled image predicted by the decoder, respectively, Δβ″ and Δε″ are the bias direction increment and bias speed increment of the built-in fine-grained decoupled image predicted by the decoder, respectively.

[0024] As a preferred technical solution, the consistency loss is:

[0025]

[0026] Among them, L c is the collaborative loss, P is the pixel set of the image, (i, j) represents the pixel point in the image, R [i，j] is the sampling Gaussian point in the generated radiation field, I prptotype and I2 correspond to the rendering results of the built-in decoupling in the prototype network and the main network under the current sampling perspective, respectively.

[0027] As a preferred technical solution, the loss function is:

[0028] L total =L S +λ R L R +λ C L C

[0029] Among them, L total is the total loss function, L s 、L R 、L CThey are consistency loss, reconstruction loss, and collaborative loss, respectively, R ,λ C As a parameter.

[0030] As a preferred technical solution, the decoupled representation learning network is a β-VAE, β-TCVAE or FactorVAE network.

[0031] As a preferred technical solution, after obtaining the input image, the method further includes:

[0032] Data cleaning and key point detection processing are performed on the input image.

[0033] Another aspect of the present invention provides an interpretable and controllable 3D reconstruction system based on neural radiation fields, characterized in that it is used to implement the aforementioned interpretable and controllable 3D reconstruction method based on neural radiation fields, and the interpretable and controllable 3D reconstruction system includes:

[0034] A two-dimensional perception module is used to obtain an input image, use a pre-trained decoupled representation learning network as a teacher network, use a semantic encoder as a student network, and use the input image as input to the semantic encoder to obtain interpretable latent variables through implicit distillation;

[0035] A 3D reconstruction module is configured to apply a randomly sampled bias to the interpretable latent variable, sample the W+ space through a trainable affine transformation and a mapping transformation, respectively, and obtain a coarse-grained decoupled image and a fine-grained decoupled image after 3D reconstruction using a pre-trained generative neural radiation field. The bias is trained to obtain a decoupled representation vector with the goal of minimizing a loss function including reconstruction loss, collaboration loss, and consistency loss.

[0036] The attribute recognition module is used to use the multimodal large model to identify the attributes with the highest discrimination in the input image and the built-in fine-grained decoupled image, establish the association between the attribute and the decoupled representation vector, and complete the three-dimensional reconstruction.

[0037] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0038] (1) Achieving interpretable 3D reconstruction: Based on neural radiation fields and multimodal large models, the present invention proposes a method for introducing fine-grained decoupling in 3D reconstruction scenes. By transforming the interpretable latent variable radioactive sample W+ space, a pre-trained generative neural radiation field is used to obtain a built-in fine-grained decoupled image, and a decoupled representation vector with abstract meaning is obtained through training. Finally, a multimodal large model is used to establish the association between the abstract decoupled representation vector and the concrete attributes, thereby enhancing the controllability and interpretability of the scene while ensuring the accuracy and efficiency of scene reconstruction.

[0039] (2) Realize fine-grained semantic dimension control: The present invention trains the bias by taking the loss functions of reconstruction loss, collaboration loss and consistency loss as the target, and guides the model to autonomously learn the bias direction of the latent variable corresponding to the representation attribute, thereby achieving fine-grained control of a single semantic dimension.

[0040] (3) Strong controllability of three-dimensional scenes: The present invention uses a large multimodal model to construct relationship pairs between abstract semantics and natural semantics, thereby improving the controllability and interactivity of three-dimensional scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flowchart of an interpretable and controllable three-dimensional reconstruction method based on neural radiation field in an embodiment;

[0042] Figure 2 It is a framework diagram of the model in the embodiment;

[0043] Figure 3 Graph showing the experimental results of verifying the fine-grained 3D reconstruction and decoupling capabilities of the method on real data in the embodiment;

[0044] Figure 4 The figure is a comparison and verification experimental result diagram of the present method and the existing method in the embodiment;

[0045] Figure 5 Graph showing the results of verification tests of the method and multiple multimodal large models in the embodiments;

[0046] Figure 6 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0048] Example 1

[0049] To address the aforementioned issues with the existing technologies, this embodiment provides an interpretable and controllable 3D reconstruction method based on neural radiance fields. This method constructs an interpretable 3D reconstruction framework. This framework effectively discovers and identifies underlying 3D semantics by integrating disentangled representation learning (DRL) and multimodal large language models (MLLMs). Furthermore, a specially designed neural radiance field (NeRF) model is utilized to maintain high-quality and perspective-consistent 3D reconstruction.

[0050] See also Figure 1, the method mainly includes the following steps:

[0051] Step S1: obtain an input image, use a pre-trained disentangled representation learning (DRL) network as a teacher network, a semantic encoder as a student network, and the input image as the input of the semantic encoder, and obtain interpretable latent variables through implicit distillation.

[0052] The purpose of this step is to extract knowledge from the pre-trained DRL network through implicit distillation technology and construct an orthogonal and well-decoupled latent space.

[0053] Specifically, this step first preprocesses the input image through data cleaning and keypoint detection. Then, a pre-trained disentangled representation learning network, such as β-VAE, β-TCVAE, and FactorVAE, is used as the teacher network, and a lightweight semantic encoder is used as the student network. Using implicit distillation, the semantically disentangled latent space information is transferred to the student network, resulting in the sampling of interpretable and orthogonal latent variables.

[0054] In step S2, a randomly sampled bias is applied to the interpretable latent variables, and the W+ space is sampled through trainable affine transformation and mapping transformation, respectively. The pre-trained generative neural radiation field is used to obtain the external coarse-grained decoupled image and the internal fine-grained decoupled image after three-dimensional reconstruction, respectively. With the goal of minimizing the loss function including reconstruction loss, collaboration loss and consistency loss, the bias is trained to obtain the decoupled representation vector.

[0055] This step focuses on identifying and extracting semantic factors in the learned latent space while achieving high-quality 3D reconstruction.

[0056] Specifically, this step may include steps S201-S204:

[0057] In step S201, a random sampling bias is applied to the interpretable latent variables, and a trainable affine transformation and a mapping transformation are performed to sample the W+ space.

[0058] Step S202: 3D reconstructing the external coarse-grained decoupled image and the internal fine-grained decoupled image.

[0059] Step S203: The decoder decodes and obtains the bias direction and bias speed information.

[0060] In summary, steps S201-S203 include three branches: an external decoupling branch of the global dimension, a built-in decoupling branch of the local dimension, and an original reconstruction branch.

[0061] Original reconstruction branch, this branch is based on latent variables, uses generative neural radiation fields to obtain the original reconstructed image, and obtains the reference bias direction and reference bias speed through the decoder.

[0062] The external decoupling branch first applies a randomly sampled bias to the obtained latent variables. Using a trainable affine transformation, the generative neural radiance field outputs the biased image, known as the external decoupled image. The decoder then predicts the bias direction and bias velocity of the external coarse-grained decoupled image. By subtracting these from the baseline bias direction and bias velocity, the bias direction increment and bias velocity increment of the external coarse-grained decoupled image are obtained. This branch aims to find the global or coarse-grained representation of the latent variable direction, but lacks the ability to perceive fine-grained representations.

[0063] The built-in decoupling branch, based on the StyleGAN model, first transforms the latent variable into an intermediate variable through a mapping transformation. This intermediate variable is sampled from the decoupled W+ space and contains 18 representation dimensions. According to previous research, the 9th to 18th dimensions of the intermediate variable primarily control the fine-grained representation of the generated image. Therefore, in the built-in branch, the sampled bias is linearly appended to the 9th to 18th dimensions, and a generative neural radiation field is used to generate a built-in decoupled image. The built-in decoupled image is decoded to obtain the predicted bias direction and bias velocity of the built-in coarse-grained decoupled image. By subtracting these from the baseline bias direction and bias velocity, the bias direction increment and bias velocity increment of the built-in coarse-grained decoupled image are obtained. This branch aims to guide the model to focus on capturing local fine-grained representations.

[0064] Step S204 , calculate the reconstruction loss, collaboration loss, and consistency loss, and train the bias.

[0065] The training objects include the lightweight semantic encoder in step S1, the trainable affine transformation in step S2, and the decoder.

[0066] After training, the decoder learns the mapping between image representation attributes and latent variable biases by reconstructing the constraints of the loss function and using sampled latent variable biases for self-supervision. The encoder predicts the corresponding latent variable bias direction based on the input image relative to the reference image. Thanks to the external decoupling branch, the model is able to identify latent variable directions with preliminary decoupling characteristics.

[0067] The purpose of training is to minimize the loss function including reconstruction loss (LR), synergistic loss (LS) and consistency loss (LC). Among them, the reconstruction loss function is used for internal and external decoupling branches to supervise the reconstructor through the true representation direction to guide its ability to find the representation direction:

[0068] L R=MSE((Δβ', Δε'), (β, ε))

[0069] Among them, L R is the reconstruction loss, MSE() is the cross entropy function, Δβ′ and Δε′ are the bias direction increment and bias speed increment of the external coarse-grained decoupled image or the built-in fine-grained decoupled image predicted by the decoder, respectively, and β and ε are the bias direction increment and bias speed increment of random sampling, respectively.

[0070] The collaborative loss function is used to coordinate the two decoupling modules. The built-in decoupling branch uses the decoupling characteristics of the neural mapping network to focus the model on fine-grained representation, thereby enhancing the decoupling capability of the external branch.

[0071] L S =CrossEntropy((Δβ',Δε'),(Δβ",Δε"))

[0072] Among them, L s is the collaborative loss, CrossEntropy() is the cross entropy loss function, Δβ′ and Δε′ are the bias direction increment and bias speed increment of the external coarse-grained decoupled image predicted by the decoder, respectively, Δβ″ and Δε″ are the bias direction increment and bias speed increment of the built-in fine-grained decoupled image predicted by the decoder, respectively.

[0073] The consistency loss function extracts 3D information through a prototype neural radiation field, ultimately achieving fine-grained decoupling in the 3D scene:

[0074]

[0075] Among them, L c is the collaborative loss, P is the pixel set of the image, (i, j) represents the pixel point in the image, R [i，j] is the sampling Gaussian point in the generated radiation field, I prototype and I2 correspond to the built-in decoupled rendering results of the prototype network and the main network under the current sampling perspective, respectively. The lightweight 3D prototype network (i.e., the prototype neural radiation field) constructed in this embodiment shares training data with the 3D reconstruction module (i.e., the main network), and trains a low-resolution but complete 3D information generative NeRF model. Through the semantic consistency loss function, it provides consistency information for the 3D reconstruction module in a dynamic and real-time framework. Therefore, the overall loss function can be written as:

[0076] L total =L S +λ R L R +λ C L C

[0077] Among them, L total is the total loss function, L s 、L R 、L C They are consistency loss, reconstruction loss, and collaborative loss, respectively, R ,λ C As a parameter.

[0078] In step S3, the multimodal large model is used to identify the attribute with the maximum discrimination in the input image and the built-in fine-grained decoupled image, and an association is established between the attribute and the decoupled representation vector to complete the three-dimensional reconstruction.

[0079] After analyzing and decoupling the latent space in the aforementioned steps, a decoupled representation vector based on the abstract semantics of the latent space is obtained. However, there is no direct correspondence between the abstract semantics at this time and the natural semantics that humans intuitively understand.

[0080] In the past, decoupled representation methods mainly relied on manual annotation to determine the mapping relationship between the two. This is not only labor-intensive and inefficient, but also seriously restricts the scope of application and generalization ability of decoupled representation learning in practical application scenarios. To solve this problem, this step uses the powerful physical law reasoning ability of the pre-trained multimodal model to automatically construct the relationship between abstract semantics and natural semantics in an unsupervised manner, thereby enhancing the controllability and interactivity of the model. Figure 2 As shown, the original image and the offset image are fed into a multimodal large model predictor. This predictor, based on hint engineering, allows the large model to predict inconsistent representational attributes between the two images, namely the offset attributes of the latent vector. Through training and attribute recognition, this module identifies the semantic attributes corresponding to each dimension of the latent variable, providing users with a more convenient three-dimensional control approach.

[0081] Specifically, the project prompts the large model to identify the most discriminative attribute (e.g., the "beard" attribute in the example) between the input image and the built-in decoupled image obtained in step S2. The model then maps the found text and the bias dimension in the latent code to each other, facilitating direct user input and voice control to complete 3D reconstruction.

[0082] See also Figure 3 To verify the fine-grained 3D reconstruction and decoupling capabilities on real data, this paper achieves high-quality reconstruction of 3D scenes and fine-grained decoupling of representational attributes. It should be noted that the input images were obtained from public channels or generated by AI.

[0083] See also Figure 43D decoupling and reconstruction of the present invention are better than those of other existing models in the comparative verification experiment.

[0084] See also Figure 5 This figure shows the verification test results of this method and multiple multimodal large models, through verification experiments on multiple multimodal large models.

[0085] This method not only ensures reconstruction accuracy but also endows the model with semantic perception and comprehension capabilities similar to those of the human brain. By doing so, the present invention not only ensures reconstruction accuracy but also empowers the model with semantic perception and comprehension capabilities similar to those of the human brain. This further constructs a multi-scale, multi-layered, and interpretable model-autonomous decision-making mechanism, spanning data and information to semantics, knowledge, and rules. This achieves "understanding first, decoupling second, and control third" at the semantic level, ultimately forming a new generation of interpretable and controllable 3D reconstruction framework.

[0086] Example 2

[0087] Based on Example 1, see Figure 2 This embodiment further provides an interpretable and controllable 3D reconstruction system based on neural radiation fields, which is used to implement the interpretable and controllable 3D reconstruction method based on neural radiation fields of embodiment 1. The interpretable and controllable 3D reconstruction system includes:

[0088] (1) A two-dimensional perception module, which is used to obtain an input image, use a pre-trained decoupled representation learning network as a teacher network, a semantic encoder as a student network, and the input image as the input of the semantic encoder, and obtain interpretable latent variables through implicit distillation;

[0089] (2) A 3D reconstruction module is used to apply a random sampling bias to the interpretable latent variables. The W+ space is sampled through trainable affine transformation and mapping transformation, and the pre-trained generative neural radiation field is used to obtain the external coarse-grained decoupled image and the internal fine-grained decoupled image after 3D reconstruction. The bias is trained to obtain the decoupled representation vector with the goal of minimizing the loss function including reconstruction loss, collaboration loss and consistency loss.

[0090] (3) An attribute recognition module is used to use a multimodal large model to identify the attribute with the maximum discrimination in the input image and the built-in fine-grained decoupled image, establish an association between the attribute and the decoupled representation vector, and complete three-dimensional reconstruction.

[0091] Example 3

[0092] See also Figure 6This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the interpretable and controllable three-dimensional reconstruction method based on neural radiation field as described in Example 1.

[0093] like Figure 6 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0094] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0095] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0096] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An interpretable and controllable 3D reconstruction method based on neural radiation field, characterized in that: The steps include: Obtain an input image, use a pre-trained disentangled representation learning network as a teacher network, a semantic encoder as a student network, and the input image as input to the semantic encoder, and obtain interpretable latent variables through implicit distillation; A randomly sampled bias is applied to the interpretable latent variable, and the W+ space is sampled through a trainable affine transformation and a mapping transformation, respectively. A pre-trained generative neural radiation field is used to obtain an external coarse-grained decoupled image and an internal fine-grained decoupled image after 3D reconstruction, respectively. The bias is trained with the goal of minimizing a loss function including reconstruction loss, synergy loss, and consistency loss to obtain a decoupled representation vector. A multimodal large model is used to identify the attributes with the highest discrimination in the input image and the built-in fine-grained decoupled image, and an association is established between the attribute and the decoupled representation vector to complete the three-dimensional reconstruction.

2. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: After obtaining the external coarse-grained decoupled image and the internal fine-grained decoupled image, the following steps are also included: For the interpretable latent variables, an original reconstructed image is obtained by using a pre-trained generative neural radiance field; The original reconstructed image, the external coarse-grained decoupled image, and the internal fine-grained decoupled image are respectively passed through a trainable decoder to obtain respective bias information.

3. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: The process of applying a random sampling bias to the interpretable latent variable, sampling the W+ space through mapping transformation, and obtaining a built-in fine-grained decoupled image using the pre-trained generative neural radiance field includes: For the interpretable latent variables, the latent variables are converted into intermediate variables in the W+ space through mapping transformation; The sampled bias is appended to the 9th to 18th dimensions of the intermediate variable, and the pre-trained generative neural radiance field is used to obtain the built-in fine-grained decoupled image.

4. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: The reconstruction loss is: L R =MSE((Δβ′, Δε′),(β,ε)) Among them, L R is the reconstruction loss, MSE() is the cross entropy function, Δβ′ and Δε′ are the bias direction increment and bias speed increment of the external coarse-grained decoupled image or the built-in fine-grained decoupled image predicted by the decoder, respectively, and β and ε are the bias direction and bias speed of random sampling, respectively.

5. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: The synergy loss is: L S =CrossEntropy((Δβ′, Δε′),(Δβ″, Δε″)) Among them, L s is the collaborative loss, CrossEntropy() is the cross entropy loss function, Δβ′ and Δε′ are the bias direction increment and bias speed increment of the external coarse-grained decoupled image predicted by the decoder, respectively, Δβ″ and Δε″ are the bias direction increment and bias speed increment of the built-in fine-grained decoupled image predicted by the decoder, respectively.

6. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: The consistency loss is: Among them, L c is the collaborative loss, P is the pixel set of the image, (i, j) represents the pixel point in the image, R [i,j] is the sampling Gaussian point in the generated radiation field, I prototype and I2 correspond to the rendering results of the built-in decoupling in the prototype network and the main network under the current sampling perspective, respectively.

7. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: The loss function is: L total =L S +λ R L R +λ C L C Among them, L total is the total loss function, L s 、L R 、L C They are consistency loss, reconstruction loss, and collaborative loss, respectively, R ,λ C As a parameter.

8. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: The decoupled representation learning network is a β-VAE, β-TCVAE or FactorVAE network.

9. The interpretable and controllable 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: After getting the input image, it also includes: Data cleaning and key point detection processing are performed on the input image.

10. An interpretable and controllable 3D reconstruction system based on neural radiation field, characterized in that: For implementing the interpretable and controllable three-dimensional reconstruction method based on neural radiation fields according to any one of claims 1 to 9, the interpretable and controllable three-dimensional reconstruction system comprises: A two-dimensional perception module is used to obtain an input image, use a pre-trained decoupled representation learning network as a teacher network, use a semantic encoder as a student network, and use the input image as input to the semantic encoder to obtain interpretable latent variables through implicit distillation; A 3D reconstruction module is configured to apply a randomly sampled bias to the interpretable latent variable, sample the W+ space through a trainable affine transformation and a mapping transformation, respectively, and obtain a coarse-grained decoupled image and a fine-grained decoupled image after 3D reconstruction using a pre-trained generative neural radiation field. The bias is trained to obtain a decoupled representation vector with the goal of minimizing a loss function including reconstruction loss, collaboration loss, and consistency loss. The attribute recognition module is used to use the multimodal large model to identify the attributes with the highest discrimination in the input image and the built-in fine-grained decoupled image, establish the association between the attribute and the decoupled representation vector, and complete the three-dimensional reconstruction.

Images