Indoor three-dimensional scene controllable generation model training and application method, equipment and medium

Abstract

The invention discloses an indoor three-dimensional scene controllable generation model training and application method and device and a medium, and relates to the technical field of indoor three-dimensional scene controllable generation. A three-dimensional scene generator is utilized to generate a first neural radiation field for training of an indoor three-dimensional scene, and a first composite image for control, a second composite image for control and a first composite image for training are obtained based on rendering of the first neural radiation field for training, and further calculating judgment loss, scene image condition control consistency loss and room wall boundary condition control consistency loss to update the three-dimensional scene generator and the two-dimensional image discriminator to obtain an updated generator and an updated discriminator, and continuously iterating until an iteration termination condition is reached, so as to obtain a final iteration result. And an indoor three-dimensional scene controllable generation model supporting various condition control is obtained. According to the invention, controllable generation of the indoor three-dimensional scene can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of controllable generation of indoor three-dimensional scenes, and in particular to a method, device and medium for training and applying a controllable generation model of indoor three-dimensional scenes. Background Art

[0002] In recent years, there has been an increasing demand for 3D digital content (which refers to 3D digital models created by 3D modeling software for design and subsequent processing). The production of large amounts of high-quality 3D digital content is key to many applications, such as video games, filmmaking, virtual reality, and augmented reality. As the main place for human life, the generation of indoor 3D scene data is an indispensable technology in the generation of 3D digital content. With the development of industries such as interior design, smart home, indoor robots, and games, the demand for highly realistic indoor 3D scene data is also increasing. For example, the latest methods in the research fields of indoor scene-related artificial intelligence algorithms and embodied intelligence are mostly based on data-driven deep learning algorithms. More indoor 3D scene data can significantly improve the performance of deep learning algorithms, especially a large amount of highly realistic indoor 3D scene data, which is crucial to improving the understanding of indoor space in the real world by intelligent agents such as indoor robots.

[0003] There are two traditional ways to generate indoor 3D scenes. One is to manually design, model, and construct highly complex indoor 3D scenes containing a large number of furniture and decorative items by professional modelers and interior designers. The other is to use scanning equipment to scan real indoor scenes and use 3D reconstruction algorithms to perform 3D reconstruction on the scanned single real indoor scene to obtain the indoor 3D scene of the real indoor scene. However, these two indoor 3D scene generation methods require professional modeling design software or scanning equipment, as well as certain professional knowledge, have high technical barriers, and are time-consuming and labor-intensive. The cost of collecting a large amount of rich and diverse indoor 3D scene data is very high.

[0004] In recent years, a large number of researchers have tried to use artificial intelligence technology to automatically generate indoor 3D scene data. Some researchers use deep learning algorithms to learn the data distribution of indoor 3D scenes from a small number of existing indoor 3D scene datasets (about tens of thousands of indoor 3D scenes), thereby automatically generating new indoor 3D scenes. However, this method is limited by the data quality and scale of the existing indoor 3D scene datasets, and its performance and generalization ability are limited, making it difficult to generate highly realistic indoor 3D scenes that are similar to indoor scenes in the real world. Although highly realistic indoor 3D scene datasets (tens of thousands) are very rare, 2D image data (hundreds of millions) containing a variety of indoor scenes in the real world are very rich and easy to obtain. Therefore, some researchers have tried to learn information about the 3D world from a large amount of 2D image data from different indoor scenes and generate indoor 3D scenes. However, this method has high requirements for image datasets, requiring the shooting angle or distribution of images in the image dataset to be known, or requiring multi-view image data of indoor scenes. It cannot be applied to massive Internet image data without any labeled information, and cannot controllably generate indoor 3D scenes according to user needs. Summary of the invention

[0005] The purpose of this application is to provide a method, device and medium for training and applying a controllable generation model of indoor three-dimensional scenes, which can be based on massive indoor scene-related image data collected directly from the Internet without any labeled information, to train a controllable generation model of indoor three-dimensional scenes that supports multiple condition controls, and further generate indoor three-dimensional scenes.

[0006] To achieve the above objectives, this application provides the following solutions.

[0007] In a first aspect, the present application provides a method for training a controllable generative model of an indoor three-dimensional scene, and the method for training a controllable generative model of an indoor three-dimensional scene comprises the following steps.

[0008] A training control condition and a plurality of indoor scene images without labeled information are obtained, and the indoor scene images without labeled information are used as training images; the training control condition includes the training scene images and the training room wall boundaries.

[0009] A training scene image and a plurality of training images are respectively taken as inputs, and a camera parameter predictor is used to determine a first control camera parameter corresponding to the training scene image and a first training camera parameter corresponding to each training image.

[0010] Taking the training control conditions and the noise satisfying the preset distribution as input, a first training neural radiation field of the indoor three-dimensional scene is generated by using the three-dimensional scene generator.

[0011] Based on the first training neural radiation field and the first control camera parameters, a first control synthetic image is rendered; based on the first training neural radiation field and the second control camera parameters, a second control synthetic image is rendered; based on the first training neural radiation field and all first training camera parameters, a first training synthetic image is rendered; the first control synthetic image, the second control synthetic image and the first training synthetic image all include color images and depth images.

[0012] Based on the training room wall boundary and the second control camera parameters, a depth image of the training room wall boundary is calculated.

[0013] The training image and the first training synthetic image are discriminated by using a two-dimensional image discriminator to obtain a discrimination result, and a discrimination loss is calculated according to the discrimination result.

[0014] According to the similarity between the training scene image and the first control synthetic image, the scene image conditional control consistency loss is calculated.

[0015] The room wall boundary condition control consistency loss is calculated based on the difference between the depth image of the training room wall boundary and the depth image of the second control synthetic image.

[0016] According to the discrimination loss, the scene image condition control consistency loss and the room wall boundary condition control consistency loss, the three-dimensional scene generator and the two-dimensional image discriminator are updated to obtain an updated generator and an updated discriminator.

[0017] Determine whether the iteration termination condition is reached.

[0018] If so, the iteration is stopped and the updated generator is used as a controllable generation model of the indoor 3D scene.

[0019] If not, continue iterating, using the updated generator as the 3D scene generator for the next iteration, using the updated discriminator as the 2D image discriminator for the next iteration, and return to the step of "obtaining control conditions for training and multiple indoor scene images without labeled information".

[0020] In a second aspect, the present application provides an application method for a controllable generation model of an indoor three-dimensional scene, and the application method for a controllable generation model of an indoor three-dimensional scene includes the following steps.

[0021] Obtain a control condition input by a user; the control condition is a scene image or a room wall boundary.

[0022] Taking control conditions and noise satisfying a preset distribution as input, a controllable generative model of an indoor three-dimensional scene is used to generate a neural radiation field of the indoor three-dimensional scene; the controllable generative model of the indoor three-dimensional scene is trained using the above-mentioned controllable generative model training method for the indoor three-dimensional scene.

[0023] Generate indoor 3D scenes based on neural radiation fields.

[0024] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned indoor three-dimensional scene controllable generation model training method or the above-mentioned indoor three-dimensional scene controllable generation model application method.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned indoor three-dimensional scene controllable generation model training method or the above-mentioned indoor three-dimensional scene controllable generation model application method.

[0026] According to the specific embodiments provided in this application, this application has the following technical effects.

[0027] The present application provides a method, device and medium for training and application of a controllable generative model for indoor three-dimensional scenes. The training control conditions and noise satisfying the preset distribution are used as inputs, and a first training neural radiation field for indoor three-dimensional scenes is generated by a three-dimensional scene generator. The first control synthetic image, the second control synthetic image and the first training synthetic image are rendered based on the first training neural radiation field. The discrimination loss, the scene image condition control consistency loss and the room wall boundary condition control consistency loss are further calculated to update the three-dimensional scene generator and the two-dimensional image discriminator to obtain an updated generator and an updated discriminator, and the iteration is continued until the iteration termination condition is reached to obtain a controllable generative model for indoor three-dimensional scenes that supports multiple condition controls. The present application designs an adversarial generative network framework, and can train a controllable generative model for indoor three-dimensional scenes that supports multiple condition controls based on a large amount of indoor scene-related image data collected directly from the Internet without any annotation information, and further generates indoor three-dimensional scenes. The requirements for image data sets are not high, and the problem of affecting model accuracy due to the small content of the image data set is avoided. Highly realistic indoor three-dimensional scenes can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A schematic diagram of the generation process of the existing HouseCrafter provided in Example 1 of the present application.

[0030] Figure 2 Schematic diagram of a generative adversarial network framework for controllable generation of room-level indoor three-dimensional scenes provided in Example 1 of the present application.

[0031] Figure 3 A schematic diagram of the controllable generation of a residential-grade indoor three-dimensional scene provided in Example 2 of the present application.

[0032] Figure 4 A schematic diagram of the structure of a computer device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] Example 1.

[0035] The existing method of learning the information of the three-dimensional world from a large amount of two-dimensional image data from different indoor scenes and generating indoor three-dimensional scenes can be: HouseCrafter, which can generate residential-level indoor three-dimensional scenes. The generation process of HouseCrafter is as follows Figure 1As shown, it is necessary to provide the floor plan of all rooms and the layout of the furniture in the rooms as input. According to the given scene layout (i.e., the floor plan of all rooms and the layout of the furniture in the rooms), firstly, a set of camera view sequences are sampled in the scene, and then the color image and depth image under each camera view are generated one by one by autoregression. Each time the color image and depth image under a new camera view are generated, the color image and depth image under the generated adjacent (i.e., neighboring) camera view are used as a reference. Finally, the color images and depth images under all the generated camera views are fused and reconstructed into a grid-represented indoor three-dimensional scene through the Truncated Signed Distance Function (TSDF) technology.

[0036] The technical core of HouseCrafter is to generate color images and depth images from a new camera perspective using the scene layout map and the color images and depth images generated from the neighboring camera perspectives as guidance. To this end, HouseCrafter designed a diffusion model guided by the scene layout map and reference images (i.e., the color images and depth images generated from the neighboring camera perspectives), and used the cross-attention mechanism to achieve conditional control of the scene layout map and reference images, supporting the simultaneous generation of color images and depth images. Therefore, HouseCrafter needs to train the diffusion model on approximately 2,000 scenes in the manually designed and modeled indoor scene dataset 3D-FRONT. Specifically, the scene layout map is obtained based on the three-dimensional grids of furniture objects and house walls in the scene. The camera perspectives are sampled in each scene and a total of 2 million images are rendered for training the diffusion model.

[0037] HouseCrafter has certain disadvantages: (1) This method has strict requirements on training data. It needs to know the scene layout diagrams of all residential-level indoor 3D scenes in the training data, as well as the color image and depth image sequences under dense camera view sequences in the scene. It is difficult to obtain large-scale indoor 3D scene data that meets the conditions, which will affect the generation quality and generalization ability of the diffusion model; (2) This method uses autoregression to generate color images and depth images under different camera viewpoints. It cannot guarantee the consistency of the content in color images and depth images under different camera viewpoints (for example, the appearance and shape of the same object in adjacent images are inconsistent), resulting in poor quality of the fused and reconstructed indoor 3D scene (for example, Figure 1The reconstructed 3D scene represented by the generated grid is shown on the right side of the figure. A relatively blurry and unclean 3D grid can be observed); (3) The generation efficiency of this method is low. It is necessary to autoregressively generate color images and depth images from each camera perspective, and then fuse and reconstruct the color images and depth images from all camera perspectives to obtain the final indoor 3D scene. The computational overhead and time consumption are both high.

[0038] Obviously, the above method has high requirements for the image dataset. It needs to know the shooting angle or its distribution of the images in the image dataset, or requires multi-view image data of indoor scenes. It cannot be applied to massive Internet image data without any annotation information, and the generation quality and efficiency of the above method are low.

[0039] In order to solve the above problems, this embodiment designs a method for generating indoor three-dimensional scenes based on neural radiation fields. It does not require indoor three-dimensional scene data for training. It can directly learn the information of indoor three-dimensional scenes from a large amount of indoor scene-related image data collected on the Internet, and does not require any additional annotation information. It can generate a large number of diverse room-level indoor three-dimensional scenes, so that the requirements for training data are low, and the generation quality and generalization ability of the model will not be affected by the difficulty in obtaining large-scale indoor three-dimensional scene data that meets the conditions. It can generate highly realistic indoor three-dimensional scenes. At the same time, by generating a neural radiation field, the method can directly generate indoor three-dimensional scenes later, without the need to generate images from multiple camera perspectives one by one and then perform fusion reconstruction, which can ensure the consistency of the content in color images and depth images from different camera perspectives, and the computational overhead and time consumption are relatively small.

[0040] This embodiment designs a highly realistic room-level indoor three-dimensional scene generation solution based on neural radiation fields driven by an Internet image set. In order to use Internet images (such as photos) of indoor scenes as training data to generate indoor three-dimensional scenes, this embodiment adopts a three-dimensionally perceived adversarial generative network framework based on neural radiation fields. The adversarial generative network framework includes a three-dimensional scene generator based on neural radiation fields, a differentiable voxel renderer, and a two-dimensional image discriminator. For the neural radiation field of the indoor three-dimensional scene generated by the three-dimensional scene generator, it is equivalent to using the neural radiation field to represent the indoor three-dimensional scene. The voxel renderer is used to render the generated neural radiation field into a two-dimensional synthetic image from a specified camera perspective, and then the two-dimensional image discriminator is used to learn the distribution of indoor scenes from the training images (i.e., Internet images) by using a discriminant loss function, and provide an optimized gradient direction for the rendered two-dimensional synthetic image. The voxel renderer is used to realize the back propagation of the gradient, and the optimized gradient is back propagated back to the three-dimensional scene generator, thereby realizing the use of Internet images as training data to update the three-dimensional scene generator based on the neural radiation field.

[0041] In order to ensure that the two-dimensional image discriminator can correctly learn the distribution of indoor scenes in the training images, it is necessary to ensure that the distribution of the camera perspective used when rendering the synthetic image from the generated neural radiation field is consistent with the distribution of the camera perspective of the training image, so as to avoid the three-dimensional scene generator learning a biased indoor three-dimensional scene. However, since the images from the Internet do not have any additional annotation information and the camera perspective is unknown, in order to apply the adversarial generative network framework of three-dimensional perception based on neural radiation fields to Internet images, this embodiment adds an online trained camera parameter predictor, which is trained on the generated neural radiation field and applied to the training images to predict the camera parameters of the training images, thereby guiding the sampling and rendering of the camera perspective in the generated neural radiation field, so that the three-dimensional scene generator can learn the correct indoor three-dimensional scene.

[0042] The training phase and generation phase of the entire adversarial generative network framework are as follows Figure 2 As shown in the figure, the adversarial generative network framework contains three trainable modules: a 3D scene generator, a 2D image discriminator, and a camera parameter predictor. The adversarial generative network framework also contains two camera samplers: a real camera sampler and a random camera sampler. The adversarial generative network framework also contains a voxel renderer.

[0043] In each iteration of the training phase, the 3D scene generator inputs the latent code (i.e., Gaussian noise) randomly sampled from a specified distribution (such as a Gaussian distribution) and outputs a neural radiation field of a room-level 3D scene; then, the generated neural radiation field is rendered using a voxel renderer based on the camera parameters sampled by the real camera sampler to obtain a synthetic image; the synthetic image is input to the 2D image discriminator, which updates the 3D scene generator and the 2D image discriminator through a discriminant loss function by forcing the distribution of the generated synthetic image to be closer to the distribution of the training image; after the training is completed, a trained 3D scene generator can be obtained, and the distribution of the synthetic image rendered in the neural radiation field generated by the trained 3D scene generator is close to the distribution of the training image, that is, the synthetic image rendered in the generated neural radiation field is realistic enough.

[0044] Among them, the three-dimensional scene generator can adopt the generator of StyleGAN2, including multiple layers of convolutional layers connected in sequence.

[0045] The indoor three-dimensional scene is represented by a neural radiation field. The neural radiation field used in this embodiment adopts a three-plane design, including three orthogonal feature planes and a multi-layer perceptron. The input of the neural radiation field is the coordinate position in the three-dimensional space. Bilinear interpolation is performed in the three orthogonal feature planes according to the coordinate position to obtain three features of the coordinate position. The three features are added together and the added features are input into the multi-layer perceptron, which finally outputs the density value and color value of the coordinate position in the neural radiation field.

[0046] The purpose of the real camera sampler is to avoid sampling invalid camera parameters during training. For example, it is unreasonable to sample camera parameters inside furniture, so the camera parameters at locations where there are no furniture objects (i.e., locations with lower density values) should be selected. The input of the real camera sampler is the neural radiance field and Set camera parameters from training images , one training image corresponds to a set of camera parameters, For the The camera parameters of the training images, represent , which selects a set of camera parameters from multiple sets of camera parameters, that is, outputs a set of camera parameters suitable for the indoor three-dimensional scene. Specifically, first calculate The density value of the group camera parameters at the location in the neural radiation field , and then use the softmin function to convert the density value Normalized to the sampling probability of the camera , according to the probability exist A set of camera parameters is sampled from the set of camera parameters for subsequent rendering of synthetic images in the neural radiance field. This sampling strategy reduces the possibility of sampling invalid camera parameters while retaining the randomness required for camera parameter sampling. The camera parameters include the camera's field of view, position, and rotation angle.

[0047] For the generated neural radiation field, a voxel renderer can be used to render a synthetic image from a specified camera perspective. The synthetic image includes a color image and a depth image.

[0048] The camera parameters of the training image are predicted by a camera parameter predictor, which includes multiple convolutional layers connected in sequence. The input of the camera parameter predictor is the training image, and the output is the camera parameters corresponding to the training image. In order to train the camera parameter predictor, this embodiment generates the input image and the corresponding camera parameter true value required for training from the generated neural radiation field. Specifically, a random camera sampler randomly samples multiple groups of camera parameters, combines the neural radiation field, and renders to obtain a synthetic image rendered under each group of camera parameters. The synthetic image and its corresponding camera parameters are used as training data for the camera parameter predictor, and the prediction loss function is used to calculate the loss to update the camera parameter predictor.

[0049] The prediction loss function is as follows.

[0050] .

[0051] in, To predict losses; Camera parameters predicted by the camera parameter predictor Middle The value of an element; is the actual camera parameter Middle The value of an element.

[0052] The two-dimensional image discriminator can adopt the discriminator of StyleGAN2, including multiple convolutional layers connected in sequence.

[0053] The discriminant loss function is the loss function used by the existing generative adversarial network, which will not be repeated here. The process of updating the three-dimensional scene generator and the two-dimensional image discriminator is also the update method used by the existing generative adversarial network, which will not be repeated here.

[0054] In the generation stage, the 3D scene generator can generate different indoor 3D scenes according to given different implicit codes. Thanks to the neural radiation field expression of the indoor 3D scene, the indoor 3D scene represented by a grid can be reconstructed from the neural radiation field, and the color image and depth image of the indoor 3D scene at any perspective can be obtained through the voxel renderer.

[0055] The above method can only realize the generation of indoor three-dimensional scenes, but cannot realize the controllable generation of indoor three-dimensional scenes according to user needs. In order to achieve the purpose of controllable generation, this embodiment designs a controllable generation method of indoor three-dimensional scenes based on neural radiation fields, introduces control conditions input by users, supports users to provide control conditions such as scene images (such as photos), scene text descriptions or room wall boundaries, and can controllably generate a large number of diverse room-level indoor three-dimensional scenes.

[0056] This embodiment designs a controllable generation scheme for indoor three-dimensional scenes. In order to meet the user's customized control requirements for indoor three-dimensional scene generation, this embodiment supports the generation of corresponding room-level indoor three-dimensional scenes by taking user-given scene images, scene text descriptions, or room wall boundaries and other control conditions as input. In order to enable the three-dimensional scene generator to support different types of control conditions as input and generate indoor three-dimensional scenes that meet the control conditions, the following introduces different control conditions from the perspective of signal injection of the control conditions and the constraint loss function corresponding to the control conditions.

[0057] When the user gives a scene image as a control condition, a scene image encoder (which includes multiple sequentially connected convolutional layers) is used to extract a two-dimensional feature map of the scene image, a camera parameter predictor is used to estimate the camera parameters of the scene image, and the three-dimensional space is discretized into a voxelized three-dimensional space according to the resolution of the three planes of the neural radiation field. For each voxel block in the voxelized three-dimensional space, it is projected to the two-dimensional space of the two-dimensional feature map according to the estimated camera parameters, and indexed to the pixel closest to it in the two-dimensional feature map according to the nearest neighbor interpolation principle, and the feature of the pixel position in the two-dimensional feature map is assigned to the voxel block to obtain a three-dimensional feature map. In order to facilitate the injection of scene image features (i.e., three-dimensional feature maps) into the neural radiation field expressed in three planes, the three-dimensional feature map is pooled into feature maps of three orthogonal planes along three orthogonal coordinate axes using a maximum pooling operation, and the feature maps of the three orthogonal planes are injected into the neural radiation field by signal modulation. Specifically, the feature maps of the three orthogonal planes are used as the input of each convolutional layer of the three-dimensional scene generator, which are first multiplied with the original input of the convolutional layer and then input into the convolutional layer. In order to promote the 3D scene generator to generate scenes that are consistent with the input scene image, an image consistency loss function is added during the update process of the 3D scene generator, requiring that the generated indoor 3D scene from the 2D image rendered under the corresponding camera parameters is consistent with the input scene image in the perception domain, so as to constrain the generated indoor 3D scene to meet the input scene image conditions. In order to make the network training more robust and stable, it is only required that the two images are consistent in the perception domain without the need for the two images to be consistent pixel by pixel in the two-dimensional image space. The specific approach is to require that the image features obtained after the two images pass through the same fixed pre-trained image feature extraction network (such as VGG) are consistent. The specific calculation process is: use the pre-trained image feature extraction network to extract features from the given scene image and the synthetic image obtained by rendering the neural radiation field under the camera parameters of the given scene image, and obtain the first feature map And the second feature map , The height, width and number of channels of the feature map are calculated respectively. And the second feature map The distance between them is used to calculate the image consistency loss function. .

[0058] When the user gives a scene text description as a control condition, the scene text description is first converted into a scene image using the existing text-to-image generation network, and then the above-mentioned scene image control method can be used.

[0059] When the user gives the room wall boundary, first generate a three-dimensional room based on the room wall boundary, set the pixel value of the pixel inside the three-dimensional room to 1, and set the pixel value of the pixel outside the three-dimensional room to 0, and obtain the three-dimensional room mask. Then project the three-dimensional room mask to three orthogonal planes along three orthogonal coordinate axis directions to obtain the room wall boundary mask of the three orthogonal planes. Then use the room wall boundary encoder (which includes multiple sequentially connected convolutional layers) to extract the feature maps of the three orthogonal planes respectively, and inject the feature maps of the three orthogonal planes into the neural radiation field by signal modulation. Specifically, the feature maps of the three orthogonal planes are used as the input of each convolutional layer of the three-dimensional scene generator, which are multiplied with the original input of the convolutional layer and then input into the convolutional layer. In order to make the generated indoor 3D scene meet the given room wall boundary, a boundary consistency loss function is added in the update process of the 3D scene generator. The voxel renderer is used to render the depth image of the indoor 3D scene. The depth value on each pixel is required to be equal to the depth value of the room wall boundary. In this way, the generated indoor 3D scene is constrained to meet the input room wall boundary conditions. The specific calculation process is as follows: first, a 3D room wall is generated based on the room wall boundary, and the depth image of the room wall boundary is calculated under random camera parameters. , and then calculate the depth image of the synthetic image obtained by rendering the neural radiance field under random camera parameters , are the height and width of the depth image respectively, and the distance between the depth image of the room wall boundary and the depth image of the synthetic image is calculated, that is, the boundary consistency loss function is calculated. .

[0060] After the control conditions are introduced, in each iteration of the training phase, the camera parameter predictor is first used to determine the first control camera parameters corresponding to the training scene image and the first training camera parameters corresponding to each training image. The input of the 3D scene generator is the latent code (i.e., Gaussian noise) randomly sampled from a specified distribution (such as a Gaussian distribution) and the training control conditions (including the training scene image and the training room wall boundary), and the output is the first training neural radiation field of the room-level 3D scene; the voxel renderer is used to render the generated first training neural radiation field according to the first control camera parameters to obtain a first control synthetic image, the first training neural radiation field is rendered according to the randomly set second control camera parameters to obtain a second control synthetic image, the first training neural radiation field is rendered according to the camera parameters sampled from all training camera parameters by the real camera sampler to obtain a first training synthetic image, and the synthetic image contains both a color image and a depth image; the training image and the first training synthetic image are input to the two-dimensional Image discriminator, the two-dimensional image discriminator updates the three-dimensional scene generator and the two-dimensional image discriminator by forcing the distribution of the generated first training synthetic image to be closer to the distribution of the training image, and calculating the discriminant loss through the discriminant loss function; according to the similarity between the training scene image and the first control synthetic image, the scene image condition control consistency loss is calculated to update the three-dimensional scene generator; based on the training room wall boundary and the second control camera parameters, the depth image of the training room wall boundary is calculated; according to the difference between the depth image of the training room wall boundary and the depth image of the second control synthetic image, the room wall boundary condition control consistency loss is calculated to update the three-dimensional scene generator; until the training is completed, a trained three-dimensional scene generator can be obtained, and the distribution of the synthetic image rendered in the neural radiation field generated by the trained three-dimensional scene generator is close to the distribution of the training image, that is, the synthetic image rendered in the generated neural radiation field is realistic enough and meets the given room wall boundary and scene image control conditions.

[0061] At this time, the input of the 3D scene generator is the hidden code and the given room wall boundary and scene image control conditions, and the output is the three feature planes of the neural radiation field of the indoor 3D scene. The hidden code and control conditions are modulated into each convolution layer, that is, the feature map of the three orthogonal planes of the control conditions is used as the input of each convolution layer of the 3D scene generator, which is first multiplied with the original input of the convolution layer, and then input into the convolution layer, and finally the three feature planes of the neural radiation field of the indoor 3D scene under the given room wall boundary and scene image control conditions are output.

[0062] In the generation stage, the 3D scene generator can generate different indoor 3D scenes according to different given latent codes and user input control conditions. Thanks to the neural radiation field expression of the indoor 3D scene, the mesh representation of the indoor 3D scene can be reconstructed from the neural radiation field, and the color image and depth image of the indoor 3D scene from any camera perspective can be obtained through the voxel renderer.

[0063] This embodiment provides a method for training a controllable generation model of indoor three-dimensional scenes based on neural radiation fields. According to the room wall boundaries, scene images or scene text descriptions given by the user as control conditions, a large number of diverse room-level indoor three-dimensional scenes can be controllably generated. The method for training a controllable generation model of indoor three-dimensional scenes includes the following steps.

[0064] S101, obtaining training control conditions and a plurality of indoor scene images without labeled information, and using the indoor scene images without labeled information as training images; the training control conditions include training scene images and training room wall boundaries.

[0065] The indoor scene image without labeling information may be an indoor scene image obtained from the Internet, which does not have any labeling information.

[0066] S102 , taking the training scene image and the plurality of training images as inputs respectively, and using a camera parameter predictor to determine a first control camera parameter corresponding to the training scene image and a first training camera parameter corresponding to each training image.

[0067] Before using a training scene image and multiple training images as inputs respectively and using a camera parameter predictor to determine a first control camera parameter corresponding to the training scene image and a first training camera parameter corresponding to each training image, the indoor three-dimensional scene controllable generation model training method also includes: using a training control condition and noise satisfying a preset distribution as inputs, using a current iteration of the three-dimensional scene generator to generate a second training neural radiation field for the indoor three-dimensional scene; using a random camera sampler to randomly sample to obtain multiple second training camera parameters; for each second training camera parameter, rendering a second training synthetic image based on the second training neural radiation field and the second training camera parameter; using the second training synthetic image as input and the second training camera parameter as a label, training the camera parameter predictor of the previous iteration to obtain a camera parameter predictor of the current iteration.

[0068] S103, using the training control conditions and the noise satisfying the preset distribution as input, using the three-dimensional scene generator to generate a first training neural radiation field of the indoor three-dimensional scene.

[0069] Among them, taking the training control conditions and the noise satisfying the preset distribution as input, a three-dimensional scene generator is used to generate the first training neural radiation field of the indoor three-dimensional scene, which specifically includes: taking the training scene image as input, and using the scene image encoder to extract and obtain a two-dimensional feature map; processing the two-dimensional feature map based on the first control camera parameter to obtain a three-dimensional feature map; performing maximum pooling on the three-dimensional feature map along three orthogonal coordinate axis directions to obtain scene image condition control feature maps of three orthogonal planes; generating a three-dimensional room based on the wall boundary of the training room, setting the pixel value of the pixel point inside the three-dimensional room to 1, and setting the pixel value of the pixel point outside the three-dimensional room to 0, to obtain a three-dimensional room mask; projecting the three-dimensional room mask onto three orthogonal planes along three orthogonal coordinate axis directions to obtain three The invention discloses a method for generating a neural radiation field for indoor three-dimensional scenes by using a three-dimensional scene generator. The method comprises the following steps: first, a neural radiation field for indoor three-dimensional scenes is generated by using a room wall boundary mask of three orthogonal planes; second, a room wall boundary encoder is used to perform feature extraction on the room wall boundary mask of the three orthogonal planes respectively to obtain the room wall boundary condition control feature maps of the three orthogonal planes; third, a noise that satisfies a preset distribution, a scene image condition control feature map of the three orthogonal planes and a room wall boundary condition control feature map of the three orthogonal planes are used as inputs, and a three-dimensional scene generator is used to generate a first training neural radiation field for indoor three-dimensional scenes, and specifically, the noise and the two feature maps are modulated into each convolution layer of the three-dimensional scene generator, that is, the two feature maps are used as the input of each convolution layer of the three-dimensional scene generator, and they are first multiplied with the original input of the convolution layer, and then input into the convolution layer, and finally the first training neural radiation field for indoor three-dimensional scenes under training control conditions is output.

[0070] S104, based on the first training neural radiation field and the first control camera parameters, render a first control synthetic image; based on the first training neural radiation field and the second control camera parameters, render a second control synthetic image; based on the first training neural radiation field and all the first training camera parameters, render a first training synthetic image; the first control synthetic image, the second control synthetic image and the first training synthetic image all include a color image and a depth image.

[0071] Among them, based on the first training neural radiation field and all the first training camera parameters, rendering to obtain a first training synthetic image specifically includes: taking the first training neural radiation field and all the first training camera parameters as input, and using a real camera sampler to determine appropriate first training camera parameters; rendering the first training neural radiation field based on the appropriate first training camera parameters to obtain a density value and a color value of each first visible position point in the indoor three-dimensional scene, where the first visible position point is a position point in the indoor three-dimensional scene that is visible under appropriate first training camera parameters; taking the density value and color value of each first visible position point in the indoor three-dimensional scene as input, and using a voxel renderer to render to obtain the first training synthetic image.

[0072] S105 , based on the training room wall boundary and the second control camera parameters, calculate and obtain a depth image of the training room wall boundary.

[0073] Among them, based on the training room wall boundary and the second control camera parameters, a depth image of the training room wall boundary is calculated, specifically including: generating a three-dimensional room wall based on the training room wall boundary, calculating the depth of the camera from each second visible position point of the three-dimensional room wall under the second control camera parameters, and forming a depth image of the training room wall boundary, wherein the second visible position point is a position point in the three-dimensional room wall that is visible under the second control camera parameters.

[0074] S106, using a two-dimensional image discriminator to discriminate the training image and the first training synthetic image to obtain a discrimination result, and calculating a discrimination loss according to the discrimination result.

[0075] S107, calculating the scene image condition control consistency loss according to the similarity between the training scene image and the first control synthetic image.

[0076] Among them, according to the similarity between the training scene image and the first control synthetic image, the scene image conditional control consistency loss is calculated, specifically including: using a pre-trained image feature extraction network to extract features from the training scene image and the first control synthetic image respectively, to obtain a first feature map corresponding to the training scene image and a second feature map corresponding to the first control synthetic image; using the first feature map and the second feature map as input, and using the image consistency loss function to calculate the scene image conditional control consistency loss.

[0077] Among them, the image consistency loss function is as follows.

[0078] .

[0079] in, Controlling consistency loss for scene image conditions; is the number of pixels in the height direction of the first feature map and the second feature map, =1, 2, ..., ; is the number of pixels in the width direction of the first feature map and the second feature map, =1, 2, ..., ; is the first feature map at pixel point ( )’s pixel value; is the second feature map at pixel point ( )’s pixel value.

[0080] S108, calculating the control consistency loss of the room wall boundary condition according to the difference between the depth image of the training room wall boundary and the depth image of the second control synthetic image.

[0081] Among them, according to the difference between the depth image of the training room wall boundary and the depth image of the second control synthetic image, the room wall boundary condition control consistency loss is calculated, specifically including: taking the depth image of the training room wall boundary and the depth image of the second control synthetic image as input, and using the boundary consistency loss function to calculate the room wall boundary condition control consistency loss.

[0082] Among them, the boundary consistency loss function is as follows.

[0083] .

[0084] in, Control consistency loss for room wall boundary conditions; is the number of pixels in the depth image in the height direction, =1, 2, ..., ; is the number of pixels in the width direction of the depth image, =1, 2, ..., ; The depth image of the training room wall boundary at pixel point ( )’s pixel value; The depth image of the second control synthesized image is at pixel point ( )’s pixel value.

[0085] S109, updating the three-dimensional scene generator and the two-dimensional image discriminator according to the discrimination loss, the scene image condition control consistency loss and the room wall boundary condition control consistency loss to obtain an updated generator and an updated discriminator.

[0086] S110, determining whether an iteration termination condition is met.

[0087] The iteration termination condition may be reaching the maximum number of iterations.

[0088] S111, if yes, stop the iteration and use the updated generator as the controllable generation model of the indoor three-dimensional scene.

[0089] S112, if not, continue iterating, using the updated generator as the 3D scene generator of the next iteration, using the updated discriminator as the 2D image discriminator of the next iteration, and returning to the step of "obtaining control conditions for training and multiple indoor scene images without labeled information".

[0090] This embodiment further provides an application method of a controllable generation model of an indoor three-dimensional scene, and the application method of a controllable generation model of an indoor three-dimensional scene includes the following steps.

[0091] S201, obtaining a control condition input by a user; the control condition is a scene image or a room wall boundary.

[0092] S202, taking the control condition and the noise satisfying the preset distribution as input, using the controllable generative model of the indoor three-dimensional scene to generate the neural radiation field of the indoor three-dimensional scene; the controllable generative model of the indoor three-dimensional scene is trained using the above-mentioned controllable generative model training method for the indoor three-dimensional scene.

[0093] S203, generating an indoor three-dimensional scene based on the neural radiation field.

[0094] In this embodiment, the control condition also includes a text description of the scene. At this time, the control condition and noise that meets the preset distribution are used as input, and a controllable generation model of the indoor three-dimensional scene is used to generate a neural radiation field of the indoor three-dimensional scene. Specifically, the text description of the scene is converted into a transformed scene image using a text-to-image generation network; the transformed scene image and noise that meets the preset distribution are used as input, and a controllable generation model of the indoor three-dimensional scene is used to generate a neural radiation field of the indoor three-dimensional scene.

[0095] After generating the neural radiation field of the indoor three-dimensional scene, the application method of the controllable generation model of the indoor three-dimensional scene in this embodiment also includes: rendering a synthetic image of the indoor three-dimensional scene under the preset camera parameters based on the neural radiation field and the preset camera parameters.

[0096] Example 2.

[0097] Based on the room-level indoor 3D scene generation method with controllable room wall boundaries, this embodiment can realize the generation of residential-level indoor 3D scenes. In this embodiment, an independent room-level 3D scene generator is first trained for different room types to generate indoor 3D scenes of different room types. When the user gives a residential floor plan as a control condition, the corresponding room-level 3D scene generator is used to generate room-level indoor 3D scenes that meet the room wall boundaries according to the room types and room wall boundaries of each room in the residential floor plan, and together they form the final residential-level indoor 3D scene. Figure 3 shown.

[0098] The problem to be solved in this application is mainly how to learn and understand the geometric structure, appearance texture and spatial layout of indoor three-dimensional scenes from indoor scene image data from the Internet without any additional annotations, and automatically, efficiently and controllably generate a large number of rich and diverse high-realistic room-level or residential-level indoor three-dimensional scenes. In order to solve this problem, the content of this application is the controllable generation of high-realistic room-level indoor three-dimensional scenes based on neural radiation fields driven by Internet image sets. Specifically, a controllable adversarial generation network framework with three-dimensional perception that can be trained on indoor scene image data from the Internet without any additional annotations is constructed. It can learn and understand the geometric structure, appearance texture and spatial layout of indoor three-dimensional scenes, support users to provide control conditions such as scene images, scene text descriptions, room wall boundaries or residential floor plans as input, and automatically, efficiently and controllably generate a large number of rich and diverse high-realistic room-level or residential-level indoor three-dimensional scenes that meet given control conditions.

[0099] This application has been verified on the indoor scene image dataset LSUN collected from the Internet. It can learn and understand the geometric structure, appearance texture and spatial layout of indoor three-dimensional scenes from millions of indoor scene image data from the Internet without any additional annotations, and automatically, efficiently and controllably generate a large number of rich and diverse high-realistic room-level or residential-level indoor three-dimensional scenes. Thanks to the fact that this application directly learns and trains on a large amount of unlimited indoor scene image data from the Internet, the requirements for training data are extremely low, and the generated indoor three-dimensional scenes are more consistent with the real world in terms of geometric structure, appearance texture and spatial layout, which is convenient for use in industries such as virtual reality and augmented reality, interior design, games and embodied intelligence, and improves the effect and generalization ability of research fields such as data-driven indoor scene understanding and analysis.

[0100] Example 3.

[0101] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above-mentioned indoor three-dimensional scene controllable generation model training method or the above-mentioned indoor three-dimensional scene controllable generation model application method is implemented.

[0102] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the above-mentioned indoor three-dimensional scene controllable generation model training method or the above-mentioned indoor three-dimensional scene controllable generation model application method is implemented.

[0104] Example 4.

[0105] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the above-mentioned indoor three-dimensional scene controllable generation model training method or the above-mentioned indoor three-dimensional scene controllable generation model application method.

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for training a controllable generative model for an indoor three-dimensional scene, characterized in that: The indoor three-dimensional scene controllable generation model training method comprises: Acquire training control conditions and a plurality of indoor scene images without labeled information, and use the indoor scene images without labeled information as training images; the training control conditions include training scene images and training room wall boundaries; Taking the training scene image and the plurality of training images as input respectively, using a camera parameter predictor to determine a first control camera parameter corresponding to the training scene image and a first training camera parameter corresponding to each training image; Using a training control condition and noise satisfying a preset distribution as input, a first training neural radiation field of an indoor three-dimensional scene is generated by a three-dimensional scene generator; Based on the first training neural radiation field and the first control camera parameters, a first control synthetic image is rendered; based on the first training neural radiation field and the second control camera parameters, a second control synthetic image is rendered; based on the first training neural radiation field and all the first training camera parameters, a first training synthetic image is rendered; the first control synthetic image, the second control synthetic image and the first training synthetic image all include a color image and a depth image; Based on the training room wall boundary and the second control camera parameters, a depth image of the training room wall boundary is calculated; Using a two-dimensional image discriminator to discriminate the training image and the first training synthetic image to obtain a discrimination result, and calculating a discrimination loss according to the discrimination result; Calculating the scene image conditional control consistency loss according to the similarity between the training scene image and the first control synthetic image; The room wall boundary condition control consistency loss is calculated based on the difference between the depth image of the training room wall boundary and the depth image of the second control synthetic image; According to the discrimination loss, the scene image condition control consistency loss and the room wall boundary condition control consistency loss, the three-dimensional scene generator and the two-dimensional image discriminator are updated to obtain an updated generator and an updated discriminator; Determine whether the iteration termination condition is reached; If yes, stop the iteration and use the updated generator as the controllable generation model of the indoor 3D scene; If not, continue iterating, using the updated generator as the 3D scene generator of the next iteration, using the updated discriminator as the 2D image discriminator of the next iteration, and return to the step of "obtaining control conditions for training and multiple indoor scene images without labeled information".

2. The indoor three-dimensional scene controllable generation model training method according to claim 1 is characterized in that: Before using the training scene image and the plurality of training images as inputs and using a camera parameter predictor to determine the first control camera parameter corresponding to the training scene image and the first training camera parameter corresponding to each training image, the indoor three-dimensional scene controllable generation model training method further includes: Using the training control condition and the noise satisfying the preset distribution as input, the current iterative 3D scene generator is used to generate a second training neural radiation field of the indoor 3D scene; Using a random camera sampler to randomly sample and obtain a plurality of second training camera parameters; For each second training camera parameter, rendering a second training synthetic image based on the second training neural radiation field and the second training camera parameter; The second training synthetic image is used as input and the second training camera parameters are used as labels to train the camera parameter predictor of the previous iteration to obtain the camera parameter predictor of the current iteration.

3. The indoor three-dimensional scene controllable generation model training method according to claim 1 is characterized in that: Taking the training control condition and the noise satisfying the preset distribution as input, a first training neural radiation field of an indoor three-dimensional scene is generated by using a three-dimensional scene generator, specifically including: Taking the training scene image as input, a two-dimensional feature map is extracted using a scene image encoder; Processing the two-dimensional feature map based on the first control camera parameter to obtain a three-dimensional feature map; The three-dimensional feature map is subjected to maximum pooling along three orthogonal coordinate axis directions to obtain the scene image conditional control feature maps of three orthogonal planes; Generate a three-dimensional room based on the wall boundary of the training room, set the pixel value of the pixel point inside the three-dimensional room to 1, and set the pixel value of the pixel point outside the three-dimensional room to 0, to obtain a three-dimensional room mask; Project the three-dimensional room mask onto three orthogonal planes along three orthogonal coordinate axis directions respectively to obtain the room wall boundary mask of the three orthogonal planes; The room wall boundary encoder is used to extract features of the room wall boundary masks of three orthogonal planes respectively, and the room wall boundary condition control feature maps of the three orthogonal planes are obtained; A first training neural radiation field for an indoor three-dimensional scene is generated using a three-dimensional scene generator, with noise that satisfies a preset distribution, scene image condition control feature maps of three orthogonal planes, and room wall boundary condition control feature maps of three orthogonal planes as input.

4. The indoor three-dimensional scene controllable generation model training method according to claim 1 is characterized in that: Based on the first training neural radiation field and all the first training camera parameters, rendering to obtain a first training synthetic image specifically includes: Taking the first training neural radiation field and all the first training camera parameters as input, determining appropriate first training camera parameters using a real camera sampler; Rendering the first training neural radiation field based on the appropriate first training camera parameters to obtain a density value and a color value of each first visible position point in the indoor three-dimensional scene; the first visible position point is a position point in the indoor three-dimensional scene that is visible under the appropriate first training camera parameters; The density value and color value of each first visible position point in the indoor three-dimensional scene are used as input, and a voxel renderer is used to render to obtain a first training synthetic image.

5. The indoor three-dimensional scene controllable generation model training method according to claim 1 is characterized in that: Based on the training room wall boundary and the second control camera parameters, a depth image of the training room wall boundary is calculated, specifically including: A three-dimensional room wall is generated based on the training room wall boundary, and the depth of the camera from each second visible position point on the three-dimensional room wall under the second control camera parameters is calculated to form a depth image of the training room wall boundary; wherein the second visible position point is a position point on the three-dimensional room wall that is visible under the second control camera parameters.

6. The indoor three-dimensional scene controllable generation model training method according to claim 1 is characterized in that: According to the similarity between the training scene image and the first control synthetic image, the scene image condition control consistency loss is calculated, specifically including: Using a pre-trained image feature extraction network, feature extraction is performed on the training scene image and the first control synthetic image to obtain a first feature map corresponding to the training scene image and a second feature map corresponding to the first control synthetic image; Taking the first feature map and the second feature map as input, the image consistency loss function is used to calculate the scene image condition control consistency loss; Among them, the image consistency loss function is: ； in, Controlling consistency loss for scene image conditions; is the number of pixels in the height direction of the first feature map and the second feature map, =1, 2, ..., ; is the number of pixels in the width direction of the first feature map and the second feature map, =1, 2, ..., ; is the first feature map at pixel point ( )’s pixel value; is the second feature map at pixel point ( )’s pixel value; According to the difference between the depth image of the training room wall boundary and the depth image of the second control synthetic image, the room wall boundary condition control consistency loss is calculated, specifically including: Taking the depth image of the training room wall boundary and the depth image of the second control synthetic image as input, the boundary consistency loss function is used to calculate the control consistency loss of the room wall boundary condition; Among them, the boundary consistency loss function is: ； in, Control consistency loss for room wall boundary conditions; is the number of pixels in the depth image in the height direction, =1, 2, ..., ; is the number of pixels in the width direction of the depth image, =1, 2, ..., ; The depth image of the training room wall boundary at pixel point ( )’s pixel value; The depth image of the second control synthesized image is at pixel point ( )’s pixel value.

7. An application method for a controllable generation model of an indoor three-dimensional scene, characterized in that: The indoor three-dimensional scene controllable generation model application method comprises: Obtaining a control condition input by a user; the control condition is a scene image or a room wall boundary; Taking control conditions and noise satisfying a preset distribution as input, a controllable generative model of an indoor three-dimensional scene is used to generate a neural radiation field of the indoor three-dimensional scene; the controllable generative model of the indoor three-dimensional scene is trained using the controllable generative model training method for indoor three-dimensional scenes described in any one of claims 1 to 6; Generate indoor 3D scenes based on neural radiation fields.

8. The indoor three-dimensional scene controllable generation model application method according to claim 7, characterized in that: The control condition also includes a scene text description. At this time, the control condition and the noise that meets the preset distribution are used as input, and the neural radiation field of the indoor three-dimensional scene is generated by using the controllable generation model of the indoor three-dimensional scene, which specifically includes: Use a text-to-image generation network to convert scene text descriptions into transformed scene images; Taking the transformed scene image and noise satisfying the preset distribution as input, the neural radiation field of the indoor 3D scene is generated using the controllable generation model of the indoor 3D scene.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the indoor three-dimensional scene controllable generation model training method described in any one of claims 1 to 6 or the indoor three-dimensional scene controllable generation model application method described in any one of claims 7 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a controllable generation model of an indoor three-dimensional scene described in any one of claims 1-6 or the method for applying a controllable generation model of an indoor three-dimensional scene described in any one of claims 7-8 is implemented.

Images