Three-dimensional indoor scene generation method and related equipment

Abstract

The invention discloses a three-dimensional indoor scene generation method and related equipment. The method comprises the steps that text description of an indoor scene is acquired; generating an indoor layout by the text description through a preset diffusion model; converting the indoor layout into a two-dimensional height field and a two-dimensional semantic graph; predicting a neural radiation field based on the two-dimensional height field and the two-dimensional semantic map; a plurality of viewpoints are sampled in the predicted neural radiation field, the plurality of viewpoints are rendered to obtain an RGB-D image corresponding to each viewpoint, RGB is color information of the two-dimensional image, and D is depth information of the two-dimensional image; and according to the RGB-D image corresponding to each viewpoint, constructing a truncated signed distance field body, and generating a mesh triangular patch based on the truncated signed distance field body to obtain an indoor three-dimensional scene. According to the method and the device, the high-quality three-dimensional indoor scene can be generated without real multi-view images, the efficiency of generating the three-dimensional scene is improved, and the method and the device can be widely applied to the technical field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a three-dimensional indoor scene generation method and related equipment. BACKGROUND

[0002] In the fields of embodied intelligence, virtual reality and interior design, the generation of high-fidelity and structured three-dimensional indoor scenes is a key foundation for building efficient simulation environments and conducting algorithm training and evaluation.

[0003] However, in the related art, traditional indoor three-dimensional scene modeling mainly includes three-dimensional reconstruction based on real multi-view images and computer-aided design three-dimensional scene modeling. The three-dimensional reconstruction based on multi-view images acquires real multi-view images of a scene from different angles, and performs three-dimensional reconstruction according to the real multi-view images. It highly depends on the real multi-view image dataset obtained from the real scene in advance as prior information of the three-dimensional scene, which affects the efficiency of three-dimensional scene modeling. The computer-aided design three-dimensional scene modeling requires professional modelers to spend a lot of time to design geometry and texture details, and the complexity of the scene also affects the modeling efficiency. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a three-dimensional indoor scene generation method and related equipment, which can generate a high-quality three-dimensional indoor scene without real multi-view images, and improve the efficiency of generating a three-dimensional scene.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application proposes a three-dimensional indoor scene generation method, comprising: obtaining a text description of an indoor scene; generating an indoor layout by a preset diffusion model from the text description; converting the indoor layout into a two-dimensional height field and a two-dimensional semantic map; predicting a neural radiance field based on the two-dimensional height field and the two-dimensional semantic map to obtain a predicted neural radiance field; arbitrarily sampling a plurality of viewpoints in the predicted neural radiance field, rendering the plurality of viewpoints to obtain an RGB-D image corresponding to each viewpoint, wherein RGB represents color information of a two-dimensional image, and D represents depth information of a two-dimensional image; constructing a truncated signed distance field body according to the RGB-D image corresponding to each viewpoint, and generating a grid triangular facet based on the truncated signed distance field body to obtain an indoor three-dimensional scene.

[0006] In some embodiments, the generating an indoor layout by a preset diffusion model from the text description comprises: encoding the text description to obtain a text embedding vector; Obtain the preset initial scene layout; Gaussian noise is gradually added to the initial scene layout using a preset diffusion model until an initial scene code conforming to a Gaussian distribution is obtained. Based on the text embedding vector and the initial scene code, the initial scene code is iteratively denoised using a preset diffusion model to obtain the denoised initial scene code; The initial scene encoding after denoising is decoded to obtain the interior layout, wherein the interior layout includes the bounding box corresponding to each object and the semantic label, center position, size and rotation angle of the bounding box.

[0007] In some embodiments, converting the interior layout into a two-dimensional height field and a two-dimensional semantic map includes: Based on the center position and size of the bounding box of each object, determine the projection area of ​​the bounding box of each object in the two-dimensional plane; Based on the dimensions of the bounding box of each object, determine the maximum and minimum height of the bounding box of each object in the projection area; A two-dimensional height field is obtained based on the maximum and minimum heights; The semantic label of the bounding box of each object is assigned to the corresponding projection area to obtain a two-dimensional semantic map.

[0008] In some embodiments, the step of predicting the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map to obtain the predicted neural radiation field includes: The interior layout is divided into multiple local areas, and the local two-dimensional height field and local two-dimensional semantic map corresponding to each local area are determined. Based on each of the local two-dimensional height fields and local two-dimensional semantic maps, local features of each local region are generated; Based on the local features, the neural radiation field is predicted to obtain the predicted neural radiation field.

[0009] In some embodiments, predicting the neural radiation field based on the local features to obtain the predicted neural radiation field includes: Obtain the position of any three-dimensional sampling point in the indoor layout; Determine the local features corresponding to the positions of the three-dimensional sampling points; The positions of the three-dimensional sampling points and the local features are used to calculate a hash index for querying the color features and volume density features of the three-dimensional sampling points using a hash function, wherein the expression for the hash index is: ; In the formula, represents a hash index, represents a position of a three-dimensional sampling point, represents an exclusive or operation of a bit operation, represents a prime number, represents a hash table capacity, represents a three-dimensional space coordinate index, represents a modulo operation; extracting, according to the hash index, color features and volume density features corresponding to the three-dimensional sampling points from the local features, to obtain a predicted neural radiance field.

[0010] In some embodiments, the rendering of a plurality of viewpoints in the predicted neural radiance field to obtain a corresponding RGB-D image of each viewpoint includes: obtaining camera parameters of each viewpoint, and determining camera rays corresponding to each viewpoint according to the camera parameters; integrating and calculating the predicted neural radiance field along the camera rays corresponding to each viewpoint to obtain an RGB image and a depth map corresponding to each viewpoint; stitching the RGB image and the depth map to obtain an RGB-D image.

[0011] In some embodiments, the generating of a triangular mesh based on the truncated signed distance field volume includes: frame-by-frame fusion of the RGB-D images rendered by the plurality of viewpoints into a three-dimensional voxel space to construct a truncated signed distance field volume, wherein the truncated signed distance field volume includes each voxel cube and a truncated signed distance field corresponding to each voxel cube; traversing each voxel cube in the truncated signed distance field volume, and determining edges of each object surface passing through each voxel cube according to the signed distance field corresponding to each voxel cube; calculating intersection points between the object surface and the edges by linear interpolation, and generating triangular mesh triangular patches according to the intersection points.

[0012] To achieve the above object, another aspect of the embodiment of the present application proposes a three-dimensional indoor scene generation device, which comprises: a text description acquisition module configured to acquire a text description of an indoor scene; an indoor layout generation module configured to generate an indoor layout by a preset diffusion model based on the text description; a layout conversion module configured to convert the indoor layout into a two-dimensional height field and a two-dimensional semantic map; a neural radiance field prediction module configured to predict a neural radiance field based on the two-dimensional height field and the two-dimensional semantic map to obtain a predicted neural radiance field. rendering a plurality of viewpoints in the predicted neural radiance field to obtain a corresponding RGB-D image of each viewpoint, wherein RGB represents color information of a two-dimensional image, and D represents depth information of the two-dimensional image; a triangular facet generation module configured to construct a truncated signed distance field volume according to the corresponding RGB-D image of each viewpoint, and generate a grid triangular facet based on the truncated signed distance field volume to obtain the indoor three-dimensional scene.

[0013] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0014] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0015] The embodiments of the present application also disclose a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method described above.

[0016] The embodiments of the present application at least have the following beneficial effects: the present application provides a three-dimensional indoor scene generation method, which firstly obtains a text description of an indoor scene; generates an indoor layout through a preset diffusion model; converts the indoor layout into a two-dimensional height field and a two-dimensional semantic map; predicts a neural radiance field based on the two-dimensional height field and the two-dimensional semantic map to obtain a predicted neural radiance field; renders a plurality of viewpoints in the predicted neural radiance field to obtain a corresponding RGB-D image of each viewpoint, wherein RGB represents color information of a two-dimensional image, and D represents depth information of the two-dimensional image; constructs a truncated signed distance field volume according to the corresponding RGB-D image of each viewpoint, and generates a grid triangular facet based on the truncated signed distance field volume to obtain the indoor three-dimensional scene. The present application generates an indoor layout from a text description through a diffusion model, realizes automatic indoor layout generation, and reduces manual intervention. In the absence of real multi-view images, the present application generates a neural radiance field according to a two-dimensional height field and a two-dimensional semantic map of the indoor layout, and finally generates a high-quality three-dimensional scene consistent with the structure of the text description based on the neural radiance field, thereby improving the efficiency of generating a three-dimensional scene. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application; Figure 2 is a flowchart of a three-dimensional indoor scene generation method provided by an embodiment of the present application; Figure 3 is a flowchart of generating an indoor layout provided by an embodiment of the present application; Figure 4 is a flowchart of predicting a neural radiance field provided by an embodiment of the present application; Figure 5 is a flowchart of generating a grid triangle based on a truncated signed distance field volume provided by an embodiment of the present application; Figure 6 is a structural schematic diagram of a three-dimensional indoor scene generation apparatus provided by an embodiment of the present application; Figure 7 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0019] It can be understood that the terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] It should be understood that in the present application, "at least one" refers to one or more, "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

[0022] In the related art, traditional indoor three-dimensional scene modeling mainly includes three-dimensional reconstruction based on real multi-view images and computer-aided design three-dimensional scene modeling. The three-dimensional reconstruction based on multi-view images acquires real multi-view images of a scene under different angles, and performs three-dimensional reconstruction according to the real multi-view images, which highly depends on the real multi-view image dataset obtained from the real scene in advance as prior information of the three-dimensional scene, affecting the efficiency of three-dimensional scene modeling; the computer-aided design three-dimensional scene modeling needs professional modelers to spend a lot of time to realize the design of geometry and texture details, and the complexity of the scene also affects the modeling efficiency.

[0023] Therefore, the embodiment of the present application provides a three-dimensional indoor scene generation method and related equipment. The method first acquires a text description of an indoor scene; generates an indoor layout through a preset diffusion model; converts the indoor layout into a two-dimensional height field and a two-dimensional semantic map; predicts a neural radiance field based on the two-dimensional height field and the two-dimensional semantic map to obtain a predicted neural radiance field; randomly samples multiple viewpoints in the predicted neural radiance field, renders the multiple viewpoints, and obtains an RGB-D image corresponding to each viewpoint, wherein RGB represents color information of a two-dimensional image, and D represents depth information of the two-dimensional image; constructs a truncated signed distance field body according to the RGB-D image corresponding to each viewpoint, generates a grid triangular facet based on the truncated signed distance field body, and obtains an indoor three-dimensional scene. The present application generates an indoor layout from a text description through a diffusion model, realizes automatic indoor layout generation, and reduces manual intervention. In the absence of real multi-view images, the present application generates a neural radiance field based on a two-dimensional height field and a two-dimensional semantic map of an indoor layout, finally generates a high-quality three-dimensional scene consistent with the structure of the text description based on the neural radiance field, and improves the efficiency of generating a three-dimensional scene.

[0024] The three-dimensional indoor scene generation method provided by the embodiment of the present application relates to the technical field of computer vision. The three-dimensional indoor scene generation method provided by the embodiment of the present application can be applied to a terminal, can be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto. The server end can be configured as a standalone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network. The software can be an application that implements the three-dimensional indoor scene generation method, and the like, but is not limited to the above forms.

[0025] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0026] As shown in Figure 1 FIG. 1 is a schematic diagram of an implementation environment according to an embodiment of the application. Referring to Figure 1 , the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected by wireless or wired means to complete data transmission and exchange.

[0027] The server 101 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0028] In addition, the server 101 can also be a node server in a blockchain network. The blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, and other computer technologies.

[0029] The terminal 102 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal 102 can also be a vehicle-mounted terminal of various device types as exemplified above, but is not limited thereto. The terminal 102 and the server 101 can be directly or indirectly connected by wired or wireless communication, and the embodiments of the application are not limited herein.

[0030] Exemplarily, based on Figure 1The implementation environment shown in this application embodiment provides a method for generating a three-dimensional indoor scene. The following description uses the application of this three-dimensional indoor scene generation method in server 101 as an example. It can be understood that this method can also be applied in terminal 102.

[0031] Reference Figure 2 , Figure 2 This is a flowchart illustrating a method for generating a 3D indoor scene on a server, provided as an embodiment of this application. The execution subject of this method can be any of the aforementioned computer devices (including a server or a terminal). (Refer to...) Figure 2 The method may include the following steps: S100: Obtain a text description of the indoor scene.

[0032] In this application, the text description is used to characterize the layout requirements of the target indoor scene, including semantic information such as the functional area division of the indoor space, object types, and object placement. For example, the text description may be: "A bedroom with an area of ​​20㎡, including a double bed, a bedside table, a door, a window and white walls", etc.

[0033] S200. The text description is used to generate an indoor layout through a preset diffusion model.

[0034] For example, Figure 3 This is a flowchart of generating an interior layout provided in an embodiment of this application, such as... Figure 3 As shown, the steps for generating the interior layout include S210-S250: S210. Encode the text description to obtain a text embedding vector; For example, embodiments of this application can use pre-trained models in the field of natural language processing (such as BERT, CLIP, etc.) to encode text descriptions to obtain corresponding text embedding vectors, and convert text descriptions into vector forms that can be recognized by diffusion models through text embedding.

[0035] S220. Obtain the preset initial scene layout; The preset initial scene layout is a clean initial scene layout.

[0036] S230. Gaussian noise is gradually added to the initial scene layout through a preset diffusion model until an initial scene code conforming to a Gaussian distribution is obtained. The preset diffusion model is used to learn the layout distribution of the 3D scene. The preset diffusion model first performs a forward diffusion process, which is defined as a process that takes a discrete time step. Markov chains on the grid are laid out from a clean initial scene. At the beginning, a predefined linearly increasing noise plan is followed , Gaussian noise is gradually added to obtain the noisy initial scene encoding at time step ; based on the clean initial scene layout , the noisy initial scene encoding at time step The expression of the forward diffusion process is as follows:

[0037] Wherein: , .

[0038] In the formula, I represents the unit matrix; represents a preset linearly increasing noise array, which controls the noise intensity of the t-th step; represents the probability of the initial scene encoding under the condition of the clean initial scene layout ; represents a normal distribution; represents the signal retention coefficient at the t-th time step; represents the cumulative signal retention coefficient, that is, the product of the signal retention coefficients from the first time step to the t-th time step.

[0039] S240, based on the text embedding vector and the initial scene encoding, iteratively denoising the initial scene encoding through a preset diffusion model to obtain a denoised initial scene encoding; Exemplarily, the embodiment of the present application iteratively denoises the noisy initial scene encoding through a noise estimator to recover the clean initial scene encoding, that is, the denoised initial scene encoding. The noise estimator is designed with multiple self-attention layers and cross-attention layers to realize scene encoding denoising under the condition of text. The noise estimator takes the initial scene encoding at any time step t in the diffusion process, the text embedding vector , and the time step t as input, and finds the real Gaussian noise added to the initial scene layout by minimizing the denoising objective function, which is as follows:

[0040] Wherein, is the real Gaussian noise added to the initial scene layout, is the noise predicted by the noise estimator, represents the noise ​​​, t, and mathematical expectation.

[0041] S250, decoding the denoised initial scene coding to obtain an indoor layout, wherein the indoor layout includes a bounding box corresponding to each object and a semantic label, a center position, a size and a rotation angle of the bounding box.

[0042] The obtained clean scene layout is encoded into a specific bounding box parameter set M, and the expression is as follows:

[0043] wherein N is the total number of bounding boxes, and D is the attribute dimension of the bounding box; the bounding box , represents a semantic label, i.e., a category, represents a center position, represents a size, represents a rotation angle.

[0044] In addition, the embodiments of the present application can further finely control the indoor layout by adding, deleting or modifying the bounding boxes in the indoor scene layout when generating the indoor layout.

[0045] S300, converting the indoor layout into a two-dimensional height field and a two-dimensional semantic map.

[0046] Specifically, the indoor layout is converted into a two-dimensional height field and a two-dimensional semantic map in a bird's eye view, wherein the two-dimensional height field (wherein, and respectively represent the height and width of the height field) includes the maximum height and the minimum height of the bounding box in the corresponding projection area, and the two-dimensional height field has the expression as follows:

[0047] wherein the position is located in the bounding box , and are respectively the maximum height and the minimum height of the bounding box at the position in the projection area; The two-dimensional semantic map includes the category of the bounding box, i.e., the semantic label, and for the two-dimensional semantic map , the expression is as follows:

[0048] wherein the position is located in the bounding box , bounding box of the semantic label.

[0049] Exemplarily, the step of converting the indoor layout into the two-dimensional height field and the two-dimensional semantic map comprises: determining a projection area of the bounding box of each object in a two-dimensional plane according to the center position and the size of the bounding box of each object; determining the maximum height and the minimum height of the bounding box of each object in the projection area according to the size of the bounding box of each object; obtaining the two-dimensional height field according to the maximum height and the minimum height; and assigning the semantic label of the bounding box of each object to the corresponding projection area to obtain the two-dimensional semantic map.

[0050] S400, predicting a neural radiance field based on the two-dimensional height field and the two-dimensional semantic map to obtain a predicted neural radiance field.

[0051] Specifically, predicting the neural radiance field means predicting a neural radiance field consistent with the layout geometry and semantics according to the generated two-dimensional height field and two-dimensional semantic map; the neural radiance field can be understood as a mapping relationship from a three-dimensional sampling point to a color feature and a volume density feature, which is a continuous volume scene representation and can predict the color and volume density of any three-dimensional space point. To realize accurate prediction of the neural radiance field, the embodiment of the present application adopts a generative neural hash table method combined with layout features to realize efficient and high-quality prediction. Exemplarily, Figure 4 is a flowchart for predicting a neural radiance field provided by the embodiment of the present application, as Figure 4 shown, the steps of predicting the neural radiance field comprise S410-S430: S410, dividing the indoor layout into a plurality of local regions and determining the corresponding local two-dimensional height field and local two-dimensional semantic map of each local region; In order to improve the processing efficiency of large-scale scenes and the local adaptability of feature representation, the embodiment of the present application divides the indoor layout into a plurality of local regions with a resolution of , wherein, respectively represent the number of divisions in the height, width and depth directions; further, for each local region after division , according to its spatial range, the corresponding part is intercepted from the global two-dimensional height field and two-dimensional semantic map to obtain the local two-dimensional height field and the local two-dimensional semantic map corresponding to the local region.

[0052] S420, generating local features of each local region according to each of the local two-dimensional height field and the local two-dimensional semantic map; For example, in this embodiment, an encoder is used to perform feature encoding on the local two-dimensional height field and local two-dimensional semantic map corresponding to each local region, generating local features corresponding to that local region. These local features are the core features required for the neural radiation field, used for subsequent extraction of color and volume density information of the three-dimensional sampling points. The expression for generating the local features corresponding to the local region is:

[0053] in, For local features, This is the encoder. Local features are included. It is a two-dimensional matrix, where each row is a... 3D eigenvectors It is the total capacity of the hash table. It is the dimension of each feature vector, and satisfies That is, the first three components are used to represent color, and the last component is used to represent volume density.

[0054] In some specific embodiments, the encoder It can be a convolutional neural network (CNN) or a Transformer. The encoder in this embodiment is... It is achieved by using the objective function The training results include L1 loss function and perceptual loss function. and adversarial generative loss function objective function The expression is:

[0055] in, Represents a real image. From The semantic map in the perspective view generated by cumulatively sampling semantic labels along each ray. The weights for L1 loss, To perceive the weights of the loss function, Weights to counteract the generation loss function.

[0056] S430. Based on the local features, the neural radiation field is predicted to obtain the predicted neural radiation field.

[0057] For example, based on local features, the neural radiation field is predicted, and the predicted neural radiation field includes S431-S434: S431. Obtain the position of any three-dimensional sampling point in the indoor layout.

[0058] S432. Determine the local features corresponding to the positions of the three-dimensional sampling points.

[0059] S433, calculate the hash index of querying the color feature and the volume density feature of the three-dimensional sampling point by a hash function through the position of the three-dimensional sampling point and the local feature, wherein the expression of the hash index is: In the formula, denotes the hash index, denotes the position of the three-dimensional sampling point, denotes the exclusive or operation of the bit operation, denotes a prime number, denotes the capacity of the hash table, denotes the three-dimensional space coordinate index, denotes the modulo operation; Through the calculated hash index, the color feature and the volume density feature corresponding to the three-dimensional sampling point can be quickly queried from the local feature, the prediction efficiency of the neural radiance field is improved, and efficient neural radiance field representation of a large-scale indoor scene is realized.

[0060] S434, extract the color feature and the volume density feature corresponding to the three-dimensional sampling point from the local feature according to the hash index, and obtain the predicted neural radiance field.

[0061] In the formula, the color feature includes color information, and the volume density feature includes volume density information.

[0062] The embodiment of the application can efficiently and accurately extract the color information and the volume density information corresponding to the three-dimensional sampling point from the layout feature by the generative neural hash table method, lays a foundation for subsequent new view image rendering, and does not need to rely on real multi-view images as priori.

[0063] S500, sample a plurality of viewpoints in the predicted neural radiance field, render the plurality of viewpoints, and obtain an RGB-D image corresponding to each viewpoint, wherein RGB represents color information of a two-dimensional image, and D represents depth information of a two-dimensional image.

[0064] In the predicted neural radiance field, a plurality of viewpoints are sampled, the viewpoint distribution is in the grid points of the three-dimensional voxel, and the azimuth direction is 0°-360°. Exemplarily, the step of rendering the plurality of viewpoints to obtain the RGB-D image corresponding to each viewpoint includes S510-S530: S510, obtain the camera parameter of each viewpoint, and determine the camera ray corresponding to each viewpoint according to the camera parameter; Specifically, for each viewpoint that needs to be rendered, first determine the camera parameter thereof, including the projection center of the camera in the world coordinate system ​(i.e., camera position). Based on the perspective camera model, for each pixel in the final generated 2D image, a line can be defined from the projection center. The camera ray that originates from and passes through that pixel The expression for this ray is:

[0065] in, It refers to the direction of the ray; therefore, each pixel in the image corresponds to a camera ray. It is a real number greater than 0, any The value of is obtained They are all points on the ray.

[0066] S520. Integrate the predicted neural radiation field along the camera ray corresponding to each viewpoint to obtain the RGB image and depth map corresponding to each viewpoint. For example, the embodiments of this application employ a voxel rendering method to perform integral calculations on the predicted neural radiation field along the camera ray corresponding to each pixel in order to synthesize a two-dimensional image (RGB image) from any viewpoint. For each camera ray, its pixel value The calculation is achieved through integration, as shown in the following formula:

[0067] in, and These represent color information and volume density information, respectively. Indicates cumulative transparency. Indicates in The features obtained from the location, Indicates from Semantic tags extracted from [the source].

[0068] By performing the above integral calculation sequentially on all camera rays, the RGB image corresponding to each viewpoint can be obtained.

[0069] For each camera ray Its depth value Calculated using the following formula:

[0070] By calculating the depth value of the corresponding camera ray for each pixel, a depth map for each viewpoint can be obtained.

[0071] S530. The RGB image and the depth map are stitched together to obtain an RGB-D image.

[0072] S600, constructing a truncated signed distance field volume according to the RGB-D images corresponding to each viewpoint, and generating mesh triangles based on the truncated signed distance field volume to obtain an indoor three-dimensional scene.

[0073] Exemplarily, Figure 5 is a flowchart for generating mesh triangles based on a truncated signed distance field volume provided by the embodiments of the present application, as shown in Figure 5 The step of generating mesh triangles based on a truncated signed distance field volume includes S610-S630, as shown in S610, fusing the RGB-D images rendered by the multiple viewpoints to a three-dimensional voxel space frame by frame to construct a truncated signed distance field volume, wherein the truncated signed distance field volume includes each voxel cube and the truncated signed distance field corresponding to each voxel cube; The truncated signed distance field (TSDF) is used to represent the signed distance from any sampling point in a three-dimensional space to the surface of a target object. The distance is usually truncated within a certain range. The role of the truncated signed distance field is to quantify the relative position of the space point and the object surface. When the sampling point is outside the object surface, the TSDF value is positive; when the sampling point is inside the object surface, the TSDF value is negative; when the sampling point is on the object surface, the TSDF value is 0.

[0074] The truncated signed distance field volume (TSDF volume) is a three-dimensional voxel grid composed of a plurality of voxel cubes. For each voxel cube in the TSDF volume, a corresponding truncated signed distance field value (TSDF value) is stored.

[0075] Exemplarily, the embodiments of the present application fuse the multi-view RGB-D images into a three-dimensional voxel space frame by frame through the open source library Open3D. Specifically, according to the color information and depth information in the RGB-D image, the three-dimensional space coordinates corresponding to each pixel are calculated, and they are mapped to the corresponding voxel cube in the three-dimensional voxel space. Based on the fused multi-view space point information, the truncated signed distance field (TSDF) of each voxel cube is calculated, and finally the truncated signed distance field volume (TSDF volume) containing all voxel cubes and their corresponding TSDF values is constructed.

[0076] S620, traversing each voxel cube in the truncated signed distance field volume, and determining the edges of each object surface passing through each voxel cube according to the signed distance field corresponding to each voxel cube. Specifically, in this embodiment, the constructed TSDF volume is traversed by voxel cubes; for each voxel cube, based on the sign change characteristics of its TSDF value, it is determined whether the object surface crosses the edge of the voxel cube. If the sign of the TSDF value corresponding to different edges of the voxel cube changes (i.e. from positive to negative or from negative to positive), it is determined that the object surface crosses the edge.

[0077] S630. Calculate the intersection points between each object surface and edge using linear interpolation, and generate a triangular mesh based on the intersection points.

[0078] This embodiment uses the Marching Cubes algorithm to process the edges that the object's surface traverses. The specific steps are as follows: For each voxel cube edge that is traversed by the object's surface, the intersection point between the edge and the object's surface is calculated using linear interpolation; Based on the intersection points calculated within all voxel cubes, a mesh triangular facet representing the object's surface is generated using the Marching Cubes algorithm.

[0079] Furthermore, the embodiments of this application optimize the generated triangular facets, including noise filtering, hole filling, mesh simplification, and coordinate system alignment operations; at the same time, combining the pose and semantic information of the generated interior layout, the triangular facets of each local area are merged to finally obtain a renderable and editable structured interior 3D scene model.

[0080] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0081] In one specific embodiment, the three-dimensional indoor scene generation process of this application includes the following steps: S1. Based on the acquired indoor scene text, generate the indoor layout through a diffusion model; The diffusion model is used to learn the layout distribution of a 3D scene. The diffusion model first performs a forward diffusion process, which is defined as a Markov chain at discrete time step t, starting from a clean initial scene layout. Initially, Gaussian noise is gradually added according to a predefined, linearly increasing noise schedule, resulting in the noise level at time step [missing information]. Noisy initial scene encoding ,based on Noisy initial scene encoding at time step t The expression for the forward diffusion process is as follows:

[0082] Second, the diffusion model noise estimator Encoding the initial scene with noise Iterative denoising is performed to recover the clean initial scene encoding, i.e., the denoised initial scene encoding. The noise estimator takes the initial scene encoding at any time step t in the diffusion process , a text embedding vector , and the time step t as input, and finds the true Gaussian noise added to the initial scene layout by minimizing the denoising objective function, which is

[0083] wherein, is the true Gaussian noise added to the initial scene layout, is the noise predicted by the noise estimator, denotes the mathematical expectation with respect to , t, and .

[0084] In addition, multiple self-attention layers and cross-attention layers are designed in the noise estimator to achieve scene encoding denoising under text conditions.

[0085] Finally, the obtained clean scene layout encoding is decoded into a specific bounding box parameter set M, and its expression is:

[0086] wherein, N is the total number of bounding boxes, and D is the attribute dimension of the bounding box; , denotes the semantic label, i.e., the category, denotes the center position, denotes the size, denotes the rotation angle.

[0087] S2, for generating indoor layout, the indoor scene layout can be further finely controlled by adding, deleting, or modifying the bounding boxes in the indoor scene layout.

[0088] S3, predicting the neural radiance field; First, for the generated indoor layout, it is converted into a two-dimensional height field of bird's eye view (BEV) and a two-dimensional semantic map, wherein the two-dimensional height field includes the bounding boxes of each object in the indoor layout and the maximum and minimum heights of the bounding boxes, and the semantic map includes the categories (i.e., semantic labels) of the bounding boxes; Second, the indoor layout is divided into a plurality of local areas with a resolution of , wherein, ​respectively represent the number of divisions in the height, width and depth directions; and for each local region after division, according to its spatial range, a corresponding part is intercepted from the global two-dimensional height field and the two-dimensional semantic map, respectively obtaining a local two-dimensional height field and a local two-dimensional semantic map corresponding to the local region . ; and through an encoder, the local two-dimensional height field and the local two-dimensional semantic map corresponding to each local region are feature encoded to generate the local feature corresponding to the local region.

[0089] Finally, the position and local feature of the obtained three-dimensional sampling point are calculated through a hash function to obtain the hash index of the color feature and the volume density feature of the query three-dimensional sampling point, wherein the expression of the hash index is: ; In the formula, , represents the hash index, , represents the position of the three-dimensional sampling point, , represents the exclusive or operation of the bit operation, , represents a prime number, , represents the capacity of the hash table, , represents the three-dimensional space coordinate index, , represents the modulo operation; According to the hash index, the color feature and the volume density feature corresponding to the three-dimensional sampling point are extracted from the local feature to obtain the predicted neural radiance field.

[0090] S4, generate a grid triangular facet; First, in the predicted neural radiance field, render the RGB-D image under the arbitrary sampling view points. The view points are distributed in the grid points of the three-dimensional voxel, and the azimuth angle direction is 0°~360°. For each view point that needs to be rendered, first determine its camera parameters, including the projection center O of the camera in the world coordinate system. According to the perspective camera model, for each pixel on the finally generated two-dimensional image, a camera ray passing through the pixel can be defined from the projection center O , the expression of the ray is:

[0091] In the formula, is the ray direction, therefore, each pixel in the image corresponds to a camera ray, is a real number greater than 0, and the value of is arbitrary, and the obtained are all points on the ray.

[0092] Secondly, a voxel rendering method is used to integrate the predicted neural radiation field along the camera ray corresponding to each pixel to synthesize a two-dimensional image from any viewpoint; for each camera ray, its pixel value... The calculation is achieved through integration, as shown in the following formula:

[0093] in, and These represent color information and volume density information, respectively. Indicates cumulative transparency. Indicates in The features obtained from the location, Indicates from Semantic tags extracted from [the source].

[0094] By performing the above integral calculation sequentially on all camera rays, the RGB image corresponding to each viewpoint can be obtained.

[0095] For each camera ray Its depth value Calculated using the following formula:

[0096] The depth map for each viewpoint is obtained by calculating the depth value of the corresponding camera ray for each pixel.

[0097] The RGB image and the depth map are stitched together to obtain an RGB-D image.

[0098] Finally, to extract mesh triangular patches from the predicted neural radiation field, the truncated signed distance field (TSDF) was first obtained from the RGB-D image set using Open3D; then, marching was used from the TSDF. The Cube algorithm calculates the mesh triangular facets. Specifically, it involves: fusing multi-view RGB-D images frame by frame into a 3D voxel space using the open-source library Open3D to construct a TSDF volume; traversing voxel cubes on the constructed TSDF volume; for each voxel cube, determining whether the object surface crosses an edge of that voxel cube based on the sign change characteristics of its TSDF value; if the sign of the TSDF value corresponding to different edges of the voxel cube changes, it is determined that the object surface crosses that edge, and the intersection point between the edge and the object surface is calculated using linear interpolation; based on the intersection points calculated within all voxel cubes, generating mesh triangular facets representing the object surface using the MarchingCubes algorithm; further processing the generated mesh triangular facets through noise filtering, hole filling, simplification, and coordinate system alignment, and merging the local meshes by combining layout pose and semantic information to obtain a renderable and editable indoor 3D scene model.

[0099] To sum up, the embodiment of the application provides a three-dimensional indoor scene generation method and related equipment. The scheme first acquires a text description of an indoor scene; generates an indoor layout through a preset diffusion model; converts the indoor layout into a two-dimensional height field and a two-dimensional semantic map; predicts a neural radiance field based on the two-dimensional height field and the two-dimensional semantic map to obtain a predicted neural radiance field; randomly samples multiple viewpoints in the predicted neural radiance field, renders the multiple viewpoints, and obtains an RGB-D image corresponding to each viewpoint, where RGB represents color information of a two-dimensional image, and D represents depth information of the two-dimensional image; constructs a truncated signed distance field body according to the RGB-D image corresponding to each viewpoint, generates a mesh triangle patch based on the truncated signed distance field body, and obtains an indoor three-dimensional scene. The application generates an indoor layout from a text description through a diffusion model, realizes automatic indoor layout generation, reduces manual intervention, generates a neural radiance field based on a two-dimensional height field and a two-dimensional semantic map of an indoor layout without real multi-view images, finally generates a high-quality three-dimensional scene consistent with the structure of the text description based on the neural radiance field, and improves the efficiency of generating a three-dimensional scene.

[0100] As shown in Figure 6 The embodiment of the application also provides a structural diagram of a three-dimensional indoor scene generation device. The device can implement the method described above. The device can include: A text description acquisition module 21 is configured to acquire a text description of an indoor scene. An indoor layout generation module 22 is configured to generate an indoor layout from the text description through a preset diffusion model. A layout conversion module 23 is configured to convert the indoor layout into a two-dimensional height field and a two-dimensional semantic map. A neural radiance field prediction module 24 is configured to predict a neural radiance field based on the two-dimensional height field and the two-dimensional semantic map to obtain a predicted neural radiance field. A rendering module 25 is configured to randomly sample multiple viewpoints in the predicted neural radiance field, render the multiple viewpoints, and obtain an RGB-D image corresponding to each viewpoint, where RGB represents color information of a two-dimensional image, and D represents depth information of the two-dimensional image. A triangle patch generation module 26 is configured to construct a truncated signed distance field body according to the RGB-D image corresponding to each viewpoint, generate a mesh triangle patch based on the truncated signed distance field body, and obtain an indoor three-dimensional scene.

[0101] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0102] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the three-dimensional indoor scene generation method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0103] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0104] Please refer to Figure 7 , Figure 7 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises: The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application. The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the three-dimensional indoor scene generation method of the embodiments of the present application. The input / output interface 903 is used to realize information input and output. The communication interface 904 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 905 transmits information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device. The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected with each other through a bus 905.

[0105] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the three-dimensional indoor scene generation method.

[0106] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiment, the present storage medium embodiment specifically implements the functions of the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0107] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0108] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0109] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0110] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0111] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0112] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0113] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0114] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0115] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0116] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for generating a three-dimensional indoor scene, characterized in that, include: Obtain a text description of the indoor scene; The text description is used to generate an interior layout through a preset diffusion model; The interior layout is converted into a two-dimensional height field and a two-dimensional semantic map; The neural radiation field is predicted based on the two-dimensional height field and the two-dimensional semantic map. Multiple viewpoints are arbitrarily sampled in the predicted neural radiation field, and the multiple viewpoints are rendered to obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image. Based on the RGB-D image corresponding to each viewpoint, a truncated signed distance field is constructed, and a mesh triangular patch is generated based on the truncated signed distance field to obtain an indoor 3D scene.

2. The method for generating a three-dimensional indoor scene according to claim 1, characterized in that, The step of generating an interior layout from the text description using a preset diffusion model includes: The text description is encoded to obtain a text embedding vector; Obtain the preset initial scene layout; Gaussian noise is gradually added to the initial scene layout using a preset diffusion model until an initial scene code conforming to a Gaussian distribution is obtained. Based on the text embedding vector and the initial scene code, the initial scene code is iteratively denoised using a preset diffusion model to obtain the denoised initial scene code; The initial scene encoding after denoising is decoded to obtain the interior layout, wherein the interior layout includes the bounding box corresponding to each object and the semantic label, center position, size and rotation angle of the bounding box.

3. The method for generating a three-dimensional indoor scene according to claim 2, characterized in that, The process of converting the interior layout into a two-dimensional height field and a two-dimensional semantic map includes: Based on the center position and size of the bounding box of each object, determine the projection area of ​​the bounding box of each object in the two-dimensional plane; Based on the dimensions of the bounding box of each object, determine the maximum and minimum height of the bounding box of each object in the projection area; A two-dimensional height field is obtained based on the maximum and minimum heights; The semantic label of the bounding box of each object is assigned to the corresponding projection area to obtain a two-dimensional semantic map.

4. The method for generating a three-dimensional indoor scene according to claim 1, characterized in that, The prediction of the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map, to obtain the predicted neural radiation field, includes: The interior layout is divided into multiple local areas, and the local two-dimensional height field and local two-dimensional semantic map corresponding to each local area are determined. Based on each of the local two-dimensional height fields and local two-dimensional semantic maps, local features of each local region are generated; Based on the local features, the neural radiation field is predicted to obtain the predicted neural radiation field.

5. The method for generating a three-dimensional indoor scene according to claim 4, characterized in that, The step of predicting the neural radiation field based on the local features to obtain the predicted neural radiation field includes: Obtain the position of any three-dimensional sampling point in the indoor layout; Determine the local features corresponding to the positions of the three-dimensional sampling points; The positions of the three-dimensional sampling points and the local features are used to calculate a hash index for querying the color features and volume density features of the three-dimensional sampling points using a hash function, wherein the expression for the hash index is: ； In the formula, Represents a hash index. Indicates the location of the three-dimensional sampling point. This represents the bitwise XOR operation. Representing prime numbers, Indicates the capacity of the hash table. Represents a three-dimensional spatial coordinate index. This represents the modulo operation; Based on the hash index, the color features and volume density features corresponding to the three-dimensional sampling points are extracted from the local features to obtain the predicted neural radiation field.

6. The method for generating a three-dimensional indoor scene according to claim 1, characterized in that, The step of arbitrarily sampling multiple viewpoints in the predicted neural radiation field, rendering the multiple viewpoints, and obtaining an RGB-D image corresponding to each viewpoint includes: Obtain the camera parameters for each viewpoint, and determine the camera ray corresponding to each viewpoint based on the camera parameters; The predicted neural radiation field is integrated along the camera ray corresponding to each viewpoint to obtain the RGB image and depth map corresponding to each viewpoint; The RGB image and the depth map are stitched together to obtain an RGB-D image.

7. The method for generating a three-dimensional indoor scene according to claim 2, characterized in that, The generation of mesh triangular facets based on the truncated signed distance field volume includes: The RGB-D images rendered from multiple viewpoints are fused frame by frame into a three-dimensional voxel space to construct a truncated signed distance field, wherein the truncated signed distance field includes each voxel cube and the truncated signed distance field corresponding to each voxel cube. In the truncated signed distance field, each voxel cube is traversed, and the edges of each object surface that pass through each voxel cube are determined according to the signed distance field corresponding to each voxel cube. The intersection points between each object surface and edge are calculated using linear interpolation, and a triangular mesh is generated based on these intersection points.

8. A three-dimensional indoor scene generation device, characterized in that, The device includes: The text description acquisition module is used to acquire text descriptions of indoor scenes; The interior layout generation module is used to generate an interior layout from the text description using a preset diffusion model. The layout conversion module is used to convert the indoor layout into a two-dimensional height field and a two-dimensional semantic map; The neural radiation field prediction module is used to predict the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map, so as to obtain the predicted neural radiation field. The rendering module is used to sample multiple viewpoints arbitrarily in the predicted neural radiation field, render the multiple viewpoints, and obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image. The triangular patch generation module is used to construct a truncated signed distance field volume based on the RGB-D image corresponding to each viewpoint, and generate mesh triangular patches based on the truncated signed distance field volume to obtain an indoor 3D scene.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the three-dimensional indoor scene generation method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional indoor scene generation method as described in any one of claims 1 to 7.

Images