A three-dimensional indoor scene generation method and related device

Abstract

The application discloses a three-dimensional indoor scene generation method and related equipment, and the method comprises the steps of obtaining a text description of an indoor scene; generating an indoor layout through a preset diffusion model; converting the indoor layout into a two-dimensional height field and a two-dimensional semantic graph; predicting a neural radiance field based on the two-dimensional height field and the two-dimensional semantic graph; sampling multiple viewpoints in the predicted neural radiance field, rendering the multiple viewpoints, and obtaining an RGB-D image corresponding to each viewpoint, wherein RGB is color information of a two-dimensional image, and D is depth information of the two-dimensional image; constructing a truncated signed distance field body according to the RGB-D image corresponding to each viewpoint, generating a grid triangular facet based on the truncated signed distance field body, and obtaining an indoor three-dimensional scene. The application can generate a high-quality three-dimensional indoor scene without real multi-view images, improve the efficiency of generating a three-dimensional scene, and can be widely applied in the technical field of computer vision.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and related equipment for generating three-dimensional indoor scenes. Background Technology

[0002] In fields such as embodied intelligence, virtual reality, and interior design, the generation of highly realistic and structured 3D interior scenes is a key foundation for building efficient simulation environments and conducting algorithm training and evaluation.

[0003] However, in related technologies, traditional indoor 3D scene modeling mainly includes 3D reconstruction based on real multi-view images and computer-aided design 3D scene modeling. 3D reconstruction based on multi-view images involves acquiring real multi-view images of the scene from different perspectives and reconstructing the 3D scene based on these images. This method heavily relies on obtaining a dataset of real multi-view images from the actual scene as prior information, impacting the efficiency of 3D scene modeling. Computer-aided design 3D scene modeling requires professional modelers to spend a significant amount of time designing geometric and textural details; the complexity of the scene also affects modeling efficiency. Summary of the Invention

[0004] The main objective of this application is to propose a method and related equipment for generating three-dimensional indoor scenes, which can generate high-quality three-dimensional indoor scenes without the need for real multi-view images, thereby improving the efficiency of generating three-dimensional scenes.

[0005] To achieve the above objectives, one aspect of this application proposes a method for generating a three-dimensional indoor scene, including:

[0006] Obtain a text description of the indoor scene;

[0007] The text description is used to generate an interior layout through a preset diffusion model;

[0008] The interior layout is converted into a two-dimensional height field and a two-dimensional semantic map;

[0009] The neural radiation field is predicted based on the two-dimensional height field and the two-dimensional semantic map.

[0010] Multiple viewpoints are arbitrarily sampled in the predicted neural radiation field, and the multiple viewpoints are rendered to obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image.

[0011] Based on the RGB-D image corresponding to each viewpoint, a truncated signed distance field is constructed, and a mesh triangular patch is generated based on the truncated signed distance field to obtain an indoor 3D scene.

[0012] In some embodiments, generating an interior layout from the text description using a preset diffusion model includes:

[0013] The text description is encoded to obtain a text embedding vector;

[0014] Obtain the preset initial scene layout;

[0015] Gaussian noise is gradually added to the initial scene layout using a preset diffusion model until an initial scene code conforming to a Gaussian distribution is obtained.

[0016] Based on the text embedding vector and the initial scene code, the initial scene code is iteratively denoised using a preset diffusion model to obtain the denoised initial scene code;

[0017] The initial scene encoding after denoising is decoded to obtain the interior layout, wherein the interior layout includes the bounding box corresponding to each object and the semantic label, center position, size and rotation angle of the bounding box.

[0018] In some embodiments, converting the interior layout into a two-dimensional height field and a two-dimensional semantic map includes:

[0019] Based on the center position and size of the bounding box of each object, determine the projection area of ​​the bounding box of each object in the two-dimensional plane;

[0020] Based on the dimensions of the bounding box of each object, determine the maximum and minimum height of the bounding box of each object in the projection area;

[0021] A two-dimensional height field is obtained based on the maximum and minimum heights;

[0022] The semantic label of the bounding box of each object is assigned to the corresponding projection area to obtain a two-dimensional semantic map.

[0023] In some embodiments, the step of predicting the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map to obtain the predicted neural radiation field includes:

[0024] The interior layout is divided into multiple local areas, and the local two-dimensional height field and local two-dimensional semantic map corresponding to each local area are determined.

[0025] Based on each of the local two-dimensional height fields and local two-dimensional semantic maps, local features of each local region are generated;

[0026] Based on the local features, the neural radiation field is predicted to obtain the predicted neural radiation field.

[0027] In some embodiments, predicting the neural radiation field based on the local features to obtain the predicted neural radiation field includes:

[0028] Obtain the position of any three-dimensional sampling point in the indoor layout;

[0029] Determine the local features corresponding to the positions of the three-dimensional sampling points;

[0030] The positions of the three-dimensional sampling points and the local features are used to calculate a hash index for querying the color features and volume density features of the three-dimensional sampling points using a hash function, wherein the expression for the hash index is:

[0031] ;

[0032] In the formula, Represents a hash index. Indicates the location of the three-dimensional sampling point. This represents the bitwise XOR operation. Representing prime numbers, Indicates the capacity of the hash table. Represents a three-dimensional spatial coordinate index. This represents the modulo operation;

[0033] Based on the hash index, the color features and volume density features corresponding to the three-dimensional sampling points are extracted from the local features to obtain the predicted neural radiation field.

[0034] In some embodiments, the step of arbitrarily sampling multiple viewpoints in the predicted neural radiation field, rendering the multiple viewpoints to obtain an RGB-D image corresponding to each viewpoint includes:

[0035] Obtain the camera parameters for each viewpoint, and determine the camera ray corresponding to each viewpoint based on the camera parameters;

[0036] The predicted neural radiation field is integrated along the camera ray corresponding to each viewpoint to obtain the RGB image and depth map corresponding to each viewpoint;

[0037] The RGB image and the depth map are stitched together to obtain an RGB-D image.

[0038] In some embodiments, generating mesh triangular patches based on the truncated signed distance field volume includes:

[0039] The RGB-D images rendered from multiple viewpoints are fused frame by frame into a three-dimensional voxel space to construct a truncated signed distance field, wherein the truncated signed distance field includes each voxel cube and the truncated signed distance field corresponding to each voxel cube.

[0040] In the truncated signed distance field, each voxel cube is traversed, and the edges of each object surface that pass through each voxel cube are determined according to the signed distance field corresponding to each voxel cube.

[0041] The intersection points between each object surface and edge are calculated using linear interpolation, and a triangular mesh is generated based on these intersection points.

[0042] To achieve the above objectives, another aspect of this application provides a three-dimensional indoor scene generation apparatus, the apparatus comprising:

[0043] The text description acquisition module is used to acquire text descriptions of indoor scenes;

[0044] The interior layout generation module is used to generate an interior layout from the text description using a preset diffusion model.

[0045] The layout conversion module is used to convert the indoor layout into a two-dimensional height field and a two-dimensional semantic map;

[0046] The neural radiation field prediction module is used to predict the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map, so as to obtain the predicted neural radiation field.

[0047] The rendering module is used to sample multiple viewpoints arbitrarily in the predicted neural radiation field, render the multiple viewpoints, and obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image.

[0048] The triangular patch generation module is used to construct a truncated signed distance field volume based on the RGB-D image corresponding to each viewpoint, and generate mesh triangular patches based on the truncated signed distance field volume to obtain an indoor 3D scene.

[0049] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0050] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0051] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0052] The embodiments of this application include at least the following beneficial effects: This application provides a method for generating three-dimensional indoor scenes. The method first obtains a text description of the indoor scene; then, it generates an indoor layout from the text description using a preset diffusion model; the indoor layout is converted into a two-dimensional height field and a two-dimensional semantic map; based on the two-dimensional height field and the two-dimensional semantic map, a neural radiation field is predicted to obtain the predicted neural radiation field; multiple viewpoints are arbitrarily sampled in the predicted neural radiation field, and the multiple viewpoints are rendered to obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image, and D represents the depth information of the two-dimensional image; based on the RGB-D image corresponding to each viewpoint, a truncated signed distance field is constructed, and a mesh triangular facet is generated based on the truncated signed distance field to obtain the three-dimensional indoor scene. This application generates an indoor layout from the text description using a diffusion model, achieving automated indoor layout generation and reducing manual intervention; and even without real multi-view images, it generates a neural radiation field based on the two-dimensional height field and the two-dimensional semantic map of the indoor layout, and finally generates a high-quality three-dimensional scene consistent with the text description structure based on the neural radiation field, improving the efficiency of generating three-dimensional scenes. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0054] Figure 2 This is a flowchart of a three-dimensional indoor scene generation method provided in an embodiment of this application;

[0055] Figure 3 This is a flowchart of generating an interior layout provided in an embodiment of this application;

[0056] Figure 4 This is a flowchart of the prediction of neural radiation field provided in an embodiment of this application;

[0057] Figure 5 This is a flowchart of generating mesh triangular patches based on truncated signed distance fields, provided in an embodiment of this application.

[0058] Figure 6 This is a schematic diagram of the structure of a three-dimensional indoor scene generation device provided in an embodiment of this application;

[0059] Figure 7This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0061] It is understood that the terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0062] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0064] In related technologies, traditional indoor 3D scene modeling mainly includes 3D reconstruction based on real multi-view images and computer-aided design 3D scene modeling. 3D reconstruction based on multi-view images involves acquiring real multi-view images of the scene from different perspectives and then reconstructing the 3D scene based on these images. This method heavily relies on obtaining a dataset of real multi-view images from the actual scene as prior information, thus affecting the efficiency of 3D scene modeling. Computer-aided design 3D scene modeling requires professional modelers to spend a significant amount of time designing geometric and textural details; the complexity of the scene also impacts modeling efficiency.

[0065] In view of this, this application provides a method and related equipment for generating a 3D indoor scene. The method first obtains a text description of the indoor scene; then, it generates an indoor layout using a preset diffusion model; the indoor layout is converted into a two-dimensional height field and a two-dimensional semantic map; based on the two-dimensional height field and the two-dimensional semantic map, a neural radiation field is predicted to obtain the predicted neural radiation field; multiple viewpoints are arbitrarily sampled in the predicted neural radiation field, and the multiple viewpoints are rendered to obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image; based on the RGB-D image corresponding to each viewpoint, a truncated signed distance field is constructed, and a mesh triangular facet is generated based on the truncated signed distance field to obtain the 3D indoor scene. This application generates an indoor layout from a text description using a diffusion model, achieving automated indoor layout generation and reducing manual intervention; and even without real multi-view images, it generates a neural radiation field based on the two-dimensional height field and the two-dimensional semantic map of the indoor layout, and finally generates a high-quality 3D scene consistent with the text description structure based on the neural radiation field, improving the efficiency of 3D scene generation.

[0066] The three-dimensional indoor scene generation method provided in this application relates to the field of computer vision technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the three-dimensional indoor scene generation method, but is not limited to the above forms.

[0067] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0068] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided in an embodiment of this application. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0069] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0070] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0071] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment does not impose any limitations.

[0072] For example, based on Figure 1The implementation environment shown in this application embodiment provides a method for generating a three-dimensional indoor scene. The following description uses the application of this three-dimensional indoor scene generation method in server 101 as an example. It can be understood that this method can also be applied in terminal 102.

[0073] Reference Figure 2 , Figure 2 This is a flowchart illustrating a method for generating a 3D indoor scene on a server, provided as an embodiment of this application. The execution subject of this method can be any of the aforementioned computer devices (including a server or a terminal). (Refer to...) Figure 2 The method may include the following steps:

[0074] S100: Obtain a text description of the indoor scene.

[0075] In this application, the text description is used to characterize the layout requirements of the target indoor scene, including semantic information such as the functional area division of the indoor space, object types, and object placement. For example, the text description may be: "A bedroom with an area of ​​20㎡, including a double bed, a bedside table, a door, a window and white walls", etc.

[0076] S200. The text description is used to generate an indoor layout through a preset diffusion model.

[0077] For example, Figure 3 This is a flowchart of generating an interior layout provided in an embodiment of this application, such as... Figure 3 As shown, the steps for generating the interior layout include S210-S250:

[0078] S210. Encode the text description to obtain a text embedding vector;

[0079] For example, embodiments of this application can use pre-trained models in the field of natural language processing (such as BERT, CLIP, etc.) to encode text descriptions to obtain corresponding text embedding vectors, and convert text descriptions into vector forms that can be recognized by diffusion models through text embedding.

[0080] S220. Obtain the preset initial scene layout;

[0081] The preset initial scene layout is a clean initial scene layout.

[0082] S230. Gaussian noise is gradually added to the initial scene layout through a preset diffusion model until an initial scene code conforming to a Gaussian distribution is obtained.

[0083] The preset diffusion model is used to learn the layout distribution of the 3D scene. The preset diffusion model first performs a forward diffusion process, which is defined as a process that takes a discrete time step. Markov chains on the grid are laid out from a clean initial scene. Initially, a predefined, linearly increasing noise schedule was implemented. Gaussian noise is gradually added to it to obtain the result at time step Noisy initial scene encoding ; Layout based on a clean initial scene In time step Noisy initial scene encoding The expression for the forward diffusion process is as follows:

[0084]

[0085] in:

[0086] , .

[0087] In the formula, I represents the identity matrix; This represents a preset, linearly increasing noise array, which controls the noise intensity at step t. This indicates the layout in a clean initial scene. Under the condition of initial scene coding The probability of; Represents a normal distribution; This represents the signal retention coefficient at time step t; This represents the cumulative signal retention coefficient, which is the product of the signal retention coefficients from time step 1 to time step t.

[0088] S240. Based on the text embedding vector and the initial scene code, the initial scene code is iteratively denoised using a preset diffusion model to obtain the denoised initial scene code.

[0089] For example, embodiments of this application construct a noise estimator. Encoding the noisy initial scene Iterative denoising is performed to recover a clean initial scene code, i.e., the denoised initial scene code. The noise estimator incorporates multiple self-attention and cross-attention layers to achieve scene code denoising under text conditions. The noise estimator uses the initial scene code at any time step t during the diffusion process. Text embedding vectors Using time step t as input, the goal is to minimize the denoising objective function to find the true Gaussian noise added to the initial scene layout. for:

[0090]

[0091] in, To add realistic Gaussian noise to the initial scene layout, The noise predicted by the noise estimator Indicates about ,t, and The mathematical expectation.

[0092] S250. Decode the denoised initial scene encoding to obtain the interior layout, wherein the interior layout includes the bounding box corresponding to each object and the semantic label, center position, size and rotation angle of the bounding box.

[0093] Encode the resulting clean scene layout Decoding results in a specific set of bounding box parameters M, whose expression is:

[0094]

[0095] Where N is the total number of bounding boxes, and D is the attribute dimension of the bounding boxes; bounding boxes , Represents semantic tags, that is, categories, Indicates the center position, Indicates size, Indicates the rotation angle.

[0096] Furthermore, in the embodiments of this application, for generating interior layouts, the bounding boxes in the interior scene layout can be added, deleted, or modified to further refine the control over the interior layout.

[0097] S300, The indoor layout is converted into a two-dimensional height field and a two-dimensional semantic map.

[0098] Specifically, this is achieved by transforming the internal layout into a two-dimensional height field and a two-dimensional semantic map from a bird's-eye view. The two-dimensional height field... (in, and (representing the height and width of the height field, respectively, including the maximum and minimum height of the bounding box within the corresponding projection area, a two-dimensional height field) The expression is:

[0099]

[0100] Among them, location Located in the bounding box middle, and These are the positions of the bounding box in the projection area. Maximum and minimum height at the location;

[0101] A two-dimensional semantic graph includes the categories of bounding boxes, i.e., semantic labels. Its expression is:

[0102]

[0103] Among them, location Located in the bounding box middle, bounding box Semantic tags.

[0104] For example, the steps of converting an interior layout into a two-dimensional height field and a two-dimensional semantic map include: determining the projection area of ​​the bounding box of each object in a two-dimensional plane based on the center position and size of the bounding box of each object; determining the maximum height and minimum height of the bounding box of each object in the projection area based on the size of the bounding box of each object; obtaining a two-dimensional height field based on the maximum height and minimum height; and assigning the semantic label of the bounding box of each object to the corresponding projection area to obtain a two-dimensional semantic map.

[0105] S400. Based on the two-dimensional height field and the two-dimensional semantic map, the neural radiation field is predicted to obtain the predicted neural radiation field.

[0106] Specifically, predicting the neural radiation field refers to predicting a neural radiation field consistent with the layout geometry and semantics based on the generated two-dimensional height field and two-dimensional semantic map. This neural radiation field can be understood as a mapping relationship from three-dimensional sampling points to color features and volume density features, a continuous volumetric scene representation capable of predicting the color of any three-dimensional point. and bulk density To achieve accurate prediction of neural radiation fields, embodiments of this application employ a generative neural hash table method incorporating layout features to achieve efficient and high-quality prediction. For example, Figure 4 This is a flowchart of the prediction of neural radiation field provided in the embodiments of this application, such as... Figure 4 As shown, the steps for predicting the neural radiation field include S410-S430:

[0107] S410. Divide the indoor layout into multiple local areas, and determine the local two-dimensional height field and local two-dimensional semantic map corresponding to each local area;

[0108] In order to improve the processing efficiency and local adaptability of feature representation for large-scale scenes, this embodiment of the application divides the indoor layout into sections with a resolution of [resolution value missing]. Several local regions, among which, These represent the number of divisions in the height, width, and depth directions, respectively; further, for each of the divided local regions... Based on its spatial extent, the corresponding portion is extracted from the global two-dimensional height field and two-dimensional semantic map to obtain the local two-dimensional height field corresponding to that local region. and local two-dimensional semantic graph .

[0109] S420. Generate local features for each local region based on each of the local two-dimensional height fields and local two-dimensional semantic maps;

[0110] For example, in this embodiment, an encoder is used to perform feature encoding on the local two-dimensional height field and local two-dimensional semantic map corresponding to each local region, generating local features corresponding to that local region. These local features are the core features required for the neural radiation field, used for subsequent extraction of color and volume density information of the three-dimensional sampling points. The expression for generating the local features corresponding to the local region is:

[0111]

[0112] in, For local features, This is the encoder. Local features are included. It is a two-dimensional matrix, where each row is a... 3D eigenvectors It is the total capacity of the hash table. It is the dimension of each feature vector, and satisfies That is, the first three components are used to represent color, and the last component is used to represent volume density.

[0113] In some specific embodiments, the encoder It can be a convolutional neural network (CNN) or a Transformer. The encoder in this embodiment is... It is achieved by using the objective function The training results include L1 loss function and perceptual loss function. and adversarial generative loss function objective function The expression is:

[0114]

[0115] in, Represents a real image. From The semantic map in the perspective view generated by cumulatively sampling semantic labels along each ray. The weights for L1 loss, To perceive the weights of the loss function, Weights to counteract the generation loss function.

[0116] S430. Based on the local features, the neural radiation field is predicted to obtain the predicted neural radiation field.

[0117] For example, based on local features, the neural radiation field is predicted, and the predicted neural radiation field includes S431-S434:

[0118] S431. Obtain the position of any three-dimensional sampling point in the indoor layout.

[0119] S432. Determine the local features corresponding to the positions of the three-dimensional sampling points.

[0120] S433. Using the positions of the three-dimensional sampling points and the local features, calculate the hash index for querying the color features and volume density features of the three-dimensional sampling points through a hash function, wherein the expression for the hash index is:

[0121] ;

[0122] In the formula, Represents a hash index. Indicates the location of the three-dimensional sampling point. This represents the bitwise XOR operation. Representing prime numbers, Indicates the capacity of the hash table. Represents a three-dimensional spatial coordinate index. This represents the modulo operation;

[0123] By using the calculated hash index, the color and volume density features corresponding to the 3D sampling point can be quickly retrieved from the local features, improving the prediction efficiency of the neural radiation field and realizing efficient neural radiation field representation of large-scale indoor scenes.

[0124] S434. Extract the color features and volume density features corresponding to the three-dimensional sampling points from the local features according to the hash index to obtain the predicted neural radiation field.

[0125] Among them, color features include color information, and volume density features include volume density information.

[0126] This application embodiment uses a generative neural hash table method to efficiently and accurately extract color and volume density information corresponding to 3D sampling points from layout features, laying the foundation for subsequent new perspective image rendering, without relying on real multi-view images as priors.

[0127] S500. Randomly sample multiple viewpoints in the predicted neural radiation field, render the multiple viewpoints to obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image.

[0128] In this process, several viewpoints are arbitrarily sampled in the predicted neural radiation field. These viewpoints are distributed among the grid points of a three-dimensional voxel, with azimuth angles ranging from 0° to 360°. For example, the step of rendering multiple viewpoints to obtain the RGB-D image corresponding to each viewpoint includes steps S510-S530:

[0129] S510. Obtain the camera parameters for each viewpoint, and determine the camera ray corresponding to each viewpoint based on the camera parameters;

[0130] Specifically, for each viewpoint that needs to be rendered, its camera parameters are first determined, including the camera's projection center in the world coordinate system. (i.e., camera position). Based on the perspective camera model, for each pixel in the final generated 2D image, a line can be defined from the projection center. The camera ray that originates from and passes through that pixel The expression for this ray is:

[0131]

[0132] in, It refers to the direction of the ray; therefore, each pixel in the image corresponds to a camera ray. It is a real number greater than 0, any The value of is obtained They are all points on the ray.

[0133] S520. Integrate the predicted neural radiation field along the camera ray corresponding to each viewpoint to obtain the RGB image and depth map corresponding to each viewpoint.

[0134] For example, the embodiments of this application employ a voxel rendering method to perform integral calculations on the predicted neural radiation field along the camera ray corresponding to each pixel in order to synthesize a two-dimensional image (RGB image) from any viewpoint.

[0135] For each camera ray, its pixel value The calculation is achieved through integration, as shown in the following formula:

[0136]

[0137] in, and These represent color information and volume density information, respectively. Indicates cumulative transparency. Indicates in The features obtained from the location, Indicates from Semantic tags extracted from [the source].

[0138] By performing the above integral calculation sequentially on all camera rays, the RGB image corresponding to each viewpoint can be obtained.

[0139] For each camera ray Its depth value Calculated using the following formula:

[0140]

[0141] By calculating the depth value of the corresponding camera ray for each pixel, a depth map for each viewpoint can be obtained.

[0142] S530. The RGB image and the depth map are stitched together to obtain an RGB-D image.

[0143] S600. Based on the RGB-D image corresponding to each viewpoint, construct a truncated signed distance field volume, and generate a mesh triangular facet based on the truncated signed distance field volume to obtain an indoor three-dimensional scene.

[0144] For example, Figure 5 This is a flowchart of generating mesh triangular patches based on truncated signed distance fields, as provided in the embodiments of this application. Figure 5 As shown, the steps for generating mesh triangular patches based on truncated signed distance fields include S610-S630:

[0145] S610. The RGB-D images rendered from multiple viewpoints are fused frame by frame into a three-dimensional voxel space to construct a truncated signed distance field, wherein the truncated signed distance field includes each voxel cube and the truncated signed distance field corresponding to each voxel cube.

[0146] The truncated signed distance field (TSDF) is used to characterize the signed distance from any sampling point in three-dimensional space to the surface of a target object. The distance is usually truncated within a certain range. Its function is to quantify the relative position of a spatial point and the object surface. When the sampling point is located outside the object surface, the TSDF value is positive; when the sampling point is located inside the object surface, the TSDF value is negative; and when the sampling point is located on the object surface, the TSDF value is 0.

[0147] A truncated signed distance field volume (TSDF volume) is a three-dimensional voxel mesh composed of several voxel cubes. For each voxel cube in the TSDF volume, a corresponding truncated signed distance field value (TSDF value) is stored.

[0148] For example, in this embodiment of the application, multi-view RGB-D images are fused frame by frame into a three-dimensional voxel space using the open-source library Open3D. Specifically, based on the color and depth information in the RGB-D images, the three-dimensional spatial coordinates corresponding to each pixel are calculated and mapped to the corresponding voxel cube in the three-dimensional voxel space; based on the fused multi-view spatial point information, the truncated signed distance field (TSDF) of each voxel cube is calculated, and finally a truncated signed distance field volume (TSDF volume) containing all voxel cubes and their corresponding TSDF values ​​is constructed.

[0149] S620. Traverse each voxel cube in the truncated signed distance field and determine the edges of each object surface that pass through each voxel cube according to the signed distance field corresponding to each voxel cube.

[0150] Specifically, in this embodiment, the constructed TSDF volume is traversed by voxel cubes; for each voxel cube, based on the sign change characteristics of its TSDF value, it is determined whether the object surface crosses the edge of the voxel cube. If the sign of the TSDF value corresponding to different edges of the voxel cube changes (i.e. from positive to negative or from negative to positive), it is determined that the object surface crosses the edge.

[0151] S630. Calculate the intersection points between each object surface and edge using linear interpolation, and generate a triangular mesh based on the intersection points.

[0152] This embodiment uses the Marching Cubes algorithm to process the edges that the object's surface traverses. The specific steps are as follows: For each voxel cube edge that is traversed by the object's surface, the intersection point between the edge and the object's surface is calculated using linear interpolation; Based on the intersection points calculated within all voxel cubes, a mesh triangular facet representing the object's surface is generated using the Marching Cubes algorithm.

[0153] Furthermore, the embodiments of this application optimize the generated triangular facets, including noise filtering, hole filling, mesh simplification, and coordinate system alignment operations; at the same time, combining the pose and semantic information of the generated interior layout, the triangular facets of each local area are merged to finally obtain a renderable and editable structured interior 3D scene model.

[0154] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0155] In one specific embodiment, the three-dimensional indoor scene generation process of this application includes the following steps:

[0156] S1. Based on the acquired indoor scene text, generate the indoor layout through a diffusion model;

[0157] The diffusion model is used to learn the layout distribution of a 3D scene. The diffusion model first performs a forward diffusion process, which is defined as a Markov chain at discrete time step t, starting from a clean initial scene layout. Initially, Gaussian noise is gradually added according to a predefined, linearly increasing noise schedule, resulting in the noise level at time step [missing information]. Noisy initial scene encoding ,based on Noisy initial scene encoding at time step t The expression for the forward diffusion process is as follows:

[0158]

[0159] Secondly, the diffusion model noise estimator Encoding the noisy initial scene Iterative denoising is performed to recover a clean initial scene code, i.e., the denoised initial scene code. The noise estimator uses the initial scene code at any time step t during the diffusion process. Text embedding vectors Using time step t as input, the goal is to minimize the denoising objective function to find the true Gaussian noise added to the initial scene layout. for:

[0160]

[0161] in, To add realistic Gaussian noise to the initial scene layout, The noise predicted by the noise estimator Indicates about ,t, and The mathematical expectation.

[0162] In addition, the noise estimator incorporates multiple self-attention layers and cross-attention layers to achieve scene coding denoising under text conditions.

[0163] Finally, the resulting clean scene layout is encoded. Decoding results in a specific set of bounding box parameters M, whose expression is:

[0164]

[0165] Where N is the total number of bounding boxes, and D is the attribute dimension of the bounding boxes; , Represents semantic tags, that is, categories, Indicates the center position, Indicates size, Indicates the rotation angle.

[0166] S2. For generating interior layouts, you can further refine the control of the interior layout by adding, deleting, or modifying the bounding boxes in the interior scene layout.

[0167] S3, Predicting neural radiation fields;

[0168] First, the generated interior layout is converted into a two-dimensional height field and a two-dimensional semantic map of the bird's-eye view BEV. The two-dimensional height field includes the bounding boxes of each object in the interior layout and the maximum and minimum heights of the bounding boxes. The semantic map includes the categories of the bounding boxes (i.e., semantic labels).

[0169] Secondly, the interior layout is divided into sections with a resolution of [resolution value missing]. Several local regions, among which, These represent the number of divisions in the height, width, and depth directions, respectively. For each local region after division, based on its spatial extent, a corresponding portion is extracted from the global two-dimensional height field and two-dimensional semantic map to obtain the local two-dimensional height field corresponding to that region. and local two-dimensional semantic graph Furthermore, the encoder is used to encode the local two-dimensional height field and local two-dimensional semantic map corresponding to each local region, thereby generating the local features corresponding to that local region.

[0170] Finally, the obtained 3D sampling points' locations and local features are used to calculate the hash index for querying the color and volume density features of the 3D sampling points using a hash function. The expression for the hash index is:

[0171] ;

[0172] In the formula, Represents a hash index. Indicates the location of the three-dimensional sampling point. This represents the bitwise XOR operation. Representing prime numbers, Indicates the capacity of the hash table. Represents a three-dimensional spatial coordinate index. This represents the modulo operation;

[0173] The color features and volume density features corresponding to the three-dimensional sampling points are extracted from the local features based on the hash index to obtain the predicted neural radiation field.

[0174] S4. Generate mesh triangular facets;

[0175] First, several viewpoints are arbitrarily sampled within the predicted neural radiation field, and RGB-D images are rendered at these viewpoints. The viewpoints are distributed across a 3D voxel grid, with azimuth angles ranging from 0° to 360°. For each viewpoint to be rendered, its camera parameters are first determined, including the camera's projection center O in the world coordinate system. Based on the perspective camera model, for each pixel in the final generated 2D image, a camera ray can be defined originating from the projection center O and passing through that pixel. The expression for this ray is:

[0176]

[0177] in, It refers to the direction of the ray; therefore, each pixel in the image corresponds to a camera ray. It is a real number greater than 0, any The value of is obtained They are all points on the ray.

[0178] Secondly, a voxel rendering method is used to integrate the predicted neural radiation field along the camera ray corresponding to each pixel to synthesize a two-dimensional image from any viewpoint; for each camera ray, its pixel value... The calculation is achieved through integration, as shown in the following formula:

[0179]

[0180] in, and These represent color information and volume density information, respectively. Indicates cumulative transparency. Indicates in The features obtained from the location, Indicates from Semantic tags extracted from [the source].

[0181] By performing the above integral calculation sequentially on all camera rays, the RGB image corresponding to each viewpoint can be obtained.

[0182] For each camera ray Its depth value Calculated using the following formula:

[0183]

[0184] The depth map for each viewpoint is obtained by calculating the depth value of the corresponding camera ray for each pixel.

[0185] The RGB image and the depth map are stitched together to obtain an RGB-D image.

[0186] Finally, to extract mesh triangular patches from the predicted neural radiation field, the truncated signed distance field (TSDF) was first obtained from the RGB-D image set using Open3D; then, marching was used from the TSDF. The Cube algorithm calculates the mesh triangular facets. Specifically, it involves: fusing multi-view RGB-D images frame by frame into a 3D voxel space using the open-source library Open3D to construct a TSDF volume; traversing voxel cubes on the constructed TSDF volume; for each voxel cube, determining whether the object surface crosses an edge of that voxel cube based on the sign change characteristics of its TSDF value; if the sign of the TSDF value corresponding to different edges of the voxel cube changes, it is determined that the object surface crosses that edge, and the intersection point between the edge and the object surface is calculated using linear interpolation; based on the intersection points calculated within all voxel cubes, generating mesh triangular facets representing the object surface using the MarchingCubes algorithm; further processing the generated mesh triangular facets through noise filtering, hole filling, simplification, and coordinate system alignment, and merging the local meshes by combining layout pose and semantic information to obtain a renderable and editable indoor 3D scene model.

[0187] In summary, this application provides a method and related equipment for generating a 3D indoor scene. The method first obtains a text description of the indoor scene; then, it generates an indoor layout using a preset diffusion model; the indoor layout is converted into a two-dimensional height field and a two-dimensional semantic map; a neural radiation field is predicted based on the two-dimensional height field and the two-dimensional semantic map to obtain the predicted neural radiation field; multiple viewpoints are arbitrarily sampled in the predicted neural radiation field, and these viewpoints are rendered to obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image; a truncated signed distance field is constructed based on the RGB-D image corresponding to each viewpoint, and mesh triangular facets are generated based on the truncated signed distance field to obtain the 3D indoor scene. This application generates an indoor layout from a text description using a diffusion model, achieving automated indoor layout generation and reducing manual intervention; and even without real multi-view images, it generates a neural radiation field based on the two-dimensional height field and the two-dimensional semantic map of the indoor layout, finally generating a high-quality 3D scene consistent with the text description structure based on the neural radiation field, thus improving the efficiency of 3D scene generation.

[0188] like Figure 6 As shown in the diagram, this application also provides a structural schematic diagram of a three-dimensional indoor scene generation device, which can implement the above-described method. The device may include:

[0189] Text description acquisition module 21 is used to acquire text descriptions of indoor scenes;

[0190] The interior layout generation module 22 is used to generate an interior layout from the text description through a preset diffusion model;

[0191] Layout conversion module 23 is used to convert the indoor layout into a two-dimensional height field and a two-dimensional semantic map;

[0192] The neural radiation field prediction module 24 is used to predict the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map to obtain the predicted neural radiation field.

[0193] The rendering module 25 is used to sample multiple viewpoints arbitrarily in the predicted neural radiation field, render the multiple viewpoints, and obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image.

[0194] The triangular patch generation module 26 is used to construct a truncated signed distance field volume based on the RGB-D image corresponding to each viewpoint, and generate mesh triangular patches based on the truncated signed distance field volume to obtain an indoor three-dimensional scene.

[0195] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0196] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described three-dimensional indoor scene generation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0197] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0198] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0199] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0200] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 using the three-dimensional indoor scene generation method of the embodiments of this application.

[0201] The input / output interface 903 is used to implement information input and output;

[0202] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0203] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0204] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0205] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional indoor scene generation method.

[0206] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0207] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0208] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0209] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0210] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0213] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0215] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0216] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating a three-dimensional indoor scene, characterized in that, include: Obtain a text description of the indoor scene; The text description is used to generate an indoor layout through a preset diffusion model, including: encoding the text description to obtain a text embedding vector; Obtain the preset initial scene layout; Gaussian noise is gradually added to the initial scene layout using a preset diffusion model until an initial scene code conforming to a Gaussian distribution is obtained. Based on the text embedding vector and the initial scene code, the initial scene code is iteratively denoised using a preset diffusion model to obtain the denoised initial scene code; The denoised initial scene encoding is decoded to obtain the interior layout, wherein the interior layout includes the bounding box corresponding to each object and the semantic label, center position, size and rotation angle of the bounding box; The interior layout is converted into a two-dimensional height field and a two-dimensional semantic map; The neural radiation field is predicted based on the two-dimensional height field and the two-dimensional semantic map. Multiple viewpoints are arbitrarily sampled in the predicted neural radiation field, and the multiple viewpoints are rendered to obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image. Based on the RGB-D image corresponding to each viewpoint, a truncated signed distance field is constructed, and a mesh triangular facet is generated based on the truncated signed distance field to obtain an indoor 3D scene; The generation of mesh triangular facets based on the truncated signed distance field volume includes: The RGB-D images rendered from multiple viewpoints are fused frame by frame into a three-dimensional voxel space to construct a truncated signed distance field, wherein the truncated signed distance field includes each voxel cube and the truncated signed distance field corresponding to each voxel cube. In the truncated signed distance field, each voxel cube is traversed, and the edges of each object surface that pass through each voxel cube are determined according to the signed distance field corresponding to each voxel cube. The intersection points between each object surface and its edges are calculated using linear interpolation, and mesh triangular patches are generated based on these intersection points.

2. The method for generating a three-dimensional indoor scene according to claim 1, characterized in that, The process of converting the interior layout into a two-dimensional height field and a two-dimensional semantic map includes: Based on the center position and size of the bounding box of each object, determine the projection area of ​​the bounding box of each object in the two-dimensional plane; Based on the dimensions of the bounding box of each object, determine the maximum and minimum height of the bounding box of each object in the projection area; A two-dimensional height field is obtained based on the maximum and minimum heights; The semantic label of the bounding box of each object is assigned to the corresponding projection area to obtain a two-dimensional semantic map.

3. The method for generating a three-dimensional indoor scene according to claim 1, characterized in that, The prediction of the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map, to obtain the predicted neural radiation field, includes: The interior layout is divided into multiple local areas, and the local two-dimensional height field and local two-dimensional semantic map corresponding to each local area are determined. Based on each of the local two-dimensional height fields and local two-dimensional semantic maps, local features of each local region are generated; Based on the local features, the neural radiation field is predicted to obtain the predicted neural radiation field.

4. The method for generating a three-dimensional indoor scene according to claim 3, characterized in that, The step of predicting the neural radiation field based on the local features to obtain the predicted neural radiation field includes: Obtain the position of any three-dimensional sampling point in the indoor layout; Determine the local features corresponding to the positions of the three-dimensional sampling points; The positions of the three-dimensional sampling points and the local features are used to calculate a hash index for querying the color features and volume density features of the three-dimensional sampling points using a hash function, wherein the expression for the hash index is: ； In the formula, Represents a hash index. Indicates the location of the three-dimensional sampling point. This represents the bitwise XOR operation. Representing prime numbers, Indicates the capacity of the hash table. Represents a three-dimensional spatial coordinate index. This represents the modulo operation; Based on the hash index, the color features and volume density features corresponding to the three-dimensional sampling points are extracted from the local features to obtain the predicted neural radiation field.

5. The method for generating a three-dimensional indoor scene according to claim 1, characterized in that, The step of arbitrarily sampling multiple viewpoints in the predicted neural radiation field, rendering the multiple viewpoints, and obtaining an RGB-D image corresponding to each viewpoint includes: Obtain the camera parameters for each viewpoint, and determine the camera ray corresponding to each viewpoint based on the camera parameters; The predicted neural radiation field is integrated along the camera ray corresponding to each viewpoint to obtain the RGB image and depth map corresponding to each viewpoint; The RGB image and the depth map are stitched together to obtain an RGB-D image.

6. A three-dimensional indoor scene generation device, characterized in that, The device includes: The text description acquisition module is used to acquire text descriptions of indoor scenes; An interior layout generation module is used to generate an interior layout from the text description through a preset diffusion model, including: encoding the text description to obtain a text embedding vector; Obtain the preset initial scene layout; Gaussian noise is gradually added to the initial scene layout using a preset diffusion model until an initial scene code conforming to a Gaussian distribution is obtained. Based on the text embedding vector and the initial scene code, the initial scene code is iteratively denoised using a preset diffusion model to obtain the denoised initial scene code; The denoised initial scene encoding is decoded to obtain the interior layout, wherein the interior layout includes the bounding box corresponding to each object and the semantic label, center position, size and rotation angle of the bounding box; The layout conversion module is used to convert the indoor layout into a two-dimensional height field and a two-dimensional semantic map; The neural radiation field prediction module is used to predict the neural radiation field based on the two-dimensional height field and the two-dimensional semantic map, so as to obtain the predicted neural radiation field. The rendering module is used to sample multiple viewpoints arbitrarily in the predicted neural radiation field, render the multiple viewpoints, and obtain an RGB-D image corresponding to each viewpoint, where RGB represents the color information of the two-dimensional image and D represents the depth information of the two-dimensional image. A triangular patch generation module is used to construct a truncated signed distance field volume based on the RGB-D image corresponding to each viewpoint, and generate mesh triangular patches based on the truncated signed distance field volume to obtain an indoor 3D scene. The generation of mesh triangular patches based on the truncated signed distance field volume includes: The RGB-D images rendered from multiple viewpoints are fused frame by frame into a three-dimensional voxel space to construct a truncated signed distance field, wherein the truncated signed distance field includes each voxel cube and the truncated signed distance field corresponding to each voxel cube. In the truncated signed distance field, each voxel cube is traversed, and the edges of each object surface that pass through each voxel cube are determined according to the signed distance field corresponding to each voxel cube. The intersection points between each object surface and its edges are calculated using linear interpolation, and mesh triangular patches are generated based on these intersection points.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the three-dimensional indoor scene generation method as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional indoor scene generation method as described in any one of claims 1 to 5.

Images