Diffusion model-based three-dimensional scene generation method and related equipment

Abstract

The invention discloses a three-dimensional scene generation method based on a diffusion model and related equipment, and the method comprises the steps: obtaining a three-dimensional layout and a style prompt text, and the three-dimensional layout comprises a plurality of objects and backgrounds; modeling each object through three-dimensional Gaussian to obtain a three-dimensional Gaussian parameter of each object, and optimizing the three-dimensional Gaussian parameter of each object through a preset geometric diffusion model; modeling the background through the polygonal mesh to obtain a background polygonal mesh; carrying out geometric reconstruction on the optimized three-dimensional Gaussian parameters to generate an object polygon mesh; combining the background polygon mesh with the object polygon mesh to generate a three-dimensional scene mesh; generating a three-dimensional scene texture based on the three-dimensional scene grid and the style prompt text; and rendering the three-dimensional scene texture to obtain a rendered three-dimensional scene. The scene structure can be accurately controlled, the editability of the generated scene result can be improved, and the method can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and related equipment for generating 3D scenes based on a diffusion model. Background Technology

[0002] 3D scene assets are the fundamental building blocks for constructing virtual environments, supporting visualization, navigation, and interactive behaviors similar to the real physical world. These assets are widely used in various fields such as robot simulation, virtual reality (VR), augmented reality (AR), digital twins, game development, and film and television production. With the development of deep learning and generative models, 3D scene generation technology has gradually become an important research direction in 3D content creation. These methods learn the implicit spatial structure and semantic relationships within large amounts of data to achieve the automatic generation of 3D scenes from abstract inputs (such as text and layouts).

[0003] However, existing 3D scene generation methods utilize Large Language Models (LLM) or Visual Language Models (VLM) to parse user-input text prompts, relying primarily on text prompts as input conditions. Text descriptions struggle to accurately and completely express the spatial positions, scale ratios, and orientation relationships between multiple objects in a complex 3D scene, especially when there are many objects or complex spatial relationships. The generated results often deviate from user expectations, making precise control over the scene structure difficult. Secondly, existing 3D scene generation methods typically represent scenes as a holistic fused mesh or implicit functions, resulting in unclear boundaries between foreground objects and background structures (such as the ground, walls, and ceiling). This representation hinders independent modification, replacement, or reuse of individual objects in subsequent processes, failing to meet the editability requirements of modern 3D content creation and graphics pipelines. Summary of the Invention

[0004] The main objective of this application is to propose a three-dimensional scene generation method and related equipment based on a diffusion model, which can achieve precise control over the scene structure and realize object-level separable three-dimensional scene generation, thereby improving the editability of the generated results.

[0005] To achieve the above objectives, one aspect of this application proposes a three-dimensional scene generation method based on a diffusion model, comprising: Obtain a 3D layout and style hint text, wherein the 3D layout includes multiple objects and a background in the scene; Each object is modeled using a 3D Gaussian model to obtain the 3D Gaussian parameters of each object in the scene space. The 3D Gaussian parameters of each object in the scene space are then optimized using a preset geometric diffusion model to obtain the final optimized 3D Gaussian parameters of each object. The background is modeled using a polygonal mesh, resulting in a background polygonal mesh. For each object, perform geometric reconstruction using the final optimized 3D Gaussian parameters to generate a polygonal mesh corresponding to each object; The background polygon mesh and the object polygon mesh are combined to generate a three-dimensional scene mesh; Based on the 3D scene mesh and the style prompt text, a 3D scene texture is generated; The three-dimensional scene texture is rendered to obtain the rendered three-dimensional scene.

[0006] In some embodiments, modeling each object using a 3D Gaussian model to obtain the 3D Gaussian parameters of each object in the scene space includes: Define the initial three-dimensional Gaussian parameters for each of the objects in the coordinate system; Obtain the bounding box parameters of each object in the 3D layout; Based on the bounding box parameters, the initial three-dimensional Gaussian parameters in the coordinate system are spatially transformed to obtain the three-dimensional Gaussian parameters of each object in the scene space.

[0007] In some embodiments, optimizing the three-dimensional Gaussian parameters of each object in the scene space using a preset geometric diffusion model to obtain the final optimized three-dimensional Gaussian parameters of each object includes: The three-dimensional layout is rendered to obtain a semantic graph; Render the three-dimensional Gaussian parameters of each object in the scene space to obtain a normal-depth map; The semantic map and the normal-depth map are input into a preset first geometric diffusion model to calculate a first optimization gradient, wherein the expression for the first optimization gradient is:

[0008] in, This represents the first optimization gradient. Represents a normal-depth map. This represents the latent variable encoding of the VAE corresponding to the normal-depth map. Represents a semantic graph. This represents the time step weight function. This represents the pre-defined first geometric diffusion model. Indicates sampling noise Satisfies Gaussian distribution , This indicates the calculation of sampling time t and sampling noise. Expected value Represents partial derivatives, This represents the three-dimensional Gaussian parameters, and t represents the sampling time; Based on the first optimization gradient, the three-dimensional Gaussian parameters of each object in the scene space are optimized to obtain the initial optimized three-dimensional Gaussian parameters. Obtain the semantic information corresponding to each object, input the semantic information and the normal-depth map into a preset second geometric diffusion model, and calculate the second optimization gradient, wherein the expression of the second optimization gradient is:

[0009] in, For the second optimization gradient, For the pre-defined second geometric diffusion model, This represents the initial optimized 3D Gaussian parameters. Represents semantic information, Indicates sampling noise Satisfies Gaussian distribution ; The initial optimized 3D Gaussian parameters are optimized based on the second optimization gradient to obtain the final optimized 3D Gaussian parameters for each object.

[0010] In some embodiments, the geometric reconstruction of the final optimized 3D Gaussian parameters for each object to generate a polygonal mesh corresponding to each object includes: Multiple camera viewpoints are defined on the sphere surrounding each object; The final optimized 3D Gaussian parameters of each object are rendered using the multiple camera perspectives to obtain a multi-view depth map for each object. The multi-view depth maps corresponding to each object are fused together to generate a polygonal mesh for each object.

[0011] In some embodiments, generating a 3D scene texture based on the 3D scene mesh and the style cue text includes: Obtain the preset first object perception neural texture parameters; Determine the UV coordinates of each object and the UV coordinates of the background in the 3D scene mesh; Based on the semantic map, the normal-depth map, and the style cue text, calculate the third optimization gradient; The first object perception neural texture parameters are optimized based on the third optimization gradient to obtain the second object perception neural texture parameters. Obtain the normal map of the 3D layout, and calculate the fourth optimization gradient based on the style hint text and the normal map; Based on the fourth optimization gradient, the texture parameters of the second object are optimized to obtain the texture parameters of the third object perception neural network. A 3D scene texture is generated based on the UV coordinates of each object, the UV coordinates of the background, and the sensory neural texture parameters of the third object.

[0012] In some embodiments, calculating the third optimization gradient based on the semantic map, the normal-depth map, and the style cue text includes: The semantic map, the normal-depth map, and the style hint text are input into a preset first appearance diffusion model to calculate the first gradient component and the second gradient component. The preset first appearance diffusion model calculates a third optimized gradient based on the first and second gradient components, wherein the expression for the third optimized gradient is:

[0013] In the formula, This represents the third optimization gradient. and This represents the time step weight function. Represents partial derivatives, No classification guidance value. This represents the neural texture parameters for the first object perception. Represents an RGB image. Indicates sampling noise Satisfies Gaussian distribution , Represents the first gradient component. This represents the second gradient component.

[0014] In some embodiments, calculating the fourth optimization gradient based on the style hint text and the normal map includes: The style hint text and the normal map are input into a preset second appearance diffusion model, and the fourth optimization gradient is calculated, including:

[0015] In the formula, This represents the fourth optimization gradient. This represents the preset second appearance diffusion model. This indicates style-related text, where 't' represents the sampling time. This represents the neural texture parameters for second object perception. Represents a normal diagram.

[0016] To achieve the above objectives, another aspect of this application proposes a three-dimensional scene generation apparatus based on a diffusion model, the apparatus comprising: The layout and text acquisition module is used to acquire 3D layout and style hint text, wherein the 3D layout includes multiple objects and background in the scene; The first modeling module is used to model each of the objects using a three-dimensional Gaussian model to obtain the three-dimensional Gaussian parameters of each object in the scene space, and to optimize the three-dimensional Gaussian parameters of each object in the scene space using a preset geometric diffusion model to obtain the final optimized three-dimensional Gaussian parameters of each object. The second modeling module is used to model the background using a polygonal mesh to obtain a background polygonal mesh; The object mesh generation module is used to perform geometric reconstruction on the final optimized 3D Gaussian parameters of each object, and generate the object polygon mesh corresponding to each object; A 3D scene mesh generation module is used to combine the background polygon mesh and the object polygon mesh to generate a 3D scene mesh; A 3D scene texture generation module is used to generate 3D scene textures based on the 3D scene mesh and the style prompt text; The rendering module is used to render the three-dimensional scene texture to obtain the rendered three-dimensional scene.

[0017] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0018] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0019] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0020] The embodiments of this application include at least the following beneficial effects: This application provides a 3D scene generation method based on a diffusion model. The method first obtains a 3D layout and style hint text, wherein the 3D layout includes multiple objects and a background in the scene; each object is modeled using a 3D Gaussian model to obtain the 3D Gaussian parameters of each object in the scene space, and the 3D Gaussian parameters of each object in the scene space are optimized using a preset geometric diffusion model to obtain the final optimized 3D Gaussian parameters of each object; the background is modeled using a polygonal mesh to obtain a background polygonal mesh; the final optimized 3D Gaussian parameters of each object are geometrically reconstructed to generate an object polygonal mesh corresponding to each object; the background polygonal mesh and the object polygonal mesh are combined to generate a 3D scene mesh; a 3D scene texture is generated based on the 3D scene mesh and style hint text; and the 3D scene texture is rendered to obtain the rendered 3D scene. This application introduces a 3D layout as a core guiding condition, clarifying the basic composition of objects and backgrounds in the scene. Combined with 3D Gaussian modeling and geometric diffusion model optimization, it enhances the controllability of structure and spatial relationships during the generation of complex 3D scenes, avoiding the uncertainty brought about by relying solely on text prompts. At the same time, by adopting a modeling method that separates objects and backgrounds, the meshes of objects and backgrounds are combined to generate 3D scene meshes, realizing the decoupled representation of objects and backgrounds. This allows for independent editing, replacement, and reuse of individual objects, significantly improving the operability of the generated scene in practical graphics applications and meeting the requirements of modern 3D content creation and graphics pipelines for the editability of scene assets. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart of a three-dimensional scene generation method based on a diffusion model provided in an embodiment of this application; Figure 3 This is a flowchart of generating a three-dimensional scene texture provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a three-dimensional scene generation device based on a diffusion model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0023] It is understood that the terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0026] Existing 3D scene generation methods utilize Large Language Models (LLM) or Visual Language Models (VLM) to parse user-input text prompts. They primarily rely on text prompts as input conditions. However, text descriptions struggle to accurately and completely express the spatial positions, scale ratios, and orientation relationships between multiple objects in a complex 3D scene. This is especially true when there are many objects or complex spatial relationships, often resulting in deviations from user expectations and hindering precise control over the scene structure. Secondly, existing 3D scene generation methods typically represent scenes as a holistic fused mesh or implicit functions, resulting in unclear boundaries between foreground objects and background structures (such as the ground, walls, and ceiling). This representation hinders independent modification, replacement, or reuse of individual objects in subsequent workflows, failing to meet the editability requirements of modern 3D content creation and graphics pipelines.

[0027] In view of this, this application provides a method and related device for generating a 3D scene based on a diffusion model. The method first obtains a 3D layout and style hint text, where the 3D layout includes multiple objects and a background in the scene. Each object is modeled using a 3D Gaussian model to obtain its 3D Gaussian parameters in the scene space. These parameters are then optimized using a preset geometric diffusion model to obtain the final optimized 3D Gaussian parameters for each object. The background is modeled using a polygonal mesh to obtain a background polygonal mesh. The final optimized 3D Gaussian parameters for each object are geometrically reconstructed to generate a corresponding object polygonal mesh. The background polygonal mesh and the object polygonal mesh are combined to generate a 3D scene mesh. A 3D scene texture is generated based on the 3D scene mesh and style hint text. The 3D scene texture is then rendered to obtain the rendered 3D scene. This application introduces a 3D layout as a core guiding condition, clarifying the basic composition of objects and backgrounds in the scene. Combined with 3D Gaussian modeling and geometric diffusion model optimization, it enhances the controllability of structure and spatial relationships during the generation of complex 3D scenes, avoiding the uncertainty brought about by relying solely on text prompts. At the same time, by adopting a modeling method that separates objects and backgrounds, the meshes of objects and backgrounds are combined to generate 3D scene meshes, realizing the decoupled representation of objects and backgrounds. This allows for independent editing, replacement, and reuse of individual objects, significantly improving the operability of the generated scene in practical graphics applications and meeting the requirements of modern 3D content creation and graphics pipelines for the editability of scene assets.

[0028] The diffusion-based 3D scene generation method provided in this application relates to the field of computer vision technology. This diffusion-based 3D scene generation method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the diffusion-based 3D scene generation method, but is not limited to the above forms.

[0029] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0030] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided in an embodiment of this application. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0031] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0032] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0033] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment does not impose any limitations.

[0034] For example, based on Figure 1 The implementation environment shown in this application embodiment provides a three-dimensional scene generation method based on a diffusion model. The following description uses the application of this three-dimensional scene generation method based on a diffusion model in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0035] Reference Figure 2 , Figure 2 This is a flowchart illustrating a diffusion-based 3D scene generation method for a server, provided as an embodiment of this application. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). See also... Figure 2 The method may include the following steps: S100. Obtain the 3D layout and style hint text, wherein the 3D layout includes multiple objects and the background in the scene.

[0036] In this context, a 3D layout refers to a design generated by the user using modern graphics pipelines (such as Blender and Unreal Engine). A 3D layout comprises two core components: first, several objects in the scene and their bounding box parameters; and second, the background of the interior scene. Each object's bounding box parameters include its 3D position, size (i.e., dimensions), rotation angle, and semantic information (e.g., bed, sofa, table, etc.), described using textual language. The interior scene background includes the ceiling, floor, and walls.

[0037] Style hint text is entered by the user in text form and is used to define the visual style of the entire 3D scene, such as Chinese style, European style, cartoon style, minimalist style, etc.

[0038] S200. Model each object using a 3D Gaussian model to obtain the 3D Gaussian parameters of each object in the scene space. Optimize the 3D Gaussian parameters of each object in the scene space using a preset geometric diffusion model to obtain the final optimized 3D Gaussian parameters of each object.

[0039] This application embodiment uses an efficient three-dimensional Gaussian representation as a surrogate geometric representation of objects; exemplarily, the step of modeling each object using three-dimensional Gaussian to obtain the three-dimensional Gaussian parameters of each object includes S210-S230: S210. Define the initial three-dimensional Gaussian parameters of each object in the coordinate system; Specifically, for each object defined in the 3D layout, in the normal coordinate system, each object is represented as a set of initial 3D Gaussian parameters. For example, the first... The initial three-dimensional Gaussian parameters of an object Defined as follows:

[0040] Each three-dimensional Gaussian is obtained through its position. Rotation ,scale Opacity and semantic category color Parametric representation, where, Let (x, y, z) be a real number field. Represents a 3x3 matrix, where each element is a real number field.

[0041] S220. Obtain the bounding box parameters of each object in the three-dimensional layout; The bounding box parameters are used to guide the spatial transformation of the initial 3D Gaussian parameters, ensuring that the transformed object Gaussian can be consistent with the 3D layout constraints.

[0042] S230. Based on the bounding box parameters, perform a spatial transformation on the initial three-dimensional Gaussian parameters in the coordinate system to obtain the three-dimensional Gaussian parameters of each object in the scene space.

[0043] In this embodiment of the application, to correctly place an object in a normalized coordinate system into the actual scene space using Gaussian, it is necessary to perform a spatial transformation on the initial three-dimensional Gaussian parameters of each object based on its bounding box parameters; for example, for a given... The initial 3D Gaussian parameters of an object, whose Gaussian transformation in normal space is applied to the scene space. The transformation method is as follows: , , , , ; in, This indicates the position of the Gaussian point in scene space after the transformation. This indicates the rotation of the Gaussian point in scene space after the transformation. This indicates the scale of the Gaussian point in the scene space after the transformation. This represents the opacity of the Gaussian point in scene space after the transformation. This represents the semantic category color of the Gaussian point in the scene space after the transformation. Represents a size vector The largest component in, Indicates the first Rotation matrix of bounding boxes, No. Translation vector of each bounding box.

[0044] Furthermore, the steps for optimizing the 3D Gaussian parameters of each object in the scene space using a preset geometric diffusion model include S240-S290: S240. Render the three-dimensional layout to obtain a semantic map; Specifically, in this embodiment of the application, a semantic map is extracted and rendered from the three-dimensional layout by rendering the camera pose of the defined three-dimensional scene. The semantic map represents the object category or background category (such as wall, ground, bed, sofa, etc.) to which the corresponding scene location belongs.

[0045] S250. Render the three-dimensional Gaussian parameters of each object in the scene space to obtain a normal-depth map; For example, the 3D Gaussian representation of all objects in scene space is rendered using the same camera pose as the semantic map to obtain a normal-depth map, ensuring consistent spatial alignment between the two. The normal-depth map integrates the geometric normal information and depth information of the scene.

[0046] S260. Input the semantic map and normal-depth map into the preset first geometric diffusion model, calculate the first optimization gradient, and optimize the three-dimensional Gaussian parameters of each object in the scene space based on the first optimization gradient to obtain the initial optimized three-dimensional Gaussian parameters.

[0047] Specifically, scene geometry optimization is performed using a score distillation sampling method, leveraging the optimization gradient provided by a pre-trained layout-guided geometry diffusion model (i.e., a preset first geometry diffusion model). The expression for the first optimization gradient is:

[0048] in, This represents the first optimization gradient (i.e., the loss function). (gradient of the three-dimensional Gaussian parameters relative to the entire object) Represents a normal-depth map. This represents the latent variable encoding of the VAE corresponding to the normal-depth map. Represents a semantic graph. This represents the time step weight function. This represents the pre-defined first geometric diffusion model. Indicates sampling noise Satisfies Gaussian distribution , This indicates the relationship between sampling time t and sampling noise. The expected value of these two random variables is calculated. The two random variables satisfy a uniform distribution U and a normal distribution, respectively. (That is, Gaussian distribution) Represents partial derivatives, This represents the three-dimensional Gaussian parameters, and t represents the sampling time; For example, in embodiments of this application, the calculated first optimization gradient can be used to iteratively optimize the three-dimensional Gaussian parameters of all objects in the scene using optimization algorithms such as gradient descent (e.g., the AdamW optimizer).

[0049] S270. Obtain the semantic information corresponding to each object, input the semantic information and normal-depth map into the preset second geometric diffusion model, calculate the second optimization gradient, and optimize the initially optimized three-dimensional Gaussian parameters based on the second optimization gradient to obtain the final optimized three-dimensional Gaussian parameters for each object.

[0050] Because the scene geometry optimization process only updates the Gaussian of the visible parts of the objects in the scene, and due to the limited number of camera view samples and the existence of object occlusion, the invisible parts of the objects cannot be effectively supervised, thus remaining in an unupdated state during the optimization process. This ultimately leads to incomplete object geometry, which adversely affects subsequent object-level editing and applications. Therefore, this application introduces object geometry optimization as a supplement, performed independently for each object, using a preset object-level geometry diffusion model (i.e., a preset second geometry diffusion model) to generate a second optimization gradient. The expression for the second optimization gradient is:

[0051] in, This represents the second optimization gradient. This represents the score distillation sampling loss function. This represents the pre-defined second geometric diffusion model. This represents the initial optimized 3D Gaussian parameters; Furthermore, although the scene geometry optimization process can ensure that the generated scene geometry is generally consistent with the input layout, it still cannot strictly constrain the object geometry within its corresponding bounding box. Therefore, embodiments of this application introduce a regularization loss term during the optimization process to ensure that the generated object Gaussians are tightly distributed within their corresponding bounding boxes.

[0052] During optimization, the Gaussian points of an object typically expand outwards from a smaller region. To address this phenomenon, this application applies a penalty to Gaussian points located in the normalized coordinate space that exceed the object's bounding box, with a regularization loss... The definition is as follows:

[0053] in, Indicates the first The normalized result of the bounding box size of an object.

[0054] S300: Model the background using a polygonal mesh to obtain a background polygonal mesh.

[0055] For example, in this embodiment of the application, the scene background elements (such as walls, ground, ceiling, etc.) have relatively simple geometric structures. To balance modeling efficiency and accuracy, a more computationally efficient polygonal mesh is used for modeling, which is formally represented as follows:

[0056] in, V represents the background polygonal mesh, and V and E represent the set of vertices and the set of edges of the polygonal mesh, respectively.

[0057] S400. Perform geometric reconstruction on the final optimized 3D Gaussian parameters for each object to generate a polygonal mesh corresponding to each object.

[0058] Extracting object meshes from 3D Gaussian representation is a crucial step in constructing scene meshes. Current mainstream methods employ a density query-based strategy: densely sampling Gaussian points in space, calculating volume density based on their opacity, and further generating the mesh using the Marching Cubes algorithm. However, this type of method is highly sensitive to the selection of the number of Gaussian points and the isosurface threshold, resulting in insufficient robustness of the mesh extraction process and poor stability of the generated results. To address the problem of insufficient robustness and poor stability of the mesh extraction process due to the high sensitivity to the selection of the number of Gaussian points and the isosurface threshold, this application proposes a depth-fusion-based 3D Gaussian mesh extraction strategy. Its core advantage lies in utilizing the high-resolution geometric information provided by the depth map, effectively avoiding dependence on hyperparameters such as the number of Gaussian points or the isosurface threshold, thereby achieving higher quality and more robust mesh extraction results. For example, the steps for generating the object polygon mesh corresponding to each object include S410-S430: S410. Determine multiple camera viewpoints on the sphere surrounding each object; Specifically, for each object, multiple camera angles are arranged on the surface of its enclosing sphere (which can completely contain the object's three-dimensional Gaussian distribution). All cameras are pointed toward the center of the object's Gaussian distribution and toward the center of the enclosing sphere to ensure that the object can be observed completely from all directions.

[0059] S420. Render the final optimized 3D Gaussian parameters of each object using the multiple camera perspectives to obtain a multi-view depth map corresponding to each object.

[0060] S430. Perform depth fusion on the multi-view depth map corresponding to each object to generate an object polygon mesh corresponding to each object.

[0061] For example, in this application embodiment, a voxel fusion method is used to fuse multi-view depth maps. By aligning the depth information from different viewpoints, the complete three-dimensional surface of the object is reconstructed, and finally, an object polygon mesh for each object is generated.

[0062] S500: Combine the background polygon mesh with the object polygon mesh to generate a three-dimensional scene mesh.

[0063] S600. Based on the three-dimensional scene mesh and the style prompt text, generate a three-dimensional scene texture.

[0064] To achieve object-level texture operations, this application introduces an object-aware neural texture representation. Its core is to integrate the texture maps of each object and the background into a unified scene texture and organize it using a mesh-based UV packing strategy, thereby supporting subsequent independent texture operations on individual objects and the background.

[0065] For example, Figure 3 This is a flowchart of generating a 3D scene texture provided in an embodiment of this application, such as... Figure 3 As shown, the steps for generating 3D scene textures include S610-S670: S610, Obtain the preset first object perception neural texture parameters; Among them, the first object perception neural texture parameter is the initialized object perception neural texture parameter. The object perception neural texture parameter is obtained by parameterization through multi-resolution hash encoding combined with multilayer perceptron (MLP) and is used to map two-dimensional texture coordinates into color values.

[0066] S620. Determine the UV coordinates of each object and the UV coordinates of the background in the three-dimensional scene mesh.

[0067] To associate object-perception neural texture parameters with specific 3D geometric surfaces and assign an independent texture space to each object, embodiments of this application employ a mesh-based UV packing strategy. For example, the entire 2D texture coordinate space (UV space) is divided into a... The grid, in which, Where m and n are positive integer values, for example, m=8, n=64, which means that the coordinate space needed for 64 objects is divided into an 8x8 grid; for the 3D scene mesh, the first... Given an object and a background, assign it a unique cell in the mesh, with any point on its surface... UV coordinates The definition is as follows:

[0068] in, This represents the UV unwrapping function, which unwrappes the object... or three-dimensional surface points in the background Mapped to two-dimensional texture coordinates.

[0069] This application embodiment uses a mesh-based UV packing strategy to map the textures of all objects and the background onto the same texture map, but isolates them in different mesh units, supporting texture extraction for individual objects and the background.

[0070] S630. Calculate the third optimization gradient based on the semantic graph, the normal-depth graph, and the style hint text.

[0071] To generate textures that conform to style cues and are consistent with the scene's geometry and semantic layout, this application constructs a layout-guided appearance diffusion model (i.e., a preset first appearance diffusion model). This model, based on a pre-trained StableDiffusion model, introduces additional conditional controls (i.e., semantic map, normal map, and depth map). The advantage of utilizing multiple conditions is that it effectively eliminates the ambiguity caused by using only a single condition. For example, a layout-guided appearance diffusion model is constructed by integrating ControlNet with Stable Diffusion. Specifically, three independent ControlNets are used to extract features from the semantic map, normal map, and depth map, respectively, and these features are concatenated and fused into the Stable Diffusion decoder to generate an RGB image. This effectively eliminates the ambiguity caused by using only a single condition and improves the accuracy of texture generation.

[0072] Based on this pre-defined first appearance diffusion model, the invariant score distillation method is used to optimize the neural texture parameters of the first object. First, using the current neural texture parameters of the first object and a 3D scene mesh, an RGB image of the scene is rendered using differentiable rasterization and encoded as a VAE latent variable. Then, a third optimization gradient is calculated to update the neural texture parameters of the first object. Exemplarily, the steps for calculating the third optimization gradient include S631-S632: S631. Input the semantic map, the normal-depth map and the style hint text into a preset first appearance diffusion model, and calculate the first gradient component and the second gradient component; S632. The preset first appearance diffusion model calculates a third optimized gradient based on the first gradient component and the second gradient component, wherein the expression of the third optimized gradient is:

[0073] In the formula, The loss function represents the invariant score distillation loss. This represents the third optimization gradient. and This represents the time step weight function. Represents partial derivatives, No classification guidance value. This represents the neural texture parameters for the first object perception. Represents an RGB image. Represents the first gradient component. The second gradient component is represented by the first gradient component, which refers to the invariant gradient component under the same full condition but at different time steps, and the second gradient component refers to the category-guided gradient component under full and no conditions. Wherein, the first gradient component Second gradient component The calculation expression is:

[0074] in, The first appearance diffusion model is preset. Encoding VAE latent variables for RGB images, In order to pass through Perform c-step DDIM inference to obtain the VAE latent variable codes. This indicates conditional input, including style hint text and semantic graphs. and normal-depth map , and These are the time step weight functions, No classification guidance value. Sampling time, It is an empty set (meaning no conditions are input).

[0075] In addition, to enhance the detail of the rendered image, this application introduces reconstruction loss during scene optimization. It is defined as an RGB image and the result of decoding. The obtained reconstructed image Euclidean distance between them:

[0076] S640. Optimize the first object perception neural texture parameters based on the third optimization gradient to obtain the second object perception neural texture parameters. In this embodiment, the first object perception neural texture parameters can be iteratively updated and optimized using the gradient descent algorithm. This round of optimization mainly focuses on the overall scene appearance to ensure the consistency of texture style and matching with the macro layout. The updated parameters are denoted as the second object perception neural texture parameters.

[0077] S650. Obtain the normal map of the three-dimensional layout, and calculate the fourth optimization gradient based on the style hint text and the normal map; Since scene-level texture optimization mainly relies on supervision of rendering from an external perspective, the texture of areas invisible due to occlusion (such as the side of furniture facing away from a wall) may not be sufficiently optimized, resulting in incomplete object textures. To address this issue, this application introduces object texture optimization as a supplement, performed independently for each individual object. First, an independent normal map of the object is rendered to provide its own geometric details. Then, using an object-level appearance diffusion model (i.e., a preset second appearance diffusion model), based on the object's normal map and the scene's style cues, a score distillation sampling method is used to calculate the fourth optimization gradient of the perceptual neural texture parameters for the object. Exemplarily, the steps for calculating the fourth optimization gradient include: The style hint text and the normal map are input into a preset second appearance diffusion model, and the fourth optimization gradient is calculated, including:

[0078] In the formula, This represents the fourth optimization gradient. This represents the preset second appearance diffusion model. This indicates style-related text, where 't' represents the sampling time. This represents the neural texture parameters for second object perception.

[0079] S660. Based on the fourth optimization gradient, optimize the texture parameters of the second object to obtain the third object perception neural texture parameters.

[0080] S670. Generate a three-dimensional scene texture based on the UV coordinates of each object, the UV coordinates of the background, and the sensory neural texture parameters of the third object.

[0081] After optimization, based on the third object perception neural texture parameters (i.e. the optimized hash code and MLP), the UV coordinates corresponding to each object and background surface point are input into the neural field, and the RGB color values ​​are queried and output, thereby coloring the entire 3D scene mesh and generating the final 3D scene texture.

[0082] S700. Render the three-dimensional scene texture to obtain the rendered three-dimensional scene.

[0083] For example, in embodiments of this application, the 3D scene texture of the textured scene mesh described above can be imported into a modern graphics pipeline (such as Blender, Unreal, etc.) for rendering, simulation, or editing to obtain the final rendered 3D scene.

[0084] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0085] In a specific embodiment, the 3D scene generation process based on the diffusion model of this application includes the following steps: S1. Obtain the 3D layout and style text prompts input by the user; Users can design 3D layouts within scenes using modern graphics pipelines (such as Blender and Unreal Engine). A 3D layout consists of two parts: bounding boxes for several objects within the scene, and the background of the interior scene. The bounding box parameters for each object in the scene include the object's 3D position, size, rotation angle, and semantic information (e.g., bed, sofa, etc.), which is described using textual language. The background of the interior scene refers to elements such as the ceiling, floor, and walls, represented using polygons; for example, a wall can be represented by a rectangle. Style hints are input by the user to describe the style of the entire scene, using textual language, such as Chinese style, European style, cartoon style, etc.

[0086] S2, 3D scene mesh generation; The object is modeled using a 3D Gaussian model, and the background is modeled using a polygonal mesh, thus effectively achieving a decoupled representation of the object and the background.

[0087] S21. Model the objects using 3D Gaussian, optimize the obtained 3D Gaussian parameters, and generate a polygonal mesh for each object based on the optimized 3D Gaussian parameters. Specifically: S21.1 Define a three-dimensional Gaussian parameter set for each object in a normal coordinate system; S21.2 To satisfy the input layout constraints during scene geometry generation, a layout-guided geometry diffusion model is used to optimize the 3D Gaussian parameters of each object in the scene space for scene geometry optimization. Since the scene geometry optimization process only updates the Gaussian parameters of objects in the visible part of the scene, and due to the limited number of camera view samples and the existence of object occlusion, the invisible parts of the objects cannot be effectively supervised and remain unupdated during optimization, ultimately resulting in incomplete object geometry. This incompleteness will adversely affect subsequent object-level editing and application. Therefore, an object-level geometry diffusion model is used to further optimize the object geometry. Specifically, based on the object-level geometry diffusion model, a score distillation sampling method is used to further optimize the 3D Gaussian parameters of randomly sampled objects under the conditional constraints of the object text description. Furthermore, a penalty is applied to Gaussian points located in the normal coordinate space that are outside the object's bounding box to effectively limit the disordered expansion of the object geometry outside the bounding box during optimization, thereby improving the compliance of the generated results with layout constraints and enhancing the rationality of the scene structure.

[0088] S21.3 After scene geometry optimization and object geometry optimization, a deep fusion-based 3D Gaussian mesh extraction strategy is used to extract the polygonal mesh corresponding to each object. Specifically, for the 3D Gaussian parameters corresponding to each object, a depth map is first rendered using multiple camera views located on a sphere, with all cameras facing the center of the object's Gaussian mesh. Subsequently, a voxel fusion method is used to fuse the obtained multi-view depth maps to reconstruct the corresponding object mesh. By utilizing the high-resolution geometric information provided by the depth map, the dependence on hyperparameters such as the number of Gaussian points or the isosurface threshold is effectively avoided, thereby achieving a higher quality and more robust mesh extraction effect.

[0089] S22. Model the background using a polygonal mesh to obtain a background polygonal mesh; S23. Combine the polygonal mesh of each object with the extracted object mesh to construct a complete scene mesh.

[0090] S3, 3D scene texture generation; S31. Construct the initial object perception neural texture parameters; To achieve object-level texture operations, this application introduces an object-aware neural texture representation. Its core is to integrate the texture maps of each object and background into a unified scene texture, and organize it using a mesh-based UV packing strategy. This supports subsequent independent texture operations on individual objects and the background. The object-aware neural texture parameters are obtained through parameterization using multi-resolution hash encoding combined with a multilayer perceptron (MLP), and are used to map two-dimensional texture coordinates to color values. The UV coordinates of the object and background are determined by a mesh-based UV packing strategy, which divides the entire 2D texture coordinate space (UV space) into a grid. The grid, in which, For the first in the 3D scene mesh For an object, the UV coordinates are determined using the UV unwrapping function, in order to unwrap the object... 3D surface points Mapped to two-dimensional texture coordinates.

[0091] S32, Scene texture optimization; To generate textures that conform to style cues and are consistent with the scene's geometry and semantic layout, this application constructs a layout-guided appearance diffusion model. This model, based on pre-trained Stable Diffusion, introduces additional conditional controls (i.e., semantic map, normal map, and depth map). The advantage of utilizing multiple conditions is that it effectively eliminates the ambiguity caused by using only a single condition. For example, a layout-guided appearance diffusion model is constructed by integrating ControlNet with Stable Diffusion. Specifically, three independent ControlNets are used to extract features from the semantic map, normal map, and depth map, respectively. These features are then concatenated and fused into the Stable Diffusion decoder to generate an RGB image, thereby effectively eliminating the ambiguity caused by using only a single condition and improving the accuracy of texture generation.

[0092] Secondly, based on the layout-guided appearance diffusion model, the invariant score distillation method is used to optimize the initial object neural texture parameters, that is, to optimize the texture of the overall scene; in addition, a reconstruction loss is introduced in the optimization process to enhance the detail of the RGB image.

[0093] S33, Object texture optimization; To alleviate the problem of incomplete object texture optimization caused by object self-occlusion during the above-mentioned scene appearance optimization process, an object-level appearance diffusion model (i.e., ControlNet based on normal conditions) is adopted to supplement and optimize the object texture of each object. Specifically, the object-level appearance diffusion model uses the score distillation sampling (SDS) method to optimize the randomly sampled object perception neural texture parameters (object perception neural texture parameters obtained from scene optimization) under the conditions of object rendering normal map and style cue words. This is to effectively optimize the object texture of occluded areas, generate complete and high-fidelity object textures, and ensure their consistency with the overall style and geometric layout of the scene.

[0094] S34. After optimization, based on the final optimized object perception neural texture parameters (i.e., optimized hash code and MLP), input the UV coordinates corresponding to each object and background surface point into the neural field, query and output the RGB color values, thereby coloring the entire 3D scene mesh and generating the final 3D scene texture.

[0095] S4. Import the 3D scene texture of the textured scene mesh into a modern graphics pipeline (such as Blender, Unreal, etc.) for subsequent rendering, simulation, editing, etc., to obtain the final 3D scene.

[0096] In summary, this application provides a method and related equipment for generating 3D scenes based on a diffusion model. The method first obtains a 3D layout and style hint text, where the 3D layout includes multiple objects and a background in the scene. Each object is modeled using a 3D Gaussian model to obtain its 3D Gaussian parameters in the scene space. These parameters are then optimized using a preset geometric diffusion model to obtain the final optimized 3D Gaussian parameters for each object. The background is modeled using a polygonal mesh to obtain a background polygonal mesh. The final optimized 3D Gaussian parameters for each object are geometrically reconstructed to generate a corresponding object polygonal mesh. The background polygonal mesh and the object polygonal mesh are combined to generate a 3D scene mesh. A 3D scene texture is generated based on the 3D scene mesh and style hint text. Finally, the 3D scene texture is rendered to obtain the rendered 3D scene. This application introduces a 3D layout as a core guiding condition, clarifying the basic composition of objects and backgrounds in the scene. Combined with 3D Gaussian modeling and geometric diffusion model optimization, it enhances the controllability of structure and spatial relationships during the generation of complex 3D scenes, avoiding the uncertainty brought about by relying solely on text prompts. At the same time, by adopting a modeling method that separates objects and backgrounds, the meshes of objects and backgrounds are combined to generate 3D scene meshes, realizing the decoupled representation of objects and backgrounds. This allows for independent editing, replacement, and reuse of individual objects, significantly improving the operability of the generated scene in practical graphics applications and meeting the requirements of modern 3D content creation and graphics pipelines for the editability of scene assets.

[0097] like Figure 4 As shown in the figure, this application embodiment also provides a structural schematic diagram of a 3D scene generation device based on a diffusion model. This device can implement the above-described method and may include: The layout and text acquisition module 21 is used to acquire a 3D layout and style hint text, wherein the 3D layout includes multiple objects and a background in the scene; The first modeling module 22 is used to model each of the objects using a three-dimensional Gaussian model to obtain the three-dimensional Gaussian parameters of each object in the scene space, and to optimize the three-dimensional Gaussian parameters of each object in the scene space using a preset geometric diffusion model to obtain the final optimized three-dimensional Gaussian parameters of each object. The second modeling module 23 is used to model the background using a polygonal mesh to obtain a background polygonal mesh; The object mesh generation module 24 is used to perform geometric reconstruction on the final optimized three-dimensional Gaussian parameters of each object and generate an object polygon mesh corresponding to each object. The 3D scene mesh generation module 25 is used to combine the background polygon mesh and the object polygon mesh to generate a 3D scene mesh; The 3D scene texture generation module 26 is used to generate a 3D scene texture based on the 3D scene mesh and the style prompt text; The rendering module 27 is used to render the three-dimensional scene texture to obtain the rendered three-dimensional scene.

[0098] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0099] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned three-dimensional scene generation method based on the diffusion model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0100] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0101] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the diffusion model-based 3D scene generation method of the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0102] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described diffusion-based 3D scene generation method.

[0103] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0104] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0106] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0110] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating 3D scenes based on a diffusion model, characterized in that, include: Obtain a 3D layout and style hint text, wherein the 3D layout includes multiple objects and a background in the scene; Each object is modeled using a 3D Gaussian model to obtain the 3D Gaussian parameters of each object in the scene space. The 3D Gaussian parameters of each object in the scene space are then optimized using a preset geometric diffusion model to obtain the final optimized 3D Gaussian parameters of each object. The background is modeled using a polygonal mesh, resulting in a background polygonal mesh. For each object, perform geometric reconstruction using the final optimized 3D Gaussian parameters to generate a polygonal mesh corresponding to each object; The background polygon mesh and the object polygon mesh are combined to generate a three-dimensional scene mesh; Based on the 3D scene mesh and the style prompt text, a 3D scene texture is generated; The three-dimensional scene texture is rendered to obtain the rendered three-dimensional scene.

2. The method for generating a 3D scene based on a diffusion model according to claim 1, characterized in that, The process of modeling each object using a 3D Gaussian model to obtain the 3D Gaussian parameters of each object in the scene space includes: Define the initial three-dimensional Gaussian parameters for each of the objects in the coordinate system; Obtain the bounding box parameters of each object in the 3D layout; Based on the bounding box parameters, the initial three-dimensional Gaussian parameters in the coordinate system are spatially transformed to obtain the three-dimensional Gaussian parameters of each object in the scene space.

3. The method for generating a 3D scene based on a diffusion model according to claim 1, characterized in that, The optimization of the three-dimensional Gaussian parameters of each object in the scene space using a preset geometric diffusion model yields the final optimized three-dimensional Gaussian parameters for each object, including: The three-dimensional layout is rendered to obtain a semantic graph; Render the three-dimensional Gaussian parameters of each object in the scene space to obtain a normal-depth map; The semantic map and the normal-depth map are input into a preset first geometric diffusion model to calculate a first optimization gradient, wherein the expression for the first optimization gradient is: in, This represents the first optimization gradient. Represents a normal-depth map. This represents the latent variable encoding of the VAE corresponding to the normal-depth map. Represents a semantic graph. This represents the time step weight function. This represents the pre-defined first geometric diffusion model. Indicates sampling noise Satisfies Gaussian distribution , This indicates the calculation of sampling time t and sampling noise. Expected value Represents partial derivatives, This represents the three-dimensional Gaussian parameters, and t represents the sampling time; Based on the first optimization gradient, the three-dimensional Gaussian parameters of each object in the scene space are optimized to obtain the initial optimized three-dimensional Gaussian parameters. Obtain the semantic information corresponding to each object, input the semantic information and the normal-depth map into a preset second geometric diffusion model, and calculate the second optimization gradient, wherein the expression of the second optimization gradient is: in, For the second optimization gradient, For the pre-defined second geometric diffusion model, This represents the initial optimized 3D Gaussian parameters. Represents semantic information, Indicates sampling noise Satisfies Gaussian distribution ; The initial optimized 3D Gaussian parameters are optimized based on the second optimization gradient to obtain the final optimized 3D Gaussian parameters for each object.

4. The method for generating a 3D scene based on a diffusion model according to claim 1, characterized in that, The step of geometrically reconstructing the final optimized 3D Gaussian parameters for each object to generate a polygonal mesh corresponding to each object includes: Multiple camera viewpoints are defined on the sphere surrounding each object; The final optimized 3D Gaussian parameters of each object are rendered using the multiple camera perspectives to obtain a multi-view depth map for each object. The multi-view depth maps corresponding to each object are fused together to generate a polygonal mesh for each object.

5. The method for generating a 3D scene based on a diffusion model according to claim 3, characterized in that, The process of generating a 3D scene texture based on the 3D scene mesh and the style prompt text includes: Obtain the preset first object perception neural texture parameters; Determine the UV coordinates of each object and the UV coordinates of the background in the 3D scene mesh; Based on the semantic map, the normal-depth map, and the style cue text, calculate the third optimization gradient; The first object perception neural texture parameters are optimized based on the third optimization gradient to obtain the second object perception neural texture parameters. Obtain the normal map of the 3D layout, and calculate the fourth optimization gradient based on the style hint text and the normal map; Based on the fourth optimization gradient, the texture parameters of the second object are optimized to obtain the texture parameters of the third object perception neural network. A 3D scene texture is generated based on the UV coordinates of each object, the UV coordinates of the background, and the sensory neural texture parameters of the third object.

6. The method for generating a 3D scene based on a diffusion model according to claim 5, characterized in that, The calculation of the third optimization gradient based on the semantic map, the normal-depth map, and the style cue text includes: The semantic map, the normal-depth map, and the style hint text are input into a preset first appearance diffusion model to calculate the first gradient component and the second gradient component. The preset first appearance diffusion model calculates a third optimized gradient based on the first and second gradient components, wherein the expression for the third optimized gradient is: In the formula, This represents the third optimization gradient. and This represents the time step weight function. Represents partial derivatives, No classification guidance value. This represents the neural texture parameters for the first object perception. Represents an RGB image. Indicates sampling noise Satisfies Gaussian distribution , Indicates the first gradient component. This represents the second gradient component.

7. The method for generating a 3D scene based on a diffusion model according to claim 5, characterized in that, The calculation of the fourth optimization gradient based on the style hint text and the normal map includes: The style hint text and the normal map are input into a preset second appearance diffusion model, and the fourth optimization gradient is calculated, including: In the formula, This represents the fourth optimization gradient. This represents the preset second appearance diffusion model. This indicates style-related text, where 't' represents the sampling time. This represents the neural texture parameters for second object perception. Represents a normal diagram.

8. A three-dimensional scene generation device based on a diffusion model, characterized in that, The device includes: The layout and text acquisition module is used to acquire 3D layout and style hint text, wherein the 3D layout includes multiple objects and background in the scene; The first modeling module is used to model each of the objects using a three-dimensional Gaussian model to obtain the three-dimensional Gaussian parameters of each object in the scene space, and to optimize the three-dimensional Gaussian parameters of each object in the scene space using a preset geometric diffusion model to obtain the final optimized three-dimensional Gaussian parameters of each object. The second modeling module is used to model the background using a polygonal mesh to obtain a background polygonal mesh; The object mesh generation module is used to perform geometric reconstruction on the final optimized 3D Gaussian parameters of each object, and generate the object polygon mesh corresponding to each object; A 3D scene mesh generation module is used to combine the background polygon mesh and the object polygon mesh to generate a 3D scene mesh; A 3D scene texture generation module is used to generate 3D scene textures based on the 3D scene mesh and the style prompt text; The rendering module is used to render the three-dimensional scene texture to obtain the rendered three-dimensional scene.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the three-dimensional scene generation method based on the diffusion model as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional scene generation method based on the diffusion model as described in any one of claims 1 to 7.

Images