Three-dimensional scene generation method based on visual language large model guidance and geometric constraint diffusion

Abstract

The application relates to a three-dimensional scene generation method based on visual language large model guidance and geometric constraint diffusion, which comprises the following steps: S1, constructing a three-dimensional layout prior of a scene based on a visual language large model; S2, generating a geometric attribute corresponding to each object by using the visual language large model, so as to generate a three-dimensional initial scene; S3, optimizing the geometric structure and appearance details of the obtained three-dimensional initial scene based on noise resampling of score distillation; and after the geometric constraint based on the three-dimensional Gaussian scene geometric constraint is adopted in the process of optimizing the geometric structure, a final three-dimensional scene is generated. Compared with the prior art, the application has the advantages that the geometric consistency under multiple perspectives can be effectively maintained, the spatial structure information in the text description can be more accurately realized, and the application is suitable for the automatic generation of complex three-dimensional scenes and the like.

Description

Technical Field

[0001] This invention relates to the field of scene generation in computer vision, and in particular to a method for generating three-dimensional scenes based on large visual language model guidance and geometric constraint diffusion. Background Technology

[0002] In recent years, automatically generating 3D scenes from text or images has become an important research direction in computer vision and computer graphics. Diffusion models have made breakthrough progress in 2D image generation tasks, making it possible to generate high-quality visual content from natural language descriptions. Building on this, researchers have begun to explore transferring the capabilities of 2D generative models to 3D space, automatically generating 3D objects or complex scenes through text prompts.

[0003] Most methods for generating 3D scenes from text employ fractional distillation sampling to introduce 2D generation models into the 3D representation optimization process, constructing a distillation method for 2D-to-3D information to generate 3D content. Subsequent research has further explored different 3D representations, such as neural radiation field (NeRF) or 3DGS-based generation methods. However, because these methods typically optimize directly based on 2D diffusion models, they lack explicit modeling of scene structure, thus limiting their application in generating complex multi-object scenes. For example, during multi-view rendering, the generated results are prone to geometric distortion, structural instability, or chaotic object relationships, making it difficult for the generated scene to meet the geometric consistency requirements of realistic 3D space. Meanwhile, 3D scenes are usually composed of multiple objects with clear semantic relationships, thus requiring consistent geometric structure and reasonable spatial layout under multi-view conditions. Existing methods primarily rely on textual prompts as prior information for 3D layout during generation, lacking explicit constraints on object spatial relationships and geometric layout, making it difficult to accurately express the structural information in the text description. For example, when the text contains complex semantics such as the relative positions, scale relationships, or spatial orientations of objects, the generated results often struggle to maintain a stable 3D structure. Furthermore, since the optimization process relies heavily on the pseudo-true image generated by the two-dimensional diffusion model, its unstable quality will further affect the geometric consistency and visual quality of the three-dimensional generation results.

[0004] A search of Chinese Patent Publication No. CN120543704A reveals a method and apparatus for generating dynamic 3D scenes from text. The method includes: acquiring a first natural language description input by a user and multiple scene images taken from different angles; generating a static 3D scene based on the first natural language description and the multiple scene images; acquiring a second natural language description input by the user into a large language model, the second natural language description including relative scale information and constraint information of the multiple objects, the relative scale information describing the actual physical dimensions of the multiple objects in the static 3D scene, and the constraint information describing physical conflicts existing during the movement of the multiple objects; and generating inferred motion trajectories and inferred dimensions of the multiple objects in the static 3D scene based on the second natural language description and the static 3D scene, to obtain a dynamic 3D scene. This solution enhances the realism and richness of autonomous driving simulation testing. However, although the existing patent also uses a language model for semantic parsing, its focus is on extracting object information from the text and generating a static 3D scene by combining multi-view images. It does not explicitly construct an independent geometric layout prior that can be used to guide the subsequent generation process before generation. At the same time, although the existing patent also uses fractional distillation gradient to optimize the scene, its optimization process is relatively conventional. It mainly relies on the joint image diffusion model and the 3D perception diffusion model to calculate the gradient, focusing on the consistency of the overall geometry and texture, without actively managing and decoupling the optimization conflicts between different perspectives.

[0005] Therefore, how to simultaneously improve the accuracy of geometric layout, the consistency of multi-view optimization, and the physical rationality of 3D geometric structure in text-driven 3D scene generation, so as to generate higher quality 3D scenes that are more consistent with the text description, has become a technical problem that needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a 3D scene generation method based on a large visual language model and geometric constraint diffusion. By introducing a large visual language model, the geometric layout and object relationships of the scene are explicitly modeled and constrained. During the diffusion generation process, structural consistency and semantic rationality are jointly optimized, thereby improving the geometric accuracy and visual quality of 3D scene generation. This method can effectively maintain geometric consistency under multiple perspectives and more accurately realize the spatial structure information in the text description, making it suitable for the automatic generation of complex 3D scenes.

[0007] The objective of this invention can be achieved through the following technical solutions: According to a first aspect of the present invention, a method for generating a 3D scene based on a large visual language model and geometric constraint diffusion is provided, comprising: Step S1: Construct a 3D layout prior for the scene based on a large visual language model; Step S2: Use the large visual language model to generate the geometric attributes corresponding to each object, thereby generating the initial 3D scene; Step S3: Based on fractional distillation, noise resampling is used to optimize the geometry and appearance details of the obtained 3D initial scene; and after applying 3D Gaussian-based scene geometric constraints during the geometry optimization process, the final 3D scene is generated.

[0008] As a preferred technical solution, the visual language large model in step S1 adopts GPT-4v. GPT-4v performs structured parsing of the input text, extracts object category, spatial relationship and geometric attribute information from the input text description, and constructs the three-dimensional layout prior of the scene accordingly.

[0009] As a preferred technical solution, in step S2, the geometric properties of each object are defined to obtain the geometric layout of the entire scene; then, a coarse point cloud representation of each object is generated using a three-dimensional point cloud diffusion model, and the point cloud is converted into a three-dimensional Gaussian representation to generate a three-dimensional initial scene.

[0010] As a preferred technical solution, the geometric attributes include the object's center coordinates, rotation angle, scale factor, height, width, and length.

[0011] As a preferred technical solution, the noise resampling based on fractional distillation in step S3 includes: calculating the similarity between rendered images from different perspectives in the current batch at fixed iterations, decoupling the optimization between different perspectives, and making the optimization directions of different perspectives more diversified.

[0012] As a preferred technical solution, the scene geometric constraints based on three-dimensional Gaussian in step S3 include rotation constraints, position constraints, and scale constraints. Geometric constraints are applied to the three-dimensional Gaussian from three dimensions: rotation, position, and scale, to ensure the geometric rationality of the generated scene.

[0013] As a preferred technical solution, the rotation constraint, position constraint and scale constraint are combined to obtain the overall geometric loss function. This loss function improves the geometric consistency of the generated scene while maintaining the visual realism of the rendering by fusing the constraints of rotation, position and scale.

[0014] As a preferred technical solution, the rotation constraint is specifically as follows: first, the target rotation is defined, then the target rotation is converted into a Gaussian rotation attribute through a Gaussian transformation matrix, and finally, the loss of the rotation constraint is modeled based on the Fisher distribution, so that the Gaussian rotation gradually approximates the target rotation; The position constraint specifically refers to constraining the position attributes of Gaussian based on the relative positions of objects in the text. The scale constraint is specifically defined as follows: Let the scale vector of the i-th Gaussian object be si, and ensure the consistency of the object scale by constraining the norm of si or the size relationship of each dimension.

[0015] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0016] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0017] Compared with the prior art, the present invention has the following advantages: 1) Before starting 3D generation, this invention first uses a visual language model to perform deep structured decoding of the input text, explicitly constructing a priori geometric layout of the scene. This priori not only includes which objects need to be generated in the scene, but also clearly defines the spatial relationships between these objects and their respective geometric attributes. In other words, this invention has already formed a clear scene blueprint before actually starting to generate 3D content. With the pre-planned structured layout information as a guide, the 3D scene generated by this invention is more reasonable in terms of object placement, relative size, and spatial hierarchy, and is more in line with the expected arrangement in the text description. 2) This invention introduces a noise matching strategy between time steps. Specifically, instead of using single-step prediction results in isolation, this invention performs noise matching and calibration between different diffusion time steps to obtain a more stable optimized gradient signal. This strategy effectively suppresses the cumulative effect of single-step prediction errors, making the entire optimization process smoother and more reliable. At the same time, an optimization decoupling mechanism based on viewpoint similarity is designed. This mechanism periodically calculates the similarity between two-dimensional images rendered from different viewpoints and uses this similarity information to actively adjust the optimization direction of different viewpoints. When the optimization direction of one viewpoint conflicts significantly with other viewpoints, the mechanism will appropriately decouple the optimization paths of these viewpoints, allowing each viewpoint to develop in a more diversified direction, rather than mutually restricting each other and getting trapped in local optima. The synergistic effect of these two mechanisms brings significant advantages; time-step noise matching makes the optimization process more stable and reduces geometric artifacts caused by noise interference; the viewpoint decoupling mechanism effectively avoids mutual interference between multi-view optimization signals, so that the generated 3D scene can maintain good consistency and integrity when viewed from any angle.

[0018] 3) This invention directly imposes constraints on the three core geometric properties of the 3D Gaussian representation, namely rotation, position and scale. The rotation constraint ensures that the orientation of the object conforms to the physical laws and the expectations in the text description; the position constraint ensures that the spatial relationship between objects is accurate and avoids unreasonable interweaving or floating; the scale constraint ensures that the relative size of each object in the scene conforms to the real proportion. The advantages of this multi-dimensional geometric constraint are multifaceted. First, since the constraint acts directly on the basic properties of the 3D Gaussian rather than indirectly through the deformation network, the constraint effect is more direct and effective. Second, by simultaneously constraining the three dimensions of rotation, position, and scale, complete control over the 3D geometric structure is achieved, making the generated object more reasonable in terms of posture, position, and size. Third, these geometric constraints, together with the aforementioned layout priors and optimization control mechanisms, constitute a comprehensive quality assurance system from macroscopic layout to microscopic geometry.

[0019] 4) The explicit geometric layout prior of this invention provides an accurate structural framework for the scene at the macro level; the time-step noise matching and viewpoint decoupling mechanism ensures the stability of the optimization process and multi-view consistency at the meso level; the multi-dimensional geometric constraints of 3D Gaussian ensure the physical rationality of each object at the micro level. In terms of layout accuracy, this invention can more accurately reproduce the spatial structure information implicit in the text description, and the placement and interrelationships of objects are more reasonable. In terms of multi-view consistency, the scene generated by this invention maintains a coherent geometric structure when viewed from any angle, avoiding obvious deformation or inconsistency when switching perspectives. In terms of physical rationality, the objects generated by this invention are more in line with the physical laws of the real world in terms of rotational posture, spatial position, and relative scale, reducing unnatural interpenetration or suspension problems. In terms of optimization stability, the generation process of this invention is smoother and more reliable, reducing artifacts and quality fluctuations caused by noise accumulation. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the three-dimensional scene generation method of the present invention; Figure 2 This is a flowchart of the noise resampling and geometric constraint process of the present invention; Figure 3 For comparison of the generation quality of this invention with other methods Figure 1 ; Figure 4 For comparison of the generation quality of this invention with other methods Figure 2 ; Figure 5 For comparison of the generation quality of this invention with other methods Figure 3 ; Figure 6For comparison of the generation quality of this invention with other methods Figure 4 ; Figure 7 This is a schematic diagram of the scene visualization results generated by the present invention; Figure 8 This is a flowchart illustrating the specific process of the three-dimensional scene generation method of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] Example 1 like Figure 1 and Figure 8 As shown, the present invention provides a 3D scene generation method based on a large visual language model and geometric constraint diffusion, which specifically includes: Step S1: Prior generation of scene layout guided by visual language model In complex 3D scene generation tasks, scenes are typically composed of multiple objects with clear semantic relationships. For example, the text description "There is a vase on the table, and a book next to it" not only contains object category information but also implicitly implies the spatial relationships and structural layout between objects. If 3D generation is performed directly based solely on text prompts, diffusion models often struggle to accurately recover the relative positions and scale relationships between objects, leading to problems such as chaotic spatial layout or unstable geometric structures in the generated results. Therefore, explicitly constructing a priori geometric layout of the scene before 3D generation is crucial for improving the stability and geometric consistency of complex scene generation.

[0023] The specific process is as follows: Structured text parsing: Powerful visual language models (such as GPT-4V) are used to analyze the input scene text description. The model extracts key structured information, including: Object categories: Identify all objects mentioned in the description (such as "table", "vase", "book", "sculpture", "house", "tree", etc.).

[0024] Spatial relationships: Analyze the relative positions between objects (such as "above", "to the left of", "beside", "behind", "in the middle", etc.).

[0025] Geometric attributes: Infer the object's posture, approximate orientation, size, and other implicit geometric information (such as "placed horizontally" or "facing a wall").

[0026] Constructing a 3D layout description: Based on the information parsed above, generate specific geometric attribute definitions for each object in the scene in 3D space. These attributes typically include: Center coordinates: The position of an object in three-dimensional space.

[0027] Rotation angle: The orientation of an object in three-dimensional space (Euler angle or quaternion).

[0028] Scale factor: The size of an object in each dimension (length, width, height).

[0029] Bounding box dimensions: the height, width, and length of the object.

[0030] In this way, we transform a natural language description into a structured 3D layout description containing multiple objects and their spatial relationships, providing a clear spatial prior for subsequent 3D generation.

[0031] Initializing the 3D representation of objects: For each object defined in the layout, this invention uses a pre-trained 3D point cloud diffusion model to generate a coarse 3D point cloud representation of the object. Subsequently, these point clouds are converted into differentiable renderable 3D Gaussian representations as the initial state for the entire scene generation.

[0032] Step S2, 3D scene optimization based on fractional distillation and noise resampling After obtaining the initial 3D Gaussian representation of the scene, it needs to be optimized to make its appearance and geometric details conform to the text description. Traditional methods (such as SDS) often lead to multi-view inconsistencies (such as the Janus problem) when optimizing complex scenes due to the instability of the diffusion model prediction. This invention introduces two key technologies to improve optimization stability: Cross-time-step noise matching: When using a diffusion model to predict noise in a 2D image generated from a differentiable 3D scene through differentiable rendering, this invention does not rely solely on the noise predicted in a single step. Instead, it establishes noise consistency constraints across different time steps. Specifically, by establishing correlations between latent variables in adjacent time steps, the optimization gradient is not only derived from the noise prediction of the current step but is also "pulled" by the predictions of adjacent steps. This reduces the randomness brought about by single-step prediction, provides a more stable optimization signal for updating 3D parameters, and effectively reduces geometric distortion caused by the accumulation of single-step prediction errors in the diffusion model.

[0033] Noise Resampling Mechanism: To further promote the diversity of generated content across multiple perspectives and avoid repetitive structures due to convergent optimization directions across all perspectives, this invention introduces a noise resampling strategy. This strategy is executed at fixed intervals of iteration: Calculate the similarity between multiple images rendered from different perspectives in the current batch.

[0034] Based on similarity analysis, optimization paths between different perspectives are decoupled. By processing the optimization signals from each perspective differently, different perspectives are encouraged to update in more diverse directions.

[0035] This mechanism can effectively alleviate the Janus problem (i.e., the appearance of multiple identical features, such as multiple "heads", from different perspectives) and improve the multi-view consistency and geometric rationality of the generated scene.

[0036] Step S3, Explicit geometric constraints based on 3D Gaussian While diffusion models can generate visual representations of 3D scenes that match textual descriptions, relying solely on 3D Gaussian splashing for optimization often fails to guarantee the stability of the generated geometry. In complex multi-object scenes, the lack of explicit geometric constraints can easily lead to problems such as object misalignment, inconsistent scale, and pose distortion during optimization, resulting in a chaotic spatial layout, distorted relationships between objects, and ultimately affecting visual realism. Therefore, introducing geometric constraints during the geometric optimization process is essential. For 3D Gaussian scene generation tasks, geometric constraints typically encompass dimensions such as position, rotation, scale, and spatial relationships between objects. These constraints not only contain semantic information but also involve limitations on spatial geometry. Therefore, the overall structure of the scene can be indirectly controlled by constraining the properties of the Gaussian model. Specifically, this section applies geometric constraints to the 3D Gaussian model in three dimensions: rotation, position, and scale, while also incorporating spatial relationship constraints between objects to ensure the geometric rationality of the generated scene.

[0037] Rotation Constraints: To ensure that the pose of the generated object is consistent with the prior text, Gaussian rotation properties need to be constrained. Before scene optimization, the target rotation needs to be defined (such as "table placed horizontally" or "chair facing the wall"). The target rotation is converted into Gaussian rotation properties using a Gaussian transformation matrix. Then, the loss of rotation constraints is modeled based on the Fisher distribution, so that the Gaussian rotation gradually approximates the target rotation, avoiding pose distortion or abnormal deviation.

[0038] Positional Constraints: In complex 3D scenes, the spatial relationships between objects are key information describing the scene structure. To ensure that the object layout in the generated scene matches the spatial relationships described in the text, the Gaussian positional properties need to be constrained based on the relative positions of objects in the text (e.g., "the vase is on the left side of the table," "the book is on the right side of the vase," etc.).

[0039] Scale Constraints: To avoid excessive differences in object scale or shape distortion during generation, the method in this section constrains the scale of each Gaussian object. Let the scale vector of the i-th Gaussian object be si. By constraining the norm of si or the size relationship of each dimension, the consistency of object scale is ensured, avoiding situations where "large objects contain small objects" or "scale abrupt changes" and maintaining the visual rationality of the scene.

[0040] Joint geometric loss: Combining the constraints mentioned above, an overall geometric loss can be obtained. This loss function, by fusing constraints of rotation, position, and scale, significantly improves the geometric consistency of the generated scene while maintaining the visual realism of the rendering, thereby generating a high-quality 3D scene that conforms to textual semantics and has a stable spatial structure.

[0041] Specific implementation examples: When the scene generation prompt "The Parthenon stands center, flanked by two pairs of Greek warrior sculptures in front and back. Two altar-topped columns stand before the front sculptures. A staircase with a cat sculpture is to the front left, with two columns before it. A bull statue stands behind the Parthenon" is input into this invention and other 3D scene generation methods (ProlificDreamer, Text2Nerf, Set-the-Scene, DreamScen), the result is as follows: Figure 3 The comparison images shown demonstrate that the scene generated by this invention exhibits higher geometric consistency and a more stable spatial structure. For example... Figure 7 The image shown is a schematic diagram illustrating the generated prompt words for the input scenario of this invention.

[0042] When the scene is input into this invention and other 3D scene generation methods (ProlificDreamer, Text2Nerf, Set-the-Scene, DreamScen), and the generation prompt "A pavilion in a meadow between a green tree and cherry blossom tree, facing a fountain with a dog beside. Behind the dog are two benches, with a rabbit sitting on one of the benches. A poplar tree stands next to the bench in the corner, with a windmill opposite the tree," it can be translated as: "A pavilion in a meadow between a green tree and a cherry blossom tree, facing a fountain. A dog lies beside the fountain, and behind the dog are two benches, with a rabbit sitting on one of the benches. A poplar tree stands next to the bench in the corner, with a windmill opposite the tree." The result is as follows: Figure 4 The comparison images shown demonstrate that the scene generated by this invention exhibits higher geometric consistency and a more stable spatial structure. For example... Figure 7 The image shown is a schematic diagram illustrating the generated prompt words for the input scenario of this invention.

[0043] When the scene generation prompt "On the moon are an astronaut and a robot side by side, facing each other, with a human lunar base and a rocket behind them, and a spaceship above. Next to the alien is a meteorite, and above it is a UFO" is input into this invention and other 3D scene generation methods (ProlificDreamer, Text2Nerf, Set-the-Scene, DreamScen), it can be translated as "On the moon, an astronaut and a robot stand side by side, facing each other. Behind them are a human lunar base and a rocket, and above them is a spaceship. Next to the alien is a meteorite, and above the meteorite is a UFO.", the following results can be obtained: Figure 5 The comparison images shown demonstrate that the scene generated by this invention exhibits higher geometric consistency and a more stable spatial structure. For example... Figure 7The image shown is a schematic diagram illustrating the generated prompt words for the input scenario of this invention.

[0044] When the scene generation prompt "Trees on the right, a farmhouse in the middle, a greenhouse on the left, and a tractor in front of it. In the center, carrots are planted in the grass with fences around them. Two cows graze on the grass in front of the farmhouse, and two sheep are near the carrots. Next to the tractor stands a horse" is input into this invention and other 3D scene generation methods (ProlificDreamer, Text2Nerf, Set-the-Scene, DreamScen), the result is as follows: Figure 6 The comparison images shown demonstrate that the scene generated by this invention exhibits higher geometric consistency and a more stable spatial structure. For example... Figure 7 The image shown is a schematic diagram illustrating the generated prompt words for the input scenario of this invention.

[0045] Table 1 Table 1 shows the evaluation metrics obtained by this invention and other 3D scene generation methods (Prolific Dreamer, Text2Nerf, Set-the-Scene, Dream Scene) after generating prompts based on the same scene input.

[0046] The table includes: 1. BRISQUE: Blind Image Spatial Quality Evaluator. The smaller the value, the better the image quality.

[0047] 2. NIQE: Natural Image Quality Evaluator. A metric for evaluating the quality of natural images; the lower the value, the better the image quality.

[0048] 3. CLIP-Score: Image-text alignment score based on the CLIP model. The higher the value, the better the generated content matches the text description.

[0049] 4. PIQE: Perception-based Image Quality Evaluator. The smaller the value, the better the image quality.

[0050] Data Analysis and Conclusions 1) Image quality (BRISQUE, NIQE, PIQE): This invention performs optimally across all three "lower is better" metrics (BRISQUE=29.7051, NIQE=3.2767, PIQE=14.8321), lower than all comparable methods (e.g., Prolific Dreamer's BRISQUE=44.1761, NIQE=4.7702; Dream Scene's BRISQUE=30.9875, NIQE=3.5624). This indicates that the image quality generated by this invention is superior.

[0051] 2) Text alignment score (CLIP-Score): The CLIP-Score of this invention is 0.8376, which is higher than all comparison methods (such as Dream Scene=0.7103, Text2Nerf=0.6726); indicating that the content generated by this invention has a higher degree of matching with the text description.

[0052] The comparison shows that the present invention achieves leading results in both image quality and text alignment, verifying the advanced nature of the method.

[0053] The above is an introduction to the method embodiments. The following embodiments using electronic devices and storage media will further illustrate the solution of the present invention.

[0054] This invention also provides an electronic device including a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0055] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0056] The processing unit executes the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S3 by any other suitable means (e.g., by means of firmware).

[0057] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0058] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0059] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0060] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating 3D scenes based on a large visual language model and geometric constraint diffusion, characterized in that, include: Step S1: Construct a 3D layout prior for the scene based on a large visual language model; Step S2: Use the large visual language model to generate the geometric attributes corresponding to each object, thereby generating the initial 3D scene; Step S3: Optimize the geometry and appearance details of the obtained 3D initial scene based on fractional distillation noise resampling; The final 3D scene is generated after geometric constraints based on 3D Gaussian are applied during the optimization of the geometric structure.

2. The method for generating a 3D scene based on a large visual language model and geometric constraint diffusion as described in claim 1, characterized in that, The visual language big model in step S1 uses GPT-4v to perform structured parsing of the input text, extracting object categories, spatial relationships, and geometric attribute information from the input text description, and constructing a three-dimensional layout prior of the scene accordingly.

3. The method for generating a 3D scene based on a large visual language model and geometric constraint diffusion as described in claim 1, characterized in that, In step S2, the geometric properties of each object are defined to obtain the geometric layout of the entire scene; then, a coarse point cloud representation of each object is generated using a 3D point cloud diffusion model, and the point cloud is converted into a 3D Gaussian representation to generate the initial 3D scene.

4. The method for generating a 3D scene based on a large visual language model and geometric constraint diffusion as described in claim 3, characterized in that, The geometric properties include the object's center coordinates, rotation angle, scale factor, height, width, and length.

5. The method for generating a 3D scene based on a large visual language model and geometric constraint diffusion as described in claim 1, characterized in that, The noise resampling based on fractional distillation in step S3 includes: calculating the similarity between rendered images from different perspectives in the current batch at fixed iterations, decoupling the optimization between different perspectives, and making the optimization directions of different perspectives more diversified.

6. The method for generating a 3D scene based on a large visual language model and geometric constraint diffusion as described in claim 1, characterized in that, The scene geometric constraints based on 3D Gaussian in step S3 include rotation constraints, position constraints, and scale constraints. Geometric constraints are applied to 3D Gaussian from three dimensions: rotation, position, and scale, to ensure the geometric rationality of the generated scene.

7. The method for generating a 3D scene based on a large visual language model and geometric constraint diffusion as described in claim 6, characterized in that, The rotation constraint, position constraint, and scale constraint are combined to obtain the overall geometric loss function. This loss function improves the geometric consistency of the generated scene while maintaining the visual realism of the rendering by fusing the constraints of rotation, position, and scale.

8. A method for generating 3D scenes based on a large visual language model and geometric constraint diffusion as described in claim 6, characterized in that, The rotation constraint is specifically defined as follows: first, the target rotation is defined, then the target rotation is converted into a Gaussian rotation attribute through a Gaussian transformation matrix, and finally, the loss of the rotation constraint is modeled based on the Fisher distribution, so that the Gaussian rotation gradually approximates the target rotation. The position constraint specifically refers to constraining the position attributes of Gaussian based on the relative positions of objects in the text. The scale constraint is specifically defined as follows: Let the scale vector of the i-th Gaussian object be si, and ensure the consistency of the object scale by constraining the norm of si or the size relationship of each dimension.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images