Virtual three-dimensional space generation method, system and equipment based on large model and medium

Abstract

The invention relates to the technical field of computer vision, in particular to a virtual three-dimensional space generation method, system and device based on a large model and a medium, and the method comprises the steps: carrying out the low-rank fine tuning of a pre-trained image large model, and enabling the pre-trained image large model to adapt to the drawing of a space scene; generating a three-dimensional space white mold according to the input parameters and extracting depth and edge information of the three-dimensional space white mold; depth and edge dual ControlNet is introduced, geometric information is taken as a condition, and a panoramic image is generated in combination with a style text; rendering the image to the surface of the model through a depth replacement method; and comparing the depth information of the image and the model, performing deformation processing on the geometric shape of the model, and finally outputting a texture and geometry fused three-dimensional space model. According to the method, automatic generation of the three-dimensional space from text description to the stylized model is realized, and style diversity, geometric accuracy and shape richness are effectively guaranteed.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, device, and medium for generating virtual three-dimensional space based on a large model. Background Technology

[0002] The digital generation and stylized design of 3D interior spaces have wide applications in fields such as architectural design, virtual reality, game development, and home furnishing e-commerce. Traditional 3D interior space generation technologies mainly rely on the following methods: Rule-based and component-library-based methods: These methods combine and assemble predefined geometric rules and a rich library of interior component models (such as walls, doors, windows, and furniture) to quickly generate structurally sound 3D white models. However, the generated results heavily rely on the size and design of the template library, leading to a tendency towards formulaic spatial layouts and styles, making it difficult to achieve highly diverse, personalized, and artistic stylistic expressions. The visual representation of walls and the overall space (texture, material, lighting) usually requires designers to perform extensive manual texturing and rendering, resulting in low levels of automation and intelligence.

[0003] Methods based on Building Information Modeling (BIM) or scanning reconstruction: BIM methods can generate accurate models with rich information, but focus on engineering data rather than visual style; 3D reconstruction technology based on laser scanning or images can restore the geometry of existing spaces, but cannot create new design styles, and it is difficult to make controlled stylistic edits of the reconstruction results.

[0004] Deep learning-based generative methods: In recent years, with the development of deep learning technology, especially the breakthroughs in large-scale image generation models such as diffusion models, new ideas have been provided for the generation of visual content in indoor scenes. Existing technologies attempt to directly generate indoor scene images or simple 3D representations using text descriptions. However, these methods face significant bottlenecks when applied to the generation of 3D indoor spaces: Poor geometric controllability: Directly generated images often fail to accurately match specific and complex indoor 3D geometric structures (such as accurate wall boundaries and spatial perspective relationships), resulting in inconsistent generated content with the target space in terms of size, proportion, and structure, making them unsuitable for direct use in driving the creation or texturing of 3D models.

[0005] Disconnect between style and structure: Existing methods lack a mechanism to tightly couple high-level stylistic semantics (such as "cyberpunk") with underlying spatial geometric information. Simply splicing textual prompts and spatial parameters can easily lead to superficial stylization effects or damage the rational structure and functionality of the space.

[0006] Lack of 3D spatial interaction: Existing technologies mostly remain at the level of generating 2D images or simple 3D voxels, failing to achieve the reverse influence and dynamic adjustment from stylized images to 3D geometric models. In other words, the generated content is a "skin pasted on," rather than an organic part that can interact with spatial geometry and evolve together. This results in the generated 3D space lacking depth and realistic shape complexity, with the style and shape remaining monotonous.

[0007] Therefore, existing technologies have failed to effectively address the core issue of achieving highly diverse and intelligent style generation for 3D interior spaces while ensuring geometric accuracy, and further enabling style-driven adaptive deformation of spatial geometry. There is an urgent need for an intelligent generation method that can deeply integrate semantic style control and geometric constraints, and achieve bidirectional driving of style and 3D geometry. Summary of the Invention

[0008] In view of this, the purpose of this invention is to propose a method, system, device and medium for generating virtual three-dimensional space based on a large model, so as to solve the problem that the existing technology cannot achieve highly diversified and intelligent style generation of three-dimensional indoor space while ensuring geometric accuracy.

[0009] To achieve the above objectives, this invention provides a method for generating a virtual three-dimensional space based on a large model, comprising the following steps: Step S1: Construct training data for the spatial scene image rendering task, and use the low-rank matrix adaptation method to fine-tune the pre-trained existing large image model to obtain the fine-tuned large image model. Step S2: Based on the input three-dimensional indoor space parameters, automatically generate the corresponding three-dimensional indoor space model, and extract the depth and edge information of the space from the model; Step S3: Introduce depth information ControlNet and edge information ControlNet into the fine-tuned image large model. Using the depth information and edge information extracted in step S2 as conditional inputs, and combined with the input stylized text description information, the fine-tuned image large model generates a stylized panoramic image corresponding to the three-dimensional indoor space model. Step S4: Based on the depth permutation method, render the generated stylized panoramic image onto the surface of the 3D indoor space model generated in step S2; Step S5: Perform depth detection on the stylized panoramic image to obtain image depth information, and combine it with the depth information of the three-dimensional indoor space model itself to perform corresponding deformation processing on the geometry of the three-dimensional indoor space model in order to improve its shape complexity. Step S6: Generate and output a multi-style, multi-shape 3D interior space model that integrates stylized panoramic image textures and deformed geometric shapes.

[0010] Preferably, the spatial scene image rendering task is a 360-degree panoramic image rendering task. The construction of training data includes collecting or generating 360-degree panoramic images containing multiple indoor scenes and their corresponding text descriptions, and preprocessing the 360-degree panoramic images to form paired data.

[0011] Preferably, the spatial geometric parameters include at least one of size, structural layout, type and location of basic components.

[0012] Preferably, in step S2, the extraction of spatial depth and edge information includes: The depth information is obtained by rendering the three-dimensional spatial model; The three-dimensional spatial model is rendered using wireframe or normal rendering, and edge detection is performed on the rendering results to obtain its edge information.

[0013] Preferably, when generating the stylized panoramic image, classifier-free guidance technology is used to independently adjust the guidance scale of the stylized text description, the depth information, and the edge information.

[0014] Preferably, the depth information ControlNet and the edge information ControlNet are constructed and trained using the fine-tuned UNet structure of the large image model as the base network.

[0015] Preferably, the deformation processing of the geometry of the three-dimensional indoor space model includes: The depth information is converted into a height field, which drives the mesh vertices of the three-dimensional spatial model to shift along the normal direction; Based on the difference between the depth information of the stylized image and the depth information of the 3D spatial model itself, the vertices of the model mesh are nonlinearly offset.

[0016] This invention also provides a virtual three-dimensional space generation system based on a large model, comprising: The model fine-tuning module is used to construct a scene image training dataset and fine-tune the pre-trained large image model using a low-rank matrix adaptation method. The spatial generation and information extraction module is used to automatically generate a three-dimensional spatial model based on the input spatial geometric parameters, and extract the depth and edge information of the space. The conditional image generation module integrates a fine-tuned large image model, depth information ControlNet, and edge information ControlNet. It is used to receive stylized text descriptions, depth information, and edge information to generate stylized images. The rendering and deformation processing module is used to render the generated stylized image onto the surface of the 3D spatial model based on the depth displacement method, and to perform geometric deformation on the 3D spatial model based on the comparison results of the image depth information and the spatial depth information itself. The output module is used to output the final three-dimensional spatial model.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor performing the steps of the above-described method for generating virtual three-dimensional space based on a large model.

[0018] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for generating virtual three-dimensional space based on a large model.

[0019] The beneficial effects of this invention are: 1. This invention constructs a complete process of "parametric modeling - geometric information extraction - conditional image generation - geometric deformation," using text descriptions and geometric parameters as unified inputs to automatically output a 3D model that blends texture and geometry. This reduces reliance on extensive manual operations and allows users to control the generation process through text and parameters.

[0020] 2. This invention introduces depth and edge dual ControlNets in parallel to the fine-tuned large image model and generates it based on geometric information graphs. This allows the model to strictly align with the perspective, contour and structure of the input three-dimensional space while adhering to the text style description, thus solving the problem of geometric misalignment between the generated content and the target.

[0021] 3. This invention utilizes a depth-permutation-based rendering method to add high-frequency geometric details to the model surface using image depth information. Furthermore, by comparing the visual depth of the generated image with the original depth of the model and driving mesh vertex deformation, the macroscopic shape of the 3D model can adaptively adjust according to the cues of the stylized image, thus overcoming the shape limitations of parametric white models.

[0022] 4. This invention employs Low-Rank Matrix Adaptation (LoRA) to fine-tune large models, significantly reducing training costs. Simultaneously, the deformed model can re-enter the process for iterative optimization, ultimately ensuring consistency in style, texture, and geometry, thereby enhancing the system's robustness and the controllability of output quality. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the process for generating a virtual three-dimensional space based on a large model, according to an embodiment of the present invention. Figure 2 This is a block diagram of a virtual 3D space generation system based on a large model, according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0026] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] Example 1: like Figure 1 As shown in the embodiments of this specification, a method for generating a virtual three-dimensional space based on a large model is provided, specifically including the following steps: Step S1: Construct a training dataset for the task of spatial scene image rendering. In a preferred embodiment, this task is specifically a 360-degree panoramic image rendering task. The corresponding training data consists of collecting or generating 360-degree panoramic images containing various indoor scenes (such as living rooms, bedrooms, and offices) and their corresponding text descriptions (such as "a bright living room in a modern minimalist style"). To improve training effectiveness, the panoramic images can be preprocessed, including isometric cylindrical projection correction to unify the format, color normalization, and resolution normalization to a specific size. These image-text pairs constitute the paired data for fine-tuning the training.

[0028] Subsequently, the pre-trained existing large image model (such as StableDiffusion) is fine-tuned using the Low-Rank Matrix Adaptation (LoRA) method. Specifically, a trainable low-rank decomposition matrix is ​​injected into the Transformer block or attention module of the large image model, while most of the original parameters of the model are frozen. The rank r of the low-rank matrix is ​​typically set to a small value, for example, in the range of 4 to 64, to achieve efficient and effective parameter updates. The fine-tuning process uses the standard noise prediction loss function of the diffusion model, and a perceptual loss can be additionally introduced to improve the detail quality of the generated images. Through fine-tuning training, the fine-tuned large image model is obtained.

[0029] Step S2: Obtain 3D interior space parameters. These can be obtained through user input or automatically by parsing BIM files. The parameters include floor plan vertex information, wall heights, number of doors and windows, etc. Based on these parameters, a corresponding 3D interior space mesh model is automatically constructed using a 3D modeling engine or procedural generation algorithm. Next, depth and edge information are extracted from this model.

[0030] One implementation method: Extracting depth information involves rendering the 3D spatial model using a rendering engine to obtain its depth buffer map, which contains depth information about the distance from each point in the scene to the camera.

[0031] Edge information extraction involves performing wireframe rendering or normal rendering on the 3D spatial model to obtain its structural outline or normal map. Subsequently, an edge detection algorithm (such as the Canny algorithm) is applied to the rendering result to obtain the spatial edge information.

[0032] Step S3: Introduce depth information ControlNet and edge information ControlNet into the fine-tuned large image model obtained in Step S1. Both ControlNets are constructed using the UNet encoder structure of the fine-tuned large image model as the base network. Specifically, a copy of the UNet encoder is made as the backbone of each ControlNet. During training, all parameters of the main large image model are fixed, and only the parameters of these two ControlNet copies and their "zero-initialized" convolutional layers connected to the main UNet are trained. Independent training is performed using a dataset containing panorama-depth-edge map pairs to learn how to map geometric conditions to generating control signals.

[0033] The system receives stylized text descriptions from the user and combines them with the depth and edge information extracted in step S2 as conditional inputs. The fine-tuned large-scale image model is generated under the guidance of dual ControlNets. To finely control the balance between style, 3D spatial perception, and contour accuracy in the generated result, this method employs classifier-free guidance (CFG) technology and independently adjusts the guidance scales for text, depth, and edge conditions. For example, the edge condition scale can be increased to ensure accurate door and window contours, while the text condition scale can be appropriately increased to stylize elements. Finally, the model outputs a stylized panoramic image that geometrically corresponds precisely to the 3D interior space model in step S2.

[0034] Step S4: Render the stylized panoramic image generated in step S3 onto the surface of the 3D indoor space model, specifically including: assigning the stylized panoramic image as a texture map to the 3D model.

[0035] Step S5: Perform depth detection on the stylized panoramic image generated in step S3. Specifically, a pre-trained monocular depth estimation neural network model (such as MiDaS) can be used to process the image and obtain the visual depth information contained in the stylized image.

[0036] Next, the obtained visual depth information is compared with the depth information of the 3D indoor space model itself from step S2, and the difference between the two at corresponding spatial locations is calculated. Based on this difference, the geometry of the 3D indoor space model undergoes corresponding non-uniform deformation processing. Specifically, this includes: Displacement deformation: This transforms depth difference information into a displacement field, driving the vertices of the 3D model mesh to undergo nonlinear displacement. For example, if the image depth indicates that a certain area should have a protruding column or a recessed niche, but the original model has a flat surface at that location, then the vertices in the corresponding area will move in the positive or negative direction along the normal.

[0037] Applying constraints: During the deformation process, deformation constraints are introduced to ensure the reasonableness of the results, including: collision detection to prevent self-intersection of the wall after deformation, local deformation protection for areas such as doors and windows that need to maintain functionality, and setting a global deformation amplitude threshold to prevent excessive distortion.

[0038] This step enables the geometry in three-dimensional space to adaptively evolve toward the more complex structures implied by the stylized image, thereby achieving the generation of "multi-shapes".

[0039] Step S6: After the above steps, the final 3D interior space model is generated and output. This model integrates the multi-style panoramic images generated in Step S3 as textures and has geometrically enhanced shapes obtained after deformation in Step S5. The final output asset is a complete 3D model that can be directly used in game engines, virtual reality, or further design, achieving the goal of intelligent 3D interior space generation with multiple styles and shapes.

[0040] In a preferred embodiment, between steps S5 and S6, an iterative optimization step may also be included: taking the deformed 3D spatial model as a new input, repeating part of the process from steps S2 to S5 (such as re-extracting depth / edge information, regenerating and deforming), performing multiple rounds of iteration until the difference between the image depth and the model depth is lower than a preset threshold, thereby achieving a higher degree of unification and optimization of style, texture and geometry.

[0041] Example 2: This embodiment provides a virtual 3D space generation system based on a large model, such as... Figure 2 As shown, it includes: The model fine-tuning module is used to build or access a training dataset containing pairings of scene images and text descriptions, and to efficiently fine-tune the parameters of the basic image generation model using the low-rank matrix adaptation (LoRA) technique, thereby outputting a fine-tuned model adapted to specific spatial scenes (such as indoor panoramas).

[0042] The spatial generation and information extraction module receives user-input parametric descriptions of 3D space and automatically constructs corresponding 3D mesh models (white models) using a built-in rule engine or generation algorithm. Simultaneously, this module integrates rendering and image processing units, enabling real-time extraction and output of depth and edge information maps from the generated white models.

[0043] The conditional image generation module integrates the fine-tuned large image model and connects in parallel with depth and edge information ControlNet. It receives stylized text descriptions from the user and depth and edge condition maps from the previous module. Through a multi-condition guided generation process, it outputs a stylized panoramic image that matches the text stylistically and is geometrically aligned strictly with the input conditions.

[0044] The rendering and deformation processing module utilizes depth displacement technology to use the generated panoramic image as a texture and, combined with its depth information, renders an effect with geometric details on the surface of the 3D model. Next, the module analyzes the depth information of the stylized image through the depth estimation submodule, compares it with the depth information of the original 3D model, and drives the deformation of the model's mesh vertices based on the differences. This enhances visual realism while increasing the geometric complexity of the 3D model itself.

[0045] The output module is used to encapsulate the processed 3D model data and output it in an asset file format that can be used in mainstream 3D engines, completing the automated generation process from parametric description to high-fidelity, stylized 3D space.

[0046] Example 3: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the virtual three-dimensional space generation method based on a large model as provided in Embodiment 1.

[0047] Example 4: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the virtual three-dimensional space generation method based on a large model as provided in Embodiment 1.

[0048] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0049] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0050] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0051] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0052] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the various method embodiments described above.

[0053] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating virtual three-dimensional space based on a large model, characterized in that, The method includes the following steps: Step S1: Construct training data for the spatial scene image rendering task, and use the low-rank matrix adaptation method to fine-tune the pre-trained existing large image model to obtain the fine-tuned large image model. Step S2: Based on the input three-dimensional indoor space parameters, automatically generate the corresponding three-dimensional indoor space model, and extract the depth and edge information of the space from the model; Step S3: Introduce depth information ControlNet and edge information ControlNet into the fine-tuned image large model. Using the depth information and edge information extracted in step S2 as conditional inputs, and combined with the input stylized text description information, the fine-tuned image large model generates a stylized panoramic image corresponding to the three-dimensional indoor space model. Step S4: Based on the depth permutation method, render the generated stylized panoramic image onto the surface of the 3D indoor space model generated in step S2; Step S5: Perform depth detection on the stylized panoramic image to obtain image depth information, and combine it with the depth information of the three-dimensional indoor space model itself to perform corresponding deformation processing on the geometry of the three-dimensional indoor space model in order to improve its shape complexity. Step S6: Generate and output a multi-style, multi-shape 3D interior space model that integrates stylized panoramic image textures and deformed geometric shapes.

2. The virtual three-dimensional space generation method based on a large model according to claim 1, characterized in that, The spatial scene image rendering task is a 360-degree panoramic image rendering task. The construction of training data includes collecting or generating 360-degree panoramic images containing multiple indoor scenes and their corresponding text descriptions, and preprocessing the 360-degree panoramic images to form paired data.

3. The virtual three-dimensional space generation method based on a large model according to claim 1, characterized in that, The spatial geometric parameters include at least one of the following: size, structural layout, type and location of basic components.

4. The method for generating virtual three-dimensional space based on a large model according to claim 1, characterized in that, In step S2, the extraction of depth and edge information of the space includes: The depth information is obtained by rendering the three-dimensional spatial model; The three-dimensional spatial model is rendered using wireframe or normal rendering, and edge detection is performed on the rendering results to obtain its edge information.

5. The method for generating virtual three-dimensional space based on a large model according to claim 1, characterized in that, When generating the stylized panoramic image, classifier-free guidance technology is used to independently adjust the guidance scale of the stylized text description, the depth information, and the edge information.

6. The method for generating virtual three-dimensional space based on a large model according to claim 1, characterized in that, The depth information ControlNet and edge information ControlNet are constructed and trained using the fine-tuned UNet structure of the large image model as the base network.

7. The method for generating virtual three-dimensional space based on a large model according to claim 1, characterized in that, The deformation processing of the geometry of the three-dimensional indoor space model includes: The depth information is converted into a height field, which drives the mesh vertices of the three-dimensional spatial model to shift along the normal direction; Based on the difference between the depth information of the stylized image and the depth information of the 3D spatial model itself, the vertices of the model mesh are nonlinearly offset.

8. A virtual three-dimensional space generation system based on a large model, characterized in that, include: The model fine-tuning module is used to construct a scene image training dataset and fine-tune the pre-trained large image model using a low-rank matrix adaptation method. The spatial generation and information extraction module is used to automatically generate a three-dimensional spatial model based on the input spatial geometric parameters, and extract the depth and edge information of the space. The conditional image generation module integrates a fine-tuned large image model, depth information ControlNet, and edge information ControlNet. It is used to receive stylized text descriptions, depth information, and edge information to generate stylized images. The rendering and deformation processing module is used to render the generated stylized image onto the surface of the 3D spatial model based on the depth displacement method, and to perform geometric deformation on the 3D spatial model based on the comparison results of the image depth information and the spatial depth information itself. The output module is used to output the final three-dimensional spatial model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the virtual three-dimensional space generation method based on a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the virtual three-dimensional space generation method based on a large model as described in any one of claims 1 to 7.

Images