3D scene generation method and device guided by structure control information, equipment and medium

Abstract

The application provides a 3D scene generation method and device based on structure control information guidance, equipment and medium, belonging to the technical field of image processing, based on the scene generation requirement, the 3D building model of target building structure is constructed;Based on the 3D building model, the wireframe structure diagram and the depth map of multiple perspectives are generated;The wireframe structure diagram and the depth map of multiple perspectives are used as structure control information, and the pre-trained image generation model is guided to generate multiple target style images under different perspectives in combination with target 3D scene text prompt words;The generated multiple target style images under different perspectives are used as texture information, and are mapped to the 3D building model to obtain a target 3D scene model.The application ensures the consistency of scene style according to the structured controllable 3D scene model.Meanwhile, the target 3D scene text prompt word is guided to generate, and the scene homogenization problem caused by asset reuse in the existing digital asset library technology is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, device, and medium for generating 3D scenes guided by structural control information. Background Technology

[0002] With the rapid development of computer technology, virtual shooting and digital scene generation technologies are gradually becoming key supports for the creation of ancient-style films and television dramas. These technologies integrate high-precision modeling, real-time rendering, and realistic lighting simulation capabilities, successfully achieving the digital reconstruction of historical scenes and fictional ancient-style worlds, providing creators with greater freedom of visual expression and production efficiency.

[0003] Existing technologies generally utilize laser scanning as a high-precision modeling method. Based on laser scanning and photogrammetry, high-fidelity architectural data is acquired, and standardized, reusable component resources (such as eaves, brackets, archways, carved window lattices, etc.) are constructed to form a "digital building block" asset library. These assets can be combined, copied, scaled, and other operations to quickly construct 3D scenes of different eras and styles.

[0004] However, this method has the following problems: 1. In large buildings or complex ruins, a complete scan often takes several days or even weeks, resulting in high time costs and a heavy data processing burden. In addition, due to the massive volume of point cloud data, high demands are placed on the performance of subsequent processing equipment and software, and data compression and structural simplification are still required in the actual production process.

[0005] 2. The initial stage of digital asset modeling relies heavily on manual intervention, including retopological modeling of components, UV unwrapping, and material partitioning, resulting in high production cycles and labor costs. Secondly, the splicing of modular assets can lead to insufficient stylistic consistency; for example, mixing elements of Song Dynasty gardens with components of Ming Dynasty palaces can easily cause visual confusion without strict guidelines. Furthermore, digital asset library technology is limited by pre-set models, and asset reuse can easily lead to scene homogenization, resulting in scenes in different film and television works appearing too similar, lacking innovation and unique style, and failing to meet complex or personalized design needs. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method, apparatus, device and medium for generating 3D scenes guided by structural control information, so as to ensure the consistency of scene style and reduce scene homogenization.

[0007] Firstly, a method for generating 3D scenes guided by structural control information is provided, including:

[0008] Build a 3D architectural model of the target building structure based on scene generation requirements;

[0009] Generate multi-view wireframe structure diagrams and depth maps based on 3D building models;

[0010] Using wireframe structure diagrams and depth maps from multiple perspectives as structural control information, and combined with text prompts for the target 3D scene, a pre-trained image generation model is used to generate multiple target-style images from different perspectives.

[0011] Multiple target style images generated from different perspectives are used as texture information and mapped onto a 3D building model to obtain the target 3D scene model.

[0012] Optionally, constructing a 3D architectural model of the target building structure based on scene generation requirements includes:

[0013] Determine the target building structure based on the scene generation requirements;

[0014] Acquire a multi-view RGB image sequence of the target building that conforms to the target building structure;

[0015] Using the VGGT model, inference is performed on multi-view RGB image sequences to obtain point cloud data of the target 3D scene;

[0016] The point cloud data of the target 3D scene is preprocessed to obtain a 3D building model that conforms to the target building structure.

[0017] Optionally, preprocessing the point cloud data of the target 3D scene to obtain a 3D building model that conforms to the target building structure includes:

[0018] Correcting structural defects in the initial point cloud model;

[0019] Using a preset lightweighting tool, the corrected point cloud model is lightweighted to obtain a 3D building model that conforms to the target building structure.

[0020] Optionally, the image generation model includes a text encoder, a variational autoencoder, and a noise prediction network U-Net. The training process of the image generation model includes:

[0021] Obtain a training dataset of 3D scene images;

[0022] Text annotation is performed on each 3D scene image in the 3D scene image training dataset to obtain text description labels;

[0023] The 3D scene image training dataset and its corresponding text description labels are input into the initial image generation model for iterative training to update the parameters of the initial 3D scene generation model and obtain the optimal 3D scene generation model. During training, a low-rank adapter is added to the attention layer of the noise prediction network U-Net to update the parameters of the initial 3D scene generation model through a low-rank adaptation strategy.

[0024] Optionally, using multi-view wireframe structure diagrams and depth maps as structural control information, and combining them with text prompts for the target 3D scene to guide a pre-trained image generation model to generate multiple target-style images from different perspectives, including:

[0025] A text encoder based on an image generation model extracts text vector representations of text prompts in a target 3D scene;

[0026] Feature maps of wireframe structure maps and depth maps are extracted based on a pre-trained ControlNet network model;

[0027] The feature maps of the wireframe structure map and depth map, as well as the text vector representation of the text prompts of the target 3D scene, are injected into the noise prediction network U-Net of the image generation model to guide the randomly generated feature image to be denoised to generate the target style image in the latent space.

[0028] The variational autoencoder decodes the target style image in the latent space into a real target style image.

[0029] Optionally, the ControlNet network model's network structure consists of a downsampling portion of the noise prediction network U-Net and zero-convolutional layers; guiding the randomly generated feature image to be denoised to generate the target style image in the latent space includes:

[0030] In the downsampling part of the noise prediction network U-Net, the feature maps of the wireframe structure map and depth map, the feature map to be denoised, and the text vector representation of the text prompt words of the target 3D scene are fused for denoising.

[0031] After fusion and denoising processing for a preset number of time steps, the fused image output from the last layer of the downsampling part of the noise prediction network U-Net is upsampled and restored to obtain the target style image in the latent space.

[0032] Optionally, the downsampling fusion process is as follows:

[0033] The feature images to be denoised from each downsampled layer of the noise prediction network U-Net are fused with the feature maps of the wireframe structure map and depth map extracted from the same layer in the ControlNet network model to obtain a fused image.

[0034] Text vector representations of target 3D scene text prompts are injected into the fused image; wherein, the denoised feature image of the first time step of the noise prediction network U-Net is randomly generated, and the fused image after fusion processing at each time step is used as the denoised feature image of the next time step.

[0035] Secondly, a 3D scene generation device guided by structural control information is provided, comprising:

[0036] Building units are used to construct 3D architectural models of target building structures based on scene generation requirements.

[0037] The generation unit is used to generate multi-view wireframe structure diagrams and depth maps based on 3D building models;

[0038] The guided generation unit is used to guide the pre-trained image generation model to generate multiple target-style images from different perspectives by using wireframe structure diagrams and depth maps from multiple views as structural control information, combined with text prompts for the target 3D scene.

[0039] The mapping unit is used to map multiple generated target style images from different perspectives as texture information onto a 3D building model to obtain the target 3D scene model.

[0040] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0041] The memory is used to store computer programs;

[0042] When the processor executes the program stored in the memory, it implements the method described in the first aspect.

[0043] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements any of the methods described in the first aspect.

[0044] This invention provides a method, apparatus, device, and medium for generating 3D scenes guided by structural control information. It constructs a 3D architectural model of the target building structure based on scene generation requirements; generates multi-view wireframe structure diagrams and depth maps based on the 3D architectural model; uses the multi-view wireframe structure diagrams and depth maps as structural control information, and combines them with text prompts for the target 3D scene to guide a pre-trained image generation model to generate multiple target style images from different perspectives; and maps these generated target style images from different perspectives onto the 3D architectural model as texture information to obtain the target 3D scene model. This invention, through the full-process embedding of structural control information, breaks through the geometric constraints of traditional generation models, achieving structured controllability of 3D scenes, and thus ensuring the consistency of scene style based on the structured and controllable generated 3D scene model. Simultaneously, through text prompts for the target 3D scene, the scene style and element combinations can be freely adjusted according to creative needs, breaking through the limitations of reality and preset templates. This effectively solves the problem of scene homogenization caused by asset reuse in existing digital asset library technologies.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart of a 3D scene generation method guided by structure control information provided in an embodiment of the present invention is shown;

[0048] Figure 2 The figure shows the effect of using VGGT for three-dimensional reconstruction provided by an embodiment of the present invention;

[0049] Figure 3 This diagram illustrates the structure of ControlNet operating on Stable Diffusion according to an embodiment of the present invention.

[0050] Figure 4 An example image of a target style image generated based on structural control information and text prompts, as provided in an embodiment of the present invention, is shown.

[0051] Figure 5 This diagram illustrates the overall process of the 3D ancient style scene generation method provided in this embodiment of the invention.

[0052] Figure 6 This diagram illustrates the structure of a 3D scene generation device guided by structural control information, as provided in an embodiment of the present invention.

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0055] Considering the following problems with existing technologies: 1. In large buildings or complex sites, complete scanning often takes several days or even weeks, resulting in high time costs and heavy data processing burdens. In addition, due to the massive volume of point cloud data, high demands are placed on the performance of subsequent processing equipment and software, and data compression and structural simplification are still required in the actual production process.

[0056] 2. The initial stage of digital asset modeling relies heavily on manual intervention, including retopological modeling of components, UV unwrapping, and material partitioning, resulting in high production cycles and labor costs. Secondly, the splicing of modular assets can lead to insufficient stylistic consistency; for example, mixing elements of Song Dynasty gardens with components of Ming Dynasty palaces can easily cause visual confusion without strict guidelines. Furthermore, digital asset library technology is limited by pre-set models, and asset reuse can easily lead to scene homogenization, resulting in scenes in different film and television works appearing too similar, lacking innovation and unique style, and failing to meet complex or personalized design needs.

[0057] Based on this, embodiments of the present invention provide a method and apparatus for generating 3D scenes guided by structural control information. The present invention combines text and structural control, which can not only be applied to the generation of 3D scenes in film and games, but also extended to the fields of virtual restoration of cultural heritage, education and popular science, and cultural tourism promotion. For example, virtual restoration models of the Mogao Grottoes in Dunhuang in different historical periods can be generated through text commands, which has both academic value and artistic expression.

[0058] The following is a description through examples.

[0059] This invention provides a method for generating 3D scenes guided by structural control information, such as... Figure 1 As shown, it includes the following steps:

[0060] Step S101: Construct a 3D building model of the target building structure based on the scene generation requirements.

[0061] Step S102: Generate multi-view wireframe structure diagrams and depth maps based on the 3D building model.

[0062] In one feasible implementation, the process of generating the wireframe structure diagram and the depth map includes the following steps:

[0063] Step S102A: Camera Settings

[0064] Configure wireframe materials (such as the aiWireframe material in the Arnold renderer) for the 3D architectural model generated in the above steps. Next, establish a multi-view camera layout around the model (including front, rear, left, right, top, and bottom views), and ensure that all cameras are configured in orthographic projection mode. Adjust the position and orientation of each camera to fully cover all key viewpoints of the model.

[0065] Step S102B: Wireframe Rendering

[0066] Switch to the Arnold renderer and activate the aiWireframe material in the Arnold render settings. Set the output format to PNG and the resolution to 512x512 pixels. Render each viewpoint sequentially according to the pre-defined camera sequence and save the generated wireframe images.

[0067] Step S102C: Depth Map Rendering

[0068] In the Arnold renderer's AOVs (Arbitrary Output Variables) settings, add a Z-channel depth channel and select 8-bit PNG as the output format. Based on the dimensions of the 3D architectural model, appropriately set the Near Clip Plane and Far Clip Plane to ensure that the 3D architectural model is not clipped during rendering while achieving optimal depth value accuracy. Perform a batch rendering process, generating depth maps for each viewpoint, and normalize the exported depth maps to a single-channel grayscale image within the range [0, 1]. This step is crucial for subsequent processing or analysis of the model's depth information.

[0069] Step S103: Using multi-view wireframe structure diagrams and depth maps as structural control information, and combining them with text prompts for the target 3D scene, guide the pre-trained image generation model to generate multiple target style images from different perspectives.

[0070] Step S104: The generated target style images from different perspectives are used as texture information and mapped onto the 3D building model to obtain the target 3D scene model.

[0071] In one feasible implementation, such as in Maya software, camera parameters can be set and UV mapping based on camera projection can be implemented via a Python script (MayaPython API).

[0072] The entire process can be briefly divided into the following steps:

[0073] Step S104A: Synchronize camera parameters;

[0074] Ensure that the camera parameters of the current model are consistent with those used when rendering the wireframe structure and depth map. Use the MayaPython API (such as the cmds module) to set parameters such as the camera rotation angle, clipping plane distance, and focal length to guarantee a consistent viewpoint.

[0075] Step S104B: Perform camera projection UV mapping;

[0076] Camera projection technology is used to generate UV layouts adapted to various camera viewpoints (U and V represent the horizontal and vertical directions in texture space). Specific operations include: creating a camera projection node, connecting the camera to the projection node, selecting the target mesh of the 3D building model (by meshing the 3D building model to obtain a 3D building mesh model), and projecting it to generate UV coordinates for each camera viewpoint.

[0077] This process supports looping and parameterized settings, making it easy to quickly generate multi-view UV maps for complex models.

[0078] Step S104C: Apply UDIM texture mapping technology;

[0079] Efficient mapping of multi-view textures is achieved using UDIM (U-Dimension) texture mapping technology. Each camera viewpoint corresponds to an independent UDIM region, and the corresponding texture image is named according to the UDIM number. By associating each camera viewpoint with its corresponding UDIM number, precise matching between 3D spatial structure and 2D image content is achieved.

[0080] Step S104D: Final rendering and quality optimization;

[0081] In the Arnold renderer, by combining finely adjusted lighting and shadow settings, high-quality 3D models of ancient-style scenes are rendered. This not only ensures the accuracy of the model's three-dimensional structure but also endows it with unique ancient-style aesthetic qualities, presenting a work that combines artistic beauty and technical precision.

[0082] This invention, through structural control information and text prompts for the target 3D scene, guides the generation of a target-style image. The scene style and element combinations can be freely adjusted according to creative needs, such as merging Tang Dynasty palaces with fantasy elements, or generating a specific dynasty's cityscape based on historical research, breaking through the limitations of reality and preset templates. This avoids the problems of "overly realistic" laser scanning or "homogenized" digital resource libraries.

[0083] Based on the above embodiments, constructing a 3D building model of the target building structure based on scene generation requirements includes:

[0084] Step S101A: Determine the target building structure based on the requirements generated in the scenario.

[0085] In this step, the scene generation requirements include architectural style and scene style, such as Tang Dynasty architectural style, Song Dynasty architectural style, etc., and scene style such as fantasy scene, etc.

[0086] Step S101B: Obtain a multi-view RGB image sequence of the target building that conforms to the target building structure.

[0087] In this step, a multi-view RGB image sequence of Tang Dynasty-style architecture can be acquired using an image acquisition device, such as a high-precision camera. This multi-view RGB image sequence is obtained by the same fixed camera from different angles.

[0088] Step S101C: Using the VGGT (Visual Geometry Grounded Transformer, a visual model based on a pure feedforward Transformer architecture) model, inference is performed on the multi-view RGB image sequence to obtain the point cloud data of the target 3D scene.

[0089] In this step, VGGT is a general-purpose 3D vision model based on a pure feedforward Transformer architecture. It can directly infer core geometric information such as camera parameters, depth maps, point clouds, and 3D point trajectories. Moreover, all of this is completed in a single forward inference, requiring no post-processing optimization. Compared to traditional 3D reconstruction methods that rely on bundle adjustment and other optimization techniques, which are computationally complex, time-consuming, and require repeated iterations, VGGT directly eliminates this cumbersome process. Employing a pure feedforward design, it handles all geometric inference tasks in a single forward propagation, significantly improving modeling efficiency.

[0090] After obtaining the RGB image sequence from multiple perspectives, the RGB image sequence is first subjected to uniform preprocessing and alignment.

[0091] Then, it is input into the VGGT model to obtain the camera intrinsic and extrinsic parameters, depth map, and dense point cloud of the scene for each image, such as... Figure 2 As shown, the dense point cloud of the scene is used as the initial point cloud model of the scene.

[0092] Step S101D: Preprocess the point cloud data of the target 3D scene to obtain a 3D building model that conforms to the target building structure.

[0093] In one feasible implementation, preprocessing the initial point cloud model to obtain a 3D building model that conforms to the target building structure includes:

[0094] Step S101D1: Correct the structural defects in the initial point cloud model.

[0095] In embodiments of the present invention, structural defects include topological errors (such as non-manifold geometry, overlapping surfaces), missing parts, or unreasonable structures.

[0096] In one example, for overlapping surfaces or unreasonable structures, the editing tools of the modeling software can be used to adjust the position of vertices, edges and faces to correct the shape of the model and make it more in line with the structural features of a real 3D scene.

[0097] For complex structural elements, such as brackets and carvings, subdivision surface techniques can be used to increase the detail and precision of the model. For example, Autodesk Maya and 3ds Max modeling software both offer the "Subdivision Surfaces" option, which allows users to refine the model surface by increasing the number of polygons, thereby obtaining a smoother result.

[0098] Step S101D2: Using the preset lightweight tool, perform lightweight processing on the corrected point cloud model to obtain a 3D building model that conforms to the target building structure.

[0099] In one example, the "Mesh > Reduce" function of Autodesk Maya modeling software is used to automatically reduce the size of the model.

[0100] In this step, the adjusted model is optimized, for example, by reducing the number of faces and vertices. While ensuring the clarity of the model's geometry, the complexity of the model is reduced, making it lightweight to meet the needs of real-time rendering or further processing.

[0101] Based on the above embodiments, the image generation model can adopt the Stable Diffusion model, including a text encoder, a variational autoencoder (VEA), and a noise prediction network U-Net.

[0102] The text encoder employs the CLIP model, which converts the input text information into a vector representation (embedding). This embedding vector is used to capture the semantic information of the text and guide the U-Net network during image generation.

[0103] In Stable Diffusion, the Visual Image Encoding (VAE) is primarily used for image encoding and decoding. The encoder portion of the VAE compresses the original image into a lower-dimensional representation (latent space representation), while the decoder portion reconstructs the image from this low-dimensional representation. In this embodiment of the invention, the VAE is used to map images to the latent space of the noise prediction network U-Net and back to the image from the latent space, rather than being used directly to generate images.

[0104] The U-Net noise prediction network is a core component of the Stable Diffusion model, responsible for learning how to progressively remove noise added to an image, thereby gradually generating the target image. The U-Net architecture includes multiple downsampling and upsampling paths, allowing the model to capture features at different scales. During training, the model receives noisy images and corresponding clean image pairs as input, learning to predict the noise added to the images. During generation, noise is progressively reduced through a reverse process, ultimately generating a sharp image.

[0105] Based on the network structure of this image generation model, the training process of this image generation model includes:

[0106] Step 1: Obtain the 3D scene image training dataset.

[0107] In this step, images can be obtained through various channels, such as collecting real 3D scene images from historical documents, museum collections, and photographs of ancient buildings; or collecting fictional 3D scene images from art works such as films, games, and illustrations.

[0108] The system also preprocesses the acquired 3D scene images, such as removing blurry, duplicate, or unacceptable images, and performing uniform format conversion and size adjustment (e.g., 512×512 pixels).

[0109] Step 2: Text annotation is performed on each 3D scene image in the 3D scene image training dataset to obtain text description labels.

[0110] In this step, an accurate and detailed text description is written for each image. The description includes information such as the era to which the scene belongs, regional style, architectural structure, and atmosphere of the scene.

[0111] In one example, combining manual annotation with AI-assisted annotation can improve annotation efficiency and accuracy. For instance, the open-source Florence-2 model can be used to help generate basic descriptions.

[0112] Alternatively, use tools like Label Studio for manual annotation, adding "Ancientry" as the first label. Organize the images and corresponding text descriptions into a CSV or JSON dataset for later use.

[0113] Step 3: Input the 3D scene image training dataset and its corresponding text description labels into the initial image generation model for iterative training, so as to update the parameters of the initial 3D scene generation model and obtain the optimal 3D scene generation model.

[0114] Specifically, a low-rank adapter is added to the attention layer of the noise prediction network U-Net during training, and the parameters of the initial 3D scene generation model are updated through a low-rank adaptation strategy.

[0115] The Low-Rank Adaptation (LoRA) strategy updates the original weight matrix W of the 3D scene generation model using the following formula:

[0116] (1);

[0117] In the formula, W represents the parameters of the initial 3D scene generation model. These are the final initial 3D scene generation model parameters after being influenced by the LoRA model. A and B are low-rank matrices, and BA represents the parameters of the LoRA model. r is the parameter of the low-rank matrix, controlling the rank of the matrix decomposition; α is the scaling factor, balancing the contributions of the original weights and the LoRA weights. α / r, as an overall scaling factor, determines the degree to which the LoRA module modifies the original weights.

[0118] When α = r, the scaling factor is 1, and the LoRA weights are directly superimposed on the original weights. Increasing α enhances the influence of the LoRA module, allowing the model to adapt to new data more quickly, but may lead to overfitting. Decreasing α weakens the influence of the LoRA module, preserving more knowledge from the original model. By setting the LoRA parameters appropriately, near-full parameter fine-tuning can be achieved while maintaining efficient training.

[0119] Therefore, at the beginning of training, the Stable Diffusion model is first fine-tuned by configuring LoRA parameters, low-rank matrix dimension r=8, scaling factor α = 8 to balance the original weights and LoRA weights, and training parameters such as training epochs, batch size, and learning rate.

[0120] The constructed 3D scene dataset is divided into a training set and a validation set in an 8:2 ratio.

[0121] Load the core components of the Stable Diffusion model: the image encoder VAE, the text encoder CLIP, and the noise prediction network U-Net, where only the attention layer (Q / K / V projection matrix) of U-Net is added with a low-rank adapter, and other layers are frozen to reduce the number of trainable parameters.

[0122] The image and text description labels from the 3D scene image training dataset are input into the Stable Diffusion model. The LoRA module is used to adjust the parameters of specific layers of the model, enabling the Stable Diffusion model to learn the features and styles of the 3D scene.

[0123] Every certain number of steps, the quality of the images generated by the model is evaluated on the validation set. Specifically, the quality of the generated images can be evaluated using the mean squared error loss function and the similarity index between the generated images and the real images.

[0124] Then, based on the evaluation results, backpropagate to update the parameters of the low-rank matrices A and B, while keeping the remaining weights of the pre-trained model unchanged. Checkpoints should be saved periodically during the training process.

[0125] Finally, by comparing the effects of different checkpoints, underfitting and overfitting models are eliminated, and the LoRA weight file with the best visual quality is selected. This is then combined with the basic Stable Diffusion model to obtain the image generation model.

[0126] Based on the above embodiments, using multi-view wireframe structure diagrams and depth maps as structural control information, and combining them with target 3D scene text prompts to guide a pre-trained image generation model to generate multiple target style images from different perspectives, including:

[0127] Step S103A: The text encoder based on the image generation model extracts the text vector representation of the target 3D scene text prompts.

[0128] Step S103B: Extract feature maps of wireframe structure map and depth map based on pre-trained ControlNet network model.

[0129] In this embodiment of the invention, the ControlNet network model is a neural network structure used to enhance Stable Diffusion. This ControlNet network model constructs a dual-control-branch guided Stable Diffusion model. One ControlNet branch is dedicated to processing wireframes and extracting contour structures; the other ControlNet branch processes depth maps and provides spatial and perspective guidance. The two are fused into each layer of the noise prediction network U-Net by adding residuals, thereby strengthening structural consistency.

[0130] Step S103C: Inject the feature maps of the wireframe structure map and depth map, as well as the text vector representation of the target 3D scene text prompts, into the noise prediction network U-Net of the image generation model to guide the randomly generated feature image to be denoised to generate the target style image in the latent space.

[0131] In this embodiment of the invention, the network structure of the ControlNet network model consists of a downsampling portion of the noise prediction network U-Net and zero convolutional layers; as shown... Figure 3 As shown, the encoding part of the ControlNet network model has the same structure as the downsampling part of the noise prediction network U-Net. This design ensures that the original model's generative ability is not compromised, while also allowing for the flexible introduction of new spatial constraints. This design enables ControlNet to efficiently incorporate subsequent structural information constraints, thereby achieving fine control over the image generation process while maintaining the stability of the original Stable Diffusion model.

[0132] Based on this, guiding the generation of target style images in the latent space from randomly generated feature images to be denoised includes:

[0133] Step S103C1: In the downsampling part of the noise prediction network U-Net, the feature maps of the wireframe structure map and depth map, the feature map to be denoised, and the text vector representation of the target 3D scene text prompt words are fused and denoised.

[0134] In one specific implementation, the downsampling fusion process is as follows:

[0135] Step A: Fuse the feature images to be denoised from each downsampled layer of the U-Net noise prediction network with the feature maps of the wireframe structure map and depth map extracted from the same layer in the ControlNet network model to obtain a fused image.

[0136] In the U-Net noise prediction network, the initial time step of the noise prediction network is randomly generated, and the fused image after each time step is used as the noise prediction feature image for the next time step.

[0137] In this step, the feature map tensors of the wireframe structure map and the depth map are fused element-wise by addition, and the fusion formula is as follows:

[0138] (2);

[0139] in, The noise prediction network U-Net has a feature map to be denoised. A feature map of a wireframe structure diagram; Feature maps of the depth map; weights As weight, by adjusting The value of controls the influence strength of the ControlNet network model to ensure that the style of the image comes from the text prompts, while the structure comes from the structure control graph of the ControlNet network model.

[0140] Step B: Inject text vector representations of the target 3D scene text prompts into the fused image.

[0141] Based on structural control, the input target 3D scene text prompts play a decisive role, guiding the model to generate 3D scene content with classical charm.

[0142] Step S103C2: After fusion and denoising processing for a preset number of time steps, the fused image output from the last layer of the downsampling part of the noise prediction network U-Net is upsampled and restored to obtain the target style image in the latent space.

[0143] Step S103D: Decode the target style image in the latent space into a real target style image based on a variational autoencoder.

[0144] like Figure 4 As shown, based on the finely tuned Stable Diffusion model and ControlNet network model, and simultaneously inputting edge wireframe maps, depth condition maps, and text prompts, the target style image is obtained by multiple inferences through multiple perspectives of the multi-view paired edge wireframe structure map-depth map.

[0145] The core innovation of this invention lies in achieving structure guidance through the dual-branch structure (wireframe outline + depth perspective) of the ControlNet network model. Its residual fusion mechanism ensures that textual semantics dominate style generation while strictly adhering to 3D structural constraints. Finally, through camera projection UV mapping technology, multi-view style images are precisely texture-mapped according to structural coordinates, forming a 3D scene that combines geometric accuracy with cultural charm. This method, through the full-process embedding of structural information, breaks through the geometric constraints of traditional generative models, achieving structured and controllable generation of 3D scenes. Simultaneously, it effectively solves the scene homogenization problem caused by asset reuse in existing digital asset library technologies.

[0146] To make the method of this invention clearer, let's take generating an ancient Chinese style scene as an example, such as... Figure 5 As shown, a flowchart illustrating the process of generating this ancient-style scene is presented.

[0147] Based on the same inventive concept, embodiments of the present invention provide a 3D scene generation device guided by structural control information, such as... Figure 6 As shown, the device includes:

[0148] Construction unit 601 is used to construct a 3D building model of the target building structure based on scene generation requirements;

[0149] The generation unit 602 is used to generate multi-view wireframe structure diagrams and depth maps based on 3D building models;

[0150] The guided generation unit 603 is used to guide the pre-trained image generation model to generate multiple target style images from different perspectives by using wireframe structure diagrams and depth maps from multiple perspectives as structural control information, combined with text prompts for the target 3D scene.

[0151] The mapping unit 604 is used to map multiple generated target style images from different perspectives as texture information onto a 3D building model to obtain a target 3D scene model.

[0152] Based on the same technical concept, embodiments of the present invention also provide an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0153] Memory 703 is used to store computer programs;

[0154] The processor 701 is used to implement the steps of a 3D scene generation method guided by structural control information when executing a program stored in the memory 703.

[0155] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0156] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0157] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0158] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0159] The computer program product of the 3D scene generation method guided by structural control information provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0160] The device for generating 3D scenes guided by structural control information provided in this embodiment of the invention can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this embodiment of the invention are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0161] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0164] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0166] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating 3D scenes guided by structural control information, characterized in that, include: Build a 3D architectural model of the target building structure based on scene generation requirements; Based on the 3D building model, generate wireframe structure diagrams and depth maps from multiple perspectives; Using the multi-view wireframe structure diagram and depth map as structural control information, and combined with the target 3D scene text prompts, a pre-trained image generation model is used to generate multiple target style images from different perspectives; the image generation model includes a text encoder, a variational autoencoder, and a noise prediction network U-Net. Using the multi-view wireframe structure diagram and depth map as structural control information, and combined with the target 3D scene text prompts to guide the pre-trained image generation model, multiple target style images from different perspectives are generated, including: A text encoder based on a pre-trained image generation model extracts text vector representations of text prompts in a target 3D scene; Feature maps of the wireframe structure map and the depth map are extracted based on the pre-trained ControlNet network model; The feature maps of the wireframe structure map and the depth map, as well as the text vector representation of the target 3D scene text prompts, are injected into the noise prediction network U-Net of the image generation model to guide the randomly generated feature image to be denoised to generate the target style image in the latent space. The variational autoencoder decodes the target style image in the latent space into a real target style image; the training process of the image generation model includes: Obtain a training dataset of 3D scene images; Text annotation is performed on each 3D scene image in the 3D scene image training dataset to obtain text description labels; The 3D scene image training dataset and its corresponding text description labels are input into the initial image generation model for iterative training to update the parameters of the initial 3D scene generation model and obtain the optimal 3D scene generation model. During training, a low-rank adapter is added to the attention layer of the noise prediction network U-Net to update the parameters of the initial 3D scene generation model through a low-rank adaptation strategy. Multiple target style images generated from different perspectives are used as texture information and mapped onto the 3D building model to obtain the target 3D scene model.

2. The method according to claim 1, characterized in that, The 3D architectural model of the target building structure, which is constructed based on scene generation requirements, includes: Determine the target building structure based on the scene generation requirements; Obtain a multi-view RGB image sequence of the target building that conforms to the target building structure; Using the VGGT model, inference is performed on the multi-view RGB image sequence to obtain point cloud data of the target 3D scene; The point cloud data of the target 3D scene is preprocessed to obtain a 3D building model that conforms to the target building structure.

3. The method according to claim 2, characterized in that, The step of preprocessing the point cloud data of the target 3D scene to obtain a 3D building model that conforms to the target building structure includes: Correcting structural defects in the initial point cloud model; Using a preset lightweighting tool, the corrected point cloud model is lightweighted to obtain a 3D building model that conforms to the target building structure.

4. The method according to claim 1, characterized in that, The ControlNet network model's network structure consists of a downsampling portion of the noise prediction network U-Net and zero-convolutional layers; guiding the randomly generated feature image to be denoised to generate the target style image in the latent space includes: In the downsampling part of the noise prediction network U-Net, the feature maps of the wireframe structure map and depth map, the feature map to be denoised, and the text vector representation of the target 3D scene text prompt words are fused and denoised. After fusion and denoising processing for a preset number of time steps, the fused image output from the last layer of the downsampling part of the noise prediction network U-Net is upsampled and restored to obtain the target style image in the latent space.

5. The method according to claim 4, characterized in that, The downsampling fusion process is as follows: The feature images to be denoised from each downsampled layer of the noise prediction network U-Net are fused with the feature maps of the wireframe structure map and depth map extracted from the same layer in the ControlNet network model to obtain a fused image. The text vector representation of the target 3D scene text prompt is injected into the fused image; wherein, the denoising feature image of the first time step of the noise prediction network U-Net is randomly generated, and the fused image after fusion processing at each time step is used as the denoising feature image of the next time step.

6. A 3D scene generation device guided by structural control information, characterized in that, include: Building units are used to construct 3D architectural models of target building structures based on scene generation requirements. The generation unit is used to generate wireframe structure diagrams and depth maps from multiple perspectives based on the 3D building model; The guided generation unit is used to guide a pre-trained image generation model to generate multiple target style images from different perspectives, using the multi-view wireframe structure diagram and depth map as structural control information, combined with the target 3D scene text prompts; the image generation model includes a text encoder, a variational autoencoder, and a noise prediction network U-Net. Specifically used for: Using the multi-view wireframe structure diagram and depth map as structural control information, and combined with the target 3D scene text prompts to guide the pre-trained image generation model, multiple target style images from different perspectives are generated, including: A text encoder based on a pre-trained image generation model extracts text vector representations of text prompts in a target 3D scene; Feature maps of the wireframe structure map and the depth map are extracted based on the pre-trained ControlNet network model; The feature maps of the wireframe structure map and the depth map, as well as the text vector representation of the target 3D scene text prompts, are injected into the noise prediction network U-Net of the image generation model to guide the randomly generated feature image to be denoised to generate the target style image in the latent space. Based on the variational autoencoder, the target style image in the latent space is decoded into a real target style image; The training process of the image generation model includes: Obtain a training dataset of 3D scene images; Text annotation is performed on each 3D scene image in the 3D scene image training dataset to obtain text description labels; The 3D scene image training dataset and its corresponding text description labels are input into the initial image generation model for iterative training to update the parameters of the initial 3D scene generation model and obtain the optimal 3D scene generation model. During training, a low-rank adapter is added to the attention layer of the noise prediction network U-Net to update the parameters of the initial 3D scene generation model through a low-rank adaptation strategy. The mapping unit is used to map multiple generated target style images from different perspectives as texture information onto the 3D building model to obtain the target 3D scene model.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images