Three-dimensional scene generation method and device, storage medium, electronic device and product

Abstract

The application discloses a three-dimensional scene generation method and device, a storage medium, an electronic device and a product. The method comprises the following steps: receiving a text description or a reference image as input data; obtaining a panoramic image and corresponding depth information of a target scene based on the input data; inputting the panoramic image and the depth information into a trained feedforward neural network; and processing the input data by the feedforward neural network to generate and output a parameter set based on 3D Gaussian representation for representing a three-dimensional scene. The scheme directly predicts 3D Gaussian parameters through single forward propagation of scene information by the feedforward neural network, avoids multiple iterations required by a traditional optimization-based method, improves the generation efficiency of the three-dimensional scene, and can support real-time interactive applications.

Description

Technical Field

[0001] This invention relates to the field of 3D scene generation technology, and in particular to a 3D scene generation method, apparatus, storage medium, electronic device and product. Background Technology

[0002] In applications such as robot simulation training, virtual reality, and digital twins, there is a need to rapidly generate high-quality, high-fidelity 3D scenes for interactive use. Related technologies often employ optimization-based 3D scene generation methods, such as those based on neural radiation fields or 3D Gaussian representations. The general principle behind these methods is to iteratively optimize scene parameters to gradually approximate the geometry and appearance of the target scene.

[0003] However, since such methods rely on repeated iterative optimization processes, their scene generation speed is inevitably limited by the number of iterations, resulting in high computational costs and making it difficult to meet the generation efficiency requirements of dynamic interactive application scenarios with strict real-time or near-real-time requirements. Summary of the Invention

[0004] This invention provides a method, apparatus, storage medium, electronic device, and product for generating three-dimensional scenes. It can directly predict 3D Gaussian parameters by performing a single forward propagation of scene information through a feedforward neural network, avoiding the multiple iterations required by traditional optimization-based methods, thereby improving the efficiency of three-dimensional scene generation and supporting real-time interactive applications.

[0005] This invention provides the following solution: A first aspect of the present invention provides a method for generating a three-dimensional scene, comprising: Receive text descriptions or reference images as input data; Based on the input data, a panoramic view of the target scene and the corresponding depth information are obtained; The panoramic image and the depth information are input into a trained feedforward neural network; The feedforward neural network processes the panoramic image and the depth information to generate and output a set of parameters for characterizing the three-dimensional scene, the set of parameters being based on 3D Gaussian representation.

[0006] In one embodiment, obtaining a panoramic view of the target scene and its corresponding depth information based on the input data includes: Based on the input data, a panoramic image of the target scene is generated; Based on the panoramic image, a corresponding depth map is generated as the depth information.

[0007] In one embodiment, generating a panoramic view of the target scene based on the input data includes: When the input data is a text description, the panoramic image is generated based on the text description using a conditional image generation model. When the input data is a reference image, the reference image is expanded into the panoramic image using an image expansion model.

[0008] In one embodiment, the trained feedforward neural network is obtained via a self-supervised training method; the self-supervised training method includes: Semantic description information is obtained from the image rendered based on the parameter set generated by the feedforward neural network; Using the semantic description information as a condition, an image generation model is used to generate a teacher image; Based on the rendered image and the teacher image, calculate the perceptual difference loss; The feedforward neural network is trained based on the perceptual difference loss.

[0009] In one embodiment, calculating the perceptual difference loss based on the rendered image and the teacher image includes: Deep features of the rendered image and the teacher image are extracted using a pre-trained visual feature extraction network. The distance between the deep features is calculated as the perceptual difference loss.

[0010] In one embodiment, the self-supervised training method further includes: Based on the depth information, a geometric prior loss function is constructed. The geometric prior loss function is used to constrain the spatial position of the three-dimensional elements in the parameter set. The geometric prior loss function is constructed based on the deviation between the position information of the three-dimensional elements in the parameter set and the depth information. The feedforward neural network is trained based on the geometric prior loss function.

[0011] In one embodiment, the self-supervised training method further includes: The parameter set is rendered from multiple virtual perspectives to obtain rendering results from multiple perspectives; Based on the rendering results from the multiple perspectives and the normalized cross-correlation metric, the consistency loss is calculated. The feedforward neural network is trained based on the consistency loss.

[0012] In one embodiment, the parameters of each three-dimensional element in the parameter set include center position coordinates, opacity, scaling factor, and color representation coefficient.

[0013] In one embodiment, the parameters of the three-dimensional element further include rotation representation parameters; the scaling factor is processed by a nonlinear activation function to ensure that it is a positive value; and the rotation representation parameters are normalized.

[0014] In one embodiment, after generating and outputting the parameter set, the method further includes: The parameter set is exported as a standard format file for loading into the simulation engine for rendering.

[0015] A second aspect of the present invention provides a three-dimensional scene generation apparatus, comprising: The input acquisition module is used to receive text descriptions or reference images as input data; The information processing module is used to obtain a panoramic view of the target scene and the corresponding depth information based on the input data. The data input module is used to input the panoramic image and the depth information into a trained feedforward neural network; The scene generation module is used to process the panoramic image and the depth information by the feedforward neural network to generate and output a set of parameters for characterizing the three-dimensional scene, the set of parameters being based on 3D Gaussian representation.

[0016] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the three-dimensional scene generation method provided in any implementation of the first aspect.

[0017] A fourth aspect of the present invention provides an electronic device, comprising: One or more processors; and A memory associated with one or more processors, the memory being used to store program instructions, which, when read and executed by one or more processors, perform the steps of the three-dimensional scene generation method provided by any implementation of the first aspect above.

[0018] A fifth aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the three-dimensional scene generation method provided in any implementation of the first aspect.

[0019] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: To address the problem of slow scene generation speed and inability to meet real-time interactive requirements caused by the need for multiple iterations of scene parameter optimization in 3D simulation scene generation technology, this invention provides a 3D scene generation method. This method receives text descriptions or reference images as input data and uses a trained feedforward neural network to perform a single forward propagation on the input data, directly predicting the 3D Gaussian parameter set used to represent the 3D scene. This invention transforms the original iterative parameter optimization process into a deterministic mapping function represented by neural network parameters, thereby reducing the computational complexity of the generation process from being dependent on the number of iterations to constant level. Furthermore, this application supports multimodal inputs including semantics and images. Therefore, this invention effectively improves the speed of 3D scene generation and can effectively meet the needs of real-time interactive applications.

[0020] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the above drawings without creative effort.

[0022] Figure 1 A flowchart illustrating the steps of a three-dimensional scene generation method provided in an embodiment of the present invention.

[0023] Figure 2 This is a system architecture diagram of the three-dimensional scene generation method provided in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the self-supervised training mechanism in the three-dimensional scene generation method provided in the embodiments of the present invention.

[0025] Figure 4 This is a schematic block diagram of a three-dimensional scene generation device provided in an embodiment of the present invention.

[0026] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0028] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a” and “the” as used in the embodiments of this invention and the appended claims are also used to include the plural forms unless the context clearly indicates otherwise.

[0029] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0030] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0031] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0032] In the fields of computer vision, robot simulation, and AI-generated content technology, a common approach to achieving rapid, high-fidelity 3D virtual scene construction is to iteratively optimize the parameters of the scene representation model using a large amount of collected multi-view image data, based on differentiable rendering representations such as neural radiation fields or 3D Gaussian splashing. This approach is widely adopted primarily to obtain 3D scenes with high visual realism and the ability to synthesize high-quality new perspectives, meeting the needs of high-end visualization and some offline applications. However, when this approach is applied to technical environments requiring real-time or high-frequency dynamic scene generation, such as robot simulation training, rapid prototyping, or interactive content creation in embodied intelligence, its performance is less than ideal. A fundamental contradiction lies in the fact that the inherent iterative optimization process based on stochastic gradient descent, used to optimize scene rendering quality and geometric accuracy, inevitably compromises the efficiency and real-time performance of scene generation. For example, in the simulation training of robot navigation algorithms, when it is necessary to quickly generate a new indoor or outdoor environment according to task instructions, the above contradiction will cause the scene construction time to be as long as several minutes or even several hours, which cannot support the rapid iteration and large-scale parallel training of the algorithm. This results in extremely low system throughput and unacceptable response latency, which seriously restricts the practicality and flexibility of the simulation system.

[0033] Through in-depth analysis, this invention reveals that the root cause of the aforementioned contradiction lies in the fact that related technical solutions model the parameter solving process of a 3D scene as a non-convex optimization problem. This requires hundreds or thousands of forward rendering and backward gradient propagation iterations to gradually converge to a local optimum. This computational paradigm is strongly correlated with the number of iterations, making it difficult to reduce its time complexity.

[0034] To overcome the aforementioned contradictions, this invention proposes a different technical approach. Its core concept lies in transforming the parameter prediction process, which originally required iterative optimization, into a deterministic, data-driven single-step function mapping through a fully trained feedforward neural network. This fundamentally innovates the computational bottleneck of the generation process. Consequently, without significantly sacrificing the visual quality and geometric consistency of the generated scene, the computational complexity of scene generation is significantly reduced from O(n), which is related to the number of iterations, to a constant level of O(1). In other words, this invention provides a technical means to directly generate a set of 3D scene parameters based on a single-step inference using a feedforward neural network, solving the speed bottleneck problem faced by existing optimization-based methods in real-time or high-efficiency scene generation, achieving high-fidelity 3D scene rapid generation at the millisecond to second level.

[0035] In some embodiments, this invention can be deployed on computing clusters or high-performance servers, and mainly includes a multimodal input interface, a panoramic image and depth information generation module, a feedforward neural network model, a parameter post-processing module, and a standard format export interface. Text descriptions or reference images from users enter the system through the input interface. The data flow within the system is as follows: input data is first converted into a panoramic image and its corresponding depth map; these two, as paired data, are input together into a pre-trained feedforward neural network; after a single forward propagation, the network outputs a structured set of parameters; this set of parameters undergoes optional post-processing and is finally packaged into a standard format file. This file can be transmitted to a downstream simulation engine via a network or storage medium. After loading this file, the simulation engine can use its built-in 3D Gaussian splash renderer to render color images and depth maps from any viewpoint in real time, and can simulate sensor data such as LiDAR point clouds for robots or intelligent agents to perform perception, navigation, and operational training. In this environment, the core of this invention lies in the efficient and deterministic mapping process from input data to the generation of a three-dimensional scene parameter set.

[0036] It is understood that, in this application, the term "feedforward neural network" broadly refers to any parameterized function model that does not have recurrent or feedback connections, and where information flows unidirectionally from the input layer through hidden layers to the output layer. Examples include, but are not limited to: fully connected neural networks, convolutional neural networks, visual transformers, and various architectures or hybrid structures composed of encoders and decoders. Its core function is to learn and solidify the complex mapping relationship from input features to output parameters.

[0037] The term "3D Gaussian representation" or "parameter set based on 3D Gaussian representation" refers to the set of parameters required for any method that uses ellipsoids in three-dimensional space as basic primitives to discretize a continuous three-dimensional scene. Each primitive is defined by a set of parameters that define its position, size, orientation, transparency, and color attributes in space.

[0038] The term "single forward propagation" refers to a complete transmission of data from the input layer to the output layer of a neural network, without involving iterative loops that update the network's parameters based on the current output. Its time consumption is essentially constant and approximately linearly related to the complexity of the input data.

[0039] refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a three-dimensional scene generation method provided in an embodiment of the present invention. The method can be executed by a software program deployed on a server or terminal device, and includes the following:

[0040] Step 110: Receive a text description or reference image as input data.

[0041] The input data serves as the source information for generating the target 3D scene. In some embodiments, the input data may come from direct user input, such as text prompts entered through a graphical user interface, or a single uploaded reference image. In other embodiments, the input data may also come from automatic invocation by other systems or programs, such as scene description statements received from a task planning module, or example images retrieved from a database.

[0042] It is understood that the system of the present invention provides a bimodal input interface, in which users can choose to input a natural language description or upload a photographed or drawn reference image. By receiving text descriptions or reference images as input data, it supports multimodal input of semantics and images.

[0043] Step 120: Based on the input data, obtain a panoramic view of the target scene and the corresponding depth information.

[0044] Understandably, when the input data is a text description, a conditional image generation model can be used to generate a 360° isometric cylindrical projection panorama based on that text description. Correspondingly, based on this generated panorama, a pre-trained monocular depth estimation network can be invoked to predict the depth value of each pixel in the image, thereby generating a depth map precisely aligned with the panorama, serving as depth information. The depth map provides distance information of objects in the scene relative to the virtual camera, and is crucial for reconstructing 3D geometry.

[0045] More generally, obtaining panoramic images and depth information can be achieved in various ways. For example, this can include, but is not limited to: directly extracting keyframes from a real panoramic video stream captured by a multi-camera array as image information, extracting the panoramic image from the image information based on a generative model, and obtaining the corresponding depth information using stereo matching or a time-of-flight sensor; or intelligently extrapolating a single image to generate a panoramic image and then estimating the depth; or directly jointly generating a panoramic image and the corresponding depth map based on text conditions. All these solutions can achieve the higher-level function of providing a two-dimensional representation containing color and geometric information for 3D reconstruction.

[0046] Step 130: Input the panoramic image and depth information into the trained feedforward neural network.

[0047] Step 130 connects the data preprocessing module to the core generation module. Understandably, this application requires stitching the panoramic image and depth map together along the channel dimension to form a multi-channel input tensor. For example, the RGB panoramic image and depth map are stitched together to form a 4-channel tensor. This tensor is normalized to the numerical range agreed upon during network training and then fed into the network. The feedforward neural network here has been trained offline, and its parameters are fixed, requiring no updates during the inference phase.

[0048] For example, an RGB panoramic image (3 channels) and a depth map (1 channel) are concatenated into a 4-channel tensor. This tensor is normalized to a numerical range agreed upon during network training (e.g., [0, 1] or [-1, 1]) and then fed into the network. The feedforward neural network here has been trained offline, and its parameters are fixed, requiring no updates during the inference phase. Step 140: The feedforward neural network processes the panoramic image and depth information to generate and output a set of parameters for representing the 3D scene. The set of parameters is based on 3D Gaussian representation.

[0049] The feedforward neural network F_θ internally acts as a mapping function, the architecture of which can contain a visual encoder (such as ResNet or ViT) and a parameter decoder. The encoder is responsible for extracting multi-level features from the concatenated input tensor; the decoder (which can consist of a series of deconvolutional or fully connected layers) then "decodes" these features into a dense set of parameters G that represents the three-dimensional spatial distribution.

[0050] Specifically, the feedforward neural network outputs a matrix of dimension [N, M], where N represents the number of three-dimensional elements (i.e., 3D Gaussian ellipsoids) predicted by the network, and M represents the number of parameters required to describe each element.

[0051] In some embodiments, reference Figure 2 The parameters of each 3D element in the parameter set G preferably include the center position coordinates μ_i, opacity α_i, scaling factor s_i, and color representation coefficient c_i. The center position coordinates μ_i ∈ R³ define the center point of the Gaussian ellipsoid in 3D space. The opacity α_i ∈ [0, 1] determines the transparency of the element during rendering. The scaling factor s_i ∈ R³ is a 3D vector that defines the scaling ratio of the ellipsoid along the x, y, and z axes, i.e., the shape and size of the ellipsoid.

[0052] To ensure the physical meaning (positive magnitude), this application can apply a non-linear activation function, such as the Softplus function: s_i = log(1 + exp(s_i_raw)), to the raw value of the network output, to guarantee that the scaling factor is always positive. The color representation coefficient c_i is used to define the color appearance of the element.

[0053] In some embodiments, to support view-dependent color variations (such as highlights), c_i can be a set of coefficients for spherical harmonics. For example, using third-order spherical harmonics, each color channel (RGB) requires 9 coefficients, so c_i∈R^(3 9). Thus, color can be represented as a function of the viewing direction d: c(d) = Σ(SH_coeffs Y(d)), where Y(d) is a spherical harmonic basis function. In this way, the network predicts not only the geometry and density of the scene, but also its complex appearance.

[0054] Furthermore, to describe the rotational orientation of the ellipsoid in space, the parameters of the three-dimensional elements can also include a rotational representation parameter q_i. The rotation can be represented by the quaternion q_i∈R. 4 The quaternion is used because it is computationally efficient and has no singularities. To ensure that the quaternion represents a valid rotation (unit rotation), the original quaternion output by the network needs to be normalized so that its modulus is 1: q_i = q_i_raw / ||q_i_raw||.

[0055] It should be noted that the generation process of the above parameter set is completed in one go. In a single forward propagation, the network directly outputs all parameters of all N Gaussian elements without any iterative optimization based on rendering feedback. This drastically reduces the generation time from minutes in traditional methods to seconds or even milliseconds, fundamentally solving the efficiency bottleneck. The panoramic image containing depth information provided in step 120 above provides the network with crucial geometric priors, enabling it to more accurately predict the spatial distribution (μ_i) of 3D elements, not just their appearance. The synergy of these two aspects enables rapid and geometrically sound 3D scene generation, thus jointly solving the specific technical problem of balancing speed and quality in real-time simulation scene construction.

[0056] To further optimize the adaptability and richness of the input data in the above embodiments, and to address the practical problem that users may only provide text or a single image as a starting point, this application also provides the following preferred solutions. As mentioned above, obtaining panoramic images and depth information based on input data can be achieved in various ways.

[0057] In some embodiments, step 120 specifically includes: generating a panoramic view of the target scene based on the input data. This step aims to transform potentially incomplete or non-panoramic input into a complete 360-degree environmental view.

[0058] Specifically, when the input data is a text description, a conditional image generation model is used to generate panoramic images based on the text description. For example, a diffusion model such as StableDiffusion can be used, but its training data or fine-tuning strategy is specifically designed for the task of generating isometric cylindrical projection panoramic images, thereby enabling the direct synthesis of high-quality, seamless panoramic scenes based on text prompts.

[0059] When the input data is a reference image, an image augmentation model is used to expand the reference image into a panoramic view. The reference image might just be a screenshot from one perspective, such as a photo of an indoor corner. The image augmentation model (an advanced image inpainting and outlining model) can intelligently complete the surrounding field of view based on the image content, generating a reasonable panoramic environment. For example, the model will extend the generation of other parts of the room based on the textures of the walls and floors in the image.

[0060] Then, based on the panoramic image, a corresponding depth map is generated as depth information. After obtaining the panoramic image, a general or panorama-optimized monocular depth estimation model is used to predict the depth of each pixel. This design makes the input modality more flexible and user-friendly, lowering the barrier to entry for users. Its advantage lies in the fact that regardless of whether the user is good at describing in words or providing visual references, the system can convert them into standardized, information-complete network input, greatly expanding the application scenarios of this invention.

[0061] Those skilled in the art will understand that alternative structures for achieving the same purpose include: using a large multimodal model (LMM) to simultaneously understand text and images and guide panoramic image generation; or constructing a unified generative model that accepts text, images, or a combination of both as input and directly outputs panoramic and depth maps. For example, an end-to-end diffusion model could start with noise latent and conditional inputs and directly output a panoramic image with depth.

[0062] Specifically, see Figure 2 , Figure 2 This is a system architecture diagram of the three-dimensional scene generation method provided in an embodiment of the present invention.

[0063] like Figure 2As shown, the text description T is fed into a text encoder to obtain a conditional embedding, which guides a latent diffusion model to denoise and generate a panoramic image I_pano. For the image input I_ref, features are first extracted by an encoder, and then spatially expanded by a decoder (or diffusion model), outputting I_pano. Subsequently, a depth estimation network f_depth processes I_pano in a feedforward manner, outputting D_pano. The above models (diffusion model, image expansion model, and depth estimation model) can all use current mainstream architectures, and their parameters have been trained before deployment. By adopting the above-described specifically defined bimodal input processing flow, this preferred scheme can greatly improve the ease of use and flexibility of the system, ensuring the diversity of generated scenes and consistency with user intent from the source. This further helps to solve the problem of the demanding input requirements of traditional methods (requiring multiple viewpoints), thereby synergistically enhancing the overall technical effect of the present invention in rapidly generating 3D scenes. In addition, the conditional image generation model on which the panoramic image is generated is not limited to the diffusion model; generative adversarial networks (GANs) or autoregressive models can also be used, as long as they have the ability to generate panoramic images based on text or images.

[0064] Furthermore, the training process is crucial to ensure that the feedforward neural network can learn the complex mapping from two-dimensional information to high-quality three-dimensional representations. As mentioned earlier, the trained feedforward neural network is preferably obtained through self-supervised training. Self-supervised training does not require expensive, paired, multi-view real 3D data as labels, but instead uses the results generated by the model itself to construct supervision signals, making it particularly suitable for situations where ground truth data is lacking, such as open-domain scene generation.

[0065] To effectively address the challenges of network training without ground truth data and improve the quality of novel perspective synthesis in generated scenes, this application provides a specific implementation of a self-supervised training method. This method includes the following steps: First, semantic description information is obtained from the image rendered based on the parameter set generated by the feedforward neural network.

[0066] Furthermore, during the training loop, for a batch of input data (panoramic image + depth map), the network outputs a corresponding parameter set G. Using a differentiable 3D Gaussian splash renderer, this parameter set is rendered from a random or specified virtual viewpoint v, resulting in a 2D image I_render. This rendered image is then input into a multimodal large language model (MLLM). This MLLM possesses visual understanding capabilities and can describe the image content. For example, the MLLM can output semantic description information S such as "This is a rendering of a modern living room, with a gray sofa in the center and a floor-to-ceiling window on the left." This step transforms the rendering result into high-level semantics.

[0067] Next, using semantic description information as conditions, an image generation model is used to generate a teacher image. Using the aforementioned semantic description S as textual conditions, input into a powerful image generation model (such as a high version of StableDiffusionXL) can generate a high-quality, highly realistic image I_teacher. This image is considered a "teacher" because it comes from a powerful generative model, and its image quality is typically higher than the coarse results rendered by the currently trained feedforward network.

[0068] Subsequently, based on the rendered image and the teacher image, a perceptual difference loss is calculated. The goal of this loss function is to narrow the distance between the network-rendered image and the high-quality teacher image in the perceptual feature space, rather than achieving strict pixel-level consistency.

[0069] Specifically, in a preferred implementation, calculating the perceptual difference loss includes: extracting deep features of the rendered image and the teacher image respectively using a pre-trained visual feature extraction network; and calculating the distance between the deep features as the perceptual difference loss.

[0070] For example, a VGG-19 network pre-trained on ImageNet can be used. Inputting I_render and I_teacher into VGG-19 respectively extracts feature maps φ_j(I_render) and φ_j(I_teacher) at specific layers (e.g., ReLU3_3, ReLU4_3). The perceptual difference loss L_self can be defined as the sum of the mean squared errors (MSEs) between these feature maps: L_self = Σ_j||φ_j(I_render) - φ_j(I_teacher)||².

[0071] Understandably, the aforementioned perceptual difference loss is insensitive to pixel location but sensitive to texture, style, and semantic content, effectively guiding the network to improve the quality of the rendered image. Finally, a feedforward neural network is trained based on the perceptual difference loss. Through the backpropagation algorithm, the gradient of the perceptual difference loss passes through the renderer (because it is differentiable) and is passed to the parameters θ of the feedforward neural network, driving the network to update so that the set of parameters it generates can produce results that are closer to the teacher's image perception in future renderings.

[0072] It is worth noting that the above design constructs a self-supervised closed loop of "rendering-understanding-reconstruction". Its advantage lies in leveraging the semantic understanding capabilities of large language models and the image priors of large image generation models to provide a continuous stream of high-quality supervision signals to the feedforward network. This significantly improves the visual fidelity and semantic accuracy of the generated scene without any real 3D labels. Those skilled in the art will understand that obtaining semantic description information is not limited to MLLM; image annotation models or fixed template descriptions can also be used. Furthermore, the model for generating teacher images is not limited to diffusion models; Generative Adversarial Networks (GANs) or other Transformer-based generative models are equally applicable.

[0073] Specifically, see Figure 3 The Chinese self-supervised training section, Figure 3 This is a schematic diagram of the self-supervised training mechanism in the three-dimensional scene generation method provided in the embodiments of the present invention.

[0074] like Figure 3 As shown, the network output G is rendered to obtain I_render, which is then fed into the MLLM module to obtain S. S is then used as a conditionally guided diffusion model to generate I_teacher. I_render and I_teacher are fed into a pre-trained VGG network to extract features and calculate L_self. By adopting the above-mentioned MLLM-guided perceptual difference self-supervised training mechanism, this preferred scheme can effectively train the feedforward network under conditions without any real 3D data supervision, so that the generated 3D scene can maintain high visual quality when rendered from any new perspective, thus solving the core training problem of open domain scene generation.

[0075] In the self-supervised training process, to further constrain the geometric rationality of the generated scene and avoid spatially distorted or deformed structures, the self-supervised training method of this application may also include geometric prior constraints. Specifically, based on depth information, a geometric prior loss function is constructed, which is used to constrain the spatial position of the three-dimensional elements in the parameter set; then, a feedforward neural network is trained based on the geometric prior loss function. The depth information D_pano, as part of the input, inherently contains geometric cues of the scene.

[0076] During training, this known information can be used to supervise the network's prediction of 3D element positions μ_i. For example, the geometric prior loss function can be constructed based on the deviation between the position and depth information of the 3D elements in the parameter set. A specific calculation method is as follows: For the i-th Gaussian ellipsoid predicted by the network, its center position is μ_i. Project this 3D point onto the 2D image plane in the coordinate system of the virtual camera used to generate the panoramic image to obtain the projected pixel coordinates (π_u(μ_i), π_v(μ_i)), and calculate its depth value d(μ_i).

[0077] Meanwhile, in the input depth map D_pano, the depth value D(u_i, v_i) of the corresponding pixel coordinates (u_i, v_i) (where (u_i, v_i) can be the feature position related to the Gaussian element, or the pixel coordinates assigned in other ways) is extracted.

[0078] The geometric prior loss L_geo can be composed of the projection position error and the depth error: L_geo=Σ_i[||(π_u(μ_i),π_v(μ_i))-(u_i,v_i)||²+λ_d·|d(μ_i)-D(u_i,v_i)|].

[0079] Where λ_d is the balancing weight. This loss function forces the network to align the predicted 3D point cloud with the input image after projection, and its depth values ​​match the input depth estimate, thus effectively transferring the 2D depth prior to the 3D spatial prediction.

[0080] In summary, by introducing this loss, geometric consistency constraints can be explicitly injected during the training phase, so that the final generated scene has a stable spatial structure even when viewed from a perspective not seen during training, which is very suitable for robot simulation applications with high geometric accuracy requirements.

[0081] To ensure that the generated 3D scene looks reasonable not only from a fixed perspective but also from any viewing angle, and to avoid viewpoint-dependent artifacts (such as the "paper-thin" effect or floating objects), the self-supervised training method in this application may also include multi-view... Figure 1 Consistency constraints. Specifically, the parameter set is rendered from multiple virtual perspectives to obtain rendering results from multiple perspectives; based on the rendering results from multiple perspectives, consistency loss is calculated; and then a feedforward neural network is trained based on the consistency loss.

[0082] In the training iterations, for the same parameter set G, in addition to rendering the viewpoint used to generate the teacher image, multiple other virtual viewpoints {V_1, V_2, ..., V_k} are randomly sampled. A differentiable renderer is used to render G from these viewpoints respectively, resulting in a set of images {I_v1, I_v2, ..., I_vk}. The goal of the consistency loss is to ensure that these different views from the same 3D scene maintain content consistency.

[0083] Specifically, calculating consistency loss based on normalized cross-correlation (NCC) is an effective method. NCC is insensitive to brightness variations and focuses more on the consistency of image structure. For example, a reference viewpoint V_ref (such as the viewpoint used to generate the teacher image mentioned earlier) can be chosen, whose rendering result is I_ref. For the rendering result I_vk of each other viewpoint V_k, a homography transformation H_{k→ref} from V_k to V_ref can be calculated (if the camera is purely rotated, or the scene is approximately planar), or a more general optical flow estimation can be used to warp I_vk to the reference viewpoint to obtain Warp(I_vk, H_{k→ref}). Then, the NCC between I_ref and Warp(I_vk, H_{k→ref}) can be calculated. (Multi-view) Figure 1 The consistency loss L_mvc can be defined as the average negative NCC: L_mvc=(1 / |V|)Σ_k(1-NCC(I_ref,Warp(I_vk,H_{k→ref}))).

[0084] Understandably, minimizing this loss implies maximizing the structural relevance between different views. This design forces the network to consider the true volumetric properties of 3D elements in space when predicting parameters, ensuring they are reasonable from different perspectives, thereby effectively improving the 3D consistency and stability of the generated scene.

[0085] In one embodiment, reference Figure 3 In the self-supervised training process of this application, the aforementioned perceptual difference loss, geometric prior loss, and multi-view loss can be applied. Figure 1 The consistency losses are weighted and summed to obtain the total loss function used to train the feedforward neural network. The expression of the total loss function is: Ltotal = λ1·Lself + λ2·Lgeo + λ3·Lmvc, where Ltotal is the total loss, Lself is the perceptual difference loss, Lgeo is the geometric prior loss, and Lmvc is the multi-view loss. Figure 1 Consistency loss is defined by λ1, λ2, and λ3, which are preset weight coefficients for the corresponding loss terms. These weight coefficients can be adaptively adjusted or pre-set according to training requirements and the optimization objectives of the generated scene. During the backpropagation phase of training, the gradient is calculated based on the total loss. This gradient is then passed to the feedforward neural network via a differentiable 3D Gaussian splash renderer, updating all learnable parameters of the network in an end-to-end manner until the total loss converges to a preset threshold, completing the training of the feedforward neural network. Through the joint constraints of multiple loss terms, the visual fidelity, geometric rationality, and multi-view rendering consistency of the generated scene can be simultaneously considered, comprehensively improving the parameter prediction accuracy and 3D scene generation effect of the feedforward neural network. In some embodiments, as described above, the parameters of each 3D element in the parameter set specifically include center position coordinates, opacity, scaling factor, and color representation coefficients. This is the core parameter set for the 3D Gaussian representation. The center position coordinates μ_i are typically represented as a 3D vector, such as (X, Y, Z) values ​​in a right-handed coordinate system. The opacity α_i is a scalar, typically constrained between 0 and 1 by a Sigmoid activation function: α_i = σ(α_i_raw). The scaling factor s_i, as before, must be positive and is usually post-processed by applying Softplus or an exponential function (exp) to the original network output. The color representation coefficients c_i, under the simple Lambertian (diffuse) assumption, can be a 3D RGB vector; to support more complex appearances, spherical harmonic coefficients are used. Furthermore, to fully describe the pose of an ellipsoid, the parameters of the 3D element also include rotation representation parameters q_i, typically using quaternions. To ensure that the quaternions represent valid unit rotations, the rotation representation parameters are normalized, i.e., q_i = q_i_raw / ||q_i_raw||. The definitions of these specific parameters and the post-processing methods together ensure that the values ​​output by the network are physically interpretable and can be directly used for downstream rendering, thus guaranteeing the validity of the generated results.

[0086] In some embodiments, after generating and outputting the parameter set, this method further includes an export step to enable its use by a wide range of downstream applications, particularly simulation engines: exporting the parameter set as a standard format file for loading into the simulation engine for rendering. For example, the parameters of all Gaussian elements (position, scaling, rotation quaternions, spherical harmonics, opacity) can be organized and written to a file in a point cloud extension format (.ply, a polygon file format) or in the emerging .splat format (designed specifically for 3D Gaussian splashes). This file contains a lightweight representation of the complete scene. The simulation engine only needs to parse this file and call its built-in or integrated 3DGS renderer to display the 3D scene in real-time and interactively, supporting advanced features such as dynamic lighting and physical collision detection. This allows the output of this invention to be seamlessly integrated into existing robot simulation and digital twin workflows.

[0087] In some embodiments, the application scenario of this application is exemplified as follows: rapid simulation training of robot navigation algorithms. In the R&D departments of autonomous driving or mobile robot companies, algorithm engineers need to train robots to navigate in different indoor environments (such as offices, warehouses, and homes). Traditional methods require art teams to manually model a few fixed scenes in advance, resulting in a severe lack of training data diversity and an inability to dynamically generate scenes according to specific training needs (such as "obstacle avoidance in a corridor full of boxes"). With this invention, engineers only need to type "a dimly lit warehouse corridor with stacks of brown cardboard boxes of varying heights on both sides and cables on the ground" into the system's text input box and click the "Generate" button. The system completes panoramic image generation, depth estimation, and 3D scene parameter prediction within seconds and automatically exports a .splat file. Subsequently, this file is automatically loaded into the IsaacSim simulation engine. The engine instantly constructs the corresponding 3D environment. The simulation system controls the virtual robot to generate the starting point and target point and starts the navigation algorithm. The robot uses virtual LiDAR and cameras to perceive this newly generated environment and begins path planning and obstacle avoidance training. If the algorithm frequently collides at a certain location, engineers can immediately modify the scene description, for example, by adding "there is an overturned box in the middle of the corridor," and generate a new scene again within seconds for targeted training. In this scenario, the invention improves the efficiency of constructing simulation training scenes by hundreds of times, achieving on-demand, dynamic, and unlimited scene supply. The perceived value for users (algorithm engineers) lies in their ability to iterate algorithms at unprecedented speeds, quickly test the robustness of algorithms in various complex and long-tailed scenarios, thereby significantly shortening the development cycle and improving the performance of the final product. Furthermore, from an infringement determination perspective, any system capable of receiving text / images, outputting 3D Gaussian parameters in a single pass through a feedforward network, and using it for simulation rendering may fall within the protection scope of this invention.

[0088] refer to Figure 4 , Figure 4This is a schematic block diagram of a three-dimensional scene generation device provided in an embodiment of the present invention. Based on the same inventive concept as the method embodiment, this application also provides a three-dimensional scene generation device. The three-dimensional scene generation device 400 includes an input acquisition module 410, an information processing module 420, a data input module 430, and a scene generation module 440. The input acquisition module 410 is used to receive text descriptions or reference images as input data. The information processing module 420 is used to obtain a panoramic view of the target scene and the corresponding depth information based on the input data. The data input module 430 is used to input the panoramic view and depth information into a trained feedforward neural network. The scene generation module 440 is used to process the panoramic view and depth information by the feedforward neural network to generate and output a parameter set for representing the three-dimensional scene, the parameter set being based on a 3D Gaussian representation. The functions of these modules correspond one-to-one with the aforementioned method steps, and their specific implementation details have been described in detail above and will not be repeated here.

[0089] Furthermore, the information processing module 420 may specifically include a receiving unit, an image generation unit, and a depth estimation unit. The receiving unit receives text descriptions or reference images as input data. The image generation unit generates a panoramic image of the target scene based on the input data. The depth estimation unit generates a corresponding depth map as depth information based on the panoramic image. The specific operating mode of the image generation unit is as follows: when the input data is a text description, it uses a conditional image generation model to generate a panoramic image based on the text description; when the input data is a reference image, it uses an image expansion model to expand the reference image into a panoramic image.

[0090] The 3D scene generation device 400 may further include a training module for obtaining a trained feedforward neural network through self-supervised training. The training module may further include a semantic acquisition unit, a teacher image generation unit, a perceptual loss calculation unit, and a first training unit. The semantic acquisition unit is used to acquire semantic description information from an image rendered based on a parameter set generated by the feedforward neural network. The teacher image generation unit is used to generate a teacher image using an image generation model, conditioned on the semantic description information. The perceptual loss calculation unit is used to calculate the perceptual difference loss based on the rendered image and the teacher image. Specifically, it can be used to extract deep features from the rendered image and the teacher image using a pre-trained visual feature extraction network, and calculate the distance between the deep features as the perceptual difference loss. The first training unit is used to train the feedforward neural network based on the perceptual difference loss.

[0091] In addition, the training module may include a geometric loss construction unit and a second training unit. The geometric loss construction unit is used to construct a geometric prior loss function based on depth information. This geometric prior loss function is used to constrain the spatial positions of the 3D elements in the parameter set. Preferably, this geometric prior loss function is constructed based on the deviation between the positional information and the depth information of the 3D elements in the parameter set. The second training unit is used to train the feedforward neural network based on the geometric prior loss function.

[0092] The training module may also include a multi-view rendering unit, a consistency loss calculation unit, and a third training unit. The multi-view rendering unit renders the parameter set from multiple virtual perspectives, obtaining rendering results from multiple views. The consistency loss calculation unit calculates the consistency loss based on the rendering results from multiple views, for example, based on a normalized cross-correlation metric. The third training unit trains a feedforward neural network based on the consistency loss.

[0093] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0094] In addition, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods in the foregoing method embodiments.

[0095] And an electronic device, comprising: One or more processors; and A memory associated with one or more processors, the memory being used to store program instructions that, when read and executed by one or more processors, perform the steps of any of the methods in the foregoing method embodiments.

[0096] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods in the foregoing method embodiments.

[0097] in, Figure 5The architecture of an electronic device 500 is illustrated by way of example, which may include a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, and a memory 520. The processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520 can communicate with each other via a communication bus 530.

[0098] The processor 510 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided by this invention.

[0099] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 520 can store the operating system 521 for controlling the operation of the electronic device 500, and the basic input / output system (BIOS) 522 for controlling the low-level operations of the electronic device 500. Additionally, it can store a web browser 523, a data storage management system 524, and a 3D scene generation device 400, etc. The aforementioned 3D scene generation device 400 can be the application program that specifically implements the aforementioned steps in this embodiment of the invention. In summary, when the technical solution provided by this invention is implemented through software or firmware, the relevant program code is stored in the memory 520 and is called and executed by the processor 510.

[0100] Input / output interface 513 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0101] Network interface 514 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0102] Bus 530 includes a pathway for transmitting information between various components of the device, such as processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520.

[0103] It should be noted that although the above-described device only shows the processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, memory 520, bus 530, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the present invention, and not necessarily all the components shown in the figures.

[0104] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of the present invention.

[0105] The technical solution provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for generating a three-dimensional scene, characterized in that, include: Receive text descriptions or reference images as input data; Based on the input data, a panoramic view of the target scene and the corresponding depth information are obtained; The panoramic image and the depth information are input into a trained feedforward neural network; The feedforward neural network processes the panoramic image and the depth information to generate and output a set of parameters for characterizing the three-dimensional scene, the set of parameters being based on 3D Gaussian representation.

2. The method according to claim 1, characterized in that, The step of obtaining a panoramic view of the target scene and its corresponding depth information based on the input data includes: Based on the input data, a panoramic image of the target scene is generated; Based on the panoramic image, a corresponding depth map is generated as the depth information.

3. The method according to claim 2, characterized in that, The step of generating a panoramic image of the target scene based on the input data includes: When the input data is a text description, the panoramic image is generated based on the text description using a conditional image generation model. When the input data is a reference image, the reference image is expanded into the panoramic image using an image expansion model.

4. The method according to claim 1, characterized in that, The trained feedforward neural network is obtained through a self-supervised training method; the self-supervised training method includes: Semantic description information is obtained from the image rendered based on the parameter set generated by the feedforward neural network; Using the semantic description information as a condition, an image generation model is used to generate a teacher image; Based on the rendered image and the teacher image, calculate the perceptual difference loss; The feedforward neural network is trained based on the perceptual difference loss.

5. The method according to claim 4, characterized in that, The calculation of perceptual difference loss based on the rendered image and the teacher image includes: Deep features of the rendered image and the teacher image are extracted using a pre-trained visual feature extraction network. The distance between the deep features is calculated as the perceptual difference loss.

6. The method according to claim 4, characterized in that, The self-supervised training method also includes: Based on the depth information, a geometric prior loss function is constructed. The geometric prior loss function is used to constrain the spatial position of the three-dimensional elements in the parameter set. The geometric prior loss function is constructed based on the deviation between the position information of the three-dimensional elements in the parameter set and the depth information. The feedforward neural network is trained based on the geometric prior loss function.

7. The method according to claim 4, characterized in that, The self-supervised training method also includes: The parameter set is rendered from multiple virtual perspectives to obtain rendering results from multiple perspectives; Based on the rendering results from the multiple perspectives and the normalized cross-correlation metric, the consistency loss is calculated. The feedforward neural network is trained based on the consistency loss.

8. The method according to claim 1, characterized in that, The parameters for each 3D element in the parameter set include center position coordinates, opacity, scaling factor, and color representation coefficient.

9. The method according to claim 8, characterized in that, The parameters of the three-dimensional element also include rotation representation parameters; the scaling factor is processed by a nonlinear activation function to ensure that it is a positive value; the rotation representation parameters are normalized.

10. The method according to claim 1, characterized in that, After generating and outputting the parameter set, the process also includes: The parameter set is exported as a standard format file for loading into the simulation engine for rendering.

11. A three-dimensional scene generation device, characterized in that, include: The input acquisition module is used to receive text descriptions or reference images as input data; The information processing module is used to obtain a panoramic view of the target scene and the corresponding depth information based on the input data. The data input module is used to input the panoramic image and the depth information into a trained feedforward neural network; The scene generation module is used to process the panoramic image and the depth information by the feedforward neural network to generate and output a set of parameters for characterizing the three-dimensional scene, the set of parameters being based on 3D Gaussian representation.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1-10.

13. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program performs the steps of the method described in any one of claims 1-10.

Images