Simulation scene generation method and device, electronic equipment and readable storage medium

Abstract

The invention provides a simulation scene generation method and device, electronic equipment and a readable storage medium, and the method comprises the steps: carrying out the grid modeling of a target road element based on the real physical data of the target road element, and obtaining a grid-modeled target road element; performing Gaussian three-dimensional modeling on the grid modeled target road elements to obtain an initial Gaussian primitive set of the target simulation scene; and inputting the initial Gaussian primitive set and the target road video data into a trained simulation scene optimization model, optimizing visual attributes of each Gaussian primitive in the initial Gaussian primitive set, and obtaining a target simulation scene corresponding to the target road video data. Therefore, the geometric structure and the visual appearance of the object in the scene are decoupled, the geometric accuracy of the object is ensured through the grid model, and the visual attribute of the object is optimized through the trained simulation scene optimization model, so that the generation confidence and the generation efficiency of the simulation scene can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving simulation technology, and in particular to a simulation scene generation method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the development of advanced autonomous driving technology, end-to-end simulation testing faces new challenges. Unlike traditional modular testing methods, end-to-end model testing requires the use of complete raw sensor data for overall simulation, which places higher demands on the realism of the simulation scenarios and the confidence level of the data. At the same time, the training of end-to-end systems relies on massive amounts of training data, especially corner case data; however, the density of valuable data in real-world data is low, and its quantity and diversity are insufficient to support the training needs of end-to-end systems.

[0003] Existing methods for generating simulation scenes mainly include two approaches: traditional image rendering techniques (such as rasterization rendering and ray tracing) and 3D Gaussian Splatting (3DGS). While traditional image rendering techniques can generate high-quality images, they are computationally expensive in dynamic scenes and complex environments, lack realism, and produce data with low confidence, making it difficult to meet the stringent requirements of end-to-end system training. Although 3DGS performs well in reconstructing and rendering static scenes, it suffers from insufficient generalization ability when handling dynamic scenes, similarly failing to meet the diverse scene requirements of end-to-end simulation testing. Because high-fidelity rendering requires fine details and complex physical simulations, while real-time rendering demands computation in a very short time, existing simulation scene generation methods present a contradiction between real-time rendering and high fidelity, resulting in low confidence and efficiency in simulation scene generation. Summary of the Invention

[0004] In view of this, embodiments of this application provide at least one simulation scene generation method, apparatus, electronic device, and readable storage medium, which decouples the geometric structure and visual appearance of objects in the scene, ensures the geometric accuracy of objects through a mesh model, and optimizes the visual attributes of objects through a trained simulation scene optimization model, thereby effectively improving the confidence and generation efficiency of simulation scene generation.

[0005] This application mainly includes the following aspects: In a first aspect, embodiments of this application provide a method for generating a simulation scene, the method comprising: Based on the real physical data of the target road elements, mesh modeling is performed on the target road elements to obtain mesh-modeled target road elements; the target road elements include roads and vehicles. Gaussian 3D modeling is performed on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene; The initial Gaussian element set and the target road video data are input into the trained simulation scene optimization model to optimize the visual attributes of each Gaussian element in the initial Gaussian element set, thereby obtaining the target simulation scene corresponding to the target road video data.

[0006] Secondly, embodiments of this application also provide a simulation scene generation device, the simulation scene generation device comprising: The mesh modeling module is used to perform mesh modeling on the target road elements based on the real physical data of the target road elements, so as to obtain mesh-modeled target road elements; the target road elements include roads and vehicles. The Gaussian modeling module is used to perform Gaussian 3D modeling on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene; The visual optimization module is used to input the initial Gaussian element set and the target road video data into the trained simulation scene optimization model, optimize the visual attributes of each Gaussian element in the initial Gaussian element set, and obtain the target simulation scene corresponding to the target road video data.

[0007] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the simulation scene generation method described above.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the simulation scene generation method described above.

[0009] This application provides a simulation scene generation method, apparatus, electronic device, and readable storage medium. The method includes: performing mesh modeling on the target road elements based on real physical data of the target road elements to obtain mesh-modeled target road elements; the target road elements include roads and vehicles; performing Gaussian 3D modeling on the mesh-modeled target road elements to obtain an initial Gaussian element set for the target simulation scene; inputting the initial Gaussian element set and target road video data into a trained simulation scene optimization model to optimize the visual attributes of each Gaussian element in the initial Gaussian element set, thereby obtaining a target simulation scene corresponding to the target road video data. In this way, this application decouples the geometric structure and visual appearance of objects in the scene, ensures the geometric accuracy of objects through a mesh model, and optimizes the visual attributes of objects through a trained simulation scene optimization model, which can effectively improve the confidence and generation efficiency of simulation scene generation.

[0010] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a simulation scene generation method provided in an embodiment of this application is shown; Figure 2 This illustration shows one of the functional block diagrams of a simulation scene generation device provided in an embodiment of this application; Figure 3 This is a second functional block diagram of a simulation scene generation device provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0014] To facilitate understanding of this application, the technical solutions provided in this application will be described in detail below with reference to specific embodiments.

[0015] Please see Figure 1 , Figure 1 This is a flowchart illustrating a simulation scene generation method provided in an embodiment of this application. Figure 1 As shown, the simulation scene generation method provided in this application includes: S101, Based on the real physical data of the target road elements, perform mesh modeling on the target road elements to obtain mesh-modeled target road elements; the target road elements include roads and vehicles.

[0016] To address the inherent conflict between real-time rendering and high fidelity in existing simulation scene generation methods, this application decouples the geometric structure and visual appearance of objects in the simulation scene. Specifically, a high-quality, topologically-based geometric model base is constructed using a physical mesh model as a priori constraint to ensure the geometric accuracy of the simulation scene. Simultaneously, a lightweight neural network model is trained and constrained to collaboratively optimize the visual properties of the 3D Gaussian model, thereby imparting high realism to the simulation scene. The resulting 3D model is not only visually highly realistic but also geometrically stable and reliable, perfectly meeting the stringent requirements of high-confidence verification for end-to-end large models (such as autonomous driving). Specifically, firstly, accurate mesh-modeled target road elements are constructed using real physical data of the target road elements, such as road geometry and vehicle dimensions. These mesh-modeled target road elements provide the foundation for subsequent Gaussian 3D modeling, ensuring the geometric accuracy of the simulation scene.

[0017] S102, Gaussian 3D modeling is performed on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene.

[0018] Here, each vertex in the target road element of the mesh model is transformed into a Gaussian element, which has initial visual properties such as color, opacity, and size. In this way, complex 3D scenes can be represented in an efficient and flexible manner as a tensor of a set of 3D Gaussian elements (the initial set of Gaussian elements), preparing for subsequent optimization steps.

[0019] S103, the initial Gaussian element set and the target road video data are input into the trained simulation scene optimization model to optimize the visual attributes of each Gaussian element in the initial Gaussian element set, thereby obtaining the target simulation scene corresponding to the target road video data.

[0020] Here, by inputting the initial Gaussian tuple and target road video data into the trained simulation scene optimization model, the visual attributes of the Gaussian tuples are automatically adjusted, making the generated simulation scene more visually realistic and highly consistent with the target road video data. This process not only improves the realism of the simulation scene but also ensures the high confidence of the generated data, thus meeting the requirements of end-to-end simulation testing for high-quality simulation scenes.

[0021] Further, the step of performing Gaussian 3D modeling on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene includes: Step a1: Obtain the position coordinates of each triangular mesh vertex of the target road element in the three-dimensional space.

[0022] Here, the mesh model generated by mesh modeling consists of vertices, edges, and faces, where faces are typically triangles. The position coordinates of each triangular mesh vertex are calculated using mesh modeling software or algorithms. These position coordinates are the foundation of Gaussian 3D modeling, ensuring the accurate positioning of each Gaussian element in space. These coordinates are usually represented in the form of 3D vectors, for example... ,in This represents the x-coordinate of the primitive in three-dimensional space. Represents the ordinate of the primitive in three-dimensional space. This represents the depth coordinates of the primitive in three-dimensional space.

[0023] Step a2: Based on the respective position coordinates, create an initial high-order element for each triangular mesh vertex and determine the geometric properties of each initial high-order element.

[0024] Here, for each triangular mesh vertex in the mesh model, an initial Gaussian element is created. Each Gaussian unit It is defined by multiple parameters, including geometric and visual attributes. Among them, the Gaussian element... The geometric properties include position, normals, and covariance matrix. The position property is directly inherited from the vertex's position coordinates and remains unchanged without participating in optimization. The initial normal is inherited from the calculated normal of the triangular mesh at that vertex and is also fixed. These geometric properties ensure the correct representation of Gaussian elements in space.

[0025] Step a3: Assign initial visual attributes to each of the initial Gaussian elements; the initial visual attributes include initial color, initial opacity, and initial size.

[0026] Here, each initial Gaussian element The visual attributes include color, opacity, and size. The initial color can be obtained by projecting vertices onto multiple input images and performing color fusion to arrive at an initial RGB color value; the initial opacity and initial size can be obtained using preset empirical values. These visual attributes ensure a reasonable visual representation of Gaussian units, providing a foundation for subsequent optimization.

[0027] Step a4: Based on the geometric and visual properties of each of the initial Gaussian elements, determine the initial Gaussian element set of the target simulation scene.

[0028] Here, all the created initial Gaussian elements are gathered together to form the initial Gaussian element set of the target simulation scene. This set contains all Gaussian elements. The geometric and initial visual properties of these Gaussian elements provide input data for subsequent optimization steps. By optimizing the visual properties of these Gaussian elements, a simulation scene highly consistent with the target road video data can be generated.

[0029] Further, the simulation scenario optimization model is trained according to the following steps: Step b1: Train the initial scene optimization model based on the sample road video data and the initial Gaussian set.

[0030] Here, during the training of the simulation scene optimization model, the initial scene optimization model is trained based on sample road video data and an initial Gaussian ensemble. The sample road video data provides visual information about the real scene, while the initial Gaussian ensemble contains the geometric and visual attributes of the scene. By inputting this data into the initial scene optimization model, the model can learn how to adjust the visual attributes of the Gaussian ensembles to generate a simulation scene highly consistent with the real scene. The goal of this step is to minimize the difference between the simulation image and the real image, that is, to gradually improve the structural similarity index (SSIM) between the simulation image and the real image, making it approach 1.

[0031] Step b2: During the training process, calculate the reconstruction loss function, geometric consistency loss function, prior loss function, temporal smoothing loss function, and spatial smoothing loss function of the initial scene optimization model.

[0032] Here, the training process involves calculating multiple loss functions to optimize the model for the initial scene. These loss functions work together to ensure that the model can be optimized from multiple dimensions, thereby generating a high-quality simulation scene. The formula for the total loss function can be expressed as: ; Specifically, The total loss function is used to reconstruct the loss function. This function measures the difference between the rendered image and the real image, with the core objective of minimizing this difference to make the rendered image visually as close to the real image as possible. Meanwhile, the geometric consistency loss function... The focus is on ensuring that the model's geometry is highly consistent with the input geometry, encompassing consistency in depth maps and normal directions, to guarantee the geometric accuracy of the scene. This is a weighting factor for the geometric consistency loss. Prior loss function. This plays a regularization role, aiming to prevent the model from overfitting, thereby ensuring the rationality and smoothness of the optimization results, enabling the model to generate stable and reliable simulation scenarios. This is the weighting factor for the prior loss. Temporal smoothing loss function. This is used to ensure that changes in the visual attributes of Gaussian elements remain smooth between adjacent frames, which is crucial for generating continuous and natural dynamic scenes. These are the weighting factors of the temporal smoothing loss function. Spatial smoothing loss function. This is aimed at ensuring the spatial continuity and smoothness of Gaussian elements, effectively avoiding isolated fragments or noise, thereby improving the overall visual quality of the scene. These are the weighting factors of the spatial smoothing loss function. By comprehensively considering these loss functions, the model continuously adjusts and optimizes the visual attributes of Gaussian units during training, while keeping the geometric attributes unchanged, ultimately generating a realistic and efficient simulation scene that meets the stringent requirements of end-to-end simulation testing.

[0033] Step b3: When the reconstruction loss function, geometric consistency loss function, prior loss function, temporal smoothing loss function, and spatial smoothing loss function of the initial simulation scene optimization model all meet the corresponding preset conditions, training is stopped, and the trained simulation scene optimization model is obtained.

[0034] Here, the training process stops when the reconstruction loss function, geometric consistency loss function, prior loss function, temporal smoothing loss function, and spatial smoothing loss function of the initial simulation scene optimization model all meet their respective preset conditions. These preset conditions are usually thresholds set based on the model's performance metrics. When the values ​​of all loss functions reach or fall below these thresholds, the model is considered to have converged, and the training process can stop. At this point, the resulting model can generate high-quality, high-confidence simulation scenes, meeting the requirements of end-to-end simulation testing.

[0035] Further, the reconstruction loss function of the initial simulation scene optimization model is determined according to the following steps: Step c1: Based on the initial Gaussian set, the initial simulation scene optimization model is used to render each real image in the sample road video data to obtain the corresponding simulation images.

[0036] Here, the model is optimized using an initial simulation scene. Based on an initial Gaussian tuple, each real image in the sample road video data is rendered to generate corresponding simulation images. This process involves giving a standardized perspective (such as the camera intrinsic and extrinsic parameters of a certain vehicle type or a publicly available dataset). All Gaussian tuples apply the visual attributes of the Gaussian tuples currently learned by the model to the rendering process, generating simulation images corresponding to the real images, thus providing a foundation for subsequent loss calculations. Through this step, the model can generate simulation images similar to the real scene based on the geometric and visual attributes of the Gaussian tuples, thereby providing a reference for optimizing model parameters.

[0037] Step c2: Determine the reconstruction loss function of the initial simulation scene optimization model based on the differences between each rendered simulation image and the corresponding real images before rendering.

[0038] Here, the reconstruction loss function of the initial simulation scene optimization model is determined based on the differences between each rendered simulation image and its corresponding unrendered real image. Specifically, it quantifies the similarity between the model-generated simulation image and the real image by calculating the differences between the simulation image and the real image. This loss function reflects the model's ability to reproduce the real scene in the current state; the smaller the value, the closer the model-generated simulation image is to the real image, and the better the model's reconstruction performance. By minimizing this reconstruction loss function, the model can continuously adjust its parameters, optimize the visual attributes of Gaussian units, and thus improve the realism and fidelity of the generated simulation scene. This process is the core of model optimization, ensuring that the generated simulation scene is visually highly consistent with the real scene, thereby meeting the requirements of end-to-end simulation testing for high-quality simulation scenes.

[0039] The specific calculation formula is as follows: ; in, This represents the total number of samples in the sampled road video data, which is also the total number of image frames. Indicates the first in the sample road video data The actual observation data corresponding to the real image of the frame, and Then it means that the model generates the corresponding first... Predicted output of the simulated image of the frame. Let Gausky's set of elements be represented. The reconstruction loss function is obtained by calculating the L1 norm (Manhattan distance) between the simulated and real images across all frames and averaging the results. This loss function measures the overall similarity between the simulated image and the real image; the smaller the value, the closer the simulated image is to the real image.

[0040] Furthermore, the geometric consistency loss function includes depth Figure 1 Geometric consistency loss function and normal consistency loss function; determine the geometric consistency loss function of the initial simulation scene optimization model according to the following steps: Step d1: Based on the difference between the depth maps of each rendered simulation image and the depth maps of the corresponding unrendered real images at the same target viewpoint, determine the depth of the initial simulation scene optimization model. Figure 1 Consistency loss function.

[0041] Here, when determining the geometric consistency loss function of the initial simulation scene optimization model, the depth is determined by comparing the differences between the depth maps of each rendered simulation image under the target viewpoint and the depth maps of the corresponding unrendered real images under the same target viewpoint. Figure 1 Consistency loss function. The depth map provides information about the distance of objects in the scene from the camera. By calculating the difference between the simulated depth map and the real depth map, the model's performance in reconstructing the 3D structure of the scene can be evaluated. Depth Figure 1 The formula for calculating the consistency loss function is as follows: ; in, This represents the total number of frames in the sample road video data. Indicates the first The depth map of the real image of the frame, and Then it means the corresponding first The depth map of the simulated image for each frame. The depth is obtained by calculating the L1 norm between the depth maps of the real and simulated images for all frames, and averaging the results to stabilize the loss function. Figure 1 Consistency loss function .

[0042] Step d2: Based on the difference between the normal directions of each rendered simulation image and the normal directions of the corresponding real images before rendering, determine the normal consistency loss function of the initial simulation scene optimization model.

[0043] Here, the normal consistency loss function is determined by comparing the differences between the normal directions of the model surface in each rendered simulation image and the normal directions estimated by traditional algorithms in the corresponding unrendered real images. The normal direction reflects the orientation of the scene surface and is crucial for the realism of lighting and reflection. By comparing the normal directions of simulated and real images, the accuracy of the model in capturing scene surface details and lighting effects can be evaluated. The formula for calculating the normal consistency loss function is as follows: ; in, Indicates the first The normal of the real image of the frame, and Then it means the corresponding first The normals of the simulated image in each frame are calculated. The L1 norm between the normals of the real image and the normals of the simulated image in all frames is calculated, and the average value is taken to make the loss function more stable, thus obtaining the normal consistency loss function. .

[0044] Step d3, for the depth Figure 1 The geometric consistency loss function of the initial simulation scene optimization model is obtained by weighted averaging the consistency loss function and the normal consistency loss function.

[0045] Here, regarding depth Figure 1 Consistency loss function Consistency loss function with normal We obtain the geometric consistency loss function by performing a weighted average. This comprehensive geometric consistency loss function combines the differences in depth and normals, providing a comprehensive geometric evaluation metric for model optimization. In this way, the model can consider both depth and normal information during training, thereby more accurately reconstructing the scene's geometry and improving the realism and fidelity of the generated simulation scene. The formula for calculating the geometric consistency loss function is as follows: ; in, and They are depths Figure 1 The weighting factors for consistency loss and normal consistency loss are used to balance their contributions to the geometric consistency loss function.

[0046] Furthermore, the prior loss function includes an opacity regularization loss function, a scale regularization loss function, and a geometric consistency loss function; the prior loss function of the initial simulation scene optimization model is determined according to the following steps: Step e1: Determine the opacity regularization loss function of the initial simulation scene optimization model based on the opacity attribute values ​​of each Gaussian element in the initial Gaussian element set.

[0047] When determining the prior loss function for the initial simulation scene optimization model, the opacity regularization loss function is determined by evaluating the opacity attribute value of each Gaussian element in the initial Gaussian element set. The opacity regularization loss function aims to force the model to make a binary decision: either completely transparent or completely opaque, thereby avoiding semi-transparent states and ensuring clean boundaries, which contributes to generating more realistic visual effects. The formula for calculating the opacity regularization loss function is as follows: ; in, This represents the total number of Gaussian elements. Indicates the first The opacity attribute of a Gaussian unit, where 0 represents complete transparency and 1 represents complete opacity, is one of the visual attributes that neural network models need to optimize. It is a parabola opening downwards, when When the variable is 0 or 1, the function value is 0; when... At a value of 0.5, the function reaches its maximum value of 0.25. This formula quantifies the degree of regularization of opacity by calculating the product of the opacity value of each Gaussian unit and its complement; the smaller the value, the closer the opacity of the Gaussian unit is to 0 or 1.

[0048] Step e2: Determine the scale regularization loss function of the initial simulation scene optimization model based on the scaling attribute tensor of each Gaussian in the initial Gaussian set.

[0049] Here, the scale regularization loss function is determined by evaluating the scaling attribute tensor of each Gaussian primitive in the initial Gaussian primitive set. The scale regularization loss function prevents Gaussian primitives from being too large or too small. Oversized primitives can lead to model blurring, while undersized primitives can increase unnecessary computational burden and potentially affect rendering efficiency. Therefore, this loss function aims to maintain a reasonable Gaussian primitive size to ensure that the generated simulation scene remains visually plausible and stable. The formula for calculating the scale regularization loss function is as follows: ; in, Indicates the first Each Gaussian cell has a scaling attribute tensor, which is a three-dimensional vector representing the scaling ratio along three axes. This is one of the visual attributes that the neural network model aims to optimize. By calculating the square of the L2 norm of the logarithm of the scaling attribute tensor for each Gaussian cell, a stronger penalty is applied to large deviations. This involves taking the logarithm of the scaling attribute tensor to prevent it from becoming zero or negative. The formula quantifies the regularization of the Gaussian cell scale; a smaller value indicates that the Gaussian cell size is closer to the preset reasonable range.

[0050] Step e3: Perform a weighted average of the opacity regularization loss function, the regularization loss function, and the geometric consistency loss function to obtain the prior loss function of the initial simulation scene optimization model.

[0051] Here, the opacity regularization loss function Scale regularization loss function and geometric consistency loss function We obtain the prior loss function by performing a weighted average. This comprehensive prior loss function combines differences in opacity, scale, and geometric consistency, providing a holistic regularization metric for model optimization. In this way, the model can consider these factors simultaneously during training, resulting in more stable and realistic simulation scenes. The formula for calculating the prior loss function is as follows: ; in, , and These are the weighting factors for the opacity regularization loss, scale regularization loss, and geometric consistency loss, respectively, used to balance the contributions of these three factors to the prior loss function.

[0052] Further, the temporal smoothing loss function and spatial smoothing loss function of the initial simulation scenario optimization model are determined according to the following steps: Step f1: Determine the temporal smoothing loss function of the initial simulation scene optimization model based on the parameter changes of the visual attributes of the commonly visible Gaussian units in each pair of temporally consecutive simulation images in the rendered simulation images.

[0053] Here, when determining the temporal smoothing loss function for the initial simulation scene optimization model, the temporal smoothing loss function is determined by evaluating the parameter changes of the visual attributes of commonly visible Gaussian units in each pair of temporally consecutive simulation images after rendering. The temporal smoothing loss function aims to ensure that the visual attributes of Gaussian units change smoothly between consecutive frames, thereby generating more natural and continuous dynamic scenes. The formula for calculating the temporal smoothing loss function is as follows: ; in, This represents the total number of frames in the sample road video data. This represents the sum of the number of all frames from the first frame to the second-to-last frame. Indicates the first Frame and the The set of Gaussian elements that are commonly visible in a frame. and They represent the first The first Gaussian Yuan in the Frame and the The visual attribute parameter vector of the frame. By calculating the changes in the visual attribute parameters of the commonly visible Gaussian elements in each pair of consecutive frames and taking the average value to make the loss function more stable, the temporal smoothing loss function is obtained. .

[0054] Step f2: Determine the spatial smoothing loss function of the initial simulation scene optimization model based on the color difference and normal difference between each Gaussian element in the initial Gaussian element set and each Gaussian element in the neighborhood set of the Gaussian element.

[0055] Here, the spatial smoothing loss function is determined by evaluating the color and normal differences between each Gaussian element in the initial Gaussian element set and the Gaussian elements in the neighborhood set of those Gaussian elements. The spatial smoothing loss function aims to ensure that the visual attribute changes of Gaussian elements in space are smooth, thereby generating a more natural and continuous static scene. The formula for calculating the spatial smoothing loss function is as follows: ; in, This represents the total number of Gaussian elements. Indicates the first A neighborhood set of Gaussian elements, and They represent the first The and the first Each high-order color value, and They represent the first The and the first The normal vector of Gaussian elements, This is a weighting factor for the normal difference. The spatial smoothing loss function is obtained by calculating the color difference and normal difference between each Gaussian element and its neighborhood Gaussian elements, and then averaging them. .

[0056] Furthermore, the total loss function During the optimization process, only the parameters of the visual attributes are variable, while the gradients of the geometric attributes are set to 0. The optimization process can be represented as follows: ; in, Represents the visual attribute parameters of all Gaussian elements (such as color, opacity, scaling attribute tensors, etc.). This makes the total loss The set of optimal visual attribute parameters that is minimized.

[0057] This formula shows that the essence of the optimization process is to find a set of visual attribute parameters that make the rendered image most closely resemble the real image by minimizing the total loss function.

[0058] The optimization constraints can be expressed as: ; to explicitly define gradient constraints on geometric properties, ensuring that geometric properties remain unchanged during optimization.

[0059] in, Representing the Geometric properties of each Gaussian element (such as position, rotation, scaling, etc.); This represents other geometric parameters that need to be fixed (such as the direction of the normal).

[0060] The formula requires the total loss function For all geometric properties and The partial derivative is 0, which means that the gradient of the geometric properties will not be updated during the optimization process. That is, the geometric properties remain consistent before and after optimization, and only the visual properties are adjusted.

[0061] The following example illustrates the adjustment of visual attributes.

[0062] Taking the training of the reconstruction loss function as an example, the color value is adjusted from RGB(0.8,0.2,0.2) to RGB(0.9,0.15,0.15), the opacity is adjusted from 0.7 to 0.85, and the scaling attribute tensor is adjusted from (1.2,1.2,1.2) to (1.1,1.1,1.1), making the rendered simulated image closer to the real image and improving the SSIM index from 0.85 to 0.92.

[0063] Taking the training of the geometric consistency loss function as an example, the position is fine-tuned from (0.5, 1.2, 0.3) to (0.52, 1.18, 0.31), the rotation is adjusted from the quaternion (0.7, 0.1, 0.1, 0.7) to (0.71, 0.09, 0.11, 0.69), and the scaling attribute tensor is adjusted from (1.0, 1.0, 1.0) to (1.05, 0.95, 1.0). This reduces the model's depth map error from 0.12 to 0.08 and the normal direction error from 15° to 8°.

[0064] Taking the training of the prior loss function as an example, the opacity is adjusted from 0.45 to 0.1 (close to complete transparency), and the scaling attribute tensor is adjusted from (3.0, 3.0, 0.1) to (1.5, 1.5, 0.8) to avoid the primitives being in a semi-transparent state, prevent the primitive size from being too large or too small, and improve the stability of the model.

[0065] Taking the temporal smoothing loss function as an example, the color value changes from RGB(0.9,0.1,0.1) in frame k to RGB(0.85,0.15,0.15) in frame k+1. The opacity is adjusted from 0.8 to 0.82, and the scaling attribute tensor is adjusted from (1.0,1.0,1.0) to (1.02,0.98,1.0). This reduces the amplitude of visual attribute changes between adjacent frames, avoids flickering and abrupt changes, and improves video smoothness.

[0066] Taking the spatial smoothing loss function as an example, the color values ​​of adjacent Gaussian units are adjusted from RGB(0.9,0.1,0.1) to RGB(0.85,0.15,0.15). The normal vector of the first Gaussian element is adjusted from (0,0,1) to (0.05,0.05,0.99) to approximate the first Gaussian element. The normal vectors of the Gaussian elements (0.1, 0.1, 0.99) are used to eliminate surface noise and isolated fragments, making the model surface more continuous and smooth, and improving visual quality.

[0067] This application provides a simulation scene generation method, comprising: performing mesh modeling on the target road elements based on real physical data of the target road elements to obtain mesh-modeled target road elements; the target road elements include roads and vehicles; performing Gaussian 3D modeling on the mesh-modeled target road elements to obtain an initial Gaussian primitive set for the target simulation scene; inputting the initial Gaussian primitive set and target road video data into a trained simulation scene optimization model to optimize the visual attributes of each Gaussian primitive in the initial Gaussian primitive set, thereby obtaining a target simulation scene corresponding to the target road video data. In this way, this application decouples the geometric structure and visual appearance of objects in the scene, ensures the geometric accuracy of objects through the mesh model, and optimizes the visual attributes of objects through the trained simulation scene optimization model, which can effectively improve the confidence and generation efficiency of simulation scene generation.

[0068] Based on the same application concept, this application also provides a simulation scene generation device corresponding to the simulation scene generation method provided in the above embodiments. Since the principle of the device in this application is similar to the simulation scene generation method in the above embodiments of this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0069] Please see Figure 2 , Figure 2 This is one of the functional block diagrams of a simulation scene generation device provided in an embodiment of this application. For example... Figure 2 As shown, the simulation scene generation device 200 provided in this application embodiment includes: The mesh modeling module 210 is used to perform mesh modeling on the target road elements based on the real physical data of the target road elements, so as to obtain mesh-modeled target road elements; the target road elements include roads and vehicles.

[0070] The Gaussian modeling module 220 is used to perform Gaussian 3D modeling on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene.

[0071] The visual optimization module 230 is used to input the initial Gaussian element set and the target road video data into the trained simulation scene optimization model, optimize the visual attributes of each Gaussian element in the initial Gaussian element set, and obtain the target simulation scene corresponding to the target road video data.

[0072] Furthermore, when the Gaussian modeling module 220 performs Gaussian 3D modeling on the meshed target road elements to obtain the initial Gaussian element set of the target simulation scene, the Gaussian modeling module 220 is specifically used for: Obtain the position coordinates of each triangular mesh vertex of the target road element in the mesh model in three-dimensional space; Based on the respective position coordinates, an initial high-order element is created for each triangular mesh vertex, and the geometric properties of each initial high-order element are determined; Assign initial visual attributes to each of the initial Gaussian elements; the initial visual attributes include initial color, initial opacity, and initial size; Based on the geometric and visual properties of each initial Gaussian element, the initial Gaussian element set of the target simulation scene is determined.

[0073] Further, please refer to Figure 3 , Figure 3 This is a second functional block diagram of a simulation scene generation device provided in an embodiment of this application. Figure 3 As shown, the simulation scene generation device 200 also includes: The model training module 240 is used to train the initial scene optimization model based on the sample road video data and the initial Gaussian set.

[0074] The loss calculation module 250 is used to calculate the reconstruction loss function, geometric consistency loss function, prior loss function, temporal smoothing loss function, and spatial smoothing loss function of the initial scene optimization model during training.

[0075] The model determination module 260 is used to stop training and obtain a trained simulation scene optimization model when the reconstruction loss function, geometric consistency loss function, prior loss function, temporal smoothing loss function and spatial smoothing loss function of the initial simulation scene optimization model all meet the corresponding preset conditions.

[0076] Furthermore, the loss calculation module 250 is used to determine the reconstruction loss function of the initial simulation scene optimization model according to the following steps: Based on the initial Gaussian set, the initial simulation scene optimization model is used to render each real image in the sample road video data to obtain the corresponding simulation images. Based on the differences between each rendered simulation image and the corresponding unrendered real images, the reconstruction loss function of the initial simulation scene optimization model is determined.

[0077] Furthermore, the loss calculation module 250 is used to determine the geometric consistency loss function of the initial simulation scene optimization model according to the following steps: Based on the differences between the depth maps of each rendered simulation image and the depth maps of the corresponding unrendered real images at the same target viewpoint, the depth of the initial simulation scene optimization model is determined. Figure 1 Consistency loss function; Based on the difference between the normal directions of each rendered simulation image and the normal directions of the corresponding real images before rendering, the normal consistency loss function of the initial simulation scene optimization model is determined. For the depth Figure 1 The geometric consistency loss function of the initial simulation scene optimization model is obtained by weighted averaging the consistency loss function and the normal consistency loss function.

[0078] Furthermore, the loss calculation module 250 is used to determine the prior loss function of the initial simulation scene optimization model according to the following steps: Based on the opacity attribute value of each Gaussian element in the initial Gaussian element set, the opacity regularization loss function of the initial simulation scene optimization model is determined. The scaling regularization loss function of the initial simulation scene optimization model is determined based on the scaling attribute tensor of each Gaussian in the initial Gaussian set. The prior loss function of the initial simulation scene optimization model is obtained by weighted averaging the opacity regularization loss function, the regularization loss function, and the geometric consistency loss function.

[0079] Furthermore, the loss calculation module 250 is used to determine the temporal smoothing loss function and spatial smoothing loss function of the initial simulation scenario optimization model according to the following steps: Based on the parameter changes of the visual attributes of the commonly visible Gaussian units in each pair of temporally consecutive simulation images in the rendered simulation images, the temporal smoothing loss function of the initial simulation scene optimization model is determined. Based on the color and normal differences between each Gaussian element in the initial Gaussian element set and each Gaussian element in the neighborhood set of the Gaussian element, the spatial smoothing loss function of the initial simulation scene optimization model is determined.

[0080] This application provides a simulation scene generation device, comprising: a mesh modeling module for performing mesh modeling on the target road elements based on real physical data of the target road elements, resulting in mesh-modeled target road elements; the target road elements include roads and vehicles; a Gaussian modeling module for performing Gaussian 3D modeling on the mesh-modeled target road elements, resulting in an initial Gaussian element set for the target simulation scene; and a visual optimization module for inputting the initial Gaussian element set and target road video data into a trained simulation scene optimization model, optimizing the visual attributes of each Gaussian element in the initial Gaussian element set, and obtaining a target simulation scene corresponding to the target road video data. In this way, this application decouples the geometric structure and visual appearance of objects in the scene, ensures the geometric accuracy of objects through a mesh model, and optimizes the visual attributes of objects through a trained simulation scene optimization model, effectively improving the confidence and generation efficiency of simulation scene generation.

[0081] Based on the same application concept, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0082] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 and the memory 420 communicate through the bus 430. When the machine-readable instructions are executed by the processor 410, the steps of the simulation scene generation method provided in the above embodiment are executed. For specific implementation, please refer to the method embodiment, which will not be repeated here.

[0083] Based on the same concept, this application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the steps of the simulation scene generation method provided in the above embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0085] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0088] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0090] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for generating a simulation scene, characterized in that, The method includes: Based on the real physical data of the target road elements, mesh modeling is performed on the target road elements to obtain mesh-modeled target road elements; the target road elements include roads and vehicles. Gaussian 3D modeling is performed on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene; The initial Gaussian element set and the target road video data are input into the trained simulation scene optimization model to optimize the visual attributes of each Gaussian element in the initial Gaussian element set, thereby obtaining the target simulation scene corresponding to the target road video data.

2. The simulation scene generation method according to claim 1, characterized in that, The step of performing Gaussian 3D modeling on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene includes: Obtain the position coordinates of each triangular mesh vertex of the target road element in the mesh model in three-dimensional space; Based on the aforementioned position coordinates, an initial high-order element is created for each triangular mesh vertex, and the geometric properties of each initial high-order element are determined. Assign initial visual attributes to each of the initial Gaussian elements; the initial visual attributes include initial color, initial opacity, and initial size; Based on the geometric and visual properties of each initial Gaussian element, the initial Gaussian element set of the target simulation scene is determined.

3. The simulation scene generation method according to claim 1, characterized in that, The simulation scenario optimization model is trained according to the following steps: The initial scene optimization model is trained based on the sample road video data and the initial Gaussian set; During training, the reconstruction loss function, geometric consistency loss function, prior loss function, temporal smoothing loss function, and spatial smoothing loss function of the initial scene optimization model are calculated. When the reconstruction loss function, geometric consistency loss function, prior loss function, temporal smoothing loss function, and spatial smoothing loss function of the initial simulation scene optimization model all meet the corresponding preset conditions, training stops, and the trained simulation scene optimization model is obtained.

4. The simulation scene generation method according to claim 3, characterized in that, The reconstruction loss function of the initial simulation scene optimization model is determined according to the following steps: Based on the initial Gaussian set, the initial simulation scene optimization model is used to render each real image in the sample road video data to obtain the corresponding simulation images. Based on the differences between each rendered simulation image and the corresponding unrendered real images, the reconstruction loss function of the initial simulation scene optimization model is determined.

5. The simulation scene generation method according to claim 4, characterized in that, The geometric consistency loss function includes the depth map consistency loss function and the normal consistency loss function; The geometric consistency loss function of the initial simulation scenario optimization model is determined according to the following steps: Based on the difference between the depth map of each rendered simulation image under the target view and the depth map of each corresponding real image before rendering under the target view, the depth map consistency loss function of the initial simulation scene optimization model is determined. Based on the difference between the normal directions of each rendered simulation image and the normal directions of the corresponding real images before rendering, the normal consistency loss function of the initial simulation scene optimization model is determined. The geometric consistency loss function of the initial simulation scene optimization model is obtained by weighted averaging the depth map consistency loss function and the normal consistency loss function.

6. The simulation scene generation method according to claim 5, characterized in that, The prior loss functions include the opacity regularization loss function, the scale regularization loss function, and the geometric consistency loss function; The prior loss function of the initial simulation scenario optimization model is determined according to the following steps: Based on the opacity attribute values ​​of each Gaussian element in the initial Gaussian element set, the opacity regularization loss function of the initial simulation scene optimization model is determined. The scaling regularization loss function of the initial simulation scene optimization model is determined based on the scaling attribute tensor of each Gaussian element in the initial Gaussian element set. The prior loss function of the initial simulation scene optimization model is obtained by weighted averaging the opacity regularization loss function, the regularization loss function, and the geometric consistency loss function.

7. The simulation scene generation method according to claim 4, characterized in that, The temporal smoothing loss function and spatial smoothing loss function of the initial simulation scenario optimization model are determined according to the following steps: Based on the parameter changes of the visual attributes of the commonly visible Gaussian units in each pair of temporally consecutive simulation images in the rendered simulation images, the temporal smoothing loss function of the initial simulation scene optimization model is determined. Based on the color and normal differences between each Gaussian element in the initial Gaussian element set and each Gaussian element in the neighborhood set of the Gaussian element, the spatial smoothing loss function of the initial simulation scene optimization model is determined.

8. A simulation scene generation device, characterized in that, The simulation scene generation device includes: The mesh modeling module is used to perform mesh modeling on the target road elements based on the real physical data of the target road elements, so as to obtain mesh-modeled target road elements; the target road elements include roads and vehicles. The Gaussian modeling module is used to perform Gaussian 3D modeling on the mesh-modeled target road elements to obtain the initial Gaussian element set of the target simulation scene; The visual optimization module is used to input the initial Gaussian element set and the target road video data into the trained simulation scene optimization model, optimize the visual attributes of each Gaussian element in the initial Gaussian element set, and obtain the target simulation scene corresponding to the target road video data.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the simulation scene generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the simulation scene generation method as described in any one of claims 1 to 7.

Images