Simulation Method, Device, Electronic Device and Storage Medium for Virtual Driving Scenarios

Abstract

The present application relates to a method, device, electronic device, and storage medium for simulating a virtual driving scenario. The method includes: obtaining environmental perception data collected in a target driving scenario, where the environmental perception data includes radar point cloud data and multi-view image data; using the radar point cloud data and multi-view image data to construct a dynamic and static Gaussian map corresponding to the target driving scenario, where the dynamic and static Gaussian map includes multiple nodes and edges between the multiple nodes, each node is used to represent an object element and the semantic information corresponding to the object element, and each edge is used to represent the mutual relationship between the object elements located at both ends of the edge; introducing a physical simulation component into the dynamic and static Gaussian map to simulate a virtual driving scenario corresponding to a target weather type, where the target weather type matches the type of the physical simulation component. This can improve the construction efficiency of virtual driving scenarios corresponding to various weather types and reduce the time cost and labor cost.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and particularly to a method, device, electronic device, and storage medium for simulating virtual driving scenarios. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, autonomous driving technology has become a frontier field of research. Autonomous driving systems rely on advanced sensors, computer vision, and deep learning algorithms to achieve precise control of vehicles and improve driving safety and comfort. In order to ensure that autonomous driving systems can also perform well under variable weather conditions, it is necessary to use virtual driving scenarios under various weather conditions to fully test their performance under various weather conditions.

[0003] However, in the prior art, virtual driving scenarios under various weather conditions are usually constructed by obtaining environmental perception data under multiple different weather conditions, which requires a large amount of time and labor costs. Therefore, how to improve the construction efficiency of virtual driving scenarios under various weather conditions has become a technical problem to be solved urgently. Summary of the Invention

[0004] This application provides a method, device, electronic device, and storage medium for simulating virtual driving scenarios to solve the problem that in the prior art, constructing virtual driving scenarios under various weather conditions by obtaining environmental perception data under multiple different weather conditions requires a large amount of time and labor costs.

[0005] In a first aspect, an embodiment of this application provides a method for simulating a virtual driving scenario, the method includes:

[0006] Obtain environmental perception data collected in a target driving scenario, where the environmental perception data includes radar point cloud data and multi-view image data;

[0007] Use the radar point cloud data and the multi-view image data to construct a dynamic and static Gaussian map corresponding to the target driving scenario, where the dynamic and static Gaussian map includes a plurality of nodes and edges between the plurality of nodes, each node is used to represent an object element and the semantic information corresponding to the object element, and each edge is used to represent the mutual relationship between the object elements located at both ends of the edge;

[0008] Introduce a physical simulation component into the dynamic and static Gaussian map to simulate a virtual driving scenario corresponding to a target weather type, where the target weather type matches the type of the physical simulation component.

[0009] Optionally, constructing the dynamic and static Gaussian map corresponding to the target driving scenario by using the radar point cloud data and the multi-view image data includes:

[0010] Identifying dynamic foreground elements and static background elements in the target driving scenario by using the radar point cloud data to obtain a dynamic point cloud map and a static point cloud map, where the dynamic point cloud map is a point cloud map formed by the point clouds corresponding to the dynamic foreground elements, and the static point cloud map is a point cloud map formed by the point clouds corresponding to the static background elements;

[0011] Respectively determining the semantic information corresponding to the dynamic foreground elements and the semantic information corresponding to the static background elements by using the multi-view image data;

[0012] Constructing a dynamic Gaussian map based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and constructing a static Gaussian map based on the static point cloud map and the semantic information corresponding to the static background elements, where each node in the dynamic Gaussian map represents a dynamic foreground element, and each node in the static Gaussian map represents a static background element;

[0013] Merging the dynamic Gaussian map and the static Gaussian map to obtain a dynamic and static Gaussian map.

[0014] Optionally, the radar point cloud data includes lidar point cloud data and millimeter-wave radar point cloud data;

[0015] The identifying dynamic foreground elements and static background elements in the target driving scenario by using the radar point cloud data to obtain a dynamic point cloud map and a static point cloud map includes:

[0016] Fusing the dynamic point clouds in the lidar point cloud data and the millimeter-wave radar point cloud data according to a first preset weight ratio to generate a dynamic point cloud map, where the weight of the millimeter-wave radar point data in the first preset weight ratio is higher than the weight of the lidar point cloud data;

[0017] Fusing the static point clouds in the lidar point cloud data and the millimeter-wave radar point cloud data according to a second preset weight ratio to generate a static point cloud map, where the weight of the millimeter-wave radar point data in the second preset weight ratio is lower than the weight of the lidar point cloud data.

[0018] Optionally, before constructing the dynamic Gaussian map based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and constructing the static Gaussian map based on the static point cloud map and the semantic information corresponding to the static background elements, the method further includes:

[0019] Calculate the first variance of the lidar point cloud and the second variance of the millimeter-wave radar point cloud in each region of two adjacent frames of the dynamic point cloud maps among multiple frames of the dynamic point cloud maps, and dynamically adjust the weight ratio of each region according to the change rates of the first variance and the second variance to obtain a new dynamic point cloud map;

[0020] Calculate the third variance of the lidar point cloud and the fourth variance of the millimeter-wave radar point cloud in each region of two adjacent frames of the static point cloud maps among multiple frames of the static point cloud maps, and dynamically adjust the weight ratio of each region according to the change rates of the third variance and the fourth variance to obtain a new static point cloud map;

[0021] Constructing a dynamic Gaussian map based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and constructing a static Gaussian map based on the static point cloud map and the semantic information corresponding to the static background elements, includes:

[0022] Construct the dynamic Gaussian map based on the new dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and construct the static Gaussian map based on the new static point cloud map and the semantic information corresponding to the static background elements.

[0023] Optionally, introducing a physical simulation component into the dynamic and static Gaussian maps to simulate a virtual driving scenario corresponding to a target weather type, including:

[0024] Determine the target regions in the dynamic and static Gaussian maps where the physical simulation component needs to be introduced, where the physical simulation component includes at least one of a smoke simulation component, a water accumulation simulation component, and a snow accumulation simulation component, and the smoke simulation component, the water accumulation simulation component, and the snow accumulation simulation component are all represented by a series of Gaussian spheres in the dynamic and static Gaussian maps;

[0025] Obtain the parameter values of the physical simulation component, where the parameter values of the physical simulation component include the distribution density and color of the Gaussian spheres corresponding to the physical simulation component in the target region;

[0026] Introduce the physical simulation component into the target region and render the target region according to the parameter values of the physical simulation component to simulate a virtual driving scenario corresponding to the target weather type.

[0027] Optionally, when the physical simulation component is the smoke simulation component, the parameter values of the physical simulation component further include the internal density of each Gaussian sphere in the smoke simulation component, and the internal density of each Gaussian sphere in the smoke simulation component is determined by a value input by the user;

[0028] When the physical simulation component is the ponding simulation component, the parameter values of the physical simulation component further include the internal density of each Gaussian sphere in the ponding simulation component, and the internal density of each Gaussian sphere in the ponding simulation component is determined according to the water depth and ripple intensity at the position where each Gaussian sphere is located;

[0029] When the physical simulation component is the snow accumulation simulation component, the parameter values of the physical simulation component further include the internal density of each Gaussian sphere in the snow accumulation simulation component, and the internal density of each Gaussian sphere in the snow accumulation simulation component gradually decreases along its respective radial direction starting from the center point of each Gaussian sphere.

[0030] Optionally, after introducing a physical simulation component into the dynamic and static Gaussian map to simulate a virtual driving scenario corresponding to a target weather type, the method further includes:

[0031] Receiving a deletion operation on a first object element, and in response to the deletion operation, deleting the first object element from the virtual driving scenario; or,

[0032] Receiving an insertion operation on a second object element, and in response to the insertion operation, inserting the second object element into the virtual driving scenario; or,

[0033] Receiving an editing operation on the semantic label of a third object element, and in response to the editing operation, adjusting the position or color of the third object element.

[0034] In a second aspect, an embodiment of the present application further provides a simulation device for a virtual driving scenario, the device includes:

[0035] An acquisition module, configured to acquire environmental perception data collected in a target driving scenario, where the environmental perception data includes radar point cloud data and multi-view image data;

[0036] A construction module, configured to use the radar point cloud data and the multi-view image data to construct a dynamic and static Gaussian map corresponding to the target driving scenario, where the dynamic and static Gaussian map includes a plurality of nodes and edges between the plurality of nodes, each node is used to represent an object element and the semantic information corresponding to the object element, and each edge is used to represent the mutual relationship between the object elements located at both ends of the edge;

[0037] A simulation module, configured to introduce a physical simulation component into the dynamic and static Gaussian map to simulate a virtual driving scenario corresponding to a target weather type, where the target weather type matches the type of the physical simulation component.

[0038] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0039] The memory is used to store a computer program;

[0040] The processor is configured to, when executing the program stored on the memory, implement the method for simulating a virtual driving scenario according to any one of the first aspects.

[0041] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for simulating a virtual driving scenario according to any one of the first aspects is implemented.

[0042] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art: In the method provided by the embodiments of the present application, by acquiring the environmental perception data collected in the target driving scenario, where the environmental perception data includes radar point cloud data and multi-view image data; using the radar point cloud data and the multi-view image data, a dynamic and static Gaussian map corresponding to the target driving scenario is constructed, where the dynamic and static Gaussian map includes a plurality of nodes and the edges between the plurality of nodes, and each node is used to represent an object element and the semantic information corresponding to the object element, and each edge is used to represent the mutual relationship between the object elements located at both ends of the edge; a physical simulation component is introduced into the dynamic and static Gaussian map to simulate a virtual driving scenario corresponding to the target weather type, where the target weather type matches the type of the physical simulation component. In the above manner, the environmental perception data collected in the target driving scenario can be used to construct a dynamic and static Gaussian map, and then different physical simulation components can be introduced into the dynamic and static Gaussian map, so that virtual driving scenarios corresponding to different weather types can be simulated, avoiding the need to use environmental perception data under multiple different weather conditions to separately construct virtual driving scenarios corresponding to different weather types, thereby improving the construction efficiency of virtual driving scenarios corresponding to each weather type and reducing the time cost and labor cost required for constructing each virtual driving scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present invention and used together with the description to explain the principles of the present invention.

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] One or more embodiments are exemplarily illustrated by the pictures in the corresponding accompanying drawings. These exemplary illustrations do not limit the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the drawings in the figures do not constitute a proportional limitation.

[0046] Figure 1 It is a schematic flowchart of a method for simulating a virtual driving scenario provided by an embodiment of the present application;

[0047] Figure 2 It is a schematic structural diagram of an autonomous driving system provided by an embodiment of the present application;

[0048] Figure 3 It is a schematic diagram of the processing logic of a scenario construction module provided by an embodiment of the present application;

[0049] Figure 4 It is a schematic structural diagram of a camera dirt detection device provided by an embodiment of the present application;

[0050] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0052] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0053] In order to solve the problem that in the prior art, obtaining environmental perception data under various different weather conditions to respectively construct virtual driving scenarios under each weather condition requires a large amount of time cost and labor cost, the present application provides a method, device, electronic device and storage medium for simulating a virtual driving scenario, which can improve the construction efficiency of virtual driving scenarios corresponding to each weather type.

[0054] See Figure 1 , Figure 1 which is a schematic flowchart of a method for simulating a virtual driving scenario provided by an embodiment of the present application. As Figure 1 shown, the method for simulating the virtual driving scenario may include the following steps:

[0055] Step 101, obtain environmental perception data collected in a target driving scenario, where the environmental perception data includes radar point cloud data and multi-view image data.

[0056] Specifically, the above target driving scenario may be any driving scenario, and the embodiments of the present application do not make specific limitations. The above environmental perception data may be pre-collected or collected in real time. The above environmental perception data may include radar point cloud data and multi-view image data, where the radar point cloud data may be millimeter-wave radar point cloud collected by using one or more millimeter-wave radars, or lidar point cloud collected by using one or more lidars. The multi-view image data may be image data collected by using multi-angle cameras.

[0057] Step 102, use the radar point cloud data and multi-view image data to construct a dynamic and static Gaussian map corresponding to the target driving scenario, where the dynamic and static Gaussian map includes a plurality of nodes and edges between the plurality of nodes, and each node is used to represent an object element and the semantic information corresponding to the object element, and each edge is used to represent the mutual relationship between the object elements located at both ends of the edge.

[0058] Specifically, the above dynamic and static Gaussian map is constructed by using three-dimensional (3D) Gaussian splashing technology. The dynamic and static Gaussian map may include a plurality of nodes and edges between the plurality of nodes, and each node is used to represent an object element and the semantic information corresponding to the object element, and each edge is used to represent the mutual relationship between the object elements located at both ends of the edge. Among them, each node includes a plurality of Gaussian spheres corresponding to its object element, and the semantic information of each Gaussian sphere may include position P=(x, y, z), covariance matrix ∑, spherical harmonic coefficients of color C related to the view, and opacity α.

[0059] When constructing a dynamic and static Gaussian map corresponding to a target driving scenario using radar point cloud data and multi-view image data, first, the dynamic foreground elements and static background elements in the target driving scenario can be identified using the radar point cloud data and / or multi-view image data. Then, the point clouds corresponding to each dynamic foreground element and the point clouds corresponding to each static background element can be determined from the radar point cloud data, and the semantic information corresponding to each dynamic foreground element and the semantic information corresponding to each static background element can be determined from the multi-view image data. Finally, based on the point clouds corresponding to each dynamic foreground element, the point clouds corresponding to each static background element, the semantic information corresponding to each dynamic foreground element, and the semantic information corresponding to each static background element, a dynamic and static Gaussian map is constructed. Of course, when constructing the dynamic and static Gaussian map, after determining the point clouds corresponding to each dynamic foreground element and the point clouds corresponding to each static background element from the radar point cloud data, and determining the semantic information corresponding to each dynamic foreground element and the semantic information corresponding to each static background element from the multi-view image data, a dynamic Gaussian map can be constructed according to the point clouds corresponding to each dynamic foreground element and the semantic information corresponding to each dynamic foreground element, and a static Gaussian map can be constructed according to the point clouds corresponding to each static background element and the semantic information corresponding to each static background element. Finally, the dynamic Gaussian map and the static Gaussian map are merged to obtain the dynamic and static Gaussian map. Of course, other methods can also be used to construct the dynamic and static Gaussian map, and the embodiments of the present application do not make specific limitations.

[0060] Step 103: Introduce a physical simulation component into the dynamic and static Gaussian map to simulate a virtual driving scenario corresponding to the target weather type, where the target weather type matches the type of the physical simulation component.

[0061] Specifically, the above physical simulation component refers to a simulation component that can be introduced into the dynamic and static Gaussian map. The physical simulation component can be represented by a series of Gaussian spheres in the dynamic and static Gaussian map, and the physical simulation component is used to simulate different weather effects in the virtual driving scenario. Specifically, the physical simulation component can be a combination of one or more of a smoke simulation component, a water accumulation simulation component, and a snow accumulation simulation component. When the physical simulation component is a smoke simulation component, it can use a series of Gaussian spheres to simulate smoke, and each Gaussian sphere is used to simulate a smoke particle. When the physical simulation component is a water accumulation simulation component, it can use a series of Gaussian spheres to simulate water accumulation, and each Gaussian sphere is used to simulate a water droplet. When the physical simulation component is a snow accumulation simulation component, it can use a series of Gaussian spheres to simulate snow accumulation, and each Gaussian sphere is used to simulate a snow seed.

[0062] In this embodiment, the environmental perception data collected in the target driving scenario can be used to construct a dynamic and static Gaussian map, and then different physical simulation components can be introduced into the dynamic and static Gaussian map, so that virtual driving scenarios corresponding to different weather types can be simulated, avoiding the need to use environmental perception data under a variety of different weather conditions to separately construct virtual driving scenarios corresponding to different weather types, thereby improving the construction efficiency of virtual driving scenarios corresponding to each weather type and reducing the time cost and labor cost required for constructing each virtual driving scenario.

[0063] Further, step 102 above, using the radar point cloud data and multi-view image data to construct a dynamic and static Gaussian map corresponding to the target driving scenario, includes:

[0064] Using the radar point cloud data to identify the dynamic foreground elements and static background elements in the target driving scenario to obtain a dynamic point cloud map and a static point cloud map, where the dynamic point cloud map is a point cloud map formed by the point clouds corresponding to the dynamic foreground elements, and the static point cloud map is a point cloud map formed by the point clouds corresponding to the static background elements;

[0065] Using the multi-view image data to respectively determine the semantic information corresponding to the dynamic foreground elements and the semantic information corresponding to the static background elements;

[0066] Based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, construct a dynamic Gaussian map, and based on the static point cloud map and the semantic information corresponding to the static background elements, construct a static Gaussian map, where each node in the dynamic Gaussian map represents a dynamic foreground element, and each node in the static Gaussian map represents a static background element;

[0067] Merge the dynamic Gaussian map and the static Gaussian map to obtain a dynamic and static Gaussian map.

[0068] In one embodiment, when constructing the dynamic and static Gaussian map, the radar point cloud data can be first used to identify the dynamic foreground elements and static background elements in the target driving scenario to obtain a dynamic point cloud map and a static point cloud map, then the multi-view image data can be used to respectively determine the semantic information corresponding to the dynamic foreground elements and the semantic information corresponding to the static background elements, then based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, construct a dynamic Gaussian map, and based on the static point cloud map and the semantic information corresponding to the static background elements, construct a static Gaussian map, and finally merge the dynamic Gaussian map and the static Gaussian map to obtain a dynamic and static Gaussian map. Through this dynamic and static Gaussian map, complex scenes can be effectively processed, high-quality view synthesis can be provided, and the consistency between multiple cameras can be maintained, which is crucial for the development and testing of autonomous driving systems because it allows the system to consider various possible interactions and events when simulating various traffic situations.

[0069] It should be noted that when identifying dynamic foreground elements and static background elements in the target driving scenario using lidar point cloud data, the Doppler effect of the radar can be directly used to identify dynamic foreground elements and static background elements from lidar point cloud data or millimeter-wave radar point cloud data. Alternatively, after fusing the lidar point cloud data and millimeter-wave radar point cloud data according to a certain weight ratio, the Doppler effect of the radar can be used to identify dynamic foreground elements and static background elements from the fused point cloud data.

[0070] When using multi-view image data to determine the semantic information corresponding to dynamic foreground elements and the semantic information corresponding to static background elements, the Segment Anything Model (SAM for short) can be used for semantic segmentation to accurately segment the regions corresponding to different objects in the image. Then, the frame difference method is used to compare the pixel differences between consecutive frames to quickly detect moving objects in the image, thereby determining the dynamic foreground elements and static background elements in the image, as well as the semantic information corresponding to each element.

[0071] When constructing a dynamic Gaussian map based on the dynamic point cloud map and the semantic information corresponding to dynamic foreground elements, each dynamic foreground element can be used as a node, and edges between nodes are created according to the mutual relationships between the dynamic foreground elements to obtain a dynamic Gaussian map. Similarly, when constructing a static Gaussian map based on the static point cloud map and the semantic information corresponding to static background elements, each static background element can be used as a node, and edges between nodes are created according to the mutual relationships between the static background elements to obtain a static Gaussian map.

[0072] Each node in the Gaussian map is associated with a Gaussian sphere model (i.e., a model composed of a series of Gaussian spheres) to capture the shape, color, and motion trajectory of the object elements. In the Gaussian map, for any Gaussian sphere in space, it can be represented by the following formula:

[0073] L GS (P,θ,φ)=∑ i G(x i ,y i ,z i ,μ i ,Σ i )·c i ;

[0074] where μ i represents the initial center point of the i-th Gaussian sphere, and this initial center point can be determined using lidar point cloud data; P=(x i ,y i ,z i ), P represents the position of the i-th Gaussian sphere; Σ irepresents the covariance matrix of the i-th Gaussian sphere; c i represents the spherical harmonic coefficient of the i-th Gaussian sphere, c i = C(r i , g i , b i , α i , θ i , Φ i ), where r i , g i , b i represents the color attribute of the i-th Gaussian sphere, α i represents the opacity of the i-th Gaussian sphere, θ i and Φ i represent the direction of the i-th Gaussian sphere in space, usually the azimuth angle and elevation angle in the spherical coordinate system, which are used to define the direction characteristics of the Gaussian sphere. When creating a Gaussian map, the scene information of the new area needs to be gradually integrated into the existing Gaussian map. When integrating the scene information of the new area, an incremental learning strategy can be adopted to gradually update the Gaussian map to adapt to the dynamic changes of the scene. This integration process uses the dynamic and static image data from multiple cameras around the vehicle as the supervision signal to ensure the global consistency of the scene.

[0075] In this embodiment, the radar point cloud data and multi-view image data can be used to accurately construct the dynamic and static Gaussian map corresponding to the target driving scene, which is convenient for subsequent editing based on the dynamic and static Gaussian map to obtain a variety of different virtual driving scenes.

[0076] Furthermore, the above radar point cloud data may include lidar point cloud data and millimeter wave radar point cloud data;

[0077] The above steps of using the radar point cloud data to identify the dynamic foreground elements and static background elements in the target driving scene to obtain the dynamic point cloud map and the static point cloud map include:

[0078] Fusing the dynamic point clouds in the lidar point cloud data and the millimeter wave radar point cloud data according to the first preset weight ratio to generate a dynamic point cloud map, where the weight of the millimeter wave radar point data in the first preset weight ratio is higher than the weight of the lidar point cloud data;

[0079] Fusing the static point clouds in the lidar point cloud data and the millimeter wave radar point cloud data according to the second preset weight ratio to generate a static point cloud map, where the weight of the millimeter wave radar point data in the second preset weight ratio is lower than the weight of the lidar point cloud data.

[0080] Specifically, the above-mentioned first preset weight ratio and the second preset weight ratio can be set according to the actual situation. For example, in the first preset weight ratio, the weight of millimeter-wave radar point data accounts for 0.8, 0.9, etc., and in the second preset weight ratio, the weight of lidar point cloud data accounts for 0.8, 0.9, etc.

[0081] In one embodiment, since lidar point cloud data has high precision and high resolution, but is greatly affected by the environment, lidar point cloud data is more suitable for depicting static background elements; while millimeter-wave radar point data has better robustness to harsh environments, but has lower spatial resolution. Therefore, millimeter-wave radar point data performs better in constructing dynamic foregrounds. Therefore, when generating a dynamic point cloud map, a higher weight can be given to millimeter-wave radar point cloud data because millimeter-wave radar has better performance in detecting dynamic objects and can capture the motion characteristics and position information of objects. When generating a static point cloud map, a higher weight can be given to lidar point cloud data because lidar performs more excellently in obtaining high-precision geometric information of static scenes.

[0082] Through the above method, the respective advantages of lidar point cloud data and millimeter-wave radar point cloud data can be fully utilized, and the two can be fused to improve the overall environmental perception ability.

[0083] Further, before the above steps of constructing a dynamic Gaussian map based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and constructing a static Gaussian map based on the static point cloud map and the semantic information corresponding to the static background elements, the method further includes:

[0084] Calculate the first variance of the lidar point cloud in each region of two adjacent frames of dynamic point cloud maps in multiple frames of dynamic point cloud maps and the second variance of the millimeter-wave radar point cloud in each region, and dynamically adjust the weight ratio of each region according to the change rate of the first variance and the second variance to obtain a new dynamic point cloud map;

[0085] Calculate the third variance of the lidar point cloud in each region of two adjacent frames of static point cloud maps in multiple frames of static point cloud maps and the fourth variance of the millimeter-wave radar point cloud in each region, and dynamically adjust the weight ratio of each region according to the change rate of the third variance and the fourth variance to obtain a new static point cloud map;

[0086] The above steps of constructing a dynamic Gaussian map based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and constructing a static Gaussian map based on the static point cloud map and the semantic information corresponding to the static background elements, include:

[0087] Construct a dynamic Gaussian map based on the new dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and construct a static Gaussian map based on the new static point cloud map and the semantic information corresponding to the static background elements.

[0088] In one embodiment, after fusing the lidar point cloud data and the millimeter-wave radar point cloud data according to a preset weight ratio to generate a dynamic point cloud map and a static point cloud map, it is also possible to calculate the first variance of the lidar point cloud in each region of two adjacent frames of the dynamic point cloud map in multiple frames of the dynamic point cloud map and the second variance of the millimeter-wave radar point cloud in each region, and dynamically adjust the weight ratio of each region according to the change rates of the first variance and the second variance. Specifically, a higher weight can be given to the point cloud data with a smaller variance change rate in each region of the dynamic point cloud map, while a lower weight can be given to the point cloud data with a larger variance change rate, and a new dynamic point cloud map can be obtained according to the adjusted weight ratio. At the same time, it is also possible to calculate the third variance of the lidar point cloud in each region of two adjacent frames of the static point cloud map in multiple frames of the static point cloud map and the fourth variance of the millimeter-wave radar point cloud in each region, and according to the change rates of the third variance and the fourth variance. Specifically, a higher weight can be given to the point cloud data with a smaller variance change rate in each region of the static point cloud map, while a lower weight can be given to the point cloud data with a larger variance change rate, and a new static point cloud map can be obtained according to the adjusted weight ratio.

[0089] After obtaining the new dynamic point cloud map and the new static point cloud map, a dynamic Gaussian map can be reconstructed based on the new dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and a static Gaussian map can be reconstructed based on the new static point cloud map and the semantic information corresponding to the static background elements. The reconstructed dynamic Gaussian map and static Gaussian map are merged to obtain a dynamic and static Gaussian map.

[0090] In this embodiment, considering that the data obtained by the lidar or millimeter-wave radar in special scenarios may have large errors, resulting in significant differences between the lidar point cloud data and the millimeter-wave radar point cloud data in some regions, at this time, the initially obtained multiple frames of dynamic point cloud maps and multiple frames of static point cloud maps can be used to judge the variance change rates of the different radar point clouds in each region, and then the weights of each region can be re-allocated, so that the finally obtained dynamic and static Gaussian map is more accurate.

[0091] Further, in step 103 above, introducing a physical simulation component into the dynamic and static Gaussian map to simulate a virtual driving scenario corresponding to the target weather type includes:

[0092] Determine the target regions in the dynamic and static Gaussian map where the physical simulation component needs to be introduced, where the physical simulation component includes at least one of a smoke simulation component, a water accumulation simulation component, and a snow accumulation simulation component, and the smoke simulation component, the water accumulation simulation component, and the snow accumulation simulation component are all represented by a series of Gaussian spheres in the dynamic and static Gaussian map;

[0093] Obtain the parameter values of the physical simulation component, where the parameter values of the physical simulation component include the distribution density and color of the Gaussian spheres corresponding to the physical simulation component within the target area;

[0094] Introduce the physical simulation component into the target area, and render the target area according to the parameter values of the physical simulation component to simulate the virtual driving scenario corresponding to the target weather type.

[0095] In one embodiment, when simulating the virtual driving scenario, the target area in the dynamic and static Gaussian map where the physical simulation component needs to be introduced can be determined first, and the parameter values of the physical simulation component can be obtained. Specifically, the target area can be determined by the coordinate parameters of the target area input by the user, or can be directly selected by the user using a mouse or the like in the dynamic and static Gaussian map, or can be determined according to the positions of these nodes after the user selects the corresponding nodes in the dynamic and static Gaussian map. The target area can be a partial area in the dynamic and static Gaussian map or the entire area in the dynamic and static Gaussian map, and the embodiments of the present application do not make specific limitations. The parameter values of the physical simulation component can include parameters such as the distribution density and color of the Gaussian spheres corresponding to the physical simulation component within the target area. After determining the target area in the dynamic and static Gaussian map where the physical simulation component needs to be introduced and obtaining the parameter values of the physical simulation component, then introduce the physical simulation component into the target area, and render the target area according to the parameter values of the physical simulation component to simulate the virtual driving scenario corresponding to the target weather type.

[0096] In this embodiment, the user can introduce the physical simulation component into the target area through simple interaction, so that the 3D Gaussian technology can learn parameter information such as the position, density, and color of each Gaussian sphere, and then simulate virtual driving scenarios corresponding to various weathers (such as ponding, smoke, snow accumulation, etc.), thereby improving the construction efficiency of virtual driving scenarios corresponding to each weather type and reducing the time cost and labor cost required to construct each virtual driving scenario.

[0097] Furthermore, when the physical simulation component is a smoke simulation component, the parameter values of the physical simulation component further include the internal density of each Gaussian sphere in the smoke simulation component, and the internal density of each Gaussian sphere in the smoke simulation component is determined by the value input by the user;

[0098] When the physical simulation component is a ponding simulation component, the parameter values of the physical simulation component further include the internal density of each Gaussian sphere in the ponding simulation component, and the internal density of each Gaussian sphere in the ponding simulation component is determined according to the water depth and ripple intensity at the position where each Gaussian sphere is located;

[0099] When the physical simulation component is a snow accumulation simulation component, the parameter values of the physical simulation component further include the internal density of each Gaussian sphere in the snow accumulation simulation component, and the internal density of each Gaussian sphere in the snow accumulation simulation component gradually decreases along its respective radial direction starting from the center point of each Gaussian sphere.

[0100] Specifically, for the smoke simulation component, it can be used to simulate smoke. Since smoke is an aerosol formed by the uniform distribution of countless tiny particles, a series of Gaussian spheres can be used to represent smoke, and each Gaussian sphere represents a smoke particle. Among them, the density of smoke (i.e., the internal density of each Gaussian sphere) can be determined according to the set constant σ smog and the color of the smoke can be obtained by adjusting the color attribute of the Gaussian function, usually a constant color c smog .

[0101] For the ponding simulation component, it can be used to simulate ponding. Since ponding usually exists on a horizontal plane or a surface close to horizontal, and the water surface is often irregular and fluctuates, a series of Gaussian spheres located above the horizontal plane can be used to represent ponding. In order to simulate the water surface information, a normal vector needs to be added to the original Gaussian representation, so as to better process the light information. The height of the water surface is determined by the formula n water (P z - h water ) = 0, where n water represents the normal vector of the gravity direction, which is estimated by the camera pose and the vanishing point detection, and h water represents the position of the water surface, which is determined by the water depth. In addition, the Fast Fourier Transform (FFT) can be used to simulate the ripples on the water surface, which is achieved by superimposing sine waves with different frequencies and amplitudes on the water surface Gaussian function. The Gaussian sphere density of the ponding (i.e., the internal density of each Gaussian sphere in the ponding simulation component) can be adjusted according to the water depth and the intensity of the ripples, and the formula is as follows:

[0102] σ water (P) = σ 0 / (1 + α · FFT(P z - h water ) + β · N(P));

[0103] where, σ 0 is the set basic density, α is a regulation coefficient used to control the degree of influence of the FFT transformation on the density, P is a point in space, and h wateris the position of the water surface. The FFT function encodes the height variation of the water wave into the density of the Gaussian sphere, thereby simulating the irregular water surface fluctuations. N is the noise function, which considers the influence of irregularity on the density of the Gaussian sphere, and β is the adjustment coefficient related to the noise function. In addition, the color of the accumulated water can be obtained by adjusting the color attribute of the Gaussian function, usually a constant color c water .

[0104] For the snow accumulation simulation component, it can be used to simulate snow accumulation. Since snow is more likely to accumulate on the ground and other object surfaces, the snow accumulation can be represented by a series of Gaussian spheres attached to the ground and other object surfaces. The normal vector n can be obtained through camera pose and vanishing point detection snow , and then the area where the snow accumulation Gaussian spheres should be attached is screened through n snow P = 0. The internal density of each Gaussian sphere in the snow accumulation simulation component can be expressed by the following formula:

[0105] σ snow (P) = ∑ i K(r, R, σ o ) = σ o ·exp(-x 2 / 2a 2 -y 2 / 2b 2 -z 2 / 2c 2 );

[0106] where, σ snow is the internal density of each Gaussian sphere in the snow accumulation simulation component, K is the kernel function, a, b, and c are the semi-axis lengths of the ellipsoid along the local x, y, and z axes respectively, x, y, and z are the positions of a certain Gaussian sphere on the x, y, and z axes, and σ o is the set density. In addition, the diffuse reflection color of the snow at position P can be calculated by the formula of the approximate Bidirectional Reflectance Distribution Function (BRDF):

[0107] c snow (P) = c diff ·illum avg (P);

[0108] where, c diff is the diffuse reflection color, and illum avg (P) is the average illumination of the scene at position P.

[0109] In the above manner, on the basis of the 3D Gaussian splash technology, a physical simulation component is introduced to achieve a realistic simulation of physical entities such as smoke, ponding, and snow accumulation, enhancing the realism of the scene.

[0110] Further, after step 103 of introducing a physical simulation component into the dynamic and static Gaussian maps and simulating to obtain a virtual driving scene corresponding to the target weather type, the method further includes:

[0111] Receiving a deletion operation on a first object element, and in response to the deletion operation, deleting the first object element from the virtual driving scene; or,

[0112] Receiving an insertion operation on a second object element, and in response to the insertion operation, inserting the second object element into the virtual driving scene; or,

[0113] Receiving an editing operation on the semantic label of a third object element, and in response to the editing operation, adjusting the position or color of the third object element.

[0114] In one embodiment, after simulating to obtain a virtual driving scene corresponding to the target weather type, it is also possible to receive a deletion operation on a first object element, and in response to the deletion operation, delete the first object element from the virtual driving scene. And it is also possible to receive an insertion operation on a second object element, and in response to the insertion operation, insert the second object element into the virtual driving scene. In this way, the user can freely delete and insert specific object elements in the virtual driving scene.

[0115] In addition, after simulating to obtain a virtual driving scene corresponding to the target weather type, it is also possible to receive an editing operation on the semantic label of a third object element, and in response to the editing operation, adjust the position or color of the third object element. In this way, precise editing of specific Gaussian points can be achieved, such as adjusting the position, color, etc., without affecting other areas.

[0116] It should be noted that for each Gaussian point g in the scene i , a semantic label s can be assigned through a mapping function f i , which is represented by the following formula:

[0117] s i = f(ε i );

[0118] where ε i represents the various parameters of the Gaussian point, f represents the mapping function, and s iRepresents a semantic label. The implementation principle of the mapping function is as follows: The semantic labels corresponding to the image segmentation are mapped back to 3D Gaussian points through an inverse rendering process, and then the semantic labels of each pixel in the SAM segmentation mask are used to determine the semantic labels of the corresponding 3D Gaussian points.

[0119] During the densification process of the scene, the representation of the environment is refined by adding Gaussian spheres. The newly generated Gaussian sphere g' inherits the semantic label of its closest parent Gaussian sphere g: s g′ = s parent(g) . This ensures that as the training progresses, each new Gaussian sphere can accurately reflect its semantic role in the simulated environment.

[0120] Through the above method, precise control and adjustment of the virtual driving scene can be achieved, thereby improving the authenticity and reliability of the simulation. Moreover, this is crucial for the development and testing of autonomous driving algorithms as it allows developers to reproduce and edit various traffic situations and scenarios in the simulated environment.

[0121] In one embodiment, the simulation method of the virtual driving scene provided by the embodiments of the present application is implemented based on an innovative 3D Gaussian splashing technology. This method aims to provide a highly realistic road condition scene for the autonomous driving system. By introducing physical simulation components and performing semantic label editing, it can not only efficiently render three-dimensional scenes under complex weather conditions but also achieve fine editing and adjustment of these scenes. The simulation method of the virtual driving scene can be applied to the autonomous driving system. As Figure 2 shown, the autonomous driving system mainly includes the following four modules:

[0122] 1. Scene construction module: This module uses dynamic and static 3D Gaussian splashing technology to automatically construct a virtual driving scene with rich details and accurate physical properties based on the data collected by on-vehicle cameras and radars. This process significantly reduces the resource consumption required for constructing and maintaining a large-scale virtual scene library in traditional methods, while improving the efficiency and cost-effectiveness of scene construction.

[0123] 2. Scene editing module: Based on the virtual driving scene, the scene editing module allows users to edit and adjust specific semantic regions of the generated scene in an intuitive manner, including simulating various adverse weather road conditions such as waterlogging, fog, and snow. These interactions enable precise control of weather effects and the scene, providing a highly realistic test platform for the evaluation of autonomous driving systems.

[0124] 3. Simulation test module: Utilizing the outputs of the scene construction and scene editing modules, the simulation test module can comprehensively evaluate the performance of the autonomous driving system under simulated adverse weather road conditions.

[0125] 4. Decision Support Module: Based on the data provided by the Simulation Test Module, the Decision Support Module uses data analysis and machine learning algorithms to provide strategies and suggestions for the performance optimization of the autonomous driving system. This module aims to enhance the overall performance of the autonomous driving system through continuous feedback loops, ensuring its safe and effective operation under various weather conditions.

[0126] Specifically, the processing logic of the Scenario Construction Module is as Figure 3 shown. Its core task is to reconstruct an accurate driving road condition scenario from multi-view image data and radar point cloud data through advanced 3D Gaussian splashing technology. This process first involves preprocessing the collected multi-view image data and radar point cloud data to optimize the accuracy of subsequent 3D reconstruction, obtaining the road condition environment. Then, static background elements and dynamic foreground elements are identified based on the road condition environment. After that, static Gaussian maps and dynamic Gaussian maps are respectively constructed based on the static background elements and dynamic foreground elements. Then, the static Gaussian map and the dynamic Gaussian map are merged to obtain a dynamic and static Gaussian map. Moreover, a physically simulated enhanced Gaussian splashing technology is used to introduce physical simulation components into the dynamic and static Gaussian map, and rendering is performed after introducing the physical simulation components to obtain a virtual driving scenario.

[0127] The Scenario Editing Module mainly provides a user interface that allows users to add or modify the effects of adverse weather road conditions through simple interactions. Users can select specific weather effects, such as waterlogging, fog, snow, etc., and can also simulate deleting and inserting specific elements, and adjust the corresponding parameters to simulate the impact degree of these edits on the scenario.

[0128] The Simulation Test Module can use the CARLA (short for Car Learning to Act) simulator to collect autonomous driving and manual driving scenarios, provide access to vehicle sensors, and describe roads according to the OpenDRIVE specification, and is used to evaluate the performance of the autonomous driving system under various adverse weather road condition conditions. During the test, the system simulates various adverse weather road condition conditions, including waterlogging, snow accumulation, and fog, as well as road conditions with the insertion and deletion of specific elements through editing. Through these tests, the perception, decision-making, and execution capabilities of the system under extreme weather conditions can be evaluated to ensure its safety and reliability in the real world. In addition, the Simulation Test Module also supports real-time user interaction, allowing dynamic adjustment of weather conditions during the simulation to evaluate the adaptability and response speed of the autonomous driving system to emergencies.

[0129] The decision support module is the core of the intelligent decision-making of the present invention. It is responsible for analyzing the data collected by the simulation test module and providing strategies and suggestions for the performance optimization of the autonomous driving system. This module adopts advanced data analysis tools and machine learning algorithms to deeply analyze the simulation test results and identify the performance bottlenecks and potential risks of the system under specific weather and driving scenarios. By analyzing the performance of the autonomous driving system in the simulation test, the decision support module can put forward specific optimization suggestions, such as improving the perception algorithm, optimizing the decision-making model, adjusting the control strategy, etc. These suggestions aim to improve the accuracy and reliability of the autonomous driving system under harsh weather and road conditions and reduce potential safety risks. In addition, the decision support module also designs a continuous feedback loop mechanism. By continuously collecting simulation test data and real-world driving data, it continuously adjusts and optimizes the parameters and strategies of the autonomous driving system. This data-driven iterative optimization process helps the autonomous driving system gradually adapt to more complex driving environments and improve its safety and efficiency under various weather conditions.

[0130] Compared with the existing technologies, the present application has the following beneficial effects:

[0131] 1. The present application can effectively simulate driving scenarios under different weather and road conditions, including harsh weather and road conditions such as waterlogging, snow accumulation, and fog, so as to more comprehensively evaluate the performance and robustness of the autonomous driving system.

[0132] 2. The present application combines the processing of static backgrounds and dynamic foregrounds, constructs dynamic and static elements through dynamic and static Gaussian maps, and realizes the accurate capture and rendering of complex intelligent driving scenarios. Therefore, it can reconstruct high-quality three-dimensional scenes, greatly reducing the time and effort for building a virtual scene library and also reducing the maintenance cost.

[0133] 3. By adding semantic tags to Gaussian points, the present application can achieve precise editing of specific Gaussian points, such as adjusting the position, color, or opacity, and can simulate road condition information in various situations to comprehensively evaluate the performance and robustness of the autonomous driving system.

[0134] 4. Based on the 3D Gaussian splashing technology, the present application introduces a physical simulation component to achieve a realistic simulation of physical entities such as smoke, waterlogging, and snow accumulation, enhancing the realism of the scene.

[0135] See Figure 4 , Figure 4 which is a schematic structural diagram of a simulation device for a virtual driving scenario provided by an embodiment of the present application. As Figure 4 shown, the device 400 includes:

[0136] An acquisition module 401, configured to acquire environmental perception data collected in a target driving scenario, where the environmental perception data includes lidar point cloud data and multi-view image data;

[0137] A construction module 402, configured to construct a dynamic and static Gaussian map corresponding to the target driving scenario by using the lidar point cloud data and the multi-view image data, where the dynamic and static Gaussian map includes a plurality of nodes and edges between the plurality of nodes, each node is used to represent an object element and the semantic information corresponding to the object element, and each edge is used to represent the mutual relationship between the object elements located at both ends of the edge;

[0138] A simulation module 403, configured to introduce a physical simulation component into the dynamic and static Gaussian map, and simulate a virtual driving scenario corresponding to a target weather type, where the target weather type matches the type of the physical simulation component.

[0139] Further, the construction module 402 includes:

[0140] An identification sub-module, configured to identify dynamic foreground elements and static background elements in the target driving scenario by using the lidar point cloud data, and obtain a dynamic point cloud map and a static point cloud map, where the dynamic point cloud map is a point cloud map formed by the point clouds corresponding to the dynamic foreground elements, and the static point cloud map is a point cloud map formed by the point clouds corresponding to the static background elements;

[0141] A first determination sub-module, configured to respectively determine the semantic information corresponding to the dynamic foreground elements and the semantic information corresponding to the static background elements by using the multi-view image data;

[0142] A construction sub-module, configured to construct a dynamic Gaussian map based on the dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and construct a static Gaussian map based on the static point cloud map and the semantic information corresponding to the static background elements, where each node in the dynamic Gaussian map represents a dynamic foreground element, and each node in the static Gaussian map represents a static background element;

[0143] A merging sub-module, configured to merge the dynamic Gaussian map and the static Gaussian map to obtain a dynamic and static Gaussian map.

[0144] Further, the lidar point cloud data includes lidar point cloud data and millimeter-wave radar point cloud data; the identification sub-module includes:

[0145] A first generation unit, configured to fuse the dynamic point clouds in the lidar point cloud data and the millimeter-wave radar point cloud data according to a first preset weight ratio to generate a dynamic point cloud map, where the weight of the millimeter-wave radar point data in the first preset weight ratio is higher than the weight of the lidar point cloud data;

[0146] A second generation unit, configured to fuse the static point clouds in the lidar point cloud data and the millimeter-wave radar point cloud data according to a second preset weight ratio to generate a static point cloud map, where the weight of the millimeter-wave radar point data in the second preset weight ratio is lower than the weight of the lidar point cloud data.

[0147] Further, the construction module 402 further includes:

[0148] A first calculation sub-module, configured to calculate the first variance of the lidar point cloud in each region of two adjacent frames of dynamic point cloud maps in multiple frames of dynamic point cloud maps and the second variance of the millimeter-wave radar point cloud in each region, and dynamically adjust the weight ratio of each region according to the change rate of the first variance and the second variance to obtain a new dynamic point cloud map;

[0149] A second calculation sub-module, configured to calculate the third variance of the lidar point cloud in each region of two adjacent frames of static point cloud maps in multiple frames of static point cloud maps and the fourth variance of the millimeter-wave radar point cloud in each region, and dynamically adjust the weight ratio of each region according to the change rate of the third variance and the fourth variance to obtain a new static point cloud map;

[0150] A construction sub-module, further configured to construct a dynamic Gaussian map based on the new dynamic point cloud map and the semantic information corresponding to the dynamic foreground elements, and construct a static Gaussian map based on the new static point cloud map and the semantic information corresponding to the static background elements.

[0151] Further, the simulation module 403 includes:

[0152] A second determination sub-module, configured to determine a target region in the dynamic and static Gaussian maps where a physical simulation component needs to be introduced, where the physical simulation component includes at least one of a smoke simulation component, a water accumulation simulation component, and a snow accumulation simulation component, and the smoke simulation component, the water accumulation simulation component, and the snow accumulation simulation component are all represented by a series of Gaussian spheres in the dynamic and static Gaussian maps;

[0153] An acquisition sub-module, configured to acquire the parameter values of the physical simulation component, where the parameter values of the physical simulation component include the distribution density and color of the Gaussian spheres corresponding to the physical simulation component in the target region;

[0154] A simulation sub-module, configured to introduce the physical simulation component into the target region and render the target region according to the parameter values of the physical simulation component to simulate a virtual driving scenario corresponding to the target weather type.

[0155] Further, when the physical simulation component is a smoke simulation component, the parameter values of the physical simulation component further include the internal density of each Gaussian sphere in the smoke simulation component, and the internal density of each Gaussian sphere in the smoke simulation component is determined by a value input by the user;

[0156] When the physical simulation component is a ponding simulation component, the parameter value of the physical simulation component further includes the internal density of each Gaussian sphere in the ponding simulation component, and the internal density of each Gaussian sphere in the ponding simulation component is determined according to the water depth and ripple intensity at the position where each Gaussian sphere is located;

[0157] When the physical simulation component is a snow accumulation simulation component, the parameter value of the physical simulation component further includes the internal density of each Gaussian sphere in the snow accumulation simulation component, and the internal density of each Gaussian sphere in the snow accumulation simulation component gradually decreases along its respective radial direction starting from the center point of each Gaussian sphere.

[0158] Furthermore, the device 400 further includes:

[0159] A deletion module, configured to receive a deletion operation on a first object element, and in response to the deletion operation, delete the first object element from the virtual driving scene; or,

[0160] An insertion module, configured to receive an insertion operation on a second object element, and in response to the insertion operation, insert the second object element into the virtual driving scene; or,

[0161] An editing module, configured to receive an editing operation on the semantic label of a third object element, and in response to the editing operation, adjust the position or color of the third object element.

[0162] It should be noted that the device 400 can implement the simulation method of the virtual driving scene provided in any of the foregoing method embodiments, and can achieve the same technical effects, which will not be elaborated herein one by one.

[0163] As Figure 5 shown, an embodiment of the present application further provides an electronic device, including a processor 511, a communication interface 512, a memory 513, and a communication bus 514. Among them, the processor 511, the communication interface 512, and the memory 513 complete communication with each other through the communication bus 514.

[0164] The memory 513 is used to store a computer program;

[0165] In an embodiment of the present application, when the processor 511 executes the program stored on the memory 513, it implements the simulation method of the virtual driving scene provided in any of the foregoing method embodiments.

[0166] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the simulation method of the virtual driving scene provided in any of the foregoing method embodiments.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or rather the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0169] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "comprising", "including", "containing", and "having" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be executed in the specific order described or illustrated, unless the execution order is explicitly stated. It should also be understood that additional or alternative steps can be used.

[0170] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for simulating a virtual driving scene, characterized in that: The method comprises: Acquire environmental perception data collected in a target driving scenario, wherein the environmental perception data includes radar point cloud data and multi-view image data; Using the radar point cloud data and the multi-view image data, construct a dynamic and static Gaussian graph corresponding to the target driving scene, wherein the dynamic and static Gaussian graph includes a plurality of nodes and edges between the plurality of nodes, each of the nodes is used to represent an object element and semantic information corresponding to the object element, and each of the edges is used to represent the relationship between the object elements located at both ends of the edge; Introducing a physical simulation component into the dynamic and static Gaussian graph to simulate a virtual driving scene corresponding to a target weather type, wherein the target weather type matches the type of the physical simulation component; The step of constructing a dynamic and static Gaussian graph corresponding to the target driving scene by using the radar point cloud data and the multi-view image data includes: Using the radar point cloud data, dynamic foreground elements and static background elements in the target driving scene are identified to obtain a dynamic point cloud map and a static point cloud map, wherein the dynamic point cloud map is a point cloud map formed by point clouds corresponding to the dynamic foreground elements, and the static point cloud map is a point cloud map formed by point clouds corresponding to the static background elements; Determine the semantic information corresponding to the dynamic foreground element and the semantic information corresponding to the static background element respectively by using the multi-view image data; Based on the semantic information corresponding to the dynamic point cloud image and the dynamic foreground element, a dynamic Gaussian image is constructed, and based on the semantic information corresponding to the static point cloud image and the static background element, a static Gaussian image is constructed, wherein each node in the dynamic Gaussian image represents a dynamic foreground element, and each node in the static Gaussian image represents a static background element; The dynamic Gaussian graph and the static Gaussian graph are combined to obtain a dynamic and static Gaussian graph.

2. The method according to claim 1, characterized in that: The radar point cloud data includes laser radar point cloud data and millimeter wave radar point cloud data; The method of using the radar point cloud data to identify dynamic foreground elements and static background elements in the target driving scene to obtain a dynamic point cloud map and a static point cloud map includes: Fusion of the laser radar point cloud data and the dynamic point clouds in the millimeter wave radar point cloud data according to a first preset weight ratio to generate a dynamic point cloud map, wherein the weight of the millimeter wave radar point data in the first preset weight ratio is higher than the weight of the laser radar point cloud data; The static point clouds in the laser radar point cloud data and the millimeter wave radar point cloud data are fused according to a second preset weight ratio to generate a static point cloud map, wherein the weight of the millimeter wave radar point data in the second preset weight ratio is lower than the weight of the laser radar point cloud data.

3. The method according to claim 2, characterized in that Before constructing a dynamic Gaussian graph based on the dynamic point cloud graph and the semantic information corresponding to the dynamic foreground elements, and constructing a static Gaussian graph based on the static point cloud graph and the semantic information corresponding to the static background elements, the method further includes: Calculate the first variance of the laser radar point cloud in each region of two adjacent frames of the dynamic point cloud image in multiple frames and the second variance of the millimeter wave radar point cloud in each region, and dynamically adjust the weight ratio of each region according to the change rate of the first variance and the second variance to obtain a new dynamic point cloud image; Calculate the third variance of the laser radar point cloud in each area of ​​two adjacent frames of the static point cloud image in the multi-frame static point cloud image and the fourth variance of the millimeter wave radar point cloud in each area, and dynamically adjust the weight ratio of each area according to the change rate of the third variance and the fourth variance to obtain a new static point cloud image; The constructing of a dynamic Gaussian graph based on the dynamic point cloud graph and the semantic information corresponding to the dynamic foreground elements, and the constructing of a static Gaussian graph based on the semantic information corresponding to the static point cloud graph and the static background elements, comprises: The dynamic Gaussian graph is constructed based on the new dynamic point cloud graph and the semantic information corresponding to the dynamic foreground elements, and the static Gaussian graph is constructed based on the new static point cloud graph and the semantic information corresponding to the static background elements.

4. The method according to claim 1, characterized in that: The step of introducing a physical simulation component into the dynamic and static Gaussian graph to simulate and obtain a virtual driving scene corresponding to the target weather type includes: Determine a target area in the dynamic and static Gaussian map where the physical simulation component needs to be introduced, wherein the physical simulation component includes at least one of a smoke simulation component, a water accumulation simulation component, and a snow accumulation simulation component, and the smoke simulation component, the water accumulation simulation component, and the snow accumulation simulation component are all represented by a series of Gaussian balls in the dynamic and static Gaussian map; Acquire a parameter value of the physical simulation component, wherein the parameter value of the physical simulation component includes a distribution density and a color of a Gaussian ball corresponding to the physical simulation component in the target area; The physical simulation component is introduced into the target area, and the target area is rendered according to the parameter value of the physical simulation component to simulate and obtain a virtual driving scene corresponding to the target weather type.

5. The method according to claim 4, characterized in that In the case where the physical simulation component is the smoke simulation component, the parameter value of the physical simulation component further includes the internal density of each Gaussian sphere in the smoke simulation component, and the internal density of each Gaussian sphere in the smoke simulation component is determined by a value input by a user; In the case where the physical simulation component is the water accumulation simulation component, the parameter value of the physical simulation component further includes the internal density of each Gaussian ball in the water accumulation simulation component, and the internal density of each Gaussian ball in the water accumulation simulation component is determined according to the water depth and ripple strength at the location of each Gaussian ball; In the case where the physical simulation component is the snow simulation component, the parameter value of the physical simulation component also includes the internal density of each Gaussian sphere in the snow simulation component, and the internal density of each Gaussian sphere in the snow simulation component gradually decreases along the respective radial directions starting from the center point of each Gaussian sphere.

6. The method according to claim 1, characterized in that After introducing the physical simulation component into the dynamic and static Gaussian graph to simulate and obtain a virtual driving scene corresponding to the target weather type, the method further includes: receiving a deletion operation on a first object element, and deleting the first object element from the virtual driving scene in response to the deletion operation; or, receiving an insert operation on a second object element, and in response to the insert operation, inserting the second object element into the virtual driving scene; or, An editing operation on the semantic label of the third object element is received, and in response to the editing operation, the position or color of the third object element is adjusted.

7. A simulation device for a virtual driving scene, characterized in that: The device comprises: An acquisition module, used to acquire environmental perception data collected in a target driving scene, wherein the environmental perception data includes radar point cloud data and multi-view image data; A construction module, configured to construct a dynamic and static Gaussian graph corresponding to the target driving scene using the radar point cloud data and the multi-view image data, wherein the dynamic and static Gaussian graph includes a plurality of nodes and edges between the plurality of nodes, each of the nodes is used to represent an object element and semantic information corresponding to the object element, and each of the edges is used to represent the relationship between the object elements located at both ends of the edge; A simulation module, used for introducing a physical simulation component into the dynamic and static Gaussian graph to simulate and obtain a virtual driving scene corresponding to a target weather type, wherein the target weather type matches the type of the physical simulation component; Wherein, the building blocks include: an identification submodule, for identifying dynamic foreground elements and static background elements in the target driving scene using the radar point cloud data, and obtaining a dynamic point cloud map and a static point cloud map, wherein the dynamic point cloud map is a point cloud map formed by point clouds corresponding to the dynamic foreground elements, and the static point cloud map is a point cloud map formed by point clouds corresponding to the static background elements; A first determination submodule, configured to respectively determine semantic information corresponding to the dynamic foreground element and semantic information corresponding to the static background element using the multi-view image data; A construction submodule, configured to construct a dynamic Gaussian graph based on the dynamic point cloud graph and the semantic information corresponding to the dynamic foreground element, and to construct a static Gaussian graph based on the semantic information corresponding to the static point cloud graph and the static background element, wherein each node in the dynamic Gaussian graph represents a dynamic foreground element, and each node in the static Gaussian graph represents a static background element; The merging submodule is used to merge the dynamic Gaussian graph and the static Gaussian graph to obtain a dynamic and static Gaussian graph.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the simulation method of the virtual driving scene described in any one of claims 1 to 6 when executing the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for simulating a virtual driving scene according to any one of claims 1 to 6 is implemented.

Images