Automatic driving scene reconstruction method, automatic driving vehicle and storage medium

Abstract

Embodiments of the present application provide an automatic driving scene reconstruction method, an automatic driving vehicle and a storage medium. The method comprises: acquiring multi-sensor data, the multi-sensor data being data collected by multiple sensors of a vehicle and representing current environmental information and a current vehicle state; constructing an initial scene model according to the multi-sensor data and generating an output attribute feature corresponding to any one sensor, the initial scene model being a model shared by the multiple sensor data, and the output attribute feature representing an attribute of output data of the sensor; training the initial scene model based on the multi-sensor data and the output attribute feature to obtain a target scene model; and generating target scene data through the target scene model, the target scene data being scene data corresponding to any one sensor. The method is used to improve consistency of cross-modal data of vehicle sensors.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving scene reconstruction method, an autonomous driving vehicle, and a storage medium. Background Technology

[0002] Autonomous driving systems typically rely on both onboard cameras and LiDAR to acquire environmental information. To obtain more comprehensive environmental information, it is usually necessary to reconstruct a renderable 3D scene from the real-world data collected.

[0003] In related technologies, camera data and LiDAR data are typically modeled and rendered separately. One approach uses explicit meshes, voxels, or point clouds for scene reconstruction, then designs different rendering pipelines for the camera and LiDAR respectively. Another approach uses neural rendering methods such as neural radiation fields or 3D Gaussian splashing to achieve new perspective image synthesis. Other approaches include neural network prediction methods based on depth map backprojection and point cloud completion. These methods can lead to inconsistencies in geometric information, object boundaries, and occlusion relationships between camera images and LiDAR point clouds.

[0004] Therefore, improving the consistency of cross-modal data in multi-sensor scene reconstruction has become a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides an autonomous driving scene reconstruction method, an autonomous driving vehicle, and a storage medium to improve the consistency of multi-sensor cross-modal data.

[0006] In a first aspect, embodiments of this application provide an autonomous driving scene reconstruction method, including:

[0007] Acquire multi-sensor data, which refers to data collected by various sensors of the vehicle that characterize the current environmental information and the current vehicle status;

[0008] An initial scene model is constructed based on multi-sensor data, and output attribute features corresponding to any one sensor are generated. The initial scene model is a model shared by multiple sensor data, and the output attribute features represent the attributes of the sensor's output data.

[0009] The initial scene model is trained based on multi-sensor data and output attribute features to obtain the target scene model;

[0010] Target scene data is generated by a target scene model. The target scene data can be scene data corresponding to any type of sensor.

[0011] In one possible embodiment, an initial scene model is constructed based on multi-sensor data, including:

[0012] Timestamp alignment of multi-sensor data;

[0013] The initial scene model is obtained by constructing a model based on three-dimensional Gaussian representation using the aligned multi-sensor data.

[0014] In one possible embodiment, generating output attribute features corresponding to any type of sensor includes:

[0015] Obtain the observation conditions, which must include at least the target's viewpoint;

[0016] Output attribute features are generated based on observation conditions and sensor type.

[0017] In one possible embodiment, training the initial scene model includes:

[0018] Supervised training is performed using multi-sensor data as input and output data corresponding to output attribute features as output, so as to minimize the error between the output data and the sensor data.

[0019] In one possible implementation, supervised training includes one of the following:

[0020] Supervised training is performed sequentially on the output attribute features corresponding to each sensor.

[0021] Simultaneously, joint supervised training is performed on the output attribute features.

[0022] In one possible embodiment, supervised training further includes:

[0023] The output attribute features are grouped, and supervised training is performed on each group in turn.

[0024] Simultaneously, supervised training was conducted for all groups.

[0025] In one possible embodiment, the multi-sensor data includes:

[0026] Image data acquired by the image sensor and its corresponding timestamps;

[0027] Laser point cloud data collected by lidar and its corresponding timestamps;

[0028] The pose data and corresponding timestamps collected by the pose sensor.

[0029] In one possible embodiment, the output attribute features include:

[0030] One or more of the color features, transparency features, and image depth features of the image data;

[0031] One or more of the following: distance characteristics, echo intensity characteristics, and echo loss characteristics of laser point cloud data.

[0032] In one possible embodiment, generating target scene data through a target scene model includes:

[0033] In response to the target instructions input by the user, the target scene model is adjusted to generate target scene data. The target instructions represent the user's scene generation requirements.

[0034] In one possible embodiment, generating target scene data includes:

[0035] Generate target scene data in the image format corresponding to the image sensor;

[0036] Generate target scene data in point cloud format corresponding to the LiDAR.

[0037] In one possible embodiment, the target instruction includes:

[0038] The target pose, target viewpoint, target timestamp, and scene editing command are one or more of the following: scene editing command is used to edit the target scene model.

[0039] In one possible embodiment, the multi-sensor data further includes:

[0040] Environmental sensors collect environmental parameters and corresponding timestamps. The environmental parameters represent the visibility information of the current weather.

[0041] In one possible embodiment, constructing an initial scene model based on multi-sensor data further includes:

[0042] When the visibility, which represents the current weather, is less than a preset threshold, Gaussian density compensation is applied to the three-dimensional Gaussian representation model to obtain the initial scene model.

[0043] Secondly, embodiments of this application provide an autonomous driving vehicle, including: a memory and a processor;

[0044] The memory stores instructions that the computer executes;

[0045] The processor executes computer execution instructions stored in memory, causing the processor to perform the methods described above.

[0046] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided above.

[0047] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0048] The autonomous driving scene reconstruction method, autonomous driving vehicle, and storage medium provided in this application acquire multi-sensor data and construct an initial scene model shared by multiple sensor data based on the multi-sensor data. At the same time, it generates output attribute features corresponding to any one sensor. It can take into account the environmental information, vehicle state information, and output attributes of different sensors under a unified scene representation. Then, it trains the initial scene model based on the multi-sensor data and output attribute features, and generates target scene data corresponding to any one sensor through the trained target scene model. Compared with the prior art where scene models are constructed separately for each sensor, this application can construct a shared target scene model based on multiple sensor data. It is equivalent to aligning and uniformly editing the sensor data through the target scene model, and then generating consistent expression data for each sensor based on the shared target scene model. Therefore, it can improve the cross-modal consistency of multiple sensor data. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 1 ;

[0051] Figure 2 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 2 ;

[0052] Figure 3 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 3 ;

[0053] Figure 4 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 4 ;

[0054] Figure 5 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 5 ;

[0055] Figure 6 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 6 ;

[0056] Figure 7 A flowchart illustrating a specific autonomous driving scene reconstruction method provided in this application;

[0057] Figure 8A schematic diagram of the autonomous driving scene reconstruction device provided in this application;

[0058] Figure 9 This is a structural diagram of the autonomous vehicle provided in this application.

[0059] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0060] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0061] First, let me explain the terms used in this application:

[0062] LiDAR (Light Detection and Ranging), also known as lidar, measures distance using laser emission and reception to generate 3D point cloud data. Its advantages include high-precision distance measurement and resistance to illumination interference, but it is limited by its scanline structure and viewing angle.

[0063] 3DGS: refers to 3D Gaussian Splatting, a neural rendering method that uses optimizable Gaussian primitives to represent scenes and supports real-time rendering. However, it is mainly geared towards image generation and lacks modeling of LiDAR characteristics.

[0064] Scaffold Gaussian model: a structured Gaussian representation method that organizes Gaussian primitives through anchor points and local offsets, reducing redundant parameters and improving storage and optimization efficiency.

[0065] Neural rendering: a technique that uses neural networks to generate images or point clouds, combining explicit geometry and implicit functions to achieve new perspective synthesis or sensor simulation.

[0066] Raydrop: Rays that are not received as echoes in a lidar system may be caused by obstruction or excessive distance. Their probability distribution needs to be modeled in the simulation.

[0067] The specific application scenarios of this application are autonomous driving simulation, scene playback or sensor fusion testing, etc. In these scenarios, it is usually necessary to combine data from multiple sensors such as cameras and LiDAR on the vehicle to predict the road environment, traffic participants and the vehicle's own state, and further generate scene data of different perspectives and types to support perception algorithm verification, decision planning evaluation and closed-loop testing.

[0068] In existing scene reconstruction solutions, a common approach is to create corresponding models for camera images and LiDAR point clouds separately, and then achieve the joint output of multi-sensor results through registration, projection, or rendering. For example, one type of solution first uses point clouds, meshes, or voxels to recover the scene geometry, and then generates image data and point cloud data separately; another type of solution uses neural rendering methods to generate visual results and obtains the relevant LiDAR output through an additional transformation process.

[0069] While this type of technology can achieve scene reconstruction to a certain extent, inconsistencies in the generated results can easily arise due to the different data representation and processing links that different sensors often rely on. These inconsistencies can lead to discrepancies in geometric structure, target boundaries, and foreground / background occlusion relationships. In particular, in detailed areas such as vehicles, pedestrians, and roadside facilities, there may be positional deviations, contour misalignments, or distortions in depth relationships between the scene results from one sensor and those from another.

[0070] Meanwhile, existing solutions often fail to adequately represent the output attributes of LiDAR, making it difficult to simultaneously learn the output features of multiple sensors. Furthermore, training and maintaining multiple models separately leads to redundant data storage, lengthy training processes, and reduced reconstruction efficiency. This not only affects the realism and editability of the simulation scenario but also reduces the reliability of multi-sensor fusion algorithm verification.

[0071] Based on the above scenarios, it can be seen that in the existing technology, by modeling data from different sensors separately, there is a technical problem of inconsistency in cross-modal data from multiple sensors.

[0072] The autonomous driving scene reconstruction method provided in this application obtains an initial scene model through unified modeling based on multi-sensor data, while simultaneously generating output attribute features corresponding to any one sensor. Subsequently, the initial scene model is trained based on the multi-sensor data and output attribute features to obtain a target scene model. Finally, target scene data is generated from the target scene model. This unified modeling and training approach improves the consistency between scene data from different sensors and solves the technical problem of inconsistency in cross-modal data from multiple sensors.

[0073] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0074] Figure 1 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0075] S101. Acquire multi-sensor data, which are data collected by various sensors of the vehicle that represent the current environmental information and the current vehicle status.

[0076] In this application, multi-sensor data serves as the basic input for scene reconstruction, representing information such as the road environment, traffic participants, and the vehicle's own motion state.

[0077] Multi-sensor data comprises data collected by different onboard sensors. Depending on the data representation method of each sensor, multi-sensor data can be image data, LiDAR data, or even include sensor parameters. Specifically, it can include: multi-camera images, LiDAR point cloud data, camera parameters, LiDAR parameters, sensor pose, and timestamps. Camera images provide color, texture, and boundary supervision, while LiDAR point cloud data provides distance, echo, and occlusion supervision.

[0078] The current environmental information includes the spatial distribution information of road boundaries, lane lines, roadside facilities, static obstacles, vehicles, pedestrians, and other traffic participants. The current vehicle state includes the vehicle's position, attitude, speed, or state variables related to pose solving at the time of data acquisition.

[0079] LiDAR frames and sensor poses are associated by time or scene frames and organized into a unified training index.

[0080] In practice, the implementing entity can be a scene reconstruction module deployed on the vehicle, roadside computing node, or server. This module reads the data collection results from various vehicle sensors through a data bus, network interface, or storage medium.

[0081] The sensors can be those capable of collecting environmental information and vehicle status; the data collected together constitute multi-sensor data for subsequent unified modeling.

[0082] In practice, data from different sources can be uniformly organized and correlated, enabling data from different sources to be used for subsequent scene modeling under the same reference system. If the original data contains anomalies or errors, it can be processed accordingly before outputting the dataset for scene reconstruction.

[0083] In one possible embodiment, multi-sensor data may include data characterizing environmental information and data characterizing vehicle state. Different types of data can be organized and stored separately, and correspondences established to form the data units required for subsequent model building. It should be understood that the above examples are merely illustrative and not limiting.

[0084] This step unifies and correlates data collected from multiple vehicle sensors, ensuring a consistent data foundation for subsequent scene models at the input level. This processing establishes relationships between data from different sources, providing a directly usable data foundation for building scene models that share data from multiple sensors.

[0085] S102. Construct an initial scene model based on multi-sensor data and generate output attribute features corresponding to any one of the sensors. The initial scene model is a model shared by multiple sensor data, and the output attribute features represent the attributes of the sensor's output data.

[0086] The initial scene model in this step differs from the approach of building independent models for different sensor data. This step maps data from different sensors into the same model parameter space.

[0087] The initial scene model can be understood as the initialization of the scene model using the aforementioned multi-sensor data, including the aforementioned multi-sensor data and attribute parameters related to the sensor outputs. In order to more accurately describe the scene information, this model can be represented in three dimensions.

[0088] Output attribute features are used to describe the attribute content that needs to be generated and supervised in the output results of the target sensor. Different sensors may correspond to different attribute dimensions. For example, color features when outputting image data, echo intensity features when outputting laser point cloud data, etc.

[0089] By configuring corresponding attributes in the same initial scene model, the model can output representational data from different sensors under a unified model structure. Although the attribute features of the data from different sensors are different, the meaning they express is the same. For example, for obstacles in front of a vehicle, image data represents the obstacle's condition through color, while laser point cloud data represents the obstacle's condition through point cloud intensity. Although the attribute features used for data representation are different, they both represent the condition of the obstacle in front of the vehicle.

[0090] In practice, the implementing entity can first construct an initial three-dimensional representation of the scene based on multi-sensor data, and establish the correlation between the scene spatial representation and the sensor output.

[0091] When generating output attribute features, the executing entity establishes attribute encoding based on the output requirements of the target sensor, so as to output attribute features corresponding to the target sensor and establish a correlation with the spatial representation in the initial scene model.

[0092] Based on the above analysis, this step constructs an initial scene model shared by multiple sensor data sources and generates output attribute features matching the sensor output format on this unified model. This allows different sensor data to no longer rely on separate representation links, but rather to be represented geometrically and in terms of attributes on the same 3D scene. After this processing, the geometric positions, target boundaries, and occlusion relationships of the scene results from different sensors all originate from the same spatial model, which can directly support subsequent training.

[0093] S103. Based on multi-sensor data and output attribute features, train the initial scene model to obtain the target scene model.

[0094] In this application, the target scene model is a unified scene model that has been trained and optimized, and its parameters are obtained by the joint constraints of real observation data from multiple sensors and corresponding output attribute features.

[0095] The core of the training process lies in simultaneously applying the observations of scene structure from multi-sensor data and the requirements of the output attribute features on the sensor output format to the initial scene model, so that the model can learn the mapping relationship between various sensor outputs while maintaining a unified geometric expression.

[0096] In this step, multi-sensor data is primarily used to provide realistic observation constraints, while output attribute features are used to specify the sensor output target corresponding to the current training sample. The two are combined to form the training data. After training, the initial scene model is updated to a target scene model, which can output scene results matching the sensor type given any sensor conditions.

[0097] In practice, the execution entity selects training samples from multi-sensor data, determines the observation viewpoint based on the training samples, projects, samples, or integrates the initial scene model into the corresponding output space, and then compares the model output with the sensor data.

[0098] A loss function is constructed based on the comparison results, and an optimization algorithm is used to update the parameters in the initial scene model so that the model output gradually approximates the actual observations. It should be understood that the specific training method can be set according to actual needs, and this application does not limit it in this regard.

[0099] Based on the above analysis, this step trains the same initial scene model using multi-sensor data and output attribute features, ensuring that the model parameters meet the constraints of multi-sensor observations and target sensor attribute representation. The resulting target scene model does not require maintaining multiple independent models, thus maintaining consistency in the output results of different sensors within the same scene structure. In particular, it maintains correspondence with real multi-sensor observations in areas such as vehicle contours, pedestrian boundaries, occlusion relationships, and depth ranking.

[0100] S104. Generate target scene data through the target scene model. The target scene data is the scene data corresponding to any type of sensor.

[0101] In this application, after the target scene model completes training, it enters the inference or generation stage. The executing entity generates target scene data from the same target scene model according to the sensor type to be output and the corresponding observation conditions.

[0102] Target scene data refers to data results that match the output format of the target sensor. Since these outputs all come from the same target scene model, the generated results have a unified source in terms of geometry, target boundaries, and occlusion relationships.

[0103] In practice, the executing entity first receives instructions from the target sensor, which include at least the sensor type and target observation information. Subsequently, the executing entity calls the generation process corresponding to the sensor in the target scene model to transform, calculate, and output the unified scene representation, obtaining target scene data corresponding to the output format of the target sensor.

[0104] For scenarios requiring scene playback and simulation testing, the execution entity can also receive target information at consecutive moments and repeatedly call the target scene model in chronological order to generate multiple frames of scene data, forming time-series scene data that can be used for closed-loop verification.

[0105] In one possible embodiment, the target scene model considers geometric relationships, occlusion relationships, and attribute relationships simultaneously when generating target scene data.

[0106] Specifically, the same spatial representation can be expressed as different output contributions from different sensor perspectives, and the executing entity decodes the corresponding output from shared parameters based on the observation patterns of the target sensor.

[0107] When the vehicle ahead is partially obscured by roadside facilities, the visible portion and effective observation distribution in the outputs of different sensors can be determined by the same occlusion calculation result; for road edges and building edges, the boundary position and structural turning position in the outputs of different sensors can be given by the same spatial model.

[0108] For example, if a request to generate a viewpoint from a specific sensor is received, scene data consistent with the imaging or measurement parameters of that sensor is output. It should be understood that the above example is merely illustrative and not limiting.

[0109] Based on the above analysis, this step generates scene data for any type of sensor using the same target scene model, enabling the generated results to directly serve autonomous driving simulation, scene playback, and sensor fusion testing.

[0110] Since the data from different sensors are all output by a shared model, the positional and occlusion relationships between the results from different sensors remain consistent, thus providing a unified and repeatable scenario basis for perception algorithm verification, decision planning evaluation, and closed-loop testing.

[0111] The autonomous driving scene reconstruction method provided in this application uses data from multiple sensors for unified scene modeling, simultaneously expressing geometric structures and sensor attributes in a shared scene model. Model parameters are then updated through training, ultimately outputting scene results from different sensor formats using the same target scene model. This eliminates the reliance on separate representation and processing chains for multi-sensor scene reconstruction. Consequently, consistency in spatial location, target contour, depth ordering, and foreground / background occlusion relationships is guaranteed for scene results from different sensors. The repetitive modeling process in model training and scene generation is compressed, and the scene reconstruction results can be directly used for simulation generation, editing playback, and fusion algorithm verification. This achieves improved consistency of cross-modal data in multi-sensor scene reconstruction. It should be understood that the above examples are merely illustrative and not limiting.

[0112] In one possible implementation, the multi-sensor data includes:

[0113] Image data acquired by the image sensor and its corresponding timestamps;

[0114] Laser point cloud data collected by lidar and its corresponding timestamps;

[0115] The pose data and corresponding timestamps collected by the pose sensor.

[0116] In this multi-sensor data structure, timestamps are used to characterize the acquisition time of each sensor's data. They are recorded in conjunction with a unified clock source or synchronization pulse, enabling image data, laser point cloud data, and pose data to be correlated under the same time reference.

[0117] When the above data is used to reconstruct the scene model, the time sequence is first determined based on the timestamp, and then the spatial relationship of each sensor in the vehicle coordinate system is determined by combining the pose data, so as to provide a consistent input basis for the subsequent construction of the initial scene model.

[0118] Thus, image data provides appearance and texture information, laser point cloud data provides geometric constraint information, and pose data provides motion state and coordinate transformation information. The three together support multi-sensor joint modeling and scene reconstruction.

[0119] By adopting this multi-sensor data format, images, point clouds, and pose information can participate in modeling under a unified time reference, which facilitates the formation of a joint representation of the same scene and ensures that the subsequently generated target scene data remains consistent in appearance, geometry, and spatial relationships.

[0120] Because timestamps and pose data are input synchronously, the model's adaptability to sampling differences between sensors and motion changes is enhanced, thereby improving the stability and reproducibility of scene reconstruction results.

[0121] Based on the foregoing embodiments, the output attribute features further include:

[0122] One or more of the color features, transparency features, and image depth features of the image data;

[0123] One or more of the following: distance characteristics, echo intensity characteristics, and echo loss characteristics of laser point cloud data.

[0124] In practical implementation, the color features of image data can be formed by RGB (Red, Green, Blue), HSV (Hue, Saturation, Value) or their normalized representations, and used as rendering supervision signals for training; transparency features can be represented by the alpha value of the target edge pixels or the cumulative transmittance of volume rendering; image depth features can be represented by depth maps, disparity maps or the distance field from pixels to the camera.

[0125] The distance characteristics of laser point cloud data can be obtained by converting the measurement radius or Cartesian coordinates of each point. The echo intensity characteristics can be obtained by using the normalized echo intensity value. The echo loss characteristics can be represented by echo loss markers, zero-value masks or sparse indicator vectors.

[0126] The aforementioned features can be jointly bound to the geometric representation in the target scene model, enabling the model to output the corresponding sensor attributes under the same spatial representation.

[0127] During training, the model takes multi-sensor data as input, outputs color, transparency and depth features for the image side, and distance, echo intensity and echo loss features for the LiDAR side, and performs error constraints on the output results with the corresponding supervision data.

[0128] For different sensor attributes in the same scene, the model can build independent output heads separately, or it can share intermediate features and then branch out the output. In practical applications, the above output heads can also be other network structures, which are not limited in this application.

[0129] In this way, the target scene model can simultaneously learn attributes such as visible appearance, spatial occlusion, and missing point cloud scans, thereby maintaining attribute consistency when generating image data and laser point cloud data, and making the results from different sensors correspond to the same scene geometry.

[0130] Figure 2 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, the autonomous driving scene reconstruction method is described in detail, which includes:

[0131] S201. Acquire data from multiple sensors;

[0132] S202. Timestamp alignment of multi-sensor data;

[0133] In practical implementation, the acquisition time can be extracted from each frame of image, each frame of point cloud, and the corresponding pose record in the multi-sensor data, and then matched according to the preset time tolerance.

[0134] When the data from a certain sensor is not completely consistent with the target time point, the corresponding data can be supplemented by linear interpolation, nearest neighbor matching, or resampling according to the sampling frequency, so that all types of data are mapped to the same time slice.

[0135] S203. Construct a model based on three-dimensional Gaussian representation based on the aligned multi-sensor data to obtain the initial scene model.

[0136] Subsequently, the aligned image features, point cloud geometric information, and pose information are input into the scene modeling module. This module initializes a three-dimensional Gaussian element set based on multi-source observation results and assigns attributes such as center coordinates, covariance matrix, color, or echo intensity to each Gaussian element, thereby forming an initial scene model that can be used for subsequent training and rendering.

[0137] The unified 3D Gaussian representation adopts the Scaffold Gaussian structure, which carries shared geometry and implicit features through a scaffolded organization. The initial Gaussian can be derived from LiDAR point clouds, camera reconstruction results, map information, random sampling points, or a combination of the above sources. Unlike related technologies that establish separate image models and LiDAR models, this application maintains only one set of shared 3D geometric bases to carry spatial positions, local shapes, visibility, and occlusion relationships in the scene, and serves as a common source for subsequent image attributes and LiDAR attributes.

[0138] In practical applications, the modeling module can also adopt different numbers of Gaussian element density configurations, which are not limited in this application.

[0139] This method ensures that data from different sensors are at the same time by aligning timestamps, and then uses a three-dimensional Gaussian representation to uniformly express the aligned data. This allows the initial scene model to simultaneously contain geometric structure and sensor attribute information, thereby providing a consistent basic representation for the subsequent generation of target scene data and reducing scene deviations caused by time asynchrony.

[0140] S204. Obtain observation conditions, which shall include at least the target viewpoint;

[0141] In the specific implementation, after acquiring the target viewpoint, the system maps the target viewpoint to the corresponding sensor's extrinsic parameter direction and view frustum range, and determines the expression dimension of the output attribute features accordingly.

[0142] For each camera type, the output attributes can correspond to characteristics such as color, brightness, resolution, pixel arrangement, and perspective projection.

[0143] For lidar types, output attribute characteristics can correspond to features such as point cloud density, scanning beam, echo intensity, measurement distance, and spatial sampling distribution.

[0144] S205. Generate output attribute features based on observation conditions and sensor type;

[0145] The system can also constrain the visibility, occlusion relationship and boundary salience of the target within the field of view based on the observation conditions, and encode these constraints into attribute parameters that match the sensor type, thereby forming output attribute features that can be used for subsequent scene model training.

[0146] The module generates rendering attributes required by different sensors based on a unified 3D Gaussian representation. For the image branch, the attribute prediction module generates attributes related to color, transparency, or image depth; for the LiDAR branch, the attribute prediction module generates attributes related to distance, echo intensity, raydrop, or point cloud generation. These attributes can be adjusted according to observation conditions such as the target camera viewpoint, the target LiDAR viewpoint, or scanning constraints. Since these attributes all rely on the same 3D Gaussian representation, even if the image and LiDAR output formats are different, they can still maintain consistency in object contours, occlusion boundaries, spatial positions, and depth levels.

[0147] If the selected sensor model has different field of view or sampling frequency in engineering implementation, the range of attribute parameters can be adjusted accordingly to adapt to the corresponding observation state. In practical applications, other models of this component can also be selected, and this application does not limit this.

[0148] During operation, the target perspective first determines the sensor's observation direction of the scene, then selects the appropriate attribute representation template based on the sensor type, and converts the observation conditions into a set of attribute parameters. Finally, it outputs the attribute features corresponding to the sensor's observation results. These output attribute features serve as training constraints for the shared scene model, ensuring that data from different sensors can be consistently represented within a unified modeling framework.

[0149] By adopting this method, the output attribute features can be synchronously matched with the specific observation direction and sensor type, making the generated results closer to the observation characteristics of the corresponding sensor, and providing unified attribute constraints for subsequent scene training based on a shared model, thereby improving the consistency and usability of multi-sensor scene data.

[0150] S206. Based on multi-sensor data and output attribute features, train the initial scene model to obtain the target scene model;

[0151] S207. Generate target scene data through the target scene model. The target scene data is the scene data corresponding to any type of sensor.

[0152] Figure 3 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 3 ,like Figure 3 As shown, in this embodiment... Figure 1 Based on the examples, the autonomous driving scene reconstruction method is described in detail, which includes:

[0153] S301, Acquire data from multiple sensors;

[0154] S302. Construct an initial scene model based on multi-sensor data and generate output attribute features corresponding to any one of the sensors;

[0155] S303. Supervised training is performed using multi-sensor data as input and output data corresponding to output attribute features as output to minimize the error between the output data and the sensor data.

[0156] In practical implementation, the initial scene model can use a shared 3D Gaussian representation to carry the scene geometry and appearance information, and the aligned multi-sensor data can be sent into the model as input samples.

[0157] During the training phase, a unified 3D Gaussian representation is updated jointly using image-supervised training and LiDAR-supervised training. Image-supervised training constrains scene color, texture, contour, and visible regions, while LiDAR-supervised training constrains distance, scan structure, echo attributes, and occlusion relationships. The system can employ alternating training, phased training, or joint training methods, allowing both types of supervision to play a role on the same 3D representation. Through this joint optimization approach, the image branch can supplement the appearance and boundary information of sparse regions in the LiDAR image, while the LiDAR branch can constrain distance and spatial structure in image reconstruction, thereby improving the quality of scene reconstruction.

[0158] After receiving the input, the model generates corresponding image, depth map, point cloud attributes or other sensor output data based on the sensor type and target view indicated by the output attribute features. Then, it calculates the error between the output data and the real sensor data collected at the same time.

[0159] Errors can take the form of pixel-level reconstruction errors, depth deviations, point cloud position deviations, or combinations thereof, and the scene model parameters are updated through backpropagation, so that the model gradually learns the scene representation shared between multi-sensor data.

[0160] The working principle is that supervised training uses real sensor data as the constraint target, so that the initial scene model can fit the observation results of different sensors under a unified representation.

[0161] As training iterates, the difference between the model output and the real sensor data gradually decreases, and the scene geometry, target contours and occlusion relationships can be stably encoded into the shared model, thereby ensuring the consistency of any sensor scene data generated based on this model in the future.

[0162] By adopting this approach, the initial scene model can learn the mapping relationship between multiple sensors within a single training framework, reducing parameter redundancy and training repetition caused by separate modeling, and bringing the error between the output data and sensor data to a smaller range, thereby improving the realism of the scene reconstruction results and the consistency of multiple sensors.

[0163] S304. Generate target scene data through the target scene model. The target scene data is the scene data corresponding to any type of sensor.

[0164] Figure 4 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 4 ,like Figure 4 As shown, in this embodiment... Figure 3 Based on the examples, the autonomous driving scene reconstruction method is described in detail, and supervised training is performed, including any one of the following:

[0165] S401. Supervised training is performed sequentially on the output attribute features corresponding to each sensor.

[0166] In practice, the real sensor data used for supervised training can come from the time-stamped acquisition results of multiple sensors. The supervised target for the camera is the image pixel distribution, and the supervised target for the LiDAR is the point cloud spatial location and echo intensity distribution.

[0167] For sequential training scenarios, a rotational training strategy can be adopted, so that the output attribute features of a certain sensor correspond to individual supervisory weights in the current round.

[0168] S402. Simultaneously, joint supervised training is performed on the output attribute features.

[0169] For joint training scenarios, different loss weights can be set for different sensors to balance image details and three-dimensional geometric constraints.

[0170] All of the above training can be implemented based on deep neural networks. The network parameters and the scene parameters represented by the three-dimensional Gaussian are updated together. After training, a target scene model that can output scene data from any sensor can be obtained.

[0171] By adopting the above-mentioned supervised training method, the model can be trained according to the needs of single sensor independent optimization or multi-sensor joint optimization, thereby making the output attribute features corresponding to different sensors consistent with the real observation results, and improving the stability and consistency of the scene model in multimodal output.

[0172] Since the training process can converge separately or simultaneously for different sensors, the target scene model can maintain more consistent geometric relationships and target contours when generating different types of scene data such as images and point clouds, thereby improving the usability of autonomous driving scene reconstruction results.

[0173] S403. Group the output attribute features and perform supervised training on each group in turn; at the same time, perform supervised training on all groups.

[0174] In practice, a multi-stage training approach can also be adopted.

[0175] First, after grouping the output attribute features, the model can first construct a local supervision target for a certain group, and adjust the related feature mapping relationship according to the data distribution corresponding to the group. Then, it can switch to the next group to continue training so that each type of attribute feature can converge.

[0176] When supervising training is performed on all groups, the supervision targets of each group are used together in the loss calculation, so that the model can learn the relationship between different attributes in a unified parameter space.

[0177] After training using any of the methods described above, the system obtains a unified 3D Gaussian representation. This representation includes shared 3D geometry and corresponding multimodal attribute generation capabilities, and can serve as the foundational model for subsequent new perspective rendering, road test scene replay, scene editing, or closed-loop simulation. Compared to training image models and LiDAR models separately, this embodiment only requires maintaining the same set of 3D scene representations after training, reducing the cost of repetitive geometry storage and maintenance.

[0178] The above-mentioned supervised training method takes into account both group optimization and overall joint optimization within the same training framework, enabling the target scene model to learn the differences and coupling relationships between different output attributes at the same time. This ensures the consistency of geometric structure, target boundary and attribute expression when generating scene data corresponding to any sensor, and improves the stability and usability of multi-sensor scene reconstruction results.

[0179] Figure 5 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 5 ,like Figure 5 As shown, in this embodiment... Figure 1 Based on the examples, the autonomous driving scene reconstruction method is described in detail, which includes the following steps:

[0180] S501. Acquire multi-sensor data, which are data collected by various sensors of the vehicle that represent the current environmental information and the current vehicle status.

[0181] S502. Construct an initial scene model based on multi-sensor data and generate output attribute features corresponding to any one sensor. The initial scene model is a model shared by multiple sensor data, and the output attribute features represent the attributes of the sensor's output data.

[0182] S503. Based on multi-sensor data and output attribute features, train the initial scene model to obtain the target scene model;

[0183] S504. In response to the target instruction input by the user, adjust the target scene model to generate target scene data. The target instruction represents the user's scene generation requirements.

[0184] In practice, after the user inputs the target command, the system first parses the scene generation requirements, extracts control parameters such as road type, weather condition, target object distribution, view position and sensor type of the scene to be generated, and then maps the control parameters to the adjustment conditions of the target scene model.

[0185] Subsequently, based on the adjustment conditions, the scene parameters, observation parameters, or output branch parameters in the target scene model are updated, so that the model switches to the generation state corresponding to the target instruction while maintaining the learned scene structure constraints, and outputs the corresponding target scene data.

[0186] When the target instruction instructs the generation of image results from different perspectives, the model adjusts the rendering direction and visibility relationship according to the target perspective; when the target instruction instructs the generation of LiDAR-related results, the model adjusts the distance, echo intensity, or echo loss representation according to the sensor type, thereby obtaining target scene data that matches the user's needs.

[0187] During the operation, the target instruction serves as an external control input, which plays a directional constraint role on the target scene model, enabling the model to perform targeted adjustments based on a unified scene representation, and the adjusted model output is directly used as the target scene data.

[0188] Because this adjustment process is completed around the user scenario generation requirements, the generated results can be consistent with the specified scenario type and sensor type, and support the output of scenario data with different content as needed.

[0189] During the rendering or application phase, the system loads a unified 3D Gaussian representation. This phase can forgo training and optimization, instead organizing the unified Gaussian set to be rendered based on target instructions (which may include target sensor pose, target viewpoint, target timestamp, or scene editing instructions). For scene editing tasks, the system performs operations such as deletion, insertion, movement, or trajectory changes on the same 3D representation through a unified scene editing module. Because editing operations act on the same 3D representation, a single edit can simultaneously affect both the image rendering branch and the LiDAR rendering branch.

[0190] By adopting the above method, the target scene model can be adjusted in a targeted manner under the drive of user instructions and output the corresponding scene results, making the correspondence between the generated data and user needs clearer. It can also generate different scene content within a unified model, reduce the processing overhead caused by repeated modeling, and improve the consistency and controllability of the generated target scene data.

[0191] S505. Generate target scene data in the image format corresponding to the image sensor;

[0192] S506. Generate target scene data in point cloud format corresponding to the LiDAR.

[0193] In this implementation, after the target scene model completes training, it simultaneously outputs two types of results based on a unified scene geometric representation and sensor output attribute features:

[0194] One is image data for image sensors. When generating the data, the scene is rendered according to the target viewpoint, imaging range and image attribute features, so that the lane lines, vehicles, pedestrians and roadside facilities are consistent with the sensor imaging in terms of color, transparency and depth.

[0195] Secondly, it is 3D point cloud data for LiDAR. When generating the data, the scene is sampled according to the radar installation pose, scanning range and point cloud attribute characteristics, so that the point set, distance distribution, echo intensity and echo loss status of each spatial location are consistent with the LiDAR output.

[0196] The system inputs the same unified Gaussian set into both the image rendering module and the LiDAR rendering module. The image rendering module generates camera images, depth, or transparency based on camera parameters, obtaining image data. The LiDAR rendering module generates distance, intensity, raydrop, range image, or point cloud data based on the LiDAR pose, field of view, and scanline constraints, obtaining LiDAR point cloud data. While both modules can use different rendering methods, they consume the same Gaussian set and the same editing results, thus avoiding the problem of asynchronous image and point cloud editing.

[0197] The multimodal rendering output module outputs camera images, LiDAR point clouds, range images, or other intermediate results. Since both types of output originate from the same 3D Gaussian representation, the outputs maintain consistency in geometric location, visible area, occlusion boundaries, and editing effects. This step is particularly suitable for autonomous driving simulations and data generation scenarios. For example, when replaying a new viewpoint, the system can simultaneously generate camera images and LiDAR data from the same viewpoint; when editing a scene, the system can simultaneously generate edited images and point clouds; and when performing closed-loop simulations, the system can provide geometrically consistent multi-sensor inputs for perception algorithms.

[0198] The target scene model can call the image decoding head and the point cloud decoding head respectively to complete the output based on the same scene state. The two types of output share the same scene geometry, thus maintaining the consistency of the target object's boundary, relative position, and occlusion relationship.

[0199] In practical applications, the image output can be in the form of RGB image, RGBA (Red Green Blue Alpha) image with transparency channel or grayscale image, and the point cloud output can be in the form of a three-dimensional point set containing coordinate, echo intensity and timestamp fields. This application does not limit this.

[0200] During operation, the system receives the target instruction, determines the type of sensor and observation conditions to be generated, and drives the target scene model to complete the corresponding format data decoding and reconstruction output, ultimately obtaining target scene data that can be directly used for simulation display, perception testing or fusion verification.

[0201] With this approach, both the image results and the point cloud results are generated uniformly from the same target scene model, which can maintain the consistency of scene geometry and sensor properties, reduce the deviation caused by reconstruction of multiple models separately, and improve the stability and usability of multi-sensor simulation output.

[0202] In one possible implementation, the target instructions include:

[0203] The target pose, target viewpoint, target timestamp, and scene editing command are one or more of the following: scene editing command is used to edit the target scene model.

[0204] In this embodiment, the target command is input by the user in the scene playback interface, simulation control interface, or test task configuration interface. After the system parses it, it maps the target pose, target viewpoint, and target timestamp to the spatial transformation parameters, rendering parameters, and temporal index of the target scene model, and maps the scene editing command to the editing parameters of object instances, material properties, motion trajectories, or occlusion relationships within the model.

[0205] If the target instruction contains multiple contents at the same time, the system will merge the parameters into the same target scene generation request according to the preset priority, so as to drive the target scene model to output the corresponding scene data at the corresponding pose, viewpoint and time.

[0206] Scene editing commands can be entered via labeled text, graphical drag-and-drop commands, or structured configuration fields. The system converts these commands into executable editing operations on the target scene model and recalculates the spatial relationships and visibility results of related objects after editing is complete.

[0207] By employing target instructions composed of target pose, target viewpoint, target timestamp, and scene editing instructions, the target scene model can output under specified spatial and temporal states, and generate scene data that meets task requirements under user-specified editing conditions, thus enabling the generated results to simultaneously possess spatiotemporal consistency and editability.

[0208] In one possible implementation, the multi-sensor data also includes: environmental parameters collected by environmental sensors and corresponding timestamps, whereby the environmental parameters characterize the visibility information of the current weather.

[0209] In this embodiment, the environmental parameters and their timestamps collected by the environmental sensors are input into the scene reconstruction processing unit along with the multi-sensor data. The processing unit aligns the environmental parameters with the image data, point cloud data and pose data at the corresponding time according to the timestamps, and writes the weather information as an additional condition for scene modeling into the initial scene model.

[0210] For scenarios with rain, fog, backlight, or low visibility, environmental parameters can provide constraints for subsequent Gaussian density compensation, occlusion relationship correction, and sensor output attribute training, enabling the model to express the observational differences corresponding to different weather conditions under the same scene structure.

[0211] By introducing environmental parameters with timestamps, the target scene model can simultaneously utilize vehicle state, road geometry, and weather information for joint modeling within a unified multi-sensor framework. This ensures that the generated image data or point cloud data remains consistent with actual weather conditions and improves the stability and consistency of multi-sensor reconstruction results in complex weather scenarios.

[0212] Figure 6 Flowchart of the autonomous driving scene reconstruction method provided in this application Figure 6 ,like Figure 6 As shown, in this embodiment... Figure 1 Based on the examples, the autonomous driving scene reconstruction method is described in detail, which includes the following steps:

[0213] S601, Acquire data from multiple sensors;

[0214] S602. When the visibility, which represents the current weather, is less than a preset threshold, Gaussian density compensation is performed on the three-dimensional Gaussian representation model to obtain the initial scene model.

[0215] In practical implementation, environmental parameters can be collected by environmental sensors and bound to the current time. Environmental parameters represent the visibility information of the current weather, and can also include one or more of the following data that can represent visibility information: precipitation intensity, fog concentration, and light intensity.

[0216] After reading the environmental parameters, the system determines whether to enter the compensation mode based on the comparison between visibility and the preset threshold. When the visibility is lower than the preset threshold, the compensation processing of the three-dimensional Gaussian representation model is started.

[0217] The compensation process can identify the possible spatial range of the occluded target based on image information, laser point cloud information and pose information from multi-sensor data, and insert new Gaussian components within this range, or adjust the number, radius, weight and covariance matrix of the original Gaussian components, so that the model can still maintain a relatively complete geometric expression in low visibility environments.

[0218] The compensated Gaussian components continue to participate in the construction of the initial scene model and serve as a common scene basis for subsequent training.

[0219] When the process is working, it first determines the current weather conditions through environmental parameters, then determines whether the scene is in a low visibility state by combining the visibility threshold, and then performs local density enhancement on the three-dimensional Gaussian representation model, finally forming an initial scene model suitable for unified modeling of multiple sensors.

[0220] Because the compensated model has a more complete spatial representation in the occluded area, it can maintain the consistency of target contours, depth relationships and geometric continuity when training and generating scenes based on the model.

[0221] By adopting the above method, the geometric details in low visibility scenes can be supplemented and expressed. The initial scene model has higher descriptive integrity for areas weakened by fog, rain, snow or dark light, thus providing a more stable scene foundation for subsequent unified training of multiple sensors and improving the spatial consistency and reconstruction reliability of the generated scene data.

[0222] S603. Generate the output attribute features corresponding to any type of sensor;

[0223] S604. Based on multi-sensor data and output attribute features, train the initial scene model to obtain the target scene model;

[0224] S605. Generate target scene data through the target scene model. The target scene data is the scene data corresponding to any type of sensor.

[0225] Figure 7 A flowchart illustrating a specific autonomous driving scene reconstruction method provided in this application is shown below. Figure 7 As shown, the application scenario of this embodiment is a closed-loop simulation test scenario for an autonomous driving system. The developer reconstructs a high-precision three-dimensional scene model based on real multi-sensor data to generate different sensor output data. The reconstruction object adopts a certain urban road scene and is used for autonomous driving simulation test in complex traffic flow.

[0226] The multi-sensor data includes images from multiple cameras, image data and corresponding timestamps collected by image sensors, laser point cloud data and corresponding timestamps collected by lidar, sensor pose data, pose data and corresponding timestamps collected by pose sensors. In one possible implementation, it also includes environmental parameters and corresponding timestamps collected by environmental sensors. The environmental parameters are used to characterize the current weather information and can correspond to extreme weather, low light at night, or rain and fog scenarios.

[0227] S701. Acquire multi-sensor data, perform timestamp alignment and spatial correlation on the multi-sensor data, and generate a unified training index.

[0228] The urban road scene collected contains camera images and LiDAR point clouds from multiple fixed perspectives. The system ensures that the image frames and LiDAR frames are synchronized in time by aligning with timestamps, and maps sensor data from different perspectives to the same coordinate system through sensor pose information. This associates image data, LiDAR point cloud data, pose data, and environmental parameters at the same or adjacent times with a unified time series.

[0229] S702. Based on a unified time series and a unified coordinate system, the system constructs a model based on three-dimensional Gaussian representation according to the aligned multi-sensor data.

[0230] The spatial distribution, geometric structure, and observation information of the scene are mapped to multiple Gaussian voxels, forming an initial scene model shared by data from various sensors. This construction process initializes a Scaffold Gaussian structure based on LiDAR point clouds, generates sparse anchor points through an anchor point network, and adjusts the Gaussian distribution according to the point cloud density through a local offset network. In traffic sign areas, the Gaussian density is higher to preserve edge contours, while in open areas, the Gaussian density is lower to reduce redundant storage, ultimately generating a Scaffold Gaussian set containing a shared geometric base.

[0231] When the visibility of the current weather, represented by environmental parameters, is less than a preset threshold, Gaussian density compensation is performed on the 3D Gaussian representation model. The initial scene model is obtained by increasing the Gaussian distribution density of the corresponding area or adjusting the Gaussian parameters. In low-light or rainy / foggy scenes at night, compensation is performed on the occluded areas, so that the shared geometric base can continue to be used for subsequent multimodal attribute generation, dynamic scene editing, and long-tail scene construction. Long-tail scene construction includes extreme weather and complex traffic flow, and dynamic scene editing includes the insertion, deletion, and trajectory modification of traffic participants.

[0232] S703. After the initial scene model is established, the system acquires the observation conditions, which include at least the target viewpoint, and generates the output attribute features corresponding to any sensor type in combination with the sensor type.

[0233] Based on a shared geometric base, image attributes and laser attributes are generated through a multimodal attribute prediction network. When the sensor type is an image sensor, the output attribute features include one or more of color features, transparency features, and image depth features, and can be specifically RGB color and transparency. The image attribute prediction network generates RGB colors conditionally according to the camera viewpoint and distance to ensure that the red and yellow of the traffic signs are clearly visible in the image.

[0234] When the sensor type is LiDAR, the output attribute features include one or more of the following: distance features, echo intensity features, and echo loss features, specifically distance, intensity, and raydrop. The LiDAR attribute prediction network generates distance and intensity based on LiDAR scan line constraints, including pitch angle and horizontal field of view, ensuring that the generated LiDAR point cloud is consistent with the real data in terms of scan line structure. In intersection areas, the LiDAR point cloud covers traffic participants in multiple directions. The attribute prediction network adjusts the point cloud distribution through scan line constraints to ensure that occlusion boundaries are consistent with the image.

[0235] S704. Using multi-sensor data as input and output data corresponding to output attribute features as output, supervised training is performed on the initial scene model.

[0236] During training, the error between the output data and the corresponding original sensor data is calculated, and the parameters in the initial scene model are adjusted to minimize the error.

[0237] During supervised training, the output attribute features corresponding to each sensor can be trained sequentially, or multiple output attribute features can be trained jointly at the same time. Alternatively, the output attribute features can be grouped first, and supervised training can be performed sequentially based on each group, or supervised training can be performed on all groups at the same time.

[0238] In this urban road scenario, the system utilizes image supervision and LiDAR supervision to jointly optimize a unified Gaussian set. Image supervision includes color and texture, while LiDAR supervision includes distance and occlusion. During the training phase, image supervision constrains the RGB color and transparency of traffic signs to ensure they are clearly visible in the image, while LiDAR supervision constrains vehicle distance and occlusion relationships at intersections to ensure that the LiDAR point cloud remains consistent with the image in occluded areas.

[0239] The system employs an alternating training strategy, first optimizing image-supervised targets such as color and texture, then optimizing LiDAR-supervised targets such as distance and intensity, iterating in a loop until convergence. In pedestrian crossing scenarios, image supervision optimizes the outline and color of pedestrians, while LiDAR supervision optimizes the distance and occlusion relationships of pedestrians. After training, a target scene model is obtained, ensuring that the generated camera images and LiDAR point clouds maintain consistency in boundary alignment and spatial position of dynamic targets.

[0240] During training, the system dynamically adjusts the Gaussian distribution based on training feedback, which includes visibility, error, and contribution. The system calculates the visibility probability of Gaussian units and retains Gaussian density in high-visibility areas such as traffic sign areas, while deleting redundant Gaussian units in low-visibility areas such as open areas. Simultaneously, it adjusts the number of Gaussian units through splitting and merging operations. In complex traffic flow areas such as intersections, splitting operations increase Gaussian density to enhance geometric representation, while merging operations reduce redundant Gaussian units to lower storage pressure in low-contribution areas such as road edges.

[0241] In low-light nighttime scenarios, the system increases Gaussian density through splitting operations to compensate for occlusion and loss of LiDAR point clouds, while reducing redundant Gaussian density in open areas through merging operations to maintain resource efficiency. This adjustment process applies to the model parameters of the 3D Gaussian representation and is executed in conjunction with the error convergence process until the target scene model after adaptive Gaussian adjustment is formed.

[0242] S705. After obtaining the target scene model, in response to the target command input by the user, adjust the target scene model and generate target scene data.

[0243] The target instructions include one or more of the following: target pose, target viewpoint, target timestamp, and scene editing instructions. The scene editing instructions are used to edit the scene content, observation position, or scene state in the target scene model, and may specifically include inserting, deleting, and modifying the trajectory of traffic participants, or inserting or deleting vehicles or pedestrians.

[0244] The model infers scene data from the target scene model based on the adjusted target pose, target viewpoint, and target timestamp, and inputs the adaptively adjusted unified Gaussian set into the image rendering module and the LiDAR rendering module respectively.

[0245] In the simulation test at the intersection, the image rendering module generates an RGB image containing traffic signs, vehicles, and pedestrians based on camera parameters, while the LiDAR rendering module generates a range image and point cloud that conform to the LiDAR scanning structure based on scan line constraints. When the output object is an image sensor, target scene data in image format is generated, and image information matching the target observation conditions is produced according to color features, transparency features, and image depth features. When the output object is a LiDAR, target scene data in point cloud format is generated, and point cloud information matching the target observation conditions is produced according to distance features, echo intensity features, and echo loss features.

[0246] In pedestrian crossing scenarios, the image rendering module generates an image containing pedestrian outlines and colors, while the LiDAR rendering module generates a point cloud containing pedestrian distance and intensity. Both maintain consistency in occlusion boundaries and spatial location. In traffic sign, intersection, and pedestrian crossing scenarios, the camera image and LiDAR point cloud maintain consistent boundary alignment and occlusion relationships.

[0247] In closed-loop simulation testing, developers performed insertion, deletion, and trajectory modification on traffic participants such as vehicles and pedestrians based on a unified Gaussian set, and simultaneously generated camera images and LiDAR point clouds from different perspectives without needing to adjust the images and LiDAR models separately. In low-light or rainy / foggy nighttime scenarios, they combined environmental parameters, dynamic Gaussian adjustment, and attribute prediction networks to generate image information, range images, and point clouds that conform to the characteristics of the real environment, which were used to verify the performance of the perception algorithm in complex traffic flows such as intersections and pedestrian crossings.

[0248] In resource-constrained scenarios (such as automotive embedded systems), the allocation of Gaussian quantities can be dynamically adjusted through a real-time computing resource monitoring module. For example, more Gaussians can be allocated on high-performance GPUs to improve reconstruction accuracy, while fewer Gaussians can be allocated on low-performance CPUs to reduce inference latency.

[0249] First, the computational resource monitoring module can monitor hardware resources (such as GPU utilization and memory usage) in real time and generate resource status labels. Second, it dynamically adjusts the number of Gaussians based on resource status. For example, under high resource conditions, all Gaussians are retained to improve reconstruction accuracy; under low resource conditions, low-contribution Gaussians (such as redundant Gaussians in open areas) are deleted to reduce computational load. When resources are limited, high-contribution Gaussians (such as traffic signs and vehicle outlines) are rendered first, while low-contribution Gaussians are rendered later or merged.

[0250] Figure 8 This is a schematic diagram of the structure of the autonomous driving scene reconstruction device provided in this application, as shown below. Figure 8 As shown, the autonomous driving scene reconstruction device 80 provided in this embodiment includes:

[0251] The acquisition module 801 is used to acquire multi-sensor data, which is data collected by various sensors of the vehicle that characterizes the current environmental information and the current vehicle status.

[0252] The first generation module 802 is used to construct an initial scene model based on multi-sensor data and generate output attribute features corresponding to any one sensor. The initial scene model is a model shared by multiple sensor data, and the output attribute features represent the attributes of the sensor's output data.

[0253] Training module 803 is used to train the initial scene model based on multi-sensor data and output attribute features to obtain the target scene model;

[0254] The second generation module 804 is used to generate target scene data through the target scene model. The target scene data is the scene data corresponding to any type of sensor.

[0255] The autonomous driving scene reconstruction device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0256] Figure 9 This is a structural diagram of the autonomous vehicle provided in this application. Figure 9 As shown, the autonomous vehicle 90 provided in this embodiment includes at least one processor 901 and a memory 902. Optionally, the vehicle 90 also includes a communication component 903. The processor 901, memory 902, and communication component 903 are connected via a bus 904.

[0257] In a specific implementation, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to perform the above-described method.

[0258] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0259] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0260] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0261] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0262] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0263] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0264] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0265] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0266] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0267] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0268] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0269] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0270] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0271] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for reconstructing an autonomous driving scene, characterized in that, include: Acquire multi-sensor data, which is data representing the current environmental information and the current vehicle status collected by various sensors of the vehicle; An initial scene model is constructed based on the multi-sensor data, and output attribute features corresponding to any one of the sensors are generated. The initial scene model is a model shared by multiple sensor data, and the output attribute features represent the attributes of the sensor's output data. Based on the multi-sensor data and the output attribute features, the initial scene model is trained to obtain the target scene model; Target scene data is generated through the target scene model, and the target scene data is scene data corresponding to any of the sensors.

2. The method according to claim 1, characterized in that, An initial scene model is constructed based on the multi-sensor data, including: The multi-sensor data is timestamped and aligned. The initial scene model is obtained by constructing a model based on three-dimensional Gaussian representation based on the aligned multi-sensor data.

3. The method according to claim 1, characterized in that, Generate output attribute features corresponding to any of the aforementioned sensors, including: Acquire observation conditions, which include at least the target viewpoint; The output attribute features are generated based on the observation conditions and the type of sensor.

4. The method according to claim 1, characterized in that, Training the initial scene model includes: Using the multi-sensor data as input and the output data corresponding to the output attribute features as output, supervised training is performed to minimize the error between the output data and the sensor data.

5. The method according to claim 4, characterized in that, Supervised training includes one of the following: Supervised training is performed sequentially on the output attribute features corresponding to each of the sensors; Simultaneously, joint supervised training is performed on the output attribute features.

6. The method according to claim 4, characterized in that, Supervised training also includes: The output attribute features are grouped, and supervised training is performed sequentially based on each group; Simultaneously, supervised training was conducted for all groups.

7. The method according to claim 1, characterized in that, The multi-sensor data includes: Image data acquired by the image sensor and its corresponding timestamps; Laser point cloud data collected by lidar and its corresponding timestamps; The pose data and corresponding timestamps collected by the pose sensor.

8. The method according to claim 7, characterized in that, The output attribute features include: One or more of the color features, transparency features, and image depth features of the image data; One or more of the distance characteristics, echo intensity characteristics, and echo loss characteristics of the laser point cloud data.

9. The method according to claim 1, characterized in that, Generating target scene data through the target scene model includes: In response to a target instruction input by the user, the target scene model is adjusted to generate the target scene data, wherein the target instruction represents the user's scene generation requirements.

10. The method according to claim 9, characterized in that, Generating the target scene data includes: Generate the target scene data in the image format corresponding to the image sensor; Generate the target scene data in point cloud format corresponding to the LiDAR.

11. The method according to claim 9, characterized in that, The target instructions include: The target pose, target viewpoint, target timestamp, and scene editing instructions are selected from one or more of these, wherein the scene editing instructions are used to edit the target scene model.

12. The method according to claim 2, characterized in that, The multi-sensor data also includes: Environmental parameters and corresponding timestamps collected by environmental sensors, wherein the environmental parameters characterize the visibility information of the current weather.

13. The method according to claim 12, characterized in that, Constructing an initial scene model based on the multi-sensor data also includes: When the visibility, which is the environmental parameter representing the current weather, is less than a preset threshold, Gaussian density compensation is performed on the three-dimensional Gaussian representation model to obtain the initial scene model.

14. An autonomous vehicle, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-13.

Images