An end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model

Abstract

The application discloses an end-to-end automatic driving closed-loop simulation test method based on a 4D neural representation and a world model, comprising the following steps: acquiring multi-source space-time data of a real road scene, wherein the multi-source space-time data comprises a multi-view image sequence, a point cloud sequence and a vehicle pose trajectory; training the multi-source space-time data to obtain a four-dimensional neural radiation field model representing a dynamic scene, and extracting a corresponding time sequence of an implicit state vector set based on the four-dimensional neural radiation field model; and constructing a state transition model according to the implicit state vector set, wherein the state transition model takes the current time implicit state vector and a control instruction as input. Through four-dimensional neural representation and implicit state modeling, the application realizes high-fidelity expression and near real-time rendering of an automatic driving simulation scene; meanwhile, a closed-loop interaction mechanism is constructed based on a state evolution model, so that the adaptability of an automatic driving model to a real road environment and the test effectiveness are improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to an end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model. Background Technology

[0002] In the field of autonomous driving technology, end-to-end autonomous driving is a technological paradigm that directly maps multi-source sensor data into vehicle control commands. Its core relies on deep neural networks to achieve unified modeling from perception to decision-making. For testing and verifying this type of model, existing technologies typically employ two methods: one is based on traditional simulation platforms, which uses 3D modeling tools to construct scene elements such as roads, buildings, and traffic participants, and combines physics engines and rendering engines to generate sensor data, achieving closed-loop interaction with the model under test; the other is a scene reconstruction method based on implicit neural representations, which trains a neural radiation field model by collecting multi-view data from real roads to generate new perspective images, and then overlays simple dynamic targets to construct a simulation environment. These two methods, from the perspectives of rule-based engineering modeling and data-driven neural modeling respectively, constitute the main technical paths for current end-to-end autonomous driving simulation testing.

[0003] However, existing technologies still have significant shortcomings: Traditional simulation platform-based methods rely heavily on manual modeling and rule design, making it difficult to accurately reproduce the complex and ever-changing environmental details and traffic participant behaviors in the real world. This results in significant differences between simulation and reality, and the scene construction method limits the ability to generate long-tail scenes. While simulation methods based on neural radiation fields improve visual realism, they mainly focus on modeling static scenes and cannot naturally express the spatiotemporal evolution of dynamic traffic flow. They also lack the ability to predict the future state of the scene and cannot form an effective closed-loop interaction mechanism. Furthermore, their computational overhead is high, making it difficult to meet the needs of real-time simulation.

[0004] Therefore, how to achieve dynamic scene evolution modeling and closed-loop interaction capabilities while ensuring high-fidelity rendering, and support large-scale and diversified scene generation, has become a key issue that current technology urgently needs to address.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide an end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model to solve the problems in the background art mentioned above.

[0007] To achieve the above objectives, this application provides the following technical solution: an end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model, comprising the following steps: Acquire multi-source spatiotemporal data of real road scenes, including multi-view image sequences, point cloud sequences, and vehicle pose trajectories. Train the multi-source spatiotemporal data to obtain a four-dimensional neural radiation field model representing the dynamic scene, and extract the set of implicit state vectors of the corresponding time series based on the four-dimensional neural radiation field model. Based on the set of implicit state vectors, a state transition model is constructed. The state transition model takes the implicit state vector at the current moment and the control command as input and outputs the prediction result of the implicit state vector at the next moment, which is used to characterize the scene evolution process. An initial implicit state vector is selected, and the simulation process is executed based on the initial implicit state vector. The current implicit state vector is input into the four-dimensional neural radiation field model for rendering to obtain observation data. The observation data is then input into the end-to-end autonomous driving model to obtain the corresponding control commands. The current implicit state vector and corresponding control command are input into the state transition model to obtain the implicit state vector at the next moment. The implicit state vector at the next moment is then used as the updated current implicit state vector. Iteration is performed to achieve the closed-loop simulation process. During the closed-loop simulation, various implicit state evolution paths are generated by adjusting the initial implicit state vector or control command sequence to obtain diverse simulation scenarios for testing end-to-end autonomous driving models.

[0008] The multi-source spatiotemporal data acquisition process adopts a multi-sensor synchronous acquisition mechanism. The acquisition equipment includes a multi-view camera unit, a laser scanning unit, and an inertial measurement unit. All acquired signals are synchronously recorded through a unified time reference, and spatial coordinates are uniformly mapped through calibration parameters. At the same time, combined with filtering and outlier data removal operations, a multimodal data set that is temporally continuous and spatially consistent is obtained, thereby ensuring that the subsequent dynamic scene modeling process has complete spatiotemporal consistency and data stability.

[0009] The four-dimensional neural radiation field model adopts a spatiotemporal joint coding structure. By extending the spatial coordinates and time variables with high-frequency position coding, the low-dimensional input is mapped to a high-dimensional feature space. Feature extraction is completed through a multi-layer nonlinear network, and color information and volume density distribution are output, thereby realizing the continuous expression of dynamic scenes. The time variable participates in the feature coding process to reflect the scene change trend, thus realizing the unified modeling of the movement trajectory of traffic participants and environmental changes.

[0010] The construction process of the implicit state vector set includes time discretization and feature compression. The continuous time series is divided into multiple time segments, and the corresponding neural field feature representation is extracted for each time segment. The high-dimensional feature vector is mapped to the low-dimensional state vector through dimensionality reduction algorithm or encoding network, thereby reducing the computational complexity while ensuring the complete expression of scene semantic information, and making the state representation suitable for dynamic prediction model input.

[0011] The state transition model adopts a long short-term memory network structure, controls the retention and updating of historical information through gating units, takes implicit state vectors and control commands as joint inputs, models the changes in time series, and outputs the probability distribution of the state vector at the next moment, thereby characterizing the dynamic uncertainty and behavioral correlation in the evolution of the traffic environment, so that state changes can reflect the interactive influence between real traffic participants.

[0012] The neural rendering process employs a differentiable rendering mechanism, which maps implicit state vectors to multimodal observation data, including image data, depth information, and point cloud representations, through ray sampling and volume rendering integral calculation. At the same time, sensor pose parameters are introduced to participate in the rendering calculation, so that the generated data is consistent with the real-world data in terms of geometric structure and visual representation, thereby providing input information that conforms to actual perception conditions for the end-to-end autonomous driving model.

[0013] After receiving multimodal observation data, the end-to-end autonomous driving model performs feature extraction and decision reasoning through a deep neural network, and outputs control commands. These control commands include steering control quantities and longitudinal acceleration information, and participate in subsequent state evolution calculations, thereby realizing a direct mapping between perception information and control decisions, enabling the simulation system to fully reproduce the autonomous driving decision-making process.

[0014] The implicit state update process is implemented through an iterative mechanism. In each time loop, the state vector of the previous time step and the corresponding control input are passed to the state transition model to generate the state of the next time step. The time evolution sequence is constructed through continuous updates, thereby forming a complete dynamic simulation trajectory, so that the scene changes can continue to evolve with the control behavior and maintain time consistency.

[0015] The simulation scenario generation process is achieved by perturbing the initial implicit state vector. The perturbation methods include adjusting the position parameters, speed parameters, and environmental attribute parameters of traffic participants, thereby generating scenario configurations under different initial conditions. This enables the simulation system to construct diverse traffic environments to cover different testing needs.

[0016] The control command sequence intervention mechanism introduces random changes or strategic adjustments during the simulation process, causing branching changes in the state evolution path and generating a variety of different evolution results. It also constructs counterfactual scenarios that do not appear in real data through implicit state space interpolation operations, which are used to simulate abnormal traffic behavior or sudden events, thereby improving the simulation test's ability to cover complex scenarios.

[0017] The technical effects and advantages provided by this application in the above technical solution are as follows: This application introduces four-dimensional neural representation and implicit state modeling mechanisms, enabling the simulation environment to continuously express both spatial structure and temporal evolution, thus generating highly realistic dynamic scenes. Building upon this, by optimizing the rendering process and data representation methods, near real-time rendering efficiency is achieved while maintaining visual quality, allowing the simulation system to support high-frequency closed-loop operation. This technical approach not only enhances scene reproduction capabilities but also makes simulation results more closely resemble real driving environments, thereby improving the credibility and stability of the testing process.

[0018] This application constructs a dynamic evolution model based on implicit states to achieve the responsiveness of the simulation environment to control behaviors, enabling continuous updates to the scenario as the decision-making process progresses, thus forming a complete closed-loop interaction mechanism. Simultaneously, leveraging the controllability of the implicit state space, various test scenarios with different evolutionary paths can be generated by adjusting initial conditions and control strategies, giving the simulation system stronger scalability and coverage. End-to-end autonomous driving models validated in this environment are more adaptable to real-world road conditions, thereby enhancing the guiding value of test results for practical applications. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a block diagram of the overall architecture of the end-to-end autonomous driving closed-loop simulation test method of this application. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0022] like Figure 1 As shown, this application provides an end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model, including the following steps: Acquire multi-source spatiotemporal data of real road scenes, including multi-view image sequences, point cloud sequences, and vehicle pose trajectories. Train the multi-source spatiotemporal data to obtain a four-dimensional neural radiation field model representing the dynamic scene, and extract the set of implicit state vectors of the corresponding time series based on the four-dimensional neural radiation field model. Based on the set of implicit state vectors, a state transition model is constructed. The state transition model takes the implicit state vector at the current moment and the control command as input and outputs the prediction result of the implicit state vector at the next moment, which is used to characterize the scene evolution process. An initial implicit state vector is selected, and the simulation process is executed based on the initial implicit state vector. The current implicit state vector is input into the four-dimensional neural radiation field model for rendering to obtain observation data. The observation data is then input into the end-to-end autonomous driving model to obtain the corresponding control commands. The current implicit state vector and corresponding control command are input into the state transition model to obtain the implicit state vector at the next moment. The implicit state vector at the next moment is then used as the updated current implicit state vector. Iteration is performed to achieve the closed-loop simulation process. During the closed-loop simulation, various implicit state evolution paths are generated by adjusting the initial implicit state vector or control command sequence to obtain diverse simulation scenarios for testing end-to-end autonomous driving models.

[0023] In this embodiment, an end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model uses real road environment data as a basis. Through the organic combination of neural field modeling, implicit state expression, dynamic evolution prediction and closed-loop simulation mechanism, it realizes a high-fidelity, interactive and scalable autonomous driving test process.

[0024] First, in the data acquisition phase, a multi-sensor data acquisition vehicle continuously collects data in a real road environment. The acquisition system includes multiple sets of surround-view cameras, a forward-looking telephoto camera, LiDAR, an inertial measurement unit, and a high-precision positioning device. All sensors record data through a unified time synchronization mechanism to ensure consistency of data across different modalities over time. Multi-view RGB image sequences are used to record the appearance information of the road scene, including road surface texture, traffic signs, lighting changes, and visual features of dynamic targets. LiDAR point cloud sequences provide accurate 3D spatial structure information, capable of depicting road boundaries, obstacle shapes, and the spatial positions of traffic participants. Vehicle pose trajectories are acquired through a high-precision positioning system, providing information on the vehicle's position and attitude changes in the global coordinate system, laying the foundation for subsequent data alignment and modeling. During data preprocessing, timestamp alignment is used to synchronize multi-source data, and sensor calibration parameters are combined to perform transformations between different coordinate systems. Simultaneously, filtering algorithms are used to remove abnormal noise data, resulting in a structurally consistent and high-quality multi-source spatiotemporal data set.

[0025] In the neural field modeling stage, a four-dimensional neural radiation field model is used to represent dynamic driving scenes. In this model, three-dimensional spatial coordinates and temporal variables are used as unified inputs, mapped to color information and volume density distribution through a deep neural network, thereby achieving continuous modeling of the scene's appearance and structure. Specifically, a positional encoding mechanism is introduced to frequently expand the input variables, enabling the network to express complex textures and detailed changes, while enhancing its ability to perceive high-frequency spatial and temporal information. The network structure can employ a multi-layer fully connected network, extracting features layer by layer through nonlinear activation functions. During training, volume rendering optimization is performed based on multi-view images. By minimizing the error between the rendered result and the real image, the model gradually learns the changing patterns of the scene at different times. The introduction of temporal variables enables the model to describe dynamic processes; for example, vehicle trajectories, pedestrian paths, and changes in illumination can all be expressed through a unified function, thus achieving spatiotemporally continuous scene modeling.

[0026] After completing the neural field training, the implicit state extraction stage begins. To transform the continuous neural representation into a structured expression suitable for dynamic prediction, the time dimension is discretized, dividing continuous time into multiple time steps. At each time step, the corresponding feature representation is extracted from the neural field. Due to the high dimensionality of the neural field features, a feature compression strategy is introduced to reduce the dimensionality of the high-dimensional representation to improve subsequent computational efficiency. Specifically, the implicit features of the neural field can be mapped to a low-dimensional state space by training the encoding network, or the features can be further compressed using linear dimensionality reduction methods. The implicit state vector sequence obtained through this process can encode complete scene information in a lower dimension, including key elements such as road structure, traffic participant positions, motion states, and environmental appearance. Each time step corresponds to an implicit state vector, which serves as the core data form for subsequent dynamic modeling and simulation calculations.

[0027] In the dynamic evolution modeling phase, a state transition model is constructed based on an implicit state vector sequence. This model employs a recurrent neural network structure and introduces a gating mechanism to model the dependencies in the time series, thereby capturing both short-term changes and long-term trends. During training, the state vectors of consecutive time steps and their corresponding control commands are used as inputs to learn the mapping relationship between states. Control commands originate from driving behavior and can include steering wheel angle, acceleration, and braking signals. Through training with a large amount of real data, the model gradually learns the scene change patterns under different states and control conditions, thus establishing a predictive ability for the dynamic evolution of the traffic environment. Functionally, this model is equivalent to an approximate model of the real physical world, capable of simulating the interaction behaviors between traffic participants and the environmental change process in implicit space.

[0028] In the closed-loop simulation execution phase, an initial state is first selected from the implicit state set as the simulation starting point. This initial state can correspond to a specific moment in a real-world scenario, such as the starting point of an urban road or a section of a highway. The simulation loop then begins. At each time step, the current state is first converted into observation data using a four-dimensional neural radiation field model. The rendering process is based on a volume rendering mechanism, generating data in a format consistent with real devices by setting virtual sensor parameters, including RGB images, depth information, and point cloud data. The generated observation data is input into an end-to-end autonomous driving model, which processes the input information through a neural network structure and outputs the control command for the current moment. Subsequently, the current implicit state and control command are input into a state transfer model to predict the implicit state for the next time step, which is then used as the new current state in the next loop. Through continuous iteration, the environmental state is dynamically updated as time and control inputs change, thus forming a complete closed-loop simulation process. This process can continue until termination conditions are met, such as a collision, deviation from a preset path, or reaching the maximum simulation time.

[0029] In the scenario expansion and diversification generation stage, various state evolution paths are generated by perturbing the initial state or control command sequence. For example, slight changes can be introduced into the initial state, such as altering the position or speed distribution of traffic participants. Random perturbations can also be applied to the control commands during simulation, resulting in different evolution outcomes. Furthermore, interpolation operations can be performed in the implicit state space to combine multiple state vectors and generate new scenario configurations, thus constructing counterfactual scenarios not found in real-world data, such as sudden traffic conflicts, abnormal driving behavior, or complex intersection situations. In this way, a large number of diverse test scenarios can be generated without additional data collection, effectively improving the test coverage capability of autonomous driving models in long-tail scenarios.

[0030] Through the synergistic effect of the above stages, this implementation method realizes a complete technical process from real data-driven modeling to closed-loop simulation testing. It not only has high-fidelity visual performance capabilities, but also the ability to model dynamic environmental evolution. At the same time, it can achieve efficient computation and flexible expansion through implicit state space, thereby providing a reliable and comprehensive testing environment for end-to-end autonomous driving systems.

[0031] In another embodiment, an end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model, while maintaining consistency in the overall technical process, systematically optimizes the neural field representation structure, implicit state modeling method and dynamic evolution mechanism, thereby significantly reducing computational complexity while ensuring high-fidelity modeling capability, and improving dynamic prediction stability and overall performance of the simulation system.

[0032] During the data processing phase, multi-source spatiotemporal data from the real road environment is still acquired through a multi-sensor data acquisition system. Data types include multi-view RGB image sequences, LiDAR point cloud sequences, and vehicle pose and trajectory information. All sensors acquire data synchronously under a unified time reference, and data consistency is achieved through timestamp alignment. Spatially, calibration parameters are used to map data from different sensors to the same coordinate system, thereby achieving multimodal information fusion. In the data preprocessing stage, point cloud noise is removed using filtering algorithms, visual quality is improved through image enhancement methods, and outlier data is removed to ensure the reliability and stability of the input data.

[0033] In the neural field representation construction stage, to enhance the model's ability to express complex dynamic scenes, an efficient spatiotemporal coding structure is introduced to perform multi-scale feature mapping on the input variables. Specifically, spatial coordinates (x, y, z) and the time variable t are used as joint inputs, and the model is extended to a high-dimensional feature space through positional coding. Multi-frequency sine and cosine functions are introduced during the coding process, enabling the model to simultaneously capture low-frequency global structure and high-frequency local details. Furthermore, a hierarchical coding strategy is adopted in the network structure design, where spatial and temporal features are initially extracted separately and then fused in subsequent network layers, thereby improving the model's ability to express complex dynamic changes. For example, in a vehicle motion scene, spatial coding can characterize the vehicle's three-dimensional structural information, while temporal coding can describe the trajectory features of the vehicle's position changing over time. Combining the two can more accurately express the dynamic process.

[0034] In the implicit state extraction stage, the trained neural field is discretized temporally, dividing continuous time into multiple time steps. At each time step, the corresponding implicit feature representation is extracted. Since the output features of the neural field have high dimensionality, directly using them for subsequent dynamic modeling would incur high computational costs. Therefore, feature compression and dimensionality reduction strategies are introduced at this stage to map the high-dimensional representation to a compact state space. Specifically, an autoencoder structure can be used to compress high-dimensional features into low-dimensional state vectors through an encoding network, and information integrity is ensured through reconstruction error constraints. Simultaneously, linear dimensionality reduction methods can be used to further optimize the features to reduce redundant information. The implicit state vectors obtained through the above processing have low dimensionality and high expressive power, capable of describing complete scene information, including the positions and motion states of traffic participants, and environmental structure, under limited computational resources.

[0035] In the dynamic modeling phase, a Long Short-Term Memory (LSTM) network is employed to model the implicit state sequence, capturing long-term dependencies and short-term dynamic changes in the time series. This model regulates information flow through a gating mechanism, preserving important historical information across multiple time steps, thereby improving prediction stability. During training, the current state vector and corresponding control commands are used as inputs to learn the state transition probability relationship, i.e., predicting the next state given the current state and control input. Control commands originate from driving behavior, including steering, acceleration, and braking. Through training with a large amount of real data, the model gradually learns the dynamic changes in the traffic environment, enabling it to generate reasonable future states under different scenario conditions. To further improve model robustness, random noise or data augmentation strategies can be introduced during training to allow the model to adapt to different environmental changes.

[0036] In the closed-loop simulation execution phase, an initial state is first selected from the implicit state set as the simulation starting point. This state can correspond to any time point in the real scene. After entering the simulation loop, at each time step, the current state is converted into observation data through a neural rendering process. The rendering process is based on four-dimensional neural representation, generating multimodal data, including RGB images, depth information, and point cloud representations, by setting the pose of virtual sensors, thereby simulating the output of real sensors. The generated observation data is input into the end-to-end autonomous driving model, which performs inference calculations based on the input information and outputs control commands. Subsequently, the current state and control commands are input into the state transition model to obtain the state representation of the next time step, which is then updated to the current state to participate in the next round of the loop. Through continuous iteration, the environmental state dynamically evolves with time and control input, thus constructing a complete closed-loop simulation process.

[0037] To further enhance test coverage during simulation, random perturbation and strategic intervention mechanisms are introduced. In the initial state selection phase, slight perturbations can be applied to the scenario, such as adjusting the position, speed, or direction of some traffic participants, thereby generating different initial conditions. During simulation execution, random variations or strategic adjustments can be applied to the control command sequence, such as simulating driver errors or abnormal behavior, thus generating different state evolution paths. Furthermore, interpolation operations can be performed in the implicit state space, combining multiple state vectors to generate new scenario configurations, thereby constructing counterfactual scenarios not found in real-world data. For example, scenarios such as sudden lane changes by vehicles, abnormal pedestrian crossings, or complex traffic conflicts can be generated. Through these methods, a set of test scenarios covering various complex traffic behaviors can be constructed, including normal driving behavior, abnormal behavior, and extreme scenarios, significantly improving the comprehensiveness of autonomous driving model testing.

[0038] By optimizing the neural representation structure and state modeling method, this implementation method effectively reduces computational complexity while ensuring high-fidelity modeling capabilities, and improves dynamic prediction stability and simulation efficiency. This enables the system to achieve high-quality closed-loop simulation under limited computing resources, thereby providing a more realistic, stable and rich testing environment for end-to-end autonomous driving models.

[0039] In a further embodiment, an end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model, while maintaining the overall process, expands the scene representation method and dynamic modeling mechanism by introducing a four-dimensional Gaussian sputtering representation and a state generation method driven by a diffusion model. This significantly improves rendering efficiency and scene generation capability while ensuring high-fidelity visual expression, enabling the system to support closed-loop simulation test tasks at higher frequencies and on a larger scale.

[0040] During the data acquisition and preprocessing phase, multi-source spatiotemporal data from the real road environment are still acquired through a multi-sensor acquisition platform. Data types include multi-view RGB image sequences, LiDAR point cloud data, and vehicle pose trajectory information. During acquisition, various sensors record synchronously using a unified time reference to ensure consistency of data across different modalities in the temporal dimension. After data acquisition, timestamp alignment is used to synchronize multi-source data, and spatial coordinate transformation between different sensors is performed using calibration parameters. In the preprocessing phase, outliers and noise are removed using point cloud filtering algorithms, and visual data quality is improved through image enhancement and deblurring. Abnormal data is also filtered and removed, thus constructing a high-quality input dataset to provide a stable foundation for subsequent modeling.

[0041] In the scene representation stage, a four-dimensional Gaussian sputtering method is used to replace the traditional neural radiation field for dynamic scene modeling. This method represents the scene using a set of learnable Gaussian distributions, each corresponding to a basic element in the scene. This element is described by multiple parameters, including spatial location, scale, rotation direction, color information, and transparency. In the temporal dimension, a time variable is introduced to dynamically model the Gaussian distribution parameters, allowing each Gaussian element to change position, shape, or appearance over time. For example, during vehicle movement, the center position of the Gaussian distribution shifts over time, the scale parameter can be used to express changes in vehicle size, and the rotation parameter can reflect changes in vehicle posture. To further describe complex dynamic behaviors, a deformation field mechanism can be introduced. By learning the displacement mapping of spatial points at different times, the dynamic changes of non-rigid objects, such as pedestrian limb movement or deformation of flexible objects, can be realized. Compared to the volume-based neural radiation field method, Gaussian sputtering avoids a large number of ray sampling calculations during the rendering process by projecting the Gaussian distribution onto the image plane and performing weighted fusion, significantly reducing computational complexity and achieving near-real-time or even real-time rendering performance.

[0042] In the implicit state extraction stage, the parameters in the four-dimensional Gaussian representation are encoded to map the attribute information of all Gaussian distributions in the scene into a unified implicit state vector. This state vector can be obtained by concatenating or aggregating the key parameters of each Gaussian element, such as combining parameters like position, scale, and color to form a feature vector. To control the dimensionality of the state space, clustering methods can be used to group the Gaussian distributions, grouping spatially adjacent or semantically related Gaussian elements into one class and extracting representative features from each class, thereby achieving dimensionality reduction. Simultaneously, an attention mechanism can be used to assign weights to different regions, giving higher weights to areas with dense traffic participants or key target areas in the state representation, thus enhancing the effectiveness of state expression. The implicit state vector obtained after this processing maintains a low dimensionality while fully describing the key structures and dynamic information in the scene.

[0043] In the dynamic modeling phase, a diffusion model is introduced as a state transition mechanism to replace traditional prediction methods based on recurrent neural networks. The diffusion model models complex data distributions by progressively adding noise to the data and learning an inverse denoising process. During training, random noise is gradually added to the real state sequence to transform it into an approximate Gaussian distribution, and then the neural network is trained to learn the process of recovering from the noisy state to the original state. In this process, the current state and control commands serve as conditional inputs to the model training, enabling the model to learn the state evolution patterns under different control conditions. In the inference phase, starting from random noise, the state distribution for the next time step is gradually generated through a multi-step denoising process. This method can generate multiple possible future states, thus reflecting the uncertainty and diversity of real traffic environments. For example, in the same traffic situation, different driving behaviors may lead to different outcomes; the diffusion model can generate multiple evolution paths, thereby enhancing the realism of the simulation system.

[0044] In the closed-loop simulation execution phase, an initial state is first selected from the implicit state set as the simulation starting point. This state can originate from real-world data acquisition or be generated. At each simulation time step, the current state is rendered using a four-dimensional Gaussian sputtering representation. Observational data, including RGB images, depth information, and point cloud data, is generated based on virtual sensor parameters. The rendering process is achieved through Gaussian projection calculations, mapping the Gaussian distribution onto the image plane and performing weighted fusion to quickly generate high-quality image results. The generated observational data is input into the end-to-end autonomous driving model, which outputs control commands based on the current environmental information. Subsequently, the current state and control commands are input into the diffusion model, and the state for the next time step is generated through a multi-step denoising process and updated to the current state for the next cycle. Through continuous iteration, the continuous evolution of the environmental state in the time dimension is achieved, constructing a complete closed-loop simulation process.

[0045] In the scenario generation and expansion phase, a large number of counterfactual scenarios are generated by diversifying the initial state and control strategy. Specifically, random variations can be introduced into the initial state, such as adjusting some Gaussian distribution parameters to change the positions of traffic participants or environmental configurations. Random noise can also be introduced during the diffusion generation process to control the generation path, thus obtaining different future states. Furthermore, interventions can be made at the control strategy level, such as simulating abnormal driving behaviors like sudden acceleration, emergency braking, or irregular lane changes, thereby generating complex traffic events. Through these methods, test data covering various extreme situations and long-tail scenarios can be constructed, such as sudden traffic conflicts, abnormal behaviors, and complex intersection situations. Compared to traditional methods that rely on manual scenario design, this method can automatically generate a large number of diverse scenarios, significantly reducing testing costs and increasing test coverage.

[0046] By introducing a four-dimensional Gaussian sputtering representation and diffusion model, this implementation method achieves significant improvements in rendering efficiency, state generation capability, and scene diversity. It can not only achieve high-fidelity visual expression, but also generate dynamic scenes with diversity and uncertainty, thereby providing a more realistic, comprehensive, and efficient simulation testing environment for end-to-end autonomous driving systems.

[0047] The foregoing has only described certain exemplary embodiments of this application by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of this application. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of this application.

Claims

1. An end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model, characterized in that, Includes the following steps: Acquire multi-source spatiotemporal data of real road scenes, including multi-view image sequences, point cloud sequences, and vehicle pose trajectories. Train the multi-source spatiotemporal data to obtain a four-dimensional neural radiation field model representing the dynamic scene, and extract the set of implicit state vectors of the corresponding time series based on the four-dimensional neural radiation field model. A state transition model is constructed based on the set of implicit state vectors. The state transition model takes the implicit state vector at the current moment and the control command as input and outputs the prediction result of the implicit state vector at the next moment, which is used to characterize the scene evolution process. An initial implicit state vector is selected, and the simulation process is executed based on the initial implicit state vector. The current implicit state vector is input into the four-dimensional neural radiation field model for rendering to obtain observation data. The observation data is then input into the end-to-end autonomous driving model to obtain the corresponding control commands. The current implicit state vector and corresponding control command are input into the state transition model to obtain the implicit state vector at the next moment. The implicit state vector at the next moment is then used as the updated current implicit state vector. Iteration is performed to achieve the closed-loop simulation process. During the closed-loop simulation, various implicit state evolution paths are generated by adjusting the initial implicit state vector or control command sequence to obtain diverse simulation scenarios for testing end-to-end autonomous driving models.

2. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 1, characterized in that, The multi-source spatiotemporal data acquisition process adopts a multi-sensor synchronous acquisition mechanism. The acquisition equipment includes a multi-view camera unit, a laser scanning unit, and an inertial measurement unit. All acquired signals are synchronously recorded through a unified time reference, and spatial coordinates are uniformly mapped through calibration parameters. At the same time, combined with filtering and outlier data removal operations, a multimodal data set that is temporally continuous and spatially consistent is obtained.

3. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 1, characterized in that, The four-dimensional neural radiation field model adopts a spatiotemporal joint coding structure. By expanding the spatial coordinates and time variables with high-frequency position coding, the low-dimensional input is mapped to a high-dimensional feature space. Feature extraction is completed through a multi-layer nonlinear network, and color information and volume density distribution are output, thereby realizing the continuous expression of dynamic scenes. The time variable participates in the feature coding process to reflect the scene change trend.

4. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 1, characterized in that, The construction process of the implicit state vector set includes time discretization and feature compression. The continuous time series is divided into multiple time segments, and the corresponding neural field feature representation is extracted for each time segment. The high-dimensional feature vector is mapped to the low-dimensional state vector through dimensionality reduction algorithm or encoding network.

5. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 4, characterized in that, The state transition model adopts a long short-term memory network structure, controls the retention and updating of historical information through gating units, takes the implicit state vector and control commands as joint inputs, models the changes in the time series, and outputs the probability distribution of the state vector at the next moment, thereby characterizing the dynamic uncertainty and behavioral correlation in the evolution of the traffic environment.

6. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 5, characterized in that, The neural rendering process employs a differentiable rendering mechanism, which maps implicit state vectors to multimodal observation data, including image data, depth information, and point cloud representations, through ray sampling and volume rendering integral calculation. Sensor pose parameters are also incorporated into the rendering calculation.

7. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 1, characterized in that, After receiving multimodal observation data, the end-to-end autonomous driving model performs feature extraction and decision reasoning through a deep neural network, and outputs control commands. These control commands include steering control quantities and longitudinal acceleration information, and participate in subsequent state evolution calculations, realizing a direct mapping between perception information and control decisions.

8. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 6, characterized in that, The implicit state update process is implemented through an iterative mechanism. In each time loop, the state vector of the previous time step and the corresponding control input are passed to the state transition model to generate the state of the next time step. The time evolution sequence is constructed through continuous updates to form a complete dynamic simulation trajectory.

9. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 8, characterized in that, The simulation scene generation process is achieved by perturbing the initial implicit state vector. The perturbation methods include adjusting the position parameters, speed parameters, and environmental attribute parameters of traffic participants to generate scene configurations under different initial conditions.

10. The end-to-end autonomous driving closed-loop simulation test method based on 4D neural representation and world model according to claim 9, characterized in that, The control command sequence intervention mechanism introduces random changes or strategic adjustments during the simulation process, causing branching changes in the state evolution path and generating a variety of different evolution results. It also constructs counterfactual scenarios that do not appear in real data through implicit state space interpolation operations to simulate abnormal traffic behavior or sudden events.

Images