End-to-end automatic driving system and control method based on world model reinforcement learning

Abstract

The invention provides an end-to-end automatic driving system and control method based on world model reinforcement learning, and the system comprises a multi-modal environment data obtaining module which is used for obtaining multi-modal environment data; the state hidden variable coding network is used for generating a state hidden variable at the current moment according to the multi-modal environment data; the state transition variable coding network is used for generating a dynamic hidden variable in a state transition process according to the state hidden variable at the current moment and the automatic driving action at the previous moment; and the strategy network is initialized by adopting preset expert data, trains the reinforcement learning model based on the actor-commentator architecture by adopting a KL divergence loss and soft update mechanism, and is used for generating an automatic driving action according to a state hidden variable at the current moment and a dynamic hidden variable decision in a state transition process. The method is based on the world model, combines imitation learning and reinforcement learning, supports the long-term driving decision, accelerates the strategy convergence, and improves the safety and stability of automatic driving.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to an end-to-end autonomous driving system and control method based on world model reinforcement learning. Background Technology

[0002] Current autonomous driving systems largely rely on modular architectures, where perception, decision-making, and control are separated. While these systems offer advantages in terms of controllability, they also suffer from significant limitations in end-to-end control capabilities, autonomous learning, and adaptability to complex dynamic environments. Existing end-to-end methods, such as those based on behavior cloning, often lack the ability to dynamically model the environment; while reinforcement learning methods, although possessing strong generalization capabilities, are insufficient in terms of sample efficiency and safety.

[0003] In recent years, world models have been proposed to address the problem of collaborative modeling of perception and prediction, but a systematic implementation that organically integrates with end-to-end driving behavior is lacking. Therefore, existing technologies have not yet formed an end-to-end autonomous driving solution that can operate stably in multiple scenarios, possesses high sample efficiency, and demonstrates strong environmental understanding capabilities.

[0004] With the continuous evolution of technology, end-to-end learning methods have become a key direction in autonomous driving research. This method directly maps sensor inputs to control commands through deep learning models, offering advantages such as simple system structure and avoidance of error accumulation. However, existing end-to-end methods still face significant challenges when dealing with complex traffic interactions and dynamically changing scenarios.

[0005] Existing autonomous driving systems are mainly divided into two categories: one is the end-to-end method based on imitation learning, which trains the model by learning from expert demonstrations. This method features fast convergence and good stability, but its generalization ability is limited due to its reliance on fixed data distributions and expert trajectories, and it is prone to failure, especially in unseen scenarios or edge cases. The other category is the method based on reinforcement learning, which optimizes strategies through interaction with the environment, possessing stronger adaptability and exploratory capabilities. Nevertheless, reinforcement learning-based methods face high training costs and low efficiency in real-world environments, and struggle to achieve stable convergence under high-dimensional perceptual inputs.

[0006] In patent "CN119647561A, an end-to-end autonomous driving model training method and apparatus," an autonomous driving model is built based on an "actor-critic" framework. It employs the DDPG algorithm with adaptive parameter space noise and updates the critic-actor network parameters using experience from an experience revisit pool to adjust the performance of the autonomous vehicle. However, its use of the DDPG algorithm with adaptive parameter space noise and the updating of the critic-actor network with experience from an experience revisit pool essentially still relies on expert data and trajectories, and is not linked to reinforcement learning, resulting in limited generalization ability.

[0007] To address the aforementioned technical limitations, there is an urgent need for a new method that can balance the stability of imitation with the generalization of reinforcement learning, while effectively modeling environmental dynamics and improving decision-making capabilities in complex traffic scenarios. Summary of the Invention

[0008] To address one of the shortcomings of existing technologies, the purpose of this application is to provide an end-to-end autonomous driving system and control method based on world model reinforcement learning.

[0009] A first aspect of this application provides an end-to-end autonomous driving system based on world model reinforcement learning, comprising:

[0010] A multimodal environment data acquisition module is used to acquire multimodal environment data, which includes multi-view images, lidar point clouds, and path point information.

[0011] A state latent variable encoding network is used to generate the state latent variables at the current moment based on the multimodal environment data;

[0012] A state transition variable encoding network is used to generate dynamic latent variables in the state transition process based on the current state latent variables and the autonomous driving action of the previous time step.

[0013] A policy network is used to generate the autonomous driving action at the current moment based on the latent variables of the current state and the latent variables of the dynamics during the state transition process. The policy network includes a reinforcement learning model based on the actor-critic architecture. The policy network is initialized with preset expert data and trained using KL divergence loss and a soft update mechanism.

[0014] Optionally, the state latent variable encoding network includes an image encoder, a point cloud encoder, and a first multilayer perceptron. The image encoder is used to encode features of the multi-view image to determine image features. The point cloud encoder is used to encode the LiDAR point cloud to determine point cloud features. The first multilayer perceptron is used to encode the path point information to determine path point features.

[0015] Optionally, the state latent variable encoding network further includes an asymmetric variational autoencoder and a decoder. The asymmetric variational autoencoder is used to extract state latent variables. The input of the asymmetric variational autoencoder is the image features, the point cloud features, and the path point features. The output of the asymmetric variational autoencoder is the state latent variables.

[0016] The decoder is used to reconstruct a semantic graph from the latent state variables. The input of the decoder is the latent state variables, and the output of the decoder is a bird's-eye view semantic graph.

[0017] Optionally, the state transition latent variable encoding network includes a second multilayer perceptron, a Transformer module, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron. The second multilayer perceptron is used to generate the latent variable at the current moment. The input of the second multilayer perceptron is the state latent variable and the autonomous driving action at the previous moment. The output of the second multilayer perceptron is the latent variable at the current moment.

[0018] The Transformer module is used to generate the latent dynamic variables in the state transition process. The input of the Transformer module is the latent variable at the current moment and the latent variable at the historical moment. The output of the Transformer module is the latent dynamic variables in the state transition process.

[0019] The third multilayer perceptron is used to generate the task termination condition. The input of the third multilayer perceptron is the dynamic hidden variable in the state transition process, and the output of the third multilayer perceptron is the task termination condition.

[0020] The fourth multilayer perceptron is used to generate rewards. The input of the fourth multilayer perceptron is the dynamic latent variable in the state transition process, and the output of the fourth multilayer perceptron is the reward of the current state or action.

[0021] The fifth multilayer perceptron is used to generate the state latent variable for the next time step. The input of the fifth multilayer perceptron is the dynamic latent variable in the state transition process, and the output of the fifth multilayer perceptron is the state latent variable for the next time step.

[0022] Optionally, the reinforcement learning model based on the actor-critic architecture includes an actor network and a critic network. The actor network is used to generate the autonomous driving action based on the latent state variables at the current moment and the latent dynamic variables during the state transition process. The critic network is used to generate policy values ​​based on the autonomous driving action. The autonomous driving action includes steering wheel angle and predicted driving distance.

[0023] Optionally, the policy network further includes an action mask module, used to assess the dynamic feasibility of the autonomous driving action generated from the LiDAR point cloud, and to filter out invalid actions in the autonomous driving action based on the dynamic feasibility of the autonomous driving action.

[0024] A second aspect of this application provides an end-to-end autonomous driving control method based on world model reinforcement learning, comprising:

[0025] Acquire multimodal environment data, which includes multi-view images, lidar point clouds, and path point information;

[0026] The multimodal environment data is input into a preset state latent variable encoding network to determine the state latent variables at the current moment;

[0027] The current state hidden variables and the previous autonomous driving action are input into a preset state transition variable encoding network to determine the dynamic hidden variables in the state transition process;

[0028] The current state latent variables and the dynamic latent variables during the state transition process are input into a preset policy network to determine the autonomous driving action at the current moment. The policy network includes a reinforcement learning model based on an actor-critic architecture.

[0029] Optionally, the preset state latent variable coding network includes an image encoder, a point cloud encoder, a first multilayer perceptron, an asymmetric variational autoencoder, and a decoder.

[0030] The preset state transition latent variable encoding network includes a second multilayer perceptron, a Transformer module, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron.

[0031] The step of inputting the multimodal environment data into a preset state latent variable encoding network to determine the state latent variables at the current moment includes:

[0032] The multi-view images are input into the image encoder for feature encoding to determine image features;

[0033] The lidar point cloud is input into the point cloud encoder for point cloud encoding to determine the point cloud features;

[0034] The path point information is input into the first multilayer perceptron for encoding to determine the path point features;

[0035] The image features, point cloud features, and path point features are input into the asymmetric variational autoencoder for encoding to determine the state latent variables;

[0036] The state hidden variables are input into the decoder to reconstruct the semantic graph and determine the bird's-eye view semantic graph.

[0037] The step of inputting the current state latent variables and the previous autonomous driving action into a preset state transition variable encoding network to determine the dynamic latent variables in the state transition process includes:

[0038] The current state hidden variable and the previous autonomous driving action are input into the second multilayer sensor to determine the hidden variable at the current moment;

[0039] The latent variables at the current moment and the latent variables at historical moments are input into the Transformer module to determine the dynamic latent variables in the state transition process;

[0040] The hidden dynamic variables in the state transition process are input into the third multilayer perceptron to determine the task termination condition;

[0041] The latent dynamic variables in the state transition process are input into the fourth multilayer perceptron to determine the reward for the current state or action.

[0042] The latent dynamic variables in the state transition process are input into the fifth multilayer perceptron to determine the latent state variables at the next moment.

[0043] Optionally, the preset policy network includes the actor-critic architecture-based reinforcement learning model and a preset action mask module, wherein the actor-critic architecture-based reinforcement learning model includes an actor network and a critic network;

[0044] The step of inputting the current state latent variables and the dynamic latent variables during the state transition process into a preset policy network to determine the current autonomous driving action includes:

[0045] The current state hidden variables and the dynamic hidden variables in the state transition process are input into the actor network to determine the autonomous driving action, which includes the steering wheel angle and the predicted driving distance.

[0046] The autonomous driving action is input into the critic network to determine the policy value of the autonomous driving action;

[0047] The parameters of the preset policy network are updated based on the policy value of the autonomous driving action;

[0048] The autonomous driving action is input into a preset action mask module to filter out invalid actions and determine the valid autonomous driving action at the current moment.

[0049] Optionally, the method for training the preset state latent variable encoding network, the preset state transition variable encoding network, and the preset policy network includes:

[0050] The preset state latent variable coding network is trained using a bird's-eye view semantic map of a single time frame to determine the state latent variable coding network after the first training.

[0051] The state latent variable encoding network and the state transition variable encoding network trained for the first time are jointly trained using a bird's-eye view semantic map of a preset time series, and the state latent variable encoding network trained for the second time and the state transition variable encoding network trained for the first time are determined.

[0052] Freeze the parameters of the state latent variable encoding network trained for the second time and the state transition variable encoding network trained for the first time, initialize the preset policy network using preset expert data, and determine the initialized policy network.

[0053] Unfreeze the parameters of the second-trained state latent variable encoding network and the first-trained state transition variable encoding network, and use KL divergence loss and soft update mechanism to jointly train the second-trained state latent variable encoding network, the first-trained state transition variable encoding network, and the initialized policy network based on the reinforcement learning paradigm to determine the completed state latent variable encoding network, the completed state transition variable encoding network, and the completed policy network.

[0054] The end-to-end autonomous driving system based on world model reinforcement learning in this application realizes the latent variable encoding of environmental state and state transition through a state latent variable encoding network and a state transition variable encoding network based on the world model architecture, which supports long-term decision-making of autonomous driving. Furthermore, it adopts a policy network that combines imitation learning and reinforcement learning to solve the cold start problem of reinforcement learning, accelerate policy convergence, and improve the stability of reinforcement learning training, thereby enhancing the safety and stability of autonomous driving.

[0055] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0056] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0057] Figure 1 This is a schematic diagram illustrating the structure of an end-to-end autonomous driving system based on world model reinforcement learning, according to an exemplary embodiment.

[0058] Figure 2 This is a schematic diagram illustrating the structure of a state latent variable encoding network according to an exemplary embodiment.

[0059] Figure 3 This is a schematic diagram illustrating the structure of a state transition variable encoding network according to an exemplary embodiment.

[0060] Figure 4 This is a schematic diagram illustrating the structure of a policy network according to an exemplary embodiment.

[0061] Figure 5 This is a flowchart illustrating an end-to-end autonomous driving control method based on world model reinforcement learning, according to an exemplary embodiment. Detailed Implementation

[0062] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0063] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0064] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0065] Existing autonomous driving systems include imitation-based end-to-end methods and reinforcement learning-based methods. Imitation-based end-to-end methods rely on fixed data distributions and expert trajectories, limiting their generalization ability. Reinforcement learning-based methods have high training costs and low training efficiency in real-world environments, and struggle to achieve stable convergence under high-dimensional perceptual inputs. To address these issues, this application provides an end-to-end autonomous driving system based on world model reinforcement learning to resolve these problems.

[0066] Figure 1 This is a schematic diagram illustrating the structure of an end-to-end autonomous driving system based on world model reinforcement learning, according to an exemplary embodiment.

[0067] Reference Figure 1 As shown in one embodiment of this application, an end-to-end autonomous driving system based on world model reinforcement learning includes: a multimodal environment data acquisition module, a state latent variable encoding network, a state transition variable encoding network, and a policy network.

[0068] The multimodal environment data acquisition module is used to acquire multimodal environment data.

[0069] Specifically, multimodal environment data includes multi-view images, LiDAR point clouds, and pathpoint information.

[0070] The multimodal environment data acquisition module may include a multi-view image acquisition device and a lidar sensor. The multi-view image acquisition device acquires multi-view images of the current environment, and the lidar sensor acquires lidar point clouds of the current environment and path point information of the current vehicle's driving path.

[0071] A state latent variable encoding network is used to generate state latent variables at the current moment based on multimodal environment data.

[0072] Specifically, the state latent variable encoding network compresses high-dimensional, multimodal environmental data from environmental observations into low-dimensional, compact state latent variables that contain semantic information.

[0073] A state transition variable encoding network is used to generate dynamic latent variables in the state transition process based on the current state latent variables and the autonomous driving action of the previous time step.

[0074] Specifically, the state transition variable encoding network models the current environment transition pattern based on the current state latent variables and the autonomous driving action of the previous moment, predicts the termination condition, reward, and state distribution of the next moment, and thus generates the dynamic latent variables in the state transition process.

[0075] The policy network is used to generate autonomous driving actions based on the hidden variables of the current state and the hidden variables of the dynamics during the state transition process.

[0076] Specifically, autonomous driving actions include steering wheel angle and predicted driving distance.

[0077] The policy network includes a reinforcement learning model based on an actor-critic architecture. The policy network is initialized with pre-set expert data and trained using KL divergence loss and a soft update mechanism.

[0078] The embodiments described above employ a state latent variable encoding network and a state transition variable encoding network based on a world model architecture to process high-dimensional multimodal environmental data. This effectively compresses high-dimensional sensor data, retains key environmental information, improves adaptability to complex dynamic traffic scenarios, and enables latent variable encoding of environmental states and state transition processes, supporting long-term driving decisions. The policy network is initialized with preset expert data for imitation learning, and then KL divergence loss and a soft update mechanism are used to transition the actor-critic architecture-based reinforcement learning model to reinforcement learning training. By combining imitation learning and reinforcement learning, the policy network addresses the cold start problem of reinforcement learning, accelerates policy convergence, and improves the stability of training the actor-critic architecture-based reinforcement learning model. Furthermore, by generating autonomous driving actions through imitation learning and reinforcement learning, the safety and stability of autonomous driving are improved.

[0079] Figure 2 This is a schematic diagram illustrating the structure of a state latent variable encoding network according to an exemplary embodiment.

[0080] Reference Figure 2 As shown, in order to extract environmental state latent variables, in some specific embodiments of this application, the state latent variable encoding network includes an image encoder, a point cloud encoder, and a first multilayer perceptron.

[0081] Image encoders are used to encode features in multi-view images and determine image features.

[0082] Specifically, the image encoder extracts image features from multi-view images based on the Swing Transformer.

[0083] Point cloud encoders are used to encode point clouds from lidar radar data to determine point cloud features.

[0084] The point cloud encoder extracts point cloud features from LiDAR point clouds based on PointPillars.

[0085] The first multilayer perceptron is used to encode path point information and determine path point features.

[0086] To extract environmental state latent variables, in some specific embodiments of this application, the state latent variable encoding network further includes an asymmetric variational autoencoder and a decoder. The asymmetric variational autoencoder is used to extract state latent variables. The input of the asymmetric variational autoencoder is image features, point cloud features, and path point features. The output of the asymmetric variational autoencoder is the state latent variables.

[0087] Specifically, before inputting image features, point cloud features, and path point features into the asymmetric variational autoencoder, the image features are sequentially subjected to viewpoint transformation to obtain image features with a unified viewpoint; the point cloud features are flattened to regularize the point cloud features; and a cross-attention mechanism is used to enhance the features of the path points to obtain enhanced path point features.

[0088] Image features from a unified perspective, regularized point cloud features, and enhanced path point features are aggregated and input into an asymmetric variational autoencoder, which then outputs the latent state variables.

[0089] The image features from a unified perspective and the regularized point cloud features are added together and then input into the asymmetric variational autoencoder.

[0090] The decoder is used to reconstruct the semantic graph of the state latent variables. The input of the decoder is the state latent variables, and the output of the decoder is the bird's-eye view semantic graph.

[0091] The embodiments described above in this application employ a state latent variable coding network, using an asymmetric variational autoencoder and Transformer architecture to compress high-dimensional multimodal environmental data into low-dimensional state latent variables, while retaining key environmental information.

[0092] Figure 3 This is a schematic diagram illustrating the structure of a state transition variable encoding network according to an exemplary embodiment.

[0093] Reference Figure 3 As shown in some specific embodiments of this application, the state transition latent variable encoding network includes a second multilayer perceptron, a Transformer module, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron.

[0094] The second layer perceptron is used to generate the latent variables at the current moment. The input of the second layer perceptron is the state latent variables and the autonomous driving action at the previous moment. The output of the second layer perceptron is the latent variables at the current moment.

[0095] Specifically, the state hidden variable represents the encoded representation of the environment state.

[0096] The Transformer module is used to generate the latent dynamic variables in the state transition process. The inputs of the Transformer module are the latent variables at the current time step and the latent variables at previous time steps, and the outputs of the Transformer module are the latent dynamic variables in the state transition process.

[0097] Specifically, the latent variables of the historical moments input to the Transformer module include e t-n to e t-1The Transformer module uses its attention mechanism to capture long-term dependencies in sequences, thereby uncovering temporal correlations and outputting the latent dynamic variables h in the state transition process. t It predicts the termination condition at the current moment, the reward at the current moment, and the state distribution at the next moment.

[0098] The third multilayer perceptron is used to generate the task termination condition. The input of the third multilayer perceptron is the dynamic hidden variable in the state transition process, and the output of the third multilayer perceptron is the task termination condition.

[0099] Specifically, the third multilayer sensor predicts the traffic state at the next moment to determine the task termination condition.

[0100] The fourth multilayer perceptron is used to generate rewards. The input to the fourth multilayer perceptron are the latent dynamic variables in the state transition process, and the output of the fourth multilayer perceptron is the reward for the current state or action.

[0101] The fifth multilayer perceptron is used to generate the state distribution at the next time step. The input of the fifth multilayer perceptron is the dynamic latent variable in the state transition process, and the output of the fifth multilayer perceptron is the state distribution at the next time step.

[0102] The embodiments described above in this application employ a state transition variable encoding network to generate latent dynamic variables in the state transition process, thereby improving adaptability to complex traffic scenarios.

[0103] Figure 4 This is a schematic diagram illustrating the structure of a policy network according to an exemplary embodiment.

[0104] Reference Figure 4 As shown in some specific embodiments of this application, the policy network includes a reinforcement learning model based on an actor-critic architecture and an action mask module.

[0105] The reinforcement learning model based on the actor-critic architecture includes an actor network and a critic network. The actor network is used to generate autonomous driving actions based on the latent variables of the current state and the latent dynamic variables during the state transition process, while the critic network is used to generate policy values ​​based on the autonomous driving actions.

[0106] Specifically, the latent variables of the current state and the latent variables of the dynamics during the state transition process are used as inputs to the actor network. The actor network learns to generate action strategies, i.e. autonomous driving actions, based on the encoding environment state and encoding time-series dynamic information. The autonomous driving actions include steering wheel angle and predicted driving distance.

[0107] The critic network evaluates the action strategies generated by the actor network, generates strategy value, and provides feedback on the strategy value to optimize the action strategies.

[0108] The policy network of this application is first initialized with preset expert data, which consists of human drivers’ driving actions and corresponding driving operations in various driving scenarios, such as steering wheel turning, accelerator, and braking. Using the preset expert data as learning samples, the mapping relationship between state and action in the expert data is learned through supervision. The policy network initially has the ability to generate reasonable driving actions, realizing the imitation learning of the policy network.

[0109] The policy network employs a reinforcement learning model based on an actor-critic architecture. During the transition from imitation learning to reinforcement learning, KL divergence loss is used to measure the difference between the action policies of imitation learning and reinforcement learning, and the action policies are gradually adjusted to allow reinforcement learning to gradually cover imitation learning. By minimizing KL divergence loss, excessive deviation from the action policies of imitation learning is prevented during the reinforcement learning process.

[0110] The parameters of the critic network are updated using a soft update mechanism, gradually updating the parameters of the critic network to the parameters of the target network. The actor network updates its action policy using policy gradient based on the policy value provided by the critic network and the reward value of environmental feedback, maximizing the long-term cumulative reward.

[0111] The policy network also includes an action masking module, which is used to generate the dynamic feasibility of autonomous driving actions based on the LiDAR point cloud, and filter invalid actions in the autonomous driving actions based on the dynamic feasibility of the autonomous driving actions.

[0112] Invalid actions are those that pose a high risk of collision in the future trajectory.

[0113] Specifically, the autonomous driving actions output by the reinforcement learning model based on the actor-critic architecture are used as the input to the action masking module to filter out invalid actions and output valid autonomous driving actions a. t Such as steering wheel angle and predicted driving distance.

[0114] The steering wheel angle is controlled by steering wheel operation, while the predicted driving distance is controlled by accelerator or brake operation.

[0115] Thus, the policy network learns the optimal action policy by interacting with the environment. After each interaction, the policy network generates autonomous driving actions based on the current state, namely, steering wheel angle, accelerator or brake decisions.

[0116] In the embodiments described above, the policy network is first initialized with preset expert data for imitation learning, and then reinforced learning is performed using a reinforcement learning model based on an actor-critic architecture. The transition from imitation learning to reinforcement learning is achieved through KL divergence loss and a soft update mechanism, thereby effectively combining imitation learning and reinforcement learning. This solves the cold start problem in the early stages of reinforcement learning, reduces the training time, improves training stability, and avoids action policy degradation. The policy network also uses an action mask module to filter invalid actions to ensure driving safety.

[0117] This application also employs a phased reward design to ensure driving safety, while the lightweight model supports real-time decision-making.

[0118] This application provides an end-to-end autonomous driving system based on world model reinforcement learning, which is an end-to-end solution that balances performance, efficiency and safety. It overcomes the shortcomings of existing end-to-end autonomous driving technologies, such as low sample efficiency, poor environmental modeling ability and weak generalization ability. It can be widely applied to complex and ever-changing traffic scenarios such as urban roads and highways.

[0119] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0120] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0121] Application Example 1:

[0122] In unprotected left-turn driving scenarios, autonomous vehicles need to turn left from the main road into an intersection where there is no dedicated left-turn lane or traffic light indication. In this situation, the autonomous vehicle's control system must respond to the dynamic behavior of other traffic participants, including vehicles going straight or turning right, as well as potential pedestrians or non-motorized vehicles.

[0123] This application employs an end-to-end autonomous driving system based on world model reinforcement learning. A multimodal environment data acquisition module acquires multi-view RGB images, LiDAR point clouds, and waypoint information from the environment. A world model-based state latent variable encoding network generates the current state latent variables based on the multi-view RGB images, LiDAR point clouds, and waypoint information, accurately modeling the current traffic environment. A world model-based state transition variable encoding network predicts and simulates dynamic changes at the intersection based on the current state latent variables, generating dynamic latent variables during the state transition process and "imagining" the environment for the next few steps to determine the optimal left-turn timing. Combined with a policy network, reinforcement learning is performed based on the predicted potential environment dynamics and the current driving strategy to formulate a left-turn operation strategy.

[0124] In practice, an end-to-end autonomous driving system based on world model reinforcement learning first estimates traffic flow and predicts possible traffic conflict points, and then uses a policy network to perform reinforcement learning to control the vehicle to make a left turn at a safe time.

[0125] To ensure the stability of left-turn operations, an end-to-end autonomous driving system based on world model reinforcement learning introduces an initial imitation learning phase during training. This phase uses driving behavior data provided by experts to facilitate a smooth transition, thereby accelerating the convergence speed of the reinforcement learning model based on the actor-critic architecture and ensuring rapid adaptation to changing traffic conditions.

[0126] Application Example 2:

[0127] In scenarios where autonomous vehicles need to yield to vehicles with high right-of-way at intersections, such as emergency vehicles or fast-moving motor vehicles, the system must possess the ability to quickly assess and make decisions to ensure safe and efficient yielding.

[0128] This application presents an end-to-end autonomous driving system based on world model reinforcement learning. By combining world modeling and reinforcement learning, the system perceives and acquires surrounding traffic conditions in real time, including road layout and the trajectories of other vehicles. The system first uses a multimodal environmental data acquisition module to obtain multi-view RGB images, LiDAR point clouds, and pathpoint information from the environment. A world model-based state latent variable encoding network generates latent state variables for the current moment based on the multi-view RGB images, LiDAR point clouds, and pathpoint information, thus generating the vehicle's potential state features. A world model-based state transition variable encoding network generates dynamic latent variables during the state transition process based on the current state latent variables, simulating environmental changes over several future steps. Reinforcement learning is then performed using a policy network based on an actor-critic architecture. This world model-based reinforcement learning end-to-end autonomous driving system can predict and evaluate the dynamic behavior of vehicles with high right-of-way, and calculate the optimal avoidance path and timing.

[0129] In practical applications, end-to-end autonomous driving systems based on world model reinforcement learning can quickly identify and predict the approach of emergency vehicles and issue timely avoidance commands, adjusting their own speed and direction. Simultaneously, these systems also consider changes in the surrounding traffic environment, avoiding risks associated with excessive acceleration or sudden lane changes. Through continuous reinforcement learning optimization, these systems can make reasonable avoidance decisions in complex and dynamic traffic environments, ensuring driving safety.

[0130] Application Example 3:

[0131] In the two-way single-lane obstacle avoidance scenario, the autonomous vehicle is driving on a two-way single lane and encounters an obstacle ahead, especially on a narrow road. The vehicle needs to choose the right time to avoid the obstacle in order to avoid a collision. This scenario places high demands on the perception and decision-making capabilities of the autonomous driving system.

[0132] This application presents an end-to-end autonomous driving system based on world model reinforcement learning. The system acquires perceptual data of the surrounding environment through a multimodal environmental data acquisition module using LiDAR and cameras. This data includes multi-view RGB images of the environment, LiDAR point clouds, and pathpoint information. The system models the dynamic changes of traffic participants using a world model's state latent variable encoding network and employs dynamic latent variables from the state transition process to "imagine" and predict the dynamics of obstacles and other vehicles in the environment several steps ahead. Based on this, the system uses a policy network-based actor-critic reinforcement learning model to optimize the autonomous vehicle's detour strategy, calculating the safe detour path and optimal timing.

[0133] In practical operation, the end-to-end autonomous driving system based on world model reinforcement learning identifies the types of obstacles ahead, including their size, position, and speed, and predicts the optimal detour path based on these factors. Once the detour opportunity is determined, the system adjusts the vehicle's path using reinforcement-learned action policy instructions to ensure successful obstacle avoidance in confined spaces. By simulating multiple scenarios during the initial training phase, the system gradually improves its adaptability and decision-making efficiency in complex environments.

[0134] Figure 5 This is a flowchart illustrating an end-to-end autonomous driving control method based on world model reinforcement learning, according to an exemplary embodiment.

[0135] Reference Figure 5 As shown in the embodiment of this application, an end-to-end autonomous driving control method based on world model reinforcement learning includes S11 to S14, which can be applied to the end-to-end autonomous driving system based on world model reinforcement learning proposed in this application.

[0136] S11, acquire multimodal environment data.

[0137] Specifically, multimodal environment data includes multi-view images, LiDAR point clouds, and pathpoint information.

[0138] S12, input the multimodal environment data into the preset state latent variable encoding network to determine the state latent variables at the current moment.

[0139] S13, input the current state hidden variables and the previous autonomous driving action into the preset state transition variable encoding network to determine the dynamic hidden variables in the state transition process.

[0140] S14: Input the hidden variables of the current state and the hidden variables of the dynamics during the state transition process into the preset policy network to determine the autonomous driving action at the current moment.

[0141] Specifically, the policy network is initialized with pre-set expert data for imitation learning; the policy network includes a reinforcement learning model based on the actor-critic architecture. The policy network performs reinforcement learning through the reinforcement learning model based on the actor-critic architecture. The policy network smoothly transitions from imitation learning to reinforcement learning through KL divergence loss and soft update mechanism, and learns the optimal action policy for autonomous driving, that is, the autonomous driving action at the current moment.

[0142] The embodiments described above employ a state latent variable encoding network and a state transition variable encoding network based on a world model architecture to process high-dimensional multimodal environmental data. This effectively compresses high-dimensional sensor data, retains key environmental information, improves adaptability to complex dynamic traffic scenarios, and enables latent variable encoding of environmental states and state transition processes, supporting long-term driving decisions. The policy network is initialized with preset expert data for imitation learning, and then trained using KL divergence loss and a soft update mechanism to transition to reinforcement learning. By combining imitation learning and reinforcement learning, the policy network addresses the cold start problem of reinforcement learning, accelerates policy convergence, and improves the stability of training the reinforcement learning model based on the actor-critic architecture. Furthermore, by generating autonomous driving actions through imitation learning and reinforcement learning, the safety and stability of autonomous driving are improved.

[0143] The pre-defined state latent variable coding network of this application includes an image encoder, a point cloud encoder, a first multilayer perceptron, an asymmetric variational autoencoder, and a decoder.

[0144] In order to model the current environmental state, in some specific embodiments of this application, for S12, the multimodal environmental data is input into a preset state latent variable encoding network to determine the state latent variable at the current moment, including: S121 to S125.

[0145] S121, input the multi-view image into the image encoder for feature encoding to determine the image features.

[0146] S122, input the lidar point cloud into the point cloud encoder for point cloud encoding to determine the point cloud features.

[0147] S123, The path point information is input into the first multilayer perceptron for encoding to determine the path point features.

[0148] S124: Input the image features, point cloud features, and path point features into the asymmetric variational autoencoder for encoding to determine the state latent variables.

[0149] S125, input the state hidden variables into the decoder for semantic graph reconstruction to determine the bird's-eye view semantic graph.

[0150] The embodiments described above in this application employ a state latent variable encoding network, using an asymmetric variational autoencoder and Transformer architecture to compress high-dimensional multimodal environmental data into low-dimensional state latent variables, while retaining key environmental information.

[0151] The pre-defined state transition latent variable encoding network of this application includes a second multilayer perceptron, a Transformer module, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron.

[0152] In order to obtain the current environmental state change pattern, in some specific embodiments of this application, for S13, the current state hidden variable and the previous autonomous driving action are input into a preset state transition variable encoding network to determine the dynamic hidden variable in the state transition process, including: S131 to S135.

[0153] S131, input the current state latent variables and the previous autonomous driving action into the second multilayer sensor to determine the current latent variables.

[0154] S132, input the hidden variables at the current moment into the Transformer module to determine the dynamic hidden variables in the state transition process.

[0155] S133, input the hidden dynamic variables in the state transition process into the third multilayer perceptron to determine the task termination condition.

[0156] S134, input the latent dynamic variables in the state transition process into the fourth multilayer perceptron to determine the reward for the current state or action.

[0157] S135, input the latent dynamic variables in the state transition process into the fifth multilayer perceptron to determine the state distribution at the next moment.

[0158] The embodiments described above in this application employ a state transition variable encoding network to generate latent dynamic variables in the state transition process, thereby improving adaptability to complex traffic scenarios.

[0159] In some specific embodiments of this application, the preset policy network includes a reinforcement learning model based on an actor-critic architecture and a preset action mask module. The reinforcement learning model based on the actor-critic architecture includes an actor network and a critic network.

[0160] In order to obtain the optimal autonomous driving action strategy at the current moment, in some specific embodiments of this application, for S14, the state hidden variables at the current moment and the dynamic hidden variables in the state transition process are input into a preset policy network to determine the autonomous driving action at the current moment, including: S141 to S144.

[0161] In the application process, the preset policy network has been initialized using preset expert data.

[0162] S141, input the hidden variables of the current state and the hidden variables of the dynamics during the state transition process into the actor network to determine the autonomous driving action.

[0163] Specifically, autonomous driving actions include steering wheel angle and predicted driving distance.

[0164] S142, input the autonomous driving actions into the critic network to determine the policy value of the autonomous driving actions.

[0165] S143, update the parameters of the preset policy network according to the policy value of the autonomous driving action.

[0166] S144, input the autonomous driving action into the preset action mask module to filter invalid actions and determine the valid autonomous driving action at the current moment.

[0167] In the embodiments described above, the latent variables of the current state and the latent variables of the dynamics during the state transition process are input into a policy network that has been pre-initialized with expert data. Through reinforcement learning training based on an actor-critic architecture, the optimal action policy for autonomous driving is obtained. By combining imitation learning and reinforcement learning, the cold start problem in the early stage of reinforcement learning is solved, the training time of reinforcement learning is reduced, the training stability is improved, and the action policy degradation is avoided. Furthermore, an action mask module is used to filter invalid actions to ensure driving safety.

[0168] In order to jointly train a preset state latent variable encoding network, a preset state transition variable encoding network, and a preset policy network, some specific embodiments of this application include a method for training the preset state latent variable encoding network, the preset state transition variable encoding network, and the preset policy network, comprising S101 to S104.

[0169] S101, the bird's-eye view semantic map of a single time frame is used to train the preset state latent variable coding network, and the state latent variable coding network after the first training is determined.

[0170] S102, using a pre-set time series bird's-eye view semantic graph, the state latent variable encoding network trained for the first time and the pre-set state transition variable encoding network are jointly trained to determine the state latent variable encoding network trained for the second time and the state transition variable encoding network trained for the first time.

[0171] S103, freeze the parameters of the state latent variable encoding network after the second training and the state transition variable encoding network after the first training, initialize the preset policy network using preset expert data, and determine the initialized policy network.

[0172] S104. Unfreeze the parameters of the state latent variable encoding network after the second training and the state transition variable encoding network after the first training. Using KL divergence loss and soft update mechanism, jointly train the state latent variable encoding network after the second training, the state transition variable encoding network after the first training, and the initialized policy network based on the reinforcement learning paradigm to determine the state latent variable encoding network, the state transition variable encoding network, and the policy network that have been trained.

[0173] The embodiments described above in this application promote rapid convergence of the policy network, reduce reinforcement learning training time, and improve the stability of reinforcement learning training by jointly training a preset state latent variable encoding network, a preset state transition variable encoding network, and a preset policy network, combined with imitation learning and reinforcement learning.

[0174] The specific manner in which each step of the above control method is performed has been described in detail in the embodiments of the system, and will not be elaborated here.

[0175] The specific embodiments of this application have been described above. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. An end-to-end autonomous driving system based on world model reinforcement learning, characterized in that, include: A multimodal environment data acquisition module is used to acquire multimodal environment data, which includes multi-view images, lidar point clouds, and path point information. A state latent variable encoding network is used to generate the state latent variables at the current moment based on the multimodal environment data; A state transition variable encoding network is used to generate dynamic latent variables in the state transition process based on the current state latent variables and the autonomous driving action of the previous time step. A policy network is used to generate the autonomous driving action at the current moment based on the latent variables of the current state and the latent variables of the dynamics during the state transition process. The policy network includes a reinforcement learning model based on the actor-critic architecture. The policy network is initialized with preset expert data and trained using KL divergence loss and a soft update mechanism.

2. The end-to-end autonomous driving system based on world model reinforcement learning according to claim 1, characterized in that, The state latent variable encoding network includes an image encoder, a point cloud encoder, and a first multilayer perceptron. The image encoder is used to encode features of the multi-view image to determine image features. The point cloud encoder is used to encode the point cloud of the lidar to determine point cloud features. The first multilayer perceptron is used to encode the path point information to determine path point features.

3. The end-to-end autonomous driving system based on world model reinforcement learning according to claim 2, characterized in that, The state latent variable encoding network further includes an asymmetric variational autoencoder and a decoder. The asymmetric variational autoencoder is used to extract state latent variables. The input of the asymmetric variational autoencoder is the image features, the point cloud features, and the path point features. The output of the asymmetric variational autoencoder is the state latent variables. The decoder is used to reconstruct a semantic graph from the latent state variables. The input of the decoder is the latent state variables, and the output of the decoder is a bird's-eye view semantic graph.

4. The end-to-end autonomous driving system based on world model reinforcement learning according to claim 3, characterized in that, The state transition latent variable encoding network includes a second multilayer perceptron, a Transformer module, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron. The second multilayer perceptron is used to generate the latent variables at the current moment. The input of the second multilayer perceptron is the state latent variables and the autonomous driving action at the previous moment. The output of the second multilayer perceptron is the latent variables at the current moment. The Transformer module is used to generate the latent dynamic variables in the state transition process. The input of the Transformer module is the latent variable at the current moment and the latent variable at the historical moment. The output of the Transformer module is the latent dynamic variables in the state transition process. The third multilayer perceptron is used to generate the task termination condition. The input of the third multilayer perceptron is the dynamic hidden variable in the state transition process, and the output of the third multilayer perceptron is the task termination condition. The fourth multilayer perceptron is used to generate rewards. The input of the fourth multilayer perceptron is the dynamic latent variable in the state transition process, and the output of the fourth multilayer perceptron is the reward of the current state or action. The fifth multilayer perceptron is used to generate the state latent variable for the next time step. The input of the fifth multilayer perceptron is the dynamic latent variable in the state transition process, and the output of the fifth multilayer perceptron is the state latent variable for the next time step.

5. The end-to-end autonomous driving system based on world model reinforcement learning according to claim 1, characterized in that, The reinforcement learning model based on the actor-critic architecture includes an actor network and a critic network. The actor network is used to generate the autonomous driving action based on the latent variables of the current state and the latent dynamic variables during the state transition process. The critic network is used to generate policy value based on the autonomous driving action. The autonomous driving action includes steering wheel angle and predicted driving distance.

6. The end-to-end autonomous driving system based on world model reinforcement learning according to claim 5, characterized in that, The policy network also includes an action mask module, which is used to generate the dynamic feasibility of the autonomous driving action based on the LiDAR point cloud, and filter invalid actions in the autonomous driving action based on the dynamic feasibility of the autonomous driving action.

7. An end-to-end autonomous driving control method based on world model reinforcement learning, characterized in that, include: Acquire multimodal environment data, which includes multi-view images, lidar point clouds, and path point information; The multimodal environment data is input into a preset state latent variable encoding network to determine the state latent variables at the current moment; The current state hidden variables and the previous autonomous driving action are input into a preset state transition variable encoding network to determine the dynamic hidden variables in the state transition process; The current state latent variables and the dynamic latent variables during the state transition process are input into a preset policy network to determine the autonomous driving action at the current moment. The policy network includes a reinforcement learning model based on an actor-critic architecture.

8. The end-to-end autonomous driving control method based on world model reinforcement learning according to claim 7, characterized in that, The preset state latent variable coding network includes an image encoder, a point cloud encoder, a first multilayer perceptron, an asymmetric variational autoencoder, and a decoder. The preset state transition latent variable encoding network includes a second multilayer perceptron, a Transformer module, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron. The step of inputting the multimodal environment data into a preset state latent variable encoding network to determine the state latent variables at the current moment includes: The multi-view images are input into the image encoder for feature encoding to determine image features; The lidar point cloud is input into the point cloud encoder for point cloud encoding to determine the point cloud features; The path point information is input into the first multilayer perceptron for encoding to determine the path point features; The image features, point cloud features, and path point features are input into the asymmetric variational autoencoder for encoding to determine the state latent variables; The state hidden variables are input into the decoder to reconstruct the semantic graph and determine the bird's-eye view semantic graph. The step of inputting the current state latent variables and the previous autonomous driving action into a preset state transition variable encoding network to determine the dynamic latent variables in the state transition process includes: The current state hidden variable and the previous autonomous driving action are input into the second multilayer sensor to determine the hidden variable at the current moment; The latent variables at the current moment and the latent variables at historical moments are input into the Transformer module to determine the dynamic latent variables in the state transition process; The hidden dynamic variables in the state transition process are input into the third multilayer perceptron to determine the task termination condition; The latent dynamic variables in the state transition process are input into the fourth multilayer perceptron to determine the reward for the current state or action. The latent dynamic variables in the state transition process are input into the fifth multilayer perceptron to determine the latent state variables at the next moment.

9. The end-to-end autonomous driving control method based on world model reinforcement learning according to claim 7, characterized in that, The preset policy network includes the actor-critic architecture-based reinforcement learning model and the preset action mask module. The actor-critic architecture-based reinforcement learning model includes an actor network and a critic network. The step of inputting the current state latent variables and the dynamic latent variables during the state transition process into a preset policy network to determine the current autonomous driving action includes: The current state hidden variables and the dynamic hidden variables in the state transition process are input into the actor network to determine the autonomous driving action, which includes the steering wheel angle and the predicted driving distance. The autonomous driving action is input into the critic network to determine the policy value of the autonomous driving action; The parameters of the preset policy network are updated based on the policy value of the autonomous driving action; The autonomous driving action is input into a preset action mask module to filter out invalid actions and determine the valid autonomous driving action at the current moment.

10. The end-to-end autonomous driving control method based on world model reinforcement learning according to claim 7, characterized in that, A method for training the preset state latent variable encoding network, the preset state transition variable encoding network, and the preset policy network includes: The preset state latent variable coding network is trained using a bird's-eye view semantic map of a single time frame to determine the state latent variable coding network after the first training. The state latent variable encoding network and the state transition variable encoding network trained for the first time are jointly trained using a bird's-eye view semantic map of a preset time series, and the state latent variable encoding network trained for the second time and the state transition variable encoding network trained for the first time are determined. Freeze the parameters of the state latent variable encoding network trained for the second time and the state transition variable encoding network trained for the first time, initialize the preset policy network using preset expert data, and determine the initialized policy network. Unfreeze the parameters of the second-trained state latent variable encoding network and the first-trained state transition variable encoding network, and use KL divergence loss and soft update mechanism to jointly train the second-trained state latent variable encoding network, the first-trained state transition variable encoding network, and the initialized policy network based on the reinforcement learning paradigm to determine the completed state latent variable encoding network, the completed state transition variable encoding network, and the completed policy network.

Images