A world-model-driven, learning-based, transferable autonomous driving method and system
By constructing a simulation environment and training a generative world model, and utilizing multi-sensor image fusion and recurrent neural networks, the problems of poor transferability and safety of autonomous driving models in new environments were solved, and safe transfer from the virtual world to the real world was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2023-05-25
- Publication Date
- 2026-05-26
AI Technical Summary
Existing autonomous driving models have poor transferability when facing new environments and tasks, and cannot effectively adapt to the development of general artificial intelligence. Furthermore, a single camera cannot capture the full picture of the environment, which affects the safety of autonomous driving.
A simulation environment is constructed, data is collected and a generative world model is trained, and multi-sensor image fusion and recurrent neural networks are used to generate bird's-eye view and control commands to realize the migration from the virtual world to the real world.
It enables the virtual-to-real migration of autonomous driving models, improves the accuracy and safety of environmental perception, simplifies training steps, and increases training efficiency.
Smart Images
Figure CN116501065B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, and in particular to a world model-driven, learning-based, transferable autonomous driving method and system. Background Technology
[0002] Today, artificial intelligence is transitioning from specialized AI to general AI. Generative large models, represented by ChatGPT, have already made a name for themselves in the field of natural language processing and have become the mainstream general AI model for natural language processing. Autonomous driving is a manifestation of the cross-integration of the automotive industry with new-generation information technologies such as artificial intelligence, automated control, and big data in the transportation sector. High-level autonomous driving systems need to cope with almost all complex traffic environments and complete driving tasks safely and efficiently.
[0003] However, most current autonomous driving models use a modular approach, which requires a lot of manual engineering, involving manual annotation of individual modules and configuration between modules. For new environments and new tasks, the algorithms need to be redesigned and upgraded manually, resulting in poor transferability and inability to adapt to the development and requirements of general artificial intelligence.
[0004] With the continuous upgrading of in-vehicle hardware systems and cloud computing capabilities, it has become possible to apply generative large models in real time. In the future, the gradual upgrading of hardware systems will inevitably break through the hardware bottleneck of general artificial intelligence.
[0005] In their paper "Model-Based Imitation Learning for Urban Driving," Anthony Hu et al. proposed a method to complete integrated autonomous driving tasks by using world model modeling. However, using only a single camera cannot obtain the full picture of the environment, thus failing to guarantee the safety of autonomous driving. Furthermore, this method does not consider how to train the autonomous driving model from the virtual world to the real world for application. Summary of the Invention
[0006] The purpose of this invention is to provide a world model-driven learning-transferable autonomous driving method and system, which uses a generative large-scale world model to solve the integrated autonomous driving problem and realize the virtual-real transfer of autonomous driving.
[0007] The objective of this invention can be achieved through the following technical solutions:
[0008] A world-model-driven, learning-based, transferable autonomous driving method includes the following steps:
[0009] Build a simulation environment and set the same autonomous vehicle configuration as in the real environment within the simulation environment;
[0010] Large-scale data is collected in a simulation environment, and small-scale data is collected in a real environment. The data includes bird's-eye view and camera images taken by cameras in the front, rear, left, and right directions on the autonomous vehicle. The bird's-eye view is a predefined world thumbnail. Traffic vehicles and pedestrians in the bird's-eye view are marked with rectangles, and red traffic lights are marked with lines in the cut-off area.
[0011] A world model is constructed and trained in a simulation environment using a large amount of data. The world model takes camera images as input, uses a generative world model to model the environment, saves and transmits information from historical moments through hidden variables, and outputs a bird's-eye view and control commands. The control commands are used to control the autonomous vehicle.
[0012] Using the trained world model, domain-adaptive transfer training is performed in a real environment based on small batches of data.
[0013] The model trained through adaptive transfer can be deployed in real-world autonomous vehicles to achieve universal, integrated autonomous driving with virtual-real transfer capabilities.
[0014] The configuration of the autonomous vehicle includes vehicle parameters and sensor parameters.
[0015] The world model of history and the present moment is represented as follows:
[0016]
[0017] Among them, o k Given the image input at time k, x k Image features are obtained by multi-sensor image fusion and encoding of image input; assuming posterior features s k and prior features z k All conform to a normal distribution, and the posterior feature s k Through hidden features h containing historical moment information k The action a in the previous moment k-1 and image features x k Sample generation, prior features z k By hiding feature h k and the action a from the previous moment k-1 Sample generation, action a K-1 The lateral and longitudinal accelerations at time k-1; the hidden variable h at the next time step. k+1 Encoding is performed using a recurrent neural network; a k b represents the action at the current moment. k Both represent the bird's-eye view at the current moment and are based on the hidden feature h. k and posterior features s k generate.
[0018] The world model for future moments cannot obtain image input; it relies on imagination to obtain future actions and bird's-eye view trends. Specifically, the world model does not generate posterior features at the future k+T time step, but directly uses the hidden feature h. k+T and prior features z k+T Generate action a k+T and bird's-eye view b k+T And the hidden feature h in the next moment k+T+1 Based on hidden feature h k+T and prior features z k+T Generate, represented as:
[0019]
[0020] The specific steps of training the world model in a simulation environment using large amounts of data are as follows:
[0021] Time t k To t k+T-1 The data is used as historical moment data, and the time t k+T To t k+T+F The data is used as future time data, t k To t k+T+F All data is input into the world model for training, maximizing the joint probability of action sequences and bird's-eye view sequences, i.e., p(a k:k+T+F ,b k:k+T+F The maximum is obtained by variational inference, which yields the lower bound of the joint probability, i.e.:
[0022]
[0023] Among them, D KLF Let E represent the relative entropy of the two distributions, and E represent the expectation.
[0024] A world-model-driven, learning-transferable autonomous driving system, comprising:
[0025] The environment configuration module is used to build a simulation environment and set the same autonomous vehicle configuration as the real environment in the simulation environment;
[0026] The data acquisition module is used to collect large amounts of data in a simulation environment and small amounts of data in a real environment. The data includes bird's-eye view and camera images taken by cameras in the front, rear, left, and right directions on the autonomous vehicle. The bird's-eye view is a predefined world thumbnail. Traffic vehicles and pedestrians in the bird's-eye view are marked with rectangles, and red traffic lights are marked with lines in the cut-off area.
[0027] The world model construction and training module is used to construct a world model and train the world model in a simulation environment using a large amount of data. The world model takes camera images as input, uses a generative world model to model the environment, saves and transmits information from historical moments through hidden variables, and outputs a bird's-eye view and control commands. The control commands are used to control the autonomous vehicle.
[0028] The model transfer module is used to perform domain-adaptive transfer training in a real environment based on a small batch of data, utilizing the trained world model.
[0029] The model deployment module is used to deploy adaptively transferred models in real-world autonomous vehicles, enabling universal and integrated autonomous driving through virtual-real migration.
[0030] The configuration of the autonomous vehicle includes vehicle parameters and sensor parameters.
[0031] In the world model construction and training module, the world models for the past and the current moment are represented as follows:
[0032]
[0033] Among them, o k Given the image input at time k, x k Image features are obtained by multi-sensor image fusion and encoding of image input; assuming posterior features s k and prior features z k All conform to a normal distribution, and the posterior feature s k Through hidden features h containing historical moment information k The action a in the previous moment k-1 and image features x k Sample generation, prior features z k By hiding feature h k and the action a from the previous moment k-1 Sample generation, action a k-1 The lateral and longitudinal accelerations at time k-1; the hidden variable h at the next time step. k+1 Encoding is performed using a recurrent neural network; a k b represents the action at the current moment. k Both represent the bird's-eye view at the current moment and are based on the hidden feature h. k and posterior features s k generate.
[0034] In the world model construction and training module, the world model at future time moments cannot obtain image input; instead, it relies on imagination to obtain future actions and the bird's-eye view trend. Specifically, the world model does not generate posterior features at the future k+T time moment, but directly uses the hidden feature h.k+T and prior features z k+T Generate action a k+T and bird's-eye view b k+T And the hidden feature h in the next moment k+T+1 Based on hidden feature h k+T and prior features z k+T Generate, represented as:
[0035]
[0036] In the world model construction and training module, the specific method for training the world model in a simulation environment using large amounts of data is as follows:
[0037] Time t k To t k+T-1 The data is used as historical moment data, and the time t k+T To t k+T+F The data is used as future time data, t k To t k+T+F All data is input into the world model for training, maximizing the joint probability of action sequences and bird's-eye view sequences, i.e., p(a k:k+T+F ,b k:k+T+F The maximum is obtained by variational inference, which yields the lower bound of the joint probability, i.e.:
[0038]
[0039] Among them, D KL Let E represent the relative entropy of the two distributions, and E represent the expectation.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] (1) This invention proposes a virtual-real migration method for applying world models to autonomous driving, which enables autonomous driving to be trained in the virtual world and applied in the real world.
[0042] (2) The generative world modeling method proposed in this invention uses images from four cameras (front, back, left, and right) as input, which enables the world model to know the overall world environment. The sensor fusion method is used to fuse and encode the images obtained from the four cameras to generate posterior features. The generated posterior features take into account the overall world environment and the feature description is more accurate.
[0043] (3) In the world model of the present invention, the generation of hidden variables is carried out by recurrent neural networks to generate hidden variables for the next time step, while giving more consideration to the features of the previous time step.
[0044] (4) The world model of the present invention only outputs bird’s-eye view and action, without generating the current image input, which reduces the network structure, simplifies the training steps, and focuses the loss function training more on bird’s-eye view generation, action generation and prior and posterior matching, thereby improving training efficiency. Attached Figure Description
[0045] Figure 1 This is a flowchart of the method of the present invention.
[0046] Figure 2 This is a schematic diagram of the world model architecture of the present invention. Detailed Implementation
[0047] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0048] Example 1
[0049] This embodiment provides a world model-driven, learning-based, transferable autonomous driving method, such as... Figure 1 As shown, it includes the following steps:
[0050] S1: Build a simulation environment and set up the same autonomous vehicle configuration as in the real environment.
[0051] In this embodiment, the configuration of the autonomous vehicle includes vehicle parameters and sensor parameters.
[0052] S2: Collect large amounts of data in a simulation environment and small amounts of data in a real environment.
[0053] In this embodiment, the collected data includes bird's-eye view images and camera images taken by cameras in the front, rear, left, and right directions on the autonomous vehicle. The bird's-eye view image is as follows: Figure 2 As shown, this is a predefined world thumbnail. In the bird's-eye view, vehicles, pedestrians, etc. are marked with rectangles, and red traffic lights are marked with lines in the cut-off area.
[0054] S3: Build a world model and train it in a simulation environment using a large amount of data.
[0055] like Figure 2 As shown, in this embodiment, the world model uses camera images as input, utilizes a generative world model to model the environment, saves and transmits information from historical moments through hidden variables, outputs a bird's-eye view to improve the interpretability of the model, and outputs control commands to control the autonomous vehicle.
[0056] For historical and current world models, such as Figure 2As shown, the image at time k is input to o. k Image features x are obtained by multi-sensor image fusion and encoding. k This process can be represented as x k =f e (o k Assume posterior features s k and prior features z k All conform to a normal distribution, and are obtained through hidden features h that contain historical time information. k The action a in the previous moment k-1 and image features x k Sample generation of posterior features s k It can be represented as q(s) k )~N(μ θ (h k ,a k-1 ,x k ),σ θ (h k ,a k-1 ,x k )), by hiding feature h k and the action a from the previous moment k-1 Sample to generate prior features z k , can be represented as The hidden variables for the next time step are encoded by a recurrent neural network and can be represented as h. k+1 =f v (h k ,s k Finally, by hiding the feature h... k and posterior features s k Generate action a at the current moment k and bird's-eye view b k , can be represented as a k =f π1 (h k ,s k ), b k =f π2 (h k ,s k That is, the world model of history and the present moment can be represented as:
[0057]
[0058] Among them, action a k-1 This represents the lateral and longitudinal accelerations at time k-1, which are the control commands used to control autonomous vehicles.
[0059] The world model for future moments cannot obtain image input; it relies on imagination to determine future actions and aerial view trends. Specifically, the world model does not generate posterior features at future time k+T; instead, it directly uses the hidden features h. k+T and prior features z k+T Generate action a k+T and bird's-eye view b k+T And the hidden feature h in the next moment k+T+1 Based on hidden feature h k+T and prior features z k+T Generate, h k+T+1 =f φ (h k+T ,z k+T That is, the world model for future moments is represented as:
[0060]
[0061] After the model is built, the world model is trained in a simulation environment using a large amount of data, specifically as follows:
[0062] Time t k To t k+T-1 The data is used as historical moment data, and the time t k+T To t k+T+F The data is used as future time data, t k To t k+T+F All data is input into the world model for training, maximizing the joint probability of action sequences and bird's-eye view sequences, i.e., p(a k:k+T+F ,b k:k+T+F The maximum is obtained by variational inference, which yields the lower bound of the joint probability, i.e.:
[0063]
[0064] Among them, D KL Let E represent the relative entropy of the two distributions, and E represent the expectation.
[0065] S4: Utilize the trained world model to perform domain-adaptive transfer training in a real environment based on small batches of data.
[0066] Since both the simulation dataset (large batch data) and the real dataset (small batch data) use the same bird's-eye view format, and both the simulation and real environments use the same vehicle configuration, transfer training is equivalent to parameter fine-tuning after pre-training a large model. Transfer training methods are common knowledge in this field, and will not be elaborated upon here to avoid obscuring the purpose of this application.
[0067] S5: Deploy the adaptively transferred training model in real-world autonomous vehicles to achieve universal integrated autonomous driving with virtual-real migration.
[0068] Example 2
[0069] This embodiment provides a world model-driven, learning-based, transferable autonomous driving system for implementing the method described in Embodiment 1, including:
[0070] (1) Environment configuration module, used to build a simulation environment and set the same autonomous vehicle configuration as the real environment in the simulation environment.
[0071] (2) Data acquisition module, used to acquire large amounts of data in the simulation environment and small amounts of data in the real environment.
[0072] (3) World model construction and training module, which is used to construct the world model and train the world model in the simulation environment using a large amount of data. The world model takes camera images as input, uses generative world model to model the environment, saves and transmits information of historical moments through hidden variables, and outputs bird's-eye view and control commands. The control commands are used to control the autonomous vehicle.
[0073] (4) Model transfer module, which is used to perform domain adaptive transfer training in the real environment based on small batches of data using the trained world model.
[0074] (5) Model deployment module, used to deploy the adaptive transfer training model in real-world autonomous vehicles to achieve universal integrated autonomous driving with virtual-real transfer.
[0075] The specific implementation process of the system is shown in Example 1, and will not be repeated here.
[0076] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A world-model-driven, learning-based, transferable autonomous driving method, characterized in that, Includes the following steps: Build a simulation environment and set the same autonomous vehicle configuration as in the real environment within the simulation environment; Large-scale data is collected in a simulation environment, and small-scale data is collected in a real environment. The data includes bird's-eye view and camera images taken by cameras in the front, rear, left, and right directions on the autonomous vehicle. The bird's-eye view is a predefined world thumbnail. Traffic vehicles and pedestrians in the bird's-eye view are marked with rectangles, and red traffic lights are marked with lines in the cut-off area. A world model is constructed and trained in a simulation environment using a large amount of data. The world model takes camera images as input, uses a generative world model to model the environment, saves and transmits information from historical moments through hidden variables, and outputs a bird's-eye view and control commands. The control commands are used to control the autonomous vehicle. Using the trained world model, domain-adaptive transfer training is performed in a real environment based on small batches of data. The model trained by adaptive transfer can be deployed in real-world autonomous vehicles to achieve universal and integrated autonomous driving with virtual-real transfer capabilities. The world model for both the past and the present moment is represented as follows: in, for k Image input at any time, Image features are obtained by multi-sensor image fusion and encoding of image input; posterior features are assumed. and prior features All conform to a normal distribution, posterior features Hidden features containing historical moment information The actions of the previous moment and image features Sampling generation, prior features By hiding features and the action of the previous moment Sample generation, action express k Lateral and longitudinal accelerations at time -1; hidden variables at the next time step. Encoding is performed using a recurrent neural network; Indicates the action at the current moment. Both the current bird's-eye view and the current view are based on hidden features. and posterior features generate.
2. The world model-driven, learning-based, transferable autonomous driving method according to claim 1, characterized in that, The configuration of the autonomous vehicle includes vehicle parameters and sensor parameters.
3. The world model-driven, learning-based, transferable autonomous driving method according to claim 1, characterized in that, The world model of the future cannot obtain image input; it relies on imagination to obtain future actions and bird's-eye view trends. Specifically, the world model in the future... No posterior features are generated at any time; hidden features are used directly. and prior features Generate Actions and bird's-eye view And the hidden features in the next moment. Based on hidden features and prior features Generate, represented as: 。 4. The world model-driven, learning-based, transferable autonomous driving method according to claim 3, characterized in that, The specific steps of training the world model in a simulation environment using large amounts of data are as follows: Time to The data is used as historical moment data, and the moment... to The data will be used as future time data. to All data is input into the world model for training, aiming to maximize the joint probability of action sequences and bird's-eye view sequences. The maximum is obtained by variational inference, which yields the lower bound of the joint probability, namely: in, Represents the relative entropy of two distributions. E It expresses expectation.
5. A world-model-driven, learning-based, transferable autonomous driving system, characterized in that, include: The environment configuration module is used to build a simulation environment and set the same autonomous vehicle configuration as the real environment in the simulation environment; The data acquisition module is used to collect large amounts of data in a simulation environment and small amounts of data in a real environment. The data includes bird's-eye view and camera images taken by cameras in the front, rear, left, and right directions on the autonomous vehicle. The bird's-eye view is a predefined world thumbnail. Traffic vehicles and pedestrians in the bird's-eye view are marked with rectangles, and red traffic lights are marked with lines in the cut-off area. The world model construction and training module is used to construct a world model and train the world model in a simulation environment using a large amount of data. The world model takes camera images as input, uses a generative world model to model the environment, saves and transmits information from historical moments through hidden variables, and outputs a bird's-eye view and control commands. The control commands are used to control the autonomous vehicle. The model transfer module is used to perform domain-adaptive transfer training in a real environment based on a small batch of data, utilizing the trained world model. The model deployment module is used to deploy adaptively transferred training models in real-world autonomous vehicles, achieving universal and integrated autonomous driving with virtual-real migration. In the world model construction and training module, the world model for the past and the current moment is represented as follows: in, for k Image input at any time, Image features are obtained by multi-sensor image fusion and encoding of image input; posterior features are assumed. and prior features All conform to a normal distribution, posterior features Hidden features containing historical moment information The actions of the previous moment and image features Sampling generation, prior features By hiding features and the action of the previous moment Sample generation, action express k Lateral and longitudinal accelerations at time -1; hidden variables at the next time step. Encoding is performed using a recurrent neural network; Indicates the action at the current moment. Both the current bird's-eye view and the current view are based on hidden features. and posterior features generate.
6. The world model-driven, learning-based, transferable autonomous driving system according to claim 5, characterized in that, The configuration of the autonomous vehicle includes vehicle parameters and sensor parameters.
7. A world-model-driven, learning-based, transferable autonomous driving system according to claim 5, characterized in that, In the world model construction and training module, the future world model cannot obtain image input; instead, it relies on imagination to obtain future actions and bird's-eye view trends. Specifically, the world model in the future... No posterior features are generated at any time; hidden features are used directly. and prior features Generate Actions and bird's-eye view And the hidden features in the next moment. Based on hidden features and prior features Generate, represented as: 。 8. A world-model-driven, learning-based, transferable autonomous driving system according to claim 7, characterized in that, In the world model construction and training module, the specific method for training the world model in a simulation environment using large amounts of data is as follows: Time to The data is used as historical moment data, and the moment... to The data will be used as future time data. to All data is input into the world model for training, aiming to maximize the joint probability of action sequences and bird's-eye view sequences. The maximum is obtained by variational inference, which yields the lower bound of the joint probability, namely: in, Represents the relative entropy of two distributions. E It expresses expectation.