End-to-end automatic driving system and method

Abstract

The invention provides an end-to-end automatic driving system and method. The system comprises an environment sensing module, a feature extraction module, a data fusion module, a decision generation model module and a decision analysis module. The environment sensing module comprises a plurality of types of sensors, and each sensor collects environment information data of different modes. The feature extraction module extracts features of each modal environment information data. And the data fusion module fuses the features of the modal environment information data to obtain fused features. The decision generation model module determines a driving decision by using a trained decision generation model based on the fusion features. And the decision analysis module converts the driving decision output by the decision generation model module into a specific vehicle control instruction. The method has higher adaptability to complex and dynamic driving scenes, reduces the cost of data annotation during training, and improves the generalization ability of the model in various driving scenes.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving system and method. Background Art

[0002] With the development of artificial intelligence technology, intelligent driving systems have been widely studied and applied around the world. Traditional autonomous driving systems usually rely on multiple independent modules, such as environmental perception, path planning, motion control, etc. The information transmission and fusion between these modules are completed through predefined interfaces and protocols. However, this segmented modular architecture often leads to information loss and computational delays, which in turn affects the overall performance of the system.

[0003] In recent years, multimodal large model technology has made significant progress. These models can effectively fuse information from different sensors (such as cameras, lidar, radar, etc.) to a certain extent and make end-to-end decisions. This end-to-end approach based on large models can effectively solve the information transmission bottleneck in traditional methods and significantly improve the system's perception, decision-making and control capabilities. However, most of the current research on autonomous driving based on multimodal large models focuses on a single scenario or a specific task, lacking a complete end-to-end system to solve multi-task decision-making and control problems in complex road environments. Therefore, designing an end-to-end intelligent driving system based on multimodal large models has important research value and application prospects. Summary of the invention

[0004] The purpose of the present invention is to provide an end-to-end autonomous driving system and method to solve the above-mentioned problems, which can have stronger adaptability to complex and dynamic driving scenarios, reduce the cost of data annotation during training, and improve the generalization ability of the model in various driving scenarios.

[0005] The present invention proposes an end-to-end autonomous driving system, comprising:

[0006] The environment perception module includes multiple types of sensors, each of which collects environmental information data of different modes, and the environmental information includes static information and dynamic information;

[0007] A feature extraction module extracts the features of each modal environment information data;

[0008] The data fusion module fuses the features of each modal environmental information data to obtain fusion features;

[0009] A decision generation model module, based on the fusion features, uses a trained decision generation model to determine a driving decision, wherein the driving decision includes route planning;

[0010] The decision analysis module converts the driving decision output by the decision generation model module into a specific vehicle control instruction, wherein the vehicle control instruction includes at least one of the following items: the steering angle of the steering wheel, the accelerator pedal depth, and the braking force.

[0011] In one embodiment, the sensor includes a camera, a radar, a GPS, an inertial sensor;

[0012] The camera is used to collect environmental image data;

[0013] Radar is used to collect three-dimensional information of the environment and generate three-dimensional point cloud data;

[0014] GPS is used to collect the absolute position information of the vehicle;

[0015] Inertial sensors are used to collect dynamic state information of objects.

[0016] In one embodiment, the end-to-end autonomous driving system further includes a central hardware trigger, which sends a synchronization signal to each sensor to uniformly trigger the operation of each sensor.

[0017] In one embodiment, the end-to-end autonomous driving system further includes a time stamping module, a data buffering module, and a data alignment module;

[0018] The time marking module is connected to the environment perception module and is used to mark the environment information data of each modality with a timestamp;

[0019] The data buffer module is connected to the time stamp module and is used to store environmental information data with a timestamp;

[0020] The data alignment module is connected to the data buffer module and is used to search for and align the environmental information data of each modality in the data buffer module according to the timestamp.

[0021] In one embodiment, the feature extraction module extracts image features of environmental image data through a convolutional neural network, and extracts point cloud features of three-dimensional point cloud data through a PCL library algorithm.

[0022] In one embodiment, the data fusion module is a deep learning network model with shared weights, which fuses the features of each modal environmental information data based on pre-fusion technology.

[0023] In one embodiment, the decision generation model adopts a Transformer network structure that only includes a decoder, and uses a masked multi-head attention mechanism to understand the fusion features and make driving decisions.

[0024] In one embodiment, the end-to-end autonomous driving system further includes a feature conversion module, which is respectively connected to the data fusion module and the decision generation model module, and converts the fused features into BEV features through BEV Former.

[0025] The present invention also proposes an end-to-end autonomous driving method, which is applied to the end-to-end autonomous driving system as described above, and comprises the following steps:

[0026] Collecting environmental information data of different modes, wherein the environmental information includes static information and dynamic information;

[0027] Extract and fuse the environmental information data of each modality to obtain fusion features;

[0028] Using the trained decision generation model, determining a driving decision based on the fused features, the driving decision including route planning;

[0029] The driving decision is converted into a specific vehicle control instruction and the vehicle is controlled to travel according to the control instruction, wherein the vehicle control instruction includes at least one of the following items: the steering angle of the steering wheel, the depth of the accelerator pedal, and the force of the brake.

[0030] In one embodiment, the process of training the decision generation model includes:

[0031] Collect environmental information data of different modes;

[0032] Annotate real driving strategies based on environmental information data;

[0033] Inputting environmental information data into the decision generation model for training to obtain a predicted driving strategy;

[0034] The loss function is calculated based on the predicted driving strategy and the actual driving strategy, and the parameters of the decision generation model are iteratively updated through the loss function until the loss function is less than the preset threshold.

[0035] Compared with the prior art, the end-to-end autonomous driving system and method of the present invention have the following beneficial effects:

[0036] 1) The present invention adopts an end-to-end learning method, which can directly map from input (raw sensor data) to output (driving strategy) without the need for step-by-step processing in modules, so that the entire information flow flows within a model, and there is no problem of information loss or delay in the traditional modular architecture. All input information can be fully processed and integrated in a model to generate more accurate and consistent decisions, thereby better optimizing the performance of the entire system, reducing the complexity of interfaces and protocols between different modules, reducing error transmission and accumulation between modules, and reducing the R&D and maintenance costs of the system. System updates and optimizations become simpler, and system upgrades can be achieved by retraining or fine-tuning the model without the need for module-by-module debugging and verification.

[0037] 2) The present invention utilizes a large-scale pre-training model similar to GPT or BERT, performs pre-training through massive unsupervised data, and then performs fine-tuning through a small amount of supervised data, which can reduce the cost of data annotation during training and improve the generalization ability of the model in various driving scenarios.

[0038] 3) The present invention introduces the group query attention mechanism into the autonomous driving model, which improves the computing efficiency, enhances the parallelization capability, and enables the model to have better generalization ability. It can directly learn the global information and dynamic changes of the environment from multi-sensor data, so that the system has stronger adaptability to complex and dynamic driving scenarios. For example, it can more accurately predict the behavior of surrounding vehicles and pedestrians, thereby making more appropriate driving decisions, and can quickly respond to different driving scenarios, reducing the delay that may be caused by traditional modular systems when switching scenes.

[0039] 4) The fusion of multimodal data can provide redundant information and enhance the ability to understand the environment, which is very important for dealing with complex traffic scenarios (such as night driving, bad weather, crowded city streets, etc.). It helps to reduce the perception and decision-making errors of the autonomous driving system and improve the safety and reliability of autonomous driving.

[0040] 5) Through large-scale pre-training and multimodal fusion, the autonomous driving system of the present invention can better cope with unknown scenarios and environmental changes, has stronger adaptability and generalization capabilities, and reduces the problem of overfitting to a single scenario or environment.

[0041] 6) By real-time monitoring of vehicle status to adjust the control strategy, and returning feedback information to the decision-making model module for real-time optimization, a closed-loop control system is formed, which improves the response speed and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the structure of an end-to-end autonomous driving system according to an embodiment of the present invention;

[0043] Figure 2 An architecture diagram of a decision generation model in an end-to-end autonomous driving system according to an embodiment of the present invention;

[0044] Figure 3 This is an operational flow chart of an end-to-end autonomous driving system for road obstacle avoidance according to an embodiment of the present invention.

[0045] Reference numerals

[0046] 101. Environmental perception module, 102. Feature extraction module, 103. Data fusion module, 104. Decision generation model module, 105. Decision analysis module, 106. Feature conversion module. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments of the specification. It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0048] Secondly, "one embodiment" or "an embodiment" in this application refers to a specific feature, structure or characteristic that can be included in at least one implementation of the present invention. The "in one embodiment" and "an embodiment" that appear in different places in this specification do not all refer to the same embodiment, nor are they separate or selective embodiments that are mutually exclusive with other embodiments. The terms "including" and "comprising" indicate the presence of the claimed features, but do not exclude the presence of one or more other features. The term "and / or" used in this application includes any and all combinations of one or more of the relevant listed items.

[0049] The present invention proposes an end-to-end autonomous driving system, including an environment perception module 101, a feature extraction module 102, a data fusion module 103, a decision generation model module 104 and a decision analysis module 105, see Figure 1. The environmental perception module 101 includes multiple types of sensors, each of which collects environmental information data of different modes, and the environmental information includes static information and dynamic information. Static information refers to the position and size of environmental objects, and dynamic information refers to the motion state information of environmental objects. The environmental perception module 101 is the input layer of the automatic driving system, which is responsible for perceiving and analyzing the information of the vehicle's surrounding environment. It collects multi-modal environmental information data through multiple types of sensors, thereby achieving a comprehensive understanding of the environment. The feature extraction module 102 is used to extract the features of each modal environmental information data. The data fusion module 103 is used to fuse the features of each modal environmental information data to obtain fusion features. The fusion feature data specifically includes 2D and 3D target information, environmental semantic information, motion state information, etc., which provides input for the subsequent decision generation model module 104. The decision generation model module 104, based on the fusion features, uses the trained decision generation model to determine the driving decision, which includes route planning, etc. The decision generation model module 104 is the core of the intelligent driving system, which is responsible for integrating and analyzing various environmental information from the environmental perception module 101 to make global scene understanding and driving decisions. The decision parsing module 105 converts the driving decision output by the decision generation model module 104 into a specific vehicle control instruction, so that the vehicle can perform the driving task according to the predetermined path and decision requirements, and ensures that the vehicle can smoothly and safely execute these instructions. The vehicle control instruction includes at least one of the following items: the steering angle of the steering wheel, the accelerator pedal depth, and the braking force.

[0050] The decision analysis module 105 involves complex control algorithms and optimization techniques to achieve precise vehicle control. Specifically, a PID controller (Proportional-Integral-Derivative Controller) and / or model predictive control (MPC) is used for vehicle lateral control (Lateral Control), which is responsible for controlling and adjusting the direction of the vehicle to ensure that the vehicle travels along the planned path; a PID speed controller is used for vehicle longitudinal control (Longitudinal Control), which is responsible for controlling the acceleration and deceleration of the vehicle to ensure a safe distance from the vehicle in front or obstacles, etc., and is converted into specific values ​​of the throttle, brake and steering angles, and communicates with the vehicle hardware system (such as the electronic control unit ECU) at the same time to ensure the real-time and accuracy of the control instructions.

[0051] The PID controller accurately controls the direction of the vehicle by adjusting three key coefficients: the proportional coefficient Kp, the integral coefficient Ki and the differential coefficient Kd. The proportional coefficient Kp is adjusted according to the steering sensitivity of the vehicle and the road conditions to ensure that the vehicle can respond quickly and correct the deviation from the predetermined path. The value of Kp in one embodiment of the present invention is set between 0.5 and 1.0, and the specific value depends on the actual driving environment and vehicle performance. The integral coefficient Ki is used to eliminate static errors, that is, when the vehicle deviates from the predetermined path for a long time, the control amount is gradually adjusted through the integral action so that the vehicle gradually returns to the correct path. The value of Ki in one embodiment of the present invention is set to 0.01, which can ensure that the vehicle stably returns to the predetermined path while avoiding excessive overshoot or oscillation. The differential coefficient Kd is used to suppress the oscillation of the vehicle when turning, improve the driving stability, and effectively reduce the instability of the vehicle during the turning process by predicting the trend of the error change and adjusting it in advance. The value of Kd in one embodiment of the present invention is set to 0.01, which can avoid excessive sensitivity to noise while ensuring stability.

[0052] MPC optimizes the steering angle based on the vehicle's dynamic model and environmental constraints, and is particularly suitable for complex dynamic scenarios such as cornering and obstacle avoidance. The prediction time domain of an embodiment of the present invention is set to 2 seconds, and this value can be adjusted according to the vehicle's dynamic characteristics and road environment to ensure the foresight and accuracy of the control strategy; the control time domain is set to 1 second, that is, the execution time interval of each control command is 1 second, which helps to achieve a smooth transition of control commands and the continuity of vehicle driving.

[0053] The PID speed controller uses three core coefficients, namely the proportional coefficient Kp, the integral coefficient Ki and the differential coefficient Kd, to precisely control the acceleration and deceleration operations of the vehicle. The proportional coefficient Kp adjusts the acceleration of the vehicle in real time according to the error between the current speed and the target speed (or the required speed calculated based on the safe distance from the vehicle in front). The value of the proportional coefficient Kp of an embodiment of the present invention is usually between 0.5 and 1.0, and the specific value will be flexibly adjusted according to the acceleration performance of the vehicle and the current road conditions to ensure that the vehicle can respond quickly to speed changes and maintain stable driving. The integral coefficient Ki is used to eliminate long-term speed errors, perform integral operations on the speed error, and gradually adjust the vehicle speed so that it gradually approaches the target speed or the speed required for the safe distance. The integral coefficient Ki of an embodiment of the present invention is set to 0.01 to ensure that the vehicle can continuously and stably maintain a safe distance from the vehicle in front. The differential coefficient Kd effectively suppresses the oscillation of the vehicle speed and improves the stability of driving by predicting the changing trend of the speed error and adjusting it in advance. It can perform differential operation on the rate of change of the speed error, so as to quickly respond and adjust the acceleration, and reduce the occurrence of speed fluctuation and oscillation. The differential coefficient Kd of one embodiment of the present invention is set to 0.01.

[0054] In order to improve the safety and reliability of the automatic driving system, a fail-safe module can be added. For example, the fail-safe module can trigger an emergency response (such as emergency braking or switching to manual control) when the control fails or the sensor fails, thereby ensuring the safety of the system. A feedback module can also be added to monitor the vehicle status (such as speed, direction, position, etc.) in real time, adjust the control strategy, and return the feedback information to the decision generation model module 104 for real-time optimization to form a closed-loop control system.

[0055] The sensors of the environment perception module 101 include cameras, radars, GPS, inertial sensors (IMU), etc. The camera is used to collect environmental image data, such as traffic signs, signal lights, vehicles, pedestrians, road markings, etc. The image data provided by the camera is crucial for object detection, lane line recognition, traffic sign recognition, etc. One embodiment of the present invention has 6 cameras, and the camera parameters are: 1920x1080 pixels, 30 frames / second, field of view angle 120 degrees, and HDR image enhancement technology. The radar is used to collect three-dimensional information of the environment, including obstacles, road boundaries, road structures, etc., and generate three-dimensional point cloud data. The radar of one embodiment of the present invention is a laser radar with a scanning range of 150 meters, a resolution of 0.05, and a frequency of 20Hz. GPS is used to collect the absolute position information (longitude, latitude, altitude, etc.) of the vehicle, as well as the movement state (speed, direction, etc.) of the vehicle. GPS data is crucial for the positioning, navigation and path planning of autonomous driving vehicles. The GPS of one embodiment of the present invention is centimeter-level. Inertial sensors (IMU) are used to collect dynamic state information of objects, such as acceleration, angular velocity, pitch angle, yaw angle, etc. These data are crucial for vehicle posture estimation, motion control, and stability control. The measurement range of the inertial sensor of an embodiment of the present invention generally covers ±2g (acceleration) and ±300° / s (angular velocity). These sensors work together to provide the vehicle with environmental information of different modes, such as vision, depth, speed, distance, and orientation.

[0056] The end-to-end autonomous driving system of an embodiment of the present invention may also be provided with a central hardware trigger to send a synchronization signal to each sensor to uniformly trigger the operation of each sensor. The synchronization signal may be a level change, a pulse, or a specific time mark, etc. After receiving the synchronization signal, the sensor starts data collection, thereby ensuring that these sensors start collecting data at almost the same time. The central hardware trigger may be an external device or a control unit inside the vehicle.

[0057] An end-to-end autonomous driving system of an embodiment of the present invention also includes a time stamping module, a data buffering module and a data alignment module. The time stamping module is connected to the environment perception module 101, and is used to mark the environmental information data of each modality with a timestamp. The data buffering module is connected to the time stamping module, and is used to store environmental information data marked with a timestamp. The data alignment module is connected to the data buffering module, and is used to find and align the environmental information data of each modality in the data buffering module according to the timestamp. Through the software synchronization mechanism of the hardware trigger and the data alignment module, it is ensured that the environmental information data of each modality is aligned in space and time. In order to improve the robustness of the system, a time window (such as ±100 milliseconds) can also be defined, and the data within this window can be considered to be consistent in time to adapt to complex error situations.

[0058] The feature extraction module 102 of one embodiment of the present invention extracts the image features of the environmental image data through a convolutional neural network (CNN), and extracts the point cloud features of the three-dimensional point cloud data through a PCL library algorithm. Convolutional neural network (CNN) is a feedforward neural network with powerful image recognition capabilities. The features of the image are extracted through convolutional layers and pooling layers, and it performs well in image feature extraction and is widely used in image recognition, classification, detection and other fields. Of course, other deep learning models, such as VGG16, ResNet, Inception, etc., can also be used for image feature extraction tasks. The PCL (Point Cloud Library) library algorithm is a large cross-platform open source C++ programming library that implements a large number of general algorithms and efficient data structures related to point clouds. It is a library designed specifically for point cloud data processing, covering point cloud acquisition, filtering, segmentation, registration, retrieval, feature extraction, recognition, tracking, surface reconstruction and visualization. In addition to the PCL library algorithm, deep learning models such as VoxelNet, PointRCNN, SECOND, MV3D, 3DSSD, PV-RCNN, etc. can also be used for 3D target detection.

[0059] In order to improve the validity of the data, a data preprocessing module can be added before data feature extraction to perform denoising and other processing on the data. For example, the data preprocessing module can perform denoising, white balance, contrast enhancement and other operations on image data, and denoising, ground point removal, outlier filtering and other operations on lidar data.

[0060] The data fusion module 103 of one embodiment of the present invention is a deep learning network model with shared weights, which fuses the features of each modal environmental information data based on the pre-fusion technology. The deep learning network model with shared weights reduces the number of parameters, improves training efficiency, and enhances the generalization ability of the model by allowing multiple neurons or network layers to share the same set of weights. Specifically, it can be a Siamese neural network (Siamese Network), DenseNet (Densely Connected Convolutional Networks), etc.

[0061] Early Fusion technology is a fusion performed at the raw data level, directly merging the raw data of different sensors (such as point cloud data of LiDAR and image data of cameras). The present invention mainly realizes early fusion technology through feature fusion, that is, the features of different sensors are fused in the feature extraction stage to improve the accuracy and robustness of feature expression. Compared with mid-term fusion technology, early fusion technology can make full use of the advantages of each sensor to achieve data complementarity, and by integrating multi-sensor data, it can form more complete space occupancy data, thereby effectively reducing misjudgment caused by partial recognition, and has significant advantages in algorithm optimization and efficiency, application flexibility and scalability.

[0062] The decision generation model of an embodiment of the present invention adopts a Transformer network structure that only includes a decoder, and uses a masked multi-head attention mechanism to understand the fusion features and make driving decisions. Specifically, the architecture of the decision generation model is as follows Figure 2 As shown in the figure, it mainly includes the Embedding layer, Layer Norm layer, Attention layer, Feed Forward layer, Linear layer and Sigmoid layer. The Embedding layer is used to convert the user's input instructions or requests into vectors. The LayerNorm layer is placed before the Attention layer and the Feed Forward layer, and is mainly used to standardize the input of each layer to reduce the internal covariate shift problem and improve training efficiency. The Attention layer uses a masked multi-head attention mechanism to understand the fusion features and instruction vectors, and uses a group query attention mechanism to improve the training and reasoning speed. The number of layers (Nx) is 32, and the number of heads is 32. The Feed Forward layer is used to perform nonlinear transformation on the input data to capture higher-level features and information, and uses the Swish activation function. The Linear layer is used to perform linear transformation on the input data and map the output of the Feed Forward layer to the output space. The Sigmoid layer converts the output of the Linear layer into a probability distribution and outputs discrete driving decisions (such as whether to change lanes). The model embedding dimension is 4k, Cross Attention in the traditional Decoder is removed, and vLLM technology, tensor parallel technology, and KV Cache technology are used to accelerate reasoning. vLLM technology accelerates calculations through vectorization, tensor parallel technology improves throughput by processing data on multiple computing devices in parallel, and KV Cache technology reduces repeated calculations by caching intermediate results. The combination of these technologies can significantly improve the performance and efficiency of large models. When training this model, the optimizer uses AdamW.

[0063] The end-to-end autonomous driving system of an embodiment of the present invention is further provided with a feature conversion module 106, which is connected to the data fusion module 103 and the decision generation model module 104 respectively, and converts the fused features into BEV features through BEV Former. BEV Former is a neural network architecture for autonomous vehicle perception tasks, which achieves this goal by converting images captured by multiple cameras into feature representations of the bird's-eye view (BEV) perspective. This approach is particularly focused on processing 3D object detection and map segmentation tasks in autonomous driving. Some key features and components of BEVFormer are as follows: 1) BEV Queries: BEVFormer uses BEV queries in predefined grid shapes, which are learnable parameters for flexibly aggregating features between multiple camera views through an attention mechanism. The BEV grid of an embodiment of the present invention is divided into 200x200 grid cells and 200 BEV Queries. 2) Spatial Cross-Attention (SCA): It is a spatial cross-attention mechanism that allows BEV queries to extract information from multiple camera images and convert them into BEV features. 3) Temporal Self-Attention (TSA): Temporal self-attention mechanism, which uses historical BEV features to extract temporal information, helps to improve the speed estimation of moving objects and the detection of occluded objects. 4) Transformer architecture: BEV Former is built on the basis of Transformer, and uses its self-attention mechanism to aggregate features from different temporal and spatial dimensions. An embodiment of the present invention adopts R101-DCN backbone, AdamW is selected as the optimizer, and 2e-4 is selected as the learning rate. 5) Multi-task learning: BEV Former can simultaneously process 3D object detection and map segmentation tasks, and improve efficiency by sharing BEV features. 6) Temporal fusion: BEV Former can effectively process temporally continuous data, such as video streams, and improve the accuracy of perception through temporal fusion. Specifically, BEV Former can utilize temporal and spatial information through a BEV query mechanism (BEV queries) with pre-set parameters for grid division. The spatial cross-attention mechanism is used (that is, the BEV query mechanism extracts the required spatial features from the multi-camera features through the attention mechanism), so that the BEV query mechanism can extract features from the multi-camera perspectives of interest, thereby aggregating spatial information; in addition, the temporal self-attention mechanism is used to fuse historical information, thereby aggregating temporal information.

[0064] The end-to-end autonomous driving system of the present invention integrates data from multiple sensors such as cameras, lidar, GPS and IMU, uses deep learning and sensor fusion technology to achieve comprehensive perception of the vehicle's surrounding environment, and based on a deep learning architecture (Transformer), can perform global scene understanding and driving decisions, and convert these decisions into specific vehicle control instructions, such as throttle, brake and steering angle, to achieve safe driving. First, the decision generation model module 104 generates advanced driving strategies, which can improve the operating efficiency of the decision generation model and simplify the marking tasks of technicians during the training process. Then, it is parsed into specific control instructions to accurately control the vehicle, which can improve the safety, reliability and generalization ability of autonomous driving, reduce R&D and maintenance costs, and adapt to complex and dynamic driving environments.

[0065] The following is a detailed description of the above system using the entire process of using the above autonomous driving system from detecting road obstacles to controlling the vehicle to avoid them. Figure 3 shown.

[0066] 1. Collect multimodal environmental information data: First, use hardware triggers to send out synchronization information so that the various sensors of the environmental perception module 101 start working almost at the same time, and collect data on the vehicle's surrounding environment through sensors such as cameras, lidar, GPS, and IMU.

[0067] 2. Data timestamp calibration: Based on the timestamp, use the data alignment module to ensure that all sensor data (i.e., environmental information data of each modality) are synchronized in time.

[0068] 3. Multimodal data fusion: Deep learning and traditional sensor fusion algorithms fuse data from different sensors, extract environmental features, and perform feature fusion.

[0069] 4. Global scene understanding and obstacle detection: Use the fused data to perform global scene understanding and identify and locate road obstacles through deep learning models.

[0070] 5. Generate driving decisions: Based on the fused feature information, the decision generation model makes obstacle avoidance decisions and determines obstacle avoidance strategies and path planning.

[0071] 6. Convert into specific vehicle control instructions: According to the obstacle avoidance path planning, calculate the required steering angle, use PID controller or model predictive control (MPC) to adjust the vehicle direction, and perform lateral control (Lateral Control) of the vehicle; calculate the required throttle depth and braking force, control the acceleration and deceleration of the vehicle to adapt to the obstacle avoidance path, maintain a safe distance, and perform longitudinal control (Longitudinal Control) of the vehicle.

[0072] 7. During system operation, the fail-safe module works to ensure that an emergency response can be triggered when control fails or sensor fails; the feedback module works to adjust the control strategy according to the vehicle status and returns feedback information to the decision generation model module 104 for real-time optimization.

[0073] The present invention also proposes an end-to-end autonomous driving method, which is applied to the end-to-end autonomous driving system as described above, and comprises the following steps:

[0074] Collect environmental information data of different modes, including static information and dynamic information;

[0075] Extract and fuse the environmental information data of each modality to obtain fusion features;

[0076] Use the trained decision generation model to determine driving decisions based on fusion features, including route planning;

[0077] The driving decision is converted into a specific vehicle control instruction and the vehicle is controlled to travel according to the control instruction. The vehicle control instruction includes at least one of the following items: the steering angle of the steering wheel, the depth of the accelerator pedal, and the force of the brake.

[0078] The process of training a decision generation model includes:

[0079] Collect environmental information data of different modes;

[0080] Annotate real driving strategies based on environmental information data;

[0081] Input the environmental information data into the decision generation model for training to obtain the predicted driving strategy;

[0082] The loss function is calculated based on the predicted driving strategy and the actual driving strategy, and the parameters of the decision generation model are iteratively updated through the loss function until the loss function is less than the preset threshold.

[0083] Based on this training method, only the final driving strategy can be marked without marking more complex intermediate quantities, which greatly reduces the workload of technicians and improves R&D efficiency.

[0084] It should be noted that, in the present application, unless otherwise clearly specified and limited, "connection" and similar terms should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, an indirect connection through an intermediate medium, or the internal connection of two components. Technical personnel in the field can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0085] The present invention has the following beneficial effects:

[0086] 1) The present invention adopts an end-to-end learning method, which can directly map from input (raw sensor data) to output (driving strategy) without the need for step-by-step processing in modules, so that the entire information flow flows within a model, and there is no problem of information loss or delay in the traditional modular architecture. All input information can be fully processed and integrated in a model to generate more accurate and consistent decisions, thereby better optimizing the performance of the entire system, reducing the complexity of interfaces and protocols between different modules, reducing error transmission and accumulation between modules, and reducing the R&D and maintenance costs of the system. System updates and optimizations become simpler, and system upgrades can be achieved by retraining or fine-tuning the model without the need for module-by-module debugging and verification.

[0087] 2) The present invention utilizes a large-scale pre-training model similar to GPT or BERT, performs pre-training through massive unsupervised data, and then performs fine-tuning through a small amount of supervised data, which can reduce the cost of data annotation during training and improve the generalization ability of the model in various driving scenarios.

[0088] 3) The present invention introduces the group query attention mechanism into the autonomous driving model, which improves the computing efficiency, enhances the parallelization capability, and enables the model to have better generalization ability. It can directly learn the global information and dynamic changes of the environment from multi-sensor data, so that the system has stronger adaptability to complex and dynamic driving scenarios. For example, it can more accurately predict the behavior of surrounding vehicles and pedestrians, thereby making more appropriate driving decisions, and can quickly respond to different driving scenarios, reducing the delay that may be caused by traditional modular systems when switching scenes.

[0089] 4) The fusion of multimodal data can provide redundant information and enhance the ability to understand the environment, which is very important for dealing with complex traffic scenarios (such as night driving, bad weather, crowded city streets, etc.). It helps to reduce the perception and decision-making errors of the autonomous driving system and improve the safety and reliability of autonomous driving.

[0090] 5) Through large-scale pre-training and multimodal fusion, the autonomous driving system of the present invention can better cope with unknown scenarios and environmental changes, has stronger adaptability and generalization capabilities, and reduces the problem of overfitting to a single scenario or environment.

[0091] 6) By real-time monitoring of vehicle status to adjust the control strategy, and returning feedback information to the decision-making model module for real-time optimization, a closed-loop control system is formed, which improves the response speed and adaptability of the system.

[0092] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0093] The construction and arrangement of the present application shown in a plurality of different exemplary embodiments are only exemplary. Although only a few embodiments are described in detail in this disclosure, it should be readily understood by those who refer to this disclosure that many modifications are possible (e.g., size, structure, shape and proportion, as well as parameter values, mounting arrangements, use of materials, color, directional changes, etc.) without substantially departing from the novel teachings and advantages of the subject matter described in the application. For example, the element shown as integrally formed can be composed of multiple parts or elements, the position of the element can be inverted or otherwise changed, and the nature or number or position of the discrete element can be changed or changed. Therefore, all such modifications are intended to be included in the scope of the present invention. The order or sequence of any process or method steps can be changed or reordered according to alternative embodiments. In the claims, any "device plus function" clause is intended to cover the structure described herein that performs the function, and is not only structurally equivalent but also equivalent structure. Without departing from the spirit and essence of the present invention, a person skilled in the art can make various corresponding modifications and changes according to the present invention, but these corresponding modifications and changes should all fall within the scope of protection of the present invention.

Claims

1. An end-to-end autonomous driving system, characterized in that: include: The environment perception module includes multiple types of sensors, each of which collects environmental information data of different modes, and the environmental information includes static information and dynamic information; A feature extraction module extracts the features of each modal environment information data; The data fusion module fuses the features of each modal environmental information data to obtain fusion features; A decision generation model module, based on the fusion features, uses a trained decision generation model to determine a driving decision, wherein the driving decision includes route planning; The decision analysis module converts the driving decision output by the decision generation model module into a specific vehicle control instruction, wherein the vehicle control instruction includes at least one of the following items: the steering angle of the steering wheel, the accelerator pedal depth, and the braking force.

2. The end-to-end autonomous driving system according to claim 1, characterized in that: The sensors include cameras, radars, GPS, and inertial sensors; The camera is used to collect environmental image data; Radar is used to collect three-dimensional information of the environment and generate three-dimensional point cloud data; GPS is used to collect the absolute position information of the vehicle; Inertial sensors are used to collect dynamic state information of objects.

3. The end-to-end autonomous driving system according to claim 1, characterized in that: It also includes a central hardware trigger, which sends a synchronization signal to each sensor to uniformly trigger the operation of each sensor.

4. The end-to-end autonomous driving system according to claim 1, characterized in that: It also includes a time stamp module, a data buffer module and a data alignment module; The time marking module is connected to the environment perception module and is used to mark the environment information data of each modality with a timestamp; The data buffer module is connected to the time stamp module and is used to store environmental information data with a timestamp; The data alignment module is connected to the data buffer module and is used to search for and align the environmental information data of each modality in the data buffer module according to the timestamp.

5. The end-to-end autonomous driving system according to claim 2, characterized in that: The feature extraction module extracts image features of environmental image data through a convolutional neural network, and extracts point cloud features of three-dimensional point cloud data through a PCL library algorithm.

6. The end-to-end autonomous driving system according to claim 1, characterized in that: The data fusion module is a deep learning network model with shared weights, which fuses the features of each modal environmental information data based on the pre-fusion technology.

7. The end-to-end autonomous driving system according to claim 1, characterized in that: The decision generation model adopts a Transformer network structure that only includes a decoder, and uses a masked multi-head attention mechanism to understand the fusion features and make driving decisions.

8. The end-to-end autonomous driving system according to claim 1, characterized in that: It also includes a feature conversion module, which is connected to the data fusion module and the decision generation model module respectively, and converts the fusion features into BEV features through BEV Former.

9. An end-to-end autonomous driving method, characterized in that: The method is applied to the end-to-end autonomous driving system according to any one of claims 1 to 8, comprising the following steps: Collecting environmental information data of different modes, wherein the environmental information includes static information and dynamic information; Extract and fuse the environmental information data of each modality to obtain fusion features; Using the trained decision generation model, determining a driving decision based on the fused features, the driving decision including route planning; The driving decision is converted into a specific vehicle control instruction and the vehicle is controlled to travel according to the control instruction, wherein the vehicle control instruction includes at least one of the following items: the steering angle of the steering wheel, the depth of the accelerator pedal, and the force of the brake.

10. The end-to-end autonomous driving method according to claim 9, characterized in that: The process of training a decision generation model includes: Collect environmental information data of different modes; Annotate real driving strategies based on environmental information data; Inputting environmental information data into the decision generation model for training to obtain a predicted driving strategy; The loss function is calculated based on the predicted driving strategy and the actual driving strategy, and the parameters of the decision generation model are iteratively updated through the loss function until the loss function is less than the preset threshold.

Images