Intelligent driving end-to-end decision model iteration method and device, equipment and medium

Abstract

The invention discloses an intelligent driving end-to-end decision model iteration method and device, equipment and a medium, and relates to the technical field of intelligent driving, and the method comprises the steps: collecting multi-sensor scene state data, carrying out the preprocessing, and generating a high-dimensional feature vector; mapping the high-dimensional feature vector from an original space to a bird's-eye view angle space through a BEV fusion model, and performing feature interaction in the bird's-eye view angle space by using an attention mechanism to generate high-dimensional environment features; performing dynamic modeling on the high-dimensional environment features by using a graph neural network and multi-task learning, and outputting an environment perception graph; and designing a reward function including safety, driving efficiency and navigation matching degree by using a Markov decision process based on the environmental perception map, so as to optimize the constructed decision model based on a PPO algorithm, and continuously adjusting through multiple iterations until the decision model converges. According to the invention, through combination of multi-sensor data fusion and deep learning, efficient and steady intelligent driving decision in a complex environment is realized.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, specifically to an intelligent driving end-to-end decision model iteration method, device, equipment, and medium. Background Technology

[0002] Autonomous driving technology has received widespread attention in recent years, especially in intelligent perception, path planning, and decision-making and control. Currently, autonomous driving systems rely on various sensors, such as vision sensors, LiDAR, and radar, to perceive the environment. However, each of these sensors has its own advantages and disadvantages. A single sensor often struggles to provide complete and accurate information in complex environments and cannot maintain efficient and stable performance in dynamically changing and complex conditions. For example, cameras perform poorly in low light and inclement weather conditions, while radar and LiDAR, although capable of stable operation in adverse conditions, have relatively weak capabilities in rendering object details.

[0003] To address these shortcomings, autonomous driving systems typically employ multi-sensor data fusion, based on rule-based or classical algorithms, and utilize cross-view cognitive fusion technology. By integrating information from different perspectives of various sensors, the environmental perception capabilities of autonomous driving systems can be significantly improved. With the continuous development of multi-sensor data fusion technology, graph neural networks, and reinforcement learning, how to organically combine these technologies to enhance the perception, localization, and decision-making capabilities of autonomous driving systems has become a core challenge in autonomous driving technology.

[0004] In related technologies, end-to-end learning methods are increasingly being applied to improve the intelligence and efficiency of systems. End-to-end methods learn driving strategies directly from raw sensor data using a unified deep neural network, eliminating the need for manual feature extraction and independent model design.

[0005] However, the aforementioned end-to-end learning methods are inefficient in processing and fusing information from different sensors and perspectives, cannot effectively overcome the differences between multi-view data, and cannot achieve high-precision perception and decision-making while ensuring computational efficiency. They still exhibit certain limitations when facing complex scenarios. Summary of the Invention

[0006] This application provides an end-to-end decision model iteration method, device, equipment, and medium for intelligent driving, which can solve the technical problems of insufficient perception accuracy and slow decision-making speed faced by existing intelligent driving systems in complex environments.

[0007] Firstly, this application provides an iterative method for an end-to-end decision-making model for intelligent driving, the method comprising: Collect scene state data from multiple sensors and preprocess it to generate high-dimensional feature vectors; The BEV fusion model maps the high-dimensional feature vectors from the original space to the bird's-eye view space, and uses an attention mechanism to perform feature interaction in the bird's-eye view space to generate high-dimensional environmental features. The high-dimensional environmental features mentioned above are dynamically modeled using graph neural networks and multi-task learning, and an environmental perception map is output. Based on the aforementioned environmental perception map, a Markov decision process is used to design a reward function that includes safety, driving efficiency, and navigation matching degree. The constructed decision model is then optimized using the PPO algorithm and adjusted through multiple rounds of iteration until the decision model converges.

[0008] In conjunction with the first aspect, in one implementation, graph neural networks and multi-task learning are used to dynamically model the aforementioned high-dimensional environmental features, outputting an environmental awareness map, specifically including: Using the aforementioned high-dimensional environmental features as input, a graph neural network is constructed; the nodes of the graph neural network represent the vehicle and surrounding targets, and the edges represent the relationships between targets. By combining prior information about human driving cognition with the optimized feature learning of the graph neural network, a scene perception map is output.

[0009] In conjunction with the first aspect, in one implementation, a Markov decision process is used based on the aforementioned environmental perception map, specifically including: The driving decision-making process is described using a Markov decision process. A state set and an action set are constructed based on an environmental perception map, and a state transition function is designed.

[0010] In conjunction with the first aspect, in one implementation, the above design includes reward functions for safety, driving efficiency, and navigation matching accuracy, specifically including: Safety rewards are determined based on the distance between the vehicle and surrounding obstacles; The driving efficiency reward is determined based on the difference between the current vehicle speed and the target vehicle speed, as well as the acceleration. The navigation matching reward is determined based on the deviation between the vehicle and the target path; Based on the aforementioned safety rewards, driving efficiency rewards, and navigation matching rewards, a reward function is designed.

[0011] In conjunction with the first aspect, in one implementation, the above reward function is:

[0012] in, The first weighting coefficient, This is the second weighting coefficient. The third weighting coefficient, As a security reward, Rewards for driving efficiency Rewards are given for navigation matching accuracy.

[0013] In conjunction with the first aspect, in one implementation method, the constructed decision model is optimized based on the PPO algorithm, specifically including: Training data is obtained through trajectory sampling, and the advantage function is calculated; The policy network is optimized using a pruning objective function based on the advantage function, and the value network is optimized by minimizing the value loss function.

[0014] In conjunction with the first aspect, in one implementation method, multi-sensor scene state data is collected and preprocessed to generate a high-dimensional feature vector, specifically including: Collect scene state data from multiple sensors, timestamp the scene state data from each sensor, and align them using an extended Kalman filter algorithm to obtain multimodal data; Feature extraction and normalization are performed on the above multimodal data to generate high-dimensional feature vectors.

[0015] Secondly, this application provides an intelligent driving end-to-end decision model iteration device, the device comprising: The data processing module is used to collect scene state data from multiple sensors and preprocess it to generate high-dimensional feature vectors. The fusion extraction module is used to map the above high-dimensional feature vectors from the original space to the bird's-eye view space through the BEV fusion model, and use the attention mechanism to perform feature interaction in the bird's-eye view space to generate high-dimensional environmental features. The perception modeling module is used to dynamically model the above-mentioned high-dimensional environmental features using graph neural networks and multi-task learning, and outputs an environmental perception map. The decision optimization module is used to design a reward function that includes safety, driving efficiency and navigation matching degree based on the above environmental perception map using Markov decision process, so as to optimize the constructed decision model based on PPO algorithm, and adjust it through multiple rounds of iteration until the decision model converges.

[0016] Thirdly, this application provides an intelligent driving end-to-end decision model iteration device, which includes a processor, a memory, and an intelligent driving end-to-end decision model iteration program stored in the memory and executable by the processor. When the intelligent driving end-to-end decision model iteration program is executed by the processor, it implements the steps of the intelligent driving end-to-end decision model iteration method.

[0017] Fourthly, this application provides a computer-readable storage medium storing an intelligent driving end-to-end decision model iteration program, wherein when the intelligent driving end-to-end decision model iteration program is executed by a processor, it implements the steps of the intelligent driving end-to-end decision model iteration method described above.

[0018] The beneficial effects of the technical solutions provided in this application include: By collecting and preprocessing multi-sensor scene state data, high-dimensional feature vectors are generated. These high-dimensional feature vectors are then mapped from the original space to a bird's-eye view space using a BEV fusion model. An attention mechanism is then used to interact with these features within the bird's-eye view space, generating high-dimensional environmental features. Graph neural networks and multi-task learning are used to dynamically model these high-dimensional environmental features, outputting an environmental perception map. Based on this environmental perception map, a Markov decision process is used to design a reward function that incorporates safety, driving efficiency, and navigation matching. The constructed decision model is then optimized using the PPO algorithm, and continuously adjusted through multiple iterations until the decision model converges. Therefore, by combining multi-sensor data fusion with deep learning, efficient and robust intelligent driving decision-making in complex environments is achieved, solving the technical problems of insufficient perception accuracy and slow decision-making speed faced by intelligent driving systems in complex environments. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating an embodiment of the intelligent driving end-to-end decision model iteration method of this application; Figure 2 This is a flowchart illustrating another embodiment of the intelligent driving end-to-end decision model iterative method of this application; Figure 3 This is the core flowchart of the BEVFormer model in the embodiments of this application; Figure 4 This is a flowchart illustrating the workflow of the PPO algorithm in the embodiments of this application; Figure 5 This is a schematic diagram of the functional modules of an embodiment of the intelligent driving end-to-end decision model iteration device of this application; Figure 6 This is a schematic diagram of the hardware structure of the intelligent driving end-to-end decision model iteration device involved in the embodiments of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0021] In a first aspect, embodiments of this application provide an iterative method for an intelligent driving end-to-end decision model.

[0022] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the intelligent driving end-to-end decision model iteration method of this application. The aforementioned intelligent driving end-to-end decision model iteration method includes: S1. Collect scene state data from multiple sensors and preprocess it to generate high-dimensional feature vectors; S2. The high-dimensional feature vectors mentioned above are mapped from the original space to the bird's-eye view space through the BEV fusion model. The attention mechanism is used to perform feature interaction in the bird's-eye view space to generate high-dimensional environmental features. S3. Use graph neural networks and multi-task learning to dynamically model the above high-dimensional environmental features and output an environmental perception map; S4. Based on the above environmental perception map, a Markov decision process is used to design a reward function that includes safety, driving efficiency and navigation matching degree. The constructed decision model is optimized based on the PPO algorithm and adjusted through multiple rounds of iteration until the decision model converges.

[0023] In this embodiment, high-dimensional feature vectors are generated by collecting and preprocessing multi-sensor scene state data. These high-dimensional feature vectors are then mapped from the original space to a bird's-eye view space using a BEV fusion model. An attention mechanism is then used to perform feature interactions within the bird's-eye view space to generate high-dimensional environmental features. Graph neural networks and multi-task learning are used to dynamically model these high-dimensional environmental features, outputting an environmental perception map. Based on this environmental perception map, a Markov decision process is used to design a reward function that incorporates safety, driving efficiency, and navigation matching. This reward function is then optimized using the PPO algorithm, and the model is continuously adjusted through multiple iterations until it converges. Therefore, by combining multi-sensor data fusion with deep learning, efficient and robust intelligent driving decision-making in complex environments is achieved, solving the technical problems of insufficient perception accuracy and slow decision-making speed faced by intelligent driving systems in complex environments.

[0024] Based on the above embodiments, in this embodiment, step S1 involves collecting multi-sensor scene state data and preprocessing it to generate a high-dimensional feature vector, specifically including: First, scene state data from multiple sensors is collected, the scene state data from each sensor is timestamped, and then aligned using an extended Kalman filter algorithm to obtain multimodal data. Then, feature extraction and normalization are performed on the above multimodal data to generate high-dimensional feature vectors.

[0025] like Figure 2 As shown, specifically, in step S1 above, environmental information is collected and preprocessed to extract high-dimensional feature vectors related to driving decisions, specifically including: Step S101: Using multiple sensors such as cameras, LiDAR, millimeter-wave radar, and ultrasonic sensors, real-time data on the surrounding scene is collected as a multi-dimensional data source. Specifically, the camera acquires visual information about the road ahead, LiDAR provides precise point cloud data, millimeter-wave radar acquires distance and velocity information of objects, and ultrasonic sensors capture information about nearby obstacles. Through the combined acquisition of data from multiple sensors, the spatial layout and dynamic changes of the environment are comprehensively captured.

[0026] Step S102: Timestamp the scene state data from different sensors, and align the data using the Extended Kalman Filter (EKF) algorithm to ensure consistency of data from different sensors at the same timestamp. Accurately align the acquired data and generate time-synchronized multimodal data.

[0027] Step S103: Perform feature extraction and normalization on the multimodal data. For example, normalize physical quantities such as velocity and acceleration to a uniform scale. Extract environmental features using a convolutional neural network (CNN) to generate high-quality input features, i.e., high-dimensional feature vectors, for use by downstream modules. This high-dimensional feature vector is the basic feature of the original data after preprocessing.

[0028] In this embodiment, by fusing data from multiple sensors, such as cameras, LiDAR, millimeter-wave radar, and ultrasonic sensors, the perception capability of the intelligent driving system in complex environments is improved. Especially under conditions such as severe weather and low light, cross-view cognitive fusion technology effectively eliminates the limitations of a single sensor, making environmental perception more comprehensive and stable. Through real-time perception and prediction, the system can identify potential hazards and generate targeted adjustment strategies.

[0029] Furthermore, in one embodiment, in step S2 above, multi-level feature extraction and time-series fusion representation are performed on the perceived high-dimensional feature vector through the backbone network and the BEVFormer model, i.e., the BEV fusion model, to generate high-dimensional environmental features, specifically including: Step S201: Multi-sensor data input and initial mapping. The preprocessed high-dimensional feature vectors (including images, point clouds, etc., which exist in their respective original data spaces) are input into the backbone neural network for multi-layer feature extraction, generating corresponding feature maps which are input into the BEVFormer model. Through basic projection or query mechanisms, they are initially mapped to a unified BEV space, generating geometrically aligned initial BEV data.

[0030] Step S202: Feature Interaction and Spatial Modeling within the BEV Space. Based on the geometrically aligned initial BEV data output from S201, the self-attention mechanism of the Transformer is used to identify the correlation between different sensors. Cross-sensor and cross-region feature interactions are performed within the BEV space. Information from different sensors is weighted to enhance the understanding of the scene's spatial structure. Combined with the temporal fusion module of the BEVFormer model, perception data from different time steps are combined to capture the motion information and dynamic changes of objects, constructing a richer and more robust unified environmental representation, i.e., high-dimensional intermediate environmental features.

[0031] Step S203: Encode and compress the aforementioned intermediate environmental features to generate deep, high-dimensional environmental features. These deep, high-dimensional environmental features integrate the processing results of steps S201 and S202. Specifically, through the decoder layer in BEVFormer, the multi-sensor intermediate environmental features, after spatial alignment, temporal fusion, and correlation weighting, are encoded into a unified and dense feature tensor, generating multimodal fusion deep, high-dimensional environmental features. These features contain dynamic trajectory prediction information and unified BEV perspective semantic information, providing subsequent graph neural networks with decision-level inputs that possess temporal awareness and spatial consistency.

[0032] In this embodiment, by combining the BEVFormer model with the backbone neural network, and through multi-level feature extraction and temporal fusion, the perception accuracy and system robustness are improved, ensuring that the decision model can provide more accurate and real-time environmental data, and helping the intelligent agent to make accurate responses in dynamic and complex driving scenarios.

[0033] Furthermore, in one embodiment, step S3 above, which uses graph neural networks and multi-task learning to dynamically model the high-dimensional environmental features and output an environmental perception map, specifically includes: First, a graph neural network is constructed using the aforementioned high-dimensional environmental features as input; the nodes of the graph neural network represent the vehicle and surrounding targets, and the edges represent the relationships between targets. Then, the feature learning of the graph neural network is optimized by combining prior information on human driving cognition, and a scene perception map is output.

[0034] Specifically, in step S3 above, a knowledge network-driven human-like cognitive autonomous driving intelligent architecture is constructed using graph neural networks and multi-task learning methods, including: Step S301: Using the above-mentioned high-dimensional environmental features as input, and combining graph neural networks and multi-task learning methods, dynamically model the environmental information such as surrounding vehicles, pedestrians, and road conditions.

[0035] Step S302: Combine prior information from human driving data to optimize the feature learning of the graph neural network, making the model more in line with human driving preferences.

[0036] Step S303: Output the scene perception map as the state input for subsequent reinforcement learning.

[0037] Furthermore, in one embodiment, step S4 above, which uses a Markov decision process based on the aforementioned environmental perception map, specifically includes: The driving decision-making process is described using a Markov decision process. A state set and an action set are constructed based on an environmental perception map, and a state transition function is designed.

[0038] Furthermore, in one embodiment, in step S4 above, the design includes reward functions for safety, driving efficiency, and navigation matching accuracy, specifically including: Safety rewards are determined based on the distance between the vehicle and surrounding obstacles; The driving efficiency reward is determined based on the difference between the current vehicle speed and the target vehicle speed, as well as the acceleration. The navigation matching reward is determined based on the deviation between the vehicle and the target path; Based on the aforementioned safety rewards, driving efficiency rewards, and navigation matching rewards, a reward function is designed.

[0039] In this embodiment, a multi-layered, weighted fusion reward function is designed to comprehensively consider multiple key factors such as the safety, driving efficiency, and comfort of autonomous vehicles, ensuring that the agent makes more balanced decisions in different driving scenarios. Unlike a single reward function, the multi-objective reward function in this embodiment can simultaneously optimize the vehicle's driving path, acceleration, and braking strategies, enabling the agent to achieve the best balance between safety, driving efficiency, and comfort in various real-world driving scenarios, thus improving the system's performance in complex situations.

[0040] Furthermore, in this embodiment, the reward function is:

[0041] in, The first weighting coefficient, This is the second weighting coefficient. The third weighting coefficient, As a security reward, Rewards for driving efficiency Rewards are given for navigation matching accuracy.

[0042] Furthermore, in one embodiment, step S4 above, which optimizes the constructed decision model based on the PPO algorithm, specifically includes: Training data is obtained through trajectory sampling, and the advantage function is calculated; The policy network is optimized using a pruning objective function based on the advantage function, and the value network is optimized by minimizing the value loss function.

[0043] Specifically, in step S4 above, the PPO reinforcement learning strategy is used to optimize the training of the end-to-end decision model. Combined with Monte Carlo reward sampling, path planning and vehicle control decisions are progressively optimized. The model's decision performance is iteratively optimized through multiple rounds using an efficient iterative method, enabling it to possess high adaptability and decision-making efficiency in changing environments. This includes: Step S401: Use a Markov Decision Process (MDP) to describe the agent's decision-making process, construct a state set and an action set, and design a state transition function.

[0044] Step S402: Design a suitable reward function for the safety, driving efficiency and navigation matching of autonomous driving.

[0045] Step S403: Train the PPO algorithm by using a trajectory sampling method to obtain training data to ensure that the data conforms to the distribution of the current policy.

[0046] Step S404: Optimize the policy network using the PPO algorithm to improve the agent's decision-making ability, and simultaneously optimize the value network based on the optimization of the policy network.

[0047] Step S405: Through multiple iterations, continuously optimize the strategy until the decision model converges, thereby improving the agent's decision-making ability in complex environments.

[0048] In this embodiment, an efficient iterative learning method is used to enable the agent to continuously optimize the decision-making model during operation. Each round of training and iteration enables the agent to gradually optimize decisions under different driving conditions, improve the stability and adaptability of the system in complex environments, and ensure that the system maintains efficient and balanced decision-making capabilities during long-term operation. Combined with PPO strategy optimization, the system can automatically adjust decision-making strategies according to different scenarios and driving conditions.

[0049] This embodiment of the intelligent driving end-to-end decision model iterative method collects data about the surrounding environment through multiple sensors (such as cameras, LiDAR, millimeter-wave radar, etc.), preprocesses this data, extracts key features, and forms high-quality input feature vectors. Next, based on a reinforcement learning framework, a cross-view cognitive fusion model is designed and optimized for safety and efficiency. Finally, through an efficient training process, the autonomous driving system achieves adaptive decision-making capabilities in complex environments. This efficient end-to-end iterative approach to intelligent driving based on cross-view cognitive fusion not only solves key problems in existing autonomous driving technologies such as multi-sensor fusion, decision efficiency, and system safety, but also provides a new solution for the efficient development, training, and commercial application of autonomous driving systems.

[0050] Specifically, the iterative method for the aforementioned intelligent driving end-to-end decision-making model includes: Step A1: Use multiple sensors such as vehicle-mounted cameras, LiDAR, millimeter-wave radar, GPS, and IMU to collect scene state data in real time, generate global observation state, and preprocess the data to extract key features related to driving decisions, forming a high-quality input feature vector to provide basic support for model training. This includes the following steps: Step A101: Sensor data acquisition. Acquired data includes vehicle status: speed, acceleration, position, and heading angle; navigation information: such as current position deviation and target direction deviation; and surrounding environment information: such as the speed and relative position of surrounding vehicles, and the distribution of obstacles. Based on the acquired raw data, a global observation status is formed. , represented as:

[0051] in, This is a global observation state vector, used to represent the vehicle's state, navigation deviation, and surrounding environment information. For the vehicle's speed, For acceleration, For vehicle location, The vehicle's steering angle. This represents the current position deviation of the target path. For the directional deviation of the target path, For the speed of surrounding vehicles, The relative positions of surrounding vehicles This describes the distribution of obstacles.

[0052] Step A102: Preprocess the raw data to ensure the consistency and high quality of the input data. This includes noise filtering: using the Kalman filter algorithm to remove noise from the sensor data, especially filtering continuous data such as vehicle speed and acceleration; data normalization: normalizing features such as vehicle speed, position, and acceleration to a range between [-1, 1] to ensure that all features are on the same order of magnitude; and coordinate transformation: converting the position information from the global coordinate system to the vehicle's local coordinate system to facilitate subsequent calculations and model training.

[0053] Step A103: Extract key features from the preprocessed data to obtain vehicle features (speed, acceleration, and direction) and environmental features (speed, relative position of surrounding vehicles, and obstacle distribution), forming a high-dimensional feature vector.

[0054] Step A2: Refer to the core process of the BEVFormer model Figure 3As shown, the BEVFormer model is used for data processing based on a bird's-eye view. This efficiently fuses multi-source sensor data, processes the high-dimensional feature vectors obtained from the multi-sensor data, and improves the accuracy of environmental perception. Specifically, the steps include: Step A201: Multi-sensor data input: Input the high-dimensional feature vector into the backbone neural network for multi-layer feature extraction, generate the corresponding feature map, input it into the BEVFormer model, and after preprocessing, map it into a unified BEV space to generate initial BEV data.

[0055] Step A202: Spatial Modeling: Initial BEV data (point cloud, images, etc.) is processed using a Transformer structure. A self-attention mechanism is employed to identify correlations between different sensors, weighting the information from each sensor to enhance the understanding of spatial relationships. The self-attention mechanism helps identify correlations between different sensors, thus enabling information fusion. Combined with a temporal fusion module, sensor data from different time steps are combined to capture object motion information and dynamic changes, obtaining high-dimensional intermediate environment features.

[0056] Step A203: Generate multimodal fusion high-dimensional environmental features based on the high-dimensional intermediate environmental features, i.e., generate BEV features, as input to the driving decision module.

[0057] Step A3: Using the high-dimensional environmental features from the multimodal fusion output of the BEVFormer model in Step A2 via a deep neural network as input to the GNN, high-precision, real-time environmental perception information can be obtained through multi-dimensional fusion. The specific steps are as follows: Step A301: Using the aforementioned high-dimensional environmental features as input data, construct a Graph Neural Network (GNN). Nodes represent the vehicle and surrounding targets, and edges represent the relationships between targets (such as speed difference, distance, etc.). Each node includes position, velocity, acceleration, and dynamic information such as the target's direction of motion and predicted trajectory. Edge features include distance difference, velocity difference, and the target's motion state relative to the vehicle, thus completing dynamic scene modeling. Through the GNN's dynamic modeling of interactive relationships in the environment, a message passing mechanism captures the mutual influence between targets, generating a graph embedding containing scene structure information. This graph embedding, together with the original attributes and relationship features of the nodes, constitutes the final scene perception map.

[0058] Step A302: Combining prior information about human driving cognition, such as maintaining a reasonable following distance, avoiding sudden braking, and avoiding dangerous overtaking, guide the graph neural network to focus on safe areas, optimize its feature learning mechanism, and make the network output more in line with human driving preferences, avoiding frequent acceleration, braking, and sharp turns.

[0059] Step A303: The final GNN output is a scene perception map containing the graph structure of the vehicle and surrounding targets. It includes dynamic information such as the position, velocity, and acceleration of each node, as well as the relationships between nodes, and integrates multimodal features from different sensors as state inputs for subsequent reinforcement learning.

[0060] Step A4: Construct a reinforcement learning training model based on cross-view cognitive fusion. Utilize multi-sensor information fusion technology and combine it with the PPO reinforcement learning algorithm to design the training model, ensuring that the agent can autonomously make safe and efficient decisions in complex driving scenarios. Refer to the PPO algorithm workflow diagram. Figure 4 The specific implementation includes the following steps: Step A401: Use a Markov Decision Process (MDP) to describe the agent's decision-making process. The main components of the MDP include: a state space S (state set) defined as: a vectorized representation of the multimodal fusion high-dimensional environmental features output by the BEVFormer and the scene perception map generated by the GNN; and an action set A: defining the actions the autonomous vehicle can take based on the current state, such as throttle control commands. and steering wheel control commands State transition function T: represents the probability that the system transitions to the next state after taking a certain action in a specific state. Action space. Defined as vehicle control commands, including accelerator, steering wheel angle, etc., the range of motion can be expressed as: ,in

[0061] Step A402: Design a suitable reward function for the safety, driving efficiency, and navigation matching accuracy of autonomous driving. Safety reward: Based on the distance design between the vehicle and surrounding obstacles, ensure that the agent maintains a safe following distance; Efficiency reward: Encourage the vehicle to reach higher speeds and accelerate smoothly; Navigation matching reward: Evaluate the consistency between the current decision and the target path based on the degree of deviation between the vehicle and the target path.

[0062] The overall reward function R is expressed as:

[0063] in, The first weighting coefficient, This is the second weighting coefficient. The third weighting coefficient, For the safety reward function based on obstacle distance, A driving efficiency reward function that encourages higher speeds and smoother acceleration. A navigation matching reward function that evaluates the deviation between the vehicle and the target path.

[0064] Step S403: Train the PPO algorithm by acquiring training data using a trajectory sampling method to ensure the data conforms to the distribution of the current policy. First, sample multiple trajectory segments from the current policy; each segment contains the state. ,action ,award and the next state Where t is the time step index within a single trajectory, and the reward is... The reward function is calculated using the reward function in step A402; then, the reward function is associated with the model optimization objective, utilizing the current state of the value network at each time step. Value estimation and the state at the next moment Value estimation Combined with rewards Calculate the time series difference error at time t. :

[0065] in, Discount factor; Secondly, the advantage function is calculated using data from the trajectory. Used to guide policy updates:

[0066] in, The smoothing parameter is used to balance the bias and variance; T represents the termination time of the current trajectory (task sequence); k is the index used for summation, indicating the number of steps to move forward from the current time t. Let be the time series difference error at time t+k.

[0067] Finally, calculate the cumulative discounted return for each trajectory time step. As the target value of the corresponding state sample in value network optimization :

[0068] By trimming the objective function Update strategy network parameters Using the same trajectory data, minimize the loss function of the value network. To improve the accuracy of state value prediction, specifically including: First, the PPO algorithm is used to optimize the policy network, improving the agent's decision-making ability. The PPO algorithm maximizes cumulative reward by adjusting the parameters of the policy network and uses a pruning objective function to limit the variation between old and new policies, ensuring stability during training. Specifically, the goal of the PPO algorithm is to achieve the following pruning objective function... Update the strategy:

[0069] in, This represents the expected experience at time step t. This is the current strategy. It's an old strategy. It is the dominant function. It is a small hyperparameter used in the pruning operation to control the difference between the old and new strategies.

[0070] Then, based on the optimization of the policy network, the value network is also optimized simultaneously. The goal of the value network is to minimize the following loss function to improve the accuracy of state value estimation:

[0071] in, Let be the loss function for the value network, and let i be the index of a sample in the training batch of the value network. The value of the state estimated for the value network. denoted as the actual return value calculated by the return function, and N is the total number of state samples participating in this round of value network optimization.

[0072] Step A404: Through multiple iterations, continuously optimize the policy to improve the agent's decision-making ability in complex environments. In each iteration, repeat the sampling and updating process to continuously optimize the policy and value network until the model converges.

[0073] At this point, driving actions can be generated in real time through the policy network based on the trained decision model and the current state.

[0074] The method in this embodiment achieves multimodal data fusion through the BEVFormer model, combines scene modeling using graph neural networks and policy optimization using PPO reinforcement learning. This cross-view data fusion technology, combined with reinforcement learning algorithms, efficiently extracts useful information from multimodal data, enabling the agent to perceive and understand complex and ever-changing traffic environments in real time and dynamically adjust its decision-making strategy based on environmental changes. When facing urban traffic, highways, and emergencies, it can respond quickly and make adjustments, such as changing lanes or adjusting speed, ensuring driving safety and comfort. In the event of traffic congestion, sudden obstacles, or emergencies, the agent can react rapidly based on real-time perception data, avoiding potential safety risks, optimizing the driving experience, and effectively improving the perception accuracy and decision-making speed of the intelligent driving system in complex environments.

[0075] Furthermore, based on an end-to-end deep learning framework, driving decisions can be generated directly from raw sensor data, avoiding the tedious manual feature extraction and multi-stage modeling process. By using convolutional neural networks for feature extraction and combining deep reinforcement learning to optimize driving strategies, the speed and accuracy of decision response are significantly improved.

[0076] This embodiment demonstrates significant advantages over existing technologies in terms of environmental perception accuracy, driving decision-making efficiency, system robustness, safety, and driving experience. It better addresses complex and ever-changing autonomous driving scenarios, improves the performance of intelligent driving systems in practical applications, and meets the stringent requirements of autonomous driving technology for safety, intelligence, and efficiency. Furthermore, it not only provides innovative improvements in perception and decision-making performance but also offers strong guarantees for the safety and reliability of future autonomous driving technologies.

[0077] Secondly, embodiments of this application also provide an intelligent driving end-to-end decision model iteration device.

[0078] In one embodiment, reference is made to Figure 5 , Figure 5 This is a functional block diagram of an embodiment of the intelligent driving end-to-end decision model iteration device of this application. The aforementioned intelligent driving end-to-end decision model iteration device includes a data processing module, a fusion extraction module, a perception modeling module, and a decision optimization module.

[0079] The aforementioned data processing module is used to collect scene state data from multiple sensors and perform preprocessing to generate high-dimensional feature vectors; The aforementioned fusion extraction module is used to map the high-dimensional feature vectors from the original space to the bird's-eye view space through the BEV fusion model, and to use the attention mechanism to perform feature interaction in the bird's-eye view space to generate high-dimensional environmental features. The aforementioned perception modeling module is used to dynamically model the high-dimensional environmental features using graph neural networks and multi-task learning, and outputs an environmental perception map. The aforementioned decision optimization module is used to design a reward function that includes safety, driving efficiency, and navigation matching degree based on the aforementioned environmental perception map using a Markov decision process. This function is then used to optimize the constructed decision model based on the PPO algorithm and is adjusted through multiple rounds of iteration until the decision model converges.

[0080] Furthermore, in one embodiment, the above-described perception modeling module is also used for: Using the aforementioned high-dimensional environmental features as input, a graph neural network is constructed; the nodes of the graph neural network represent the vehicle and surrounding targets, and the edges represent the relationships between targets. By combining prior information about human driving cognition with the optimized feature learning of the graph neural network, a scene perception map is output.

[0081] Furthermore, in one embodiment, the decision optimization module is also used for: The driving decision-making process is described using a Markov decision process. A state set and an action set are constructed based on an environmental perception map, and a state transition function is designed.

[0082] Furthermore, in one embodiment, the decision optimization module is also used for: Safety rewards are determined based on the distance between the vehicle and surrounding obstacles; The driving efficiency reward is determined based on the difference between the current vehicle speed and the target vehicle speed, as well as the acceleration. The navigation matching reward is determined based on the deviation between the vehicle and the target path; Based on the aforementioned safety rewards, driving efficiency rewards, and navigation matching rewards, a reward function is designed.

[0083] Furthermore, in one embodiment, the above reward function is:

[0084] in, The first weighting coefficient, This is the second weighting coefficient. The third weighting coefficient, As a security reward, Rewards for driving efficiency Rewards are given for navigation matching accuracy.

[0085] Furthermore, in one embodiment, the decision optimization module is also used for: Training data is obtained through trajectory sampling, and the advantage function is calculated; The policy network is optimized using a pruning objective function based on the advantage function, and the value network is optimized by minimizing the value loss function.

[0086] Furthermore, in one embodiment, the data processing module is also used for: Collect scene state data from multiple sensors, timestamp the scene state data from each sensor, and align them using an extended Kalman filter algorithm to obtain multimodal data; Feature extraction and normalization are performed on the above multimodal data to generate high-dimensional feature vectors.

[0087] The functions of each module in the above-mentioned intelligent driving end-to-end decision model iteration device correspond to the steps in the above-mentioned intelligent driving end-to-end decision model iteration method embodiment, and their functions and implementation processes will not be described in detail here.

[0088] Thirdly, embodiments of this application provide an intelligent driving end-to-end decision model iteration device, which can be a device with data processing capabilities such as a personal computer (PC), a laptop, or a server.

[0089] Reference Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of the intelligent driving end-to-end decision model iteration device involved in the embodiments of this application. In the embodiments of this application, the intelligent driving end-to-end decision model iteration device may include a processor, a memory, a communication interface, and a communication bus.

[0090] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0091] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the intelligent driving end-to-end decision model iteration device, as well as interfaces used for interconnecting the intelligent driving end-to-end decision model iteration device with other devices (such as other computing devices or user devices). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user devices can be displays, keyboards, etc.

[0092] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0093] The processor can be a general-purpose processor, which can call the intelligent driving end-to-end decision model iteration program stored in the memory and execute the intelligent driving end-to-end decision model iteration method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the intelligent driving end-to-end decision model iteration program is called can be referred to the various embodiments of the intelligent driving end-to-end decision model iteration method of this application, and will not be repeated here.

[0094] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0095] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0096] The present application stores an intelligent driving end-to-end decision model iteration program on a computer-readable storage medium, wherein when the intelligent driving end-to-end decision model iteration program is executed by a processor, it implements the steps of the intelligent driving end-to-end decision model iteration method as described above.

[0097] The method implemented when the intelligent driving end-to-end decision model iteration program is executed can be referred to in various embodiments of the intelligent driving end-to-end decision model iteration method of this application, and will not be repeated here.

[0098] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0099] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0100] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0101] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0102] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0104] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An iterative method for an end-to-end decision-making model for intelligent driving, characterized in that, The method includes: Collect scene state data from multiple sensors and preprocess it to generate high-dimensional feature vectors; The high-dimensional feature vectors are mapped from the original space to the bird's-eye view space through the BEV fusion model, and the attention mechanism is used to perform feature interaction in the bird's-eye view space to generate high-dimensional environmental features. The high-dimensional environmental features are dynamically modeled using graph neural networks and multi-task learning, and an environmental perception map is output. Based on the environmental perception map, a Markov decision process is used to design a reward function that includes safety, driving efficiency, and navigation matching degree. The constructed decision model is then optimized using the PPO algorithm and adjusted through multiple rounds of iteration until the decision model converges.

2. The intelligent driving end-to-end decision model iterative method as described in claim 1, characterized in that, The high-dimensional environmental features are dynamically modeled using graph neural networks and multi-task learning to output an environmental awareness map, specifically including: Using the high-dimensional environmental features as input, a graph neural network is constructed; the nodes of the graph neural network represent the vehicle and surrounding targets, and the edges represent the relationships between targets. By combining prior information about human driving cognition with the optimized feature learning of the graph neural network, a scene perception map is output.

3. The intelligent driving end-to-end decision model iterative method as described in claim 1, characterized in that, Based on the aforementioned environmental perception map, a Markov decision process is used, specifically including: The driving decision-making process is described using a Markov decision process. A state set and an action set are constructed based on an environmental perception map, and a state transition function is designed.

4. The intelligent driving end-to-end decision model iterative method as described in claim 3, characterized in that, The design incorporates reward functions for safety, driving efficiency, and navigation matching accuracy, specifically including: Safety rewards are determined based on the distance between the vehicle and surrounding obstacles; The driving efficiency reward is determined based on the difference between the current vehicle speed and the target vehicle speed, as well as the acceleration. The navigation matching reward is determined based on the deviation between the vehicle and the target path; Based on the aforementioned safety reward, driving efficiency reward, and navigation matching reward, a reward function is designed.

5. The intelligent driving end-to-end decision model iterative method as described in claim 4, characterized in that, The reward function is: in, The first weighting coefficient, This is the second weighting coefficient. The third weighting coefficient, As a security reward, Rewards for driving efficiency Rewards are given for navigation matching accuracy.

6. The intelligent driving end-to-end decision model iterative method as described in claim 4, characterized in that, The decision model constructed based on the PPO algorithm is optimized, specifically including: Training data is obtained through trajectory sampling, and the advantage function is calculated; The policy network is optimized using a pruning objective function based on the advantage function, and the value network is optimized by minimizing the value loss function.

7. The intelligent driving end-to-end decision model iterative method as described in claim 1, characterized in that, Collect and preprocess multi-sensor scene state data to generate high-dimensional feature vectors, specifically including: Collect scene state data from multiple sensors, timestamp the scene state data from each sensor, and align them using an extended Kalman filter algorithm to obtain multimodal data; The multimodal data is subjected to feature extraction and normalization to generate a high-dimensional feature vector.

8. An iterative device for an end-to-end decision-making model for intelligent driving, characterized in that, The device includes: The data processing module is used to collect scene state data from multiple sensors and preprocess it to generate high-dimensional feature vectors. The fusion extraction module is used to map the high-dimensional feature vector from the original space to the bird's-eye view space through the BEV fusion model, and to perform feature interaction in the bird's-eye view space using an attention mechanism to generate high-dimensional environmental features. The perception modeling module is used to dynamically model the high-dimensional environmental features using graph neural networks and multi-task learning, and output an environmental perception map. The decision optimization module is used to design a reward function that includes safety, driving efficiency and navigation matching degree based on the environmental perception map using a Markov decision process, so as to optimize the constructed decision model based on the PPO algorithm, and adjust it through multiple rounds of iteration until the decision model converges.

9. An intelligent driving end-to-end decision model iteration device, characterized in that, The intelligent driving end-to-end decision model iteration device includes a processor, a memory, and an intelligent driving end-to-end decision model iteration program stored in the memory and executable by the processor, wherein when the intelligent driving end-to-end decision model iteration program is executed by the processor, it implements the steps of the intelligent driving end-to-end decision model iteration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an intelligent driving end-to-end decision model iteration program, wherein when the intelligent driving end-to-end decision model iteration program is executed by a processor, it implements the steps of the intelligent driving end-to-end decision model iteration method as described in any one of claims 1 to 7.

Images