Automatic driving vehicle navigation method based on deep reinforcement learning
Through pre-training deep Q networks and online learning mechanisms, and combining multi-sensor data to build high-definition maps, the problems of large memory consumption and long training time in traditional autonomous driving navigation methods are solved, and efficient and safe navigation in complex environments are achieved.
Patent Information
- Application Number
- CN202510721049.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-08-15
AI Technical Summary
The existing autonomous driving vehicle navigation methods consume a lot of memory in traditional reinforcement learning, making it difficult to cope with complex driving environments, deep neural network training time is long and the robustness and generalization capabilities are insufficient, making it difficult to meet the real-time and safety requirements of autonomous driving.
A pre-trained deep Q network is used to build a high-definition map with lidar and visual camera mechanism, output the optimal actions through the deep Q network, and learn fine-tuning online after the vehicle performs the actions. Combining the risk assessment model and the adaptive ε-greedy strategy, the actions are selected using the UCB algorithm, and the network parameters are optimized using the experience playback pool and buffer to achieve self-evolution of the strategy.
Significantly reduce memory usage, improve navigation accuracy and security, enhance robustness and generalization capabilities, quickly adapt to complex driving environments, and ensure real-time and security of navigation.
Smart Images
Figure CN120489148A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of autonomous driving vehicles, and specifically relates to an autonomous driving vehicle navigation method based on deep reinforcement learning. Background Art
[0002] The vehicle autonomous driving navigation method refers to a method that integrates multiple sensors, data processing algorithms and intelligent control systems to enable the vehicle to independently complete tasks such as road identification, path planning, obstacle avoidance and control without human intervention. It can not only improve road traffic safety, but also improve travel efficiency and reduce energy consumption. It is of great significance to the development of future intelligent transportation systems.
[0003] Current autonomous vehicle navigation methods use reinforcement learning to improve the accuracy and safety of autonomous vehicle navigation. However, existing reinforcement learning methods have the following shortcomings in autonomous vehicle navigation:
[0004] (1) Traditional reinforcement learning requires storing a large number of state-action Q-value tables, which consumes a lot of memory and is difficult to cope with complex driving environments;
[0005] (2) Introducing deep neural networks into reinforcement learning (DRL) can solve the problem of excessive state space, but the network parameter tuning is complex and the training time is long;
[0006] (3) The existing DRL algorithm does not perform well in terms of real-time performance, robustness, and generalization ability, and it is difficult to meet the stringent requirements of autonomous driving. Summary of the Invention
[0007] To solve the problems raised in the above background technology, the present invention provides an autonomous driving vehicle navigation method based on deep reinforcement learning, which has the characteristics of being able to adapt to complex driving environments.
[0008] To achieve the above objectives, the present invention provides the following technical solution: a deep reinforcement learning-based autonomous driving vehicle navigation method, comprising:
[0009] S1: Collects environmental data around the autonomous vehicle using lidar and visual cameras and builds a high-definition map in real time;
[0010] S2: Output the Q value of each potential action based on the constructed HD map and the current state of the vehicle through the deep Q network, and select the action with the largest Q value as the current optimal action;
[0011] S3: Control the autonomous vehicle to continuously execute the current optimal action until the end point, that is, complete the navigation
[0012] The deep Q network is a pre-trained deep Q network, and after the autonomous driving vehicle completes each action, the deep Q network is fine-tuned through online learning.
[0013] Furthermore, in one implementation, the specific steps of step S2 are:
[0014] S201: Assessing potential action risks based on the current state of the vehicle using a risk assessment model, where the risk assessment model is an independent neural network pre-trained using historical data;
[0015] S202: Select the action with the largest Q value as the current optimal action based on the evaluation results through the adaptive ε-greedy strategy for the autonomous driving vehicle to execute in the environment.
[0016] Furthermore, in one implementation, the step S202 further includes:
[0017] Dynamically adjust the exploration rate ε based on risk assessment results, reducing the exploration tendency under high-risk conditions;
[0018] According to the UCB algorithm, within a safe range, the action with a high estimated value or high uncertainty, that is, the action with the largest Q value, is selected as the current optimal action.
[0019] Furthermore, in one implementation, after the autonomous driving vehicle completes each action, the online learning fine-tuning of the deep Q network specifically includes: after the autonomous driving vehicle completes each action, calculating the immediate reward based on the execution result and the predetermined goal, and storing the experience in the experience replay pool; randomly extracting experience samples from the experience replay pool, for each sample, using the target network to calculate the maximum Q value in the new state, and combining the immediate reward and the discount factor to calculate the target Q value, using the mean square error to calculate the difference between the predicted Q value and the target Q value, updating the network weights through the back propagation algorithm based on the difference to minimize the loss function, and periodically copying part of the parameters of the current network to the target network to achieve online learning fine-tuning of the deep Q network.
[0020] Furthermore, in one implementation, a buffer is set in the experience replay pool, and experience tuples are stored in the buffer, each experience tuple includes the current state, executed action, reward obtained and the next state; after each action is completed, the generated experience tuple is added to the end of the buffer, and if the buffer capacity is full, the old experience tuple is replaced; the experience tuples in the buffer are calculated based on the preset rules, wherein the preset rules are based on the absolute value of the time difference error of the experience tuple; non-uniform sampling learning is performed based on the priority of the experience tuple, wherein the sampling amount of the experience tuple with high priority is high, and the sampling amount of the experience tuple with low priority is low.
[0021] Furthermore, in one implementation, the step of fine-tuning the deep Q network is provided with an evaluation period. When the evaluation period is over, the performance of the fine-tuned deep Q network is evaluated through historical data, and corresponding adjustments are made based on the evaluation results.
[0022] Furthermore, in one implementation, the specific steps of step S1 are:
[0023] S101: collects vehicle environmental data in real time through lidar and visual cameras;
[0024] S102: Preprocessing the collected vehicle environment data, including time synchronization and spatial registration;
[0025] S103: Process the lidar point cloud data through PointNet and output the extracted lidar vector features;
[0026] S104: Processing the image data of the visual camera layer by layer through a convolutional neural network, and outputting the extracted visual vector features;
[0027] S105: Fuse lidar vector features and visual vector features through a Transformer neural network with a multi-head attention mechanism, and output the fused multimodal features.
[0028] S106: Upsampling and convolution operations are performed on the fused multimodal features through the U-Net neural network to restore spatial resolution, perform pixel-level classification, and output a scene semantic segmentation map to identify the categories of different objects and regions;
[0029] S107: The SLAM algorithm combines the geometric information and scene semantic segmentation results provided by the lidar and visual camera to output a high-definition map of the vehicle environment.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] 1. The present invention designs a deep Q network to map state-action data into an optimal action strategy, which can significantly reduce memory usage and enable it to adapt to complex driving environments.
[0032] 2. The deep Q network of the present invention is generated by training and fine-tuning the pre-trained deep Q network through the historical navigation data of the autonomous driving vehicle, which can accelerate the speed of learning convergence. At the same time, the deep Q network is designed with a deep reinforcement learning algorithm, which can continuously incorporate new experience data for online optimization, realize strategy self-evolution, and improve navigation accuracy and safety.
[0033] 3. The present invention sets up a risk perception model to predict high-risk areas in high-definition maps, and collaborates with the deep Q network to moderately explore new strategies while ensuring safety, thereby improving the robustness and generalization ability of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a flowchart of the autonomous driving vehicle navigation method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION
[0035] In order to better describe the embodiments of the present invention, some names are explained below:
[0036] Deep Q-Network (DQNQ) is a method that combines deep learning and reinforcement learning to solve the problem of decision-making. It is an end-to-end learning method that can learn how to map to action values (i.e., policies) directly from raw inputs (such as pixels) without any manual feature engineering.
[0037] Q-value: In deep reinforcement learning, it refers to the estimated value of the state-action value function (Q function). In the Deep Q-Network (DQN), the Q-value represents the expected reward that can be obtained by taking an action in a given state.
[0038] Discount factor: A factor used to discount future rewards, which can be set based on experience.
[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0040] See also Figure 1 , which is a flow chart of a method for navigating an autonomous vehicle based on deep reinforcement learning according to an embodiment of the present invention. The method for navigating an autonomous vehicle based on deep reinforcement learning according to an embodiment of the present invention includes:
[0041] Step S1: Collect environmental data around the autonomous vehicle based on lidar and visual cameras and build a high-definition map in real time;
[0042] Step S2: Output the Q value of each potential action through the deep Q network based on the constructed high-definition map and the current state of the vehicle, and select the action with the largest Q value as the current optimal action; wherein, the deep Q network is a pre-trained deep Q network, and after the autonomous driving vehicle completes each action, the deep Q network is fine-tuned through online learning.
[0043] Step S3: Control the autonomous driving vehicle to continuously execute the current optimal action until the end point, thus completing the navigation.
[0044] By pre-training the network and self-evolution of the strategy, the learning cycle is effectively shortened. The trained deep Q network is used as the starting point for autonomous vehicle exploration, avoiding learning from scratch. The convergence speed is significantly improved, enabling the algorithm to quickly adapt to complex and changing traffic scenarios.
[0045] There are many ways to implement the aforementioned step S1, which is not limited in this embodiment. The specific implementation methods are as follows:
[0046] S101: Uses lidar and visual cameras to collect real-time environmental data from the vehicle. This environmental data is raw sensor data, such as images and point clouds.
[0047] S102: Preprocessing the collected vehicle environment data, including time synchronization and spatial registration;
[0048] S103: Process the lidar point cloud data through PointNet and output the extracted lidar vector features;
[0049] S104: Processing the image data of the visual camera layer by layer through a convolutional neural network, and outputting the extracted visual vector features;
[0050] S105: Fuse lidar vector features and visual vector features through a Transformer neural network with a multi-head attention mechanism, and output the fused multimodal features.
[0051] A Transformer neural network was constructed, and a multi-head attention mechanism was designed. The extracted LiDAR and visual vector features were used as input for Transformer training. During training, the encoder-decoder structure captured the long-range dependencies between features and effectively fused them into a unified representation until convergence. This resulted in a Transformer neural network with a multi-head attention mechanism.
[0052] S106: Upsampling and convolution operations are performed on the fused multimodal features through the U-Net neural network to restore spatial resolution, perform pixel-level classification, and output a scene semantic segmentation map to identify the categories of different objects and regions;
[0053] S107: The SLAM algorithm combines the geometric information and scene semantic segmentation results provided by the lidar and visual camera to output a high-definition map of the vehicle environment.
[0054] This application reconstructs the map and target information around the vehicle in all directions through the semantic fusion of multi-source heterogeneous sensor data such as machine vision and lidar.
[0055] In some embodiments, step S2 is further refined into the following steps S210 and S201:
[0056] Step S201: Evaluate potential action risks based on the current state of the vehicle using a risk assessment model, where the risk assessment model is an independent neural network pre-trained using historical data.
[0057] In this step, the risk assessment model identifies risk factors in the HD map and assigns a risk weight. Risk factors include environmental obstacles, dynamic factors, and environmental conditions. The risk assessment model then calculates a comprehensive risk score based on the weighted sum of the risk weights of the risk factors in the HD map area. Based on this comprehensive risk score, the model identifies the risk level of potential actions. The model can design a reward function that rewards those who explore high-risk areas but pass safely. This introduces an intrinsic reward mechanism (calculating immediate rewards based on execution results and predetermined goals) to encourage the agent to explore unknown or uncertain states. The model can also set safety boundaries to ensure that key safety constraints are not violated when exploring new strategies. The model can also set active learning sampling: identifying knowledge blind spots or areas of high uncertainty and proactively generating relevant scenarios for focused learning. By utilizing a balancing mechanism and active learning strategy, the action strategy is continuously optimized while ensuring safety, improving the system's adaptability and robustness.
[0058] Step S202: An adaptive ε-greedy strategy dynamically adjusts the exploration rate ε based on the evaluation results, reducing the exploration tendency in high-risk situations. Based on the UCB (Upper Confidence Bound) algorithm, within a safe range, an action with a high estimated value or high uncertainty, i.e., the action with the largest Q value, is selected as the current optimal action for the autonomous vehicle to execute in the environment. In this embodiment of the present invention, the input layer directly receives raw sensor data, such as images and point clouds, and then automatically learns effective feature representations using structures such as deep convolutional networks. The policy network uses fully connected layers to map the extracted features into action probability distributions or Q values.
[0059] The specific steps for online learning and fine-tuning the deep Q network after the autonomous vehicle completes each action are as follows:
[0060] After the autonomous vehicle completes each action, it calculates an immediate reward based on the execution result and the predetermined goal, and stores the experience in the experience replay pool;
[0061] Experience samples are randomly drawn from the experience replay pool. For each sample, the target network (the current deep Q-network) is used to calculate the maximum Q-value in the new state. The target Q-value is then calculated by combining the immediate reward and a discount factor. The mean squared error (MSE) is used to calculate the difference between the predicted and target Q-values. Based on this difference, the network weights are updated using a backpropagation algorithm to minimize the loss function. The parameters of the current network are periodically copied to the target network (using the temporal difference error to update the entire network parameters), enabling online learning and fine-tuning of the deep Q-network. The update strategy can adopt a progressive network update strategy, such as Polyak averaging, to slowly update the target network and maintain learning stability. This learning-while-planning mechanism continuously absorbs and internalizes new experience, enabling self-improvement and evolution of the strategy.
[0062] In a preferred embodiment, a buffer is set in the experience replay pool, and the buffer stores experience tuples, each of which includes the current state, the action performed, the reward obtained, and the next state;
[0063] After each action is completed, the generated experience tuple is added to the end of the buffer. If the buffer capacity is full, the old experience tuple is replaced. In the embodiment of the present application, the buffer size can be set to place 1000 to 10000 experience tuples, which can meet the algorithm requirements without occupying too much storage resources.
[0064] The priority of the experience tuple in the buffer is calculated based on a preset rule, wherein the preset rule is based on the absolute value of the time series difference error of the experience tuple;
[0065] Non-uniform sampling learning is performed based on the priority of experience tuples, where experience tuples with high priority have a high sampling volume, and experience tuples with low priority have a low sampling volume. Important or rare experiences are given higher priority so that they can obtain a higher sampling probability and accelerate the learning of key knowledge.
[0066] In a preferred embodiment, an evaluation cycle is set in the step of fine-tuning the deep Q network. When the evaluation cycle arrives, the performance of the fine-tuned deep Q network is evaluated through historical data, and corresponding adjustments are made according to the evaluation results. This is conducive to regularly evaluating the performance of the current strategy and dynamically adjusting the learning rate and exploration rate according to the evaluation results.
[0067] This embodiment of the present invention uses a deep neural network as a carrier to directly map continuous observation states to optimal actions or strategies. This replaces the multiple subsystems of traditional autonomous driving systems, such as perception, decision-making, planning, and control, significantly reducing memory usage and achieving a highly integrated process. A closed-loop feedback mechanism enables strategies to be evaluated and adjusted based on actual results, continuously identifying and correcting problems to ensure continuous improvement in system performance. Actual driving data is continuously collected and fed back into the training process, enabling continuous optimization of strategies. This end-to-end approach integrates traditional modular systems into a unified deep network, simplifying the system architecture and improving overall performance and adaptability.
[0068] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.
Claims
1. A deep reinforcement learning-based autonomous vehicle navigation method, characterized in that: include: S1: Collects environmental data around the autonomous vehicle using lidar and visual cameras and builds a high-definition map in real time; S2: Output the Q value of each potential action based on the constructed HD map and the current state of the vehicle through the deep Q network, and select the action with the largest Q value as the current optimal action; S3: Control the autonomous vehicle to continuously execute the current optimal action until the destination, thus completing navigation; The deep Q network is a pre-trained deep Q network, and after the autonomous driving vehicle completes each action, the deep Q network is fine-tuned through online learning.
2. The autonomous driving vehicle navigation method according to claim 1, wherein: The specific steps of step S2 are: S201: Assessing potential action risks based on the current state of the vehicle using a risk assessment model, where the risk assessment model is an independent neural network pre-trained using historical data; S202: Using an adaptive ε-greedy strategy, the action with the largest Q value is selected as the current optimal action based on the evaluation results for the autonomous driving vehicle to execute in the environment.
3. The autonomous driving vehicle navigation method according to claim 2, wherein: The step S202 further includes: Dynamically adjust the exploration rate ε based on risk assessment results, reducing the exploration tendency under high-risk conditions; According to the UCB algorithm, within a safe range, the action with a high estimated value or high uncertainty, that is, the action with the largest Q value, is selected as the current optimal action.
4. The autonomous driving vehicle navigation method according to claim 2 or 3, characterized in that: After the autonomous vehicle completes each action, online learning fine-tuning of the deep Q network specifically includes: After the autonomous vehicle completes each action, it calculates an immediate reward based on the execution result and the predetermined goal, and stores the experience in the experience replay pool; Experience samples are randomly drawn from the experience replay pool. For each sample, the target network is used to calculate the maximum Q value in the new state, and the target Q value is calculated by combining the immediate reward and the discount factor. The mean square error is used to calculate the difference between the predicted Q value and the target Q value. The network weights are updated based on the difference through the backpropagation algorithm to minimize the loss function. The parameters of the current network are partially copied to the target network periodically to achieve online learning and fine-tuning of the deep Q network.
5. The autonomous driving vehicle navigation method according to claim 4, wherein: A buffer is set in the experience replay pool, and experience tuples are stored in the buffer, each experience tuple includes the current state, the executed action, the reward obtained and the next state; after each action is completed, the generated experience tuple is added to the end of the buffer, and if the buffer capacity is full, the old experience tuple is replaced; the experience tuples in the buffer are calculated based on the priority of the preset rules, wherein the preset rules are based on the absolute value of the time difference error of the experience tuple; non-uniform sampling learning is performed based on the priority of the experience tuple, wherein the sampling amount of the experience tuple with high priority is high, and the sampling amount of the experience tuple with low priority is low.
6. The autonomous driving vehicle navigation method according to claim 4, wherein: The step of fine-tuning the deep Q network is set with an evaluation cycle. When the evaluation cycle is up, the performance of the fine-tuned deep Q network is evaluated through historical data, and corresponding adjustments are made based on the evaluation results.
7. The autonomous driving vehicle navigation method according to any one of claims 1 to 3, characterized in that: The specific steps of step S1 are: S101: collects vehicle environmental data in real time through lidar and visual cameras; S102: Preprocessing the collected vehicle environment data, including time synchronization and spatial registration; S103: Process the lidar point cloud data through PointNet and output the extracted lidar vector features; S104: Processing the image data of the visual camera layer by layer through a convolutional neural network, and outputting the extracted visual vector features; S105: Fuse lidar vector features and visual vector features through a Transformer neural network with a multi-head attention mechanism, and output the fused multimodal features. S106: Upsampling and convolution operations are performed on the fused multimodal features through the U-Net neural network to restore spatial resolution, perform pixel-level classification, and output a scene semantic segmentation map to identify the categories of different objects and regions; S107: The SLAM algorithm combines the geometric information and scene semantic segmentation results provided by the lidar and visual camera to output a high-definition map of the vehicle environment.