Multi-mode reinforcement learning vehicle decision planning method with compensation feedback
Through the multimodal reinforcement learning method, multiple sensor data are integrated and reward functions are designed, combined with local trajectory planner, the problems of low decision efficiency and model overfitting in complex environments are solved, and more accurate environmental perception is achieved and independent decision-making improves the adaptability and safety of smart vehicles.
Patent Information
- Application Number
- CN202410941500.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional vehicle decision planning methods are difficult to achieve accurate and efficient decision-making in complex traffic environments, and a single modal input cannot provide sufficient comprehensive information, resulting in smart vehicles not having a deeper understanding of the environment, the model may be overfitted, reducing generalization capabilities, and There may be a problem that the output action is out of actual or non-optimal output.
Using a multimodal reinforcement learning method with compensation feedback, we design a reinforcement learning interactive environment, fuse multiple sensor data (such as point cloud, image, vehicle state), build a comprehensive environmental state representation, design reward function and deep reinforcement learning The model, combined with the local trajectory planner, implements dynamic constraints and compensation to improve the optimization of decision strategies.
Provide more comprehensive and accurate environmental perception, improve the independent decision-making ability of smart vehicles, enhance adaptability and safety to complex traffic scenarios, reduce dangerous driving behaviors, and improve driving autonomy and intelligence level.
Smart Images

Figure CN118917179A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent driving technology, and in particular relates to a multi-modal reinforcement learning vehicle decision planning method with compensation feedback. Background Art
[0002] With the rapid development of economic globalization and the continuous expansion of urbanization, the number of cars is increasing, making urban traffic networks more complex, bringing about problems such as traffic congestion, safety issues and air pollution. In the face of these challenges, the emergence of intelligent vehicles has become a key solution. Intelligent vehicles need to make accurate and efficient decisions in complex and changing traffic environments to ensure the safety and smoothness of driving. Therefore, the research on vehicle decision planning methods is crucial to improving the performance and reliability of autonomous driving systems. Traditional vehicle decision planning methods rely on accurate environmental models, which are difficult to achieve in complex traffic scenarios, and these methods have difficulty handling high-dimensional state spaces and action spaces, resulting in low decision efficiency in complex environments.
[0003] With the rapid development of artificial intelligence and deep learning technology, deep reinforcement learning combines traditional reinforcement learning methods with deep neural networks, bringing new possibilities to autonomous driving systems. This technology enables autonomous vehicles to make accurate and efficient decisions in complex and changing traffic environments, improving driving safety, comfort and efficiency. However, there are still some challenges in directly applying reinforcement learning to decision-making planning for autonomous driving. For example, the commonly used single-modal input cannot provide enough comprehensive information to describe the complex environment, resulting in the agent's understanding of the environment not being deep and accurate enough. In addition, single-modal input may contain a lot of redundant information, which will increase the complexity of the model and may cause the model to overfit the data and reduce its generalization ability. In addition, in some cases, the decision-making planning of autonomous driving may result in output actions that are out of touch with reality or non-optimal outputs, which require constraint compensation to correct them, and at the same time, the training process of the agent needs to be accelerated. Summary of the invention
[0004] The purpose of the present invention is to provide a multimodal reinforcement learning vehicle decision planning method with compensation feedback, aiming to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] A multimodal reinforcement learning vehicle decision planning method with compensation feedback includes the following steps:
[0007] Step 1: Reinforcement learning interactive environment design: determine the scenarios and conditions for the interaction between the intelligent vehicle and the environment, and design the interaction rules and behaviors between the intelligent vehicle and other traffic participants;
[0008] Step 2: Multimodal fusion of environmental status: Collect data from multiple sensors, integrate and fuse data from different modalities, and construct a comprehensive representation of environmental status;
[0009] Step 3: Reward function design: According to the characteristics and objectives of the decision-making task, a reward function is designed to motivate the intelligent vehicle to take correct decision-making actions;
[0010] Step 4: Build a deep reinforcement learning model: Design the structure of the deep reinforcement learning model, including Actor and Critic; use multimodal environmental states as input and train the model to learn the optimal decision-making strategy;
[0011] Step 5. Design a local trajectory planner feedback mechanism: Design a local trajectory planner to generate a local trajectory based on the current state of the intelligent vehicle and the environmental conditions; design a feedback mechanism to combine the action strategy output by the deep reinforcement learning model with the local trajectory planner to achieve dynamic constraints and compensation.
[0012] Furthermore, the specific process of step 2 is: integrating and fusing data from different sensors, including point clouds, images, and status information of the vehicle and surrounding vehicles, to construct a comprehensive environmental status representation; specifically including the following steps:
[0013] Step 21, pre-process the data from different sensors: perform denoising and image enhancement on image data; perform filtering and feature extraction on point cloud data;
[0014] Step 22, extract features from the preprocessed data for subsequent fusion: for image data, use a convolutional neural network to extract image features; for point cloud data, use a point cloud processing algorithm to extract point cloud features; for vehicle status information, extract the relative coordinates (x, y) between the vehicle and the target point, the vehicle's forward direction and the angle between the target point, speed, vehicle length and width, throttle, direction, brake, lane ID, lane width, lane curvature, deviation from the center of the lane, left lane ID, right lane ID, and the number of surrounding vehicles; extract the relative (x, y) between the surrounding vehicles and the vehicle, the length and width of the surrounding vehicles, the lane ID of the surrounding vehicles, and the speed of the surrounding vehicles;
[0015] Step 23: Fuse the extracted features of different modes to construct a comprehensive representation of the environment state. The fusion method uses weighted summation or uses a neural network model for end-to-end learning.
[0016] Furthermore, the specific process of step 3 is as follows:
[0017] Step 31, collision reward and penalty function design: During training, vehicle collision information is collected through vehicle collision sensors, and a collision reward and penalty function r is designed based on the collision information. collision as follows:
[0018]
[0019] Among them, flag collision Is the collision flag; if the collision flag is True, it means that the vehicle has collided, and the agent is given a penalty of -200. If the collision flag is False, it means that there is no collision, and no reward or punishment is given. At the same time, the collision flag is one of the termination conditions for each round of agent training. If the collision flag is True, the training of this round is terminated.
[0020] Step 32: Target reward function design: Design a target reward function based on whether the driving task is successfully completed:
[0021]
[0022]
[0023] Among them, flag success is the flag for reaching the end of the mission scenario; if the agent successfully reaches the target location, the reward value is a positive number, indicating that the mission is completed; if the agent fails to reach the target location, the reward value is zero, indicating that the mission is not completed or failed;
[0024] Step 33: Guide the design of reward and punishment function: Guide the reward and punishment function r guidance The road center distance and heading deviation in the path constraint are used as rewards and penalties, including the road center distance reward and penalty function r d and heading deflection angle reward and penalty function r θ :
[0025] r guidance =r d +r θ +r n ;
[0026] Road center distance reward and penalty function r d Guiding the vehicle to its target driving position:
[0027]
[0028] When the distance between the center of the road and lat Less than the limit distance D limit When the vehicle is farther from the center of the lane, the penalty value is greater; when the distance from the center of the road is D lat Greater than the limit distance D limit When , the penalty value is always -1;
[0029] The local planning path is close to the reward and punishment function r θ Guiding the vehicle's target driving direction:
[0030]
[0031] Among them, θ error is the deviation between the local planning path heading vector and the current agent heading vector;
[0032] When the heading deviation angle is less than or equal to 1°, the vehicle moves in the correct direction and is given a reward of 1; when the heading deviation angle is greater than 1° and within the limit range θ limit The more the vehicle deviates from the correct course, the lower the reward value will be. When the course deviation angle is greater than the limit range, a penalty of -1 will be given.
[0033] Target guidance function r n Guide the vehicle to move towards the target point:
[0034]
[0035] Assume that the current road center point is W N , then D x2 The distance between the vehicle and the center point of the road W N+2 The distance, D x3 The distance between the vehicle and the center point of the road W N+3 distance;
[0036] Step 34: Design of speed reward and penalty function: The speed reward and penalty function guides the vehicle to gradually approach or maintain the target speed according to the difference between the current vehicle speed and the target speed:
[0037]
[0038] Among them, r v is the speed reward and penalty value; α is the speeding reward and penalty coefficient, β is the deceleration reward and penalty coefficient; v min and v max are the lower and upper limits of the target speed range respectively;
[0039] If the current vehicle speed v is within the target speed range, that is, v min ≤v≤v max , a reward of 10 is given; if the current vehicle speed v exceeds the target speed range, that is, v <v min or v>v max , then give corresponding punishment or reward according to the degree of deviation of vehicle speed from the target speed;
[0040] Step 35: The total reward R of the vehicle agent is the weighted sum of the collision reward, target reward, guidance reward, and speed reward:
[0041] R=β1r collision +β2r target +β3r guiddnce +r v ;
[0042] Among them, β1 is the collision reward and punishment coefficient, β2 is the target reward and punishment coefficient, and β3 is the guidance reward and punishment coefficient.
[0043] Furthermore, the specific process of step 4 is as follows:
[0044] Step 41: Select a deep learning architecture: Use the SAC algorithm as the optimization strategy. The SAC algorithm generates a corresponding behavior strategy based on the current state and environmental feedback, and optimizes the strategy based on the reward signal. The SAC algorithm combines the Actor and the Critic and introduces entropy maximization. The steps of the SAC algorithm are as follows:
[0045] Use Actor to generate actions, the goal is to maximize the expected cumulative reward, and define the following objective function:
[0046]
[0047] in, is the expected value, which represents the expectation of the empirical data distribution D (i.e., the distribution of state-action pairs), γ is the discount factor, r(s, a) is the immediate reward obtained after selecting action a in state s, α is the entropy coefficient, is the entropy of strategy π, π θ It is a parameterized strategy;
[0048] Given a state s t , through the Actor network, we get all the action probabilities π(a|s t ), and obtain action a according to probability sampling t , a t Input into the environment to get s t+1 and r t , gain experience(s t , a t ,s t+1 , r t ), put it into the experience pool;
[0049] Sample data from the experience pool (s t , a t ,s t+1 , r t ), update the network parameter ω; t Q(s t , a t ) value as s tThe predictive value estimate of s is obtained according to the optimal Bellman equation t The true value of the state is estimated to be E π [r t +γQ(s t+1 ,π(s t+1 ))|s t ], where E π is s under strategy π t The expected value of the state, r t is the immediate reward obtained at time step t; the Q Critic network is trained using the mean square loss function as the loss, and the loss function is defined as:
[0050]
[0051] Among them, E B is the expected loss value, θ targ are the parameters of the target network;
[0052] Sample data from the experience pool (s t , a t ,s t+1 , r t ), update the network parameter θ; the true value of the output of the V Critic network is r t +γV(s t+1 )-αlogπ(a′ t |s t ), where V(S t+1 ) is in state s t+1 The value estimate under a′ t is the policy π in state s t The action sampled from ; the loss of the VCritic network is calculated based on the true value:
[0053]
[0054] The loss function for gradient descent training is defined as the TD error to measure the correction amplitude of the algorithm. The action a selected by the strategy is calculated in the form of TD error. t To conduct an assessment:
[0055] δ t =r t +γQ(s t+1 , a t+1 )-Q(s t , a t );
[0056] Among them, δ t is the TD error at time step t, Q is the state value of the Critic;
[0057] Step 42: Define the state space and action space: Considering the position, velocity, and acceleration of the ego vehicle and surrounding vehicles, the state space is represented by s = (s ego ,s surroundings ), the semantic segmentation image in the state space is represented as s image , the point cloud in the state space is represented as s point , considering the destination and route information of the current vehicle, represented as s task =(x goal , route); get the state space representation S = (s, s image ,s point ,s task );
[0058] The action space is represented as A = (a, θ), where a is the vehicle acceleration, a∈[a min , a max ]; θ is the vehicle steering angle, θ∈[-θ max ,θ max ].
[0059] Furthermore, the specific process of step 5 is as follows:
[0060] The trajectory planner uses Lattice Planner. The trajectory planner predicts the trajectory points in the future, calculates the expected action size in the future through the predicted trajectory points, and then connects the points to form a lateral trajectory based on the existing sampling points. The method of connecting according to the fifth-order polynomial after given boundary conditions is adopted. The formula is as follows:
[0061] d(t)=a5t 5 +a4t 4 +a3t 3 +a2t 2 +a1t+a0;
[0062] Among them, a i (i=1, 2, ..., 5) are coefficients of the polynomial, and t is time;
[0063] The compensation feedback mechanism is introduced to calculate the difference between the future expected action predicted by the local trajectory planner and the action output by the vehicle agent during the actual driving process, and the compensation value is obtained. The calculation formula of the compensation value is Δ=λ×(a t -a n ), where Δ represents the compensation value, λ is the compensation coefficient, and a t is the expected action size in the future, a n is the action size of the current network output;
[0064] After the compensation value is calculated, it is returned to the SAC network as loss information:
[0065]
[0066] Compared with the prior art, the present invention has the following beneficial effects:
[0067] 1. Through the fusion of multimodal data, the present invention can provide more comprehensive and accurate environmental perception, enabling intelligent vehicles to better understand the surrounding environment, including road conditions, vehicle driving status, etc.
[0068] 2. The present invention utilizes a deep reinforcement learning model to realize autonomous decision-making of intelligent vehicles. Compared with traditional rule-making methods, it has a higher level of intelligence and adaptability, and can better adapt to different traffic scenarios and complex road conditions.
[0069] 3. The present invention adopts a reinforcement learning algorithm. Intelligent vehicles can quickly optimize decision-making strategies through continuous interaction and learning, gradually improve driving skills, adapt to different driving environments and road conditions more quickly, and improve driving autonomy and intelligence.
[0070] 4. The present invention introduces a compensation feedback mechanism, which enables intelligent vehicles to adjust their decision-making strategies in a timely manner, reduce the occurrence of dangerous driving behaviors, and improve the resilience and safety of intelligent vehicles. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 It is a flow chart of the present invention.
[0072] Figure 2 This is a network structure diagram of the present invention.
[0073] Figure 3 This is a network structure diagram of multimodal fusion in the present invention.
[0074] Figure 4 This is a network architecture diagram of the SAC algorithm in the present invention. DETAILED DESCRIPTION
[0075] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0076] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.
[0077] like Figure 1 As shown, a multimodal reinforcement learning vehicle decision planning method with compensation feedback provided by an embodiment of the present invention includes the following steps:
[0078] Step 1: Reinforcement learning interactive environment design: determine the scenarios and conditions for the interaction between the intelligent vehicle and the environment, and design the interaction rules and behaviors between the intelligent vehicle and other traffic participants;
[0079] Step 2: Multimodal fusion of environmental status: Collect data from multiple sensors, integrate and fuse data from different modalities, and construct a comprehensive representation of environmental status;
[0080] Step 3: Reward function design: According to the characteristics and objectives of the decision-making task, a reward function is designed to motivate the intelligent vehicle to take correct decision-making actions;
[0081] Step 4: Build a deep reinforcement learning model: Design the structure of the deep reinforcement learning model, including Actor and Critic; use multimodal environmental states as input and train the model to learn the optimal decision-making strategy;
[0082] Step 5. Design a local trajectory planner feedback mechanism: Design a local trajectory planner to generate a local trajectory based on the current state of the intelligent vehicle and the environmental conditions; design a feedback mechanism to combine the action strategy output by the deep reinforcement learning model with the local trajectory planner to achieve dynamic constraints and compensation.
[0083] As a preferred embodiment of the present invention, in step 1, the reinforcement learning interactive environment design refers to determining the scenarios and conditions for the intelligent vehicle to interact with the surrounding environment, and designing the interaction rules and behaviors between the intelligent vehicle and other traffic participants (such as other vehicles, pedestrians, etc.), so that the intelligent agent can generate corresponding rewards based on the current environmental state. The main purpose of this step is to create a simulated or real environment in which the intelligent vehicle can learn and improve its decision-making strategy. The specific process is as follows:
[0084] First, the scene is set up to determine the driving scene of the intelligent vehicle, including road type (urban roads, highways, etc.), traffic conditions (congestion, free flow, etc.), weather conditions (sunny, rainy, etc.). Different scenes will have different impacts on the decision-making of intelligent vehicles, so they need to be set according to actual needs. Secondly, other traffic participants are determined, including other vehicles, pedestrians, bicycles, etc. These traffic participants will have an impact on the decision-making of intelligent vehicles and may become a consideration for decision-making. In addition, it is necessary to design interaction rules between intelligent vehicles and other traffic participants, including traffic rules, driving etiquette, etc. These rules will affect the behavior selection and decision-making strategy of intelligent vehicles. Simulate or simulate the behavior of intelligent vehicles and other traffic participants to create a real driving environment. This can be done through simulation software or actual test vehicles on real roads. Determine the information exchange method and feedback mechanism between the intelligent vehicle and the environment, including sensor data collection, information transmission, etc. By reasonably designing the reinforcement learning interaction environment, a realistic learning scene can be provided for the intelligent vehicle, prompting it to learn and improve its decision-making ability, thereby improving driving safety and efficiency.
[0085] As a preferred embodiment of the present invention, in step 2, the multimodal fusion of environmental status refers to the effective integration and fusion of data from different sensors, such as point clouds, images, and status information of the vehicle and surrounding vehicles. The purpose of this step is to improve the understanding and perception of the environment, so as to make decisions and plans more effectively. This method can overcome the limitations of a single sensor, enable the system to obtain and understand environmental information from different angles, and improve the comprehensiveness and accuracy of the system's understanding of the environment.
[0086] The specific process is as follows:
[0087] First, the data from different sensors are preprocessed to ensure the quality and consistency of the data. For example, image data is processed by denoising and image enhancement; point cloud data is processed by filtering and feature extraction. Feature extraction is performed on the preprocessed data for subsequent fusion: for image data, convolutional neural network (CNN) is used to extract image features; for point cloud data, point cloud processing algorithm (PointNet++) is used to extract point cloud features; for vehicle status information, the relative coordinates (x, y) between the vehicle and the target point, the angle between the vehicle's forward direction and the target point, speed, vehicle length and width, throttle, direction, brake, lane ID, lane width, lane curvature, deviation from the center of the lane, left lane ID, right lane ID, and number of surrounding vehicles are extracted; the relative (x, y) between the surrounding vehicles and the vehicle, the length and width of the surrounding vehicles, the lane ID of the surrounding vehicles, and the speed of the surrounding vehicles are extracted, and data features are extracted through the fully connected layer (MLP). The extracted features of different modes are fused to construct a comprehensive environmental state representation. The fusion method can be a simple weighted summation (see Figure 3 ), and further processed by MLP to output the final joint eigenvalues. More complex neural network models can also be used for end-to-end learning.
[0088] It should be noted that in the fusion process, the weight distribution between different modalities and the correlation between different modalities need to be considered. The fused environmental state representation is converted into an input format acceptable to the model for subsequent decision-making and control. The state can be represented in the form of a vector or tensor and then input into the reinforcement learning model as a state feature.
[0089] As a preferred embodiment of the present invention, in step 3, the design of the reward function is crucial in reinforcement learning, which defines how well the agent acts in the environment. In the field of intelligent driving, the design of the reward function directly affects the effectiveness of the driving strategy learned by the agent. The following are several key aspects that need to be considered when designing the reward function:
[0090] 1) Collision reward and penalty function: During training, vehicle collision information is collected through vehicle collision sensors, and a collision reward and penalty function r is designed based on the collision information. collision as follows:
[0091]
[0092] Among them, flag collisionIt is the collision flag, and the collision reward and punishment are mainly based on punishment. If the collision flag is True, it means that the vehicle has collided, and the agent is given a penalty of -200. If the collision flag is False, it means that there is no collision and no reward or punishment is given. At the same time, the collision flag is one of the termination conditions for each round of agent training. If the collision flag is True, the training of this round is terminated.
[0093] 2) Target reward function: The target reward function is designed based on whether the driving task is successfully completed. It is to assist the intelligent agent to recognize the importance of completing the driving task and then learn the driving strategy.
[0094]
[0095]
[0096] Among them, flag success It is the flag for reaching the end of the mission scenario; if the agent successfully reaches the target position, the reward value is a positive number, indicating that the mission is completed; if the agent fails to reach the target position, the reward value is zero, indicating that the mission is not completed or failed.
[0097] 3) Guiding reward and punishment function: Guiding reward and punishment function r guidance The road center distance and heading deviation in the path constraint are used as rewards and penalties, including the road center distance reward and penalty function r d and heading deflection angle reward and penalty function r θ :
[0098] r guidance =r d +r θ +r n ;
[0099] Road center distance reward and penalty function r d Guiding the vehicle to its target driving position:
[0100]
[0101] The road center distance reward and penalty function is mainly a penalty term, which penalizes the behavior decision away from the lane center. lat Less than the limit distance D limit The farther the vehicle is from the center of the lane, the greater the penalty value. limit The penalty value is always -1.
[0102] The local planning path is close to the reward and punishment function r θ Guiding the vehicle's target driving direction:
[0103]
[0104] The local planning path is close to the reward and penalty function, which is mainly a reward item, which rewards the action decision that can drive to the current desired heading planned by the local trajectory planner. error is the deviation between the heading vector of the local planning path and the heading vector of the current agent. When the heading deviation angle is less than or equal to 1°, the vehicle drives in the correct direction and is given a reward of 1; when the heading deviation angle is greater than 1° and within the limit range θ limit When the deviation angle is greater than the limit, the vehicle deviates seriously from the correct course and needs to be punished with a value of -1.
[0105] Target guidance function r n Guide the vehicle to move towards the target point:
[0106]
[0107] Assume that the current road center point is W N , then D x2 The distance between the vehicle and the center point of the road W N+2 The distance, D x3 The distance between the vehicle and the center point of the road W N+3 distance.
[0108] 4) Speed reward and penalty function: The speed reward and penalty function guides the vehicle to gradually approach or maintain the target speed based on the difference between the current speed and the target speed:
[0109]
[0110] Among them, r v is the speed reward and penalty value; α is the speeding reward and penalty coefficient, β is the deceleration reward and penalty coefficient, which is used to adjust the size of the reward and penalty; V min and v max are the lower and upper limits of the target speed range respectively. If the current vehicle speed v is within the target speed range, that is, v min ≤v≤v max , a reward of 10 is given if the current vehicle speed v exceeds the target speed range, that is, v <V min or v>V max , then a corresponding penalty or reward is given according to the degree to which the vehicle speed deviates from the target speed, so as to prompt the vehicle to adjust the speed back to the target speed range as soon as possible. The selection of the reward and penalty coefficient α can be adjusted according to the specific situation to balance the effect of speed control.
[0111] The total reward of the vehicle agent is the weighted sum of collision reward and penalty, target reward, guidance reward and penalty, and speed reward and penalty. The total reward is represented by R:
[0112] R=β1rcollision +β2r target +β3r guiddnce +r v ;
[0113] Among them, β1 is the collision reward and punishment coefficient, β2 is the target reward and punishment coefficient, and β3 is the guidance reward and punishment coefficient.
[0114] As a preferred embodiment of the present invention, the specific process of step 4 is as follows:
[0115] 1) Select a suitable deep learning architecture: This invention takes the SAC algorithm (Soft Actor-Critic) as an optimization strategy as an example. The SAC algorithm is an advanced deep reinforcement learning algorithm that can effectively process continuous action space and high-dimensional state space, and has good performance and convergence. The present invention takes the fused multimodal features as input, including image features, point cloud features, and vehicle status information. After preprocessing and feature extraction, these features are input into the SAC algorithm. The SAC algorithm generates corresponding behavioral strategies based on the current state and environmental feedback, and optimizes the strategies based on the reward signals, so that the intelligent agent can make accurate and efficient decisions in complex traffic environments. It combines deterministic strategies (Actor) and value functions (Critic), and introduces entropy maximization to promote the exploratory nature of the strategy. The network architecture of the SAC algorithm is as follows: Figure 4 As shown. The main steps and formulas of the SAC algorithm are as follows:
[0116] Use Actor to generate actions, the goal is to maximize the expected cumulative reward, and define the following objective function:
[0117]
[0118] in, is the expected value, which represents the expectation of the empirical data distribution D (i.e., the distribution of state-action pairs); γ is the discount factor; r(s, a) is the immediate reward obtained after selecting action a in state s; α is the entropy coefficient, which is used to balance exploration and exploitation; is the entropy of strategy π, indicating the uncertainty of the strategy; θ It is a parameterized strategy.
[0119] Given a state s t , through the Actor network, we get all the action probabilities π(a|s t ), according to the probability sampling, action a is obtained t , a t Input into the environment to get s t+1 and r t , gain experience(s t , a t ,st+1 , r t ), put it into the experience pool;
[0120] Sample data from the experience pool (s t , a t ,s t+1 , r t ), update the network parameter ω; t Q(s t , a t ) value as s t The predictive value estimate of s is obtained according to the optimal Bellman equation t The true value of the state is estimated to be E π [r t +γQ(s t+1 ,π(s t+1 ))|s t ], where E π is s under strategy π t The expected value of the state, r t is the immediate reward obtained at time step t; the Q Critic network is trained using the mean square loss function as the loss, and the loss function is defined as:
[0121]
[0122] Among them, E B is the expected loss value, θ targ are the parameters of the target network;
[0123] Sample data from the experience pool (s t , a t ,s t+1 , r t ), update the network parameter θ; the true value of the output of the V Critic network is r t +γV(s t+1 )-αlogπ(a′ t |s t ), where V(s t+1 ) is in state s t+1 The value estimate under a′ t is the policy π in state s t The action sampled from ; the loss of the VCritic network is calculated based on the true value:
[0124]
[0125] The loss function for gradient descent training is defined as: reinforcement learning measures the algorithm correction amplitude through temporal difference (TD) error, and calculates the TD error to correct the action a selected by the strategy. t To conduct an assessment:
[0126] δ t =r t +γQ(s t+1 , a t+1 )-Q(s t , a t );
[0127] Among them, δ t is the TD error at time step t, and Q is the state value of the Critic.
[0128] The SAC pseudo code is shown in Table 1:
[0129] Table 1 SAC pseudo code
[0130]
[0131]
[0132] 2) Define state space and action space: Considering the position, velocity, and acceleration of the ego vehicle and surrounding vehicles, the state space is represented by s = (s ego ,s surroundings ), the semantic segmentation image in the state space is represented as s image , the point cloud in the state space is represented as s point , considering the destination and route information of the current vehicle, represented as s task =(x goal , route); Taking the above information into consideration, the state space can be expressed as S = (s, s image ,s point ,s task );
[0133] The agent can choose the acceleration to control the acceleration or deceleration of the vehicle. The action space can be defined as a continuous acceleration range, a∈[a min , a max ]. If the lane changes or there is an obstacle, the agent can choose to turn to avoid it. The action space can be defined as the range of the vehicle’s steering angle, θ∈[-θ max ,θ max]. Taking the above actions into consideration, the action space is expressed as A = (a, θ), where a is the vehicle acceleration and θ is the vehicle steering angle. Through the state space and action space designed in this way, the intelligent driving system can select appropriate acceleration and steering according to the current vehicle state, surrounding environment information and mission scenario information to achieve safe and efficient autonomous driving.
[0134] As a preferred embodiment of the present invention, in step 5, the local trajectory planner plays a key role in the method of multimodal reinforcement learning vehicle decision planning with compensation feedback proposed by the present invention. It generates the optimal local trajectory based on the environmental state information, providing important guidance for the safe driving of the vehicle. The present invention takes LatticePlanner as an example. The trajectory planner predicts the trajectory points in the future. After the trajectory planner generates future trajectory points, the expected action size in the future can be calculated through these trajectory points to guide the future driving direction and speed of the vehicle. This expected action size in the future is based on a comprehensive consideration of the current state of the vehicle and environmental conditions, including factors such as the vehicle's position, speed, and surrounding obstacles. According to the existing sampling points, it is also necessary to connect the points to form a lateral trajectory. The method adopted is to connect them according to a quintic polynomial after the boundary conditions are given. The formula is as follows:
[0135] d(t)=a5t 5 +a4t 4 +a3t 3 +a2t 2 +a1t+a0;
[0136] Among them, a i (i=1,2,...,5) are coefficients of the polynomial, and t is time;
[0137] Then the compensation feedback mechanism is introduced. By comparing the future trajectory generated by the local trajectory planner with the state information of the vehicle during actual driving, the difference between the two is calculated, and the actions of the intelligent driving system are adjusted and corrected based on this difference. The compensation feedback mechanism compares the expected action size calculated by the future trajectory with the output action of the SAC network to obtain a compensation value. This compensation value is the difference between the future expected action predicted by the local trajectory planner and the action output by the vehicle agent during actual driving.
[0138] In the SAC network, after the trajectory planner generates future trajectories, the expected action size in the future is calculated based on these trajectories to guide the future driving direction and speed of the vehicle agent. Then, the expected action size in the future is compared with the action size output by the current network, and the difference between them is calculated. This difference value represents the deviation between the action output by the current network and the expected action in the future, which is the basis for compensation.
[0139] The calculation formula of compensation value is Δ=λ×(a t -a n ), where Δ represents the compensation value; λ is the compensation coefficient, which is used to adjust the size of the compensation value to control the compensation amplitude; a t is the expected action size in the future; a n is the action size of the current network output;
[0140] After the compensation value is calculated, it is returned to the SAC network as loss information:
[0141]
[0142] During the update process of the SAC network, the parameters of the network are adjusted to reduce the difference between the current network output action and the expected action in the future, taking into account the guidance of the compensation value. In this way, the SAC network can better adapt to changes in the environment and the requirements of the task, thereby improving the performance and safety of the intelligent driving system. At the same time, the intelligent driving system corrects the output action according to this compensation value, so that the vehicle better conforms to the expected driving path.
[0143] By introducing a local trajectory planner and compensation feedback mechanism, the intelligent driving system can better adapt to environmental changes and uncertain traffic conditions, thereby improving driving safety and stability. The application of this mechanism enables the intelligent driving system to more reliably cope with various complex driving scenarios, providing drivers and pedestrians with a safer and more comfortable travel experience.
[0144] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.
Claims
1. A multimodal reinforcement learning vehicle decision planning method with compensation feedback, characterized in that: The following steps are involved: Step 1: Reinforcement learning interactive environment design: determine the scenarios and conditions for the interaction between the intelligent vehicle and the environment, and design the interaction rules and behaviors between the intelligent vehicle and other traffic participants; Step 2: Multimodal fusion of environmental status: Collect data from multiple sensors, integrate and fuse data from different modalities, and construct a comprehensive representation of environmental status; Step 3: Reward function design: According to the characteristics and objectives of the decision-making task, a reward function is designed to motivate the intelligent vehicle to take correct decision-making actions; Step 4: Build a deep reinforcement learning model: Design the structure of the deep reinforcement learning model, including Actor and Critic; use multimodal environmental states as input and train the model to learn the optimal decision-making strategy; Step 5: Design a local trajectory planner feedback mechanism: Design a local trajectory planner to generate a local trajectory based on the current state of the intelligent vehicle and the environment; A feedback mechanism is designed to combine the action policy output by the deep reinforcement learning model with the local trajectory planner to achieve dynamic constraints and compensation.
2. The multimodal reinforcement learning vehicle decision planning method with compensation feedback according to claim 1 is characterized in that: The specific process of step 2 is: integrating and fusing data from different sensors, including point clouds, images, and status information of the vehicle and surrounding vehicles, to construct a comprehensive environmental status representation; specifically including the following steps: Step 21, pre-process the data from different sensors: perform denoising and image enhancement on image data; perform filtering and feature extraction on point cloud data; Step 22, extract features from the preprocessed data for subsequent fusion: for image data, use a convolutional neural network to extract image features; for point cloud data, use a point cloud processing algorithm to extract point cloud features; for vehicle status information, extract the relative coordinates (x, y) between the vehicle and the target point, the vehicle's forward direction and the angle between the target point, speed, vehicle length and width, throttle, direction, brake, lane ID, lane width, lane curvature, deviation from the center of the lane, left lane ID, right lane ID, and the number of surrounding vehicles; extract the relative (x, y) between the surrounding vehicle and the vehicle, the length and width of the surrounding vehicle, the lane ID of the surrounding vehicle, and the speed of the surrounding vehicle; Step 23: Fuse the extracted features of different modes to construct a comprehensive representation of the environment state. The fusion method uses weighted summation or uses a neural network model for end-to-end learning.
3. The multimodal reinforcement learning vehicle decision planning method with compensation feedback according to claim 1, characterized in that: The specific process of step 3 is as follows: Step 31, collision reward and penalty function design: During training, vehicle collision information is collected through vehicle collision sensors, and a collision reward and penalty function r is designed based on the collision information. collision as follows: Among them, flag collision Is the collision flag; if the collision flag is True, it means that the vehicle has collided, and the agent is given a penalty of -200. If the collision flag is False, it means that there is no collision, and no reward or punishment is given. At the same time, the collision flag is one of the termination conditions for each round of agent training. If the collision flag is True, the training of this round is terminated. Step 32: Target reward function design: Design a target reward function based on whether the driving task is successfully completed: Among them, flag success is the flag for reaching the end of the mission scenario; if the agent successfully reaches the target location, the reward value is a positive number, indicating that the mission is completed; if the agent fails to reach the target location, the reward value is zero, indicating that the mission is not completed or failed; Step 33: Guide the design of reward and punishment function: Guide the reward and punishment function r guidance The road center distance and heading deviation in the path constraint are used as rewards and penalties, including the road center distance reward and penalty function r d and heading deflection angle reward and penalty function r θ : r guidance =r d +r θ +r n ; Road center distance reward and penalty function r d Guiding the vehicle to its target driving position: When the distance between the center of the road and lat Less than the limit distance D limit When the vehicle is farther from the center of the lane, the penalty value is greater; when the distance from the center of the road is D lat Greater than the limit distance D limit When , the penalty value is always -1; The local planning path is close to the reward and punishment function r θ Guiding the vehicle's target driving direction: Among them, θ error is the deviation between the local planning path heading vector and the current agent heading vector; When the heading deviation angle is less than or equal to 1°, the vehicle moves in the correct direction and is given a reward of 1; when the heading deviation angle is greater than 1° and within the limit range θ limit The more the vehicle deviates from the correct course, the lower the reward value will be. When the course deviation angle is greater than the limit range, a penalty of -1 will be given. Target guidance function r n Guide the vehicle to move towards the target point: Assume that the current road center point is W N , then D x2 The distance between the vehicle and the center point of the road W N+2 The distance, D x3 The distance between the vehicle and the center point of the road W N+3 distance; Step 34: Design of speed reward and penalty function: The speed reward and penalty function guides the vehicle to gradually approach or maintain the target speed according to the difference between the current vehicle speed and the target speed: Among them, r v is the speed reward and penalty value; α is the speeding reward and penalty coefficient, β is the deceleration reward and penalty coefficient; v min and v max are the lower and upper limits of the target speed range respectively; If the current vehicle speed v is within the target speed range, that is, v min ≤v≤v max , a reward of 10 is given; if the current vehicle speed v exceeds the target speed range, that is, v <v min or v>v max , then give corresponding punishment or reward according to the degree of deviation of vehicle speed from the target speed; Step 35: The total reward R of the vehicle agent is the weighted sum of the collision reward, target reward, guidance reward, and speed reward: R=β1r collision +β2r target +β3r guiddnce +r v ; Among them, β1 is the collision reward coefficient, β2 is the target reward coefficient, and β3 is the guidance reward coefficient.
4. The multimodal reinforcement learning vehicle decision planning method with compensation feedback according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 41: Select a deep learning architecture: Use the SAC algorithm as the optimization strategy. The SAC algorithm generates a corresponding behavior strategy based on the current state and environmental feedback, and optimizes the strategy based on the reward signal. The SAC algorithm combines the Actor and the Critic and introduces entropy maximization. The steps of the SAC algorithm are as follows: Use Actor to generate actions, the goal is to maximize the expected cumulative reward, and define the following objective function: in, is the expected value, which represents the expectation of the empirical data distribution D (i.e., the distribution of state-action pairs), γ is the discount factor, r(s, a) is the immediate reward obtained after selecting action a in state s, α is the entropy coefficient, is the entropy of policy π, πθ is the parameterized policy; Given a state s t , through the Actor network, we get all the action probabilities π(a|s t ), according to the probability sampling, action a is obtained t , a t Input into the environment to get s t+1 and r t , gain experience(s t , a t ,s t+1 , r t ), put it into the experience pool; Sample data from the experience pool (s t , a t ,s t+1 , r t ), update the network parameter ω; t Q(s t , a t ) value as s t The predictive value estimate of s is obtained according to the optimal Bellman equation t The true value of the state is estimated to be E π [r t +γQ(s t+1 ,π(s t+1 ))|s t ], where E π is s under strategy π t The expected value of the state, r t is the immediate reward obtained at time step t; the Q Critic network is trained using the mean square loss function as the loss, and the loss function is defined as: Among them, E B is the expectation of the loss value, θ targ are the parameters of the target network; Sample data from the experience pool (s t , a t ,s t+1 , r t ), update the network parameter θ; the true value of the output of the V Critic network is r t +γV(s t+1 )-αlogπ(a′ t |s t ), where V(s t+1 ) is in state s t+1 The value estimate under a′ t is the policy π in state s t The actions sampled in ; the loss of the V Critic network is calculated based on the true value: The loss function for gradient descent training is defined as the TD error to measure the correction amplitude of the algorithm. The action a selected by the strategy is calculated in the form of TD error. t To conduct an assessment: δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t ); Among them, δ t is the TD error at time step t, Q is the state value of the Critic; Step 42: Define the state space and action space: Considering the position, velocity, and acceleration of the ego vehicle and surrounding vehicles, the state space is represented by s = (s ego ,s surroundings ), the semantic segmentation image in the state space is represented as s image , the point cloud in the state space is represented as s point , considering the destination and route information of the current vehicle, represented as s task =(x goal , route); get the state space representation S = (s, s image ,s point ,s task ); The action space is represented as A = (a, θ), where a is the vehicle acceleration, a∈[a min , a max ]; θ is the vehicle steering angle, θ∈[-θ max ,θ max ].
5. The multimodal reinforcement learning vehicle decision planning method with compensation feedback according to claim 1, characterized in that: The specific process of step 5 is as follows: The trajectory planner uses Lattice Planner. The trajectory planner predicts the trajectory points in the future, calculates the expected action size in the future through the predicted trajectory points, and then connects the points to form a lateral trajectory based on the existing sampling points. The method of connecting according to the fifth-order polynomial after given boundary conditions is adopted. The formula is as follows: d(t)=a5t 5 +a4t 4 +a3t 3 +a2t 2 +a1t+a0; Among them, a i (i=1,2,...,5) are coefficients of the polynomial, and t is time; The compensation feedback mechanism is introduced to calculate the difference between the future expected action predicted by the local trajectory planner and the action output by the vehicle agent during the actual driving process to obtain the compensation value. The calculation formula of the compensation value is Δ=λ×(a t -a n ), where Δ represents the compensation value, λ is the compensation coefficient, and a t is the expected action size in the future, a n is the action size of the current network output; After the compensation value is calculated, it is returned to the SAC network as loss information:
Citation Information
Cited By
Implicit state representation method based on signal-free intersection automatic driving system
CN119808880A
Complex environment-oriented self-adaptive multi-mode automatic driving vehicle taking system
CN119928915A
An adaptive multimodal autonomous vehicle pick-up system for complex environments
CN119928915B
Dynamic walking control method for indicating machine based on traffic lane management and control
CN119960465A
An indication machine dynamic walking control method based on traffic lane control
CN119960465B
