Reinforcement learning lane-changing behavior planning method and system based on imitation learning initialization

Abstract

The present invention proposes a reinforcement learning lane changing behavior planning method and system based on imitation learning initialization. The method maps the deformed posture and fuzzy speed of surrounding vehicles onto a grid map through gradient colors, represents traffic rules and road conditions with clear information using vectors, and connects the two to input into the imitation learning DAgger network, so that the autonomous driving vehicle learns to obey traffic rules and basic lane changing decisions. The imitation learning lane changing mechanism is then optimized by reinforcement learning to adapt to extreme and unknown traffic scenarios that are not included in the imitation learning data set. The potential lane changing driving mechanism can be learned from the driving mechanism and environment of vehicle interactions, simplifying the manual driving modeling, having good adaptability to the dynamic changes of the lane changing scene, and adopting a dynamic mixed reward mechanism for the lane changing task. A reinforcement learning initialization method based on imitation learning is introduced to solve the problem of low learning efficiency of reinforcement learning under sparse rewards, and only a small amount of scene data is required.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving behavior planning, and specifically relates to a reinforcement learning lane changing behavior planning method and system based on imitation learning initialization. Background Art

[0002] In traditional autonomous driving systems, autonomous vehicles receive information about the surrounding environment analyzed by the perception module and make reasonable behavior plans, such as turning left or right when changing lanes, and then the motion planning generates the corresponding motion trajectory to complete specific driving tasks. In complex and highly dynamic traffic environments, autonomous vehicles need to consider traffic rules, surrounding traffic participants and road conditions to make safe and reliable decisions, which has always been a challenge.

[0003] The current decision models are mainly divided into three categories: heuristic rule enumeration decision models, optimization theory-based decision models, and learning-based decision models. Heuristic-based decision models seem to be able to describe the human-like decision process well, but due to the dynamics and uncertainty of traffic scenes, it is difficult to create a rule base that comprehensively covers all traffic scenes. The decision model based on optimization theory considers the vehicle dynamics model and optimization constraints, and generates a comfortable and safe drivable trajectory by optimizing the objective function. However, it requires accurate modeling of traffic scenes and vehicle models, and the optimization time is short. It is impossible to make a more long-term optimal global decision in the face of complex and long visual navigation tasks. In the decision model based on learning, imitation learning learns safe and reliable driving strategies based on high-quality and large-scale human expert decision data; reinforcement learning enables autonomous vehicles to learn strategies to maximize rewards or achieve specific goals in the process of interacting with the environment, and show better generalization for unknown scenes. However, the original reinforcement learning-based method randomly initializes the strategy in the early stage of training, making it difficult for the strategy to obtain favorable rewards in the initial environment. In addition, the huge search space and sparse reward space greatly slow down the learning speed. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a reinforcement learning lane-changing behavior planning method based on imitation learning initialization. By learning from the demonstration of experts, the inefficiency of reinforcement learning initialization is alleviated, and the behavior planning ability consistent with the expert level is quickly obtained. Compared with the method using only imitation learning, the combination of reinforcement learning and imitation learning also retains the ability of reinforcement learning to adapt in the interaction with the environment to deal with unpredictable and extreme situations not covered by the demonstration data; it is completed under the framework of hierarchical behavior planning and motion planning, and the behavior planning is designed to achieve better decision-making. As for the motion planning part, advanced algorithms are used to ensure the overall performance of the framework.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a reinforcement learning lane-changing behavior planning method based on imitation learning initialization, comprising the following steps:

[0006] S1, construct a grid map M that introduces the blurred speed and deformed posture of surrounding vehicles t , traffic rules and road condition information are combined to use vector V t express;

[0007] S2, using imitation learning to initialize lane-changing decision behavior planning π b , the grid map M t and the vector V t Connect the input to the imitation learning DAgger network and output the high-level decision of the vehicle;

[0008] S3, initialize the Actor network in the PPO algorithm, collect multiple sets of driving trajectory data through interaction through SUMO simulation scenes, and under the action of the hybrid reward mechanism, the PPO algorithm optimizes the lane change decision behavior planning to obtain the optimized high-level decision b k ;

[0009] The hybrid reward mechanism automatically triggers the corresponding reward function according to the selected strategy; specifically, based on the lane change driving task, the state of the autonomous driving vehicle and traffic conditions at time t is defined as state s t , by the behavior planning π according to s t , get the decision command u t , reinforcement learning lane change decision behavior planning takes u t , transition to state s at time t+1 t+1 , and get reward r t , which is accumulated to the lane-changing decision behavior planning to complete a specific task, and the behavior planning π must adjust its own network parameters θ to obtain the long-term benefit Q(s t , b t ),

[0010]

[0011] Based on the reinforcement learning PPO algorithm, behavior planning π is performed, and the state information of the reinforcement learning lane change decision behavior planning is input, and the output b k ∈{left_change, keep, right_change, speedup, speeddown} is defined as a high-level decision if b k If there is a conflict with the road information, the behavior planning π will make a new decision; the timeline is divided into multiple fixed time periods, and in each time period [t k , t k+1 ], b kwill remain unchanged, the time span is longer than the control command, and [t k , t k+1 ] The control command cumulative reward in the time period is replaced Decision-making rewards.

[0012] In S1, a grid map M is constructed that introduces the blurred speed and deformed posture of surrounding vehicles. t The details are as follows:

[0013] The position information of surrounding traffic participants is: the information of multiple distance sensors is integrated into a grid map of the position information of surrounding traffic participants centered on the autonomous driving vehicle. In each grid, if there is a traffic participant object, its value is 1, otherwise it is 0;

[0014] The speed information of surrounding traffic participants is: referring to the real-time speed and ideal linear speed of the autonomous driving vehicle, the speed of surrounding traffic vehicles is divided into 7 fuzzy sets: {NB, NM, NS, Z, PS, PM, PB}, and the 7 fuzzy sets are displayed in gradient colors and projected onto the grid map M. t , uniformly represents the position information and speed information of surrounding traffic vehicles.

[0015] When surrounding vehicles are introduced in S1, all vehicles within the 80-meter sensor range are considered; the grid map is improved and the curved grid map is uniformly transformed into a straight road.

[0016] S1 extracts the road contour and road direction from the road contour and information layer of the high-precision map. Based on the line signs and the current positioning information of the autonomous driving vehicle, the autonomous driving vehicle learns to track the lane lines and then extracts the current road information from the high-precision map.

[0017] The high-level decisions output by S2 include left lane change behavior, right lane change behavior, acceleration behavior, constant speed behavior or deceleration behavior.

[0018] The PPO algorithm includes an Actor network and a Critic network. The Actor network and the Critic network have the same structure except for the different outputs. The input is a raster image of the blurred speed and deformed posture of the surrounding traffic participants, and a vector of traffic rules and road directions. The raster image is processed by three convolutional layers and then flattened by a fully connected network. The vectors of traffic rules and road directions are processed by fully connected networks respectively. The Actor network and the Critic network are connected and output by two fully connected networks. The Actor network is finally normalized by the SoftMax function to output high-level decision-making behavior. The last layer of the Critic network directly outputs a value.

[0019] On the other hand, the present invention provides a reinforcement learning lane-changing behavior planning method system based on imitation learning initialization, comprising a preprocessing module, an initialization module and an optimization module, wherein a reward unit is introduced into the optimization module;

[0020] The preprocessing module is used to construct a grid map M that introduces the blurred speed and deformed posture of surrounding vehicles. t , traffic rules and road condition information are combined to use vector V t express;

[0021] The initialization module is used to initialize the lane-changing decision behavior planning π using imitation learning b , the grid map M t and the vector V t Connect the input to the imitation learning DAgger network and output the high-level decision of the vehicle;

[0022] The optimization module is used to initialize the Actor network in the PPO algorithm, and collect multiple sets of driving trajectory data based on the SUMO simulation scenario. Under the action of the hybrid reward mechanism, the PPO algorithm optimizes the lane change decision behavior planning and obtains the optimized high-level decision b k ;

[0023] The hybrid reward mechanism in the reward unit automatically triggers the corresponding reward function according to the selected strategy; specifically, based on the lane change driving task, the state of the autonomous driving vehicle and traffic conditions at time t is defined as state s t , the lane-changing strategy π is based on s t , get the decision command u t , reinforcement learning lane change decision behavior planning takes u t , transition to state s at time t+1 t+1 , and get reward r t , which is accumulated to complete the specific task of lane-changing decision-making behavior planning, and the lane-changing decision-making behavior planning must adjust the lane-changing strategy π to obtain long-term benefits R t

[0024]

[0025] Based on the reinforcement learning PPO algorithm, behavior planning π is performed, and the state information of the reinforcement learning lane change decision behavior planning is input, and the output b k ∈{left_change, keep, right_change, speedup, speeddown} is defined as a high-level decision if b k If there is a conflict with the road information, the behavior planning π will make a new decision; the timeline is divided into multiple fixed time periods, and in each time period [t k , t k+1 ], bk will remain unchanged, the time span is longer than the control command, and [t k , t k+1 ] The control command cumulative reward in the time period is replaced Decision-making rewards.

[0026] An autonomous driving vehicle is also provided, which performs lane change decision behavior planning based on the method of the present invention.

[0027] The present invention also provides a computer device, including a processor and a memory, the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and when the processor executes the computer executable program, it can implement the reinforcement learning lane changing behavior planning method based on imitation learning initialization described in the present invention.

[0028] At the same time, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning lane changing behavior planning method based on imitation learning initialization described in the present invention can be implemented.

[0029] Compared with the prior art, the present invention has at least the following beneficial effects:

[0030] Reinforcement learning methods are used to train lane changes to learn behavior planning in a hierarchical behavior planning and motion planning framework, which can take into account traffic rules, surrounding traffic participants and road conditions, and display tactical performance decisions; imitation learning DAgger initializes lane change behavior planning, so that it learns to obey traffic rules and basic decisions, such as obeying traffic lights, lane tracking, etc. Reinforcement learning PPO is then used to fine-tune the lane-changing behavior planning, so that it can be trained on data that is difficult to simulate in the real world and learn more sophisticated decisions; the constructed hybrid reward mechanism effectively evaluates the rewards of state-action pairs at different levels of behavior planning and motion planning. According to the selected strategy, the algorithm will automatically trigger the corresponding reward function, which is conducive to the diversity and dynamics of the reward; at the same time, positive rewards that are conducive to options and actions will appear only when and only when the entire task and the sub-goals in the hierarchical structure are completed. In order to prevent the policy network from being lazy, sometimes blindly giving positive rewards will suppress the exploration desire of the policy network; the performance of different levels can be effectively evaluated, and the cost allocation of motion planning is used instead of the reward of behavior planning to ensure that the final learned strategy is the optimal solution of HBMP; the reinforcement learning PPO network uses a hybrid reward mechanism to optimize the training of the lane-changing behavior planning after imitation learning initialization, so that the present invention has better generalization for unknown and extreme scenarios. In addition, the state space of fuzzy technology is introduced. By fuzzy processing the speed of surrounding vehicles and projecting the fuzzy speed and vehicle posture onto a grid map, the learning speed of lane change decision behavior planning is accelerated and the performance of lane change decision behavior planning is significantly improved. Imitation learning is used on the simulation platform SUMO to train autonomous driving vehicles to learn to obey traffic rules and basic lane change decisions, and reinforcement learning is used to train complex lane change decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of a reinforcement learning lane-changing behavior planning method based on imitation learning initialization.

[0032] Figure 2 It is a grid map that introduces blurred velocities and deformed poses of surrounding vehicles.

[0033] Figure 3 Schematic diagram of the change in the speed of the autonomous driving vehicle over time after the fuzzy speed of the surrounding vehicles is introduced.

[0034] Figure 4 Schematic diagram of the change of rewards over time after deforming the surrounding vehicle poses.

[0035] Figure 5 Schematic diagram of the Actor-Critic network structure of the PPO (Proximal Policy Optimization) algorithm.

[0036] Figure 6Schematic diagram of the change in collision rate over time of the lane changing behavior mechanism after imitation learning initialization.

[0037] Figure 7 Schematic diagram of the hierarchical behavior planning and motion planning framework.

[0038] Figure 8 Schematic diagram of the mixed reward mechanism.

[0039] Fig. 9 Schematic diagram of how rewards change with the number of training rounds when deformation pose, blur speed, and hybrid reward mechanisms are added to the reinforcement learning optimization training process. DETAILED DESCRIPTION

[0040] The following is a detailed description of the exemplary embodiments of the present application in conjunction with the accompanying drawings and specific implementations, including various details of the embodiments of the present application to facilitate understanding. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, the modifications of various equivalent forms of the present invention by those skilled in the art all fall within the scope defined by the claims attached to the present application.

[0041] Figure 1 This is a flow chart of the reinforcement learning lane-changing behavior planning method based on imitation learning initialization of the present invention. The present invention is suitable for complex and highly dynamic traffic environments. The deformed postures and fuzzy speeds of surrounding vehicles are mapped to a grid map through gradient colors, and traffic rules and road conditions with clear information are represented by vectors. The two are connected and input into the imitation learning DAgger network, so that the autonomous driving vehicle learns to obey traffic rules and basic lane-changing decisions. Reinforcement learning is then used to optimize the imitation learning lane-changing decision-making behavior planning to adapt it to extreme and unknown traffic scenarios that are not in the imitation learning data set. The above training processes are all carried out in the simulation platform SUMO. Finally, the optimized lane-changing mechanism is tested on another simulation platform CARLA to verify the cross-platform robustness of the present invention.

[0042] The method of the present invention comprises the following steps:

[0043] S1: Construct a grid map that introduces the blurred speed and deformed posture of surrounding vehicles, and use vectors to uniformly represent traffic rules and road conditions information, as follows:

[0044] In order to describe the surrounding traffic participants, the present invention takes into account all vehicles within the sensor range of 80 meters in front and behind. They are described by the following characteristics:

[0045] 1) The position information of surrounding traffic participants. The information of multiple distance sensors is integrated into a grid map of the position information of surrounding traffic participants centered on the autonomous driving vehicle. In each grid, if there is a traffic participant object, its value is 1, otherwise it is 0. In addition, the present invention improves the grid map and uniformly transforms the curved grid map into a straight road. Then, the straight road data can be uniformly used for training. Figure 2 .

[0046] 2) Speed ​​information of surrounding traffic participants: In actual traffic driving, humans are actually unable to accurately perceive the speed information of surrounding vehicles. They only know vague concepts such as fast, slow, and about the same, and can make decisions to change lanes. Referring to the real-time speed and ideal linear speed of the autonomous driving vehicle, the speed of surrounding traffic vehicles is divided into 7 fuzzy sets: {NB, NM, NS, Z, PS, PM, PB}, and the 7 fuzzy sets are displayed in gradient colors and projected onto the grid map M. t , uniformly represents the position information and speed information of surrounding traffic vehicles.

[0047] The road contour and road direction are extracted from the road contour and information layer of the high-precision map. Based on the line signs and the current positioning information of the autonomous driving vehicle, the current road contour is obtained from the road contour layer of the high-precision map, so that the autonomous driving vehicle can learn to track the lane lines and then extract the current road information from the high-precision map.

[0048] For fixed and clear traffic rules and road conditions provided by high-precision maps, the road conditions and traffic rules are represented by vectors V t Indicates, as follows:

[0049] 1) The existence and direction of the left lane, the current lane, and the right lane, whether the left lane can turn left, whether the left lane can go straight, whether the current lane exists, whether the right lane can turn right, and whether the right lane can go straight are defined as [e l , d l , d c , e r , d r ], using vectors to encode lane existence status and direction to ensure that autonomous vehicles make correct lane changes.

[0050] 2) The state of the traffic light, green light, yellow light, and red light are defined as [l g , l y , l r ] to ensure that autonomous vehicles obey traffic light rules.

[0051] like Figure 3As shown, compared with simply considering the posture information of the surrounding traffic participants, adding the fuzzy initialization speed reduces the number of lane changes while ensuring the increase in the average driving speed. The reason is that when only the posture information is considered, regardless of the speed of the front vehicle, the behavioral decision will execute the same tactic. Even if the speed of the front vehicle is much higher than the ideal linear speed of the autonomous driving vehicle, it is very likely to execute a lane change decision, which is unnecessary. Introducing the speed of the surrounding traffic participants can avoid this situation, but this will expand the state space. Because once the speed of the front vehicle exceeds the ideal speed of the autonomous driving vehicle, it should be unified into one situation, that is, there is no need to change lanes. For this reason, the present invention fuzzifies the speed of the surrounding traffic vehicles, simplifies the state space, and accelerates the learning rate of the behavioral strategy.

[0052] The present invention creates a training scenario with three straight lanes, i.e., a road without turns, and the training behavior planning π d′ ; Create a circular three-lane, that is, ensure that the lane is always in an arc shape. d′ Directly applied to the curved lane, Figure 4 It is found that the behavior decision performance is still good, and is basically the same as the performance on the straight lane. The reason is that the present invention cuts the grid map along the curve direction and converts it into a straight road grid map instead of a flat grid map, which shows the robustness of the state space.

[0053] S2 builds the network structure of the reinforcement learning algorithm PPO, as follows: The PPO algorithm is a very popular single-agent reinforcement learning algorithm that uses the classic Actor-Critic architecture; the Actor and the Critic have the same input state and network structure, but the difference is that the Actor uses a policy function to interact with the environment and generate discrete high-level lane-changing decisions. The Critic uses a value function to evaluate the value of the lane-changing decision and guide the Actor's next stage of action. The network input is a deformed grid map M that integrates the location and speed information of surrounding traffic participants. t and a vector V containing traffic rules and road information t .

[0054] The network structure of the PPO algorithm is as follows Figure 5 Traffic rules and road information vectors are processed by two fully connected layers and combined with M t The dimensions remain the same. The deformed grid image is processed by three convolutional layers and one fully connected layer, which are finally connected and input into the final fully connected layer. Softmax is used after the output layer. Except that the last layer does not have Softmax assigned, the Critic has the same architecture. The Actor network is finally normalized by the SoftMax function to output high-level decision-making behavior. The last layer of the Critic network directly outputs a value.

[0055] Specifically, the input is a grid image (size 3*84*84) of the blurred speed and deformed posture of the surrounding traffic participants, and a vector (size 1*8) of traffic rules and road direction; the grid image is processed by three convolutional layers of 2*84*84, 32*39*39, and 32*7*7, and then flattened by a 128-bit fully connected network; the traffic rules and road direction vectors are processed by 32- and 64-bit fully connected networks; it can be seen that the two are connected and output by two 128- and 64-bit fully connected networks; the Actor network is finally normalized by the SoftMax function to output high-level decision-making behavior; the last layer of the Critic network directly outputs a value.

[0056] S3: Imitation learning DAgger initializes lane change decision behavior planning, as follows:

[0057] The training strategy of the present invention is to first use imitation learning to initialize the lane-changing decision behavior planning π b , so that it can learn to obey traffic rules and basic decisions, such as obeying traffic lights, lane tracking, etc. Then use reinforcement learning PPO to fine-tune the lane change decision behavior planning, so that it can be trained on data that is difficult to simulate in the real world and learn more sophisticated decisions. The present invention uses the event-level simulator SUMO, which does not require motion planning, but only needs to provide decision results, i.e., left turn, right turn, straight, etc., and will automatically complete the lane change in a shorter time. The present invention uses DAgger to train the loss of imitating the learning lane change decision behavior planning. In formula (1), b k It is the lane-changing decision behavior provided by expert data.

[0058]

[0059] at last, Figure 6 The collision rate of the lane-changing behavior mechanism changes over time after the imitation learning is initialized. In the early stage of training, the probability α selects the expert decision, resulting in an extremely high reward. As the number of training rounds increases, α decreases, the model makes behavioral decisions, and the reward decreases. In the later stage of training, the reward gradually stabilizes to the initial expert decision level, which illustrates the convergence of the imitation learning pre-training. The present invention uses the trained model to initialize the participant network in reinforcement learning. Figure 6 In the present invention, the policy network initialized by imitation learning is compared with the policy network initialized by random. It is found that the behavior planner initialized by imitation learning has a rapid reduction in collisions at the beginning of training and is always lower than the behavior planner initialized by random.

[0060] S4: Construct a hybrid reward mechanism, as follows:

[0061] Based on the lane-changing driving task, the state of the autonomous driving vehicle and traffic conditions at time t is defined as state s t , the lane-changing strategy π is based on s t , get the decision command u t , as shown in formula (2),

[0062] u t =π(s t ) (2)

[0063] Reinforcement learning lane-changing decision-making behavior planning takes u t , transition to state s at time t+1 t+1 , and get reward r t , which is accumulated to complete the specific task of lane-changing decision-making behavior planning, and the lane-changing decision-making behavior planning must adjust the lane-changing strategy π to obtain long-term benefits R t , as shown in formula (3),

[0064]

[0065] Among them, ∑ t:0...T-1 γ t r t The part records the cumulative reward of the state trajectory of the reinforcement learning lane change decision behavior planning in time [0, T-1), γ T r T represents the reward at the end time T, where γ is the decay factor. Generally speaking, r T The penalty is based on whether the reinforcement learning lane change decision behavior planning reaches the destination, r t The penalty is based on whether the lane-changing decision behavior planning based on reinforcement learning violates traffic rules, consumes time, or improves performance during trajectory driving. As can be seen from formula (3), this reward design obviously only provides extremely sparse guidance at the end time, resulting in similar results after multiple trainings, resulting in low learning efficiency of behavioral strategies and only limited tactical decisions. Even in traditional methods based on explicit semantic objects, search is not very effective.

[0066] To solve this problem, the hierarchical modeling of driving tasks is introduced into motion planning, and only high-level decisions are made to limit the search space. The reinforcement learning PPO algorithm is used as the behavior planning π in the hierarchical behavior planning and motion planning framework (HBMP), and the state information of the reinforcement learning lane change decision behavior planning is input, and the output b k ∈{left_change, keep, right_change, speedup, speeddown} is defined as a high-level behavior if b k If there is a conflict with the road information, the behavior planning π will make a new decision. The timeline is divided into multiple fixed time periods, and in each time period [tk , t k+1 ], b k will remain unchanged, the time span is longer than the control command, the present invention will [t k , t k+1 ] The control command cumulative reward in the time period is replaced The decision-making rewards at the same time will make the rewards dynamic and diversified, thus solving the sparse reward problem. Figure 7 As shown, the present invention then divides formula (3) into:

[0067]

[0068] For greater clarity, the present invention defines a policy network b k =π b (s t ), then simplify formula (4) as follows:

[0069]

[0070] Compared with the original formula (3) without explicit hierarchical behavior, the autonomous driving HBMP formula (5) is similar to factorization. The behavior planning accepts the state information of the autonomous driving vehicle, outputs high-level decisions, and then performs motion planning of the trajectory, which reduces the search space and is simpler than planning the motion and target lane at the same time.

[0071] The present invention proposes a hybrid reward mechanism to effectively evaluate the rewards of different levels of state-action pairs in behavior planning and motion planning. 1) According to the selected strategy, the algorithm will automatically trigger the corresponding reward function, which is conducive to the diversity and dynamics of rewards; 2) At the same time, positive rewards for options and actions will appear only when and only when the entire task and sub-goals in the hierarchical structure are completed. In order to prevent the strategy network from being lazy, sometimes blindly giving positive rewards will suppress the exploration desire of the strategy network. Figure 8 Demonstrated a hybrid reward mechanism.

[0072] from Fig. 9 It can be seen that the present invention trains a strategy network π that directly maps from sensory data to low-dimensional control command speed d′ , using formula (3), it is found that due to sparsity, the policy network ensures security, but only learns path tracking, and the tactical decision of the intermediate planner is not shown. Using formula (5) to train the hierarchical behavior planning π in HBMP d ,from Fig. 9 It can be seen that π d The rising speed and upper limit are both higher than π d′ ,This is because the hybrid reward mechanism makes the rewards diverse and dynamic,,showing tactical decision making.

[0073] S5: Reinforcement learning PPO uses a mixed reward mechanism to optimize the training of imitation learning after initialization of lane change decision-making behavior planning. The specific details are as follows:

[0074] On SUMO, the lane-changing decision behavior planning is initialized using the imitation learning DAgger algorithm, and then optimized using the reinforcement learning PPO algorithm. The present invention compares the performance of the HBMP structure with one of the other two benchmark decision models: the IDM-based decision model and the MPC-based decision model. The IDM-based agent uses the rule-based lane-changing decision provided by the SUMO environment. The present invention reasonably sets the internal parameters of the IDM model to avoid harsh rules (such as never overtaking from the right and being too humble to other vehicles) to ensure that there are no internal constraints that prevent the autonomous driving vehicle from reaching the ideal speed. The MPC-based decision model considers the longitudinal braking safety distance and the lane-changing safety distance, and establishes a safe vehicle distance model to judge the vehicle's driving state, which is used as an indicator of the lane-changing warning decision strategy. Finally, the HBMP structure selects from the high-level behaviors: lane keeping, left lane change, and right lane change. In order to ensure that the two typical decision models are trained in the same way as HBMP, the present invention uses the same reward function and the same input state. During the evaluation process, all decision models are rewarded according to the same criteria. They only differ in the action / option selection strategy. As shown below, it can be seen that the collision rate and average driving speed of the HBMP structure are significantly better than those of the other two decision models.

[0075]

[0076] In order to simulate real-world scenarios, the present invention built a town with various traffic scenarios on CARLA, including intersections with traffic lights, multiple lanes, pedestrians, etc. In DAgger, the probability of expert decision is α, and the probability of model decision is 1-α. At the beginning, α=0.98, and after 4000 trainings, α gradually becomes 0. The details of the PPO algorithm are as follows:

[0077]

[0078] The present invention tests the model of the present invention on CARLA. To correspond to the traffic scenario of SUMO, the present invention selects the map Town3 test model, which has richer road contours, such as multiple lanes. For each round of testing, the starting position and target end point of the autonomous driving vehicle are randomly generated on the lane. The A* global planning algorithm is used to plan the global route as a reference. Once the target is reached or the traffic rules are violated, a round of testing ends.

[0079] Based on the above steps, the specific parameters of the simulation scenario are as follows:

[0080]

[0081] The final test results are shown in the following table:

[0082]

[0083] The efficient reinforcement learning lane-changing decision-making behavior planning algorithm based on imitation learning initialization of the present invention is specifically as follows:

[0084]

[0085]

[0086] By learning from expert demonstrations, the present invention alleviates the inefficiency of reinforcement learning initialization and quickly obtains behavior planning capabilities consistent with expert levels. Compared with the method that only uses imitation learning, the combination of reinforcement learning and imitation learning also retains the ability of reinforcement learning to adapt in the interaction with the environment to cope with unpredictable and extreme situations not covered by demonstration data.

[0087] According to the characteristics of the autonomous driving planning task, a number of improvements are proposed. The present invention is completed under the hierarchical behavior planning and motion planning framework (HBMP), and behavior planning is designed to achieve better decision-making. As for the motion planning part, advanced algorithms are used to ensure the overall performance of the framework. The autonomous driving car first performs behavior planning for multi-lane traffic scenarios, such as left turn, straight driving, etc., and then the motion planning will use the decision to design the optimal path result. The comparative experimental results on SUMO prove the effectiveness of the method of the present invention. In addition, the SUMO lane changing mechanism without fine-tuning is tested on CARLA, which verifies the robustness of the present invention on different platforms.

[0088] In one possible scenario, an autonomous driving vehicle can perform lane changing decision behavior planning based on the reinforcement learning lane changing decision behavior planning method described in the present invention.

[0089] The present invention can also provide a reinforcement learning lane-changing behavior planning method system based on imitation learning initialization, comprising a preprocessing module, an initialization module and an optimization module, wherein a reward unit is introduced into the optimization module;

[0090] The preprocessing module is used to construct a grid map M that introduces the blurred speed and deformed posture of surrounding vehicles. t , traffic rules and road condition information are combined to use vector V t express;

[0091] The initialization module is used to initialize the lane-changing decision behavior planning π using imitation learning b , the grid map M t and the vector V tConnect the input to the imitation learning DAgger network and output the high-level decision of the vehicle;

[0092] The optimization module is used to initialize the Actor network in the PPO algorithm, and collect multiple sets of driving trajectory data based on the SUMO simulation scenario. Under the action of the hybrid reward mechanism, the PPO algorithm optimizes the lane change decision behavior planning and obtains the optimized high-level decision b k ;

[0093] The hybrid reward mechanism in the reward unit automatically triggers the corresponding reward function according to the selected strategy; specifically, based on the lane change driving task, the state of the autonomous driving vehicle and traffic conditions at time t is defined as state s t , the lane-changing strategy π is based on s t , get the decision command u t , reinforcement learning lane change decision behavior planning takes u t , transition to state s at time t+1 t+1 , and get reward r t , which is accumulated to complete the specific task of lane-changing decision-making behavior planning, and the lane-changing decision-making behavior planning must adjust the lane-changing strategy π to obtain long-term benefits R t

[0094]

[0095] Based on the reinforcement learning PPO algorithm, behavior planning π is performed, and the state information of the reinforcement learning lane change decision behavior planning is input, and the output b k ∈{left_change, keep, right_change, speedup, speeddown} is defined as a high-level decision if b k If there is a conflict with the road information, the behavior planning π will make a new decision; the timeline is divided into multiple fixed time periods, and in each time period [t k , t k+1 ], b k will remain unchanged, the time span is longer than the control command, and [t k , t k+1 ] The control command cumulative reward in the time period is replaced Decision-making rewards.

[0096] The present invention can also provide a computer device, including a processor and a memory, the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and when the processor executes part or all of the computer executable program, it can implement the reinforcement learning lane changing behavior planning method based on imitation learning initialization described in the present invention.

[0097] On the other hand, the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning lane changing behavior planning method based on imitation learning initialization described in the present invention can be implemented.

[0098] The computer device may be a vehicle-mounted computer, a notebook computer, a desktop computer or a workstation.

[0099] The processor may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or an off-the-shelf field programmable gate array (FPGA).

[0100] The memory described in the present invention may be an internal storage unit of a vehicle-mounted computer, a notebook computer, a desktop computer or a workstation, such as a memory or a hard disk; or an external storage unit, such as a mobile hard disk or a flash memory card.

[0101] Computer-readable storage media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media may include: read-only memory (ROM), random access memory (RAM), solid-state drive (SSD) or optical disk, etc. Among them, random access memory may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

Claims

1. A reinforcement learning lane-changing behavior planning method based on imitation learning initialization, characterized in that: The following steps are involved: S1, construct a grid map M that introduces the blurred speed and deformed posture of surrounding vehicles t , traffic rules and road condition information are combined to use vector V t express; S2, using imitation learning to initialize lane-changing decision behavior planning π b , the grid map M t and the vector V t Connect the input to the imitation learning DAgger network and output the high-level decision of the vehicle; S3, initialize the Actor network in the PPO algorithm, collect multiple sets of driving trajectory data through interaction through SUMO simulation scenes, and under the action of the hybrid reward mechanism, the PPO algorithm optimizes the lane change decision behavior planning to obtain the optimized high-level decision b k ; The hybrid reward mechanism automatically triggers the corresponding reward function according to the selected strategy; specifically, based on the lane change driving task, the state of the autonomous driving vehicle and traffic conditions at time t is defined as state s t , by the behavior planning π according to s t , get the decision command u t , reinforcement learning lane change decision behavior planning takes u t , transition to state s at time t+1 t+1 , and get reward r t , which is accumulated to the lane-changing decision behavior planning to complete a specific task, and the behavior planning π must adjust its own network parameters θ to obtain the long-term benefit Q(s t , b t ), Based on the reinforcement learning PPO algorithm, behavior planning π is performed, and the state information of the reinforcement learning lane change decision behavior planning is input, and the output b k ∈{left_change, keep, right_change, speedup, speeddown} is defined as a high-level decision if b k If there is a conflict with the road information, the behavior planning π will make a new decision; the timeline is divided into multiple fixed time periods, and in each time period [t k , t k+1 ], b k will remain unchanged, the time span is longer than the control command, and [t k , t k+1 ] The control command cumulative reward in the time period is replaced Decision-making rewards.

2. The method according to claim 1, characterized in that: In S1, a grid map M is constructed that introduces the blurred speed and deformed posture of surrounding vehicles. t The details are as follows: The position information of surrounding traffic participants is: the information of multiple distance sensors is integrated into a grid map of the position information of surrounding traffic participants centered on the autonomous driving vehicle. In each grid, if there is a traffic participant object, its value is 1, otherwise it is 0; The speed information of surrounding traffic participants is: referring to the real-time speed and ideal linear speed of the autonomous driving vehicle, the speed of surrounding traffic vehicles is divided into 7 fuzzy sets: {NB, NM, NS, Z, PS, PM, PB}, and the 7 fuzzy sets are displayed in gradient colors and projected onto the grid map M. t , uniformly represents the position information and speed information of surrounding traffic vehicles.

3. The method according to claim 2, characterized in that When surrounding vehicles are introduced in S1, all vehicles within the 80-meter sensor range are considered; the grid map is improved and the curved grid map is uniformly transformed into a straight road.

4. The method according to claim 1, characterized in that S1 extracts the road contour and road direction from the road contour and information layer of the high-precision map. Based on the line signs and the current positioning information of the autonomous driving vehicle, the autonomous driving vehicle learns to track the lane lines and then extracts the current road information from the high-precision map.

5. The method according to claim 1, characterized in that The high-level decisions output by S2 include left lane change behavior, right lane change behavior, acceleration behavior, constant speed behavior or deceleration behavior.

6. The method according to claim 1, characterized in that The PPO algorithm includes an Actor network and a Critic network. The Actor network and the Critic network have the same structure except for the different outputs. The input is a raster image of the blurred speed and deformed posture of the surrounding traffic participants, and a vector of traffic rules and road directions. The raster image is processed by three convolutional layers and then flattened by a fully connected network. The vectors of traffic rules and road directions are processed by fully connected networks respectively. The Actor network and the Critic network are connected and output by two fully connected networks. The Actor network is finally normalized by the SoftMax function to output high-level decision-making behavior. The last layer of the Critic network directly outputs a value.

7. A reinforcement learning lane-changing behavior planning method system based on imitation learning initialization, characterized in that: It includes preprocessing module, initialization module and optimization module, and the reward unit is introduced in the optimization module; The preprocessing module is used to construct a grid map M that introduces the blurred speed and deformed posture of surrounding vehicles. t , traffic rules and road condition information are combined to use vector V t express; The initialization module is used to initialize the lane-changing decision behavior planning π using imitation learning b , the grid map M t and the vector V t Connect the input to the imitation learning DAgger network and output the high-level decision of the vehicle; The optimization module is used to initialize the Actor network in the PPO algorithm, and collect multiple sets of driving trajectory data based on the SUMO simulation scenario. Under the action of the hybrid reward mechanism, the PPO algorithm optimizes the lane change decision behavior planning and obtains the optimized high-level decision b k ; The hybrid reward mechanism in the reward unit automatically triggers the corresponding reward function according to the selected strategy; specifically, based on the lane change driving task, the state of the autonomous driving vehicle and traffic conditions at time t is defined as state s t , the lane-changing strategy π is based on s t , get the decision command u t , reinforcement learning lane change decision behavior planning takes u t , transition to state s at time t+1 t+1 , and get reward r t , which is accumulated to complete the specific task of lane-changing decision-making behavior planning, and the lane-changing decision-making behavior planning must adjust the lane-changing strategy π to obtain long-term benefits R t Based on the reinforcement learning PPO algorithm, behavior planning π is performed, and the state information of the reinforcement learning lane change decision behavior planning is input, and the output b k ∈{left_change, keep, right_change, speedup, speeddown} is defined as a high-level decision if b k If there is a conflict with the road information, the behavior planning π will make a new decision; the timeline is divided into multiple fixed time periods, and in each time period [t k , t k+1 ], b k will remain unchanged, the time span is longer than the control command, and [t k , t k+1 ] The control command cumulative reward in the time period is replaced Decision-making rewards.

8. An autonomous driving vehicle, characterized in that: Lane changing decision behavior planning is performed based on the method described in any one of claims 1 to 6.

9. A computer device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store a computer executable program, and the processor reads part or all of the computer executable program from the memory and executes it. When the processor executes part or all of the computer executable program, the reinforcement learning lane changing behavior planning method based on imitation learning initialization as described in any one of claims 1 to 6 can be implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored in a computer-readable storage medium. When the computer program is executed by a processor, the method for planning lane-changing behavior through reinforcement learning based on imitation learning initialization as described in any one of claims 1 to 6 can be implemented.

Images