Automatic driving lane keeping decision-making method based on deep reinforcement learning

Abstract

The invention discloses an automatic driving lane keeping decision-making method based on deep reinforcement learning, and the method specifically comprises the following steps: 1, sequentially building a state space and an action space, designing a composite reward function, and improving an experience playback algorithm; and step 2, setting corresponding training parameters, collecting training data, training the deep reinforcement learning model, and then testing each model. The method shows remarkable superiority in a lane keeping decision task, can adapt to various weather and traffic conditions, improves the lane keeping capability of the automatic driving vehicle in a complex environment, and has a wide application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically relates to an autonomous driving lane keeping decision method based on deep reinforcement learning. Background Art

[0002] Vehicle keeping decisions play a vital role in the safety and efficiency of road traffic. However, human factors such as distraction or lack of driving experience often lead to unreasonable lane changing behaviors. In recent years, with the rapid development of autonomous driving technology, it has shown great potential in improving the safety and efficiency of traffic systems.

[0003] Lane keeping decision models in autonomous driving have become a research hotspot in the field of transportation and have achieved remarkable research results. These models are mainly divided into two categories: traditional lane keeping models and lane keeping models based on machine learning. Traditional lane keeping models are based on the principles of vehicle kinematics and dynamics. They calculate key control parameters such as steering wheel angle, throttle and brake pedal opening and closing through control algorithms, and send these parameters to the vehicle control platform to control the vehicle's driving direction and speed. Subsequently, the control signal is adjusted according to the error between the vehicle's current actual position and the expected position: when the error is within the allowable range, a smaller control increment is used; when the error exceeds the range, a larger control increment is used, so as to quickly adjust the system state to ensure control accuracy and stability. However, this model is difficult to meet the real-time computing requirements when dealing with dynamic environmental changes, and the measurement is relatively complex. Another method is based on deep reinforcement learning algorithms in machine learning. Lane keeping requires a series of actions to reach the destination, and the performance of the current action affects the final goal. Deep reinforcement learning algorithms are very suitable for dealing with this problem. Unlike traditional models, deep reinforcement learning captures road images through on-board cameras and inputs images into the network for training, without the need to establish complex constraint optimization models. The trained model can take the current state as input and directly output control actions to guide the vehicle to drive safely and stably within the lane. At present, lane keeping models based on deep reinforcement learning usually only control a single action of the vehicle and need to be used in combination with traditional models to complete the complete lane keeping process. Simple reward functions will reduce the learning ability of the model in lane keeping control, and the reward function alone cannot fully guarantee safety. The model may still make unsafe actions, increasing the risk of collision. Existing autonomous driving lane keeping decision methods have problems with complex model design and low computational efficiency. In addition, these methods are prone to collision risks in actual vehicle control engineering and are difficult to be effectively applied. Summary of the invention

[0004] The purpose of the present invention is to provide an autonomous driving lane keeping decision method based on deep reinforcement learning to solve the problem of low computational efficiency in existing methods.

[0005] The technical solution adopted by the present invention is an automatic driving lane keeping decision method based on deep reinforcement learning, which is specifically implemented according to the following steps:

[0006] Step 1: Establish state space, action space, design composite reward function, and improve experience replay algorithm;

[0007] Step 2: Set the corresponding training parameters, collect training data, train the deep reinforcement learning model, and then test each model.

[0008] The present invention is also characterized in that:

[0009] Step 1 is as follows:

[0010] Step 1.1: Establishing the state space

[0011] In the process of lane keeping, the behavior of the smart car is closely related to information such as the road and traffic participants. The features are fused through the RGB image and the bird's-eye view image and used as the state space;

[0012] Step 1.2: Action space establishment

[0013] The control of the smart car is determined by the throttle force, brake force, steering wheel angle and whether to reverse gear;

[0014] Step 1.3: Compound Reward Function Establishment

[0015] By defining more detailed reward and punishment items, the vehicle can receive corresponding rewards every time it performs an action, thus effectively guiding it towards the goal.

[0016] Step 1.4: Improve the experience replay algorithm.

[0017] In step 1.3, in order to make the smart car move towards the target, the reward function consists of distance reward and penalty term r_dis, collision reward and penalty term r_collision, speed reward and penalty term r_fast, lane departure term r_out, driving direction reward and penalty term r_α, lateral acceleration reward and penalty term r_lat and stagnation reward and penalty term c, that is:

[0018] r=λ0r_dis+λ1r_collision+λ2v_lon+λ3r_fast+λ4r_out+λ5r_α+λ6r_lat+c (1)

[0019] In the formula, λ i Represents the weight coefficient of each reward and punishment item.

[0020] The distance reward and penalty item means that in order to encourage the vehicle to reach the destination as soon as possible, a reward related to the distance to the destination is set. When the vehicle is close to the destination, a positive reward is given; when the vehicle is far away from the destination, a negative reward is given; collision reward and penalty item: avoid collision between the autonomous driving vehicle and other vehicles or obstacles in the surrounding environment. When a collision occurs, a large negative reward is given; speed reward and penalty item: to ensure the safety of the driving process and prevent the vehicle from speeding too fast, a negative reward is given when the speed exceeds the expected speed; lane departure item: to keep the vehicle driving in the lane, a negative reward should be given when the vehicle deviates from the lane. In the experiment, when the distance between the vehicle and the center line of the lane is greater than 2m, the lane departure item is set to a negative number; driving direction reward and penalty item r_α=5α 2 , α(θ) represents the function of the deviation between the vehicle's driving direction and the road direction:

[0021]

[0022] θ represents the angle between the road direction and the vehicle's travel direction, and its value is between 0° and 45°;

[0023] Lateral acceleration bonus and penalty v_lon represents the lateral velocity of the vehicle;

[0024] Stagnation reward and penalty: In order to prevent the vehicle from being stagnant for a long time to avoid penalties, a negative reward related to stagnation is set. In the experiment, when the vehicle stays in place for a long time, it will receive a penalty value of -0.1.

[0025] In step 1.4, using the priority experience replay method to improve can maximize the use of historical samples and accelerate the training process. The priority experience replay algorithm uses the TD error, i.e., δ i To determine the weights:

[0026] δ i = r(s t ,a t )+λQ - (s t+1 ,a t+1 )-Q(s t ,a t ) (3)

[0027] Among them, Q represents the online network, Q - represents the target network, r represents the reward and punishment value, s t represents the state at time t, a t Represents the action at time t.

[0028] Step 2 is implemented as follows:

[0029] Step 2.1: Set the corresponding training parameters

[0030] Set the hyperparameters for deep reinforcement learning lane keeping decision model training, including network learning rate, training batch size, number of training cycles, and experience pool capacity, which together affect the training effect and performance of the model;

[0031] Step 2.2: Collect training data

[0032] The Town01 city built into the CARLA simulator was selected for data collection. The scene includes many T-shaped intersections, various shopping malls, residential buildings, a small river running through the city, a bridge over the river for passage, and dynamic cars.

[0033] Step 2.3: Model training

[0034] Determine the initial parameters of deep reinforcement learning, including the batch size during training, network learning rate, maximum training cycle, optimizer, training discount parameter, priority experience replay coefficient, importance sampling compensation coefficient, and maximum capacity of experience pool storage; determine the parameters of the composite reward and penalty function, including distance reward and penalty items, collision reward and penalty items, vehicle speed reward and penalty items, lane departure items, driving direction reward and penalty items, lateral acceleration reward and penalty items, and stagnation reward and penalty items;

[0035] Step 2.4: Model testing

[0036] The trained model is tested in the Town02 city built into the CARLA simulator using a lane keeping decision task. The driving task variables include the camera sensor input image type, the number of traffic participants, and weather conditions. If the test result performance is consistent with the training performance, the model is acceptable. If the performance decreases, return to 2.3, change the hyperparameters, and retrain until the model is acceptable.

[0037] The beneficial effects of the present invention are:

[0038] 1) A lidar sensor is added to collect information, and a convolutional neural network model with an attention mechanism is established to extract representative low-dimensional features as the state space;

[0039] 2) Based on the fusion state, we further designed an autonomous driving lane keeping decision algorithm with deep reinforcement learning as the core, and used the priority experience replay technology to improve the original experience pool sampling method. We elaborated on the priority calculation method and model update process in detail to ensure that the algorithm can give priority to sampling "high-quality" samples, thereby improving learning efficiency;

[0040] 3) The experimental environment CarlaEnv was built using the CARLA virtual simulation platform. By setting the lane keeping decision-making task of autonomous driving in the simulation environment, the training and testing of the deep reinforcement learning model was completed, which can significantly reduce the risk of collision and improve the operational efficiency of lane keeping. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0042] Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1

[0045] The present invention is based on the deep reinforcement learning automatic driving lane keeping decision method, such as Figure 1-2 As shown, the specific implementation steps are as follows:

[0046] Step 1: Build the model

[0047] Establish state space, action space, design composite reward function, and improve experience replay algorithm;

[0048] Step 1 is as follows:

[0049] Step 1.1: Establishing the state space

[0050] In the process of lane keeping, the behavior of the smart car is closely related to information such as the road and traffic participants. The features are fused through the RGB image and the bird's-eye view image and used as the state space;

[0051] Step 1.2: Action space establishment

[0052] The control of the smart car is determined by the throttle force, brake force, steering wheel angle and whether to reverse gear;

[0053] Step 1.3: Compound Reward Function Establishment

[0054] By defining more detailed reward and punishment items, the vehicle can receive corresponding rewards every time it performs an action, thus effectively guiding it towards the goal.

[0055] In step 1.3, in order to make the smart car move towards the target, the reward function consists of distance reward and penalty term r_dis, collision reward and penalty term r_collision, speed reward and penalty term r_fast, lane departure term r_out, driving direction reward and penalty term r_α, lateral acceleration reward and penalty term r_lat and stagnation reward and penalty term c, that is:

[0056] r=λ0r_dis+λ1r_collision+λ2v_lon+λ3r_fast+λ4r_out+λ5r_α+λ6r_lat+c (1)

[0057] In the formula, λ i Indicates the weight coefficient of each reward and punishment item;

[0058] Distance reward and penalty: In order to encourage the vehicle to reach the destination as soon as possible, set rewards related to the distance to the destination. When the vehicle is close to the destination, a positive reward is given; when the vehicle is far away from the destination, a negative reward is given; Collision reward and penalty: Avoid collisions between the autonomous driving vehicle and other vehicles or obstacles in the surrounding environment. A large negative reward is given when a collision occurs; Speed ​​reward and penalty: To ensure the safety of the driving process and prevent excessive speed, a negative reward is given when the speed exceeds the expected speed (here 2m / s); Lane departure item: To keep the vehicle driving in the lane, a negative reward should be given when the vehicle deviates from the lane. In the experiment, when the distance between the vehicle and the center line of the lane is greater than 2m, the lane departure item is set to a negative number; Driving direction reward and penalty r_α=5α 2 , α(θ) represents the function of the deviation between the vehicle's driving direction and the road direction:

[0059]

[0060] θ represents the angle between the road direction and the vehicle's travel direction, and its value is between 0° and 45°;

[0061] Lateral acceleration rewards and penalties: In order to maintain the stability of the vehicle, set rewards related to lateral acceleration: v_lon represents the lateral velocity of the vehicle;

[0062] Stagnation reward and penalty: In order to prevent the vehicle from being stagnant for a long time to avoid penalties, a negative reward related to stagnation is set. In the experiment, when the vehicle stays in place for a long time, it will receive a penalty value of -0.1;

[0063] Step 1.4: Improve the experience replay algorithm;

[0064] Improve the experience replay algorithm to increase the effective utilization of historical samples;

[0065] In step 1.4, using the priority experience replay method to improve can maximize the use of historical samples and accelerate the training process. The priority experience replay algorithm uses the TD error, i.e., δ i To determine the weights:

[0066] δ i = r(s t ,a t )+λQ - (s t+1 ,a t+1 )-Q(s t ,a t ) (3)

[0067] Among them, Q represents the online network, Q - represents the target network, r represents the reward and punishment value, s t represents the state at time t, a t represents the action at time t;

[0068] Step 2: Model implementation

[0069] Set the appropriate training parameters, collect training data, and train the deep reinforcement learning model, and then test each model;

[0070] Step 2.1: Set the corresponding training parameters

[0071] Set the hyperparameters for deep reinforcement learning lane keeping decision model training, including network learning rate, training batch size, number of training cycles, and experience pool capacity, which together affect the training effect and performance of the model;

[0072] The learning rate determines the speed at which the model's weights are updated during training. Setting it too high or too low may lead to unstable training or slow convergence. The batch size determines the number of samples used in each training iteration. An appropriate batch size can improve training efficiency. The number of training cycles refers to the total number of rounds for model training. A sufficient number of training cycles can ensure that the model fully learns and converges to the optimal solution. The experience pool is used to store the experience data generated by the interaction between the model and the environment. Having a large enough storage capacity can ensure that the model effectively uses historical data for learning while avoiding data overflow problems.

[0073] Step 2.2: Collect training data

[0074] The Town01 city built into the CARLA simulator was selected for data collection. The scene includes many T-shaped intersections, various shopping malls, residential buildings, a small river running through the city, a bridge over the river for passage, and dynamic cars.

[0075] There are multiple feasible spawn points specified on the map. In each new training round, a spawn point is randomly selected as the starting position of the autonomous vehicle. The decision model controls the smart car to continue driving until the termination condition is met. This method of using random starting points allows the autonomous vehicle to drive in a variety of scenarios, thereby improving the stability and generalization ability of the decision model.

[0076] Step 2.3: Model training

[0077] Determine the initial parameters of deep reinforcement learning, including the batch size during training, network learning rate, maximum training cycle, optimizer, training discount parameter, priority experience replay coefficient, importance sampling compensation coefficient, and maximum capacity of experience pool storage; determine the parameters of the composite reward and penalty function, including distance reward and penalty items, collision reward and penalty items, vehicle speed reward and penalty items, lane departure items, driving direction reward and penalty items, lateral acceleration reward and penalty items, and stagnation reward and penalty items;

[0078] In the reinforcement learning model, the network parameters are gradually updated by initializing the connection weights of the neural network and performing multiple rounds of training (each round corresponds to a change in the starting point in the data set) until the model converges.

[0079] Step 2.4: Model testing

[0080] The trained model is tested in the Town02 city built into the CARLA simulator using a lane keeping decision task. The driving task variables include the camera sensor input image type, the number of traffic participants, and weather conditions. If the test result performance is consistent with the training performance, the model is acceptable. If the performance decreases, return to 2.3, change the hyperparameters, and retrain until the model is acceptable.

[0081] Example 2

[0082] The lane keeping decision method for autonomous driving based on deep reinforcement learning, in which step 1.1, the state space: using the observation data of RGB image and bird's-eye view, firstly reduce the dimension through the feature fusion network, then pass the global average pooling layer to obtain the low-dimensional image features as the state space S, and then define a five-dimensional feature vector in the output layer, corresponding to the Q value of the five action spaces of the smart car;

[0083] Action space: Lane keeping requires learning how to adjust speed and direction angle, so the action space A consists of throttle force, brake force, steering wheel angle, and reverse gear. The above four are discrete actions, including throttle force ([0,1]), brake force ([0,1]), steering wheel angle ([-1,1]), and reverse gear (True / False);

[0084] Composite reward function: The design of the reward function affects the learning efficiency of the intelligent agent and the quality of the final strategy. A reasonable reward function can help the algorithm quickly converge to the optimal strategy according to the requirements of the task. The present invention designs a composite reward function composed of multiple reward and punishment items. These reward and punishment items not only cover various situations that may occur during vehicle driving, but also fully consider multiple factors such as safety, driving efficiency and comfort. The reward function is composed of seven items: distance reward and punishment item, collision reward and punishment item, vehicle speed reward and punishment item, lane departure item, driving direction reward and punishment item, lateral acceleration reward and punishment item and stagnation reward and punishment item.

[0085] Improve the experience replay algorithm in step 1.4: The DQN algorithm is a heterogeneous deep reinforcement learning algorithm, which is characterized by using different policy functions to collect experience (behavior strategy) and control the agent (target strategy). Improving the experience pool sampling method of this type of algorithm using the priority experience replay method can maximize the use of historical samples and accelerate the training process. In the priority experience replay, samples with larger weights are more likely to be prioritized for training.

[0086] The DQN algorithm with priority experience replay uses TD error, i.e., δ i To determine the weights:

[0087] δ i = r(s t ,a t )+λQ - (s t+1 ,a t+1 )-Q(s t ,a t )

[0088] Q stands for online network; Q - represents the target network, r represents the reward and punishment value, s t represents the state at time t, a t represents the action at time t;

[0089] If the absolute value of TD error is larger, it means that the correction effect of the sample on the network is more significant, and the sample needs to be learned more.

[0090] The sampling probability of sample i is:

[0091]

[0092] p i Represents the priority of sample i, and its size is determined by δ i Decide;

[0093] The index α is a priority playback factor, 0≤α≤1, α=0 represents uniform sampling, and α=1 represents full use of priority sampling.

[0094] The probability sampling method can ensure that samples with smaller TD error still have a chance to be selected. In the implementation process, there are two types of priority experience replay: proportional and sorted. Proportional priority experience replay means that the probability of a sample being sampled is proportional to its weight δ, and sorted priority experience replay means that the probability of a sample being sampled is proportional to the ranking number of the sample weight δ in the experience pool. The present invention uses proportional priority experience replay:

[0095] p i =δ i |+ε (5)

[0096] ε is a very small positive number that prevents δ i is equal to 0;

[0097] In addition, the priority replay technique changes the sampling frequency of samples, which may cause the model to overfit important experience. Therefore, it is necessary to reduce the weight according to the importance of the samples during training to compensate for this deviation. The importance sampling coefficient is:

[0098]

[0099] N is the size of the experience replay queue; β is the importance sampling compensation coefficient, 0≤β≤1, and the importance of different samples in the importance sampling process is adjusted according to its size.

[0100] Example 3

[0101] Before starting training, call the Reset function to reset the environment. The self-driving car will select a birth point and receive the initial state, and then start this training round. After the training starts, the car collects environmental information through sensors as the current state. The decision-making system decides the action to be performed based on the current state and the deep reinforcement learning algorithm. After each action is performed, the environment will provide feedback through the Step function, including the state at the next moment, the reward, and the signal whether the round is over. The experience data generated by the interaction between the self-driving car and the environment is stored in the experience pool. During the training process, the car continuously extracts data for learning based on the importance of the samples in the experience pool. When the car reaches the termination condition set by the experiment, the training round ends and returns to the initial state of the environment. If the termination condition is not met before the end of the round, the above interaction process is repeated, such as Figure 2 After that, by observing the change of reward value during training, when the trend of reward value change is stable, the model training is completed, and the hyper parameters of the relevant model are shown in Table 1 below.

[0102] Table 1 Model hyperparameter values

[0103] name Numeric Batch size during training 64 DQN network learning rate 0.0005 DDPG policy network learning rate 0.0001 DDPG value function network learning rate 0.001 Maximum training cycle 10000 Optimizer Adam Training discount factor 0.99 Priority experience replay coefficient α 0.6 Importance sampling compensation coefficient β 0.4 Maximum capacity of experience pool storage 5000

[0104] The trained model was tested, and the results are shown in Tables 2 and 3 below, where Table 2 represents a sunny afternoon, and Table 3 represents a rainy afternoon. The model in the present invention can perform lane keeping in different scenarios. Compared with the baseline model, the model designed by the present invention has obvious improvements in reducing collision rate and improving completion.

[0105] Table 2 Test results for a sunny afternoon

[0106]

[0107] Table 3 Test results for rainy afternoon

[0108]

[0109] According to the bird's-eye view and camera image provided by the laser radar sensor and the original sensor data, the present invention adopts pixel-level and feature-level schemes to fuse image information, and establishes a convolutional neural network model with an attention mechanism to extract representative low-dimensional features as the state space as input, and directly outputs the Q value of the five action spaces of the smart car, thereby controlling the lane keeping of the smart car during driving. The original experience pool sampling method is improved by using a composite reward function adapted to the lane keeping task and using the priority experience playback technology, and the priority calculation method and model update process are elaborated in detail to ensure that the algorithm can give priority to sampling "high-quality" samples, thereby improving learning efficiency. Finally, the experimental environment CarlaEnv was built using the CARLA virtual simulation platform, and the training and testing of the deep reinforcement learning model were completed by setting the lane keeping decision task of autonomous driving in the simulation environment. The model construction in this aspect is simple, the calculation efficiency is high, and it is suitable for practical applications.

Claims

1. An autonomous driving lane keeping decision method based on deep reinforcement learning, characterized in that: Follow the steps below to implement it: Step 1: Establish state space, action space, design composite reward function, and improve experience replay algorithm; Step 2: Set the corresponding training parameters, collect training data, train the deep reinforcement learning model, and then test each model.

2. The autonomous driving lane keeping decision method based on deep reinforcement learning according to claim 1, characterized in that: The step 1 is specifically as follows: Step 1.1: Establishing the state space In the process of lane keeping, the behavior of the smart car is closely related to information such as the road and traffic participants. The features are fused through the RGB image and the bird's-eye view image and used as the state space; Step 1.2: Action space establishment The control of the smart car is determined by the throttle force, brake force, steering wheel angle and whether to reverse gear; Step 1.3: Compound Reward Function Establishment By defining more detailed reward and punishment items, the vehicle can receive corresponding rewards every time it performs an action, thus effectively guiding it towards the goal. Step 1.4: Improve the experience replay algorithm.

3. The autonomous driving lane keeping decision method based on deep reinforcement learning according to claim 2, characterized in that: In step 1.3, in order to make the smart car move towards the target, the reward function is composed of the distance reward and penalty term r_dis, the collision reward and penalty term r_collision, the speed reward and penalty term r_fast, the lane departure term r_out, the driving direction reward and penalty term r_α, the lateral acceleration reward and penalty term r_lat and the stagnation reward and penalty term c, that is: r=λ0r_dis+λ1r_collision+λ2v_lon+λ3r_fast+λ4r_out+λ5r_α+λ6r_lat+c (1) In the formula, λ i Represents the weight coefficient of each reward and punishment item.

4. The autonomous driving lane keeping decision method based on deep reinforcement learning according to claim 3, characterized in that: The distance reward and penalty item: in order to encourage the vehicle to reach the destination as soon as possible, a reward related to the distance to the destination is set. When the vehicle is close to the destination, a positive reward is given; when the vehicle is far away from the destination, a negative reward is given; collision reward and penalty item: avoid collision between the autonomous driving vehicle and other vehicles or obstacles in the surrounding environment. When a collision occurs, a large negative reward is given; speed reward and penalty item: to ensure the safety of the driving process and prevent the vehicle from speeding too fast, a negative reward is given when the vehicle speed exceeds the expected speed; lane departure item: to keep the vehicle driving in the lane, a negative reward should be given when the vehicle deviates from the lane. In the experiment, when the distance between the vehicle and the center line of the lane is greater than 2m, the lane departure item is set to a negative number; driving direction reward and penalty item r_α=5α 2 , α(θ) represents the function of the deviation between the vehicle's driving direction and the road direction: θ represents the angle between the road direction and the vehicle's travel direction, and its value is between 0° and 45°; Lateral acceleration bonus and penalty v_lon represents the lateral velocity of the vehicle; Stagnation reward and penalty: In order to prevent the vehicle from being stagnant for a long time to avoid penalties, a negative reward related to stagnation is set. In the experiment, when the vehicle stays in place for a long time, it will receive a penalty value of -0.

1.

5. The autonomous driving lane keeping decision method based on deep reinforcement learning according to claim 2, characterized in that: In step 1.4, using the priority experience replay method to improve can maximize the use of historical samples and accelerate the training process. The priority experience replay algorithm uses the TD error, i.e., δ i To determine the weights: d i =r(s t ,a t )+λQ - (s t+1 ,a t+1 )-Q(s t ,a t ) (3) Among them, Q represents the online network, Q - represents the target network, r represents the reward and punishment value, s t represents the state at time t, a t Represents the action at time t.

6. The autonomous driving lane keeping decision method based on deep reinforcement learning according to claim 1, characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1: Set the corresponding training parameters Set the hyperparameters for deep reinforcement learning lane keeping decision model training, including network learning rate, training batch size, number of training cycles, and experience pool capacity, which together affect the training effect and performance of the model; Step 2.2: Collect training data The Town01 city built into the CARLA simulator was selected for data collection. The scene includes many T-shaped intersections, various shopping malls, residential buildings, a small river running through the city, a bridge over the river for passage, and dynamic cars. Step 2.3: Model training Determine the initial parameters of deep reinforcement learning, including batch size during training, network learning rate, maximum training cycle, optimizer, training discount parameter, priority experience replay coefficient, importance sampling compensation coefficient, and maximum capacity of experience pool storage; Determine the parameters of the composite reward and penalty function, including distance reward and penalty items, collision reward and penalty items, vehicle speed reward and penalty items, lane departure items, driving direction reward and penalty items, lateral acceleration reward and penalty items, and stagnation reward and penalty items; Step 2.4: Model testing The trained model is tested in the Town02 city built into the CARLA simulator using a lane keeping decision task. The driving task variables include the camera sensor input image type, the number of traffic participants, and weather conditions. If the test result performance is consistent with the training performance, the model is acceptable. If the performance decreases, return to 2.3, change the hyperparameters, and retrain until the model is acceptable.

Images