Decision planning method for autonomous vehicle based on deep reinforcement learning

Abstract

The invention relates to the field of decision planning of automatic driving vehicles, in particular to an automatic driving vehicle decision planning method based on deep reinforcement learning. The method comprises the following steps: when a vehicle runs on a structured road, acquiring position information of an environmental vehicle, and introducing historical position data of the environmental vehicle into a long short-term memory network to obtain a driving intention hiding state of surrounding vehicles; and generating a polynomial curve-based planning track through an SAC algorithm according to the obtained hidden state and vehicle motion information fusion. The problem of poor safety and stability of deep reinforcement learning in the field of automatic driving decision planning is solved, the overall training difficulty of reinforcement learning is reduced, and the method can be widely applied to the technical field of automatic driving decision planning.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, particularly to the field of decision-making and planning for autonomous vehicles, and specifically to a method for decision-making and planning of autonomous vehicles on urban expressways based on deep reinforcement learning. Background Art

[0002] With the rapid development of autonomous driving technology, achieving safe and efficient autonomous driving of vehicles in complex traffic environments has become an important research direction. Autonomous vehicles need to have the ability to make long-term and short-term decisions and plans in dynamically changing traffic scenarios to cope with various complex traffic conditions.

[0003] Existing methods have certain limitations in dealing with such complex tasks. On the one hand, traditional rule-based methods usually rely on preset traffic rules and vehicle behavior models. Although they show certain effectiveness in structured and rule-defined traffic environments, their adaptability and flexibility are poor when facing complex and changeable traffic scenarios. For example, on highways, when encountering sudden traffic jams, road construction, etc., rule-based methods may not be able to make reasonable decisions and plans in a timely manner. On the other hand, end-to-end autonomous driving methods can directly learn driving strategies from sensor data through technologies such as deep learning, directly output control instructions, and reduce the dependence on artificial rules. However, since the decision-making process inside the model is difficult to explain and understand, it is called a "black box". When errors or accidents occur, it is difficult to trace and analyze the reasons, which is not conducive to the improvement of the system. In addition, end-to-end methods usually require a large amount of training data and computing resources, and their generalization ability and reliability also face challenges when facing rare or extreme traffic scenarios. At the same time, when using deep reinforcement learning to handle long-sequence driving tasks, there is a problem of insufficient ability to model time-series information, making it difficult to capture the dynamic changes of traffic flow and the complex interaction relationships between vehicles, resulting in limited foresight and accuracy of decision-making and planning. Summary of the Invention

[0004] In order to at least solve one of the problems existing in the prior art, an embodiment of the present invention provides a method for decision-making and planning of autonomous vehicles on urban expressways based on deep reinforcement learning, which solves the problems of poor safety and stability of deep reinforcement learning in the field of autonomous driving decision-making and planning, and at the same time reduces the overall training difficulty of reinforcement learning, and can be widely applied to the technical field of autonomous driving decision-making and planning.

[0005] In order to achieve the purpose of the present invention, a method for decision-making and planning of autonomous vehicles on urban expressways based on deep reinforcement learning provided by the present invention includes the steps:

[0006] Step 1: Based on the sensors of the vehicle, obtain the visual data and driving state data of the vehicle, and process them through a perception algorithm to obtain the surrounding environment vehicle positions (including the ego vehicle) and perception information such as the ego vehicle speed and heading angle as the system input state features;

[0007] Step 2: The historical position information of the environmental vehicles generates the hidden state of the driving intention of the environmental vehicles through a long short-term memory network (LSTM);

[0008] Step 3: The deep reinforcement learning SAC algorithm generates the end state of the planned trajectory according to the hidden state information and the ego vehicle motion information;

[0009] Step 4: Generate a planned trajectory in the form of an interpolated quintic polynomial based on the known start state and the generated end state, and obtain the target driving trajectory points at each moment;

[0010] Step 5: The control module built in the autonomous vehicle controls the autonomous vehicle to track the target driving trajectory points through a proportional-integral-derivative control algorithm to complete the autonomous driving task.

[0011] The present invention also provides an autonomous vehicle decision-making and planning system based on deep reinforcement learning, including the following modules:

[0012] A vehicle perception module, which is used to preprocess the ego vehicle driving state data and the surrounding environment vehicle data to obtain input state features, and the input state features include the historical position features of the environmental vehicles and the ego vehicle motion features;

[0013] An autonomous vehicle decision-making and planning model module based on the SAC algorithm, including a vehicle perception information processing module, a SAC decision-making and planning module, and an action output module. The vehicle perception information processing module is used to generate a hidden state h for judging the driving intention of the surrounding environment vehicles through an LSTM network t and splice the hidden state h t with the ego vehicle motion features to obtain a vector S e ; The SAC decision-making and planning module is used to generate the end state of the plan according to the vector S e The action output module is used to generate a planned trajectory in the form of an interpolated polynomial according to the start state and the generated end state of the plan, and obtain the target driving trajectory points at each moment.

[0014] Further, the SAC decision-making and planning module includes an Actor policy network and a Critic value network. The vector S e is input into the Actor policy network for learning the decision-making and planning strategy of the vehicle to obtain an action probability density function π(s,a), where s and a represent the state and action respectively, and the Actor policy network obtains a corresponding action A based on the action probability density function π(s,a), A = {Sf , d f , t f} , S f 、d f and t f , respectively represent the longitudinal coordinate, the lateral coordinate at the end of the planned trajectory, and the time interval from the initial position to the end position; the vehicle perception information processing module encodes the action to obtain the action vector A t , and the action vector A t is concatenated with the vector S e to obtain the vector S e ′, and the vector S e ′ is input into the Critic value network for value estimation to guide the policy generation of the Actor policy network.

[0015] Furthermore, the number of the Actor policy networks is 1, and the number of the Critic value networks is 2. During training, there are also two target value networks. The parameters of the two Critic value networks are independent of each other, and the parameters of the two target value networks are copied from the Critic value network through soft update. The parameters of the Actor policy network are optimized by maximizing the expected action entropy and Q value; the Critic value network takes the action generated by the Actor policy network and the current state as inputs, outputs the predicted Q value of the action in the current state, and optimizes the network parameters by minimizing the mean square error between the predicted Q value and the target Q value.

[0016] Furthermore, the Actor policy network includes an input layer, a fully connected layer, and an output layer. The first layer network in the output layer is used to output the mean and covariance of the action probability density function π(s, a), and the second layer network in the output layer is used to sample and output the action A according to the action probability density function π(s, a); the Critic value network includes an input layer, a fully connected layer, and an output layer, and the output layer is used to output the estimated Q(s, a) of the state-action value.

[0017] Furthermore, the training process of the autonomous driving vehicle decision-making and planning model based on the SAC algorithm is specifically as follows:

[0018] (1) Initialize the network parameters and the experience replay pool;

[0019] (2) The vehicle interacts with the environment, inputs the current input state feature s t into the Actor policy network, and selects the action a from the action probability distribution output by the Actor policy network; execute the action a t , observe the input state feature s t+1 at the next moment, and obtain the action reward r t , and add the sample (s t , at , r t , s t+1 ) Store it in the experience replay pool P;

[0020] (3) Let t = t + 1, and execute step (2) until T iterations are performed;

[0021] (4) Every time z rounds of interaction are completed, a network update process is carried out. Randomly draw a small batch of samples (s t , a t , r t , s t+1 ) from the experience replay pool P to update the weight hyperparameters of the two Critic value networks and the Actor policy network, and update the weight hyperparameters of the target value network by soft update;

[0022] (5) After the T - round iterative interaction step is completed, the network is updated, and the final network weight hyperparameters are output.

[0023] Furthermore, the reward of the deep reinforcement learning algorithm:

[0024] R = R safe + R speed + R comf

[0025] In the formula, R is the total reward, R comf is the total comfort reward, R speed is the speed reward, R safe is the safety reward.

[0026] Furthermore, in the action output module, the following operations are performed:

[0027] Plan the longitudinal trajectory s(t) and the lateral trajectory d(t) of the vehicle respectively, and parameterize the two trajectories as fifth - degree polynomials;

[0028] Differentiate the longitudinal and lateral trajectory expressions with respect to time respectively to obtain the polynomial expressions of speed and acceleration in each direction;

[0029] Assume that the vehicle is moving with uniform variable acceleration during driving and the vehicle acceleration at the end time of the planned trajectory is 0, and obtain the speed and acceleration at the end of the planned trajectory;

[0030] Solve the coefficients of each term in the trajectory expression to generate a planned trajectory with time characteristics.

[0031] The present invention also provides a computer device.

[0032] The present invention also provides a computer - readable storage medium.

[0033] Compared with the prior art, the beneficial effects of the present invention include:

[0034] Utilize the efficient exploration and fast convergence characteristics of the SAC algorithm to improve the intelligence and comfort of autonomous driving decision-making and planning. At the same time, the present invention combines the memory and processing capabilities of the LSTM network for time series data to achieve optimized decision-making and planning for autonomous driving vehicles in complex traffic environments, ultimately improving the safety, traffic efficiency, and comfort of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of the steps of a method for decision-making and planning of an autonomous driving vehicle on an urban expressway based on deep reinforcement learning provided by an embodiment of the present invention.

[0036] Figure 2 It is a schematic diagram of the Cartesian coordinate system and the Frenet coordinate system in an embodiment of the present invention.

[0037] Figure 3 It is a schematic diagram of the discrete area of the driving environment in an embodiment of the present invention.

[0038] Figure 4 It is a schematic diagram of the structure of an autonomous driving decision-making and planning model based on the SAC algorithm incorporating the LSTM network in an embodiment of the present invention.

[0039] Figure 5 It is a schematic diagram of the complete network structure of the SAC algorithm in an embodiment of the present invention.

[0040] Figure 6 It is a schematic diagram of generating a planned trajectory based on a spline curve in an embodiment of the present invention.

[0041] Figure 7 It is a schematic diagram of completing the decision-making and planning task in a simulated traffic scenario in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] The following describes the specific embodiments of the present invention in further detail with reference to the accompanying drawings through the description of the embodiments, so as to help those skilled in the art have a more complete, accurate, and in-depth understanding of the inventive concept and technical solution of the present invention.

[0043] An autonomous driving vehicle decision-making and planning system based on deep reinforcement learning provided by an embodiment of the present invention includes the following modules:

[0044] A vehicle perception module, which is used to preprocess the driving state data of the host vehicle and the data of surrounding environment vehicles to obtain input state features, and the input state features include environmental vehicle historical position features and host vehicle motion features;

[0045] The decision-making and planning model module of an autonomous driving vehicle based on the SAC algorithm includes a vehicle perception information processing module, an SAC decision-making and planning module, and an action output module. The vehicle perception information processing module is used to generate a hidden state h for judging the driving intentions of surrounding vehicles through a long short-term memory network (LSTM). t And splice the hidden state h t with the motion characteristics of the ego vehicle to obtain a vector S e ; The SAC decision-making and planning module is used to generate a planned end state according to the vector S e The action output module is used to generate a planned trajectory in the form of an interpolation polynomial based on the initial state and the generated planned end state, and obtain the target driving trajectory points at each moment.

[0046] Please refer to Figure 1 , A decision-making and planning method for an autonomous driving vehicle on an urban expressway based on deep reinforcement learning provided by an embodiment of the present invention includes the following steps:

[0047] Step 1, Collect the driving state data of the ego vehicle (abbreviated as "ego vehicle") and the data of surrounding vehicles through in-vehicle sensors and store them. The vehicle perception module preprocesses the data obtained by the in-vehicle sensors to obtain perception information.

[0048] Design the state features for the input of the decision-making and planning model of the autonomous driving vehicle. The state space of the autonomous driving decision-making and planning task in the embodiment of the present invention is the state features (perception features) required by the deep reinforcement learning algorithm preprocessed by the vehicle perception module. According to traffic regulations, non-motor vehicles and pedestrians are not allowed to cross on urban expressways. Therefore, the state space only includes the state of the ego vehicle and the state of surrounding vehicles, and does not consider traffic participants such as pedestrians and non-motor vehicles. The specific state feature design is as follows:

[0049] Represent the input state of the decision-making and planning model of the autonomous driving vehicle in the Frenet framework. Convert the vehicle position coordinates in the environment from the Cartesian coordinate system to the Frenet coordinate system. The Frenet coordinate system simplifies the road constraint expression and improves the semantic intuitiveness of the state space by decoupling the lateral and longitudinal motions. As Figure 2 is the process of converting the Cartesian coordinate system to the Frenet coordinate system, where S represents the longitudinal displacement and d represents the lateral displacement. Convert the positions of all vehicles in the environment from the Cartesian coordinate G i (x, y) to the Frenet coordinate F i (S, d), where i represents the environmental vehicle number, and i = 0 represents the ego vehicle.

[0050] The perception features include the historical position features of surrounding vehicles and the motion features of the ego vehicle. In one embodiment of the present invention, for the positions of surrounding vehicles, since continuous lane-changing scenarios are not considered, only the vehicles in the front and rear of the current lane, as well as the left and right adjacent lanes, are considered. As shown in Figure 3 , the surrounding environment of the ego vehicle is divided into 9 discrete regions. The length of the middle region where the ego vehicle is located is 10 m, and the width is the lane width. Since the environment in which the vehicle is located is a dynamic system evolving over time, the position information of the past trajectories of the vehicles (including the ego vehicle) in the 9 discrete regions is recorded to obtain the historical position features X of the surrounding vehicles t . The shape of the historical position features X of the surrounding vehicles t is a tensor of 6×9×2, which encodes the position information (S t , d t ) of the vehicles in the 9 discrete regions based on the Frenet coordinate system in the past 6 time steps, where the interval of each time step is 0.6 s. For the motion features V of the ego vehicle ego , the shape of the motion features V of the ego vehicle ego is a tensor of 2×1, which encodes two driving information items, namely the vehicle speed and the heading angle of the ego vehicle at the current moment. In each decision-making and planning process, the state features s of the reinforcement learning agent t are modeled as a tuple: s t = (X t , V ego ), including the historical position features X of the surrounding vehicles in the discrete regions preprocessed by the vehicle perception module t and the motion features V of the ego vehicle ego .

[0051] Step 2: Construct a decision-making and planning model module for autonomous driving vehicles based on the SAC algorithm.

[0052] Figure 4 is a schematic structural diagram of an autonomous driving decision-making and planning model based on the deep reinforcement learning SAC algorithm with an LSTM network introduced in the embodiment of the present invention. For the sake of illustration, only the parts related to the embodiment of the present invention are shown. The autonomous driving decision-making and planning model includes three parts, namely a vehicle perception information processing module, an SAC decision-making and planning module, and an action output module.

[0053] The vehicle perception information processing module is used to further process the input state features s of the decision-making and planning model t = (X t , V ego ). Among them, the historical position features X of the surrounding vehicles t are processed by the LSTM network to obtain the hidden state h t used to judge the driving intentions of the surrounding vehicles; the hidden state h t and the motion features V of the ego vehicleego Concatenated into a new vector S e =(h t ,V ego ), the length of the vector is 74, and the new vector S e is used as the input of the SAC decision-making and planning module.

[0054] The LSTM network is a special type of recurrent neural network (RNN) used to address the vanishing gradient and exploding gradient problems faced by traditional RNNs when dealing with long-sequence data. By introducing special cell structures and gate mechanisms, the LSTM network can effectively capture and utilize long-term dependencies in the sequence, making it perform well when processing data with time-series characteristics. Incorporating the LSTM network into the deep reinforcement learning framework can significantly improve the agent's ability to process time-series data, enhancing the adaptability, flexibility, and safety of driving strategies; by capturing the driving intentions of surrounding vehicles and optimizing driving operations, it helps to achieve a more intelligent, efficient, and comfortable driving experience. (The LSTM network in the embodiments of the present invention directly adopts an existing network structure)

[0055] The SAC decision-making and planning module includes an Actor policy network and a Critic value network. The vector S e is input into the Actor policy network of the SAC decision-making and planning module for learning the vehicle's decision-making and planning strategy, obtaining an action probability density function π(s,a) (in the form of a Gaussian function, whose mean and covariance are given by a neural network), where s and a represent the state and action respectively. This probability density function describes the selection probability distribution of each possible action under a given state, and the Actor policy network samples actions based on this to obtain corresponding actions; since the role of the Critic value network is to estimate the action value of the decision-making action, compared with the Actor policy network, an additional action vector A t is added to the input vector of the Critic value network. The action vector A t is obtained by encoding the action A through a fully connected layer FC0 (with 16 neurons and a ReLU activation function). The action vector A t is concatenated with the previous new vector S e to form a new vector S e '=(h t ,V ego ,A t ) with a length of 90. The vector S e ' is input into the Critic value network for value estimation to provide accurate evaluation information for the update of the Actor policy network.

[0056] The SAC decision-making and planning module is implemented based on the SAC algorithm. The SAC algorithm is based on the Actor-Critic framework and introduces the ideas of maximum entropy and maximum soft value. While ensuring the completion of the task, it makes the expected policy as randomized as possible, enabling the decision-making and planning model to achieve sufficient exploration during the training process. Figure 4 Only one Critic value network is shown. The other Critic value networks have the same structure but different network parameters. For the complete network structure of the specific SAC algorithm, see Figure 5 as shown. The present invention uses the idea of maximum entropy reinforcement learning to add an entropy component to the original objective function. Entropy represents the randomness of a random variable and is a measure of uncertainty. The greater the uncertainty, the greater the amount of information. Adding the maximum entropy objective to maximize the entropy of each output of the Actor policy network can randomize the policy, that is, disperse the probabilities of the output actions as much as possible, thereby avoiding premature convergence to a local optimal solution and improving the diversity and robustness of the policy. The maximum soft value is another important concept in the SAC algorithm and is also related to the maximum entropy objective. In traditional reinforcement learning algorithms, a hard maximum value is usually used, that is, the action with the highest Q value is selected as the optimal action. However, this method may lead to a too rigid policy and lack of flexibility when facing a complex environment. The SAC algorithm introduces the maximum soft value, which combines the standard Q value and the entropy term, allowing the agent, i.e., the vehicle, to make a probability distribution and selection among multiple actions based on the Q value of the action and the entropy of the policy. This probability selection method not only improves the flexibility of the policy but also can balance exploration and exploitation during the optimization process of the SAC algorithm.

[0057] As Figure 5The complete network structure of the SAC algorithm includes: an Actor policy network, two Critic value networks, and two target value networks (only the results of one value network are shown in the figure, and the other value network structures are the same but the network parameters are different). The Actor policy network includes an input layer, two output layers, and two intermediate fully connected layers FC1 (with 128 neurons and the activation function ReLU) and FC2 (with 32 neurons and the activation function ReLU). The first layer of the output layer outputs the mean and covariance of the action probability density function π(s,a), and the second layer of the output layer samples and outputs the action A according to the action probability density function π(s,a). The structures of the two Critic value networks and the two target value networks are the same. Similar to the Actor policy network, they all include an input layer, an output layer, and two fully connected layers, namely FC3 (with 128 neurons and the activation function ReLU) and FC4 (with 32 neurons and the activation function ReLU) in the figure. The output layer of the Critic value network has only one layer and directly outputs the estimation Q(s,a) of the state-action value. At the same time, the parameters of the two Critic value networks are independent of each other, while the parameters of the two target value networks are copied from the Critic value network through soft update to improve the stability of learning and stabilize the training process (the two target value networks only play a role in the training process of the model). The update of the Actor policy network depends on the predicted Q value output by the Critic value network and its own action probability density function π(s,a), and optimizes the Actor policy network parameters by maximizing the expected action entropy and Q value. The Critic value network takes the action generated by the Actor policy network and the current state as inputs and outputs the predicted Q value of the action in the current state. Its update depends on the target Q value provided by the target value network and optimizes the network parameters by minimizing the mean square error between the predicted Q value and the target Q value. The dual Critic value network is used to evaluate the action value from different perspectives. The two Critic value networks independently estimate the Q value and combine the evaluation information of the two Critic value networks to guide the update of the Actor policy network, reducing the possibility that the learning process is affected by the randomness or instability of a single network. At the same time, the policy network is updated under multiple evaluations, improving the robustness and adaptability of the policy.

[0058] To implement the decision-making and planning task based on the SAC algorithm, it is necessary to design the action space of the SAC algorithm. The action space is the operation that the vehicle will execute calculated by the Actor policy network in each round of the SAC algorithm according to the currently observed state feature s t To complete the decision-making and planning task, it is necessary to generate a planned trajectory based on a polynomial curve in the Frenet space. Each segment of the planned trajectory can be characterized by three consecutive values: S f, d f and t f , respectively representing the longitudinal coordinate, lateral coordinate at the end of the planned trajectory, and the time interval from the initial state to the end position. Each value has an acceptable range for action sampling. These ranges are mapped to [-1, 1], so that the action (continuous action space) A in RL (Reinforcement Learning) can be defined as:

[0059] A = {s f , d f , t f}

[0060] The action output module generates a corresponding planned trajectory according to the action A output by the Actor policy network of the SAC decision-making and planning module in the current state. Since the action A includes the longitudinal coordinate, lateral coordinate, and time interval t of the planned end state f , the longitudinal trajectory s(t) and lateral trajectory d(t) of the vehicle are planned respectively, and the two trajectories are parameterized as fifth-order polynomials:

[0061]

[0062] where d(t) and s(t) are fifth-order polynomials of the lateral and longitudinal trajectories with respect to time t, and a i , b i are the coefficients of each term respectively.

[0063] Taking the derivative of the longitudinal and lateral trajectory expressions with respect to time respectively, polynomial expressions of velocity and acceleration in each direction can be obtained, and 6 boundary condition expressions for the longitudinal and lateral directions are obtained:

[0064]

[0065] where t0 and t1 represent the initial and termination times of the planned trajectory respectively, and d0, s0, d, and s1 represent the lateral and longitudinal coordinates of the starting and ending points of the planned trajectory respectively, are the first and second derivatives of the lateral and longitudinal trajectory polynomials, that is, the lateral and longitudinal velocities and accelerations at time t, are the lateral and longitudinal velocities and accelerations of the planned trajectory at the initial time respectively, and d, are the lateral and longitudinal velocities and accelerations of the planned trajectory at the termination time respectively.

[0066] The time, coordinates, velocity, and acceleration of the starting point of the planned trajectory can be obtained from relevant data of the vehicle's sensors. The time, lateral, and longitudinal coordinates of the ending point of the planned trajectory are determined by the output action A = (s f , d f , t f)It is determined that by assuming that the vehicle performs a uniformly variable motion during driving and the vehicle acceleration at the end time of the planned trajectory is 0, the speed and acceleration at the end of the planned trajectory can be obtained:

[0067]

[0068] According to the above boundary conditions, the coefficients of each term in the trajectory expression can be obtained, and then a planned trajectory with time characteristics can be generated. The autonomous vehicle is controlled to track the generated planned trajectory points through a proportional-integral-derivative control algorithm.

[0069] As Figure 6 shown, exploring each area in the action space is equivalent to checking different spline curves in the driving corridor, thereby generating corresponding planned trajectories.

[0070] Step 3: Train the proposed decision-making and planning model for autonomous vehicles.

[0071] To train the decision-making and planning model for autonomous vehicles based on the SAC algorithm, it is necessary to define an environmental reward function. In reinforcement learning, the environmental reward function guides the vehicle to adjust its decision-making and planning strategy to maximize the future cumulative reward by providing immediate positive rewards or negative punishments, playing a key role in guiding learning and task definition. In one embodiment of the present invention, the environmental reward function is considered from three aspects: success rate, traffic efficiency, and comfort. The environmental reward function used includes three elements: safety reward R safe , speed reward R speed and comfort reward R comf . The safety reward R safe is determined based on the offset of the ego vehicle from the center line of the lane and whether a collision occurs, and is used to encourage the ego vehicle to keep driving in the center of the lane while punishing collision behaviors. The expression is:

[0072]

[0073] Where D is the lateral deviation at the end state of the planning. In addition to safety considerations, traffic efficiency is also an important factor considered in this method. The embodiments of the present invention hope that the vehicle can travel at a suitable speed on the road. If the speed is too high, it is easy to violate traffic regulations, and at the same time, it cannot turn normally during cornering and may drive out of the specified road; if the speed is too low, even if it can drive close to the center line, it will affect road traffic and does not meet the goal of improving traffic efficiency in the embodiments of the present invention. In addition, when designing the environmental reward function, a certain positive reward should be given to balance the penalties of other items. Otherwise, too many negative rewards will cause the ego vehicle to tend to "commit suicide" at the beginning stage of training, that is, immediately end the training of this round. Therefore, the driving efficiency item is used as a positive reward item here. If the speed planned by the ego vehicle during road driving is close to the desired speed, a positive reward is given to encourage the ego vehicle to drive normally at the target vehicle speed; at the same time, under the condition of road speed limit, if the planned speed exceeds the limit speed v max it will be penalized. The more the exceeded speed, the greater the penalty. Similarly, under the given low-speed limit condition, if the planned speed is lower than the given minimum speed v min it will be penalized, then the speed reward R speed The expression of is:

[0074]

[0075] Where, v target is the target vehicle speed, which can be adjusted according to the current road speed limit and environmental traffic conditions; v is the speed of the ego vehicle at the end state of the planning; ω v , ω low , ω high are the speed reward coefficient, low-speed penalty coefficient, and overspeed penalty coefficient respectively. These coefficients can be adjusted according to driving goals.

[0076] According to the literature, when the acceleration of the vehicle during acceleration is higher than a certain value, it will enter the aggressive driving (uncomfortable) interval, and there is also an uncomfortable driving interval during deceleration. In order to ensure that the strategy learned by the decision-making and planning model can be as smooth and peaceful as that of a skilled human driver and avoid frequent start-stop phenomena as much as possible, the acceleration of the driving trajectory obtained by the decision-making and planning is evaluated. When the trajectory acceleration output by the decision-making and planning exceeds the preset threshold, the model is penalized; in addition, humans are more sensitive to the change rate of acceleration (jerk) than acceleration, so suppressing jerk can also avoid the "jerky feeling" generated during the acceleration and deceleration process; the longitudinal driving comfort reward R comf-long is obtained by evaluating the longitudinal acceleration and jerk of the trajectory output by the decision-making and planning, and the expression is:

[0077] R comf-long =-ω1*exp(|acc|-acc′)-ω2*exp(|jerk|-jerk′)

[0078] Among them, acc is the maximum acceleration during the acceleration process or the maximum deceleration during the deceleration process, acc′ is the acceleration threshold, which can be obtained according to the set driving style, jerk is the maximum acceleration change rate during the acceleration and deceleration processes, jerk′ is the acceleration threshold, and ω1 and ω2 are the acceleration reward term coefficient and the jerk reward term coefficient respectively.

[0079] During the lane change process, the large change in the front wheel angle will cause a large lateral impact force on the passengers, which greatly affects the riding experience. Since it is not convenient to directly collect the front wheel angle information during the training process, the lateral comfort reward R is designed by evaluating the lateral displacement change rate of the output trajectory of the decision-making planning. comf-lat :

[0080] R comf-lat = -ω3 * l′ 2 -ω4 * l″ 2

[0081] Among them, l′ and l″ are the maximum acceleration and the maximum acceleration change rate in the lateral direction of the output trajectory of the decision-making planning during the lane change process respectively, and ω3 and ω4 are the lateral acceleration reward term coefficient and the lateral acceleration change rate term coefficient respectively.

[0082] In addition, in order to minimize the discomfort brought to the passengers by the lane change behavior during driving and increase the potential lane change risk, the embodiment of the present invention also restricts the vehicle lane change behavior. If the initial state and the end state of the planning result are not in the same lane, it is considered that a lane change behavior is triggered, and a penalty will be given to each lane change behavior to obtain the lane change comfort reward R. comf-lc :

[0083]

[0084] The total comfort reward R comf is:

[0085] R comf = R comf-long + R comf-lat + R comf-lc

[0086] In summary, the total reward is:

[0087] R = R safe + R speed + R comf

[0088] In one embodiment of the present invention, the CARLA simulation simulator is used to construct a traffic scenario. A certain number of environmental vehicles are randomly generated in the simulated traffic scenario, and the surrounding vehicles are equipped with IDM and a ranging sensor to have an interactive simulation environment. Each vehicle randomly samples an initial position and a target speed within a feasible range. To prepare the input features of the model, the ground truth information obtained from the simulator is directly used and Gaussian noise is added to simulate the environmental modeling error of the perception algorithm.

[0089] Train the self-driving decision-making and planning model based on the SAC algorithm with the LSTM network introduced (abbreviated as the SAC-LSTM model). The training process is as follows:

[0090] ① In the initial state, the experience replay pool P is initialized to zero, and the policy network parameters φ and the two Critic value network parameters θ1, θ2 are set. The discount factor γ and the learning rate λ are set;

[0091] ② After the initialization is completed, copy the parameters θ1, θ2 to the two target value networks to initialize the parameters of the target value networks

[0092] ③ Perform T iterations. In each iteration, the vehicle interacts with the environment once. Specifically, the agent, that is, the vehicle, at each time step t, generates an action a according to the policy π(a t ,s t ), generates an action a t , and then the vehicle executes this action in the environment to obtain the next state s t+1 and the reward r corresponding to the current action t , and then store the training sample (s t ,a t ,r t ,s t+1 ) of this time step into the experience replay pool P to reduce data correlation;

[0093] ④ Every time z environmental interaction steps are completed, enter the gradient update step. Each time the gradient is updated, randomly extract a batch (in this embodiment, the batch size batch_size = 128) of historical training samples (s t ,a t ,r t ,s t+1 ) from the experience replay pool P, and then first update the Critic value network parameters θ1, θ2. For each Critic value network, calculate the gradient according to the following objective function and update the network parameters θ i :

[0094]

[0095] Among them, J Q (θ n ) is the objective function for updating the Critic value network. n ∈ {1, 2} corresponds to two different Critic value networks. s t , a t , r t , s t+1 are the sampled training samples. represents calculating the expected value of the losses of all training samples sampled from the experience replay pool P for the current batch. is the estimated Q value of the state-action pair (s t , a t ) output by the Critic value network. y t is the target value calculated through the target value network. The difference between t and y t is the sample loss; for the target value y t , it is obtained by adding the current moment reward r in the sample to the future discounted reward. γ is the discount factor. t+1 , a t+1 ) calculated by the two target value networks for the next moment state-action pair (s t+1 |s t+1 ) and selecting the smaller value among the two target values. logπ(a t+1 |s t+1 ) is the logarithmic probability of the policy, i.e., the entropy term. a is the gradient calculation symbol. λ Q is the learning rate of the network parameters. The update objective of the network parameter θ n is to minimize the Bellman error of the objective function, enabling the Critic value network to more accurately estimate the value of the state-action. Then, the learning rate is used to control the update step size of the network parameters, and the gradients of the parameters θ1 and θ2 are updated separately.

[0096] ⑤ Immediately update the policy network. Similarly, calculate the gradient of the objective function of the Actor policy network and update the Actor policy network parameters φ:

[0097]

[0098] Among them, J π (φ) is the objective function for updating the Actor policy network. s t is the state sampled from the experience replay pool D. a t is the action sampled from the policy network.​ Denote the calculation of the expected value of the losses of all training samples sampled in the current batch, It is to take the smaller of the two Q-values obtained from the two Critic value networks to reduce overestimation. logπ(a|s) is the logarithmic probability of this policy, that is, to calculate the entropy of the action policy, and α is the weight of the entropy. ∇ is the symbol for gradient calculation. The goal of the policy network update is to maximize the expected action entropy and Q-value, that is, to maximize the return with the entropy term, so that the agent can balance a certain degree of exploration while pursuing higher expected rewards when selecting actions. Then, use the learning rate to adjust the update step size and perform gradient update on the network parameter φ.

[0099] ⑥ Finally, update the target value network parameters in a soft update manner every certain number of steps:

[0100]

[0101] where τ is the hyperparameter of soft update (taking 0.995 in one of the embodiments of the present invention), n ∈ {1, 2} corresponds to two different Critic value networks and two target value networks, and θ n are the parameters of the previously updated Critic value network, are the parameters of the target value network.

[0102] Iteratively update the SAC-LSTM model through the above steps. After the SAC-LSTM model converges, construct a simulation traffic environment to test the model. If the test results do not meet the expectations, optimize the network parameters and the reward function and continue training.

[0103] Step 4, Verification and evaluation of the SAC-LSTM model. Use the obtained decision-making and planning model of the autonomous driving vehicle to decide the planned trajectory and obtain the target driving trajectory points at each moment, so that the control module built into the autonomous driving vehicle can control the autonomous driving vehicle to track the target driving trajectory points through the proportional-integral-derivative control algorithm (PID algorithm), and control the vehicle movement to complete the autonomous driving task.

[0104] In one of the embodiments of the present invention, as Figure 7 shown, in this embodiment, the CARLA simulator is used to construct a test traffic scenario, where the curve is the planned trajectory output by the SAC-LSTM model, and the points on the curve represent the planned trajectory points at different moments. The vehicle follows these trajectory points for action control to complete the driving task and generate the data required for training and testing.

[0105] To verify whether the provided decision-making planning model has the ability to complete safe driving within a specified time and compare the effects of different algorithms, the proposed SAC-LSTM model was compared with three other reinforcement learning baseline methods commonly used to solve continuous action space problems, namely DDPG, TRPO, and SAC. DDPG: Combines the ideas of deterministic policy gradient and DQN, learns approximate value functions and policy functions through neural networks, and introduces techniques such as experience replay and target networks to improve the stability and efficiency of learning. TRPO: Adopts techniques such as double delayed update strategy, target policy noise, and clipped double Q-network, and at the same time ensures the stability of policy updates by introducing the concept of trust region, showing excellent performance in dealing with continuous action space problems. The method was tested in 1000 randomly generated scenarios to measure its performance. The test results were sorted and analyzed. As shown in Table 1, the task completion rate of the present invention is 96.8%, the average speed is 14.12%, and the comfort index is 89. All indicators are higher than the baseline methods, indicating that the method provides effective planning decisions and realizes the safety and comfort requirements during driving. Because the method introduces LSTM and can better understand the dynamic changes of traffic conditions by capturing long-term dependencies, the completion rate of this method is 3.6% higher than that of the SAC algorithm, and the average speed and comfort have also been improved to a certain extent.

[0106] . Table 1

[0107]

[0108] Comfort index: where jerk is the maximum acceleration change rate during acceleration and deceleration, and yaw rate is the front wheel yaw angular velocity

[0109] In one embodiment of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in the foregoing embodiment is implemented.

[0110] In one embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the foregoing embodiment is implemented.

[0111] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A decision-making and planning method for autonomous driving vehicles based on deep reinforcement learning, characterized in that, It includes the following steps: Collect and preprocess the self-vehicle driving state data and the surrounding environment vehicle data to obtain input state features, where the input state features include the historical position features of the environmental vehicles and the self-vehicle motion features; Input the environmental vehicle historical position features into the Long Short-Term Memory Network (LSTM) to obtain the hidden state h for judging the driving intentions of surrounding environmental vehicles t ; The SAC algorithm generates a planned end state based on the hidden state and the self-vehicle motion features; Generate a planned trajectory in the form of an interpolation polynomial based on the initial state and the generated planned end state.

2. An autonomous driving vehicle decision-making and planning system based on deep reinforcement learning, characterized in that, It includes the following modules: A vehicle perception module, which is used to preprocess the self-vehicle driving state data and the surrounding environment vehicle data to obtain input state features, where the input state features include the historical position features of the environmental vehicles and the self-vehicle motion features; The decision-making and planning model module of an autonomous driving vehicle based on the SAC algorithm includes a vehicle perception information processing module, an SAC decision-making and planning module, and an action output module. The vehicle perception information processing module is used to generate a hidden state h for judging the driving intentions of surrounding vehicles through a long short-term memory network (LSTM). t And splice the hidden state h t with the motion characteristics of the vehicle itself to obtain a vector S e ; The SAC decision-making and planning module is used to generate a planned end state according to the vector S e The action output module is used to generate a planned trajectory in the form of an interpolation polynomial based on the initial state and the generated planned end state, and obtain the target driving trajectory points at each moment.

3. The decision-making and planning system for an autonomous driving vehicle based on deep reinforcement learning according to claim 1, wherein The SAC decision-making and planning module includes an Actor policy network and a Critic value network, and the vector S e is input into the Actor policy network to learn the decision-making and planning strategy of the vehicle, obtaining the action probability density function π(s,a), where s and a represent the state and action respectively, and the Actor policy network obtains the corresponding action A based on the action probability density function π(s,a), A = {S f , d f , t f}, where S f , d f and t f represent the longitudinal coordinate, lateral coordinate of the end of the planned trajectory, and the time interval from the initial state to the end position respectively; the vehicle perception information processing module encodes the action to obtain the action vector A t and concatenates the action vector A t with the vector S e to obtain the vector S e ′, and the vector S e ′ is input into the Critic value network for value estimation.

4. The decision-making and planning system for an autonomous driving vehicle based on deep reinforcement learning according to claim 3, wherein, There is 1 Actor policy network and 2 Critic value networks. During training, there are also two target value networks. The parameters of the two Critic value networks are independent of each other, and the parameters of the two target value networks are copied from the Critic value networks through soft update. The Actor policy network parameters are optimized by maximizing the expected action entropy and Q value; the Critic value network takes the action generated by the Actor policy network and the current state as inputs and outputs the predicted Q value of the action in the current state, and the network parameters are optimized by minimizing the mean square error between the predicted Q value and the target Q value.

5. The decision-making and planning system for an autonomous driving vehicle based on deep reinforcement learning according to claim 3, characterized in that, The Actor policy network includes an input layer, a fully connected layer, and an output layer. The first layer network in the output layer is used to output the mean and covariance of the action probability density function π(s,a), and the second layer network in the output layer is used to sample and output the action A according to the action probability density function π(s,a); the Critic value network includes an input layer, a fully connected layer, and an output layer, and the output layer is used to output the estimated Q(s,a) of the state-action value.

6. The decision-making and planning system for an autonomous driving vehicle based on deep reinforcement learning according to claim 3, characterized in that The training process of the decision-making and planning model for autonomous driving vehicles based on the SAC algorithm is specifically as follows: (1) Initialize the network parameters and the experience replay pool; (2) The vehicle interacts with the environment and inputs the current input state feature s t into the Actor policy network, selects an action a from the action probability distribution output by the Actor policy network; and executes the action a t , observes the input state feature s at the next moment t+1 , and obtains the action reward r t , stores the sample (s t , a t , r t , s t+1 ) into the experience replay pool P; (3) Let t = t + 1, and execute step (2) until T iterations are executed; (4) After every z rounds of interaction, a network update process is carried out. A small batch of samples (s t , a t , r t , s t+1 ) are randomly drawn from the experience replay pool P to update the weight hyperparameters of the two critic value networks and the input state feature policy network, and the weight hyperparameters of the target value network are updated by soft update; (5) After executing the T-round iterative interaction steps, the network is updated, and the final network weight hyperparameters are output.

7. The decision-making and planning system for an autonomous driving vehicle based on deep reinforcement learning according to claim 3, wherein In order to implement the training of the decision-making and planning model for autonomous driving vehicles based on the SAC algorithm, an environmental reward function needs to be defined. The environmental reward function is: R = R safe + R speed + R comf Wherein, R is the total reward, R comf is the total comfort reward, R speed is the speed reward, R safe is the safety reward.

8. A decision-making and planning system for an autonomous driving vehicle based on deep reinforcement learning according to any one of claims 2-7, characterized in that In the action output module, perform the following operations: Separate the longitudinal trajectory s(t) and the lateral trajectory d(t) of the vehicle for planning, and parameterize the two trajectories as fifth-degree polynomials respectively; Derive the longitudinal and lateral trajectory expressions with respect to time respectively to obtain the polynomial expressions of the speed and acceleration in each direction; Assume that the vehicle is in uniform variable motion during driving and the vehicle acceleration at the end time of the planned trajectory is 0, and obtain the speed and acceleration at the end of the planned trajectory; Solve the coefficients of each term in the trajectory expression to generate a planned trajectory with time characteristics.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in claim 1.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method described in claim 1.

Images