DDPG-based unmanned overtaking decision-making method fusing timing information

Abstract

The application discloses a DDPG-based unmanned vehicle overtaking decision-making method fusing time sequence information.The application obtains state information at multiple continuous time points, such as vehicle speed, wheel rotation speed, distance from the position of adjacent other vehicles and distance from a lane line, and generates a multi-dimensional state input matrix through processing.Then the state matrix is input into an overtaking decision-making control model, and the action control signal of the vehicle, such as the accelerator, the brake and the steering angle, is directly output, so that the vehicle can complete the overtaking driving behavior.The neural network structure of the overtaking model is based on the Actor-Critic architecture, wherein the multi-dimensional state matrix input into the neural network fuses time sequence driving state information, and the proportion of the state space at different time points is controlled through a time decay factor weight coefficient.Finally, simulation experiments on a TORCS (The Open Racing Car Simulator) platform show that the overtaking model fusing the time sequence information can control the unmanned vehicle to safely and accurately complete the overtaking task.

Description

Technical Field

[0001] This invention belongs to the field of intelligent decision-making and control for autonomous driving, and relates to an autonomous driving overtaking decision-making method based on DDPG algorithm modeling. Background Technology

[0002] With the rapid development of artificial intelligence and information perception, autonomous driving technology has received widespread attention from academia and industry, ushering in a period of vigorous development and becoming a research hotspot both domestically and internationally. Autonomous driving plays a crucial role in building future smart city systems, and its application can alleviate problems such as traffic congestion, exhaust emissions, energy consumption, and land resource scarcity. Overtaking is a fundamental driving behavior in autonomous driving; it refers to the process of changing lanes from the current lane into the overtaking lane, passing vehicles on the same side, and then returning to the original lane.

[0003] Currently, methods for solving overtaking decision-making and control problems mainly include rule-based and learning-based methods. Rule-based methods typically involve manually setting rules for a specific scenario, with finite state machines being the most representative. These machines can classify and switch between different driving states to adapt to various driving scenarios. However, as scenario uncertainty increases, the number of manually set rules also becomes diverse, resulting in a large number of rules and difficulties in maintaining the rule base. Therefore, rule-based methods struggle to adapt to highly dynamic and diverse environments. Learning-based methods can be divided into reinforcement learning and deep reinforcement learning. Reinforcement learning-based methods can only solve low-dimensional state and action space problems, exhibiting significant limitations and lacking scalability. Deep reinforcement learning-based methods combine feature extraction from neural networks with the step-by-step decision-making capabilities of reinforcement learning, enabling them to handle high-dimensional state spaces and better adapt to highly dynamic and complex high-dimensional environments. Constructing overtaking driving behavior based on deep reinforcement learning is an important research approach. The DDPG algorithm is widely used in the field of autonomous driving, with its advantages lying in its ability to handle continuous action outputs while closely resembling real-world driving actions. Based on real-world human overtaking behavior, it can be inferred that a vehicle's overtaking action depends not only on its current state but also on its states at multiple past moments; that is, there is a temporal dependency between the states at each moment. The DDPG algorithm determines actions based on the current state of the autonomous vehicle, without considering the temporal dependencies across multiple moments. To address this, this invention proposes a DDPG-based autonomous driving overtaking decision-making method that integrates temporal information. Summary of the Invention

[0004] This invention proposes a DDPG-based overtaking decision-making method for autonomous driving that integrates time-series information. The aim is to address the varying degrees of influence that multiple states at different time periods have on vehicle actions, thereby enabling the autonomous vehicle to make more accurate and safer action control signals and achieve overtaking maneuvers. The end-to-end overtaking decision-making and control model considering the time-series driving state information of the autonomous vehicle is as follows: Figure 2 As shown.

[0005] This invention includes the following steps:

[0006] Step 1: Design of the multidimensional state space and action space of the DDPG-based autonomous driving overtaking model incorporating temporal information. By combining the kinematic and dynamic factors of the vehicle in the simulation environment, the input state space of the autonomous vehicle includes information such as vehicle speed, the angle between the vehicle and the road centerline, and wheel speed. Then, the dimensions of the input state are expanded, and a multidimensional state matrix considering the time-series driving state information of the autonomous vehicle is constructed. The multidimensional state matrix consists of state data from multiple moments within a continuous time period. Considering that moments closer to the current moment have a greater impact on the autonomous vehicle's decision-making and actions, a time decay factor weighting coefficient is designed to control the different proportions of the state space at different moments. Finally, a convolution operation is performed on the multidimensional state matrix to form a feature map with a time decay factor, as shown in the following process. Figure 3 As shown, the output action space of an autonomous vehicle includes three types of action control signals: throttle, steering angle, and braking.

[0007] Step 2: Building the neural network for the DDPG-based autonomous driving overtaking model, which incorporates temporal information. The neural network of the overtaking model consists of an Actor network μ(s|θ). μ ), Critic network Q(s, a|θ) Q ), Target-Actor network μ′(s′|θ μ′ ) and Target-Critic network Q′(s′,a′|θ Q′ The autonomous driving overtaking model consists of two parts: the Actor network, which outputs the control signal for the autonomous vehicle's actions based on its current observed state; and the Target-Actor network, which outputs the target action control signal based on the next observed state. The Critic network estimates the Q-value to be obtained when the autonomous vehicle observes its current state and takes its action, while the Target-Critic network estimates the Q-value to be obtained when the autonomous vehicle observes its next state and takes its target action.

[0008] Step 3: Training process of the DDPG-based autonomous driving overtaking model that integrates temporal information. Environmental state information s over consecutive time t is acquired through the autonomous vehicle's body sensors. t-1 s t-2 , ..., s t-n The process generates the multidimensional state space matrix of the autonomous vehicle, which is then input into the A of the overtaking model. c t o The network obtains the actions that the autonomous vehicle should perform. t Then, action a t After being executed by the autonomous driving simulation environment, the new environmental state s′ faced by the autonomous vehicle is generated. t And calculate the reward value r obtained by the autonomous vehicle. t Finally, it continuously interacts and trains with the autonomous driving simulation environment. The interaction process between the DDPG-based overtaking model, which incorporates temporal information, and the environment is as follows: Figure 4 As shown. The overtaking model requires defining one triggering condition and two key variables during the process of controlling an autonomous vehicle to complete an overtaking maneuver, such as... Figure 5 As shown. One trigger condition is when the car detects a vehicle ahead and is within a safe distance relative to that vehicle, thus triggering an overtaking maneuver. Two key variables are the lateral offset point and the longitudinal speed offset of the autonomous vehicle relative to the entire lane. The lateral offset point controls the autonomous vehicle's movement in different lanes, while the longitudinal speed offset controls the autonomous vehicle's movement at different target speeds. Both of these lay the foundation for the overtaking model to control the autonomous vehicle's overtaking behavior.

[0009] Step 4: The learning process of the overtaking model. The learning process is divided into two stages: lane keeping and overtaking. The lane keeping stage involves the autonomous vehicle learning to maintain a constant speed in a single-vehicle environment. By adjusting the lateral offset point and longitudinal speed offset, combined with the lane keeping reward function, the autonomous vehicle is guided to continuously learn left and right turning and acceleration / deceleration operations, ultimately achieving the goal of maintaining lanes at different speeds in different lanes. The overtaking stage involves the autonomous vehicle learning overtaking behavior in a multi-vehicle environment. This stage builds upon the lane keeping stage. When the autonomous vehicle's front sensors detect other vehicles ahead, the overtaking condition is triggered. Since the autonomous vehicle initially lacks overtaking experience, it will directly collide with the vehicle in front. After a period of learning under the guidance of the overtaking reward function, it can turn left into the left lane within a safe distance by changing the lateral offset point and maintain lane keeping. Then, by changing the value of the longitudinal speed offset, it learns to accelerate overtaking behavior, and finally returns to the original lane after overtaking. The flowchart for when the autonomous vehicle is driving in the right lane and detects a vehicle ahead and prepares to overtake is shown below. Figure 6 As shown.

[0010] Step 5: Validation process of the DDPG-based autonomous driving overtaking model incorporating time-series information. The trained overtaking model is tested on the TORCS simulation platform to verify the control accuracy of the autonomous vehicle overtaking model.

[0011] Furthermore, in step 3, the specific training process for the overtaking model is as follows:

[0012] Step 3.1: Load the autonomous driving simulation environment, initialize the vehicle's starting position, and update the autonomous vehicle's environmental state information s. t .

[0013] Step 3.2: Obtain the environmental state information s of the unmanned vehicle at consecutive time t. t-1 s t-2 , ..., s t-n At the same time, a time decay factor weighting coefficient α (0 < α < 1) is set.

[0014] Step 3.3: Combine the obtained state information of the unmanned vehicle to form a multi-dimensional state matrix A.

[0015] Step 3.4: Input the state matrix into the Actor network of the autonomous driving overtaking model, and perform convolution operations based on the time decay factor α to form multiple time decay factor feature maps A of the autonomous vehicle. α .

[0016] A α =A*diag(1, α) 1 α 2 …α n (1)

[0017] Step 3.5: Then, multiple feature maps A α The information is integrated and recombined into a feature vector. Finally, the feature vector is transformed into a set of three-dimensional vectors, which is the action 'a' output by the Actor network of the autonomous driving overtaking model. t .

[0018] Step 3.6: For action a t Add noise N for processing, then process action a t This information is applied to a simulation environment to generate new environmental state information s′ for the autonomous vehicle. t .

[0019] a t =a t +N (2)

[0020] Step 3.7: Next, calculate the overall reward value based on the reward function R. Wherein... Let R be the longitudinal velocity reward function for the autonomous vehicle. collisionThe reward function for determining whether an autonomous vehicle has collided is R. track The reward function for determining whether an autonomous vehicle has run off the lane is R. overtaking Reward function for autonomous vehicles performing overtaking maneuvers

[0021]

[0022] Step 3.8: Finally, collect the overall reward value of the autonomous vehicle, the new environment status, and the round end flag and store them in the experience replay pool.

[0023] Step 3.9: Next, start updating the Actor network and Critic network of the autonomous driving overtaking model.

[0024] Step 3.10: First, randomly select sample data of batch size from the experience replay pool. Each sample data includes a multidimensional state matrix A composed of time decay factors.

[0025] Step 3.11: Next, combine the multidimensional state matrix A of the autonomous vehicle with the actions a from the sample data. t The input state space of the Target-Critic network, which is pieced together to form an autonomous driving overtaking model.

[0026] Step 3.12: Input this state space into the Target-Critic network of the autonomous driving overtaking model to estimate the maximum Q-value y for the next state and action, where γ is the reward decay factor, and r t It's a reward value.

[0027] y=γQ′(s′ t ,μ′(s′ t |θ μ )|θ Q )+r t (4)

[0028] Step 3.13: Then minimize the loss function L(θ) Q The Critic network parameters θ of the autonomous driving overtaking model are updated in the following way: Q .

[0029]

[0030] Step 3.14: Then, based on the returned policy gradient... Update the Actor network parameters θ of the autonomous driving overtaking model μ .

[0031]

[0032] Step 3.15: Then update the Target-Actor network parameters θ using a moving average method.μ′ and Target-Critic network parameters θ Q′ , where τ is the sliding coefficient.

[0033]

[0034] Step 3.16: Finally, save the Actor network and Critic network models of the autonomous driving overtaking model. At this point, the training of the DDPG-based autonomous driving overtaking model, which incorporates temporal information, is complete.

[0035] Furthermore, the multidimensional state input matrix of the autonomous vehicle includes environmental state information s from multiple consecutive time points. t Each environmental condition includes information such as vehicle speed, the angle between the vehicle and the center line of the road, and wheel speed.

[0036] Furthermore, the motion space of the autonomous vehicle, a t It includes three types of action control signals: throttle, steering angle, and brake.

[0037] Furthermore, the overall reward function R of the autonomous vehicle includes the speed reward function. Collision reward function R collision Round termination reward function R track and overtaking reward function R overtaking .

[0038] Compared with existing methods, the DDPG-based autonomous driving overtaking decision method proposed in this invention integrates temporal information and has a more accurate action control effect. Compared with the action of a single state decision, it can better control the autonomous vehicle to complete the overtaking task. Attached Figure Description

[0039] Figure 1 This is a diagram of the overall structure of the model of the present invention.

[0040] Figure 2 This is a diagram of the end-to-end overtaking model that integrates timing information according to the present invention.

[0041] Figure 3 The time decay factor characteristic diagram of the present invention

[0042] Figure 4 This is an interactive diagram of the overtaking model and simulation environment of the present invention.

[0043] Figure 5 This is a schematic diagram of the overtaking triggering conditions and key variables of the present invention.

[0044] Figure 6 This is a flowchart of the overtaking behavior of the present invention. Detailed Implementation

[0045] To make the objectives, contents, and advantages of this invention clearer, and in conjunction with the accompanying drawings, this invention will be further described in detail. The specific steps of the overtaking action execution process and the network model's decision-making and updating process in the implementation example are as follows:

[0046] Step 1: The first step is the execution process of the autonomous vehicle overtaking maneuver. The autonomous driving simulation environment is loaded, the vehicle's starting position is initialized, and the environmental status information of the autonomous vehicle is updated.

[0047] Step 2: Construct a neural network for the DDPG-based autonomous driving overtaking model that integrates temporal information. The neural network of the overtaking model consists of an Actor network μ(s|θ). μ ), Critic network Q(s, a|θ) Q ), Target-Actor network μ′(s′|θ μ′ ) and Target-Critic network Q′(s′,a′|θ Q′ The autonomous driving overtaking model consists of two fully connected layers. The Actor network and Target-Actor network are identical, both containing ReLU activation functions in their hidden layers and Tanh and Sigmoid activation functions in their output layers. The Actor network outputs the action control signal that the autonomous vehicle should take based on its current observed state, while the Target-Actor network outputs the target action control signal that the autonomous vehicle should take based on its next observed state. Similarly, the Critic network and Target-Critic network are also identical, both consisting of three fully connected layers, with ReLU activation functions only in the first and third hidden layers. The Critic network estimates the Q-value that the autonomous vehicle should achieve when taking action based on its current observed state, while the Target-Critic network estimates the Q-value that should achieve when taking action based on its next observed state.

[0048] Step 3: Initialize the neural network parameters, experience replay buffer, and random noise N of the autonomous driving overtaking model.

[0049] Step 4: Interact with the simulation environment to collect multiple time-series driving state information of the autonomous vehicle and store it in the experience playback pool.

[0050] Step 5: Combine the multiple states of the autonomous vehicle into a multi-dimensional state matrix A.

[0051] Step 6: Input the multidimensional state matrix A of the autonomous vehicle as the environmental state space into the Actor network of the autonomous driving overtaking model.

[0052] Step 7: The Actor network of the autonomous driving overtaking model then performs state processing, convolving the multidimensional state matrix A according to the time decay factor α to form multiple time decay factor feature maps A. α ,like Figure 2 As shown.

[0053] Step 8: Integrate the information from multiple feature maps and recombine them into a single feature vector.

[0054] Step 9: This feature vector is transformed into a set of three-dimensional vectors through a fully connected operation, which are the specific actions of the autonomous vehicle output by the Actor network of the autonomous overtaking model.

[0055] Step 10: Add noise N to this action to make it a noisy action a. t .

[0056] Step 11: Perform noise processing action a t When applied to the TORCS simulation environment, new environmental observation information s′ for the unmanned vehicle is generated. t .

[0057] Step 12: Reward function based on the longitudinal velocity of the autonomous vehicle Calculation speed bonus value r1.

[0058] Step 13: Determine whether the safe distance between the autonomous vehicle and other vehicles triggers the overtaking condition and whether the overtaking procedure should be executed.

[0059] Step 14: If the overtaking conditions are met, the overtaking model begins to control the autonomous vehicle to perform the overtaking operation, such as... Figure 6 As shown.

[0060] Step 15: Calculate the reward function R when the autonomous vehicle overtakes another vehicle. ovwrtaking Start calculating the reward value r2 when the autonomous vehicle overtakes another vehicle.

[0061] Step 16: By changing the value of the lateral offset point of the unmanned vehicle, control the unmanned vehicle to perform a left turn and enter the overtaking lane.

[0062] Step 17: By changing the value of the longitudinal speed offset of the autonomous vehicle, control the autonomous vehicle to perform an acceleration operation to overtake vehicles in the same lane.

[0063] Step 18: After the autonomous vehicle completes the overtaking maneuver, the overtaking model controls the autonomous vehicle to return to its original lane.

[0064] Step 19: Simultaneously change the longitudinal speed offset of the autonomous vehicle to control it to decelerate in the original lane and continue lane keeping.

[0065] Step 20: Based on the collision reward function R of the autonomous vehiclecollision Calculate the reward value r3 for whether the autonomous vehicle collides during the overtaking process.

[0066] Step 21: Based on the round termination reward function R of the autonomous vehicle track Calculate the reward value r4 for whether the autonomous vehicle gets stuck on the track or goes off the track during its operation.

[0067] Step 22: Integrate the reward values ​​obtained by the autonomous vehicle to obtain the overall reward value Reward = r1 + r2 + r3 + r4.

[0068] Step 23: Collect the overall reward value of the autonomous vehicle (Reward) and the new environmental state (s′). t The round ends when the marker "done" is entered into the experience replay pool.

[0069] Step 24: This completes the process of controlling the autonomous vehicle to perform overtaking behavior using the DDPG-based autonomous overtaking model that integrates time-series information. Next is the neural network decision-making and update process in the overtaking model.

[0070] Step 25: Randomly select sample data of batch size from the experience replay pool of the autonomous driving overtaking model. Each sample data includes a multi-dimensional state matrix R composed according to the time decay factor.

[0071] Step 26: Next, combine the multi-dimensional state matrix A of the autonomous vehicle with the actions a in the sample data. t The input state space of the Target-Critic network, which is pieced together to form an autonomous driving overtaking model.

[0072] Step 27: Input the state space into the Target-Critic network of the autonomous driving overtaking model to estimate the maximum Q value y of the next state and action.

[0073] Step 28: Then minimize the loss function L(θ) Q The Critic network parameters θ of the autonomous driving overtaking model are updated in the following way: Q .

[0074] Step 29: Then, based on the returned policy gradient... Update the Actor network parameters θ of the autonomous driving overtaking model μ .

[0075] Step 30: Then update the Target-Actor network parameters θ of the autonomous driving overtaking model using a moving average method. μ′ and Target-Critic network parameters θ Q′

[0076] Step 31: Finally, save the Actor network and Critic network of the autonomous driving overtaking model.

[0077] In the implementation example of this invention, a one-way two-lane autonomous driving simulation platform built on an Ubuntu server is used as the overtaking scenario to demonstrate the conditions for the autonomous vehicle to complete the overtaking operation. The DDPG-based autonomous driving overtaking model, which integrates time-series information, controls the overtaking behavior of the autonomous vehicle by combining two variables: the lateral offset point and the longitudinal velocity offset, when the overtaking conditions are met. This successfully demonstrates the decision-making and control process of the DDPG-based autonomous driving overtaking model that integrates time-series information. Finally, the E-tra simulation on the TORCS platform is performed. c Experimental tests conducted on the K5 map showed that the autonomous vehicle can safely and accurately complete overtaking tasks without collisions.

Claims

1. A DDPG-based autonomous driving overtaking decision-making method integrating time-series information, characterized in that, Includes the following steps: Step 1: Design of the multidimensional state space and action space of the DDPG-based autonomous driving overtaking model that integrates time-series information; By combining the kinematic and dynamic factors analysis of the vehicle in the simulation environment, the input state space of the autonomous vehicle includes vehicle speed, the angle between the vehicle and the road centerline, and wheel speed. Then, the dimensions of the input state are expanded, and a multidimensional state matrix considering time-series driving state information is constructed; The output action space of the autonomous vehicle includes three action control signals: throttle, steering angle, and braking. Step 2: Building the neural network for the DDPG-based autonomous driving overtaking model that integrates temporal information; the neural network of the overtaking model consists of an Actor network. Critic Network Target-Actor Network and Target-Critic network The autonomous driving overtaking model consists of: the Actor network, which outputs the action control signal that the autonomous vehicle should take based on the current state observed by the autonomous vehicle; the Target-Actor network, which outputs the target action control signal that the autonomous vehicle should take based on the next state observed by the autonomous vehicle; and the Critic network, which estimates the Q value that the autonomous vehicle should obtain when taking the action based on the current state observed by the autonomous vehicle; and the Target-Critic network, which estimates the Q value that should be obtained when taking the target action based on the next state observed by the autonomous vehicle. Step 3: Training process of the DDPG-based autonomous driving overtaking model that integrates temporal information; acquiring continuous data through the autonomous vehicle's body sensors. Environmental state information within a given time period The process generates a multi-dimensional state space matrix for the autonomous vehicle, which is then input into the Actor network of the overtaking model to obtain the actions the autonomous vehicle should perform. Then the action After being executed by the autonomous driving simulation environment, the new environmental state faced by the autonomous vehicle is generated. And calculate the reward value obtained by the autonomous vehicle. Finally, the system continuously interacts and trains with the autonomous driving simulation environment. The training process includes the first step of initializing the vehicle's starting position in the autonomous driving scenario and updating the environmental state information. ; The second step is to obtain multiple consecutive... Environmental state information within a given time period At the same time, a time decay factor weighting coefficient is set. ; The third step is to combine the multiple states of the autonomous vehicle obtained into a multi-dimensional state matrix. ; The fourth step is to input the state matrix into the Actor network of the autonomous driving overtaking model, and then, based on the time decay factor... Convolution operations are performed to generate multiple time decay factor feature maps for the autonomous vehicle. ; The fifth step is to then process multiple feature maps. The information is integrated and recombined into a feature vector; finally, the feature vector is transformed into a set of three-dimensional vectors, which are the actions that the autonomous vehicle should perform, output by the Actor network of the autonomous driving overtaking model. ; Step 4: The learning process of the DDPG-based autonomous driving overtaking model that integrates temporal information; The learning process is divided into two stages: lane keeping and overtaking. In the lane keeping stage, the autonomous vehicle controlled by the overtaking model learns to maintain a constant speed in a single-vehicle environment. In the overtaking stage, the autonomous vehicle controlled by the overtaking model learns overtaking behavior in a multi-vehicle environment. Step 5: Verification process of the DDPG-based unmanned overtaking model that integrates time-series information; The trained overtaking model is tested on the TORCS simulation platform to verify the control accuracy of the unmanned vehicle overtaking model; Overall reward function of autonomous vehicles Including speed reward function Collision reward function Round termination reward function and overtaking reward function According to the longitudinal speed reward function of the autonomous vehicle Calculation speed bonus ;Based on the reward function when the autonomous vehicle overtakes Start calculating the reward value when the autonomous vehicle overtakes another vehicle. According to the collision reward function of the autonomous vehicle Calculate the reward value for whether an autonomous vehicle causes a collision during overtaking. According to the round termination reward function of the autonomous vehicle Calculate the reward value for whether the autonomous vehicle gets stuck on the track or goes off the track during operation. .

2. The autonomous driving overtaking decision-making method based on DDPG that integrates time-series information according to claim 1, characterized in that, In step 3, the training process for the autonomous overtaking model to interact with the environment also includes: Step 6, regarding the action Add noise Process, then perform the action This information is applied to an autonomous driving simulation environment to generate new environmental state information for autonomous vehicles. ; The seventh step is to proceed according to the set overall reward function for the autonomous vehicle. Calculate its overall reward value; The eighth step is to collect the overall reward value of the autonomous vehicle, the new environment status, and the round end flag and store them in the experience replay pool of the autonomous vehicle overtaking model. The ninth step is to update the Actor network and Critic network of the autonomous overtaking model. Step 10: First, randomly select a batch of sample data from the experience replay pool. Each sample data includes a multi-dimensional state matrix composed of data based on the time decay factor. ; The eleventh step is to then process the multi-dimensional state matrix of the autonomous vehicle. Actions in sample data The input state space of the Target-Critic network, which is pieced together to form an autonomous driving overtaking model; The twelfth step is to input the state space into the Target-Critic network of the autonomous driving overtaking model to estimate the maximum Q value of the next state and action of the autonomous vehicle. Step 13: Then, update the Critic network parameters of the autonomous driving overtaking model by minimizing the loss function. ; Step fourteen: Update the Actor network parameters of the autonomous driving overtaking model according to the policy gradient. ; Step 15: Then, update the Target-Actor network parameters of the autonomous driving overtaking model using a moving average method. and Target-Critic network parameters ; Step sixteen: Finally, save the Actor network and Critic network models of the autonomous driving overtaking model; at this point, the training of the DDPG-based autonomous driving overtaking decision and control model that integrates time-series information is completed.

3. The autonomous driving overtaking decision-making method based on DDPG that integrates time-series information according to claim 1, characterized in that, The multidimensional state input matrix of the autonomous vehicle includes environmental state information at multiple consecutive time points. Each state includes vehicle speed, the angle between the vehicle and the center line of the road, and wheel speed.

4. The autonomous driving overtaking decision-making method based on DDPG that integrates time-series information according to claim 1, characterized in that, The movement space of autonomous vehicles It includes three types of action control signals: throttle, steering angle, and brake.

Images