A Vehicle Adaptive Cruise Control Method Based on Projection Constraint Strategy Optimization

Abstract

This invention relates to the field of vehicle assisted driving technology and provides a vehicle adaptive cruise control method based on projection constraint strategy optimization, comprising the following specific steps: Step 1, Strategy trajectory sampling: Step 1.1, Simulation environment setup; Step 1.2, Defining state space and behavior space; Step 1.3, Reward function and safety supervisory function design; Step 1.4, Trajectory sampling process; Step 2, Strategy evaluation and optimization. This method achieves collision-free adaptive cruise control with low computational requirements and exhibits high comfort and traffic efficiency. It is trained using reinforcement learning and incorporates a safety supervisory trust domain to prevent the strategy from increasing the safety supervisory value while pursuing reward maximization. For the first time, a zero collision rate is achieved in an adaptive cruise control strategy trained by reinforcement learning, and the strategy can achieve a high degree of consistency with the expected speed when there is no interference from target vehicles.

Description

Technical Field

[0001] This invention belongs to the field of vehicle assisted driving technology, and in particular relates to a vehicle adaptive cruise control method based on projection constraint policy optimization (PCPO-ACC). Background Technology

[0002] Adaptive Cruise Control (ACC) is an advanced longitudinal driving assistance system with two operating modes. In cruise control mode, it controls the speed to a set value. If a vehicle is detected ahead, a second mode is activated, following the vehicle for a certain time to collision (TTC). Currently, different control algorithms have been proposed for ACC, considering safety, comfort, and traffic efficiency. Typically, cruise control is implemented using a simple proportional-integral-derivative (PID) controller. Special control characteristics can be achieved through design methods, such as changing the pole placement. However, for more complex systems, optimal tuning of the controller gain is challenging. More complex ACC systems are often based on Model Predictive Control (MPC). However, the computational cost of MPC greatly limits the application of models in real-world traffic scenarios. In real-world traffic scenarios, vehicles need real-time control; excessively high computational costs will lead to extremely long inference times, or even fail to compute the optimal solution, which is impractical in real-world traffic environments.

[0003] In recent years, the emergence of Deep Reinforcement Learning (DRL) has shown promise in overcoming the high computational demands of ACC policies based on traditional control models. A key characteristic of reinforcement learning is its self-learning nature. Unlike supervised learning, the learning algorithm does not rely on a labeled dataset. Instead, the optimal behavior is defined solely by an abstract objective that maximizes the collected reward. Particularly important deep reinforcement learning algorithms are model-free reinforcement learning algorithms that do not require a mathematical model of the environment transformation.

[0004] Currently, only a small number of adaptive cruise control strategies are trained using deep reinforcement learning algorithms. These trained strategies require very little onboard computing power when deployed in vehicles and offer high levels of comfort and traffic efficiency. However, these strategies cannot guarantee a zero-accident rate during the continuous learning process, and rear-end collisions occasionally occur. The long-tail effect of safety issues severely hinders the application of strategies trained using safety reinforcement learning methods in safety-critical areas (such as autonomous driving). Summary of the Invention

[0005] The purpose of this invention is to provide a vehicle adaptive cruise control method based on projection constraint strategy optimization, which aims to solve the problems mentioned in the background art.

[0006] The present invention is implemented as follows: a vehicle adaptive cruise control method based on projection constraint strategy optimization includes the following specific steps: Step 1, Strategy Trajectory Sampling: Step 1.1, Setting up the simulation environment: Set up the simulation environment in the Carla simulator. The road scene should allow the agent to perform five scenarios in a single attempt of adaptive cruise control: emergency braking of the vehicle in front, vehicle entering, vehicle exiting, smooth following, and constant speed cruise. Each attempt of the agent will include the smooth following and constant speed cruise scenarios, and the emergency braking of the vehicle in front, vehicle entering, and vehicle exiting scenarios will occur randomly. Step 1.2: Define the state space and behavior space: The state space includes the target vehicle's speed. Follow the vehicle's speed Distance between the two vehicles The state space is The desired speed of the following vehicle is controlled as the primary way to adjust the state of the following vehicle according to the strategy; therefore, the behavior space is defined as follows: ,in This represents the desired speed for controlling the vehicle at the next time step; Step 1.3, Design of Reward Function and Safety Monitoring Function: The reward function is related to the speed and acceleration of the following vehicle and whether a safety accident has occurred. The reward function guides the strategy to make the following vehicle update in the direction of safety, comfort and efficiency; the safety monitoring function is related to the distance between the target vehicle and the following vehicle and whether a collision has occurred. Step 1.4, Trajectory Sampling Process: Sampling Trajectory Including the current time step The state space below Behavioral space Reward Value Safety supervision value and the state space of the next time step , represented as ,in This represents the total number of time steps for each trajectory sampling; the agent needs to record each time step during exploration. When the total number of exploration steps reaches One trajectory sampling process is completed in a short time. Step 2, Strategy Evaluation and Optimization: Based on three neural networks: a policy neural network, a value-based neural network, and a value-based security oversight neural network; after the trajectory sampling process is completed, the policy neural network will update its network parameters. Then, the new policy neural network will map a new behavior space based on the state space in the trajectory. The value-based neural network and the value-based security oversight neural network will evaluate whether the updated network parameters can obtain a higher score and a lower security oversight value based on the behavior space. If so, the new policy neural network parameters will be applied; otherwise, this process will be repeated until a higher score and a lower security oversight value appear.

[0007] A further technical solution, step 1.1 includes the following specific steps: A ring road is constructed based on the map of the Carla simulator. The simulation environment includes five scenarios: emergency braking of the vehicle in front, vehicle entry, vehicle exit, smooth following, and cruise control. These five scenarios are implemented using four vehicles: two obstacle vehicles that may enter the main road at any time, one obstacle vehicle traveling on the main road, and one control vehicle traveling on the main road. The speed of the control vehicle is controlled in real time by the PCPO-ACC strategy. A successful exploration by the agent requires the control vehicle to complete the entire ring road journey comfortably and efficiently without colliding with the obstacle vehicles. The emergency braking of the vehicle in front, vehicle entry, and vehicle exit scenarios will occur randomly, implemented using the random function in the numpy package of the Python environment. The randomness of emergency braking, vehicle entry, and vehicle exit is independent. The emergency braking scenario is generated by the obstacle vehicle traveling on the main road, with a 50% probability of occurrence in each exploration. When emergency braking occurs, the expected speed of the obstacle vehicle on the main road in the next time step will become... v break The magnitude of its acceleration is controlled by a PID controller in the Carla simulator; vehicle entry and exit scenarios occur in a set during the same exploration. Each exploration may have 0, 1, or 2 vehicle entry and exit scenarios. If 2 scenarios occur, they are created by two obstacle vehicles that are not traveling on the main road. The position of the obstacle vehicle entering the main road is also random.

[0008] A further technical solution is that, in step 1.2, in the state space, and Obtained by the get_velocity() function in the Carla simulator.d The distance() function in the Carla simulator is used to obtain the acceleration of the vehicle in the behavior space, which is controlled by the PID controller in the Carla simulator in combination with the desired speed control.

[0009] In a further technical solution, step 1.3 involves designing a reward function that includes three aspects: safety, comfort, and efficiency. The safety reward includes TTC violation rewards and collision rewards, defined as follows: (1); The TTC violation reward is as follows: (2); If a safety distance violation occurs at the current time step, then ,otherwise ; Collision rewards are: (3); If a collision occurs during an exploration, If no collision occurs during the entire exploration, then .

[0010] Vehicle comfort is strongly correlated with acceleration, so the reward function for comfort is defined as follows: (4); in To control the vehicle in time step acceleration, Discount factor; The reward function for vehicle efficiency is defined as follows: (5); in To control the vehicle's intended cruising speed, Discount factor; In summary, the reward function is: (6); The security policing function is related to TTC violation and whether a collision occurs. The security policing function related to TTC violation is: (7); If a safety distance violation occurs at the current time step, then ,otherwise The safety monitoring function related to whether a collision has occurred is: (8); If a collision occurs during an exploration, If no collision occurs during the entire exploration, then .

[0011] In summary, the security supervision function is: (9).

[0012] In a further technical solution, step 2 includes the following specific steps: Agent in policy Next collection Trajectory at each time step The method for calculating the parameter values ​​of the policy neural network for each update is as follows: The goal of the updated policy neural network is to maximize the expected reward, expressed as: (10); At the same time, it is necessary to ensure that the updated strategy meets the security regulatory threshold constraints. (11); in To determine the number of security regulatory functions, Kakade and Langford provided an identity using strategies. Safety regulatory advantage function representation strategy Safety supervision expectations: (12); Combining formulas (11) and (12), we can obtain the strategy. The updatable set that satisfies the safety regulatory threshold is represented as: (13).

[0013] A further technical solution, in step 2, is to use three indicators. and Determine the security monitoring level of the current and updated policy, among which , representing the policy gradient of the safety regulatory value function, a larger one The scope of security oversight has changed significantly after the strategy update; This represents the degree of closeness between the current strategy's security oversight level and the security oversight threshold. A positive value indicates that the current strategy's security oversight level is above the threshold and the security oversight level is relatively high, while a negative value indicates that the current strategy's security oversight level is below the threshold and the security oversight level is relatively low. ,in The Hessian matrix representing the KL divergence between the old and new strategies. A positive value indicates that the KL divergence trust domain and the security regulatory constraint trust domain are in an overlapping state, while a negative value indicates that the KL divergence trust domain and the security regulatory constraint trust domain are in an inclusive state or an unrelated state. when Extremely small This indicates that the current policy and its surrounding neighborhood are in a state of low danger potential, and updating the policy in any direction will not pose a danger; when and If the current policy's KL divergence trust region is within the security oversight trust region, then updating the policy in any direction will not be dangerous. In these two cases, method 1, the Trust Region Policy Optimization (TRPO) algorithm, is used to update the policy network parameters. TRPO limits the maximum change in the policy during each update by using the most recently iterated policy... Iteratively update by maximizing the reward advantage function over a local neighborhood: (14); ; Based on the second-order approximation of the KL divergence constraint... The objective function at point is linearized, resulting in the following update: (15); ; in The gradient of the reward advantage function is represented by equation (12), and the network update formula is obtained by solving equation (12) using convex optimization methods: (16); when When the security regulatory trust domain and the KL divergence trust domain overlap, maximizing the expected reward value during the update might lead the policy into a relatively dangerous region. In this case, method 2 is applied. Method 2 updates in two steps: a reward enhancement step and a projection step. The reward enhancement step maximizes the reward value using the TRPO algorithm, and the projection step projects the policy network after the reward enhancement step into the security regulatory trust domain. First, the advantage function is maximized within the KL divergence trust domain. : (17); ; Secondly, by minimizing intermediate strategies Distance from the security regulatory trust domain will influence the strategy. Projected into the security supervision trust domain: (18); ; Apply the second-order approximation KL divergence constraint to By linearizing the objective function at a given point, we obtain an approximation for the reward enhancement step update: (19); ; Measured by KL divergence The distance to the security regulatory trust domain is also approximated by the second-order KL divergence pair. Linearize the cost constraints at the location: (20); ; Solve equations (16) and (17) using convex optimization methods: (twenty one); like and This indicates that the current policy is in a relatively dangerous state and the KL divergence trust domain does not overlap with the security oversight trust domain. In this case, updating the policy in any direction will not result in a safe state. Therefore, method 3 will be used to update the policy neural network, that is, a linear backtracking method will be used to update the policy neural network to minimize the security oversight value. (twenty two).

[0014] The vehicle adaptive cruise control method based on projection constraint strategy optimization provided in this embodiment of the invention has the following beneficial effects: (1) A safety supervision perception reinforcement learning method, namely projection constraint policy optimization, is proposed and used to train the vehicle's adaptive cruise strategy. This method introduces safety supervision constraints on the basis of near-end policy optimization, divides the safety level during policy update, and applies different policy update methods to different safety levels. Compared with traditional control methods (such as MPC), the model solution time is significantly reduced, solving the problem of high computational power required by the model. Compared with the adaptive cruise strategy trained by current reinforcement learning methods, this method achieves zero collision rate for the first time, proving the effectiveness of the safety supervision trust domain, solving the long-tail effect of safety reinforcement learning methods, and enabling the safety reinforcement learning method to move from theory to practice in the field of adaptive cruise control strategy. (2) The adaptive cruise method based on projection constraint strategy optimization explores the design of optimized reward function, risk function, state space and vehicle behavior space. The reward function considers the vehicle's traffic efficiency and comfort, and the risk function considers collision risk and safe distance violation risk, so that the adaptive cruise strategy under reasonable guidance can have robustness, safety, high comfort and high efficiency performance. Attached Figure Description

[0015] Figure 1 A schematic diagram of a vehicle adaptive cruise control method based on projection constraint strategy optimization provided in an embodiment of the present invention; Figure 2 The comparison results of PCPO-ACC in terms of average round reward; Figure 3 The comparison results of PCPO-ACC in terms of safety regulatory values; Figure 4 The comparison results of PCPO-ACC in terms of acceleration; Figure 5 The comparison results of PCPO-ACC in terms of collision rate; Figure 6 This is a schematic diagram of the emergency braking operation. Figure 7 A schematic diagram illustrating a smooth following operation. Figure 8 To control the real-time speed map of the vehicle and the target vehicle; Figure 9 Real-time acceleration graphs of the control vehicle and the target vehicle; Figure 10 To control the collision distance diagram between the vehicle and the target vehicle; Figure 11 A graph showing the difference between the controlled vehicle and its set expected speed; Figure 12 This is a schematic diagram of the working conditions before the vehicle enters the field. Figure 13 This is a schematic diagram of the working conditions after the vehicle enters the field; Figure 14 To control the real-time speed map of the vehicle and the target vehicle; Figure 15 Real-time acceleration graphs of the control vehicle and the target vehicle; Figure 16 To control the collision distance between the vehicle and the target vehicle; Figure 17 To control the difference between the vehicle's speed and its set expected speed. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0018] like Figure 1 As shown, an embodiment of the present invention provides a vehicle adaptive cruise control method based on projection constraint strategy optimization, which includes the following specific steps: Step 1, Strategy Trajectory Sampling: Step 1.1: Setting up the simulation environment: In the Carla simulator, a simulation environment is set up. The selected road scenario should be such that a single attempt by the agent can include five conditions in adaptive cruise control: emergency braking of the vehicle in front, vehicle entry, vehicle exit, smooth following, and constant speed cruise. In order to simulate the ACC scenario in real traffic flow, each attempt by the agent will include smooth following and constant speed cruise conditions, and random occurrence of emergency braking of the vehicle in front, vehicle entry, and vehicle exit conditions. Step 1.2: Define the state space and behavior space: Define a single-agent environment, where the state space includes the speed of the target vehicle. Follow the vehicle's speed Distance between the two vehicles The state space is The desired speed of the following vehicle is controlled as the primary way to adjust the state of the following vehicle according to the strategy; therefore, the behavior space is defined as follows: ,in This represents the desired speed for controlling the vehicle at the next time step; Step 1.3, Design of Reward Function and Security Monitoring Function: The reward function is related to the speed, acceleration, and whether a safety accident has occurred of the following vehicle. The reward function guides the strategy to make the following vehicle update in the direction of safety, comfort, and efficiency. The safety supervision function is related to the distance between the target vehicle and the following vehicle and whether a collision has occurred. Step 1.4, Trajectory Sampling Process: Sampling trajectory Including the current time step The state space below Behavioral space Reward Value Safety supervision value and the state space of the next time step , represented as ,in This represents the total number of time steps for each trajectory sampling; the agent needs to record each time step during exploration. When the total number of exploration steps reaches One trajectory sampling process is completed in a short time. Step 2: Strategy Evaluation and Optimization The strategy evaluation and optimization are based on three neural networks: a policy neural network, a value-based neural network, and a value-based safety oversight neural network. After the trajectory sampling process is completed, the policy neural network first updates its network parameters. Then, the new policy neural network maps a new behavior space based on the state space in the trajectory. The value-based neural network and the value-based safety oversight neural network evaluate whether the updated network parameters can achieve a higher score and a lower safety oversight value based on the behavior space. If so, the new policy neural network parameters are applied; otherwise, this process is repeated until a higher score and a lower safety oversight value are achieved.

[0019] In a preferred embodiment of the present invention, in step 1.1, a long ring road is selected as the road scenario so that a single attempt by the intelligent agent can include five working conditions in adaptive cruise control: emergency braking of the vehicle in front, vehicle entry, vehicle exit, smooth following, and constant speed cruise.

[0020] In a preferred embodiment of the present invention, step 1.1 includes the following specific steps: The simulation environment was built using the "Town5" map on the Carla simulator, featuring a ring road. Five scenarios were simulated: emergency braking of the vehicle ahead, vehicle entry, vehicle exit, smooth following, and cruise control. These scenarios were implemented using four vehicles: two obstacle vehicles that could potentially enter the main road, one obstacle vehicle already on the main road, and one control vehicle already on the main road. The control vehicle's speed was controlled in real-time using the PCPO-ACC strategy. A successful exploration by the agent required the control vehicle to complete the entire ring road journey comfortably and efficiently without colliding with the obstacle vehicles. To simulate ACC scenarios in real traffic conditions, emergency braking of the vehicle ahead, vehicle entry, and vehicle exit were randomized using the `random` function in the NumPy package of Python. The randomness of emergency braking and vehicle entry / exit was independent. The emergency braking scenario was initiated by the obstacle vehicle on the main road, with a 50% probability of occurring in each exploration. When emergency braking occurred, the expected speed of the obstacle vehicle on the main road at the next time step would change. v break The magnitude of its acceleration is controlled by a PID controller in the Carla simulator. Vehicle entry and exit scenarios occur in a set during the same exploration. When a vehicle entry scenario occurs, the obstacle vehicles on the main road will first create enough distance from the control vehicle so that obstacle vehicles not on the main road can enter the main road. Each exploration may have 0, 1, or 2 vehicle entry and exit scenarios. If 2 scenarios occur, they are created by the corresponding two obstacle vehicles not on the main road. In order to ensure that the trained strategy has strong robustness, the position of the obstacle vehicle entering the main road is also random.

[0021] In a preferred embodiment of the present invention, in step 1.2, in the state space, and Obtained by the get_velocity() function in the Carla simulator. d The distance() function in the Carla simulator is used to obtain the acceleration of the vehicle in the behavior space, which is controlled by the PID controller in the Carla simulator in combination with the desired speed control.

[0022] In a preferred embodiment of the present invention, the design of the reward function in step 1.3 includes three aspects: safety, comfort, and efficiency; among which safety is more important and therefore accounts for a larger proportion; the reward for safety includes TTC violation reward and collision reward, defined as follows: (1); The TTC violation reward is as follows: (2); If a safety distance violation occurs at the current time step, then ,otherwise ; Collision rewards are: (3); If a collision occurs during an exploration, If no collision occurs during the entire exploration, then .

[0023] Vehicle comfort is strongly correlated with acceleration, so the reward function for comfort is defined as follows: (4); in To control the vehicle in time step acceleration, Discount factor; The reward function for vehicle efficiency is defined as follows: (5); in To control the vehicle's intended cruising speed, Discount factor; In summary, the reward function is: (6); The security policing function is related to TTC violation and whether a collision occurs. The security policing function related to TTC violation is: (7); If a safety distance violation occurs at the current time step, then ,otherwise The safety monitoring function related to whether a collision has occurred is: (8); If a collision occurs during an exploration, If no collision occurs during the entire exploration, then .

[0024] In summary, the security supervision function is: (9).

[0025] In a preferred embodiment of the present invention, step 2 includes the following specific steps: Agent in policy Next collection Trajectory at each time step The method for calculating the parameter values ​​of the policy neural network for each update is as follows: The goal of the updated policy neural network is to maximize the expected reward, expressed as: (10); At the same time, it is necessary to ensure that the updated strategy meets the security regulatory threshold constraints. (11); in To determine the number of security regulatory functions, Kakade and Langford provided an identity using strategies. Safety regulatory advantage function representation strategy Safety supervision expectations: (12); Combining formulas (11) and (12), we can obtain the strategy. The updatable set that satisfies the safety regulatory threshold is represented as: (13); During strategy optimization, different security oversight levels will inevitably occur. To improve the efficiency of strategy training, different optimization methods are used for different security oversight levels. The policy neural network employs three optimization methods: Method 1 is used when the current strategy is at an extremely low security oversight level; Method 2 is used when the current strategy is at a high security oversight level but strategy optimization can lower the security oversight level of the new strategy; and Method 3 is used when the current strategy is at a high security oversight level and updating the strategy in any direction cannot lower the security oversight level of the new strategy. The algorithm uses three metrics and Determine the security monitoring level of the current and updated policy, among which , representing the policy gradient of the safety regulatory value function, a larger one The scope of security oversight has changed significantly after the strategy update; This represents the degree of closeness between the current strategy's security oversight level and the security oversight threshold. A positive value indicates that the current strategy's security oversight level is above the threshold and the security oversight level is relatively high, while a negative value indicates that the current strategy's security oversight level is below the threshold and the security oversight level is relatively low. ,in The Hessian matrix representing the KL divergence between the old and new strategies. A positive value indicates that the KL divergence trust domain and the security regulatory constraint trust domain are in an overlapping state, while a negative value indicates that the KL divergence trust domain and the security regulatory constraint trust domain are in an inclusive state or an unrelated state.

[0026] when Extremely small This indicates that the current policy and its surrounding neighborhood are in a state of low danger potential, and updating the policy in any direction will not pose a danger; when and The statement indicates that the KL divergence trust region of the current policy is within the safe regulatory trust region. In this case, updating the policy in any direction will not be dangerous. For these two situations, method 1, the Trust Region Policy Optimization (TRPO) algorithm, is used to update the policy network parameters. TRPO is an improvement on the traditional policy gradient algorithm. Since excessive policy changes can cause drastic fluctuations in learning performance and thus affect the agent's learning effect, TRPO limits the maximum change in policy during each update by using the most recently iterated policy... Iteratively update by maximizing the reward advantage function over a local neighborhood: (14); ; Based on the second-order approximation of the KL divergence constraint... The objective function at point is linearized, resulting in the following update: (15); ; in The gradient of the reward advantage function is represented by equation (12), and the network update formula is obtained by solving equation (12) using convex optimization methods: (16); when When the security regulatory trust domain and the KL divergence trust domain overlap, maximizing the expected reward value during the update might lead the policy into a relatively dangerous region. In this case, the algorithm applies Method 2 to update the policy neural network, which adds a step to TRPO to ensure that the updated policy meets the security regulatory threshold constraint. Method 2 update consists of two steps: a reward enhancement step and a projection step. The reward enhancement step maximizes the reward value using the TRPO algorithm, and the projection step projects the policy network after the reward enhancement step into the security regulatory trust domain. First, this algorithm maximizes the advantage function within the KL divergence trust domain. : (17); ; Secondly, by minimizing intermediate strategies Distance from the security regulatory trust domain will influence the strategy. Projected into the security supervision trust domain: (18); ; Apply the second-order approximation KL divergence constraint to By linearizing the objective function at a given point, we obtain an approximation for the reward enhancement step update: (19); ; Measured by KL divergence The distance to the security regulatory trust domain is also approximated by the second-order KL divergence pair. Linearize the cost constraints at the location: (20); ; Solve equations (16) and (17) using convex optimization methods: (twenty one); like and This indicates that the current policy is in a relatively dangerous state and the KL divergence trust domain does not overlap with the security oversight trust domain. In this case, updating the policy in any direction will not result in a safe state, and method 3 will be used to update the policy neural network. In this situation, using method 2 to update the policy neural network will result in a convex optimization problem with no solution. Therefore, the policy neural network update method will be changed, using a linear backtracking method to update the policy neural network to minimize the security oversight value. (twenty two).

[0027] As a preferred embodiment of the present invention, the above method was verified through simulation. The simulation platform was Carla version 0.9.11, and a reinforcement learning model was built based on the PyTorch framework. The GPU device was an NVIDIA GeForce RTX 3080Ti. The simulation environment included five scenarios in adaptive cruise control: emergency braking of the vehicle in front, vehicle entry, vehicle exit, smooth following, and cruise control. Since the scenarios of vehicle exit and cruise control do not pose a danger, the analysis focused on the other four scenarios. To simulate real traffic flow, emergency braking of the vehicle in front and vehicle entry were set as occasional scenarios. The expected cruise speed of the vehicle was set to... Each time step Strategy and value neural networks are Each strategy iteration collects data. A sample size is used, and the reward and safety supervision functions are optimized using the Adam optimizer. The learning rate linearly decays from 1e-3 to 0, and the training algorithm stops after 2000 iterations. The following sections describe the policy training process and the performance analysis after policy deployment.

[0028] (a) Training process analysis, such as Figure 2-5 As shown (in) Figure 2-5 In the diagram, the solid line represents the mean of the training curve, and the shaded area represents the variance. Regarding average reward, the reward value rises rapidly in the first 100 rounds. From rounds 100 to 500, the reward value oscillates around zero with a large amplitude. This phenomenon is due to the random occurrence of obstacles such as emergency braking of the vehicle in front, vehicle entry, and vehicle exit in the simulation environment. If these obstacle situations occur less frequently, the score is higher; if they occur more frequently, the score is lower. Starting from round 500, the agent gradually begins to learn a more robust strategy, finally converging at round 1250. The score curve becomes relatively stable with small variance, at which point controlling the vehicle can prevent accidents.

[0029] Regarding the safety supervision value, it drops rapidly in the first 100 rounds, and remains at a relatively high value between the 100th and 1250th rounds. The reason for this phenomenon is that the safety supervision value generated by vehicle collisions accounts for the majority of the safety supervision function. Therefore, as long as a vehicle collision occurs, a high safety supervision value will be generated until the strategy can completely avoid collisions in the 1250th round, at which point the safety supervision value drops to almost zero and remains virtually unchanged.

[0030] Regarding acceleration, it steadily decreased until reaching its lowest point at round 1250, indicating that the strategy's performance in terms of acceleration was continuously improving.

[0031] The average collision rate curve is similar to the safety regulatory value curve, fluctuating significantly before the agent learns a robust policy, but the overall trend is downward. Once the policy neural network converges, no collisions occur after updating the policy neural network based on the new sampled trajectories, demonstrating the safety of the PCPO-ACC policy.

[0032] (II) Performance analysis of emergency braking and smooth following conditions of the vehicle in front: Emergency braking conditions of the vehicle in front, such as Figure 6 As shown, the smooth following condition is as follows: Figure 7 As shown. In Figure 8-11 In the process, the first 100 time steps represent the preceding vehicle entering the operating condition, and the last 75 time steps represent the stable following condition. Figure 8 and Figure 9 In this context, Target Vehicle represents the obstacle vehicle, and Host Vehicle represents the control vehicle; from Figure 8 It can be seen that when the vehicle in front brakes suddenly, the speed of the controlled vehicle decreases in tandem with the speed of the obstacle vehicle. Once the target vehicle's speed stabilizes, the speed of the controlled vehicle also gradually stabilizes. From Figure 9 It can be seen that, except for the peak value during emergency braking, the acceleration is relatively stable, indicating that the strategy provides good comfort. Figure 10 The dashed line represents the safety threshold, and it can be seen that TTC has consistently remained above the safety threshold. From... Figure 11 It can be seen that the difference between the speed and the expected speed gradually reaches 0 after the speed of the obstacle vehicle stabilizes.

[0033] (III) Performance analysis of the vehicle under operating conditions: The working condition is as follows: Figure 12 and 13 As shown, Figure 14 and 15 In this context, "Target Vehicle" represents the obstacle vehicle, and its value is the speed of the nearest obstacle vehicle simultaneously ahead of the controlling vehicle on the main road. "Host Vehicle" represents the controlling vehicle. Figure 14 It can be seen that the obstacle vehicle entered the main road at time step 55. Before entering the main road, it was in a stable following condition. When entering the main road, the vehicle's speed was reduced to prevent collision with the obstacle vehicle. Figure 15 In the above data, except for the moment when the vehicle in front enters, the acceleration is relatively small and smooth, indicating good comfort. Figure 16 In the diagram, the dashed line represents the TTC safety threshold. It can be seen that the value consistently remains above this threshold, indicating a relatively safe strategy. Figure 17 It can be seen that once the vehicle enters the area, the vehicle speed is gradually controlled to reach the expected cruising speed, indicating that the strategy is highly efficient.

[0034] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A vehicle adaptive cruise control method based on projection constraint policy optimization, characterized in that, The specific steps include the following: Step 1, Strategy Trajectory Sampling: Step 1.1, Setting up the simulation environment: Set up the simulation environment in the Carla simulator. The road scene should allow the agent to perform five scenarios in a single attempt of adaptive cruise control: emergency braking of the vehicle in front, vehicle entering, vehicle exiting, smooth following, and constant speed cruise. Each attempt of the agent will include the smooth following and constant speed cruise scenarios, and the emergency braking of the vehicle in front, vehicle entering, and vehicle exiting scenarios will occur randomly. Step 1.2, Define state space and action space: the state space includes the speed of the target vehicle , the speed of the following vehicle , and the distance between the two vehicles , the state space is ; the desired speed of the control following vehicle is taken as the way to adjust the state of the following vehicle, and the action space is defined as , where represents the desired speed of the control vehicle at the next time step; Step 1.3, Design of Reward Function and Safety Monitoring Function: The reward function is related to the speed and acceleration of the following vehicle and whether a safety accident has occurred. The reward function guides the strategy to make the following vehicle update in the direction of safety, comfort and efficiency; the safety monitoring function is related to the distance between the target vehicle and the following vehicle and whether a collision has occurred. Step 1.4, trajectory sampling process: sampled trajectory including the state space under the current time step , the action space , the reward value , the safety supervision value and the state space of the next time step , denoted as , wherein represents the total number of time steps for each trajectory sampling; each time step needs to record when the total number of exploration time steps reaches , a trajectory sampling process is completed; Step 2, Strategy Evaluation and Optimization: Based on three neural networks: a policy neural network, a value-based neural network, and a value-based security oversight neural network; after the trajectory sampling process is completed, the policy neural network will update its network parameters. Then, the new policy neural network will map a new behavior space based on the state space in the trajectory. The value-based neural network and the value-based security oversight neural network will evaluate whether the updated network parameters can obtain a higher score and a lower security oversight value based on the behavior space. If so, the new policy neural network parameters will be applied. If not, this process will be repeated until a higher score and a lower security oversight value appear.

2. The vehicle adaptive cruise control method based on projection constraint strategy optimization according to claim 1, characterized in that, Step 1.1 includes the following specific steps: A ring road is constructed based on the map of the Carla simulator. The simulation environment includes five scenarios: emergency braking of the vehicle in front, vehicle entry, vehicle exit, smooth following, and cruise control. These five scenarios are implemented using four vehicles: two obstacle vehicles that may enter the main road at any time, one obstacle vehicle traveling on the main road, and one control vehicle traveling on the main road. The speed of the control vehicle is controlled in real time by the PCPO-ACC strategy. The scenarios of emergency braking of the vehicle in front, vehicle entry, and vehicle exit will occur randomly, implemented using the `random` function in the `numpy` package of the Python environment. The randomness of emergency braking of the vehicle in front and vehicle entry and exit is independent. The emergency braking scenario is initiated by the obstacle vehicle traveling on the main road, with a 50% probability of occurrence in each exploration. When emergency braking occurs, the expected speed of the obstacle vehicle on the main road in the next time step will become... v break The magnitude of its acceleration is controlled by a PID controller in the Carla simulator; vehicle entry and exit scenarios occur in a set during the same exploration. During each exploration, the number of vehicle entry and exit scenarios is 0, 1, or 2. If they occur twice, they are caused by two obstacle vehicles that are not traveling on the main road. The position of the obstacle vehicles entering the main road is also random.

3. The vehicle adaptive cruise control method based on projection constraint strategy optimization according to claim 1, characterized in that, In step 1.2, the state space and Obtained by the get_velocity() function in the Carla simulator. d The distance() function in the Carla simulator is used to obtain the acceleration of the vehicle in the behavior space, which is controlled by the PID controller in the Carla simulator in combination with the desired speed control.

4. The vehicle adaptive cruise control method based on projection constraint strategy optimization according to claim 1, characterized in that, In step 1.3, the design of the reward function includes three aspects: safety, comfort, and efficiency; the safety reward includes TTC violation reward and collision reward, defined as follows: （1）； The TTC violation reward is as follows: （2）； If a safety distance violation occurs at the current time step, then ,otherwise ; Collision rewards are: （3）； If a collision occurs during an exploration, If no collision occurs during the entire exploration, then ; Vehicle comfort is strongly correlated with acceleration, so the reward function for comfort is defined as follows: （4）； in To control the vehicle in time steps acceleration, Discount factor; The reward function for vehicle efficiency is defined as follows: （5）； in To control the vehicle's intended cruising speed, Discount factor; In summary, the reward function is: （6）； The security policing function is related to TTC violation and whether a collision occurs. The security policing function related to TTC violation is: （7）； If a safety distance violation occurs at the current time step, then ,otherwise The safety monitoring function related to whether a collision has occurred is: （8）； If a collision occurs during an exploration, If no collision occurs during the entire exploration, then ; In summary, the security supervision function is: （9）。 5. The vehicle adaptive cruise control method based on projection constraint strategy optimization according to claim 4, characterized in that, Step 2 includes the following specific steps: Agent in policy Next collection Trajectory at each time step The method for calculating the parameter values ​​of the policy neural network for each update is as follows: The goal of the updated policy neural network is to maximize the expected reward, expressed as: （10）； At the same time, it is necessary to ensure that the updated strategy meets the security regulatory threshold constraints: （11）； in The number of security regulatory functions, using a strategy. Safety regulatory advantage function representation strategy Safety supervision expectations: （12）； The strategy is obtained by combining formulas (11) and (12). The updatable set that satisfies the safety regulatory threshold is represented as: （13）。 6. The vehicle adaptive cruise control method based on projection constraint strategy optimization according to claim 5, characterized in that, In step 2, three indicators are used. and Determine the security monitoring level of the current and updated policy, among which , representing the policy gradient of the security regulatory value function; This represents the degree of closeness between the current strategy's security oversight level and the security oversight threshold. A positive value indicates that the current strategy's security oversight level is above the threshold, indicating a high level of security oversight, while a negative value indicates that the current strategy's security oversight level is below the threshold, indicating a low level of security oversight. ,in The Hessian matrix representing the KL divergence between the old and new strategies. A positive value indicates that the KL divergence trust domain and the security regulatory constraint trust domain are in an overlapping state, while a negative value indicates that the KL divergence trust domain and the security regulatory constraint trust domain are in an inclusive state or an unrelated state. when This indicates that the current policy and its surrounding neighborhood are in a state of low danger potential, and updating the policy in any direction will not pose a danger; when and When the KL divergence trust domain of the current policy is within the security oversight trust domain, updating the policy in any direction will not pose a risk. In both cases, TRPO is used to update the policy network parameters. TRPO limits the maximum change in the policy during each update by using the most recently iterated policy... Iteratively update by maximizing the reward advantage function over a local neighborhood: （14）； ； Based on the second-order approximation of the KL divergence constraint... The objective function at point is linearized, resulting in the following update: （15）； ； in The gradient of the reward advantage function is represented by equation (12), and the network update formula is obtained by solving equation (12) using convex optimization methods: （16）； when The context indicates that the security regulatory trust domain and the KL divergence trust domain are intersecting. In this case, maximizing the expected reward value during the update may cause the policy to enter a relatively dangerous region. This type of update is divided into a reward enhancement step and a projection step. The reward enhancement step maximizes the reward value through TRPO, and the projection step projects the policy network after the reward enhancement step into the security regulatory trust domain. First, the advantage function is maximized within the KL divergence trust domain. : （17）； ； Secondly, by minimizing intermediate strategies Distance from the security regulatory trust domain will influence the strategy. Projected into the security supervision trust domain: （18）； ； Apply the second-order approximation KL divergence constraint to By linearizing the objective function at a given point, we obtain an approximation for the reward enhancement step update: （19）； ； Measured by KL divergence The distance to the security regulatory trust domain is also approximated by the second-order KL divergence pair. Linearize the cost constraints at the location: （20）； ； Solve equations (16) and (17) using convex optimization methods: （21）； like and This indicates that the current policy is in a dangerous state and the KL divergence trust domain does not overlap with the safety oversight trust domain. In this case, updating the policy in any direction will not result in a safe state. A linear backtracking method will be used to update the policy neural network to minimize the safety oversight value. （22）。

Images