A method and system for optimizing vehicle trajectories at intersections based on offline reinforcement learning

Abstract

This invention discloses a method and system for optimizing vehicle trajectories at intersections based on offline reinforcement learning. The specific steps are as follows: A communication scenario between traffic lights and connected vehicles at an intersection is established under an intelligent connected vehicle environment. A communication distance threshold between vehicles and traffic lights is set within the scenario. When a vehicle enters the control area of ​​the traffic light, both parties can exchange information. The operating trajectories of connected vehicles and signal phase timing data at the signalized intersection are collected. The vehicle's driving process at the intersection is abstracted as a Markov decision process. Vehicle travel time, vehicle energy consumption, and collision time are used as parameters of the Markov decision reward function. A suitable dataset is constructed. The vehicle agent is trained offline using reinforcement learning to obtain a suitable intersection traffic strategy. This invention can be applied to vehicle control at intersections, controlling vehicle acceleration to enable vehicles to travel along the optimal trajectory, thereby achieving energy saving, emission reduction, and improved safety performance.

Description

Technical Field

[0001] This invention belongs to the field of vehicle-road cooperative intelligent transportation road control technology, and particularly relates to a method and system for optimizing vehicle trajectories at intersections based on offline reinforcement learning. Background Technology

[0002] With urban traffic demand increasing year by year, urban roads are suffering from severe traffic congestion, which in turn exacerbates energy consumption and environmental degradation. Intersections are the bottlenecks in urban road traffic network management, and improving vehicle management at major road intersections plays a crucial role in the efficiency of the traffic system.

[0003] Traditional traffic control strategies include timed signal control, adaptive signal control, and vehicle-driven control. Timed signal control uses historical traffic data to calibrate the signal light parameters of pre-installed facilities, including phase sequence, cycle length, and green light duration. Vehicle-driven and adaptive signal control apply detection equipment in the infrastructure to collect real-time traffic data and adjust signal light durations according to time-varying traffic demand.

[0004] With the development of connected and autonomous vehicle (CAV) technology, V2V (vehicle-to-vehicle) and V2I (vehicle-to-infrastructure) communication have provided entirely new datasets for traffic management. This bilateral communication enables real-time transmission of traffic information to vehicles and optimization of trajectories, while simultaneously collecting precise vehicle trajectory data. Real-time data communication can optimize signal phase and timing by transmitting trajectory data to traffic lights; alternatively, assuming fixed signal phase and timing, traffic light data can be used to optimize vehicle trajectories by controlling vehicle speed or acceleration, thereby reducing energy consumption, vehicle emissions, and delays, and improving vehicle safety. Typically, optimization control problems treat vehicle position and speed as state variables and acceleration as control variables. However, under complex constraints and multi-objective conditions, the computational complexity of optimization control methods is high. One approach is to discretize the time and state space, transforming the problem into a multi-stage decision problem; another approach is to divide the vehicle trajectory into different parts with continuous acceleration, thereby alleviating computational pressure.

[0005] Meanwhile, intersection control methods based on DRL (Deep Reinforcement Learning) offer a new approach to addressing the aforementioned challenges. Vehicles or traffic lights are used as reinforcement learning agents. These agents continuously update their policies based on their own actions and changes in the intersection's state, selecting the action with the highest reward to optimize energy-saving control of the intersection. However, in online reinforcement learning, the agent needs to constantly interact with the environment during training, updating its policy and selecting actions based on the latest feedback data. Since the agent may choose dangerous actions leading to traffic accidents when it is not yet mature during online learning, and real-time policy updates have high computational complexity and speed requirements, potentially leading to untimely policy updates, this paper considers offline reinforcement learning methods. Existing intersection vehicle trajectory data is used for agent training. The vehicle agent only needs to select the optimal policy from the formed policy space based on the current state to control the vehicles at the intersection. This method reduces computational complexity, optimizes vehicle trajectories, and achieves energy saving, emission reduction, and improved safety and efficiency. Summary of the Invention

[0006] The technical problem to be solved by this invention is: an intersection vehicle trajectory optimization method based on offline reinforcement learning, which collects intersection vehicle traffic trajectory data and signal phase data in an intelligent connected environment, and constructs a static dataset for training vehicle control strategies; within the communication range of traffic lights and connected vehicles, the vehicle agent adopts the optimal traffic trajectory provided by the intersection traffic strategy trained by the constructed dataset to optimize intersection vehicle control.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] This invention proposes a method for optimizing vehicle trajectories at intersections based on offline reinforcement learning, comprising the following steps:

[0009] S1. Establish a communication scenario between traffic lights and connected vehicles at intersections in an intelligent connected environment. Based on the distance between intersections, set a communication distance threshold between vehicles and traffic lights, generally in the range of 100 to 300 meters. When a vehicle enters the control range of the traffic light, the two parties can exchange information.

[0010] S2. Collect the operation data of connected vehicles at signalized intersections and the SPaT (Signal Phase and Timing) information of traffic lights to construct a static dataset.

[0011] S3. Simulate the vehicle's driving process at the intersection as a Markov decision process. Train the vehicle agent offline using reinforcement learning methods on the static dataset constructed in step S2 to obtain an applicable intersection passage strategy.

[0012] S4. Apply the trained strategy to vehicle control at the intersection, select the most suitable acceleration based on the observed environmental state data, and control the vehicle acceleration to make the vehicle run along the optimal trajectory.

[0013] Furthermore, in step S1, the specific process of establishing a communication scenario between intersection traffic lights and connected vehicles in an intelligent connected environment is as follows: In an intelligent connected traffic environment, based on V2I communication, road facilities and connected vehicles exchange vehicle trajectory and traffic light SPAT information through communication equipment; the vehicle operation results under different communication distance thresholds are compared, and a suitable communication distance threshold R is selected. c When the distance between the traffic light and the vehicle is less than R c At this time, the vehicle is within the communication area and can travel according to the strategy provided by the connected intersection system. The communication range at this time is centered on the signalized intersection and extends to R... c A circular area with a radius of 1; otherwise, the vehicle will proceed according to its own following behavior.

[0014] Furthermore, the specific process of step S2 is as follows: Based on the signalized intersection communication scenario established in step S1, the vehicle operation data collected includes the distance, speed, acceleration, and remaining green light time of the vehicle to the intersection stop line.

[0015] Furthermore, in step S3, the vehicle's operational behavior is abstracted as a decision-making process with Markov properties. Therefore, the state of a vehicle at an intersection at a given moment depends only on its state and behavior at the previous moment, and this decision-making process progresses continuously over time. Treating the vehicle as an agent, there is a continuous interaction between the agent and the environment. Offline reinforcement learning achieves the acquisition of applicable policies based solely on the collected dataset, without the agent interacting with the environment. Offline reinforcement learning avoids the dangers arising from the agent's interaction with the environment or the delays caused by statistical agent rewards. The specific steps are as follows:

[0016] S301. Simulate the process of a vehicle passing through an intersection as a Markov decision process. The specific process of a Markov decision is described by a quintuple (S, A, R, P, γ), where: S and A are the sets of environmental states and agent actions; R is the reward function, which can depend on the environmental state and agent actions or only on the state; P is the state transition function, which represents the probability of reaching another state after performing an action in a certain state; γ is the discount factor, used to measure the relative importance of the current reward and the long-term future reward.

[0017] S302. Treat the connected vehicle as an intelligent agent, and define its state S using a four-dimensional vector, with the specific formula as follows:

[0018] s t =[dr (t), v r (t), φ(t), G(t)] T ;

[0019] Where, d r (t) represents the vehicle's position; v r (t) represents the vehicle speed, provided by the connected vehicle; φ(t) represents the remaining time of the green light, and if the vehicle is traveling in the direction of the red light, the value of φ(t) is 0; G(t) is the remaining time of the red light, provided by the SPAT information transmitted by V2I communication.

[0020] S303, agent action A represents the vehicle's acceleration control, i.e., the acceleration limit a. t ∈[d m u m The continuous action space within the range; where d m u is the lower limit of acceleration. m This is the upper limit for acceleration. In addition to the acceleration limit, the acceleration also meets the vehicle's safety requirements, namely, to avoid rear-end collisions or exceeding the speed limit.

[0021] S304. Define a reward function R. To improve vehicle safety performance, it is necessary to control the vehicle's trajectory at intersections, mitigate congestion and stop-and-start behavior at intersections, thereby improving traffic efficiency and reducing vehicle energy consumption. Therefore, the time it takes for a vehicle to pass through an intersection, the energy consumption of the vehicle passing through an intersection, and TTC (Time-to-Collision) are considered as indicators of the reward function. TTC represents the time required for the following vehicle to catch up with the preceding vehicle and collide with it while maintaining the current speed difference between the preceding and following vehicles.

[0022]

[0023] Among them, TTC i (t) represents the collision time of the i-th vehicle at second t, x i v represents the position of the i-th car. i Let represent the speed of the i-th vehicle, and l represent the length of the vehicle ahead of the i-th vehicle. Considering that the aggregated vehicle travel time makes it difficult to allocate rewards to each time step for decision-making on instantaneous vehicle actions, a reward function is defined as follows:

[0024] r t =ω1(x(t)-x(t-1))+ω2F(t)+ω3TTC i (t)

[0025] Where x(t)-x(t-1) represents the distance the vehicle travels in a single time step, F(t) represents the instantaneous fuel consumption of the vehicle, ω1 represents the weight of the travel distance in a single time step, ω2 represents the weight of the vehicle's fuel consumption, and ω3 represents the weight of TTC in the reward function.

[0026] S305. Due to the complexity of signalized intersection scenarios, the state transition probabilities of the environment are difficult to calculate. Therefore, a state transition function P is defined based on a model-free reinforcement learning method. To achieve optimal vehicle control, the vehicle... (The sentence is incomplete and requires more context to translate accurately.) Learning to obtain the optimal strategy π * To maximize expected returns:

[0027]

[0028] Where Θ represents the policy space; Based on the aforementioned environmental conditions, agent actions, and reward definitions, the calculations are performed using vehicle trajectory data and historical traffic light data.

[0029] S306. The discount factor γ is used to measure the relative importance of current rewards and future long-term returns. Its value ranges from [0,1). A γ closer to 1 emphasizes long-term cumulative rewards, while a γ closer to 0 emphasizes short-term rewards. From time point S306 to the final state, the total decay of all rewards equals the return.

[0030]

[0031] S307. Use the CQL (conservative Q-learning) offline reinforcement learning algorithm to eliminate part of the extrapolation error. The Q value is the state value function under a given policy π.

[0032]

[0033] S308. To avoid overestimating the Q-value, CQL penalizes high Q-values ​​in certain states. Within the training iterations, the entropy regularization coefficient α needs to be updated at each step. The specific formula is as follows:

[0034]

[0035] S309. Update function Q, the specific formula is as follows:

[0036]

[0037] in, This is the Bellman operator for strategy π in actual computation.

[0038] S310, Update Strategy π φThe final strategy for vehicle control at this intersection is obtained, and the specific formula is as follows:

[0039]

[0040] Furthermore, in step S4, the specific process of making the vehicle run along the optimal trajectory is as follows: the connected vehicle is deployed as an intelligent agent to the intersection environment. In each departure step, the vehicle selects the corresponding action from the action space according to the strategy proposed in step S3. By controlling the acceleration of the vehicle, the vehicle runs along the optimal trajectory provided by the strategy, thereby reducing vehicle energy consumption and improving vehicle traffic efficiency and safety performance.

[0041] Furthermore, this invention also proposes an intersection vehicle trajectory optimization system based on offline reinforcement learning, comprising:

[0042] The communication scenario module under intelligent connectivity is used to enable information exchange between vehicles and traffic lights when vehicles enter the control range of traffic lights at intersections.

[0043] The dataset module is used to build datasets of vehicle trajectory and traffic light SPaT information, including information such as the distance of the vehicle to the intersection stop line, speed, acceleration, and remaining green light time.

[0044] The Markov decision module is used to simulate the process of a vehicle driving at an intersection as a Markov decision process.

[0045] The intersection traffic strategy module is used to train the vehicle agent offline using reinforcement learning methods to obtain applicable intersection traffic strategies.

[0046] The vehicle control module is used to control the vehicle's acceleration to make the vehicle run along an optimized trajectory.

[0047] Furthermore, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the intersection vehicle trajectory optimization method based on offline reinforcement learning described above.

[0048] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the intersection vehicle trajectory optimization method based on offline reinforcement learning described above.

[0049] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0050] 1. The offline reinforcement learning method used in this invention uses existing intersection vehicle trajectory data for agent training. The vehicle agent only needs to select the optimal policy from the formed policy space according to the current state to control the intersection vehicles, avoiding accidents that may be caused by the vehicle agent rushing around recklessly in the initial training stage, thus improving safety performance.

[0051] 2. This invention utilizes offline reinforcement learning to train vehicle agents, which has low computational complexity and avoids the time consumption caused by data interaction between the agent and the environment, resulting in untimely policy updates, thereby improving the efficiency of the vehicle-road cooperative system in providing control strategies to vehicles.

[0052] 3. The offline reinforcement learning method provided by this invention uses vehicle travel time, vehicle energy consumption, and TTC as parameters of the reward function to jointly optimize the vehicle's economy and safety performance. This can alleviate traffic delays and carbon emissions caused by congestion at urban intersections, thereby improving the operational efficiency of urban intersection systems and achieving energy conservation and emission reduction. Attached Figure Description

[0053] Figure 1 This is the overall flowchart of the present invention.

[0054] Figure 2 This invention describes a networked signal intersection scenario.

[0055] Figure 3 This is a single-vehicle status trajectory collected in an embodiment of the present invention.

[0056] Figure 4 This is a flowchart of the Markov decision process of the present invention.

[0057] Figure 5 This is a static dataset applicable to Markov decision processes in this embodiment of the invention.

[0058] Figure 6 This is the vehicle trajectory obtained by a vehicle according to the optimal strategy in an embodiment of the present invention.

[0059] Figure 7 This is the trend of the total reward after each round according to the CQL algorithm in this embodiment of the invention.

[0060] Figure 8 This is a flowchart illustrating the implementation process of the offline reinforcement learning method of this invention.

[0061] Figure 9 This is a comparison of the optimized vehicle trajectory, speed trajectory, and energy consumption in this embodiment of the invention with the original vehicle following model. Detailed Implementation

[0062] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0063] This invention provides a method for optimizing vehicle trajectories at intersections based on offline reinforcement learning in an intelligent connected environment, and establishes a communication scenario between traffic lights and connected vehicles at intersections in an intelligent connected environment. Figure 1 The following is an overall flowchart of the present invention, and the specific steps are as follows:

[0064] S1. Using the intersection at the entrance of the University of Central Florida in Orlando, Florida as a background, construct a communication scenario between intersection traffic lights and connected vehicles in an intelligent connected environment, such as... Figure 2 As shown, the length of the approach lane is 57 meters. Based on V2I communication, road infrastructure and connected vehicles exchange vehicle trajectory and traffic light SPAT information via communication equipment. A communication distance threshold between vehicles and traffic lights is set based on the distance between intersections. In this embodiment, the communication distance threshold R... c The distance is 300 meters, when the distance between the traffic light and the vehicle is less than R. c At this time, the vehicle is within the communication area and can travel according to the strategy provided by the connected intersection system. The communication range at this time is centered on the signalized intersection and extends to R... c A circular area with a radius of 1; otherwise, the vehicle will proceed according to its own following behavior.

[0065] S2. Collect the operation data of connected vehicles at the signalized intersection and the SPaT information of the traffic lights to construct a static dataset of vehicle trajectory and traffic light SPaT information. The collected vehicle operation data includes the vehicle's distance to the intersection stop line, speed, acceleration, remaining green light time, time until the next green light, and green light status information. In this embodiment, trajectory information is collected for this vehicle at 0.5-second intervals, as detailed below. Figure 3 As shown. Figure 3 This describes the vehicle's driving information and environmental status information recorded at 0.5-second intervals as it moves from the observation range of an intersection to the point of entering the intersection itself. Figure 3 This allows us to obtain environmental and state information of the vehicle's intelligent agent during driving, laying the foundation for its subsequent conversion into a Markov dataset.

[0066] S3. Simulate the vehicle's driving process at the intersection as a Markov decision process. Train the vehicle agent offline using reinforcement learning methods on the static dataset constructed in step S2 to obtain an applicable intersection passage strategy.

[0067] The Markov decision process refers to a process where the state of vehicles at an intersection at a given moment depends only on the state and behavior of the vehicles at the previous moment, and this decision process is continuously evolving over time. Vehicles are treated as intelligent agents, and there is a continuous interaction between the agent and the environment. For example... Figure 4 As shown, the interaction process is as follows: When a vehicle enters the connected intersection environment, the vehicle's intelligent agent selects an action based on the perceived environmental state information. The environment then provides feedback on its state and reward data to the intelligent agent based on the vehicle's selected action. The intelligent agent continues to select actions, and this process continues until the vehicle enters the intersection. The specific steps are as follows:

[0068] S301. Simulate the process of a vehicle passing through an intersection as a Markov decision process. The specific process of a Markov decision is described by a quintuple (S, A, R, P, γ), where: S and A are the sets of environmental states and agent actions; R is the reward function, which can depend on the environmental state and agent actions or only on the state; P is the state transition function, which represents the probability of reaching another state after performing an action in a certain state; γ is the discount factor, used to measure the relative importance of the current reward and the long-term future reward.

[0069] S302. Treat the connected vehicle as an intelligent agent, and define its state S using a four-dimensional vector, with the specific formula as follows:

[0070] s t =[d r (t), v r (t), φ(t), G(t)] T ;

[0071] Where, d r (t) represents the vehicle's position; v r (t) represents the vehicle speed, provided by the connected vehicle; φ(t) represents the remaining time of the green light, and if the vehicle is traveling in the direction of the red light, the value of φ(t) is 0; G(t) is the remaining time of the red light, provided by the SPAT information transmitted by V2I communication.

[0072] S303, agent action A represents the vehicle's acceleration control, i.e., the acceleration limit a. t The continuous motion space within the range of [-4.5, 2.6]. In addition to the acceleration limit, the acceleration also meets the vehicle's safety requirements, namely, to avoid rear-end collisions or exceeding the speed limit.

[0073] S304. Define a reward function R. To improve vehicle safety performance, it is necessary to control the vehicle's trajectory at intersections, mitigate congestion and stop-and-start behavior at intersections, thereby improving traffic efficiency and reducing vehicle energy consumption. Therefore, the time it takes for a vehicle to pass through the intersection and the energy consumption of the vehicle passing through the intersection are considered as indicators of the reward function. Define the reward function:

[0074] r t =ω1(x(t)-x(t-1))+ω2F(t)+ω3TTC i (t)

[0075] Where x(t)-x(t-1) represents the distance the vehicle travels within a single time step, in Figure 3 The middle column represents the difference between each row in the 'dist to stopline' column; F(t) represents the vehicle's instantaneous fuel consumption, i.e. Figure 5 The values ​​in each row of the 'fuels' column are as follows: ω1 represents the weight of travel distance within a single time step, which is 1; ω2 represents the weight of vehicle fuel consumption, which is 2; and ω3 represents the weight of TTC in the reward function.

[0076] S305. Due to the complexity of signalized intersection scenarios, the state transition probabilities of the environment are difficult to calculate. Therefore, a state transition function P is defined based on a model-free reinforcement learning method. To achieve optimal vehicle control, the vehicle... (The sentence is incomplete and requires more context to translate accurately.) Learning to obtain the optimal strategy π * To maximize the expected return. In this embodiment, the vehicle trajectory is transformed into a static dataset, such as... Figure 5 As shown. Figure 5 This describes a Markov decision dataset for agent learning, compiled from collected vehicle trajectory data and SPaT data. The 'observations' column represents the state data. t The 'actions' column represents the agent's action data. t 'tdists' and 'fuels' can be integrated into a reward r t The static dataset is set at 0.5-second intervals. The vehicle agent learns from a Markov dataset of 1000 different vehicles to arrive at the optimal strategy for navigating the intersection. The vehicle trajectory obtained based on the optimal strategy is shown below. Figure 6 After learning all trajectory data, the vehicle's intelligent agent applies the resulting strategy π. * The optimal passage strategy provided for vehicles arriving at the intersection. Figure 6 The data describes the distance, speed, acceleration, and energy consumption of vehicles passing through at 0.5-second intervals, based on the optimal strategy.

[0077]

[0078] Where Θ represents the policy space; Based on the aforementioned environmental conditions, agent actions, and reward definitions, the calculations are performed using vehicle trajectory data and historical traffic light data.

[0079] S306. The discount factor γ is used to measure the relative importance of current rewards and future long-term returns. Its value ranges from [0,1). A γ value closer to 1 emphasizes long-term cumulative rewards, while a γ value closer to 0 emphasizes short-term rewards. In this embodiment, γ = 0.99. From time t to the final state, the sum of the decay of all rewards equals the return. Figure 7 The diagram illustrates the trend of the total reward after each round according to the CQL algorithm. The training process consisted of 100 rounds, and the total reward in each round generally showed an increasing trend. Therefore, it can be concluded that after multiple rounds of learning, the agent obtained a more optimized passage strategy, namely a more efficient and energy-saving intersection passage strategy.

[0080]

[0081] Offline reinforcement learning enables the development of applicable policies using only a collected dataset, without the agent interacting with its environment. Offline reinforcement learning avoids the risks associated with agent-environment interactions and the delays caused by statistical analysis of agent rewards. Figure 8 As shown, the specific steps are as follows:

[0082] S307. Use the CQL offline reinforcement learning algorithm to eliminate part of the extrapolation error. The Q value is the state value function under a given policy π.

[0083]

[0084] The parameters of the CQL algorithm are as follows: the learning rate for the policy network and the target Q network is 0.001, and the learning rate for the Q network is 0.003. The training batch size is 256, and the synchronization coefficient of the target Q network is 0.005.

[0085] S308. To avoid overestimating the Q-value, CQL penalizes high Q-values ​​in certain states. Initialize the Q-network. θ Target Q network Q θ′ and strategy π φ The entropy regularization coefficient α = 1. Within the training iterations, the entropy regularization coefficient needs to be updated at each step. The specific formula is as follows:

[0086]

[0087] S309. Update function Q, the specific formula is as follows:

[0088]

[0089] in, This is the Bellman operator for strategy π in actual computation.

[0090] S310, Update Strategy π φ The final strategy for vehicle control at this intersection is obtained, and the specific formula is as follows:

[0091]

[0092] S4. Apply the trained strategy to vehicle control at the intersection. Based on the observed environmental data, select the most suitable acceleration and control the vehicle acceleration to make the vehicle follow the optimal trajectory. Compare the optimized vehicle trajectory, speed trajectory, and energy consumption with the original car-following model. Figure 9 As shown in the diagram. IDM represents the vehicle's trajectory in normal car-following mode, while CQL represents the vehicle's trajectory optimized using an offline reinforcement learning algorithm. Figure 9 The left-hand diagram illustrates the trajectories of vehicles crossing traffic lights under a standard car-following model and under a strategy optimized by an offline reinforcement learning algorithm. The optimized trajectory allows the vehicle to pass through the intersection at a slower speed during the green light period, while the unoptimized trajectory allows the vehicle to reach the intersection at a higher speed and then stop and wait. Figure 9 The intermediate graph illustrates the speed changes of vehicles under two different strategies during this process. Figure 9 The right-hand diagram illustrates the changes in vehicle energy consumption during this process. It can be concluded that the vehicle optimized through CQL offline reinforcement learning passed through the intersection with a smoother speed and lower energy consumption. Specifically, the connected vehicle is deployed as an intelligent agent in the intersection environment. Within each walk, the vehicle selects the appropriate action from the action space according to the strategy proposed in step S3. By controlling the vehicle's acceleration, the vehicle runs along the optimal trajectory provided by the strategy, thereby reducing vehicle energy consumption and improving traffic efficiency and safety performance.

[0093] This invention also proposes an intersection vehicle trajectory optimization system based on offline reinforcement learning, including a communication scenario module under intelligent connected vehicles, a dataset module, a Markov decision module, an intersection traffic strategy module, a vehicle control module, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention embodiment.

[0094] This invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0095] This invention also proposes a computer-readable storage medium storing a computer program. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0096] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for optimizing vehicle trajectories at intersections based on offline reinforcement learning, characterized in that, Includes the following steps: S1. Establish a communication scenario between traffic lights and connected vehicles at intersections in an intelligent connected environment, set a communication distance threshold between vehicles and traffic lights, and enable information exchange between the two parties when a vehicle enters the control range of the traffic light. S2. Collect the operation data of connected vehicles at signalized intersections and the SPAT information of traffic lights to construct a static dataset; S3. Simulate the vehicle's driving process at the signalized intersection as a Markov decision process. The static data from step S2 is collected offline and used to train the vehicle agent through reinforcement learning to obtain the signalized intersection passage strategy. Specifically: S301. The process of a vehicle passing through a signalized intersection is simulated as a Markov decision process. The Markov decision process is described by a quintuple (S, A, R, P, γ), where: S and A are the sets of environmental states and agent actions; R is the reward function; P is the state transition function; and γ is the discount factor. S302. Treat the connected vehicle as an intelligent agent, and define its state S using a four-dimensional vector, with the specific formula as follows: s t =[d r (t),v r (t),φ(t),G(t)] T ; Where, d r (t) represents the vehicle's position; v r (t) represents the vehicle speed; φ(t) represents the remaining time of the green light. If the vehicle is traveling in the direction of the red light, the value of φ(t) is 0; G(t) is the remaining time of the red light. S303, Agent A represents the vehicle's acceleration control, i.e., the acceleration limit a. t ∈[d m ,u m The continuous action space within the range of d; where d m u is the lower limit of acceleration. m This represents the upper limit of acceleration. S304. The time it takes for a vehicle to pass through a signalized intersection, the energy consumed by the vehicle passing through the signalized intersection, and TTC are used as indicators of the reward function R. TTC represents the time required for the following vehicle to catch up with the preceding vehicle and collide with it while maintaining the current speed difference between the preceding and following vehicles. Among them, TTC i (t) represents the collision time of the i-th vehicle at second t, x i Let v represent the position of the i-th vehicle, v represent the speed of the i-th vehicle, and l represent the length of the vehicle in front of the i-th vehicle. Define the reward function: r t =ω1(x(t)-x(t-1))+ω2F(t)+ω3TTC i (t); Where x(t)-x(t-1) represents the distance the vehicle travels in a single time step, F(t) represents the instantaneous fuel consumption of the vehicle, ω1 represents the weight of the travel distance in a single time step, ω2 represents the weight of the vehicle's fuel consumption, and ω3 represents the weight of TTC in the reward function. S305. A model-free reinforcement learning method defines a state transition function P. The vehicle learns from a static dataset. Learning to obtain the optimal strategy π * The specific formula is as follows: Where Θ represents the policy space; Based on the aforementioned environmental conditions, agent actions, and reward function, the data is calculated using vehicle trajectory data and historical traffic light data. S306, the discount factor γ ranges from [0,1), and the total decay of all rewards from time t to the final state is the return: S307. Use the CQL offline reinforcement learning algorithm to eliminate part of the extrapolation error. The Q value is the state value function under a given policy π. S308 and CQL penalize high Q values. Within the training iterations, the entropy regularization coefficient α needs to be updated at each step. The specific formula is as follows: S309. Update function Q, the specific formula is as follows: in, For the actual computation of strategy π, use the Bellman operator. S310, Update Strategy π φ The strategy for vehicle control at signalized intersections is obtained, and the specific formula is as follows: S4. Apply the trained strategy to vehicle control at signalized intersections, and control vehicle acceleration to make the vehicle run along the optimal trajectory.

2. The intersection vehicle trajectory optimization method based on offline reinforcement learning according to claim 1, characterized in that, In step S1, the specific process of building the communication scenario is as follows: In an intelligent connected transportation environment, based on V2I communication, road facilities and connected vehicles exchange vehicle trajectory and traffic light SPAT information through communication equipment; a communication distance threshold R is defined. c When the distance between the traffic light and the vehicle is less than R c At this time, the vehicle is within the communication area and can travel according to the strategy provided by the signalized intersection system. The communication range at this time is centered on the signalized intersection and extends to R... c A circular area with a radius of 1; otherwise, the vehicle will proceed according to its own following behavior.

3. The intersection vehicle trajectory optimization method based on offline reinforcement learning according to claim 1, characterized in that, The vehicle operation data collected in step S2 includes information such as the distance of the vehicle to the stop line of the signalized intersection, speed, acceleration, and remaining green light time.

4. The intersection vehicle trajectory optimization method based on offline reinforcement learning according to claim 1, characterized in that, In step S4, the specific process of making the vehicle run along the optimal trajectory is as follows: the connected vehicle is deployed as an intelligent agent in the signalized intersection environment. In each walk, the vehicle selects the corresponding action from the action space according to the strategy proposed in step S3, and the vehicle runs along the optimal trajectory provided by the strategy by controlling the vehicle's acceleration.

5. A system applied to the intersection vehicle trajectory optimization method based on offline reinforcement learning as described in claim 1, characterized in that, include: The intelligent connected vehicle communication scenario module is used to enable information exchange between the vehicle and the traffic light when the vehicle enters the control range of the traffic light at a signalized intersection. The dataset module is used to build datasets of vehicle trajectory and traffic light SPaT information, including information such as the distance of the vehicle to the stop line of the signalized intersection, speed, acceleration, and remaining green light time. The Markov decision module is used to simulate the driving process of a vehicle at a signalized intersection as a Markov decision process. The intersection traffic strategy module is used to train the vehicle agent offline using reinforcement learning methods to obtain the signalized intersection traffic strategy. The vehicle control module is used to control the vehicle's acceleration to make the vehicle run along an optimized trajectory.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method of any one of claims 1 to 4.

Images