Method for determining multi-vehicle passing time of unstructured intersection based on reinforcement learning

Abstract

The application relates to the technical field of non-structured road intersection networked automatic driving vehicle cooperative trajectory planning, in particular to a non-structured intersection multi-vehicle cooperative planning passing time determination method based on reinforcement learning, which comprises the following steps: for a specific non-structured intersection, modeling a passing time determination problem as a single-step decision problem, defining a state space, an action space and a reward function; constructing a reinforcement learning training environment, taking an OCP solver as an interactive environment, training a passing time prediction model under a large number of random traffic configurations of the intersection; and deploying the trained model on the cloud control platform of the intersection, real-time predicting and transferring the optimal solution to the OCP solver. The application trains a mapping model from multi-vehicle traffic configuration to optimal passing time for a specific non-structured intersection, and can predict the optimal solution for any newly arrived traffic configuration within milliseconds after the training is completed, thereby realizing the joint optimization of passing efficiency and solution feasibility.

Description

Technical Field

[0001] This invention relates to the field of cooperative trajectory planning technology for connected and automated vehicles at unstructured road intersections, and particularly to the problem of multi-vehicle cooperative trajectory planning at unstructured intersections without lane lines or traffic lights (such as open-pit mines and ports). A reinforcement learning-based method for determining travel time is proposed. When modeling multi-vehicle cooperative trajectory planning as a fixed-terminal-time optimal control problem (OCP), the terminal travel time is... The key preliminary problem of determining the optimal value of the time is addressed by training a time prediction model for a specific intersection using reinforcement learning. The model adaptively outputs the optimal time based on the traffic configuration of multiple vehicles arriving at the intersection in the current batch, providing key parameter settings for the subsequent OCP solver and achieving joint optimization of traffic efficiency and solution feasibility. Background Technology

[0002] In multi-vehicle cooperative trajectory planning at unstructured intersections, modeling the problem as an optimal control problem (OCP) and solving it numerically is a widely adopted approach. To reduce the nonlinearity of the OCP and simplify the solution process, a common and effective practice is to incorporate the travel time... By setting these variables as fixed constants instead of free optimization variables, the additional nonconvexity introduced by free terminal time is avoided. This approach significantly reduces the difficulty of the solution, enabling gradient-based NLP solvers to converge to feasible solutions more efficiently. However, The specific value of has a decisive influence on the solution of OCP: If the limit is too small, the vehicle will be unable to complete the crossing of the intersection within the specified time under the kinematic limit constraint, resulting in no feasible solution for OCP; If the speed is too high, vehicles are forced to travel at speeds below the optimal level, resulting in decreased traffic efficiency and increased energy consumption. Therefore, in a fixed... Within this framework, determining the optimal travel time for the current multi-vehicle traffic configuration that ensures successful OCP solution while maximizing traffic efficiency is a key preliminary decision-making problem in multi-vehicle cooperative trajectory planning.

[0003] Optimal travel time Determining the optimal [structure / structure] faces the challenge of high-dimensional coupling. For a given unstructured intersection, the optimal [structure / structure]... The problem depends on the coupling effect of multiple factors, including the number of vehicles arriving at the intersection in the current batch, the combination of each vehicle's initial pose (position, velocity, heading angle) and the target exit direction, the vehicle's kinematic limit parameters, and the potential conflict density between vehicles. The relationships between these factors are highly nonlinear and difficult to express explicitly using analytical formulas. Existing methods typically employ manual empirical setting of fixed values, conservative settings based on worst-case estimates, or binary search through multiple trial solutions. However, empirical fixed values ​​cannot adapt to changing traffic configurations, conservative estimates sacrifice traffic efficiency, and trial solutions require repeated calls to the OCP solver, multiplying computational costs and failing to meet real-time requirements.

[0004] In the prior art, invention patent CN114283607B discloses a multi-vehicle cooperative planning method based on distributed swarm intelligence learning. It employs a multi-agent reinforcement learning framework to achieve multi-vehicle cooperative trajectory planning, where each agent shares perception information through distributed communication and directly outputs vehicle control commands. Invention patent CN116895163A discloses a multi-vehicle cooperative control method and system based on active traffic light communication. This optimizes multi-vehicle passage speed through information interaction between traffic lights and vehicles, and guides speed based on signal phase cycles to reduce vehicle stops and delays. Invention patent CN115016474A discloses a control method, roadside equipment, cloud control platform, and system for vehicle-road cooperative autonomous driving. This achieves autonomous driving control in a vehicle-road cooperative environment by centrally planning multi-vehicle driving trajectories and issuing control commands through the cloud control platform. However, all of the above methods have the following technical defects in determining passage time: (1) Existing methods will determine the travel time. The values ​​are set to fixed empirical values, without considering the optimal conditions under different traffic configurations. The difference lies in the technical reason: optimal The problem depends on the coupling effects of multiple factors, including the number of vehicles, initial pose, target exit direction, kinematic limits, and conflict density. The relationships between these factors are highly nonlinear and cannot be determined using simple analytical formulas or lookup tables. Fixed empirical values ​​may be feasible in some scenarios, but they can lead to OCP solution failure or a significant decrease in traffic efficiency when traffic configuration deviates from empirical assumptions.

[0005] (2) While the conservative approach based on worst-case estimation can guarantee a feasible solution for OCP in most scenarios, it systematically overestimates the required travel time. The technical reason for this is that the conservative estimate is set according to the most unfavorable traffic configuration (such as the largest number of vehicles and the most intense conflicts). For a large number of traffic configurations with medium or low conflict density in practice, The speed is significantly overestimated, forcing vehicles to travel at speeds far below the optimal level, resulting in a double loss of traffic efficiency and energy consumption.

[0006] (3) The binary search method based on trial-and-error solution can gradually approach the optimal solution. However, this requires repeatedly calling the OCP solver. The technical reason is that each trial requires a complete OCP solution to determine the current state. Is it feasible? Multiple OCP calculations during the search process significantly increase the total computation time. In unstructured intersection scenarios, a single OCP calculation already has high computational complexity, and multiple calculations make real-time performance completely unreliable.

[0007] Therefore, how can we determine the optimal travel time in real time and accurately for the current traffic configuration without repeatedly calling the OCP solver? This is a key issue that urgently needs to be addressed in multi-vehicle cooperative trajectory planning at unstructured intersections. Summary of the Invention

[0008] As mentioned earlier, in multi-vehicle cooperative trajectory planning at unstructured intersections, the problem is modeled as the travel time when the fixed terminal time (OCP) is... The value of has a decisive impact on the feasibility of the solution and traffic efficiency. Existing methods using fixed empirical values ​​cannot adapt to changing traffic configurations, conservative estimations systematically sacrifice traffic efficiency, and trial-and-error searches require repeated calls to the OCP solver, which cannot meet real-time requirements. In view of this, this invention provides a method for determining the travel time of multi-vehicle cooperative planning at unstructured intersections based on reinforcement learning. For a specific unstructured intersection, a method is trained to determine the optimal travel time from multi-vehicle traffic configurations. The mapping model takes the state information and conflict characteristics of the current batch of multiple vehicles as input, and... As the output, the optimal strategy is learned through interactive feedback with the OCP solver. After training, the optimal strategy can be predicted for any newly arriving traffic configuration in milliseconds. It eliminates the need to repeatedly call the OCP solver, achieving joint optimization of traffic efficiency and solution feasibility.

[0009] The complete technical solution of this invention includes: A reinforcement learning-based method for determining travel time in unstructured intersections through multi-vehicle cooperative planning includes the following steps: 1) For specific unstructured intersections, the problem of determining travel time is modeled as a single-step decision problem. A state space, action space, and reward function are established. The state space is responsible for encoding the current batch of multi-vehicle traffic configuration into a state vector s. The action space is responsible for outputting a normalized action a from the policy network based on the state vector s, and linearly mapping it into the physical travel time. The reward function is used to calculate the physical travel time. The reward is calculated when OCP is passed in as a fixed parameter. 2) Construct a reinforcement learning training environment, using the OCP solver as the interactive environment, and train the travel time prediction model under a large number of random traffic configurations at the intersection. The training process of the travel time prediction model is as follows: The state vector s of the current multi-vehicle traffic configuration in the episode is observed, and then the policy network outputs a normalized action a, which is mapped to the physical travel time. Physical travel time Input into the OCP solver, and with Solve for the parameters, calculate the reward r based on the solution and store it in the experience buffer. After accumulating a sufficient number of episodes, perform PPO parameter update and repeat the above process until the policy converges. 3) Finally, the traffic time prediction model trained in step 2) is deployed on the cloud control platform of the intersection. Its traffic configuration features are extracted in real time and encoded into a state vector s. The policy network outputs the optimal value through forward inference. It is then passed to the OCP solver for multi-vehicle collaborative trajectory planning.

[0010] Furthermore, the state space design method is as follows: the state vector s contains the following feature components: the initial position coordinates of each CAV. initial velocity Initial heading angle Target export direction code Kinematic limit parameters of each CAV: maximum speed Maximum acceleration Maximum steering angle Potential conflict density index The potential conflict density index This is the ratio of the number of vehicle pairs that need to pass through the same conflict zone to the total number of vehicle pairs.

[0011] Furthermore, the motion space design method is as follows: The action space is a continuous scalar, and the output passage time is... : (1) In the formula, and These are the lower and upper physical bounds of the travel time, determined by the intersection dimensions and vehicle kinematic limits. The normalized action a ∈ [-1, 1] is output by the policy network and linearly mapped to the physical travel time using formula (2). : (2) in, Determined by dividing the shortest path length at the intersection by the maximum permissible speed for vehicles. It is determined by dividing the longest path length at the intersection by the minimum permissible speed for vehicles.

[0012] Furthermore, the reward function is designed as follows: The total reward function consists of a linear combination of four sub-rewards: (3) In the formula, To solve for the feasibility reward, Incentives for improving traffic flow. As a reward for trajectory quality, To solve for efficiency rewards.

[0013] Furthermore, incentives for feasibility When the OCP solver is given A positive reward is given when a feasible solution is successfully found. A large penalty is imposed when the solution fails. ( ), to severely punish those who fail to solve the problem. Select: (4).

[0014] Furthermore, rewards for improving traffic efficiency. When the OCP solution is successful, As a reward for improved traffic efficiency: (5) A weighting coefficient is assigned to reward traffic efficiency.

[0015] Furthermore, trajectory quality rewards: (6) In the formula, The objective function value of OCP. This represents the trajectory quality weighting coefficient.

[0016] Furthermore, rewards for solution efficiency. : (7) In the formula, The actual computation time of the OCP solver. To solve for the efficiency reward weight coefficient.

[0017] Furthermore, during the training process of the policy network, a traffic configuration is first randomly generated from a preset distribution: the number of vehicles... exist[ Uniform sampling within the intersection; the initial pose and entry direction of each CAV are randomly sampled from the reasonable initial area of ​​each entrance lane of the intersection; the target exit direction is randomly selected from the feasible exits; vehicle size parameters fluctuate according to a normal distribution to reflect the differences between heterogeneous vehicles; The Proximal Policy Optimization (PPO) algorithm is used to train the travel time prediction policy. The PPO algorithm limits the policy update magnitude by pruning the agent objective function. The policy network and value network adopt a multilayer perceptron (MLP) structure. The input of the policy network is the state vector s, and the output is the Gaussian distribution parameters of the normalized action a. The input of the value network is the state vector s, and the output is the value estimate V(s).

[0018] Furthermore, when an unstructured intersection undergoes geometric changes, the policy network parameters obtained from training at the original intersection are used as the initial weights for training at the new intersection, and fine-tuning training is performed under a small number of random traffic configurations at the new intersection.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: In multi-vehicle cooperative trajectory planning at unstructured intersections, the problem is modeled as the travel time when the fixed end time (OCP) is reached. The reasonable determination of the value directly determines the feasibility of solving the OCP and the traffic efficiency. Existing methods, which use fixed empirical values, cannot adapt to changing traffic configurations; conservative estimations systematically sacrifice traffic efficiency; and tentative searches require repeated calls to the OCP solver, which cannot meet real-time requirements. In this invention: Based on the invented reinforcement learning-based travel time prediction model, the agent learns under a large number of random traffic configurations by using the OCP solver as an interactive component of the training environment. Select the mapping relationship between the chosen solution and the solution. After training, the model can output the optimal solution based on the current traffic configuration within milliseconds. It eliminates the need for repeated calls to the OCP solver, overcoming the technical shortcomings of the trial-and-error search method, which increases computational overhead and fails to meet real-time requirements.

[0020] Based on the invented multidimensional composite reward function, the four objectives of solution feasibility, traffic efficiency, trajectory quality, and solution efficiency are uniformly encoded into a scalar reward signal, enabling the agent to automatically weigh these objectives during the learning process. The selection of multiple dimensions has a significant impact. The large negative penalty design of the feasibility reward forces the strategy to first ensure the success of the OCP solution. Under this premise, the passage efficiency reward drives the strategy to tend towards shorter passage times. This overcomes the technical shortcomings of fixed empirical values ​​not being able to be adaptively adjusted and conservative estimates systematically overestimating travel time.

[0021] Based on the invented training and deployment paradigm for specific intersections, the model is trained under a large number of random traffic configurations for a given intersection with a fixed geometric layout. This allows the model to fully learn the specific geometric constraints, conflict patterns, and kinematic limitations of the intersection for optimal performance. This minimizes the impact of [unspecified factors], ensuring prediction accuracy and generalization ability. The model inference time during deployment is in the millisecond range, without increasing the system's real-time burden. Attached Figure Description

[0022] To more clearly illustrate the content and specific embodiments of this invention, the accompanying drawings required for the description process are introduced. The following drawings are merely for the purpose of illustrating the subject matter of this invention. Those skilled in the art can readily provide other similar drawings based on the drawings in this invention without any inventive effort.

[0023] Figure 1 The flowchart shows the overall process of determining the travel time for multi-vehicle collaborative planning based on reinforcement learning. Detailed Implementation

[0024] The present invention will now be described in detail with reference to embodiments and accompanying drawings. However, it should be understood that the embodiments and drawings are for illustrative purposes only and do not constitute any limitation on the scope of protection of the present invention. All reasonable modifications and combinations included within the inventive spirit of the present invention fall within the scope of protection of the present invention.

[0025] This embodiment provides a reinforcement learning-based method for determining travel time in unstructured intersections using multi-vehicle cooperative planning. It is primarily used to address the travel time issue in unstructured road intersections (such as open-pit mines and ports) when multi-vehicle cooperative trajectory planning is modeled with a fixed endpoint time (OCP). This addresses the technical problem of determining the optimal value. Our method is trained and deployed for a specific unstructured intersection where the geometric boundaries, branch layout, and guide paths are all known and fixed prior information.

[0026] like Figure 1 As shown, the method of the present invention mainly includes the following execution steps: Step 1: Modeling the Problem of Determining Travel Time For a given unstructured intersection, when a batch When a CAV arrives, a unified passage time is first determined. This serves as a fixed parameter for subsequent OCP solving. The problem is modeled as a single-step decision problem: given the current multi-vehicle traffic configuration (state s), output the travel time (action a → ...). The OCP solver returns a result feedback (reward r). Specifically, this includes: 1) State-space design and state vector construction The current batch of multi-vehicle traffic configuration is encoded as a state vector s, which includes the following feature components: the initial position coordinates of each CAV. Initial velocity Initial heading angle Target export direction code Kinematic limit parameters (maximum speed) of each CAV Maximum acceleration Maximum steering angle ) and potential conflict density index The potential conflict density index This is the ratio of the number of vehicle pairs that need to pass through the same conflict zone to the total number of vehicle pairs.

[0027] The intersection geometry information is a fixed prior for the intersection and is embedded in the training environment, so it does not need to be used as state input.

[0028] For insufficient number of vehicles Traffic configurations are aligned to a fixed dimension using zero-filling.

[0029] 2) Action space and physical mapping The action space is a continuous scalar, and the output passage time is... : (1) In the formula, and These are the lower and upper physical bounds of the travel time, determined by the intersection dimensions and vehicle kinematic limits. The action space is a continuous scalar space, and its specific working process is as follows: First, the policy network takes the state vector s as input, outputs the Gaussian distribution parameters of the normalized action a∈[-1,1], and samples to obtain the normalized action a. Then, the normalized action a is converted into the physical travel time using a linear mapping formula. .

[0030] (2) in, Determined by dividing the shortest path length at the intersection by the maximum permissible speed for vehicles. The time is determined by dividing the longest path length at the intersection by the minimum permissible speed for vehicles, ensuring the physical rationality of the travel time.

[0031] Step 2: Reward Function Construction and OCP Interaction like Figure 1 As shown, the output of the RL agent (policy network + linear mapping) Then, it is passed into the OCP solver as a fixed parameter for solving, and the four sub-rewards and the total reward are calculated based on the solution results.

[0032] The reward function is the core of training the travel time prediction model and needs to comprehensively reflect... The multidimensional influences on the OCP solution results are selected, and their specific design is as follows: The total reward function consists of a linear combination of four sub-rewards: (3) In the formula, To solve for the feasibility reward, Incentives for improving traffic flow. As a reward for trajectory quality, To solve for efficiency rewards.

[0033] 1) Feasibility reward

[0034] When the OCP solver is given A positive reward is given when a feasible solution is successfully found. A large negative penalty is imposed when the solution fails (no feasible solution or timeout). ( ), to severely punish those who fail to solve the problem. Select: (4) According to formula (4), a positive reward is given for successfully solving the OCP. A large penalty is imposed for solution failures (including infeasible solutions and timeouts). .set up (like This allows the strategy to first learn to avoid factors that lead to solution failure. Establish a safe and feasible domain boundary understanding within the interval.

[0035] 2) Traffic efficiency bonus

[0036] Traffic efficiency reward Shorter travel times are encouraged, provided the solution is successful: (5) According to formula (5), when the OCP is successfully solved, As a reward for improving efficiency, strategies are encouraged to output shorter times while ensuring a successful solution. The reward factor drives the strategy to gradually approach the efficiency-optimal boundary of the feasible region.

[0037] 3) Track quality reward

[0038] Track quality reward This reflects the OCP objective function value (i.e., trajectory smoothness and energy consumption): (6) In the formula, This is the objective function value of OCP.

[0039] According to formula (6), when the OCP solution is successful, the OCP objective function value is used. The negative value of the objective function is used as a trajectory quality reward. The smaller the objective function value, the smoother the trajectory, the lower the energy consumption, and the higher the reward. This reward item guides the strategy to select elements that enable OCP to generate high-quality trajectories. .

[0040] 4) Solution efficiency reward

[0041] Solution efficiency reward This reflects the time required to solve the OCP problem. (7) In the formula, This represents the actual computation time of the OCP solver.

[0042] According to formula (7), when the OCP solution is successful, the actual solution time is used. The negative value is used as a reward for solution efficiency. A reasonable... By appropriately tightening the OCP constraints, the solver converges more easily and the solution time is shortened. This reward-based indirect guidance strategy selects methods that maximize the efficiency of the solution process. .

[0043] In the above sub-reward function, This represents the weighting coefficient for each sub-reward.

[0044] When OCP solution fails All values ​​are set to zero, and the total reward is solely determined by the feasibility penalty.

[0045] The above method involves obtaining the following parameters: the initial position coordinates, initial velocity, and initial heading angle of each CAV are obtained in real time through onboard sensors and cloud control platform observation; the target exit direction code is determined by the task instruction issued by the scheduling system; the kinematic limit parameters (maximum speed, maximum acceleration, maximum steering angle) of each CAV are fixed parameters calibrated at the vehicle factory; the potential conflict density index is obtained by the cloud control platform through combined calculation based on the origin and destination directions of each CAV; the physical lower and upper bounds of the travel time are calculated and determined based on the intersection geometry and vehicle kinematic limit parameters, specifically the shortest path length divided by the maximum permissible speed and the longest path length divided by the minimum permissible speed; the weighting coefficients in the reward function ( ) and feasibility reward parameters ( , () These are hyperparameters that users can customize according to their actual needs. In this embodiment, they are set as follows: =1、 =10、 =0.1、 =0.05、 =0.01; The number of layers and hidden layer dimensions of the policy network and value network are user-defined hyperparameters. In this embodiment, a 3-layer MLP structure is used with a hidden layer dimension of 256; The sampling range of the number of vehicles during training [ , The parameters are set according to the actual maximum capacity of the intersection. All of the above parameters can be obtained through observation, calculation, or customization.

[0046] Step 3: Training the policy network: 1) Training environment setup The core component of the training environment is the OCP solver for that specific intersection. A traffic configuration randomization strategy is employed; in each episode, the environment first randomly generates a traffic configuration from a preset distribution: the number of vehicles... exist[ Uniform sampling within the area.

[0047] The initial pose and entry direction of each CAV are randomly sampled from the reasonable initial areas of each approach lane at the intersection; the target exit direction is randomly selected from the feasible exits; vehicle size parameters fluctuate according to a normal distribution to reflect the differences between heterogeneous vehicles. Through training with a large number of random traffic configurations, the strategy is ensured to cover various typical operating conditions that may be encountered at the intersection.

[0048] 2) Training Algorithm A proximal policy optimization (PPO) algorithm is used to train the travel time prediction policy. PPO limits the policy update magnitude by pruning the agent objective function, ensuring training stability and making it suitable for single-step decision problems in continuous action spaces. The policy network adopts a multilayer perceptron (MLP) structure, with the input being the state vector s and the output being the Gaussian distribution parameters (mean and standard deviation parameters) of the normalized action a, obtained through sampling. The value network has the same structure as the policy network, also using a multilayer perceptron (MLP) structure. The value network takes the state vector s as input and outputs the state value estimate V(s).

[0049] 3) Training process During training, a large number of different multi-vehicle traffic configurations are randomly generated under a fixed intersection geometry. Each episode is a single-step interaction, with the following flow: agent observes state s → policy network outputs action a → mapping to... →OCP solver with Solve for the parameters → Calculate the reward r based on the solution results → Store in the experience buffer.

[0050] After accumulating a sufficient number of episodes, perform a PPO parameter update. Repeat the above process until the policy converges (the reward is stable and the success rate of OCP solution reaches a preset threshold).

[0051] Step 4: Deployment and Real-Time Inference After training, the policy network is deployed on the cloud control platform at the intersection. When a batch of CAVs arrives at the intersection, their traffic configuration features are extracted in real time and encoded into state vectors s. The policy network then outputs the optimal value during forward inference. .Should The policy network is passed as a fixed parameter to the downstream OCP solver for multi-vehicle cooperative trajectory planning. The inference time of the policy network is in the millisecond range (usually less than 1ms), which is negligible compared to the computation time of the OCP solver (usually hundreds of milliseconds to seconds), and does not increase the real-time burden of the system.

[0052] A transfer learning mechanism for intersection geometric changes: When an intersection undergoes geometric changes due to construction or renovation, the method described in this invention supports rapid adaptation based on transfer learning. Specifically, the policy network parameters trained at the original intersection are used as the initial weights for training at the new intersection, and fine-tuning is performed under a small number of random traffic configurations at the new intersection. Since the decision-making logic for travel time at different unstructured intersections shares commonalities (e.g., longer travel times are required for higher conflict densities), this approach addresses these challenges. The more lenient the vehicle's kinematic limits, the shorter the allowance. (etc.), transfer learning can significantly reduce the training sample size and training time at new intersections, enabling rapid deployment.

[0053] The above-described embodiments are merely some implementation methods of this application. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of this application, and these all fall within the protection scope of this application.

Claims

1. A method for determining travel time in unstructured intersections through multi-vehicle cooperative planning based on reinforcement learning, characterized in that... Includes the following steps: 1) For specific unstructured intersections, the problem of determining travel time is modeled as a single-step decision problem. A state space, action space, and reward function are established. The state space is responsible for encoding the current batch of multi-vehicle traffic configuration into a state vector s. The action space is responsible for outputting a normalized action a from the policy network based on the state vector s, and linearly mapping it into the physical travel time. The reward function is used to calculate the physical travel time. The reward is calculated when OCP is passed in as a fixed parameter. 2) Construct a reinforcement learning training environment, using the OCP solver as the interactive environment, and train the travel time prediction model under a large number of random traffic configurations at the intersection. The training process of the travel time prediction model is as follows: The state vector s of the current multi-vehicle traffic configuration in the episode is observed, and then the policy network outputs a normalized action a, which is mapped to the physical travel time. Physical travel time Input into the OCP solver, and with Solve for the parameters, calculate the reward r based on the solution and store it in the experience buffer. After accumulating a sufficient number of episodes, perform PPO parameter update and repeat the above process until the policy converges. 3) Finally, the traffic time prediction model trained in step 2) is deployed on the cloud control platform of the intersection. Its traffic configuration features are extracted in real time and encoded into a state vector s. The policy network outputs the optimal value through forward inference. It is then passed to the OCP solver for multi-vehicle collaborative trajectory planning.

2. The method for determining travel time at unstructured intersections based on reinforcement learning for multi-vehicle cooperative planning as described in claim 1, characterized in that, The state space design is as follows: the state vector s contains the following feature components: the initial position coordinates of each CAV. initial velocity Initial heading angle Target export direction code Kinematic limit parameters of each CAV: maximum speed Maximum acceleration Maximum steering angle Potential conflict density index The potential conflict density index This is the ratio of the number of vehicle pairs that need to pass through the same conflict zone to the total number of vehicle pairs.

3. The method for determining travel time at unstructured intersections based on reinforcement learning for multi-vehicle cooperative planning as described in claim 1, characterized in that... The motion space design method is as follows: The action space is a continuous scalar, and the output passage time is... : (1) In the formula, and These are the lower and upper physical bounds of the travel time, determined by the intersection dimensions and vehicle kinematic limits. The normalized action a ∈ [-1, 1] is output by the policy network and linearly mapped to the physical travel time using formula (2). : (2) in, Determined by dividing the shortest path length at the intersection by the maximum permissible speed for vehicles. It is determined by dividing the longest path length at the intersection by the minimum permissible speed for vehicles.

4. The method for determining travel time in unstructured intersections based on reinforcement learning for multi-vehicle cooperative planning as described in claim 1, characterized in that, The reward function is designed as follows: The total reward function consists of a linear combination of four sub-rewards: (3) In the formula, To solve for the feasibility reward, Incentives for improving traffic flow. As a reward for trajectory quality, To solve for efficiency rewards.

5. The method for determining travel time in unstructured intersections based on reinforcement learning for multi-vehicle cooperative planning as described in claim 4, characterized in that, Feasibility reward When the OCP solver is given A positive reward is given when a feasible solution is successfully found. A large penalty is imposed when the solution fails. ( ), to severely punish those who fail to solve the problem. Select: (4)。 6. The method for determining travel time for multi-vehicle cooperative planning at unstructured intersections based on reinforcement learning as described in claim 4, characterized in that, Rewards for traffic efficiency When the OCP solution is successful, As a reward for improved traffic efficiency: (5) A weighting coefficient is assigned to reward traffic efficiency.

7. The method for determining travel time for multi-vehicle cooperative planning at unstructured intersections based on reinforcement learning as described in claim 4, characterized in that, The trajectory quality reward is: (6) In the formula, The objective function value of OCP. This represents the trajectory quality weighting coefficient.

8. The method for determining travel time for multi-vehicle cooperative planning at unstructured intersections based on reinforcement learning as described in claim 4, characterized in that, Rewards for solution efficiency : (7) In the formula, The actual computation time of the OCP solver. To solve for the efficiency reward weight coefficient.

9. The method for determining travel time at unstructured intersections based on reinforcement learning for multi-vehicle cooperative planning as described in claim 1, characterized in that, During the training of the policy network, a traffic configuration is first randomly generated from a preset distribution: the number of vehicles... exist[ Uniform sampling within the intersection; the initial pose and entry direction of each CAV are randomly sampled from the reasonable initial area of ​​each entrance lane of the intersection. The target export direction is randomly selected from feasible exports; vehicle size parameters fluctuate according to a normal distribution to reflect the differences between heterogeneous vehicles; The Proximal Policy Optimization (PPO) algorithm is used to train the travel time prediction policy. The PPO algorithm limits the policy update magnitude by pruning the agent objective function. The policy network and value network adopt a multilayer perceptron (MLP) structure. The input of the policy network is the state vector s, and the output is the Gaussian distribution parameters of the normalized action a. The input of the value network is the state vector s, and the output is the value estimate V(s).

10. The method for determining travel time at unstructured intersections based on reinforcement learning for multi-vehicle cooperative planning as described in claim 1, characterized in that, When an unstructured intersection undergoes geometric changes, the policy network parameters obtained from training at the original intersection are used as the initial weights for training at the new intersection, and fine-tuning training is performed under a small number of random traffic configurations at the new intersection.

Images