Dual-network-oriented intelligent vehicle scheduling method

Abstract

The invention discloses a dual-network-oriented intelligent vehicle scheduling method. The method comprises the following steps: acquiring local observation information of an intelligent vehicle; historical local observation information of the intelligent vehicle is coded into a hidden state vector; inputting the hidden state vector into a strategy network and a value network, outputting probability distribution of an action by the strategy network, and outputting value estimation of a state by the value network; the MCTS algorithm uses the output of the strategy network and the value network to search, and finally outputs an optimal action; and the intelligent vehicle executes a corresponding control instruction according to the optimal action obtained by searching through the MCTS algorithm. According to the method, deep reinforcement learning, Monte Carlo tree search and an LSTM network are fused, a framework of collaborative decision of a strategy network and a value network is constructed, the traffic safety and traffic efficiency of vehicles at the non-signalized intersection are improved, and the robustness of a scheduling algorithm in a part of observable environments is enhanced; the continuous action space is discretized through a strategy network, and efficient decision making is carried out in combination with Monte Carlo tree search.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to an intelligent vehicle dispatching method for dual networks. Background Art

[0002] Unsignalized intersections are critical nodes in urban transportation networks and are also prone to accidents. Due to the lack of clear traffic signal guidance, vehicle traffic relies heavily on individual driving decisions and inter-vehicle interactions, significantly increasing the risk of collisions and reducing traffic efficiency in intersections. With the development of intelligent connected vehicles (ICVs), the question of how to fully utilize vehicle-to-everything (V2X) technology to design safe and efficient intelligent scheduling strategies and improve the traffic capacity and safety of unsignalized intersections has become a key research topic in the field of intelligent transportation.

[0003] Existing research on the problem of vehicle dispatching at unsignalized intersections mainly includes rule-based methods, bionic optimization methods, and learning-based methods. Rule-based methods are simple and easy to implement, but they are difficult to adapt to complex and changing traffic conditions and can easily lead to traffic congestion and deadlock. Bionic optimization methods have strong environmental adaptability, but they are prone to falling into local optimality in high-dimensional multimodal problems, and their high computational complexity makes it difficult to meet real-time requirements. AlphaGo's breakthrough achievements in the field of Go demonstrate that deep reinforcement learning (DRL) has great potential in handling complex sequential decision-making problems, opening up new avenues for intelligent traffic control research.

[0004] In recent years, researchers have gradually applied DRL to cooperative vehicle control at unsignalized intersections. Existing DRL-based methods mostly assume a fully observable environment or ignore the game and cooperation relationships between multiple agents. Their performance and robustness are often significantly limited in complex scenarios such as unsignalized intersections, where the environment is partially observable and multi-agent games are in play. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an intelligent vehicle scheduling method for dual networks in response to the above-mentioned deficiencies in the existing technology. The intelligent vehicle scheduling method for dual networks deeply integrates deep reinforcement learning, Monte Carlo Tree Search (MCTS) and LSTM network, constructs a framework for collaborative decision-making between policy network and value network, improves the safety and efficiency of vehicle passage at unsignaled intersections, and enhances the robustness of the scheduling algorithm in partially observable environments; discretizes the continuous action space through the policy network, and combines it with Monte Carlo tree search for efficient decision-making, thereby solving the problem that traditional reinforcement learning methods are difficult to handle high-dimensional, continuous state and action spaces.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A dual-network-oriented intelligent vehicle scheduling method comprises the following steps:

[0008] Step 1: Environmental perception: Obtain local observation information of the intelligent vehicle;

[0009] Step 2, LSTM state encoding: Encode the historical local observation information of the intelligent vehicle into a hidden state vector;

[0010] Step 3, decision-making stage: The hidden state vector is input into the policy network and the value network respectively. The policy network outputs the probability distribution of the action, and the value network outputs the value estimate of the state.

[0011] Step 4: Monte Carlo Tree Search: The MCTS algorithm uses the outputs of the policy network and the value network to search and ultimately output the optimal action.

[0012] Step 5, execution phase: The intelligent vehicle executes the corresponding control instructions based on the optimal action searched by the MCTS algorithm.

[0013] As a further improved technical solution of the present invention, the step 1 specifically includes:

[0014] Each intelligent vehicle i obtains its own local observation information at time t through on-board sensors (such as lidar, camera, GPS, etc.) include:

[0015] In the POMDP framework, the system state space is composed of the states of all vehicles participating in the traffic; the state of vehicle i at time t is expressed as in represents the position coordinates of intelligent vehicle i at time t, represents the speed of intelligent vehicle i at time t, represents the heading angle of intelligent vehicle i at time t, is the navigation information of smart vehicle i at time t, including the destination coordinates of smart vehicle i The distance between the current position of intelligent vehicle i and the destination The relative direction between the current position of intelligent vehicle i and the destination Limited by the perception capabilities of each agent, the observation information of vehicle i at time t is: in It represents the set of neighboring vehicles communicating with smart vehicle i at time t, that is, relative information, which is obtained through V2V communication.

[0016] As a further improved technical solution of the present invention, in step 2:

[0017] LSTM state encoding purpose: Encode the historical observation information of the vehicle into a fixed-length hidden state vector h t , to capture the temporal characteristics of vehicle behavior and deal with partial observability (i.e., the vehicle can only perceive information within a limited range);

[0018] Operation: Use LSTM network as encoder;

[0019] Specifically include:

[0020] The LSTM network transforms the historical local observation information O of intelligent vehicle i into t-H:t Encoded into a hidden state vector;

[0021] The input of the LSTM network is the observation information sequence O of the intelligent vehicle i in the past H control cycles t-H:t , O t-H:t ={o t-H+1 ,o t-H+2 …o t}; Among them, o t That is, the local observation information of intelligent vehicle i at time t

[0022] The output of the LSTM network is a hidden state vector h that combines the temporal feature information and contextual information in the historical observation sequence. t and cell state c t :

[0023] At each time step t, the LSTM network receives the observation o at the current time t. t , the hidden state vector h at the previous moment t-1 and cell state c t-1 As input, the network state is updated through the gating mechanism:

[0024] h t ,c t =LSTM(o t ,ht-1 ,c t-1 );

[0025] After H time steps of iterative calculation, the hidden state vector h is finally output by the LSTM network. t ;h t Serves as input to subsequent strategy networks and value networks.

[0026] The role of step 2: improves the accuracy and robustness of state estimation and provides a better basis for subsequent decision-making.

[0027] As a further improved technical solution of the present invention, in step 3:

[0028] Policy network purpose: state h based on LSTM encoding t , output the probability distribution of each possible action in the current state, guide the search direction of MCTS, and discretize the continuous action space;

[0029] The specific operations are:

[0030] The policy network receives the hidden state h t As input;

[0031] The policy network uses a multi-dimensional Gaussian distribution to model the action probability distribution:

[0032]

[0033] in It means that the mean is μ and the variance is σ 2 Gaussian distribution, μ d (h t ) represents the expected value of the d-th dimension action, reflecting the agent's optimal action estimate in this dimension; It represents the variance of the d-th dimension action, which characterizes the uncertainty of action selection. A larger variance value means a stronger exploration tendency in this dimension.

[0034] The parameters θ of the policy network π Updated through the policy gradient algorithm;

[0035] Condition: Policy network MCTS joint training;

[0036] Difference from existing technologies: Traditional MCTS is usually used in discrete action spaces and has difficulty processing continuous action spaces. This paper uses a policy network to map the continuous action space to a discrete action probability distribution, making MCTS applicable to continuous control problems.

[0037] Function: Guide the search direction of MCTS and improve search efficiency; discretize the continuous action space and reduce the complexity of the problem.

[0038] Purpose of the value network: state h based on LSTM encoding t , evaluate the long-term value of the current state, provide pruning basis for MCTS, and evaluate the pros and cons of the state;

[0039] operate:

[0040] The value network receives the hidden state h t As input;

[0041] The value network outputs a scalar value estimate V(h t θ v ), which represents an estimate of the expected long-term discounted return that the agent can obtain in the current state:

[0042]

[0043] Where T is the total duration of the scheduling round, γ is the discount factor, and r k represents the immediate reward of the environment at time k, where k is the time step index;

[0044] Parameters θ of the value network v Update by minimizing the mean square error between the predicted value and the true value;

[0045] Condition: Value network MCTS joint training.

[0046] Difference from existing technologies: Traditional MCTS usually uses the final reward of the simulation as the estimate of the state value, which has a large variance. This invention uses the value network to estimate the state value, reducing the variance and improving the accuracy of the estimation.

[0047] Function: Provides accurate state value estimation for pruning the MCTS search tree, accelerating the search process.

[0048] As a further improved technical solution of the present invention, in step 4:

[0049] Monte Carlo Tree Search (MCTS) aims to find the optimal action through iterative search using the prior probability provided by the policy network and the state evaluation provided by the value network.

[0050] Operation: The MCTS algorithm is an iterative process, and each iteration consists of four steps: selection, expansion, simulation, and backtracking;

[0051] Selection:

[0052] Starting from the root node (current state), recursively select child nodes until a leaf node is reached;

[0053] The strategy for selecting child nodes is based on the output of the policy network and the PUCT formula (Polynomial Upper Confidence Tree):

[0054] a * =argmax a [Q(h t ,a)+U(h t ,a)];

[0055]

[0056] Among them, Q(h t ,a) indicates the state h t The average cumulative reward after executing action a; Indicates that from state h t The set of all simulated trajectories obtained after starting and taking action a, and R(τ) represents the cumulative reward of trajectory τ; the upper limit control term U(h t ,a) The following improved PUCT formula is adopted:

[0057]

[0058] in, is the prior probability distribution obtained by normalizing the policy network output probability of the sampled action set; c puct is the exploration constant, b is used to traverse the current state node h t An index variable for all candidate actions, N(h t ,b) is the number of visits,∑ b N(h t ,b) represents the total number of visits to the current node;

[0059] Expansion:

[0060] If the selected leaf node is not the terminal state, then according to the probability distribution π(a|h t θ π ), sample m actions and create m new child nodes;

[0061] Simulation:

[0062] Starting from the newly created child node, use the policy network π(a|h t θ π ) until the simulation reaches the termination state or the maximum simulation depth D sim ;After the simulation is completed, a cumulative reward G is obtained;

[0063] Backpropagation:

[0064] Starting from the leaf node, backpropagate the cumulative reward G or the output of the value network along the search path, and update the number of visits N (h t ,a) and cumulative value Q(h t ,a):

[0065] N(h t ,a)←N(h t ,a)+1;

[0066]

[0067] Among them, the target value V target Varies depending on the leaf node type: If the leaf node is a new expansion node, then V target The value is the output of the value network; if the leaf node is in the simulation termination state or reaches the maximum simulation depth, the value is the cumulative reward G obtained by the simulation;

[0068] Timing relationship: The four steps of each iteration of the MCTS algorithm are executed in a sequential loop. Each iteration updates the information of the search tree, and over time it gradually converges to the optimal strategy, that is, outputs the optimal action to be performed at the current moment.

[0069] Condition: The strategy network and value network provide prior knowledge;

[0070] Different from existing technologies: Traditional MCTS is typically used in discrete action spaces and relies on stochastic simulations. This paper combines MCTS with a policy network and a value network to achieve efficient search in continuous action spaces. It also accelerates the search process by leveraging the prior knowledge provided by the policy network and the value network.

[0071] Role: Online decision-making, finding near-optimal actions in complex environments.

[0072] As a further improved technical solution of the present invention, in step 5:

[0073] Each intelligent vehicle executes corresponding control instructions, such as acceleration, deceleration, and steering, based on the optimal action obtained by MCTS search.

[0074] As a further improved technical solution of the present invention, the present invention further includes step 6:

[0075] Environment update: The environment is updated according to the actions of all vehicles and enters the next time step.

[0076] Repeat the above process (including step 1 to step 6) until all vehicles arrive at the destination.

[0077] The beneficial effects of the present invention are:

[0078] 1. Improve the safety and efficiency of vehicles passing through unsignalized intersections. Existing technologies have difficulty ensuring both safety and efficiency in high-density, complex traffic scenarios. This invention aims to reduce collision risks and improve traffic efficiency through intelligent scheduling methods.

[0079] 2. Enhance the robustness of the scheduling algorithm in partially observable environments. Most existing technologies assume that the environment is fully observable, ignoring the limitations of sensor perception range and the uncertainty of inter-vehicle communication. This invention aims to handle partial observability through time series modeling (LSTM) to improve the robustness of the algorithm.

[0080] 3. Achieve multi-agent collaborative control. Existing technologies rarely consider inter-vehicle game and cooperation. This invention aims to achieve multi-vehicle collaborative scheduling through a centralized training and distributed execution framework.

[0081] 4. Solve the problem that traditional reinforcement learning methods have difficulty handling high-dimensional, continuous state and action spaces. This invention aims to discretize the continuous action space through a policy network and combine it with Monte Carlo tree search for efficient decision-making. The calculation process is simple and can meet real-time requirements.

[0082] In summary, the present invention improves traffic efficiency: by optimizing the vehicle scheduling strategy, the waiting time of vehicles at intersections is reduced, and traffic efficiency is improved. Experiments show that compared with the baseline algorithm, the present invention improves vehicle traffic efficiency by 15%-24%. The present invention reduces the risk of collision: through time series modeling and collaborative control, the probability of vehicle collision is reduced. Experiments show that compared with the baseline algorithm, the present invention reduces the collision rate by 3%-11%. The present invention improves the success rate of passage: ensuring that more vehicles can pass through intersections safely and smoothly. Experiments show that in high-density traffic scenarios, the success rate of passage of the present invention reaches 85%-92%. The present invention enhances robustness: by processing partial observability through LSTM, the robustness of the algorithm in complex traffic scenarios is improved. Ablation experiments show that the LSTM module significantly improves the performance of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a conflict model diagram for an unsignalized intersection.

[0084] Figure 2 This is the LPV-MCTS intelligent vehicle scheduling framework diagram.

[0085] Figure 3 This is a simulation environment diagram.

[0086] Figure 4 Comparison chart of reward curves of different algorithms.

[0087] Figure 5 This is a comparison chart of algorithm performance indicators.

[0088] Figure 6 This is the reward curve for the LSTM ablation experiment. DETAILED DESCRIPTION

[0089] The specific embodiments of the present invention are further described below with reference to the accompanying drawings:

[0090] This paper addresses the partial observability and multi-agent game characteristics of unsignalized intersections by proposing an intelligent vehicle scheduling method based on a Long Short-Term Memory (LSTM) network-enhanced Policy-Value Monte Carlo Tree Search (LPV-MCTS), also known as a dual-network intelligent vehicle scheduling method. This method deeply integrates deep reinforcement learning, Monte Carlo Tree Search (MCTS), and LSTM networks to construct a framework for collaborative decision-making between the policy network and the value network. Multi-agent collaborative control is achieved through centralized training and distributed execution. The core advantages of this LPV-MCTS algorithm are: 1) Utilizing the LSTM network to encode the historical observation sequence of vehicles, it effectively captures the temporal characteristics of vehicle behavior, improving state representation and decision robustness in partially observable environments; 2) The MCTS search, driven by the collaborative policy and value networks, achieves more accurate state assessment and better sequential decision-making, achieving a better balance between safety and efficiency.

[0091] Overall technical solution process:

[0092] Environmental perception and state encoding stage: Each intelligent vehicle (ICV) uses on-board sensors (such as Figure 1 The ICV on the driving route shown in the figure obtains the surrounding environment information, including its own state (position, speed, heading angle, etc.) and the state of other vehicles (relative position, speed, etc.). This information constitutes the original observation The LSTM encoder receives these raw observations and encodes them into a hidden state h containing historical information. t .

[0093] Decision stage: the hidden state h t Input policy network and value network. The policy network outputs the probability distribution of action π(a|h t θ π ), the value estimate of the output state of the value network V(h t θv ). The MCTS algorithm uses the output of these two networks to search and finally outputs the optimal action a * .

[0094] Execution phase: Each intelligent vehicle takes the optimal action a according to the output of MCTS. * Execute corresponding control instructions (acceleration, deceleration, steering).

[0095] Environment update: The environment is updated according to the actions of all vehicles and enters the next time step.

[0096] 1. System model and problem description:

[0097] 1.1. Unsignalized intersection collaborative control environment:

[0098] This study focuses on the problem of ICVs coordinated scheduling in an unsignaled intersection environment. Figure 1 As shown in the figure, the ICVs in this scenario are equipped with on-board units (OBUs), which enable environmental perception, vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications, and can exchange information with roadside units (RSUs). In intersection conflict zones, the interweaving of vehicle trajectories increases the risk of collision, posing a severe challenge to the decision-making capabilities of ICVs. This paper aims to develop an intelligent decision-making mechanism that balances safety and traffic efficiency to achieve optimal coordinated scheduling of ICVs in conflict zones.

[0099] 1.2 Multi-agent POMDP modeling:

[0100] In an unsignaled intersection environment, each ICV can only obtain partial observation information, and the behavior of other vehicles is uncertain. This paper models the problem of multi-ICV cooperative control at an unsignaled intersection as a multi-agent partially observable Markov decision process (POMDP). In this framework, each ICV is regarded as an independent agent that interacts with other agents in a shared road environment. Formally, the POMDP is represented by a seven-tuple (S, A, O, T, Z, R, γ). Where:

[0101] (1) S is the state space, which contains all possible environmental states. Due to factors such as the limited sensor perception range and environmental occlusion, agents are usually unable to obtain the complete global state. The state of each agent i at time t is represented by Indicates. represents the position coordinates of vehicle i at time t, represents the instantaneous speed, represents the heading angle, Navigation information, including the destination coordinates of vehicle i The distance between vehicle i's current position and the destination The relative direction between vehicle i's current position and the destination

[0102] (2) A is the action space, which contains all actions that the agent can perform. This paper adopts a continuous action space, where the control action of vehicle i at time t is (Can also be abbreviated as action a or action a t ) is the continuous acceleration and steering angle and is subject to the following physical constraints:

[0103]

[0104] Among them, v max ,u max , δ max are the maximum longitudinal speed, maximum longitudinal acceleration and maximum steering angle of the vehicle specified by the road.

[0105] (3) O is the observation space, which contains all possible local observations of all agents. In this paper, the local observation of each agent i at time t is Contains its own status information And the relative status information of a limited number of other vehicles around it obtained through RSU:

[0106]

[0107] in, represents the set of n neighboring vehicles that communicate with agent i at time t.

[0108] (4) T is the state transition function, and T(s',s,a) represents the probability that the environment will transition to state s' after executing the joint action a in state s. In unsignaled intersection scenarios, the state transition function T is usually unknown due to the uncertainty of the behavior of other vehicles.

[0109] (5) Z is the observation function, and Z(o, s′, a) represents the probability that the agent obtains observation o after executing the joint action a in state s′.

[0110] (6) R is the reward function, R(s,a) represents the immediate reward obtained by all agents after performing the joint action a in state s;

[0111] (7)γ∈(0,1] is a discount factor used to balance the importance of current rewards and future rewards.

[0112] The goal of this study is to find an optimal strategy π* that maximizes the expected value of the cumulative reward given the initial state distribution:

[0113]

[0114] Among them, π represents the scheduling strategy, that is, the policy function that outputs the action according to the current observation; τ represents the state-action sequence: [(s0,a0),(s1,a1),…,(s T ,a T )]; T is the total duration of the scheduling round; r t is the total system reward at time t; represents the expected return of the state-action sequence τ under the policy π.

[0115] 2. Intelligent vehicle scheduling method based on LPV-MCTS:

[0116] To solve the vehicle scheduling problem in the scenario of no signal light intersection, this study proposes a new hybrid algorithm framework - LSTM-enhanced policy-value MCTS (LPV-MCTS). LPV-MCTS combines deep reinforcement learning, MCTS and LSTM network to build a decision-making framework driven by the "policy-value" dual network collaboration. The LPV-MCTS decision-making framework for a single intelligent vehicle is as follows: Figure 2 This paper adopts a centralized training with decentralized execution (CTDE) training paradigm. During the training phase, all agents share the same set of policy network and value network parameters to accelerate the experience-based learning process, improve training stability, and enhance sample efficiency. During the execution phase, each agent independently runs an instance of the LPV-MCTS algorithm based on its own local observations, ensuring the distributed nature and real-time performance of the system.

[0117] 2.1. State representation based on LSTM:

[0118] At unsignaled intersections, vehicles can only observe a portion of their surroundings and lack a complete global state. Furthermore, the driving intentions and future trajectories of other vehicles are uncertain, resulting in a highly dynamic environment. Traditional state representation methods based on current-time observations struggle to capture the temporal features of historical information, making it difficult to accurately estimate the current state and predict future states, thus impacting decision-making reliability. To address this issue, this paper introduces an LSTM network to encode historical state sequences and extract temporal feature information.

[0119] LSTM is a special type of recurrent neural network (RNN) that excels at processing sequential data and learning long-term dependencies. In the LPV-MCTS algorithm, LSTM is used as a state representation module. Its main function is to encode the vehicle's historical observation sequence into a fixed-length hidden state vector. This vector contains the temporal characteristics of historical information and serves as the input to the policy network and value network.

[0120] The input of the LSTM encoding layer is the observation information sequence of the vehicle in the past H control cycles. t-H:t , where H is the history window size, o t Represents the local observation information of vehicle i at time t:

[0121] O t-H:t ={o t-H+1 ,o t-H+2 …o t} (4);

[0122] At each time step t, the LSTM unit receives the observation o at the current moment. t , the hidden state h at the previous moment t-1 and cell state c t-1 As input, the network state is updated through the gating mechanism:

[0123] h t ,c t =LSTM(o t ,h t-1 ,c t-1 ) (5);

[0124] After H time steps of iterative calculation, the hidden state vector h is finally output by LSTM t By integrating temporal feature information and contextual association information in historical observation sequences, it can more comprehensively and accurately characterize the vehicle's current state and environmental situation, providing high-quality state representation for subsequent strategy learning and action decision-making.

[0125] 2.2 Dual-network collaborative decision-making architecture:

[0126] The core of LPV-MCTS lies in the dual-network collaborative decision-making architecture consisting of a policy network and a value network. The two networks share a feature extraction layer based on a multilayer perceptron (MLP), which consists of a two-layer fully connected structure with 128 neurons per layer and uses a ReLU activation function to enhance nonlinear expression capabilities. The network uses the hidden state h output by the LSTM encoder. tAs input, they are connected to their respective output layers after shared feature extraction to achieve the learning of action selection strategy and evaluation of state value.

[0127] 2.2.1, Policy Network:

[0128] Policy network π(a|h t θ π ) Based on shared features, the mapping relationship between state and action space is learned, and the probability distribution of each possible action under the current state is output. Considering the continuity and uncertainty of vehicle control actions, for each intelligent agent, the policy network uses a multi-dimensional Gaussian distribution to model the action probability distribution:

[0129]

[0130] in, Indicates that the mean is μ and the variance is σ 2 Gaussian distribution, μ d (h t ) represents the expected value of the d-th dimension action, reflecting the agent's optimal action estimate in this dimension; It represents the variance of the action in the d-th dimension, which describes the uncertainty of action selection. A larger variance value means a stronger exploration tendency in this dimension.

[0131] The policy network is optimized using the expected cumulative reward maximization criterion and the parameters are updated using the policy gradient algorithm:

[0132]

[0133] Among them, α π is the learning rate; A(h t ,a) is the advantage function, using Q(h t ,a)-V(h t ) to estimate. By comparing the state-action value function Q(h t ,a) and state value function V(h t ), evaluates the performance of action a relative to the average performance. In LPV-MCTS, V(h t ) is directly estimated by the value network, Q(h t ,a) is approximated through iterative search and value backtracking of the MCTS algorithm.

[0134] The policy network outputs π(a|h t θ π ) serves as the prior information and behavior strategy of MCTS: in the tree search stage, the output is used as the action prior probability to guide the selection and expansion of nodes; in the simulation stage, it is directly used as a behavior strategy for fast sampling to improve search efficiency.

[0135] 2.2.2 Value Network:

[0136] Value network V(h t θ v ) Based on the shared features, evaluate the long-term value of the agent's current state and output a scalar value estimate:

[0137]

[0138] The output of the value network directly reflects the expected long-term benefits that the agent can obtain in the current state, providing a reliable value estimate for node evaluation in the MCTS search process.

[0139] The optimization goal of the value network is to minimize the mean square error (MSE) between the predicted value and the true value. Its parameter θ v Optimization is performed using the gradient descent algorithm:

[0140]

[0141] Among them G t is the cumulative discounted reward from time t to the end of the round, estimated by the TD(λ) algorithm. v is the learning rate.

[0142] In the backtracking phase of MCTS, the output V(h t θ v ) is used as the value estimate of the leaf node to provide heuristic information for tree search, thereby improving sampling efficiency and decision quality.

[0143] 2.3. LPV-MCTS search algorithm:

[0144] In high-dimensional action problems, MCTS simulates and evaluates actions by expanding candidate actions at the nodes of a search tree. However, in continuous control, the dimensionality of actions and the continuous nature of states further complicate decision-making. Traditional MCTS algorithms face two key challenges when dealing with continuous action spaces: the infinity of the action space makes full expansion infeasible, and random sampling makes it difficult to guarantee the quality of sampled actions. To address this, this study discretizes the continuous action space using a probability distribution parameterized by a policy network, focusing on candidate action sets in regions of high probability density. Efficient tree search is then implemented based on an improved PUCT (Polynomial Upper Confidence Tree) formula.

[0145] For each node state h t , extract m actions with higher potential value from the probability distribution output by the policy network This parameterized sampling method based on the policy network guides the search towards more promising directions through the learned mean, while using the variance to adaptively adjust the scope of exploration and maintain moderate exploration in uncertain areas, thus ensuring the continuity and smoothness of the sampled actions.

[0146] In the search tree, we have for each state-action pair (h t ,a) Maintain two cumulative statistics: cumulative number of visits N(h t ,a) and cumulative value Q(h t ,a). The action selection follows an improved PUCT criterion:

[0147] a * =argmax a [Q(h t ,a)+U(h t ,a)] (10);

[0148]

[0149] Among them, Q(h t ,a) represents the average cumulative reward obtained by the state-action pair; Indicates that from state h t The set of all simulated trajectories obtained after starting and taking action a, and R(τ) represents the cumulative reward of trajectory τ. The upper limit control term U(h t ,a) The following improved PUCT formula is adopted:

[0150]

[0151] in, is the prior probability distribution of the strategy, which is obtained by normalizing the strategy network output probability of the sampled action set; c puct is an exploration constant used to balance the relationship between exploration and utilization; b is used to traverse the current state node h t An index variable for all candidate actions. b N(h t ,b) represents the total number of visits to the current node. This improved PUCT formula introduces prior knowledge of the policy network to guide the search direction, while the design of the number of visits also ensures sufficient exploration. Here, a refers specifically to the action of interest; b is general and serves as an index to traverse all actions in order to calculate the total number of visits to the parent node.

[0152] When the selection phase reaches a leaf node h tIf the node is not in the terminal state, it will enter the simulation stage. In the simulation stage, from h t At the beginning, an action is sampled according to the probability distribution output by the policy network, and a fast simulation is performed until the terminal state or the maximum simulation depth D is reached. sim After the simulation is completed, a cumulative reward G will be obtained.

[0153] In the backtracking phase, starting from the leaf node (the newly expanded node or the state at the end of the simulation), the value estimate or cumulative reward is back-propagated along the search path. For each state-action pair (h t ,a), first update its access count:

[0154] N(h t ,a)←N(h t ,a)+1(13)

[0155] Then, based on the target value V target Update its value estimate Q(h t ,a):

[0156]

[0157] Among them, the target value V target Varies depending on the leaf node type: If the leaf node is a new expansion node, then V target The value is the output of the value network, V(h); if the leaf node reaches the simulation termination state or the maximum simulation depth, the value is the cumulative reward G obtained from the simulation. Unlike traditional MCTS, which relies solely on simulation results to update value, the LPV-MCTS algorithm proposed in this paper combines the forward-looking estimates of the value network with the empirical information of Monte Carlo simulation to improve the accuracy and convergence speed of value estimation.

[0158] The LPV-MCTS algorithm proposed in this paper discretizes actions with high potential value using the probability distribution output by the policy network, successfully transforming the continuous action space search problem into a finite space search problem. This design enables the search process to adaptively focus on action sequences with high potential value, thereby achieving efficient exploration in the continuous control space. The LSTM encoder, policy network, and value network form a collaborative decision-making system: the LSTM encoder handles the uncertainty introduced by partial observability by encoding historical observation sequences; the value network smoothes the effects of short-term stochasticity by predicting long-term rewards; and the policy network ensures the continuity and robustness of actions through parameterized outputs. This collaborative mechanism not only improves the algorithm's decision-making quality in complex traffic scenarios but also ensures the smoothness and enforceability of control commands. As training progresses, the value network's estimates gradually approach the optimal value function, providing a reliable basis for the agent's decision-making.

[0159] 2.4. Multi-layer reward mechanism design:

[0160] To address the problem of multi-agent collaborative control at intersections without signal lights, this paper designs a hierarchical reward mechanism, which includes two dimensions: individual-level rewards and system-level rewards, to achieve effective guidance from local behavior to global optimization.

[0161] 2.4.1 Individual Level Rewards:

[0162] Individual-level rewards focus on evaluating the driving performance of a single agent at time t, encompassing three dimensions: traffic efficiency, safety, and driving comfort:

[0163] (1) Safety reward: Encourage vehicles to maintain a safe distance from other vehicles. This paper introduces a double-circle model to approximate the collision zone of vehicles and defines the safe distance between vehicles accordingly:

[0164]

[0165] Where L is the vehicle wheelbase and W is the vehicle track width. Based on the safety distance, the following safety reward function is constructed:

[0166]

[0167] Among them, w safe is the traffic safety reward weight, is the minimum distance between vehicle i and its surrounding vehicles at time t. When the distance between vehicles is less than the safe distance, the smaller the distance, the greater the penalty.

[0168] (2) Traffic efficiency reward: To encourage vehicles to pass through intersections efficiently, the reward function is defined as:

[0169]

[0170] Among them, w eff The hyperbolic tangent function is used to nonlinearly map the normalized speed, suppressing speeding while providing approximately linear positive incentives in the medium and high speed areas.

[0171] (3) Driving comfort reward: To encourage smooth driving of the vehicle, an acceleration-based comfort evaluation function is introduced:

[0172]

[0173] Among them, w comf is the driving comfort reward weight, and σ is a smoothing factor that adjusts the sensitivity of acceleration changes to comfort assessment. This function penalizes drastic acceleration changes exponentially, thereby encouraging the agent to adopt a smooth driving strategy.

[0174] 2.4.2 System-level rewards:

[0175] To improve the collaborative decision-making ability of multi-agent systems and optimize the overall performance of the transportation system, this paper introduces a global reward mechanism. This mechanism is mainly based on the following two key performance indicators:

[0176] (1) Overall traffic efficiency reward: Encourages reducing the average travel time of all vehicles. The reward function is defined as:

[0177]

[0178] Among them, w time is the weight coefficient of the average travel time; and are the time it takes for vehicle i to enter and exit the intersection. The negative sign indicates that we want to minimize the average travel time.

[0179] (2) System safety reward: Encourages reducing the number of collisions at intersections. The reward function is defined as:

[0180]

[0181] Among them, w collision is the weight coefficient of the number of collisions; C t is the total number of collisions at time t.

[0182] 2.4.3. Reward Integration:

[0183] The individual-level rewards and system-level rewards are combined through weighted combination. The total reward obtained by all agents in the environment at time t is:

[0184]

[0185] Where β is a weight coefficient used to adjust the proportion of global rewards in the total rewards. The relevant reward parameter settings are shown in the following table.

[0186] Table 1, reward parameter setting table:

[0187] parameter describe Value <![CDATA[w eff ]]> Traffic efficiency reward weight 0.3 <![CDATA[w safe ]]> Traffic safety reward weight 0.6 <![CDATA[w comf ]]> Driving comfort reward weight 0.1 <![CDATA[w time ]]> Weight coefficient of average travel time 1.0 <![CDATA[w collisiin ]]> The weight coefficient of the collision penalty 10.0 σ Smoothing factor 2 β The proportion of global rewards in the total rewards 0.6

[0188] 3. Simulation results and analysis:

[0189] 3.1 Experimental setup

[0190] 3.1.1, Simulation environment:

[0191] This study uses the open-source autonomous driving simulation platform CARLA (Car Learning to Act) for experimental verification. CARLA can simulate complex urban scenes, diverse vehicle behaviors, and realistic sensor data, providing a reliable environment for the development and testing of autonomous driving algorithms. In the experiment, the Town10 map built into CARLA was used to construct an unsignaled intersection scene, such as Figure 3 As shown in Figure 2, the intersection is a three-way intersection with four 100-meter-long lanes in each direction. The vehicle model used in the experiment is CARLA's built-in model 3. To simulate the diversity of real-world traffic flow, the vehicle's initial position, speed, and driving intention are randomly generated on the lane.

[0192] In order to comprehensively evaluate the performance of the algorithm under different traffic conditions, the experiment set three traffic density levels: low, medium, and high, corresponding to the situations where there are 5, 16, and 24 vehicles in the intersection, respectively.

[0193] In order to quantitatively evaluate the performance of the algorithm, this paper uses the following three indicators: successful passage rate B succ 、Average travel time B att and safety factor B safe :

[0194]

[0195] Among them, N succ Indicates the number of vehicles that successfully complete the navigation task, N coll The main simulation parameters are shown in Table 2.

[0196] Table 2 Simulation parameters

[0197]

[0198]

[0199] 3.2 Algorithm comparison experiment:

[0200] In order to verify the performance advantage of the LPV-MCTS algorithm in the unsignaled intersection scenario, this paper selected three typical multi-agent reinforcement learning algorithms as comparison benchmarks: Multi-Agent Deep Deterministic Policy Gradient (MADDPG), Multi-Agent Proximal Policy Optimization (MAPPO), and Multi-Agent Soft Actor-Critic (MA-SAC). The reward curves of the algorithms during training are compared. Figure 4 , performance indicators such as Figure 5 shown.

[0201] from Figure 4 Judging from the training process, the LPV-MCTS algorithm demonstrates significant advantages in several key areas: First, it exhibits superior exploration capabilities early in training, with minimal negative rewards (approximately -30) and rapid positive transitions. Second, thanks to the LSTM module's effective processing of historical information, the algorithm converges faster, reaching a stable state around 74,000 steps. Most importantly, the confidence interval of the curve is significantly smaller than that of other algorithms, demonstrating the high stability of its decisions. In comparison, while other baseline algorithms eventually converge, they exhibit significant deficiencies in training stability and convergence speed: MAPPO requires approximately 77,500 steps to converge, while MADDPG and MA-SAC require 97,000 and 106,500 steps, respectively, and all exhibit significant fluctuations. This result demonstrates the superiority of LPV-MCTS for complex decision-making problems in partially observable environments.

[0202] from Figure 5 In low-density scenarios, all algorithms demonstrated strong performance, but LPV-MCTS maintained a clear advantage. Its success rate reached 97.5%, its safety rate reached 98.5%, and its average travel time was only 12.5 seconds, outperforming the baseline algorithm in all three key metrics. In particular, compared to MAPPO, the closest performing algorithm, LPV-MCTS achieved improvements of 2.2% in success rate and 12.0% in travel time.

[0203] As traffic density increases, the performance differences between the algorithms become more pronounced. In medium-density scenarios, LPV-MCTS maintains a 94.8% success rate and a 97.2% safety rate, while keeping the average travel time to 14.8 seconds, demonstrating good scalability. In contrast, the performance of the other algorithms declines to varying degrees, particularly in terms of travel efficiency. The average travel times for MADDPG, MAPPO, and MA-SAC increase to 18.9 seconds, 16.5 seconds, and 19.4 seconds, respectively.

[0204] In high-density traffic scenarios, the complex traffic environment places higher demands on the algorithm's decision-making capabilities. Experimental results show that LPV-MCTS maintains a stable performance advantage: a 91.2% success rate, a 95.8% safety rate, and an average travel time of 17.2 seconds. This result validates the effectiveness of the proposed algorithm in handling highly dynamic and complex traffic environments. Notably, the performance of other algorithms degrades significantly in this scenario, particularly in terms of travel efficiency, with average travel times generally exceeding 20 seconds.

[0205] The experimental results above demonstrate that LPV-MCTS demonstrates excellent performance across a wide range of traffic densities, maintaining high efficiency and safety in complex, high-density scenarios. This fully demonstrates the effectiveness of the LSTM-based temporal information processing and policy-value network collaborative decision-making mechanism in improving algorithm performance.

[0206] 3.3 LSTM ablation experiment:

[0207] To further explore the impact of the LSTM module on decision-making performance, this study quantitatively analyzed the contribution of the LSTM module through ablation experiments in a medium-density traffic flow environment. In the experiment, other network architectures, hyperparameter configurations, and environmental parameters were kept unchanged, and only the performance differences with and without the LSTM module were compared. The experimental results are as follows: Figure 6 and as shown in Table 3.

[0208] Table 3 Convergence analysis table

[0209]

[0210] Depend on Figure 6The experimental results in Table 3 show that the LPV-MCTS algorithm significantly outperforms the PV-MCTS algorithm in terms of convergence speed, final performance, and stability. Specifically, the LPV-MCTS algorithm converges to a positive reward more quickly in the early stages of training, while the PV-MCTS algorithm takes longer to reach a similar level. This demonstrates that the LSTM module, by memorizing historical information, effectively accelerates the agent's transition from the exploration phase to the effective learning phase. Regarding convergence speed, the LPV-MCTS algorithm reaches 90% of its maximum reward value in approximately 78,017 steps, while the PV-MCTS algorithm requires approximately 108,029 steps to achieve the same performance level, a convergence speed improvement of approximately 27.8%. Furthermore, the final reward value of the LPV-MCTS algorithm is 20.85 higher than that of the PV-MCTS algorithm, indicating that the LSTM module significantly improves the algorithm's performance ceiling. More importantly, the LPV-MCTS algorithm's reward curve after convergence is smoother, with less fluctuation, demonstrating greater stability. In contrast, the PV-MCTS algorithm's reward curve exhibits greater fluctuations, even showing a trend of performance degradation in the later stages of training. PV-MCTS requires 90,000 steps to stabilize. This suggests that, without the LSTM module, the PV-MCTS algorithm struggles to effectively handle the uncertainty in some observable environments, resulting in unstable learning and limited performance.

[0211] Ablation experiments clearly demonstrate the crucial role of the LSTM module in the LPV-MCTS algorithm. By effectively leveraging historical observations, the LSTM module not only accelerates algorithm convergence and improves final performance, but also enhances the algorithm's stability in partially observable environments. These advantages enable the LPV-MCTS algorithm to more effectively solve complex decision-making problems at unsignaled intersections, providing reliable technical support for the coordinated control of autonomous vehicles.

[0212] 4. Conclusion

[0213] This study addresses the problem of coordinated scheduling of autonomous vehicles at intersections without signal lights and proposes an intelligent scheduling algorithm based on LSTM-enhanced policy-value MCTS (LPV-MCTS). This algorithm organically combines deep reinforcement learning, Monte Carlo tree search, and LSTM networks to construct a "policy-value" dual-network collaborative decision-making framework, and achieves efficient multi-agent collaborative control through centralized training and distributed execution. The core of the LPV-MCTS algorithm is to use the LSTM network to encode the historical observation sequence of the vehicle, thereby capturing the temporal characteristics of vehicle behavior and improving the state representation ability and decision robustness in partially observable environments. At the same time, through the MCTS search driven by the policy network and the value network, more accurate state evaluation and better sequential decision-making can be achieved.

[0214] Simulation results demonstrate that the proposed LPV-MCTS algorithm can effectively solve the problem of coordinated dispatching of autonomous vehicles at intersections without signal lights, achieving significant results in improving traffic efficiency, ensuring driving safety, and enhancing system robustness. This algorithm provides a new and effective solution for the coordinated control of multi-agent systems in complex dynamic environments, with important theoretical significance and application value. Future research will further explore the generalization capabilities of the LPV-MCTS algorithm in more complex and diverse traffic scenarios, consider extending the algorithm to heterogeneous agent scenarios, explore how to integrate the LPV-MCTS algorithm with real-world sensor data and vehicle control systems to achieve migration from simulation to reality, and study improvements to the algorithm under conditions with communication constraints.

[0215] The protection scope of the present invention includes but is not limited to the above embodiments. The protection scope of the present invention is based on the claims. Any replacement, deformation, and improvement of the technology that can be easily thought of by those skilled in the art fall within the protection scope of the present invention.

Claims

1. A dual-network oriented intelligent vehicle dispatching method, characterized in that: The following steps are involved: Step 1: Obtain local observation information of the intelligent vehicle; Step 2: Encode the historical local observation information of the intelligent vehicle into a hidden state vector; Step 3: Input the hidden state vector into the policy network and the value network respectively. The policy network outputs the probability distribution of the action, and the value network outputs the value estimate of the state. Step 4: The MCTS algorithm uses the output of the policy network and the value network to search and finally output the optimal action; Step 5: The intelligent vehicle executes the corresponding control instructions based on the optimal action searched by the MCTS algorithm.

2. The dual-network-oriented intelligent vehicle dispatching method according to claim 1, characterized in that: The step 1 specifically includes: Intelligent vehicle i obtains its own local observation information through on-board sensors at time t in, represents the set of neighboring vehicles communicating with smart vehicle i at time t; represents the state of intelligent vehicle i at time t, represents the position coordinates of intelligent vehicle i at time t; represents the speed of intelligent vehicle i at time t; represents the heading angle of intelligent vehicle i at time t; is the navigation information of smart vehicle i at time t, including the destination coordinates of smart vehicle i The distance between the current position of intelligent vehicle i and the destination The relative direction between the current position of intelligent vehicle i and the destination 3. The dual-network-oriented intelligent vehicle dispatching method according to claim 1, characterized in that: The step 2 specifically includes: The LSTM network encodes the historical local observation information of intelligent vehicle i into a hidden state vector; The input of the LSTM network is the observation information sequence O of the intelligent vehicle i in the past H control cycles t-H:t , O t-H:t ={o t-H+1 ,o t-H+2 …o t }; Among them, o t That is, the local observation information of intelligent vehicle i at time t At each time step t, the LSTM network receives the observation o at the current time t. t , the hidden state vector h at the previous moment t-1 and cell state c t-1 As input, the network state is updated through the gating mechanism: h t ,c t =LSTM(o t ,h t-1 ,c t-1 ); After H time steps of iterative calculation, the hidden state vector h is finally output by the LSTM network. t .

4. The dual-network-oriented intelligent vehicle dispatching method according to claim 2, characterized in that: The step 3 specifically includes: (1) The hidden state vector h t Input policy network; The policy network uses a multi-dimensional Gaussian distribution to model the action probability distribution: in It means that the mean is μ and the variance is σ 2 Gaussian distribution, μ d (h t ) represents the expected value of the d-th dimension action; represents the variance of the d-th dimension action, the parameter θ of the policy network π Updated through the policy gradient algorithm; (2) The hidden state vector h t Enter the value network; The value network outputs a scalar value estimate V(h t θ v ): Where T is the total duration of the scheduling round, γ is the discount factor, and r k represents the immediate reward of the environment at time k, where k is the time step index; Parameters θ of the value network v Updates are made by minimizing the mean squared error between the predicted value and the true value.

5. The dual-network oriented intelligent vehicle dispatching method according to claim 1, characterized in that: The step 4 specifically includes: The MCTS algorithm uses the prior probabilities provided by the policy network and the value estimates provided by the value network to find the optimal action through iterative search. Each iteration of the MCTS algorithm consists of four steps: selection, expansion, simulation, and backtracking. choose: Starting from the root node, recursively select child nodes until a leaf node is reached; The strategy for selecting child nodes is based on the output of the policy network and the PUCT formula: a * =argmax a [Q(h t ,a)+U(h t ,a)]; Among them, Q(h t ,a) indicates the state h t The average cumulative reward after executing action a; Indicates that from state h t The set of all simulated trajectories obtained after starting and taking action a, and R(τ) represents the cumulative reward of trajectory τ; the upper limit control term U(h t ,a) The following improved PUCT formula is adopted: in, is the prior probability distribution obtained by normalizing the policy network output probability of the sampled action set; c puct is the exploration constant, b is used to traverse the current state node h t An index variable for all candidate actions, N(h t ,b) is the number of visits, Σ b N(h t ,b) represents the total number of visits to the current node; Extensions: If the selected leaf node is not the terminal state, then according to the probability distribution π(a|h t θ π ), sample m actions and create m new child nodes; simulation: Starting from the newly created child node, use the policy network π(a|h t θ π ) until the simulation reaches the termination state or the maximum simulation depth D sim ; After the simulation is completed, a cumulative reward G is obtained; Backtrack: Starting from the leaf node, backpropagate the cumulative reward G or the output of the value network along the search path, and update the number of visits N (h t ,a) and cumulative value Q(h t ,a): N(h t ,a)←N(h t ,a)+1; Among them, the target value V target Varies depending on the leaf node type: If the leaf node is a new expansion node, then V target The value is the output of the value network; if the leaf node is in the simulation termination state or reaches the maximum simulation depth, the value is the cumulative reward G obtained by the simulation; The four steps of each iteration of the MCTS algorithm are executed in a sequential loop, and each iteration updates the information of the search tree, allowing it to gradually converge to the optimal strategy over time.

Images