Robust monte carlo tree search based decision control method for autonomous vehicle

Abstract

The application relates to a kind of automatic driving car decision control methods based on robust Monte Carlo tree search, comprising: establishing Markov decision model with disturbance lane-changing scene;Respectively construct adversarial disturbance network and deep reinforcement learning network based on Actor-Critic, generate adversarial disturbance using adversarial disturbance network, and use deep reinforcement learning network to guide action selection;Combined with the output of adversarial disturbance network and deep reinforcement learning network, the output of automatic driving car lane-changing decision action instruction is output using Monte Carlo tree search mode, which is used to control the driving state of automatic driving car.Compared with prior art, the application actively simulates environmental disturbance, and deeply integrates disturbance mechanism into search tree structure, introduces adversarial disturbance in the search and training process, which can make automatic driving car fully consider the possibility of future scene, effectively improve the safety and robustness of automatic driving car in lane-changing decision.

Description

Technical Field

[0001] This invention relates to the field of autonomous vehicle control technology, and in particular to an autonomous vehicle decision control method based on robust Monte Carlo tree search. Background Technology

[0002] With the development of autonomous driving technology, ensuring safe and robust decisions by vehicles in complex and ever-changing traffic scenarios has become a core challenge. Currently, decision-making and planning methods for autonomous vehicles (AVs) mainly include rule-based methods, deep reinforcement learning (DRL), and Monte Carlo tree search (MCTS).

[0003] Deep reinforcement learning is widely used due to its ability to handle high-dimensional state spaces and nonlinear problems. However, pure reinforcement learning methods are often limited by out-of-distribution risks in the environment. When faced with extreme conditions or traffic participants with diverse intentions, the models often exhibit vulnerability, making it difficult to guarantee the safety of decisions. To improve the foresight of decisions, some existing solutions introduce Monte Carlo tree search, which simulates possible future evolutions through online simulation. However, traditional Monte Carlo tree search methods typically assume that the environment follows a fixed transition probability distribution, failing to fully consider the adversarial perturbations or extreme uncertainties that may exist in the traffic environment.

[0004] It can be said that existing vehicle decision-making technologies still have significant limitations in practical applications. Most decision-making models are trained based on standard traffic scenarios and lack active simulation of extreme and adversarial scenarios. This results in poor robustness of the system when facing unexpected environmental disturbances, making it prone to making incorrect and dangerous lane-changing decisions, which reduces vehicle safety and robustness. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an autonomous vehicle decision control method based on robust Monte Carlo tree search, which can ensure that autonomous vehicles can effectively cope with extreme and adversarial scenarios and improve the accuracy of vehicle lane-changing decisions.

[0006] The objective of this invention can be achieved through the following technical solution: an autonomous vehicle decision-making and control method based on robust Monte Carlo tree search, comprising the following steps: S1. Establish a Markov decision model with a lane-changing scenario involving disturbances; S2. Based on the Markov decision model, an adversarial perturbation network and an Actor-Critic-based deep reinforcement learning network are constructed respectively. The adversarial perturbation network is used to generate adversarial perturbations, and the deep reinforcement learning network is used to guide action selection. S3. Combining the outputs of the adversarial perturbation network and the deep reinforcement learning network, the Monte Carlo tree search method is used to output lane-changing decision instructions for autonomous vehicles, which are used to control the driving state of autonomous vehicles.

[0007] Furthermore, S1 specifically involves constructing a highway simulation scenario with multiple lanes and modeling the decision-making process of the autonomous vehicle as a Markov decision model containing adversarial perturbations. The optimization objective of this Markov decision model is to find a strategy that maximizes the cumulative reward for the entire simulation round. This Markov decision model is expressed in the form of a six-tuple, which includes a partially observable state space, an action space, a multi-objective optimization reward function, a state transition probability, a loss factor, and a set of adversarial perturbations.

[0008] Furthermore, the partially observable state space includes vehicle motion features and local environmental features. The vehicle motion features include acceleration, heading angle, longitudinal speed, and the current lane number. The local environmental features are centered on the vehicle and acquire traffic information of the lane where the vehicle is located and the adjacent lanes on the left and right through onboard sensors. The relative speed and relative position of the nearest forward and backward vehicles in the lane are recorded in a fixed order. If a vehicle in a specific location is missing within the observation range, the corresponding state bit is filled with a preset value of 0.

[0009] Furthermore, the action space is specifically a set of discrete actions designed for the lateral high-level decision-making layer of autonomous vehicles, including lane keeping actions, left lane changing actions, and right lane changing actions.

[0010] Furthermore, the multi-objective optimization reward function includes traffic efficiency rewards, frequent lane-changing penalties, and safety penalties.

[0011] Furthermore, the state transition probability is used to characterize the autonomous vehicle in its current state. Next action Then, the simulation scenario evolves to the state of the next moment. The probability distribution; The depreciation factor is used to adjust the weighting relationship between current rewards and future long-term rewards when calculating cumulative rewards.

[0012] Furthermore, the adversarial perturbation set is used to define the observation state. Modifiable boundary range threshold That is, for any state There exists a perturbed state. ,in , Let be the perturbation vector. To counteract the set of disturbances.

[0013] Furthermore, the process of constructing the adversarial perturbation network in S2 includes: It employs a multi-layer fully connected neural network architecture, with the input being the state vector of the current environment. The output is a perturbation vector with the same dimension as the state space. The perturbation vector Subject to boundary range threshold Constraints ; During network training, Jensen-Shannon divergence is used as the core optimization metric, and the loss function is designed to maximize the original policy distribution. Distribution of policies after perturbation The difference between them is that in each training iteration, the adversarial network adjusts its own weights based on the performance of the current policy network through the backpropagation algorithm, searching for the direction that causes the vehicle's decision to fluctuate the most.

[0014] Furthermore, the process of constructing the Actor-Critic-based deep reinforcement learning network in S2 includes: An Actor network is constructed as the policy network, and a Critic network is constructed as the value network. The input of the policy network is the state space. The output is the action space. Probability distribution of discrete actions The input to the value network is the state space. The output is the expected cumulative reward scalar value for this state. ; The policy network is trained with adversarial perturbations. The perturbation vectors generated by the adversarial perturbations are injected into the environmental observations in real time. The training objective is to make the policy work in the perturbation state. The output action distribution and the original state The distribution is consistent with the following; The temporal difference method is used to update and train the value network.

[0015] Further, S3 includes the following steps: S31, Real-time status As the root node of the Monte Carlo tree search, the probability distribution of the root node's actions is calculated using the trained policy network, and then used as the initial prior probability to be assigned to each branch action to be expanded. S32. Starting from the root node, recursively select child nodes for expansion using the upper confidence limit criterion. During the node expansion process, use an adversarial perturbation network to perturb the simulated state of the current node. S33. When the search reaches an unexpanded leaf node or reaches a preset depth, stop the simulation and directly output the value evaluation value of the leaf node using the trained value network. ; S34. Evaluate the value of the leaf nodes. Propagate backward along the search path to the root node, updating the visit count of each node along the way during the backtracking process. Cumulative action value ; S35. When the preset number of iterations or computation time threshold is reached, stop the search. By comparing the statistical data of each action branch under the root node, select the branch with the most or average visits. The action with the highest value is sent to the vehicle's underlying actuator as the optimal decision instruction at the current moment; S36. After the vehicle executes the optimal decision instruction, it returns to step S31, reconstructs the search tree based on the updated sensor observation data, and enters the next decision cycle.

[0016] Compared with the prior art, the present invention has the following advantages: This invention first establishes a Markov decision model with a lane-changing scenario involving disturbances. Based on this, an adversarial disturbance network and an Actor-Critic-based deep reinforcement learning network are constructed. Finally, the outputs of the adversarial disturbance network and the deep reinforcement learning network are combined (the adversarial disturbance network outputs adversarial disturbances, and the deep reinforcement learning network outputs information to guide action selection). Using a Monte Carlo tree search method, the lane-changing decision action instructions for the autonomous vehicle are obtained. By actively simulating environmental disturbances and deeply integrating the disturbance mechanism into the search tree structure, the safety and robustness of autonomous vehicles in lane-changing decisions can be improved.

[0017] This invention models the decision-making process of autonomous vehicles in complex traffic environments as a six-tuple comprising a partially observable state space, an action space, a multi-objective optimization reward function, a state transition probability, a loss factor, and an adversarial perturbation set. The adversarial perturbation set defines the boundary range within which the observed state can be modified, thereby incorporating perturbations into the Markov decision process. This facilitates autonomous vehicles in learning obstacle avoidance strategies under extreme evolution and ensures that their lane-changing decisions have the ability to cover unknown risks.

[0018] This invention, after constructing a decision-making model incorporating uncertainty, actively generates adversarial perturbations by building and training a parameterized adversarial network. To ensure the generated perturbations most effectively simulate uncertainty, Jensen-Shannon divergence is used as the core optimization metric, and the loss function of the adversarial network is designed to maximize the difference between the original policy distribution and the perturbed policy distribution. In each training iteration, the adversarial network adjusts its weights based on the current policy network performance using a backpropagation algorithm, searching for the direction that maximizes the fluctuation in the vehicle's decision. Using the trained adversarial perturbation network, the evolution of uncertainty in traffic scenarios can be fully simulated, i.e., the extreme situations that may exist in future traffic scenarios can be fully simulated.

[0019] This invention constructs a deep reinforcement learning network based on the Actor-Critic framework. The Actor network aims to establish a mapping from state to action, while the Critic network aims to evaluate the merits of the current state. This invention uses this deep reinforcement learning network as a policy guidance module for subsequent Monte Carlo tree search, which can improve search efficiency and learning stability, and ensure that vehicles make optimal decisions in dynamic and uncertain traffic environments.

[0020] This invention constructs a perturbation-based Monte Carlo tree for search decision-making, incorporating the perturbation evolution of future scenarios into the tree node expansion process. By constructing a search tree that includes the evolution of future uncertainties, robust lane-changing decisions can be generated for autonomous vehicles. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram illustrating the application process of an example. Figure 3 This is a schematic diagram of a highway lane-changing scenario with disturbances constructed in the embodiment. Figure 4 The network architecture diagram of the adversarial network and Actor-Critic deep reinforcement learning in the embodiment is shown. Figure 5 A schematic diagram of the Monte Carlo tree search decision process; Figure 6 This is a schematic diagram of the simulation results of autonomous vehicle decision-making in a certain highway scenario in the embodiment. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0023] Example like Figure 1As shown, an autonomous vehicle decision-making and control method based on robust Monte Carlo tree search includes the following steps: S1. Establish a Markov decision model with a lane-changing scenario involving disturbances; S2. Based on the Markov decision model, an adversarial perturbation network and an Actor-Critic-based deep reinforcement learning network are constructed respectively. The adversarial perturbation network is used to generate adversarial perturbations, and the deep reinforcement learning network is used to guide action selection. S3. Combining the outputs of the adversarial perturbation network and the deep reinforcement learning network, the Monte Carlo tree search method is used to output lane-changing decision instructions for autonomous vehicles, which are used to control the driving state of autonomous vehicles.

[0024] This embodiment applies the above solution, and the main process is as follows: Figure 2 As shown, it mainly includes: I. Constructing a simulation scenario model for partially observable highway lane-changing decisions under increased disturbances; 2. Design an adversarial training network to simulate the evolution of uncertainty in traffic scenarios by generating adversarial perturbations through constraints; III. Using deep reinforcement learning networks with an Actor-Critic structure to guide action selection in Monte Carlo tree search; Fourth, construct a Monte Carlo tree search decision tree under perturbation, integrate the perturbation evolution of future scenarios into the tree node expansion process, and use the learned optimal strategy to guide the action selection of Monte Carlo tree search.

[0025] In step one, this embodiment establishes a highway decision-making environment capable of simulating the complexity and uncertainty of the real world, such as... Figure 3 As shown, a four-lane highway scenario is constructed in the simulation platform. An autonomous vehicle is designated as the decision-making controlled object, and several human-driven vehicles are deployed as the interaction background. This scheme actively introduces adversarial perturbations during the construction process to simulate the diversity of traffic participants' intentions and the potential evolution of extreme behaviors. Figure 3 (a) and (b) are simulation schemes of existing technologies. In the scenario, the vehicle movement is set to a fixed intelligent agent mode for autonomous vehicle decision training. The autonomous vehicle believes that the vehicle in the adjacent lane will maintain a constant speed and thus makes a lane-changing decision. However, in reality, there is a risk of the vehicle suddenly accelerating or cutting in, i.e., a disturbance situation, which causes the autonomous vehicle's original decision to result in a traffic accident. Figure 3 Figure (c) illustrates the Markov decision model constructed by this invention for a highway lane-changing scenario with perturbations. This scheme incorporates such worst-case perturbations into the six-tuple of the Markov decision process. In this process, autonomous vehicles learn obstacle avoidance strategies under extreme evolutionary conditions to ensure that their lane-changing decisions cover unknown risks. The optimization objective of the entire decision-making problem is to find a strategy that maximizes the cumulative reward over the entire simulation round, as shown in the following equation: Regarding the state space of the above Markov decision model This scheme defines a partially observable space. Vehicle motion characteristics include acceleration, heading angle, longitudinal velocity, and the current lane number. Local environmental characteristics are centered on the vehicle, using onboard sensors to acquire traffic information about the vehicle's current lane and adjacent lanes. The relative speeds and positions of the nearest forward and backward vehicles within each lane are recorded in a fixed order. If a vehicle at a specific location is missing from the observation range, its corresponding status bit is filled with a preset value of 0.

[0026] Action space of Markov decision models This solution defines it as a set of discrete actions designed for the lateral advanced decision-making layer of autonomous vehicles. Specifically, this set includes lane keeping actions, left lane changing actions, and right lane changing actions.

[0027] To guide the Markov decision model in balancing various driving needs in highway scenarios, this solution constructs a multi-objective optimization reward function. The calculation formula is as follows: In the formula, These are preset weighting coefficients; This indicates a traffic efficiency bonus, directly proportional to vehicle speed; This indicates a penalty for frequent lane changes, specifically for the behavior of changing lanes too frequently with too short intervals. This represents the safety penalty, i.e., the penalty value when a collision occurs. Additionally, it represents the state transition probabilities in the model. Used to characterize the probability distribution of the simulation scenario evolving to the next state after an autonomous vehicle performs an action in the current state; loss factor This is a preset constant used to adjust the weighting relationship between current rewards and future long-term rewards.

[0028] As a core feature of this scheme, an adversarial perturbation set is introduced into the Markov decision model. Unlike existing technologies, this set defines the boundary range within which the observed state can be modified, that is, for any state... There exists a perturbed state. ,in .

[0029] In step two, after constructing the Markov decision model with uncertainties, this scheme actively generates adversarial perturbations by constructing and training a parameterized adversarial network to simulate extreme evolutions that may occur in traffic scenarios. For example... Figure 4 As shown, this adversarial perturbation network adopts a multi-layer fully connected neural network architecture, and its specific construction and training process is as follows: The input to this network is the state vector of the current environment. The network contains two hidden layers, each fully connected with 256 neurons, and both layers use the ReLU activation function for non-linear mapping. The network's output layer outputs a perturbation vector with the same dimension as the state space. To ensure the generated perturbation is physically plausible, the network output is typically constrained by a preset amplitude coefficient. In this scheme, the perturbation vector... Subject to boundary range threshold Constraints .

[0030] To ensure that the generated perturbations most effectively simulate uncertainty, this scheme employs Jensen-Shannon divergence as the core optimization metric. The loss function of the adversarial network is designed to maximize the original policy distribution. Distribution of policies after perturbation The differences between them. In each training iteration, the adversarial network adjusts its own weights through backpropagation based on the performance of the current policy network, searching for the direction that causes the vehicle's decision to fluctuate the most.

[0031] In step three, to make optimal decisions in a dynamic and uncertain traffic environment, this scheme designs a deep reinforcement learning network based on the Actor-Critic framework and uses it as a policy guidance module for subsequent Monte Carlo tree search. Figure 4 As shown, the Actor network aims to establish a mapping from state to action. Its input... The state vector is processed through two ReLU fully connected layers with 256 neurons each to extract features. The output layer then outputs the probability distribution of three decision actions in the action space. . like Figure 4 As shown, the Critic network aims to evaluate the quality of the current state. Its network structure is similar to the Actor network; the input is also a state vector, but the output layer outputs a scalar value representing the expected cumulative reward for that state. .

[0032] During training, the adversarial perturbations generated in step two will be used... Real-time injection of environmental observations; its training objective is to make the policy work under perturbed states. The output action distribution and the original state The distribution is kept as consistent as possible to achieve robust policy network training with adversarial perturbations. The Critic network is updated using a temporal difference method, with the update formula as follows: in, These are the parameters of the Critic network, used to complete the training of the value network.

[0033] In step four, as Figure 5 As shown, a search tree containing future uncertainties is constructed using a pre-trained Actor-Critic network and an adversarial perturbation network to generate robust lane-changing decisions for autonomous vehicles.

[0034] First, the search tree is initialized and the root node is defined. In each decision cycle, the real-time status of the vehicle is obtained. The root node is used as the root node in the Monte Carlo tree search. The Actor network trained in step three is used to calculate the action probability distribution of the root node, and this distribution is used as the initial prior probability to assign to each branch action to be expanded.

[0035] Next, node selection and expansion are performed. The search tree starts from the root node and recursively selects child nodes using the upper confidence limit criterion. During node expansion, the adversarial network from step two is invoked to perturb the simulated state of the current node, and child nodes are recursively selected using the upper confidence limit criterion, as shown in the following equation: in, For the value of the action, For the number of visits, To explore constants.

[0036] Next, leaf node evaluation based on the value network is performed. The simulation stops when the search reaches an unexpanded leaf node or a preset depth. The Critic network trained in step three directly outputs the value evaluation value of the leaf node. .

[0037] Perform path value backtracking and statistical updates, and update the evaluation value of leaf nodes. Propagate backward along the search path to the root node. During the backtracking process, the visit count of each node along the way is updated synchronously. Cumulative action value .

[0038] Finally, the optimal decision action is output, and the search stops when the preset number of iterations or computation time threshold is reached. By comparing the statistical data of each action branch under the root node, the action with the most or average visits is selected. The action with the highest value is sent to the vehicle's underlying actuator as the optimal decision instruction at the current moment.

[0039] Furthermore, a closed-loop rolling decision-making process is implemented. After the vehicle executes the optimal decision instruction, it enters the next decision cycle. The system will reconstruct the search tree based on updated sensor observation data to ensure that the autonomous vehicle maintains the safety and robustness of its decisions in complex dynamic traffic flows.

[0040] To verify the effectiveness of the robust decision-making algorithm proposed in this scheme in actual dynamic traffic flow, this embodiment conducts simulation experiments. Figure 6 This visualization showcases the spatiotemporal evolution of the autonomous vehicle's decision-making process. The horizontal axis represents the vehicle's longitudinal position on the road, and the vertical axis represents its lateral position, corresponding to different lanes on a highway. The color intensity of the trajectory lines represents the passage of time. Figure 6 As shown, in the initial stage of the simulation, the autonomous vehicle maintains a stable following state within the current lane. As time progresses, when the system detects that the speed of vehicles ahead in the current lane decreases or the traffic density increases, leading to reduced traffic efficiency and a potential collision risk, the thick solid line trajectory in the figure shows that the proposed method, after fully simulating the possible cutting-in behaviors and uncertainties of surrounding vehicles, determines that changing lanes to the adjacent lane with less traffic is the optimal strategy. The system then controls the vehicle to execute a smooth and safe lane-changing maneuver, successfully escaping the low-speed traffic area. Throughout the process, the autonomous vehicle's trajectory is smooth and continuous, without overlapping with the trajectories of densely packed human-driven vehicles. This fully demonstrates that the proposed method can effectively predict future risks and generate robust decisions that balance safety and traffic efficiency in complex and ever-changing highway interaction scenarios. It solves the problem of insufficient safety of traditional decision-making methods in uncertain environments and helps reduce the risk of traffic accidents.

Claims

1. A decision-making and control method for autonomous vehicles based on robust Monte Carlo tree search, characterized in that, Includes the following steps: S1. Establish a Markov decision model with a lane-changing scenario involving disturbances; S2. Based on the Markov decision model, an adversarial perturbation network and an Actor-Critic-based deep reinforcement learning network are constructed respectively. The adversarial perturbation network is used to generate adversarial perturbations, and the deep reinforcement learning network is used to guide action selection. S3. Combining the outputs of the adversarial perturbation network and the deep reinforcement learning network, the Monte Carlo tree search method is used to output lane-changing decision instructions for autonomous vehicles, which are used to control the driving state of autonomous vehicles.

2. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 1, characterized in that, Specifically, S1 involves constructing a highway simulation scenario with multiple lanes and modeling the decision-making process of the autonomous vehicle as a Markov decision model containing adversarial perturbations. The optimization objective of this Markov decision model is to find a strategy that maximizes the cumulative reward for the entire simulation round. This Markov decision model is expressed in the form of a six-tuple, which includes a partially observable state space, an action space, a multi-objective optimization reward function, a state transition probability, a loss factor, and a set of adversarial perturbations.

3. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 2, characterized in that, The observable state space includes vehicle motion features and local environmental features. The vehicle motion features include acceleration, heading angle, longitudinal speed, and the current lane number. The local environmental features are based on the vehicle and acquire traffic information of the lane where the vehicle is located and the adjacent lanes on the left and right through onboard sensors. The relative speed and relative position of the nearest forward and backward vehicles in the lane are recorded in a fixed order. If a vehicle in a specific location is missing from the observation range, the corresponding state bit is filled with a preset value of 0.

4. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 2, characterized in that, The action space is specifically a set of discrete actions designed for the lateral high-level decision-making layer of autonomous vehicles, including lane keeping actions, left lane changing actions, and right lane changing actions.

5. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 2, characterized in that, The multi-objective optimization reward function includes traffic efficiency rewards, frequent lane-changing penalties, and safety penalties.

6. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 2, characterized in that, The state transition probability is used to characterize the autonomous vehicle in its current state. Next action Then, the simulation scenario evolves to the state of the next moment. The probability distribution; The depreciation factor is used to adjust the weighting relationship between current rewards and future long-term rewards when calculating cumulative rewards.

7. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 2, characterized in that, The set of adversarial perturbations is used to define the observation state. Modifiable boundary range threshold That is, for any state There exists a perturbed state. ,in , Let be the perturbation vector. To counteract the set of disturbances.

8. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 7, characterized in that, The process of constructing the adversarial perturbation network in S2 includes: It employs a multi-layer fully connected neural network architecture, with the input being the state vector of the current environment. The output is a perturbation vector with the same dimension as the state space. The perturbation vector Subject to boundary range threshold Constraints ; During network training, Jensen-Shannon divergence is used as the core optimization metric, and the loss function is designed to maximize the original policy distribution. Distribution of policies after perturbation The difference between them is that in each training iteration, the adversarial network adjusts its own weights based on the performance of the current policy network through the backpropagation algorithm, searching for the direction that causes the vehicle's decision to fluctuate the most.

9. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 8, characterized in that, The process of constructing the deep reinforcement learning network based on Actor-Critic in S2 includes: An Actor network is constructed as the policy network, and a Critic network is constructed as the value network. The input of the policy network is the state space. The output is the action space. Probability distribution of discrete actions The input to the value network is the state space. The output is the expected cumulative reward scalar value for this state. ; The policy network is trained with adversarial perturbations. The perturbation vectors generated by the adversarial perturbations are injected into the environmental observations in real time. The training objective is to make the policy work in the perturbation state. The output action distribution and the original state The distribution is consistent with that below; The temporal difference method is used to update and train the value network.

10. The autonomous vehicle decision-making and control method based on robust Monte Carlo tree search according to claim 9, characterized in that, S3 includes the following steps: S31, Real-time status As the root node of the Monte Carlo tree search, the probability distribution of the root node's actions is calculated using the trained policy network, and then used as the initial prior probability to be assigned to each branch action to be expanded. S32. Starting from the root node, recursively select child nodes for expansion using the upper confidence limit criterion. During the node expansion process, use an adversarial perturbation network to perturb the simulated state of the current node. S33. When the search reaches an unexpanded leaf node or reaches a preset depth, stop the simulation and directly output the value evaluation value of the leaf node using the trained value network. ; S34. Evaluate the value of the leaf nodes. Propagate backward along the search path to the root node, updating the visit count of each node along the way during the backtracking process. Cumulative action value ; S35. When the preset number of iterations or computation time threshold is reached, stop the search. By comparing the statistical data of each action branch under the root node, select the branch with the most or average visits. The action with the highest value is sent to the vehicle's underlying actuator as the optimal decision instruction at the current moment; S36. After the vehicle executes the optimal decision instruction, it returns to step S31, reconstructs the search tree based on the updated sensor observation data, and enters the next decision cycle.

Images