Monte Carlo tree search method and device based on reverse reanalysis and storage medium

Abstract

The invention relates to a Monte Carlo tree searching method and device based on reverse reanalysis and a storage medium, and the method comprises the steps: taking an initial state of a to-be-decided problem as a root node R0 of a Monte Carlo tree, assuming that a corresponding action space has n actions, then forming n child nodes of the root node R0, each sub-node is used as a root node of a sub-search tree; in the training stage, the whole buffer area is reanalyzed regularly, specifically, a reverse view angle reanalysis technology is adopted to reanalyze training samples so as to obtain value estimation of child nodes in advance, and the value estimation is used for assisting root node search; and completing a tree search traversal process according to a preset number of training iterations, and determining an action corresponding to the decision scheme. Compared with the prior art, time overhead can be saved, meanwhile, the method is not limited by hardware resources, higher universality is achieved, and hardware execution efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of game decision technology, and in particular to a Monte Carlo tree search method, device and storage medium based on reverse reanalysis. Background Art

[0002] MCTS (Monte Carlo Tree Search) is a decision-making method based on forward tree search for real-time planning. When executing MCTS, n simulations are performed starting from the current state to obtain n trajectories, which together form a search tree. Based on the information obtained during the simulation process, the quality of each node in the tree can be evaluated and a final decision can be made. Due to its excellent planning capabilities, MCTS is widely used in the field of model based RL (model-based reinforcement learning), such as the Go artificial intelligence AlphaZero, which has received widespread social attention.

[0003] MCTS has been proven to be an efficient method for planning using models. It incorporates the UCB1 (UpperConfidence Bound 1) algorithm into the tree search process and has achieved good results in a wide range of application scenarios. The UCB1 algorithm is a classic multi-armed bandit algorithm used to make the best choice among multiple choices. Specifically, AlphaZero has made remarkable achievements in combining deep reinforcement learning with MCTS, and is able to beat top human players; MuZero extends the algorithm to situations with unknown environmental models, thus performing well in a wider range of tasks. Following MuZero, many subsequent algorithms have emerged, further broadening the use scenarios of MuZero. However, the large amount of tree search calculations brings additional time overhead to these algorithms: during the data collection phase, the agent needs to perform MCTS to select an action every time it receives a new state. In addition, due to the characteristics of tree search, it is challenging to parallelize it using the commonly used vectorized environment method, further amplifying the speed disadvantage. On the other hand, during the reanalysis process, in order to obtain a higher quality update target, the latest model parameters are used to rerun MCTS on the samples in the buffer. Therefore, as the price of high sample efficiency, the time required to process samples increases with the improvement of sample efficiency, and the excessive time consumption has become a bottleneck hindering the further promotion of these algorithms.

[0004] That is to say, since MCTS is a real-time planning method, that is, the above-mentioned tree search process must be executed at every moment, which will undoubtedly bring huge time overhead, it is necessary to study how to save the time overhead of the MCTS-based algorithm.

[0005] In the prior art, SpeedyZero is a distributed RL (Reinforcement Learning) system based on the state-of-the-art model-based RL method EfficientZero, and designed for fast distributed computing. SpeedyZero achieves sample efficiency comparable to EfficientZero through massive parallelization and large-batch training, and achieves a 14.5-fold acceleration in wall clock time, achieving human-level performance in the Atari benchmark in just 35 minutes and 300,000 samples. Specifically, in order to address the training instability problems caused by massive parallelization and large-batch training, relevant researchers have developed two new algorithmic techniques: Priority Refresh and ClippedLARS. Priority Refresh is a distributed, more aggressive variant of prioritized experience replay, which actively recalculates the accurate priorities of all samples in the replay buffer, and ClippedLARS is a variant of the LARS (Least Angle Regression) algorithm, which is used to ensure stability during large-batch training. Experimental results show that SpeedyZero achieves human-level performance on the Atari 100k benchmark with only 35 minutes of training time and 300,000 samples. Compared with EfficientZero, which requires 8.5 hours of training, SpeedyZero significantly improves the training speed while maintaining comparable sample efficiency.

[0006] Although the method proposed by SpeedyZero can greatly save training time through the distributed training framework, building a distributed training framework has relatively high requirements for hardware resources. In the scenario of single-card training, due to the limitation of hardware overhead, this method cannot be applied, which limits the promotion of SpeedyZero. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a Monte Carlo tree search method, device and storage medium based on reverse reanalysis, which can save time and is not limited by hardware resources, thereby greatly improving hardware execution efficiency.

[0008] The object of the present invention can be achieved by the following technical solution: A Monte Carlo tree search method based on reverse reanalysis, comprising:

[0009] The initial state of the problem to be decided is used as the root node R0 of the Monte Carlo tree. Assuming that the corresponding action space has n actions, n child nodes of the root node R0 are formed, and each child node is the root node of a sub-search tree.

[0010] During the training phase, the entire buffer is reanalyzed periodically. Specifically, the training samples are reanalyzed using the reverse perspective reanalysis technique to obtain the value estimates of the child nodes in advance to assist the root node search.

[0011] According to the preset number of training iterations, the tree search traversal process is completed to determine the action corresponding to the decision-making plan.

[0012] Furthermore, the obtaining of the value estimation of the child node in advance specifically involves first obtaining the value of the root node by searching at a subsequent time, and then evaluating the value of the child node currently being searched based on the value of the root node.

[0013] Furthermore, the periodic re-analysis of the entire buffer is specifically to perform a reverse perspective tree search on the entire buffer in which the trajectories are collected according to a set period, that is, to search the trajectories in reverse order, first sampling n+1 trajectories from the buffer to form a batch, wherein each trajectory has a length of k+1, and using S l t represents the tth state in the lth trajectory, and then searches in the reverse direction of the trajectory: first search for all S k Search and then for all S k-1 to search.

[0014] Furthermore, the reverse perspective reanalysis technology includes a reverse reanalysis process from a node perspective and a reverse reanalysis process from a batch perspective.

[0015] Furthermore, the reverse reanalysis process from the node perspective is: first search Then the root node value Pass to To evaluate the value of a child node, They are the corresponding search trees and Nodes in .

[0016] Furthermore, the reverse reanalysis process from the batch perspective is: t+1 Search together in a sub-batch and pass the root node value to S t .

[0017] Furthermore, the set period is specifically a set fixed number of iterations.

[0018] Furthermore, the tree search traversal process includes:

[0019] is the root node Select action a root :

[0020]

[0021] Where a is the action associated with the child node, is with The corresponding action, represents the reward predicted by the dynamic model, if the Different actions, the simulation will continue to traverse according to the original MCTS setting. If the action is selected The simulation terminates immediately.

[0022] An electronic device, comprising:

[0023] a memory storing a computer program; and

[0024] A processor in communication with the memory executes a computer program stored in the memory to implement the Monte Carlo tree search method based on reverse reanalysis.

[0025] A computer storage medium stores a computer program, and when the computer program is executed by a processor, the Monte Carlo tree search method based on reverse reanalysis is implemented.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] The design of the present invention reanalyzes the training samples through the reanalysis technology of the reverse perspective, so as to obtain the value estimation of a specific subnode in advance. In order to further adapt to this design, the present invention adopts the method of periodically reanalyzing the entire buffer instead of frequently reanalyzing small batches of data. The synergy of these two designs can significantly reduce the search cost while ensuring or even improving the performance, simplifying the process of data collection and reanalysis, thereby saving time and not being limited by hardware resources, greatly improving the hardware execution efficiency.

[0028] The present invention adopts the re-analysis technology of reverse perspective to re-analyze the training samples to obtain the value estimation of the child nodes in advance, that is, firstly obtain the value of the root node by searching at the subsequent time, and then evaluate the value of the child node currently being searched according to the root node value, so as to realize a reverse information reuse from the back-to-front perspective. The reused information is the root node value obtained by searching, and only the simple node value needs to be reused. Compared with the traditional method of mechanically copying the information of the entire subtree, the reverse information reuse method proposed in the present invention is simpler and more efficient to operate, and can optimize the search effect while improving the search speed, so as to obtain a better target strategy.

[0029] The framework adopted by the present invention is to periodically reanalyze the entire experience pool. Compared with the method of reanalyzing small batches used in the past, the present invention does not need to run MCTS to reanalyze small batches for each training iteration. It only needs to sample small batches and perform gradient descent. MCTS is no longer called in the collection phase. All MCTS calls are concentrated in the reanalysis process, and during the full buffer reanalysis, it is no longer limited to the size of the small batch, and the batch size can be freely adjusted to take advantage of the large batch. The training process is thus decoupled from the reanalysis process, which not only reduces the total number of MCTS calls, but also utilizes the processing speed of large batches, that is, better utilizes the parallelization speed gain of large batches. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0031] Figure 2 Schematic diagram of the connection between MCTS and slot machine;

[0032] Figure 3 It is a schematic diagram of the reverse perspective reanalysis under the node and batch perspectives in the present invention;

[0033] Figure 4 Detailed description of the execution workflow and runtime cycle diagram of MuZero and ReZero (the present invention) in single and multiple worker cases in the embodiments. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example

[0036] like Figure 1 As shown, a Monte Carlo tree search method based on reverse reanalysis includes:

[0037] The initial state of the problem to be decided is used as the root node R0 of the Monte Carlo tree. Assuming that the corresponding action space has n actions, n child nodes of the root node R0 are formed, and each child node is the root node of a sub-search tree.

[0038] During the training phase, the entire buffer is reanalyzed periodically. Specifically, the training samples are reanalyzed using the reverse perspective reanalysis technique to obtain the value estimates of the child nodes in advance to assist the root node search.

[0039] According to the preset number of training iterations, the tree search traversal process is completed to determine the action corresponding to the decision-making plan.

[0040] To facilitate the understanding of the content of this scheme, the principle concept of this scheme is described as follows:

[0041] As a classic algorithm in the field of random slot machines, UCB1 has been used to select nodes in the search tree. Figure 2 As shown, each node in the tree can be considered as a multi-armed bandit.

[0042] Figure 2 The middle left figure shows the tree search from the perspective of the slot machine. When action A is selected, a reward r is returned. A ,in For the root node, the traversal, evaluation, and backpropagation processes that occur in the gray box can be approximated as sampling from a non-stationary distribution. Therefore, the root node can be viewed as a non-stationary slot machine. Figure 2 The middle right figure shows a one-armed bandit. Once the reward μ A It is known that μ A to evaluate arm A, thus eliminating the need to rely on a subsequent tree search process.

[0043] It is important to note that in the context of a multi-armed bandit, the reward for each arm is sampled from a fixed probability distribution. However, during a tree search, the value gained from selecting a node depends on the subsequent search process. The UCT algorithm models this process as sampling from a convergent non-stationary distribution. For ease of analysis, the root node is approximated as a random bandit, and each child node is considered an arm with a deterministic reward distribution. Based on this modeling, the public UCB1 variant formula used in AlphaZero can be analyzed:

[0044]

[0045] And the following proposition is obtained:

[0046] Proposition 1: The prior representation of arm i is P i Without loss of generality, assume μ *=μ1. Assuming that the reward distribution of each arm i is 1-subgaussian and P1>μ1, the regret R of selecting the arm using formula (1) is n satisfy:

[0047]

[0048] Assuming P1>μ1 is purely for formal reasons; in fact, as long as there is a t such that A similar upper bound on regret can always be obtained. This illustrates that selecting actions according to formula (1) can produce a sublinear growth in regret, indicating that the final access distribution will be concentrated on the optimal arm. This also confirms the rationality of selecting the action with the most visits. The design of the scheme of the present invention is inspired by the single-armed bandit model: if the expected reward of a particular arm is known a priori, there is no need to conduct exploratory trials on the arm, thereby reducing the upper bound on regret. To concretize this insight, formula (1) can be expanded and the following proposition can be derived:

[0049] Proposition 2 For a slot machine that satisfies the assumptions in Proposition 1, if the known expected value of the arm is used to replace the score in formula (1), the upper bound of the regret can be reduced. Specifically, if the known arm is the optimal arm and its value is μ1, then the regret R n satisfy:

[0050]

[0051] In other cases, if the value μ of the suboptimal arm l is known l , then unfortunately R n satisfy:

[0052]

[0053] Proposition 2 shows that, given the known expected reward of the suboptimal arm, the asymptotic upper bound of regret can be further reduced, making the visit distribution more concentrated on the optimal arm. Figure 2 As shown, when the expected μ A When is known, it is no longer necessary to evaluate arm A through real-time search results. Instead, it can be directly used A Replace the scores in the formula, thus saving time spent in traversal and backpropagation. This inspires using existing information to evaluate specific child nodes, thereby eliminating the exploration of the corresponding subtree and speeding up the search process.

[0054] Inspired by the above motivation, we hope to obtain the reward expectation of the child node in advance to evaluate it. However, the true expectation is always unknown. Therefore, this scheme uses the root node value obtained from MCTS as an approximate substitute. Specifically, in Figure 3 Two adjacent time steps in and When the search status When , the root node corresponds to , then the root node value obtained can be used to assist However, there is a time contradiction ( yes 's successor state), so this solution first needs to complete This is possible during the reanalysis process.

[0055] During the reanalysis process, since the trajectories have been collected in the replay buffer, a tree search can be performed from a reverse perspective, that is, searching the trajectories in reverse order. Figure 3 A batch of n+1 trajectories is shown, each of length k+1. represents the tth state in the lth trajectory. First, for all S k Search and then for all S k-1 Search, and so on.

[0056] This scheme samples n+1 trajectories of length k+1 to form a batch and searches in the reverse direction of the trajectory. From the node perspective, first search Then the root node value Pass to to evaluate the value of a child node, and is the corresponding search tree. From a batch perspective, all S 1 Search together in a sub-batch and pass the root node value to S 0 .

[0057] In the state After searching, the root node value is obtained When When searching, Assign the value of When traversing the tree, use the following formula for the root node Select action a root :

[0058]

[0059]

[0060] Among them, a refers to the action associated with the child node, is with The corresponding action, represents the reward predicted by the dynamic model. If Different actions, the simulation will continue to traverse according to the original MCTS settings. If the action is selected The simulation is terminated immediately. time, this enhanced search process is faster than the original version. It should be noted that the analysis of Proposition 2 assumes that the expected value is known. However, in the design of this scheme, the root node value is used as an approximate estimate. Still based on the random slot machine assumption, the following theorem analyzes the impact of the error between these two values:

[0061] Theorem 1 For the sample results of a bandit and one arm that meet the assumptions of Proposal 1, if the arm is selected using formula (5), the upper bound of regret can be reduced. Specifically, if the sampled arm is the optimal arm, its n-times sample mean is Regret R n satisfy:

[0062]

[0063] In other cases, if the n-times sample mean of the suboptimal arm l is Regret R n satisfy:

[0064]

[0065] The theorem shows that using the sample mean as a substitute for the expected value can also reduce the asymptotic upper bound of regret and maintain a sublinear growth rate. This can be understood as: although there is an error between the sample mean and the expected value, since this scheme uses the average value of n samples (the root node value comes from a tree search with n simulations), as n grows, this sample mean will get closer and closer to the expected value, thus preventing linear growth.

[0066] However, this design will bring a new problem in practice. Batch processing MCTS allows parallel model inference and data processing, thereby speeding up the average search speed. Figure 3 As shown in Figure 1, in order to perform the reanalysis from the reverse perspective, the batch needs to be split into 1 / k of its original size. This weakens the benefits of parallel search and makes the algorithm slower. Therefore, this scheme proposes a new framework that is more compatible with this design.

[0067] Specifically, in the collection phase, we have changed from using MCTS to select actions to directly sampling actions based on the output of the policy network. This change can be interpreted as another way to enhance exploration in the collection phase, similar to injecting noise into the root node in the MCTS algorithm. For the reanalysis phase, the design introduces a mechanism to periodically reanalyze the entire experience pool. Figure 4As shown, the entire experience pool is reanalyzed after a fixed number of training iterations. For each training iteration, there is no need to run MCTS to reanalyze the mini-batch, only to sample the mini-batch and perform gradient descent. Overall, this design provides two significant advantages: 1. Full-buffer reanalysis is similar to the fixed target network mechanism in DQN, maintaining a constant policy target for a certain number of training iterations. Reducing the frequency of policy target updates correspondingly reduces the number of MCTS calls. At the same time, this does not result in performance degradation. 2. Since MCTS is no longer called during the collection phase, all MCTS calls are concentrated in the reanalysis process. And during full-buffer reanalysis, it is no longer limited to the size of the mini-batch, allowing the batch size to be freely adjusted to take advantage of large batches.

[0068] Figure 4 In , the number inside the module indicates the number of iterations, and the number below the module indicates the time required for the module to execute. The model is updated n iterations between two collections. MuZero reanalyzes the mini-batch before each model update. ReZero (i.e., the solution of the present invention) is updated at a specific number of iterations (e.g. ), which not only reduces the total number of MCTS calls, but also fully and efficiently utilizes the processing speed of large batches.

[0069] To verify the effectiveness of this solution, this embodiment evaluates the performance of ReZero through experiments, focusing on the reduction of wall clock time required to achieve comparable performance levels. In order to make a fair comparison in terms of wall clock time, this embodiment not only uses the same computing resources each time it runs, but also maintains consistent hyperparameter settings between algorithms. Tables 1 and 2 show the acceleration effect of ReZero applied to MuZero and EfficientZero. The data shows that in most games, ReZero requires two to four times less wall clock time per 100k steps than the baseline method.

[0070] Table 1

[0071]

[0072] Table 2

[0073]

[0074] In summary, the reuse method adopted by this scheme is a reverse reuse from the back-to-front perspective. The value of the root node is first obtained by searching at the subsequent moment, and then the value of the child node currently being searched is evaluated based on the value of the root node. This is fundamentally different from the design of sequential reuse from front to back in the traditional method. The information reused by this scheme is the value of the root node obtained by the search, and only simple value values ​​need to be reused. Compared with the traditional method of copying the entire subtree, this reuse method is simpler and more efficient. The framework adopted by this scheme is to periodically reanalyze the entire experience pool. Compared with the method of reanalyzing mini-batches adopted by the traditional method, this design can decouple the training process from the reanalyze process, thereby better utilizing the parallelization speed gain of large batches.

[0075] Compared with SpeedyZero, the technical solution proposed by the present invention is simpler and easier to implement. The present invention does not require additional hardware resources, is not limited by hardware resources, has stronger versatility, and does not require the deployment of a distributed training system. It only needs to reverse the search order during reanalyze and save and reuse the root node value.

[0076] The method proposed by the present invention is orthogonal to the technical solutions proposed by Speedyzero and PTSAZero. In the re-analysis link of the existing method, the method proposed by the technical solution can be used to reuse information to improve the search efficiency.

[0077] The method of reusing subtrees from back to front is just a mechanical copy of information. The theoretical proof and experimental results in the technical solution show that the reuse method proposed by the present invention can improve the search speed while optimizing the search effect and obtain a better target strategy.

Claims

1. A Monte Carlo tree search method based on reverse reanalysis, characterized in that: include: The initial state of the problem to be decided is used as the root node R0 of the Monte Carlo tree. Assuming that the corresponding action space has n actions, n child nodes of the root node R0 are formed, and each child node is the root node of a sub-search tree. During the training phase, the entire buffer is reanalyzed periodically. Specifically, the training samples are reanalyzed using the reverse perspective reanalysis technique to obtain the value estimates of the child nodes in advance to assist the root node search. According to the preset number of training iterations, the tree search traversal process is completed to determine the action corresponding to the decision-making plan.

2. The Monte Carlo tree search method based on reverse reanalysis according to claim 1, characterized in that: The method of obtaining the value estimation of the child node in advance specifically involves first obtaining the value of the root node by searching at a subsequent time, and then evaluating the value of the child node currently being searched based on the value of the root node.

3. The Monte Carlo tree search method based on reverse reanalysis according to claim 1, characterized in that: The periodic reanalysis of the entire buffer is specifically to perform a reverse perspective tree search on the entire buffer where the trajectories are collected according to a set period, that is, to search the trajectories in reverse order, first sampling n+1 trajectories from the buffer to form a batch, wherein each trajectory has a length of k+1, and adopting represents the tth state in the lth trajectory, and then searches in the reverse direction of the trajectory: first search for all S k Search and then for all S k-1 to search for it.

4. The Monte Carlo tree search method based on reverse reanalysis according to claim 3, characterized in that: The reverse perspective reanalysis technology includes a reverse reanalysis process from a node perspective and a reverse reanalysis process from a batch perspective.

5. The Monte Carlo tree search method based on reverse reanalysis according to claim 4, characterized in that: The reverse reanalysis process from the node perspective is: first search Then the root node value Pass to To evaluate the value of a child node, They are the corresponding search trees and Nodes in .

6. The Monte Carlo tree search method based on reverse reanalysis according to claim 5, characterized in that: The reverse reanalysis process from the batch perspective is: t+1 Search together in a sub-batch and pass the root node value to S t .

7. The Monte Carlo tree search method based on reverse reanalysis according to claim 3, characterized in that: The set period is specifically a set fixed number of iterations.

8. The Monte Carlo tree search method based on reverse reanalysis according to claim 6, characterized in that: The tree search traversal process includes: is the root node Select action a root : Among them, a is the action associated with the child node, is with The corresponding action, represents the reward predicted by the dynamic model, if the Different actions, the simulation will continue to traverse according to the original MCTS setting. If the action is selected The simulation terminates immediately.

9. An electronic device, characterized in that: The electronic device comprises: a memory storing a computer program; and A processor communicatively connected to the memory executes a computer program stored in the memory to implement the Monte Carlo tree search method based on reverse reanalysis according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by the processor, the Monte Carlo tree search method based on reverse reanalysis according to any one of claims 1 to 8 is implemented.

Images