Monte Carlo tree searching method and device and computer equipment

Abstract

The invention relates to a Monte Carlo tree searching method and device and computer equipment. The method comprises the following steps: acquiring first state information of a root node to be searched; controlling at least one idle thread in the first thread pool, determining an extended child node based on the first state information, and generating a simulation task of the extended child node based on the second state information of the extended child node; at least one idle thread in the second thread pool is controlled, the simulation task is executed, a back propagation task is generated, and the back propagation task comprises a simulation reward value; at least one idle thread in the first thread pool is controlled to execute a back propagation task, second state information is updated according to the simulation reward value, and the at least one idle thread in the first thread pool and the at least one idle thread in the second thread pool are distributed to corresponding cores of the processor and executed by the processor in parallel; and repeatedly executing the steps until a preset search condition is met, and determining a search result based on the second state information.

Description

Technical Field

[0001] The present application relates to the field of computer science and technology, and in particular to a Monte Carlo tree search method, device and computer equipment. Background Art

[0002] The Monte Carlo Tree Search (MCTS) algorithm is a Monte Carlo sampling-based algorithm that is widely used in many technical fields such as robot path planning and intelligent navigation.

[0003] The Monte Carlo tree search algorithm includes four stages: selection, expansion, simulation, and back propagation. Generally speaking, the more iterations of the Monte Carlo tree search algorithm, the more nodes in the search tree, and the better the final search results. However, an increase in the number of iterations also means that more calculations are required, which will significantly increase the execution time. The Monte Carlo tree search algorithm in traditional technology is usually executed using a serial method. Even if different stages are assigned to different cores of the processor for execution, the subsequent stage still needs to wait for the previous stage to complete, and the processor execution efficiency is low. Summary of the invention

[0004] Based on this, it is necessary to provide a Monte Carlo tree search method, device and computer equipment that can improve the execution efficiency of the processor in response to the above technical problems.

[0005] In a first aspect, the present application provides a Monte Carlo tree search method. The method comprises:

[0006] Obtain first state information of the root node to be searched;

[0007] Controlling at least one idle thread in the first thread pool, determining an extended subnode based on the first state information, and generating a simulation task of the extended subnode based on the second state information of the extended subnode;

[0008] Controlling at least one idle thread in the second thread pool to execute the simulation task and generate a back-propagation task, wherein the back-propagation task includes a simulation reward value;

[0009] Controlling at least one idle thread in the first thread pool to execute the back-propagation task and update the second state information according to the simulation reward value, wherein at least one idle thread in the first thread pool and at least one idle thread in the second thread pool are respectively assigned to corresponding cores of a processor and executed in parallel by the processor;

[0010] The above steps are repeated until a preset search condition is met, and search results are determined based on the second state information.

[0011] In one embodiment, the second state information includes the number of simulations of the expanded child node and the current reward value; and executing the back propagation task and updating the second state information according to the simulated reward value includes:

[0012] After each back-propagation task is executed, the simulation times of the corresponding expanded child node is increased by 1 as the updated simulation times;

[0013] An updated current reward value is determined based on the sum of the current reward value and the simulation reward value and the updated number of simulations.

[0014] In one embodiment, updating the second state information according to the simulation reward value further includes:

[0015] After each back-propagation task is executed, a first atomic variable and a second atomic variable are obtained, wherein the first storage bit of the first atomic variable is used to store the number of simulations of the extended child node, the second storage bit of the first atomic variable is used to store the current accumulated reward value of the extended child node, the first storage bit of the second atomic variable is used to store the integer 1, and the second storage bit of the second atomic variable is used to store the simulated reward value;

[0016] Based on the received atomic variable accumulation command, the second atomic variable is accumulated to the first atomic variable to obtain an updated first atomic variable, the first storage bit of the updated first atomic variable is used to store the updated number of simulations, and the second storage bit of the updated first atomic variable is used to store the updated current reward value.

[0017] In one embodiment, after updating the second state information according to the simulation reward value, determining the expansion subnode based on the first state information includes:

[0018] Obtaining the updated number of simulations and the updated current reward value;

[0019] The expansion child node is determined based on the first state information, the updated number of simulations, and the updated current reward value.

[0020] In one embodiment, obtaining the updated number of simulations and the updated current reward value comprises:

[0021] Acquire a first atomic variable and a storage variable, wherein a first storage bit of the first atomic variable is used to store the updated number of simulations, and a second storage bit of the first atomic variable is used to store the updated current reward value;

[0022] Based on the received atomic variable read instruction, read first data of a first storage bit of the first atomic variable, and read second data of a second storage bit of the first atomic variable;

[0023] Respectively copying the first data and the second data to a first storage bit and a second storage bit of the storage variable;

[0024] The first storage bit and the second storage bit of the storage variable are read to determine the updated number of simulations and the updated current reward value.

[0025] In one embodiment, determining the search result based on the second state information includes:

[0026] Taking the updated expanded child node with the largest number of simulation times as the target search node;

[0027] The execution action from the root node to be searched to the target search node is used as the search result.

[0028] In one embodiment, repeatedly performing the above steps until a preset search condition is met includes:

[0029] After each back-propagation task is executed, the current number of iterations is increased by 1;

[0030] After the second state information is updated according to the simulation reward value, determining whether the current number of iterations is equal to a preset number of iterations;

[0031] If so, it is determined that the preset search condition is met.

[0032] In one embodiment, generating a simulation task of the extended subnode based on the second state information of the extended subnode includes:

[0033] Determining at least one optional execution action based on the second state information;

[0034] Among the at least one optional executed action, determining an unexecuted action;

[0035] A simulation node is determined based on the unexecuted action, and the simulation task is generated.

[0036] In a second aspect, the present application also provides a Monte Carlo tree search device. The device comprises:

[0037] A state information acquisition module, used to acquire first state information of a root node to be searched;

[0038] A task generation execution module, used for at least one idle thread in the first thread pool, determines an extended subnode based on the first state information, and generates a simulation task of the extended subnode based on the second state information of the extended subnode;

[0039] A back propagation task execution module, used for at least one idle thread in the second thread pool, executes the simulation task and generates a back propagation task, wherein the back propagation task includes a simulation reward value;

[0040] The task generation execution module is also used for at least one idle thread in the first thread pool to execute the back propagation task and update the second state information according to the simulation reward value;

[0041] The search result determination module is used to determine the search result based on the second state information after the preset search condition is met.

[0042] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the Monte Carlo tree search methods in the first aspect are implemented.

[0043] The Monte Carlo tree search method, apparatus and computer device described above obtain first state information of a root node to be searched; control at least one idle thread in a first thread pool to determine an expanded child node based on the first state information, and generate a simulation task of the expanded child node based on the second state information of the expanded child node; control at least one idle thread in a second thread pool to execute the simulation task and generate a back propagation task, wherein the back propagation task includes a simulation reward value; control at least one idle thread in the first thread pool to execute the back propagation task, and update the second state information according to the simulation reward value, wherein at least one idle thread in the first thread pool and at least one idle thread in the second thread pool are respectively assigned to corresponding cores of a processor and are executed in parallel by the processor; and the above steps are repeated until a preset search condition is met, and a search result is determined based on the second state information.

[0044] The Monte Carlo tree search method provided in the embodiment of the present application, through at least one idle thread in the first thread pool to execute the expansion child node, generate simulation tasks and back propagation tasks, through at least one idle thread in the second thread pool to execute the simulation task, and at least one idle thread in the first thread pool and at least one idle thread in the second thread pool are respectively assigned to the corresponding cores of the processor and executed in parallel by the processor, can effectively improve the execution efficiency of the processor, and significantly reduce the processor overhead in the selection, expansion, simulation and back propagation stages. On the other hand, the present application can also enhance the scalability of Monte Carlo tree search by decoupling the selection, expansion, simulation and back propagation stages.

[0045] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 It is a schematic diagram of the Monte Carlo tree search method in traditional technology;

[0048] Figure 2 This is a timing diagram for parallelization in traditional technology;

[0049] Figure 3 This is the timing diagram of root parallelization in traditional technology;

[0050] Figure 4 It is a timing diagram of tree parallelization in traditional technology;

[0051] Figure 5 It is the timing diagram of WU-UCT algorithm in traditional technology;

[0052] Figure 6 A diagram of an application environment of a Monte Carlo tree search method in one embodiment;

[0053] Figure 7 A schematic diagram of a flow chart of a Monte Carlo tree search method in one embodiment;

[0054] Figure 8 is a timing diagram of a Monte Carlo tree search method in one embodiment;

[0055] Fig. 9 A schematic diagram of a Monte Carlo tree search system in one embodiment;

[0056] Fig.10A schematic diagram of a process flow of a producer module executing a task in one embodiment;

[0057] Fig.11 A schematic diagram of a process for a consumer module to perform tasks in one embodiment;

[0058] Fig.12 is a structural block diagram of a Monte Carlo tree search device in an embodiment;

[0059] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The terms "module", "unit", etc. used below are a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in hardware, the implementation of software, or a combination of software and hardware is also possible and conceivable.

[0062] First, the Monte Carlo tree search method in traditional technology is further explained.

[0063] As a reinforcement learning algorithm, the goal of Monte Carlo tree search is to solve the optimal action under a given environmental state. The algorithm maintains a search tree to record known state information and execution action information. Generally speaking, the nodes of the search tree represent an environmental state, and the edges between the nodes represent the execution actions that cause the environmental state to transfer. The root node of the search tree is the environmental state to be solved. Figure 1 The figure shows a schematic diagram of the Monte Carlo tree search method, where S0 is the root node to be solved, and S1-S4 are extended child nodes. During the execution of the algorithm, the results of each simulation will be saved in the nodes of the search tree. The Monte Carlo tree search algorithm determines how to perform subsequent simulations based on the recorded node information and updates the node information of the search tree. This iteration is repeated until the set number of iterations is completed, and finally the Monte Carlo tree search algorithm gives the optimal action under the root node as the output search result.

[0064] Generally speaking, the more iterations of the Monte Carlo tree search algorithm, the more search tree nodes there are, and the better the final search performance, because this means that the algorithm explores a larger state space, the statistical data is more accurate, and the convergence effect is better. However, the increase in the number of iterations also means that more calculations are required, which will significantly increase the search time. Therefore, in order to obtain better performance within a limited time, some parallel search methods have been proposed in traditional technologies to improve computational efficiency.

[0065] The first parallel search method is leaf parallelization (LeafP), such as Figure 2 The following is a timing diagram of leaf parallelization. In this parallelization method, the overall process is basically similar to the serial Monte Carlo tree search algorithm. The biggest difference compared to the serial algorithm is that when entering the simulation phase, the leaf parallelization method starts from the same initial state and simulates in multiple Worker threads (Worker0, Worker1, Worker2, Worker3) at the same time, as shown in Figure 1. Figure 2 As shown in the figure. When all worker threads have completed the simulation, the reward values ​​obtained will be averaged, and the back propagation phase will be performed with the averaged reward values. The idea of ​​this method is very simple, and more simulations can be used to obtain more accurate reward values, thereby obtaining a more accurate evaluation of new nodes.

[0066] The second parallel search method is root parallelization (RootP), such as Figure 3 The following is a timing diagram of root parallelization. In this parallelization method, each Worker thread (Worker0, Worker1, Worker2, Worker3) maintains an independent search tree. The initial root nodes of these search trees are all states S0 to be solved. Each Worker thread executes a serial Monte Carlo tree search algorithm on its own search tree. The Worker threads do not affect each other. The timing is shown in Figure 3 As shown in the figure. Due to the randomness of the Monte Carlo tree search algorithm, the shapes of these search trees are different in the end. Therefore, when the final search is given, the nodes of these search trees need to be merged first, and the merged data is used for search. The idea of ​​this method is not complicated, and it is essentially equivalent to executing the serial Monte Carlo tree search algorithm multiple times at the same time.

[0067] The third parallel search method is tree parallelization (TreeP), such as Figure 4The following is a timing diagram of tree parallelization. Each iteration of the serial Monte Carlo tree search algorithm has a strict sequence. The next iteration can only start after the previous iteration completes backpropagation and updates the node information of the search tree. In the tree parallelization method, this sequence is broken. When a Worker thread (such as Worker0) starts to execute the first iteration, the second iteration is then executed by another Worker thread (any one of Worker1, Worker2, and Worker3). The timing is shown in Figure 1. Figure 4 As shown in the figure, the tree parallelization method only maintains a shared search tree. Each Worker thread accesses and modifies the shared search tree and performs complete iterations. That is, each Worker thread continuously executes the four stages of selection, expansion, simulation, and backpropagation based on the information in the shared search tree.

[0068] The fourth parallel search method is the WU-UCT algorithm, such as Figure 5 The following is a timing diagram of the WU-UCT algorithm. This parallelized algorithm divides the four stages of selection, expansion, simulation, and backpropagation into two categories: the first category includes the expansion and simulation stages. Both stages require interaction with the environment to calculate the state after the transfer, which takes up relatively more time and requires parallel processing; the second category includes the selection and backpropagation stages. These two stages only involve reading and writing the search tree, do not require interaction with the environment, and take up relatively less time. The WU-UCT algorithm framework consists of a Master and multiple Workers: the Master thread is responsible for executing the stages in the second category, updating the search tree in a single-threaded manner and then generating new expansion tasks and simulation tasks; multiple Workers (Worker0, Worker1) are executed in a multi-threaded manner. After each thread receives an expansion task or simulation task, it immediately executes the expansion stage or simulation stage and returns the result to the Master thread. The timing of the WU-UCT algorithm is shown in the figure below. Figure 5 shown.

[0069] In the leaf parallelization method, the end time of the simulation phase depends on the slowest Worker thread, which means that other Worker threads that end early will have idle time and the computing device utilization rate is low. In addition, leaf parallelization only simulates a single node multiple times, but it does not increase the growth rate of the search tree nodes. Compared with the serial Monte Carlo tree algorithm, each iteration expands a node, and the time of each iteration is not shortened.

[0070] In the root parallelization method, multiple worker threads essentially execute the serial Monte Carlo tree search algorithm simultaneously, and then merge the search trees of each worker thread. However, it also faces the same problem as the leaf parallelization method, and the growth rate of the search tree nodes has not increased. These worker threads will repeatedly simulate some nodes in the search tree. For example, when the random number seed is fixed, each worker thread will get the same result each time it simulates, thereby constructing the same search tree. These repeated explorations of the same nodes will not make the merged search tree have more nodes than the serial Monte Carlo tree search algorithm. Therefore, the root parallelization method has a limited effect on the improvement of node growth rate.

[0071] In the tree parallelization method, each worker thread will execute its own iteration process. N worker threads means that N iterations are executed at the same time, and each iteration expands a node. However, in fact, since the search tree in this method is shared by multiple threads, when they read and write the search tree at the same time, they will face the problem of thread synchronization. Mechanisms such as mutex locks are required to ensure the correctness of data reading and writing. In actual applications, the synchronization overhead brought by mutex locks limits the scalability of this method. When the number of threads N is large, the synchronization overhead of multi-threaded maintenance of the search tree is high, which has a performance bottleneck.

[0072] In the WU-UCT algorithm, in order to improve scalability, the system architecture uses a single Master thread to perform the selection and back-propagation phases, and assigns the computational tasks of the expansion and simulation phases to N Worker threads for parallel processing. Although the synchronization overhead of multiple threads reading and writing the search tree at the same time is avoided, there is still a performance bottleneck problem. Because the speed of maintaining the search tree by a single thread is limited, as the number of threads N increases, the time spent in the simulation and expansion phases will decrease, but the speed of the selection and back-propagation phases will not increase. Therefore, the performance upper limit of this method is completely determined by the speed of maintaining the search tree by a single thread, forming a performance bottleneck.

[0073] Therefore, the scalability of the parallelization of the Monte Carlo tree search algorithm in traditional technology is constrained by synchronization overhead and system architecture, which makes the search efficiency and processor execution efficiency of the parallelization method low.

[0074] Based on this, the embodiment of the present application provides a Monte Carlo tree search method, which can be applied to Figure 6In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains the first state information of the root node to be searched and sends it to the server 104. The server 104 controls at least one idle thread in the first thread pool, determines the extended child node based on the first state information, and generates a simulation task of the extended child node based on the second state information of the extended child node; controls at least one idle thread in the second thread pool, executes the simulation task and generates a back propagation task, and the back propagation task includes a simulation reward value; controls at least one idle thread in the first thread pool, executes the back propagation task, and updates the second state information according to the simulation reward value, and at least one idle thread in the first thread pool and at least one idle thread in the second thread pool are respectively assigned to the corresponding cores of the processor and executed in parallel by the processor. The server 104 repeats the above steps until the preset search condition is met, determines the search result based on the second state information, and sends it to the terminal 102. In other embodiments, the above steps may also be performed by the server 104 or the terminal 102 alone, and the present application does not limit this. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0075] In some embodiments, Figure 7 As shown, a Monte Carlo tree search method is provided, characterized in that the method comprises:

[0076] S201: Acquire first state information of a root node to be searched.

[0077] In an embodiment of the present application, the root node to be searched includes the root node of the search tree in the Monte Carlo tree search algorithm. The first state information may include first environmental state information and first executable action information of the root node to be searched. The root node to be searched may be connected to the extended child node through the first executable action information.

[0078] The Monte Carlo tree search method provided in the embodiment of the present application can be applied in multiple technical fields. In some embodiments, it can be applied in robot path planning search, the purpose of which is to determine the optimal path of the robot from the starting point to the end point in a complex environment. The first state information of the root node to be searched may include the environmental state information, posture information, executable action information, etc. of the robot at a certain position. The Monte Carlo tree search method provided in the embodiment of the present application can search for the optimal action to be performed by the robot in the next step, and the optimal path for the robot to reach the end point can be determined by repeatedly executing the method provided in the present application.

[0079] In some specific embodiments, the timing diagram of the Monte Carlo tree search method is as follows: Figure 8 As shown, the root node to be searched is Figure 8 As shown in S0.

[0080] S203: Control at least one idle thread in the first thread pool, determine an extended subnode based on the first state information, and generate a simulation task of the extended subnode based on the second state information of the extended subnode.

[0081] In an embodiment of the present application, the first thread pool may include a first preset number of threads. At least one idle thread in the first preset number of threads is controlled to determine the expansion subnode based on the first state information. Multiple threads in the first thread pool may execute the selection and expansion tasks in parallel and generate multiple simulation tasks at the same time.

[0082] In some specific embodiments, the expanded child nodes determined in the first iteration process may include nodes directly connected to the root node to be searched, such as Figure 8 In other embodiments, the expanded child nodes determined after multiple iterations may include nodes that are not directly connected to the root node to be searched, such as Figure 8 However, it should be noted that each iteration process starts from the root node S0 to be searched and executes the selection phase.

[0083] The second state information of the extended subnode may include the environmental state information, executable action information, simulation times and current reward value of the extended subnode. The environmental state information and executable action information of the extended subnode may be used to select the next extended subnode connected to the extended subnode, for example, according to Figure 8 The environment information and executable action information of the extended child node S2 shown can determine the extended child node S4. The current reward value of the extended child node can be used to select the extended child node in the next iteration. The number of simulations can be used to determine the optimal action of the root node to be searched after satisfying the preset search conditions.

[0084] In some embodiments, the generating a simulation task of the extended subnode based on the second state information of the extended subnode includes:

[0085] S301: Determine at least one optional execution action based on the second state information.

[0086] S303: Determine an unexecuted action among the at least one optional executed action.

[0087] S305: Determine a simulation node based on the unexecuted action, and generate the simulation task.

[0088] In an embodiment of the present application, the optional execution action of the extended subnode can be determined based on the environmental status information of the extended subnode. In at least one optional execution action, an unexecuted action is randomly determined. Based on the unexecuted action, a new simulation node is expanded, and all selected simulation nodes are recorded and packaged into a simulation task.

[0089] In the embodiment of the present application, a simulation task queue is also provided for storing the encapsulated simulation tasks. If the simulation task queue is full, the first thread pool needs to wait for other simulation tasks to be taken out before placing the simulation task into the simulation task queue.

[0090] S205: Control at least one idle thread in the second thread pool to execute the simulation task and generate a back-propagation task, wherein the back-propagation task includes a simulation reward value.

[0091] In an embodiment of the present application, the second thread pool may include a second preset number of threads. At least one idle thread in the second preset number of threads is controlled to execute the simulation task and generate a back propagation task. Multiple threads in the second thread pool may execute the simulation task in parallel and generate multiple back propagation tasks at the same time. The back propagation task includes a simulation reward value, which is generated based on the simulation task.

[0092] In the embodiment of the present application, at least one idle thread in the second thread pool is controlled to obtain a simulation task from the simulation task queue. Specifically, each idle thread can be assigned a simulation task, and multiple threads in the second thread pool can process multiple simulation tasks in parallel.

[0093] In the embodiment of the present application, a back propagation task queue is also provided for storing back propagation tasks. If the back propagation task queue is full, the second thread pool needs to wait for other back propagation tasks to be taken out before placing the back propagation task into the back propagation task queue.

[0094] S207: Control at least one idle thread in the first thread pool to execute the back propagation task and update the second state information according to the simulation reward value, wherein at least one idle thread in the first thread pool and at least one idle thread in the second thread pool are respectively assigned to corresponding cores of the processor and executed in parallel by the processor.

[0095] In an embodiment of the present application, at least one idle thread in the first thread pool is controlled to execute the back propagation task, and the second state information is updated according to the simulated reward value. Specifically, based on the simulated reward value and the current reward value of the expanded child node, an updated current reward value is determined to update the second state information.

[0096] In an embodiment of the present application, at least one idle thread in the first thread pool is controlled to obtain a back propagation task from the back propagation task queue. Specifically, each idle thread can be assigned a back propagation task, and multiple threads in the first thread pool can process multiple back propagation tasks in parallel, and generate simulation tasks for multiple extended sub-nodes.

[0097] In the embodiment of the present application, at least one idle thread in the first thread pool and at least one idle thread in the second thread pool are respectively assigned to corresponding cores of the processor and executed in parallel by the processor. It can be understood that processing different threads by different cores of the processor can effectively improve the execution efficiency of parallel tasks.

[0098] S209: Repeat the above steps until a preset search condition is met, and determine search results based on the second state information.

[0099] In an embodiment of the present application, the above steps S201-S207 are repeatedly executed until the preset search conditions are met, and the search results are determined based on the second state information. The preset search conditions may include a preset number of iterations, a preset algorithm execution time, etc. Specifically, if the number of executions of the back-propagation task reaches the preset number of iterations, it is determined that the preset search conditions are met; or after the time for the repeated execution of the above steps S201-S207 reaches the preset algorithm execution time, it is determined that the preset search conditions are met. It is understandable that if the above steps S201-S207 are not completed when the repeated execution time reaches the preset algorithm execution time, the execution continues until step S207 is completed. In some embodiments, determining the search results based on the second state information may include taking the extended sub-node with the largest number of simulations as the target search node.

[0100] In some embodiments, repeatedly performing the above steps until a preset search condition is met includes:

[0101] S401: After each back-propagation task is executed, the current number of iterations is increased by 1.

[0102] S403: After the second state information is updated according to the simulation reward value, determining whether the current number of iterations is equal to a preset number of iterations.

[0103] S405: If yes, determine that the preset search condition is met.

[0104] In the embodiment of the present application, each time a thread of the first thread pool executes a back propagation task, the number of completed current iterations is increased by 1. After the second state information is updated according to the simulation reward value, it is determined whether the current number of iterations is equal to the preset number of iterations. If the current number of iterations is equal to the preset number of iterations, it means that the preset search condition is met, and the steps S201-S207 are stopped and the search result is determined.

[0105] The following is a specific example of the present application. Figure 8 Take as an example to illustrate the four stages of the Monte Carlo tree search method.

[0106] (1) Selection phase: Starting from the root node S0 of the search tree, determine whether the current node meets the scalability condition, that is, determine whether there is an unexecuted action under the current node. If the current node is scalable, enter the expansion phase, otherwise recursively select the expansion child node according to the selection strategy until the selected expansion child node meets the scalability condition. The selection strategy can include the UCB1 algorithm, as shown in formula (1):

[0107]

[0108] In formula (1), s represents the current node, C(s) represents the set of expanded child nodes that have been explored in the subordinate nodes of the current node, and N s' Indicates the number of simulations to expand the child node s', N s Indicates the number of simulations of the current node, V s' represents the average reward value of child node s', and β is a constant greater than 0.

[0109] (2) Expansion phase: If the current node s has met the conditions for expansion, an action a is randomly selected from all unexecuted actions under the current node, and the state transition function of the environment is called to calculate the new environment state information. Based on this state, a new node s is determined. new , added to the subordinate of the current node s, becoming the extended child node of the current node s.

[0110] (3) Simulation phase: The simulation will start from the new node s new Start with random sampling to get a complete trajectory. Specifically, the simulation stage is based on node s newAs a starting point, randomly select one of all executable actions under the node to execute, call the state transfer function of the environment to calculate the new environment state, and continue to execute randomly selected actions on the new environment state until entering the terminal state.

[0111] (4) Backward propagation phase: When the simulation enters the termination state, a reward value r can be calculated. In the backward propagation phase, the reward value information r needs to be updated to the search tree. Specifically, for each node s selected in the selection phase of this iteration i , update the simulation times according to formula (2) and average reward

[0112]

[0113] After setting the preset number of iterations, the above four stages are repeated until the preset number of iterations is reached. Finally, the optimal action of the state to be solved S0 (i.e. the root node of the search tree) is output. The number of simulations is usually used as the criterion, and the execution action corresponding to the expanded subnode with the most simulations is used as the optimal action searched by the Monte Carlo tree search algorithm.

[0114] The following is a specific embodiment of the present application to illustrate the application of the Monte Carlo tree search method provided by the present application in a robot path planning scenario.

[0115] Set up a 3×3 map. The robot's initial position is at the upper left corner coordinate (0, 0). The goal is to reach the end point at the lower right corner coordinate (2, 2). Each step forward can only go right or down (i.e., the horizontal coordinate +1 or the vertical coordinate +1). There is also a trap in the grid at the coordinate (1, 0). When the robot enters the terminal state, there may be two situations: reaching the end point to get a reward value of 1; walking into the trap to get a reward value of -100. The coordinate distribution of the map can be shown in Table 1:

[0116] Table 1

[0117] Coordinates (0,0) (starting point) Coordinate (0,1) Coordinate (0,2) Coordinates (1,0) (trap) Coordinate (1,1) Coordinates (1,2) Coordinates (2,0) Coordinates (2,1) Coordinates (2,2) (end point)

[0118] If the target to be searched is: what is the best action when the robot is at the starting point at the coordinate (0, 0). Then the state corresponding to the root node S0 to be searched is the actual position coordinate (0, 0) of the robot. The executable action information under each node is go right, go down (if it reaches the end or trap, it enters the termination state and no action can be executed; when it is at the border of the map, the action that will cause the robot to cross the border will be excluded from the executable action information set).

[0119] Next, the iterative steps of the Monte Carlo tree search method are started.

[0120] First time: Start the selection phase from S0. Since there is no expanded child node at this time, randomly select one of the executable actions (walk right, walk down) to execute. If you choose to walk right, then determine to expand the child node S1, and the state of S1 is (1, 0). Start the random simulation from S1, that is, starting from (1, 0), randomly choose to walk right or down until entering the terminal state (entering the trap or reaching the end). Since it is already in the trap position at this time, the current reward value of the random simulation is -100, and backpropagation is updated. After the update, the cumulative reward of S1 is -100, and the number of simulations is 1.

[0121] Second time: Start the selection phase from S0. At this time, there is only S1 child node, but it has not tried to go right in the S0 state, which meets the extensibility condition. Therefore, it chooses to go right and determines to expand child node S2. The state of S2 is (0, 1). Then start random simulation from S2. Since it is certain that you can reach the end point from S2 in this scenario, the reward value of this simulation is 1. The current reward value of S2 after update is 1, and the number of simulations is 1.

[0122] The third time: The selection phase is executed from S0. At this time, the S0 node has two extended child nodes, S1 and S2. According to formula (1), S2 will be selected in the selection phase. The selection phase is continued for S2. S2 meets the scalability condition and is therefore expanded. Assuming that the selection goes down, S3 (1, 1) is expanded and the reward value is 1. Backpropagation updates S3 (the current reward value after update is 1, and the number of simulations after update is 1) and S2 (the current reward value after update is 2, and the number of simulations after update is 2).

[0123] Then, the iteration continues, repeating selection, expansion, simulation, and back propagation until the preset search termination condition is met. Assuming that the search is stopped after 3 iterations, then among the expanded child nodes of the root node S0, S2 has the most simulation times, and the corresponding action is to go right, so go right is used as the search result.

[0124] The Monte Carlo tree search method provided in the embodiment of the present application, through at least one idle thread in the first thread pool to execute the expansion child node, generate simulation tasks and back propagation tasks, through at least one idle thread in the second thread pool to execute the simulation task, and at least one idle thread in the first thread pool and at least one idle thread in the second thread pool are respectively assigned to the corresponding cores of the processor and executed in parallel by the processor, can effectively improve the execution efficiency of the processor, and significantly reduce the processor overhead in the selection, expansion, simulation and back propagation stages. On the other hand, the present application can also enhance the scalability of Monte Carlo tree search by decoupling the selection, expansion, simulation and back propagation stages.

[0125] The following describes a method for updating the second status information through an embodiment of the present application.

[0126] In some embodiments, the second state information includes the number of simulations of the expanded child node and the current reward value; and executing the back-propagation task and updating the second state information according to the simulated reward value includes:

[0127] S501: After each back-propagation task is executed, the number of simulations of the corresponding extended child node is increased by 1 as the updated number of simulations.

[0128] S503: Determine an updated current reward value based on the sum of the current reward value and the simulation reward value, and the updated number of simulations.

[0129] In the embodiment of the present application, the second state information includes the simulation times N of the extended sub-node. i And the current reward value W i , the current reward value W i It can be a cumulative reward value. After each back-propagation task is executed, the simulation times of the corresponding expanded child node is increased by 1 as the updated simulation times. For example, Figure 8 As shown, if the simulation nodes determined in the simulation task include S2, S4, and S6, the simulation times of nodes S2, S4, and S6 are increased by 1 respectively. i and the sum of the simulation reward value r, and the updated number of simulations N i , determine the updated current reward value V i The updated current reward value V i It can be the current average reward value.

[0130] Specifically, the updated simulation number N can be determined according to formula (3): i And the updated current reward value V i :

[0131]

[0132] In formula (3), N i +1 is assigned to N i Get the updated simulation number, W i +r assigns value to W i Get the sum of the current reward value and the simulated reward value, and then add the sum of the current reward value and the simulated reward value W i and the updated number of simulations N i Divide to determine the updated current reward value V i .

[0133] In some embodiments, after updating the second state information according to the simulation reward value, determining the expansion subnode based on the first state information includes:

[0134] S601: Obtain the updated number of simulations and the updated current reward value.

[0135] S603: Determine the expansion child node based on the first state information, the updated number of simulations, and the updated current reward value.

[0136] In the embodiment of the present application, after obtaining the updated number of simulations and the updated current reward value, the extended child node can be determined based on the first state information, the updated number of simulations, and the updated current reward value according to formula (1) in the above embodiment, which will not be repeated here.

[0137] In some embodiments, determining search results based on the second state information includes:

[0138] S701: Taking the updated expanded child node with the largest number of simulation times as the target search node.

[0139] S703: The execution action from the root node to be searched to the target search node is used as the search result.

[0140] In the embodiment of the present application, the expanded child node with the largest number of simulation times Ni after the update is used as the target search node. Figure 8 For example, if the simulation times of S1, S2, and S3 are 500, 1000, and 800 respectively, then the simulation times of the expanded child node S2 is the largest and it is used as the target search node. It can be understood that the simulation times of the upper node in the search tree are greater than the simulation times of the lower node, such as Figure 8 As shown in , the simulation times of node S2 must be greater than that of node S4, and the simulation times of node S4 must be greater than that of node S6.

[0141] The execution action from the root node to be searched to the target search node is used as the search result. For example, if the target search node is S2, the execution action from S0 to S2 is used as the search result.

[0142] In the case of multi-threaded maintenance of the search tree in traditional technology, when operating on the same node, there are three possible data contention situations: (1) two or more threads read the data of the same node at the same time; (2) two or more threads modify the data of the same node at the same time; (3) a thread is reading the data of a node, and another thread is modifying the data of the same node at the same time.

[0143] For the first possibility, simultaneous read operations will not affect the data itself and are allowed. For the second possibility, if two threads try to write the same data at the same time, for example, both threads need to write to node S i The number of simulations N i If the order or atomicity of the operations cannot be strictly guaranteed, some of the addition operations will be lost, affecting the correctness of the data. For the third possibility, if one thread reads the data of a node and another thread modifies the data of the node at the same time, for example, thread A reads node S first. i The number of simulations N i Data, thread B also modified node S i The reward value W i Data, and then thread A reads the modified W i' data, which will result in thread A finally getting data N i With W i' , rather than N i With W i In fact, N i With W i The data should be matched, so data inconsistency may occur.

[0144] The traditional way to solve this problem is to use a mutex to protect it and ensure that only one thread is operating a single node at the same time. However, the synchronization overhead caused by the mutex limits scalability. When the number of threads N is large, the synchronization overhead in the multi-threaded maintenance of the search tree cannot be ignored, resulting in a performance bottleneck.

[0145] Based on this, the present application provides a method for reading and writing atomic variables to solve the above technical problems.

[0146] In some embodiments, updating the second state information according to the simulation reward value further includes:

[0147] S801: After each back-propagation task is executed, a first atomic variable and a second atomic variable are obtained, wherein the first storage bit of the first atomic variable is used to store the number of simulations of the extended child node, the second storage bit of the first atomic variable is used to store the current accumulated reward value of the extended child node, the first storage bit of the second atomic variable is used to store the integer 1, and the second storage bit of the second atomic variable is used to store the simulated reward value.

[0148] S803: Based on the received atomic variable accumulation command, the second atomic variable is accumulated to the first atomic variable to obtain an updated first atomic variable, wherein the first storage bit of the updated first atomic variable is used to store the updated number of simulations, and the second storage bit of the updated first atomic variable is used to store the updated current reward value.

[0149] In the embodiment of the present application, after each back-propagation task is executed, the first atomic variable and the second atomic variable are obtained. The first storage bit of the first atomic variable W_N is used to store the simulation times N of the extended child node. i , the second storage bit is used to store the current cumulative reward value W of the extended child node i Specifically, if the node has been simulated multiple times, the current cumulative reward value is the sum of the current reward values ​​of the multiple simulations. The first storage bit of the second atomic variable W_N' is used to store the integer 1, and the second storage bit is used to store the simulation reward value r.

[0150] In some specific embodiments, the first atomic variable W_N may be applied synchronously when the search tree node is created. The first atomic variable and the second atomic variable may include 64-bit integer variables.

[0151] In some embodiments, the first storage bit may include the high bit of the first atomic variable, and the second storage bit may include the low bit of the first atomic variable. In other embodiments, the first storage bit may include the low bit of the first atomic variable, and the second storage bit may include the high bit of the first atomic variable. However, it should be noted that if the high bit of the first atomic variable is used to store the number of simulations of the extended child node, the high bit of the second atomic variable should also store the integer 1 accordingly. That is, the data stored by different atomic variables should match each other.

[0152] Based on the received atomic variable accumulation command W_N.fetch_add(W_N'), the second atomic variable is accumulated to the first atomic variable to obtain the updated first atomic variable, and the simulation number N is completed at the same time. i and the current reward value W i Updates.

[0153] In some embodiments, obtaining the updated number of simulations and the updated current reward value comprises:

[0154] S901: Acquire a first atomic variable and a storage variable, wherein a first storage bit of the first atomic variable is used to store the updated number of simulations, and a second storage bit of the first atomic variable is used to store the updated current reward value.

[0155] S903: Based on the received atomic variable read instruction, read the first data of the first storage bit of the first atomic variable, and read the second data of the second storage bit of the first atomic variable.

[0156] S905: Copy the first data and the second data to the first storage bit and the second storage bit of the storage variable respectively.

[0157] S907: Read the first storage bit and the second storage bit of the storage variable to determine the updated number of simulations and the updated current reward value.

[0158] In the embodiment of the present application, when the first thread pool accesses each node, it first applies for a local 64-bit integer variable W_N" as a temporary storage space, that is, a storage variable. The storage variable may be of atomic type or not. Using the atomic read operation of the atomic variable, the data of the first atomic variable W_N of the extended child node is read, and the value of W_N is copied to the local W_N" variable through the received W_N"=W_N.load() command, which is equivalent to obtaining the updated number of simulations Ni and the updated current reward value Wi at the same time. Then, by reading from the lower 32 bits and the upper 32 bits of W_N" respectively, the updated number of simulations and the updated current reward value can be determined respectively.

[0159] The following is a specific example to illustrate the execution process of the above atomic variables. Taking C++ as an example, the read and write operations of the lower 32 bits and the upper 32 bits of the atomic variable can be completed by bit operations, and the atomic variable can be completed by the std::atomic library.<long long> An atomic 64-bit integer variable can be defined, and atomic operations such as fetch_add and load are provided. Among them, fetch_add implements the modification operation of the atomic variable, and load implements the reading operation of the atomic variable, ensuring the simultaneous modification or reading of the node data and ensuring the consistency of the data. It should be noted that the add and load operations do not use the mutex mechanism in the underlying implementation, but are implemented based on the underlying hardware. Compared with the mutex method in traditional technology, atomic operations have smaller overhead and higher efficiency.

[0160] In the traditional parallelization method of the Monte Carlo tree search algorithm, in terms of maintaining the search tree, it can be divided into two forms: multi-threaded maintenance and single-threaded maintenance. The former is affected by the overhead of the mutex lock, and it is difficult to achieve a high degree of parallelism; in the single-threaded maintenance mode of the latter, the search efficiency becomes a performance bottleneck under high parallelism. The read-write method (lock-free) of atomic variables for multi-threaded maintenance of the search tree in this application can greatly alleviate the synchronization overhead problem caused by the mutex lock. Through a new data storage method, the simulation times and current reward values ​​of the search tree nodes are reintegrated into a variable and recorded using a single atomic variable. With the help of atomic operations to realize the reading and modification of data, the overhead of the mutex lock is avoided, and the parallel efficiency is improved, thereby allowing the Monte Carlo tree search method of this application to have a higher degree of parallelism and to ensure the consistency of the read data.

[0161] The Monte Carlo tree search system of the present application is described below through specific embodiments. Fig. 9 As shown, the system includes a producer module, a consumer module, and a task queue module. The producer module includes a first thread pool with P Worker threads (Worker1, Worker2...WorkerP) and a shared search tree shared by these threads. The main task of the producer module is to receive the back-propagation tasks to be completed, update the simulation times and current reward values ​​of the search tree nodes, and generate new simulation tasks according to the updated search tree. These tasks are executed by P Worker threads, and simulation tasks to be executed are continuously generated by executing the selection, expansion and back-propagation stages of the shared search tree in parallel. These simulation tasks wait for the consumer module to execute.

[0162] The consumer module contains a second thread pool with C worker threads (Worker1, Worker2, ... WorkerC). The consumer module is responsible for completing the simulation tasks to be executed. The C worker threads execute in parallel, each thread receives a simulation task to be completed, performs the calculation of the simulation stage, returns the simulation results after completion, and generates the back propagation tasks to be completed. These tasks wait for the producer module to execute.

[0163] The task queue module contains the simulation task queue and the back propagation task queue. The simulation task queue is used for message transmission from the producer module to the consumer module, and stores the simulation tasks to be executed sent by the producer module, which are retrieved and executed by the consumer module. The back propagation task queue is used for message transmission from the consumer module to the producer module, and stores the back propagation tasks to be executed sent by the consumer module, which are retrieved and executed by the producer module.

[0164] The specific values ​​of P and C can be specified by the user to adapt to the specific computing requirements in different application scenarios and realize customized parallel configuration. And by adjusting the ratio of P and C, the rate of incoming tasks and the rate of outgoing tasks in the task queue can be effectively balanced, thereby improving the execution efficiency of the system.

[0165] The following is a schematic diagram of the process of the producer module and the consumer module executing tasks through a specific embodiment.

[0166] like Fig.10 The figure shows a flowchart of the producer module executing tasks. The producer module determines whether the reverse propagation task queue is empty. If not, at least one thread in the first thread pool executes the reverse propagation phase and updates the number of completed iterations; if so, it determines whether the number of completed iterations is greater than the preset value (preset number of iterations). If so, the task execution process ends; if not, the selection phase is executed, and the expansion phase is executed to generate simulation tasks. It is determined whether the simulation task queue is full. If so, it is repeatedly determined whether the simulation task queue is full; if not, the simulation task is placed in the simulation task queue, and the above-mentioned step of determining whether the reverse propagation queue is empty is repeated.

[0167] like Fig.11 The figure shows a flowchart of the consumer module executing tasks. The consumer module determines whether the simulation task queue is empty. If so, it repeats the judgment; if not, it takes out the simulation task and records the newly expanded extended subnode and all selected nodes. Starting from the extended subnode, the simulation phase is executed, and the reward value is calculated to generate the back propagation task. Determine whether the simulation task queue is full. If so, it repeatedly determines whether the simulation task queue is full; if not, it puts the simulation task into the simulation task queue and determines whether the number of completed iterations is greater than the preset value (preset number of iterations). If so, the task execution ends; if not, it repeats the step of determining whether the above simulation task queue is empty.

[0168] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0169] Based on the same inventive concept, the embodiment of the present application also provides a Monte Carlo tree search device 1100 for implementing the Monte Carlo tree search method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more Monte Carlo tree search device 1100 embodiments provided below can refer to the limitations of the Monte Carlo tree search method above, and will not be repeated here.

[0170] In one embodiment, Fig.12 As shown, a Monte Carlo tree search device 1100 is provided, comprising:

[0171] The state information acquisition module 1101 is used to acquire the first state information of the root node to be searched;

[0172] A task generation execution module 1102 is used for at least one idle thread in the first thread pool to determine an extended child node based on the first state information, and to generate a simulation task of the extended child node based on the second state information of the extended child node;

[0173] A back propagation task execution module 1103, used for at least one idle thread in the second thread pool, to execute the simulation task and generate a back propagation task, wherein the back propagation task includes a simulation reward value;

[0174] The task generation execution module 1102 is also used for at least one idle thread in the first thread pool to execute the back propagation task and update the second state information according to the simulation reward value;

[0175] The search result determination module 1104 is used to determine the search result based on the second state information after the preset search condition is met.

[0176] Each module in the Monte Carlo tree search device 1100 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0177] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the Monte Carlo tree search method described in any of the above embodiments. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13 As shown. Those skilled in the art can understand that Fig.13The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0178] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the Monte Carlo tree search method described in any of the above embodiments are implemented.

[0179] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with general skills in the technical field to which this application belongs. "One", "one", "a kind of", "the", "these" and similar words in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions. "Multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of objects.

[0180] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A Monte Carlo tree search method, characterized in that: The method comprises: Obtain first state information of the root node to be searched; Controlling at least one idle thread in the first thread pool, determining an extended child node based on the first state information, and generating a simulation task of the extended child node based on the second state information of the extended child node; Controlling at least one idle thread in the second thread pool to execute the simulation task and generate a back-propagation task, wherein the back-propagation task includes a simulation reward value; Controlling at least one idle thread in the first thread pool to execute the back-propagation task and update the second state information according to the simulation reward value, wherein the at least one idle thread in the first thread pool and the at least one idle thread in the second thread pool are respectively assigned to corresponding cores of a processor and are executed in parallel by the processor; The above steps are repeated until a preset search condition is met, and search results are determined based on the second state information.

2. The method according to claim 1, characterized in that The second state information includes the simulation times and the current reward value of the expansion subnode; The performing of the back propagation task and updating the second state information according to the simulated reward value includes: After each back-propagation task is executed, the simulation times of the corresponding expanded child node is increased by 1 as the updated simulation times; An updated current reward value is determined based on the sum of the current reward value and the simulation reward value and the updated number of simulations.

3. The method according to claim 2, characterized in that The updating of the second state information according to the simulation reward value further includes: After each back-propagation task is executed, a first atomic variable and a second atomic variable are obtained, wherein the first storage bit of the first atomic variable is used to store the number of simulations of the extended child node, the second storage bit of the first atomic variable is used to store the current accumulated reward value of the extended child node, the first storage bit of the second atomic variable is used to store the integer 1, and the second storage bit of the second atomic variable is used to store the simulated reward value; Based on the received atomic variable accumulation command, the second atomic variable is accumulated to the first atomic variable to obtain an updated first atomic variable, the first storage bit of the updated first atomic variable is used to store the updated number of simulations, and the second storage bit of the updated first atomic variable is used to store the updated current reward value.

4. The method according to claim 2, characterized in that: After updating the second state information according to the simulation reward value, determining the expansion subnode based on the first state information includes: Obtaining the updated number of simulations and the updated current reward value; The expansion child node is determined based on the first state information, the updated number of simulations, and the updated current reward value.

5. The method according to claim 4, characterized in that The obtaining of the updated number of simulations and the updated current reward value comprises: Obtaining a first atomic variable and a storage variable, wherein a first storage bit of the first atomic variable is used to store the updated number of simulations, and a second storage bit of the first atomic variable is used to store the updated current reward value; Based on the received atomic variable read instruction, read first data of a first storage bit of the first atomic variable, and read second data of a second storage bit of the first atomic variable; Respectively copying the first data and the second data to a first storage bit and a second storage bit of the storage variable; The first storage bit and the second storage bit of the storage variable are read to determine the updated number of simulations and the updated current reward value.

6. The method according to claim 2, characterized in that Determining the search result based on the second state information includes: Taking the updated expanded child node with the largest number of simulation times as the target search node; The execution action from the root node to be searched to the target search node is used as the search result.

7. The method according to claim 1, characterized in that Repeating the above steps until the preset search condition is met includes: After each back-propagation task is executed, the current number of iterations is increased by 1; After the second state information is updated according to the simulation reward value, determining whether the current number of iterations is equal to a preset number of iterations; If so, it is determined that the preset search condition is met.

8. The method according to claim 1, characterized in that The step of generating a simulation task of the extended subnode based on the second state information of the extended subnode includes: Determining at least one optional execution action based on the second state information; Among the at least one optional executed action, determining an unexecuted action; A simulation node is determined based on the unexecuted action, and the simulation task is generated.

9. A Monte Carlo tree search device, characterized in that: The device comprises: A state information acquisition module, used to acquire first state information of a root node to be searched; A task generation execution module, used for at least one idle thread in the first thread pool, determines an extended subnode based on the first state information, and generates a simulation task of the extended subnode based on the second state information of the extended subnode; A back propagation task execution module, used for at least one idle thread in the second thread pool, executes the simulation task and generates a back propagation task, wherein the back propagation task includes a simulation reward value; The task generation execution module is also used for at least one idle thread in the first thread pool to execute the back propagation task and update the second state information according to the simulation reward value; The search result determination module is used to determine the search result based on the second state information after the preset search condition is met.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images