Unmanned cluster reconnaissance and rescue task allocation method based on monte carlo tree search

Abstract

This invention relates to a method for allocating reconnaissance and rescue missions in unmanned swarms based on Monte Carlo tree search, belonging to the field of mission planning technology. This invention formulates the mission allocation problem in reconnaissance and rescue as a sequential decision-making process, fully utilizing heuristic information from the problem, designing a state space and action space, balancing action completeness and the number of actions, thus saving computational resources while ensuring allocation effectiveness. Furthermore, an adaptive simulation number selection mechanism is proposed, which automatically selects the number of simulations based on the tasks to be assigned and the number of UAVs, avoiding a large number of repetitive simulations, thereby reducing the computational load of the algorithm and further improving the computational efficiency of mission allocation. The method of this invention can also enhance the effectiveness of mission allocation in unmanned systems and improve the mission efficiency of unmanned systems.

Description

Technical Field

[0001] This invention relates to a method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search, belonging to the field of mission planning technology. It can be used by unmanned aerial vehicles (UAVs) to quickly conduct comprehensive searches of disaster-stricken areas, locate people awaiting rescue, and deliver necessary relief supplies to key locations, thereby improving the success rate of rescue missions. Background Technology

[0002] With the advancement of unmanned intelligent technology, unmanned swarms have demonstrated enormous application potential in various fields. Unmanned swarms offer advantages such as low cost and operational flexibility, making them widely applicable to tasks like reconnaissance and surveillance, such as power line inspection, road monitoring, and personnel search and rescue. However, the increasing number of drones and the growing number of missions place higher demands on the task allocation and scheduling of drone swarms. Optimizing the selection and execution sequence of multiple drones and multiple targets to maximize the mission efficiency of unmanned swarms within limited resources is currently a hot topic in unmanned aerial vehicle (UAV) task allocation research.

[0003] The task allocation problem in unmanned swarms is typically formulated as a classic combinatorial optimization problem, with solutions offered by exact and heuristic algorithms. Exact algorithms rely on mathematical optimization to obtain precise solutions. While ensuring optimality, they are computationally expensive and generally unsuitable for large-scale problems or those involving complex constraints. Heuristic algorithms leverage domain knowledge and randomness to accelerate the solution process. Although they typically do not guarantee a globally optimal solution, they effectively generate high-quality solutions, making them particularly suitable for large-scale and complex problems. Therefore, they are widely used in real-world scenarios.

[0004] Common heuristic algorithms include genetic algorithms, ant colony optimization, and auction algorithms. However, they typically require careful parameter tuning (e.g., mutation rate in genetic algorithms or population size in particle swarm optimization) or extensive knowledge of the specific problem to achieve satisfactory performance. In contrast, Monte Carlo Tree Search (MCTS), as a heuristic search algorithm, stands out because it minimizes its dependence on knowledge of the specific problem and parameter tuning, making it widely applicable to decision problems under uncertainty. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a Monte Carlo tree search-based method for allocating reconnaissance and rescue missions in unmanned swarms. The method formulates the mission allocation problem in reconnaissance and rescue as a sequential decision-making process, fully utilizing heuristic information and designing state and action spaces to balance action completeness and the number of actions, thus conserving computational resources while ensuring allocation effectiveness. Furthermore, an adaptive simulation quantity selection mechanism is proposed to further improve computational efficiency. This method can also enhance the effectiveness of mission allocation in unmanned systems and improve their mission efficiency.

[0006] The technical solution of this invention is: a method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search, comprising the following steps:

[0007] Step 1: Mathematical modeling of the task allocation problem: Establish the set of UAVs capable of performing tasks and the set of tasks to be performed, and clarify the task objectives and task constraints;

[0008] Step 2: Design system actions and states, transforming the task allocation problem into a sequential decision problem, that is, the complete task allocation is achieved by executing a series of system actions;

[0009] Step 3: Perform Monte Carlo tree search to determine the action to be performed: Use an adaptive simulation number selection mechanism to determine the number of searches, and repeatedly execute the four steps of selection, expansion, simulation and backtracking until the number of searches reaches a given upper limit. Based on the constructed search tree, select the action corresponding to the optimal child node under the root node as the action to be performed.

[0010] Step 4: Repeat step 3 until all tasks have been assigned, and finally output the task assignment results.

[0011] Furthermore, in step (1), assuming the drones depart from and return from the same airport, each drone can perform multiple tasks, but each task is assigned to only one drone, and there are N drones capable of performing tasks. The set of drones is represented as... , The drone's serial number represents the set of tasks to be executed in the task allocation problem. , Represents the task sequence number; among which, Represents task i. Tasks representing departures from or return to the airport; the connectivity and distance between any two tasks are represented by a symmetric matrix. It means that, among them Indicates task and The distance between them Represents task i. Represents task j; Indicates from arrive The route is prohibited; each drone has different capabilities and attributes, the first... The flight time, flight speed, reconnaissance capability, and rescue capability of the drone are respectively expressed as: ;Task The execution time, reward, search demand, and rescue demand are respectively expressed as: .

[0012] Furthermore, taking the maximum task completion time of all drones and minimizing the total reward of all unassigned tasks as the task objective, the objective function is expressed as follows:

[0013]

[0014] Among them, γ = {γ0, γ1, γ2,...,γ N} represents a set of task sequences assigned to each drone, γ0 represents the set of all unassigned tasks, and γ i =(τ0,τ1,τ2,...,τ M ) is a drone U i The task allocation sequence, where τ represents the task index, τ1, τ2, ..., τ M Represents tasks numbered 1, 2...M.

[0015] Furthermore, the flight time and mission execution time of each drone must be less than its maximum flight endurance, i.e.

[0016]

[0017] Furthermore, the search and rescue requirements of the mission cannot exceed the capabilities of the drone, that is...

[0018]

[0019] Furthermore, in step (2), the system action is designed to select the j-th task and assign it to the i-th drone, where j represents the sequence number of the task among all tasks and i represents the sequence number of the drone among all drones. At the same time, the j-th task is automatically inserted into the optimal position of the current task sequence of the i-th drone. If the task is assigned to the 0th drone, it means that the task is abandoned.

[0020] Furthermore, in step (2), the state is defined as the current task allocation result, which corresponds to the task sequence of each UAV. Therefore, the initial state is represented by the initial task sequence of each UAV as (T0, T0).

[0021] Furthermore, in step (3), the adaptive simulation quantity selection mechanism sets the upper limit of the Monte Carlo tree search times to N times the product of the number of remaining tasks and the number of remaining drones, based on the number of remaining tasks and the number of remaining drones in the current state. Here, N is a positive integer, determined by the computing power of the computing device. The stronger the computing power, the larger the value of N. The default selection is 4.

[0022] Furthermore, in step (3), a Monte Carlo action search tree is constructed for each action to be executed. The root node of the tree is the current task allocation result, which is the current system state. The edges of the tree represent the actions to be executed, and the nodes of the tree represent the new task allocation state reached after the action is executed. During the action search process, the four steps of selection, expansion, simulation and backtracking are executed cyclically until the number of searches reaches a given upper limit. Finally, based on the constructed search tree, the action corresponding to the optimal child node under the root node is selected as the action to be executed.

[0023] Furthermore, the simulation starts from the newly expanded nodes, performs task allocation based on a heuristic simulation strategy, continues until all tasks are allocated, and finally evaluates the quality of the task allocation results.

[0024] Furthermore, the heuristic simulation strategy is a random task allocation strategy designed based on the heuristic information provided by the task allocation objective function and expert experience. It consists of two parts: heuristic sampling of candidate tasks and heuristic sampling of candidate UAVs.

[0025] The specific method for heuristic sampling of candidate tasks is to calculate the score of each candidate task, use the softmax function to transform the task score into a probability distribution, and then randomly select the task index according to the distribution.

[0026] Furthermore, the score of a candidate task is determined by the ratio of task reward to time, T j The task score is calculated as follows:

[0027]

[0028] Furthermore, the specific method for heuristic sampling of candidate drones is to calculate the score of each candidate drone, use the softmax function to transform the drone score into a probability distribution, and then randomly select drone indices based on this distribution.

[0029] Furthermore, the score of the candidate drone is primarily determined by the minimum distance between the drone's route and the newly selected task, for the selected task T. j U j The score is calculated as follows:

[0030]

[0031] Furthermore, the evaluation of task allocation results uses the best result from all simulations as the evaluation benchmark. If the cost of the current simulation is lower than the current benchmark, then the simulation is considered successful, earning 1 point, and the benchmark is updated to reflect the result of this simulation. If the cost of the simulation result exceeds the range of the benchmark, it is considered a failure, earning 0 points. Finally, the terminal node's score is evaluated as follows:

[0032]

[0033] Where α>1.0 represents the evaluation coefficient, J best This represents the score for the current best task allocation result.

[0034] Furthermore, in step (4), the task allocation result is selected from two results: first, the action sequence consisting of the optimal actions selected in each round of the Monte Carlo tree search; second, the optimal sequence obtained from all simulations performed in the Monte Carlo tree search, which is the evaluation benchmark mentioned in the task allocation result evaluation section. By comparing the two types of action sequences, the action sequence with lower cost is selected and used as the final task allocation result.

[0035] Innovation points and their advantages / effects

[0036] 1. The task allocation problem in reconnaissance and rescue is expressed as a sequential decision-making process. By making full use of the heuristic information of the problem, a state space and an action space are designed to balance the completeness and number of actions, thus saving computational resources while ensuring the allocation effect.

[0037] 2. An adaptive simulation number selection mechanism is proposed, which can automatically select the number of simulations based on the tasks to be assigned and the number of drones, avoiding a large number of repeated simulations, thereby reducing the computational load of the algorithm and further improving the computational efficiency of task allocation. Attached Figure Description

[0038] Figure 1 A flowchart illustrating the method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0040] The method of this invention can be used for unmanned swarm task allocation in reconnaissance and rescue operations.

[0041] like Figure 1 As shown, the specific implementation steps of the present invention are as follows:

[0042] Step 1: Mathematical modeling of the task allocation problem.

[0043] In step 1, assume that the drones depart from and return from the same airport. Each drone can perform multiple tasks, but each task is assigned to only one drone. There are N drones capable of performing tasks, and the set of drones is represented as:

[0044] U={U i}, i=1,...,N

[0045] Where 'i' represents the drone's serial number, and each drone has different capabilities and attributes, with its flight time, flight speed, reconnaissance capability, and rescue capability represented as follows:

[0046] In the task allocation problem, the set of tasks to be executed can be represented as:

[0047] T = {T j},j=0,1,....,M

[0048] in, Represents the task sequence number. Represents task i. Tasks representing departures from or return to the airport. The connectivity and distance between any two tasks are represented by a symmetric matrix. express, Indicates task and The distance between them Represents task i. This represents task j. Indicates from arrive The route is prohibited, mission The execution time, reward, search demand, and rescue demand are respectively expressed as: .

[0049] Furthermore, taking the maximum task completion time of all drones and minimizing the total reward of all unassigned tasks as the task objective, the objective function is expressed as follows:

[0050]

[0051] Among them, γ = {γ0, γ1, γ2,...,γ N} represents a set of task sequences assigned to each drone, γ0 represents the set of all unassigned tasks, and γ i =(τ0,τ1,τ2,...,τ M ) is a drone U i The task allocation sequence, τ * This represents the task index, for example: τ * =1 represents task number 1.

[0052] Furthermore, to ensure mission completion, the solution must also meet two constraints. The first is the endurance constraint, ensuring that each drone has enough energy to return to the airport. The flight time and mission execution time of each drone must be less than its maximum flight endurance, i.e.

[0053]

[0054] Furthermore, the second constraint is capability constraint, meaning that only suitable drones can be deployed to perform specific tasks; that is, the search and rescue requirements of the task cannot exceed the capabilities of the drone.

[0055]

[0056] Step 2: Design system actions and states. The system action is defined as assigning task j to drone i, and automatically inserting the j-th task into the optimal position of the current task sequence of drone i. Assigning a task to drone 0 means abandoning the task.

[0057] Furthermore, the state is defined as the current task allocation result, corresponding to the task sequence of each UAV. Therefore, the initial state is represented by the initial task sequence of each UAV as (T0, T0). Based on the above definitions of actions and states, the solution to the task allocation problem can be transformed into a series of action sequences. By executing this action sequence, i.e., selecting tasks one by one and allocating them to the corresponding UAVs, the task allocation can eventually be completed.

[0058] Step 3: Perform a Monte Carlo tree search to determine the actions to be performed. Specifically, an adaptive simulation quantity selection mechanism is used. Based on the number of remaining tasks and the number of drones in the current state, the upper limit of the number of Monte Carlo tree searches is set to N times the product of the number of remaining tasks and the number of remaining drones, where N is a positive integer determined by the computing power of the computing device; the default value is 4. Based on the above definitions of actions and states, the solution to the task allocation problem can be transformed into a series of action sequences. By executing a sequence of tasks, i.e., selecting tasks one by one and assigning them to the corresponding drones, the task allocation can be completed.

[0059] Adaptive adjustment of search count: The search count represents how many times the complete selection, expansion, simulation, and backtracking process needs to be executed. Too many searches will result in long computation times and wasted resources. Too few searches may miss important optimal solutions, leading to a decrease in solution quality.

[0060] Furthermore, the Monte Carlo tree search constructs a Monte Carlo action search tree for each action to be executed. The root node of the tree is the current task allocation result, which is the current system state. The edges of the tree represent the actions to be executed, and the nodes of the tree represent the new task allocation state reached after the action is executed. During the action search process, the four steps of selection, expansion, simulation, and backtracking are executed cyclically until the number of searches reaches a given upper limit. Finally, based on the constructed search tree, the action corresponding to the optimal child node under the root node is selected as the action to be executed.

[0061] Furthermore, the simulation begins with newly expanded nodes, performing task allocation based on a heuristic simulation strategy until all tasks are assigned, and finally evaluating the quality of the task allocation results. This invention provides a detailed design for the heuristic simulation strategy and the evaluation of task allocation results.

[0062] Furthermore, the heuristic simulation strategy is a random task allocation strategy designed based on the heuristic information provided by the task allocation objective function and expert experience. It consists of two parts: heuristic sampling of candidate tasks and heuristic sampling of candidate UAVs.

[0063] The specific method for heuristic sampling of candidate tasks is to calculate the score of each candidate task, use the softmax function to transform the task score into a probability distribution, and then randomly select the task index according to the distribution.

[0064] Furthermore, the score of a candidate task is determined by the ratio of task reward to time; the higher the ratio, the higher the score, and the greater the probability of being selected. j The task score is calculated as follows:

[0065]

[0066] Furthermore, the specific method for heuristic sampling of candidate drones is to calculate the score of each candidate drone, use the softmax function to transform the drone score into a probability distribution, and then randomly select drone indices based on this distribution.

[0067] Furthermore, the score of a candidate drone is primarily determined by the minimum distance between the drone's route and the newly selected task. A smaller minimum distance indicates that the candidate drone is more likely to complete the task in a shorter time. For the selected task T... j U j The score is calculated as follows:

[0068]

[0069] Furthermore, the task allocation results are evaluated: The purpose of evaluating the task allocation results is to assess the reward magnitude of the final result of each simulation and update the value or score of the terminal node accordingly. Throughout the Monte Carlo tree search process, this invention maintains the best result among all simulations as the evaluation benchmark. If the cost obtained in this simulation is lower than the current benchmark, then this simulation is judged as successful, earning 1 point, and the benchmark is updated to reflect the result of this simulation; if the cost of the simulation result exceeds the range of the benchmark, it is judged as a failure, earning 0 points. Finally, the terminal node's score is evaluated as follows:

[0070]

[0071] Where α>1.0 represents the evaluation coefficient, J best This represents the score for the current best task allocation result.

[0072] Step 4: Repeat Step 3 until all tasks have been assigned, and finally output the task assignment results. The task assignment results are selected from two sources: First, the action sequence formed by the optimal actions selected in each round of the Monte Carlo tree search; second, the optimal sequence obtained from all simulations performed in the Monte Carlo tree search, which is the evaluation benchmark mentioned in the task assignment result evaluation section. By comparing the two types of action sequences, the action sequence with the lower cost is selected and used as the final task assignment result.

[0073] Finally, it should be noted that the above examples are only used to illustrate the methods of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search, characterized in that, Includes the following steps: Step 1: Mathematical Modeling of the Task Assignment Problem: Establish the set of drones capable of performing tasks and the set of tasks to be performed, clarifying the task objectives and constraints; assume that drones depart from and return from the same airport, each drone can perform multiple tasks, but each task is assigned to only one drone, and there are N drones capable of performing tasks, represented by the set of drones as follows: , The drone's serial number represents the set of tasks to be executed in the task allocation problem. , Represents the task sequence number; among which, Represents task i. Tasks representing departures from or return to the airport; the connectivity and distance between any two tasks are represented by a symmetric matrix. It means that, among them Indicates task and The distance between them Represents task i. Represents task j; Indicates from arrive The route is prohibited; each drone has different capabilities and attributes, the first... The flight time, flight speed, reconnaissance capability, and rescue capability of the drone are respectively expressed as: ;Task The execution time, reward, search demand, and rescue demand are respectively expressed as: The score for a candidate task is determined by the ratio of task reward to time. The task score is calculated as follows: Step 2: Design system actions and states, transforming the task allocation problem into a sequential decision problem, that is, the complete task allocation is achieved by executing a series of system actions; Step 3: Perform Monte Carlo tree search to determine the action to be performed: Use an adaptive simulation number selection mechanism to determine the number of searches, and repeatedly execute the four steps of selection, expansion, simulation and backtracking until the number of searches reaches a given upper limit. Based on the constructed search tree, select the action corresponding to the optimal child node under the root node as the action to be performed. Step 4: Repeat Step 3 until all tasks have been assigned, and finally output the task assignment results. The task assignment results are selected from two results: First, the action sequence consisting of the optimal actions selected in each round of Monte Carlo tree search; Second, the optimal sequence obtained from all simulations in Monte Carlo tree search, which is the evaluation benchmark mentioned in the task assignment result evaluation section. By comparing the two types of action sequences, the action sequence with lower cost is selected and used as the final task assignment result.

2. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, The objective function is to minimize the maximum task completion time for all drones and the total reward for all unassigned tasks. in, It is a sequence of tasks assigned to each drone. This represents the set of all unassigned tasks. It is a drone The task allocation sequence, Indicates the task index. Represents tasks numbered 1, 2...M.

3. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 2, characterized in that, The flight time and mission execution time of each drone must be less than its maximum flight endurance, i.e. 。 4. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 3, characterized in that, The search and rescue requirements of the mission must not exceed the capabilities of the drone, that is... ; The adaptive simulation quantity selection mechanism sets the upper limit of the Monte Carlo tree search count to N times the product of the number of remaining tasks and the number of remaining drones, based on the number of remaining tasks and the number of remaining drones in the current state. Here, N is a positive integer, determined by the computing power of the computing device. The stronger the computing power, the larger the value of N. The default selection is 4.

5. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 4, characterized in that, In step 2, the system action is designed to select the first... j The task is assigned to the first i A drone, of which j This represents the sequence number of this task among all tasks. i This represents the serial number of the drone among all drones, and also indicates the first... j The task is automatically inserted into the first... i The optimal position of the 0th drone in the current mission sequence; if the mission is assigned to the 0th drone, it means that the mission is abandoned.

6. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 5, characterized in that, In step 2, the state is defined as the current task allocation result, corresponding to the task sequence of each UAV. Therefore, the initial state is represented by the initial task sequence of each UAV. .

7. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, In step 3, a Monte Carlo action search tree is constructed for each action to be executed. The root node of the tree is the current task allocation result, which is the current system state. The edges of the tree represent the actions to be executed, and the nodes of the tree represent the new task allocation state reached after the action is executed. During the action search process, the four steps of selection, expansion, simulation, and backtracking are executed cyclically until the number of searches reaches a given upper limit. Finally, based on the constructed search tree, the action corresponding to the optimal child node under the root node is selected as the action to be executed.

8. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, The simulation starts with newly expanded nodes, performs task allocation based on a heuristic simulation strategy, continues until all tasks are allocated, and finally evaluates the quality of the task allocation results.

9. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, Heuristic simulation strategy is a random task allocation strategy designed based on the heuristic information provided by the task allocation objective function and expert experience. It consists of two parts: heuristic sampling of candidate tasks and heuristic sampling of candidate UAVs.

10. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, The specific method for heuristic sampling of candidate tasks is to calculate the score of each candidate task, use the softmax function to transform the task score into a probability distribution, and then randomly select the task index according to the distribution.

11. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, The specific method for heuristic sampling of candidate drones is to calculate the score of each candidate drone, use the softmax function to transform the drone score into a probability distribution, and then randomly select drone indices based on this distribution.

12. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, The score of a candidate drone is primarily determined by the minimum distance between the drone's route and the newly selected task. of The score is calculated as follows: 。 13. The method for allocating unmanned swarm reconnaissance and rescue missions based on Monte Carlo tree search according to claim 1, characterized in that, The evaluation of task allocation results uses the best result from all simulations as the benchmark. If the cost of the current simulation is lower than the current benchmark, the simulation is considered successful, earning 1 point, and the benchmark is updated to reflect the result of this simulation. If the cost of the simulation exceeds the benchmark range, it is considered a failure, earning 0 points. Ultimately, the terminal node's score is evaluated as follows: in, Represents the evaluation coefficient. This represents the score for the current best task allocation result.

Images