Search and rescue robot group cooperative task allocation and path planning method and system

Abstract

The invention provides a search and rescue robot group cooperative task allocation and path planning method and system. The method comprises the following steps: constructing a global product Markov decision process model according to task requirements corresponding to search and rescue robot group system tasks and current environment information; based on the MDP model, circularly selecting robots, and solving an optimal strategy of each robot in the local MDP model by using an improved Q-learning algorithm; the selected robot is controlled to move according to the optimal strategy, the environment estimation weight of each moving step of the robot is updated, and the environment estimation results of all the robots are fused; updating a global product MDP according to an environment fusion result, judging whether to enter a final state of the global product MDP, and if so, stopping calculation; if not, returning is carried out, and the final state is that all the robots complete the search and rescue tasks and all the positions in the implementation process are updated; the search and rescue robot aims at overcoming the defects in the prior art and effectively improving the applicability of the search and rescue robot in the actual task environment.

Description

Technical Field

[0001] The invention belongs to the field of electric power technology and relates to a method and system for collaborative task allocation and path planning of a search and rescue robot group. Background Art

[0002] In recent years, with the rapid development of robotics technology, search and rescue robots have been widely used in natural disasters, accident rescue and other fields. Search and rescue robots perform tasks in complex and dangerous environments, and their task allocation and path planning problems have become the core challenges of robot system control.

[0003] In search and rescue missions, the specific rescue environment is usually not fully understood. For example, the location of survivors in earthquake ruins is unclear, and the internal structure of collapsed buildings is complex and unstable. These situations require search and rescue robots to have autonomous exploration and environmental perception capabilities, and to be able to effectively perform tasks in dynamic and unknown environments. At the same time, multiple robots have different perceptions of the environment, and the effective fusion of the estimation results of each robot is also the key to ensuring the success of the mission. In addition, there are certain errors in the sensors used by the robot to perceive the environment, which will affect the accuracy of path planning. At the same time, due to uncertain factors such as potential signal interference or mechanical failure, the robot may have unexpected action deviations when performing tasks, affecting the success rate of the task. In order to meet such challenges, it is particularly important to develop a path planning method that can handle complex search and rescue tasks and has high autonomy and high reliability. When dealing with complex task path planning problems under various conditions, the existing technology decouples task allocation and path planning into two problems for consideration. However, the decoupled problem often leads to a large difference between the path obtained and the optimal path of the original problem. Therefore, a new solution is needed to coordinate task allocation and path planning to improve the versatility and task execution capabilities of search and rescue robots.

[0004] Based on this background, this patent proposes a method and system for collaborative task allocation and path planning of a search and rescue robot group. This method and system realize autonomous exploration and perception of the environment through the collaborative work of multiple search and rescue robots, solve the path planning problem of complex tasks, and significantly improve the efficiency and success rate of search and rescue tasks. Through this method, search and rescue robots can better cope with complex and changing environments, complete challenging rescue tasks, and provide strong support for actual rescue work. Summary of the invention

[0005] The present invention provides a method and system for collaborative task allocation and path planning of a search and rescue robot group to overcome the shortcomings of the prior art, so that the search and rescue robot group can perform autonomous exploration, information integration and autonomous decision-making in complex and interfering mission scenarios and complete the mission safely, thereby effectively improving the applicability of the search and rescue robot in actual mission environments.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a search and rescue robot group collaborative task allocation and path planning method, comprising the following steps:

[0008] S1: Construct a global product Markov decision process model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information;

[0009] S2: Based on the MDP model, the robot is selected cyclically, and the improved Q-learning algorithm is used to solve the optimal strategy of each robot in the local MDP model. The optimal strategy refers to the shortest path that ensures the safety of each robot when completing the specified task requirements in the search and rescue process, the list of tasks performed by the robot in the process, and the set of actions performed under each task;

[0010] S3: Control the selected robot to move according to the optimal strategy, update the environment estimation weight of each robot's movement, and fuse the environment estimation results of each robot to obtain the environment fusion result;

[0011] S4: Update the global product MDP according to the environment fusion result, and determine whether it has entered the final state of the global product MDP. If it has, stop the calculation, that is, the robot system completes the task; otherwise, return to S2, and the final state is that all robots have completed their search and rescue tasks and all position updates during the implementation process.

[0012] Optionally, S1: constructing a global product Markov decision process model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information includes:

[0013] Discretize the robot's mission environment and determine the corresponding search and rescue robots and the mission requirements for each robot;

[0014] Adopting adaptive weighted fusion algorithm to fuse the environment estimation results of each robot, and obtain the weighted fusion results of the environment of all robots;

[0015] Based on the weighted fusion results of the environments of all machines, the spatial state, initial state, final state, action state, transition probability, and reward function of each robot are determined to complete the construction of the global product Markov decision process model.

[0016] Optionally, an adaptive weighted fusion algorithm is used to fuse the environment estimation results of each robot, and the weighted fusion results of the environment of all robots are obtained, including:

[0017] Determine the set of environment estimation results of all robots at the i-th time step;

[0018] The environment estimation results of each robot for each grid in the calculation environment estimation result set are weighted averaged to obtain the environment weighted fusion result.

[0019] Optionally, S2: based on the MDP model, cyclically selecting robots, and using the improved Q-learning algorithm to solve the optimal strategy of each robot in the local MDP model includes:

[0020] Select robots in a step-by-step loop and determine the local MDP model of each robot;

[0021] Calculate the robot r in the local MDP model l Complete the current mid-way visit task m j The estimated cost X includes the current robot r j The estimated distance X1 required to complete all remaining tasks and the current robot r j The judgment parameter X2 of whether the task should be completed;

[0022] Update the reward function value of the robot when moving according to the estimated consumption X;

[0023] According to the reward function value, the improved Q-learning algorithm is used to solve the optimal strategy of each robot in the local MDP model.

[0024] Optionally, calculate the robot r in the local MDP model j The estimated cost X to complete the current mid-way visit task m includes:

[0025] Calculate the estimated distance matrix based on each robot's mid-way visit mission area, terminal mission area, and prohibited visit area;

[0026] Calculate the shortest distance matrix between every two subtask areas based on the estimated distance matrix;

[0027] Solve the robot r based on the shortest distance matrix l Complete the current mid-way visit task m j The estimated consumption is X.

[0028] Optionally, according to the reward function value, the optimal strategy of each robot in the local MDP model is solved using the improved Q-learning algorithm, including:

[0029] Determine the action strategy of the robot every time it moves based on the reward function;

[0030] Control the robot's movement according to the action strategy and cache the robot's trajectory;

[0031] Based on the cached trajectory, the cumulative reward is calculated and the Q value is updated.

[0032] Finally, the final selected action is output, the action is executed, the trajectory is cached and the Q table is updated.

[0033] Optionally, the selected robot moves according to the action strategy, updates its environment estimation weight, and fuses the estimation results of each robot to obtain a new fusion result.

[0034] Optionally, update the state of the global product MDP (the positions of all robots) according to the new fusion result and the completion status of all current tasks V M 、V F 、V N ), determine whether it has entered the final state of the global product MDP. If so, the algorithm stops, that is, the robot system completes the task; otherwise, return to S2.

[0035] The present invention also provides a search and rescue robot group collaborative task allocation and path planning system, comprising:

[0036] A modeling module is used to construct a global product Markov decision process model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information;

[0037] The optimization module is used to select robots cyclically based on the MDP model and use the improved Q-learning algorithm to solve the optimal strategy of each robot in the local MDP model. The optimal strategy refers to the shortest path that ensures the safety of each robot when completing the specified task requirements in the search and rescue process, the list of tasks performed by the robot in the process, and the set of actions performed under each task;

[0038] The fusion module is used to control the selected robot to move according to the optimal strategy, update the environmental estimation weight of each robot's movement, and fuse the environmental estimation results of each robot to obtain the environmental fusion result;

[0039] The output module is used to update the global product MDP according to the results of environment fusion and determine whether it has entered the final state of the global product MDP. If so, the calculation is stopped, that is, the robot system completes the task; otherwise, it returns to S2, and the final state is that all robots have completed their search and rescue tasks and all position updates in the implementation process.

[0040] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for collaborative task allocation and path planning of a search and rescue robot group in any one of the above-mentioned embodiments are implemented.

[0041] The beneficial effects of the present invention are:

[0042] The present invention discloses a method and system for collaborative task allocation and path planning for a search and rescue robot group. When the task environment is initially unknown and each robot has a different understanding of the environment, an adaptive weighted fusion algorithm is used to fuse the environmental estimation results of each robot to improve the accuracy and consistency of the estimation. The MDP model of the multi-robot system is constructed by converting the task requirements into a series of feature vectors, and a local calculation and global update MDP framework is proposed to improve the scalability of the algorithm in the multi-robot system. In addition, a heuristic reward function design method is introduced to effectively improve the solution quality. For the problem of collaborative task allocation and path planning for a search and rescue robot group, the present invention can efficiently generate mobile strategies, so that the robot can efficiently perform search and rescue tasks while ensuring its own safety. It can be effectively used to solve problems in actual environments and has universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a search and rescue robot group collaborative task allocation and path planning method provided by the present invention;

[0044] Figure 2 A flowchart of a method for collaborative task allocation and path planning of a search and rescue robot group provided by the present invention;

[0045] Figure 3 A global environment diagram of a robot according to an embodiment of the present invention, wherein: Figure 3 a is the actual factory environment. Figure 3 b is the environment diagram after discrete decomposition;

[0046] Figure 4 A schematic diagram of a cyclic MDP provided by the present invention;

[0047] Figure 5 A schematic diagram of a mobile track task area of ​​a robot provided by the present invention;

[0048] Figure 6 A moving trajectory diagram of a robot provided by the present invention;

[0049] Figure 7 A schematic diagram of a search and rescue robot group collaborative task allocation and path planning system provided by the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] The present invention provides a search and rescue robot group collaborative task allocation and path planning method, such as Figure 1 As shown, the following steps are included:

[0052] S1: Construct a global product Markov decision process (MDP) model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information;

[0053] In one embodiment, the task environment of the search and rescue robot group collaborative task allocation and path planning is discretely decomposed, and the task requirements are converted into computable logical expressions;

[0054] A global product Markov decision process (MDP) model is constructed based on the mission requirements of the search and rescue robot and the current environmental information.

[0055] It should be noted that the task requirement here refers to the mid-way visit task area Ω of each robot M , terminal mission area Ω F , Prohibited access areaΩ N , and the number of robots of different types.

[0056] S2: Based on the MDP model, the robot is selected cyclically, and the improved Q-learning algorithm is used to solve the optimal strategy of each robot in the local MDP model. The optimal strategy refers to the shortest path that ensures the safety of each robot when completing the specified task requirements in the search and rescue process, the list of tasks performed by the robot in the process, and the set of actions performed under each task;

[0057] In a real-time control, based on the constructed MDP model, a local MDP model of each robot is established, and each robot is selected for control in turn in a loop, and a corresponding action is selected for each robot each time it moves. The improved Q-learning algorithm is used to solve the optimal strategy of each robot in the local MDP model, and each action selection is scored to obtain a reward value Q. The higher the reward value, the more optimal the action selected by the robot, and the higher the possibility of reaching the next position with the shortest distance. The robot's trajectory is cached, and finally the shortest path to ensure the safety of each robot when completing the specified task requirements in the search and rescue process, the list of tasks performed by the robot in the process, and the set of actions performed under each task are output.

[0058] S3: Control the selected robot to move according to the optimal strategy, update the environment estimation weight of each robot's movement, and fuse the environment estimation results of each robot to obtain the environment fusion result;

[0059] In one embodiment, after each robot moves based on the optimal strategy, the environment estimation of each step of the robot is re-weighted and the weighted environment estimation results of all robots are fused to obtain an environment fusion result.

[0060] S4: Update the global product MDP according to the results of environment fusion, and determine whether it has entered the final state of the global product MDP. If so, stop the calculation, that is, the robot system completes the task; otherwise, return to S2, and the final state is that all robots have completed their search and rescue tasks and all position updates during the implementation process.

[0061] In one embodiment, the global product MDP is updated each time according to the environment fusion result of the machine, and it is determined whether to update and fuse the environment of all robots to enter the final state. If it is entered, the calculation is stopped, that is, the robot system completes the task; otherwise, it returns to S2, and the final state is that all robots have completed their search and rescue tasks and all position updates in the implementation process.

[0062] Optionally, S1: constructing a global product Markov decision process (MDP) model according to the task requirements corresponding to the search and rescue robot group system task and the current environment information includes:

[0063] Discretize the robot's mission environment and determine the corresponding search and rescue robots and the mission requirements for each robot;

[0064] Adopting adaptive weighted fusion algorithm to fuse the environment estimation results of each robot, and obtain the weighted fusion results of the environment of all robots;

[0065] Based on the weighted fusion results of the environments of all machines, the spatial state, initial state, final state, action state, transition probability, and reward function of each robot are determined to complete the construction of the global product Markov decision process (MDP) model.

[0066] In one embodiment, the robot's task environment is discretized, and the decomposed environment is defined as C = {c1, c2, ..., c n}, consisting of a group of k identical mobile robots A logical expression The form is:

[0067]

[0068] Among them, subtask M represents the logical requirement for the mid-way visit task, and subtask F represents the logical requirement for the final stop task, that is, the area that the robot needs to visit (cannot visit) midway and the area where the robot finally stops.

[0069] Adaptive weighted fusion algorithm is used to fuse the environmental estimation results of each robot. Since the robot cannot accurately recognize the environment, probability is used to describe the robot's estimation of each grid. is the set of environment estimation results of all robots at the i-th time step;

[0070]

[0071] and are respectively the i-th step robot r j For grid c ll The mid-mission task environment estimation results and the terminal task environment estimation results, p k With q k Respectively represent the probability of the task corresponding to the current grid;

[0072] Adopting adaptive weighted fusion algorithm to fuse the environment estimation results of each robot, the weighted fusion results of the environment of all robots are obtained, including:

[0073] Determine the set of environment estimation results of all robots at the i-th time step;

[0074] The environment estimation results of each robot for each grid in the calculation environment estimation result set are weighted averaged to obtain the environment weighted fusion result.

[0075] Specifically, a weight is assigned to each robot's environmental estimation result for each grid.

[0076]

[0077] Among them, e u，j For robot j For the grid c u Observation error, σ is a very small number greater than 0 to prevent the denominator from being 0. Observation error The larger the error, the lower the weight of the robot. The calculated weights are used to weight the average of the environment estimates of all robots. For each grid c ll Compute the fused environment estimate:

[0078]

[0079] The weighted fusion result of the environment is obtained, and the fused environment is estimated to be:

[0080]

[0081] It should be noted that an adaptive weighted fusion algorithm is used to fuse the environmental estimation results of each robot to improve the accuracy and consistency of the estimation.

[0082] And define the robot's state space Each of these states Represents robot r j Current location, V N =[v n ]∈{0,1} represents the satisfaction of each subtask in the global MDP. If the corresponding task is completed, the corresponding position is 1, otherwise it is 0;

[0083] Initial state

[0084] Final state set in represents the state that satisfies the Boolean task, that is And

[0085] Action Space in is the combined action of all robots, Represents robot r j Move to grid c on the map x .function Restricts the available actions for a robot to move to adjacent cells, and prohibits movement to a cell currently occupied by another robot.

[0086] Transition probability Used to describe situations where motion is uncertain.

[0087] The reward function is defined as:

[0088]

[0089] when , indicating that the forbidden access task is violated. Therefore, a reward of -h will be obtained, where h is a positive integer. and When , it means that the global task is satisfied and the final state is reached, and a reward of h will be obtained. In other cases, the reward is -0.1.

[0090] Optionally, S2: based on the MDP model, cyclically selecting robots, and using the improved Q-learning algorithm to solve the optimal strategy of each robot in the local MDP model includes:

[0091] Select robots in a step-by-step loop and determine the local MDP model of each robot;

[0092] Calculate the robot r in the local MDP model l Complete the current mid-way visit task m i The estimated cost X includes the current robot r j The estimated distance X1 required to complete all remaining tasks and the current robot r l The judgment parameter X2 of whether the task should be completed;

[0093] Update the reward function value of the robot when moving according to the estimated consumption X;

[0094] According to the reward function value, the improved Q-learning algorithm is used to solve the optimal strategy of each robot in the local MDP model.

[0095] In one embodiment, a robot is selected cyclically, its local MDP model is constructed, and its optimal strategy is solved using an improved Q-learning algorithm, which specifically includes the following steps:

[0096] The local MDP is defined as follows:

[0097] 1) State Space in represents the robot r in the local MDP j location, Indicates the satisfaction of each subtask in the local MDP.

[0098] 2) Initial state in Represents robot r in the global MDP j Current location, and Respectively represent the current satisfaction of each subtask in the global MDP.

[0099] 3) Final state set in Express satisfaction The status of the subtask in , or satisfy All corresponding subtasks and stop at The status of the grid corresponding to the completed subtask.

[0100] 4) Action Space in Indicates that the robot moves to grid c on the map x .function Restricts the available actions for a robot to move to adjacent cells, and prohibits movement to a cell currently occupied by another robot.

[0101] 5) Transition probability Used to describe situations where motion is uncertain.

[0102] 6) The reward function is defined as:

[0103]

[0104] when , indicating that violating the prohibited access will receive a reward of -h, where h is also a positive integer. Satisfying any stop task will receive a reward of +h. The reward for taking a stop action is 0. When completing an intermediate access task, the reward is 1 / 1+X. In all other cases, the reward is -0.1. The parameter X represents the current robot r l The estimated cost to complete this task, including the current robot r l The estimated distance X1 required to complete all remaining tasks and the current robot r l The judgment parameter X2 is whether this task should be completed.

[0105] Update the reward function value of the robot when moving according to the estimated consumption X;

[0106] According to the reward function value, the improved Q-learning algorithm is used to solve the optimal strategy of each robot in the local MDP model.

[0107] It should be noted that the MDP model of the multi-robot system is constructed by converting the task requirements into a series of feature vectors, and a local calculation and global update MDP framework is proposed to improve the scalability of the algorithm in the multi-robot system.

[0108] Optionally, calculate the robot r in the local MDP model l Complete the current mid-way visit task m j The estimated consumption of X includes:

[0109] Calculate the estimated distance matrix based on each robot's mid-way visit mission area, terminal mission area, and prohibited visit area;

[0110] Calculate the shortest distance matrix between every two subtask areas based on the estimated distance matrix;

[0111] Solve the robot r based on the shortest distance matrix l Complete the current mid-way visit task m j The estimated consumption is X.

[0112] In one embodiment, the robot r in the local MDP model is calculated according to the following steps: l Complete the current mid-way visit task m j Estimated consumption of X:

[0113] First, the estimated distance matrix Z is calculated by the following algorithm:

[0114]

[0115]

[0116] Next, we calculate the matrix Z to represent the shortest distance between every two subtask areas, which is obtained by the following steps:

[0117] Step 1: For each mid-access task Define a matrix V i . From the corresponding The grid in c j Determine the matrix V in the rows of the matrix Z j The value of the corresponding element, where V i (j;) = Z(j, ),

[0118] Step 2: For the matrix V i For each column x=1, 2, ..., calculate V j Minimum value of all rows y to form a vector

[0119]

[0120] Step 3: Vectors are aggregated to form a matrix Z′=[w1, W2, .., W p ] T .

[0121] Finally, the robot l Complete the current mid-way visit task m j The estimated consumption X is:

[0122] X=X1+X2

[0123] 2) The calculation process of parameter X1 is:

[0124]

[0125]

[0126]

[0127] Where J is a large constant.

[0128] 3) The calculation process of parameter X2 is:

[0129]

[0130]

[0131]

[0132] Optionally, according to the reward function value, the optimal strategy of each robot in the local MDP model is solved using the improved Q-learning algorithm, including:

[0133] Determine the action strategy of the robot every time it moves based on the reward function;

[0134] Control the robot's movement according to the action strategy and cache the robot's trajectory;

[0135] Based on the cached trajectory, the cumulative reward is calculated and the Q value is updated.

[0136] Finally, the final selected action is output, the action is executed, the trajectory is cached and the Q table is updated.

[0137] In one embodiment, the Q-learning algorithm based on multi-step learning and hybrid strategy solves the optimal strategy of each robot in the local MDP model:

[0138] 1) Initialize the Q table, set all Q(s,a) = 0, set the learning rate α, discount factor γ, exploration rate ε, temperature parameter τ and multi-step learning step size n.

[0139] 2) Softmax function definition: Define the Softmax function to convert the Q value into a probability distribution.

[0140]

[0141] 3) Action selection strategy: Combine epsilon-greedy and Softmax strategies to select actions. Generate a random number δ∈[0,1],

[0142]

[0143] 4) Trajectory cache initialization: Initialize the trajectory cache to an empty queue.

[0144] 5) Function to update the Q table: Use the cached trajectory information to calculate the cumulative reward and update the Q value. Cumulative reward: The trajectory reveals the state-action pair (s, a), the terminal state s',

[0145] Q(s,a)←Q(s,a)+α×(R+γ n×max Q(s',a')-Q(s,a))

[0146] 6) Training process: In each round, actions are selected, actions are executed, trajectories are cached, and the Q table is updated until the predetermined number of training times or convergence is reached. Finally, the final selection of actions, execution of actions, caching of trajectories, and updating of the Q table are output.

[0147] Optionally, the selected robot moves according to the action strategy, updates its environment estimation weight, and fuses the estimation results of each robot to obtain a new fusion result.

[0148] Optionally, update the state of the global product MDP (the positions of all robots) according to the new fusion result and the completion status of all current tasks V M 、V F 、V N ), determine whether it has entered the final state of the global product MDP. If so, the algorithm stops, that is, the robot system completes the task; otherwise, return to S2.

[0149] Based on the task allocation and path planning method of a search and rescue robot group system provided by the present invention, a complete and specific embodiment is given:

[0150] Reference Figure 2 , a search and rescue robot group collaborative task allocation and path planning method, comprising the following steps:

[0151] Step 1, such as Figure 3 As shown, the actual factory environment is Figure 3 a), the environment after discrete decomposition is Figure 3 b) Use the grid method to divide the global environment into 2500 regions, using the set C = {c1, c2, ..., c 2500} indicates that the numbering sequence increases from left to right and from top to bottom. The black area is the explosion collapse area and cannot be passed through. The white area is the free area and can be passed through. The robot group contains two robots Mission requirements: Π1-Π5 are pollution monitoring tasks, Π6-Π 10 For the personnel search and rescue mission, we finally returned to two temporary monitoring stations. 11 and Π 12 The logical expression corresponding to the task is:

[0152]

[0153] Step 2: The current estimate of the environment by each robot in the robot group is: is the set of environmental estimation results of the two robots at the i-th time step, where

[0154]

[0155] and are respectively the i-th step robot r j For grid c u The mid-mission task environment estimation results and the terminal task environment estimation results, p k With q k Respectively represent the probability of the current grid corresponding to the task;

[0156] Preferably, in step 3, a global product Markov decision process (MDP) model is constructed in combination with task requirements and current environment information, specifically including the following steps:

[0157] Step 3.1 State Space Each of these states Represents robot r j Current location, V N =[v n ]∈{0,1} represents the satisfaction of each subtask in the global MDP. If the corresponding task is completed, the corresponding position is 1, otherwise it is 0;

[0158] Step 3.2 Initial state C I ={c3,c 99};

[0159] Step 3.3 Final state set in represents the state that satisfies the Boolean task, i.e. And

[0160] Step 3.4 Action Space in is the combined action of all robots, Represents robot r j Move to grid c on the map x .function Restricts the available actions for a robot to move to adjacent cells, and prohibits movement to a cell currently occupied by another robot.

[0161] Step 3.5 Transition Probability Used to describe situations where motion is uncertain.

[0162] Step 3.6 The reward function is defined as:

[0163]

[0164] when , indicating that the forbidden access mission has been violated. Therefore, a -10 reward will be obtained. and , it means that the global task is satisfied and the final state is reached, and a reward of 10 will be obtained. In other cases, the reward is -0.1.

[0165] Preferably, in step 4, the robot r is selected cyclically j ,like Figure 4 As shown, construct its local MDP model and use the improved Q-learning algorithm to solve its optimal strategy, which specifically includes the following steps:

[0166] In step 4.1, the local MDP is defined as follows:

[0167] 1) State Space in represents the robot r in the local MDP i location, Indicates the satisfaction of each subtask in the local MDP.

[0168] 2) Initial state in Represents robot r in the global MDP j Current location, and Respectively represent the current satisfaction of each subtask in the global MDP.

[0169] 3) Final state set in Express satisfaction The status of the subtask in , or satisfy All corresponding subtasks and stop at The status of the grid corresponding to the completed subtask.

[0170] 4) Action Space in Indicates that the robot moves to grid c on the map x .function Restricts the available actions for a robot to move to adjacent cells, and prohibits movement to a cell currently occupied by another robot.

[0171] 5) Transition probability Used to describe situations where motion is uncertain.

[0172] 6) The reward function is defined as:

[0173]

[0174] X is calculated using the algorithm mentioned above.

[0175] Step 4.2, Q-learning algorithm based on multi-step learning and hybrid strategy:

[0176] 1) Initialize the Q table, set all Q(s, a) = 0, set the learning rate α = 0.2, the discount factor γ = 0.9, the initial exploration rate ε = 1, the initial temperature parameter τ = 100 and the multi-step learning step n = 10, and the total training rounds E = 500. The exploration rate reduction rule is Where θ is the current training round, E is the total training round, and the exploration rate will continue to decrease with the increase of training rounds, increasing the guidance of the algorithm. The temperature drop rule is μ=0.99 is the cooling rate.

[0177] 2) Softmax function definition: Define the Softmax function to convert the Q value into a probability distribution.

[0178]

[0179] Where τ is the current temperature.

[0180] 3) Action selection strategy: Combine epsilon-greedy and Softmax strategies to select actions. Generate a random number δ∈[0,1],

[0181]

[0182] 4) Initialize the track cache: Initialize the track cache to an empty queue. Select an action, execute the action, and cache the track.

[0183] 5) Function to update the Q table: Use the cached trajectory information to calculate the cumulative reward and update the Q value. Cumulative reward: The starting state-action pair in the trajectory is (s, a), and the ending state is s′.

[0184] Q(s,a)←Q(s,a)+0.2×((R+0.9 3 ×maxQ(s′,a′)-Q(s,a))

[0185] 6) Training process: In each round, actions are continuously selected, executed, trajectories are cached, and the Q table is updated until 500 rounds of training are reached.

[0186] Preferably, in step 5, the selected robot moves according to the obtained strategy, updates its environment estimation weight, and fuses the estimation results of each robot.

[0187] Preferably, in step 6, the state of the global product MDP (the positions of all robots) is updated and the completion status of all current tasks V M 、V F V N ), determine whether it has entered the final state of the global product MDP. If so, the algorithm stops, that is, the robot system completes the task; otherwise, return to step 4.

[0188] The final path is Figure 5 As shown, robot r1 is at the temporary monitoring station Π in the upper left corner. 11 Set out to perform the mission, robot R2 from the temporary monitoring station Π in the upper right corner 12 Start to execute the task, and finally obtain the task path, such as Figure 6 As shown, it can be seen that robot r1 completes detection tasks Π1, Π3 and Π4 and returns to Π 11 In the monitoring station area, robot r2 completes detection task Π2 and search and rescue tasks Π6, Π7, Π8, Π9, Π 10 Finally return to Π 12 The inspection station effectively ensures the completion of the global task.

[0189] The present invention also provides a search and rescue robot group collaborative task allocation and path planning system, such as Figure 7 As shown, including:

[0190] A modeling module is used to construct a global product Markov decision process (MDP) model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information;

[0191] The optimization module is used to select robots cyclically based on the MDP model and use the improved Q-learning algorithm to solve the optimal strategy of each robot in the local MDP model. The optimal strategy refers to the shortest path that ensures the safety of each robot when completing the specified task requirements in the search and rescue process, the list of tasks performed by the robot in the process, and the set of actions performed under each task;

[0192] The fusion module is used to control the selected robot to move according to the optimal strategy, update the environmental estimation weight of each robot's movement, and fuse the environmental estimation results of each robot to obtain the environmental fusion result;

[0193] The output module is used to update the global product MDP according to the results of environment fusion and determine whether it has entered the final state of the global product MDP. If so, the calculation is stopped, that is, the robot system completes the task; otherwise, it returns to S2, and the final state is that all robots have completed their search and rescue tasks and all position updates in the implementation process.

[0194] It should be noted that the search and rescue robot group collaborative task allocation and path planning system provided by the present invention can realize the above-mentioned search and rescue robot group collaborative task allocation and path planning method, which will not be repeated here.

[0195] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the search and rescue robot group collaborative task allocation and path planning method.

Claims

1. A method for collaborative task allocation and path planning of a search and rescue robot group, characterized in that: The following steps are involved: S1. Construct a global product Markov decision process model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information; S2. Based on the MDP model, select robots cyclically and use the improved Q-learning algorithm to solve the optimal strategy of each robot in the local MDP model. The optimal strategy refers to the shortest path that ensures the safety of each robot when completing the specified task requirements in the search and rescue process, the list of tasks performed by the robot in the process, and the set of actions performed under each task; S3, controlling the selected robot to move according to the optimal strategy, updating the environment estimation weight of each movement of the robot, and fusing the environment estimation results of each robot to obtain an environment fusion result; S4. Update the global product MDP according to the environment fusion result, and determine whether it has entered the final state of the global product MDP. If it has entered, stop the calculation, that is, the robot system completes the task; otherwise, return to S2, and the final state is that all robots have completed their search and rescue tasks and all position updates in the implementation process.

2. A search and rescue robot group collaborative task allocation and path planning method as claimed in claim 1, characterized in that: In step S1, constructing a global product Markov decision process model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information includes: Discretize the task environment of the robot and determine the corresponding search and rescue robot and the task requirements corresponding to each robot; Adopting adaptive weighted fusion algorithm to fuse the environment estimation results of each robot, and obtain the weighted fusion results of the environment of all robots; Based on the weighted fusion results of the environments of all the machines, the spatial state, initial state, final state, action state, transition probability, and reward function of each robot are determined to complete the construction of the global product Markov decision process model.

3. A search and rescue robot group collaborative task allocation and path planning method as claimed in claim 2, characterized in that: Adopting adaptive weighted fusion algorithm to fuse the environment estimation results of each robot, the weighted fusion results of the environment of all robots are obtained, including: Determine the set of environment estimation results of all robots at the i-th time step; The weighted average of the environment estimation results of each robot on each grid in the environment estimation result set is calculated to obtain an environment weighted fusion result.

4. A search and rescue robot group collaborative task allocation and path planning method as claimed in claim 1, characterized in that: In step S2, based on the MDP model, the robot is cyclically selected, and the optimal strategy of each robot in the local MDP model is solved using the improved Q-learning algorithm, including: Select robots in a step-by-step loop and determine the local MDP model of each robot; Calculate the robot r1 in the local MDP model to complete the current midway visit task m j The estimated cost X includes the estimated distance X1 required for the current robot r1 to complete all remaining tasks and the judgment parameter X2 of whether the current robot r1 should complete this task; Update the reward function value of the robot when moving according to the estimated consumption X; According to the reward function value, an improved Q-learning algorithm is used to solve the optimal strategy of each robot in the local MDP model.

5. A search and rescue robot group collaborative task allocation and path planning method as claimed in claim 1, characterized in that: Calculate the robot r1 in the local MDP model to complete the current midway visit task m j The estimated consumption of X includes: Calculate the estimated distance matrix based on each robot's mid-way visit mission area, terminal mission area, and prohibited visit area; Calculate the shortest distance matrix between every two subtask areas based on the estimated distance matrix; Based on the shortest distance matrix, the robot r1 completes the current mid-way visit task m j The estimated consumption is X.

6. A search and rescue robot group collaborative task allocation and path planning method as claimed in claim 4, characterized in that: According to the reward function value, the optimal strategy for solving each robot in the local MDP model using the improved Q-learning algorithm includes: Determine the action strategy of the robot each time it moves according to the reward function; Controlling the movement of the robot according to the action strategy, and caching the trajectory of the robot; According to the cached trajectory, calculate the cumulative reward and update the Q value; Finally, the final selected action is output, the action is executed, the trajectory is cached and the Q table is updated.

7. A search and rescue robot group collaborative task allocation and path planning method as claimed in claim 6, characterized in that: The selected robot moves according to the action strategy, updates its environment estimation weight, and fuses the estimation results of each robot to obtain a new fusion result.

8. A search and rescue robot group collaborative task allocation and path planning method as claimed in claim 7, characterized in that: The state of the global product MDP is updated according to the new fusion result to determine whether it has entered the final state of the global product MDP. If so, the algorithm stops, that is, the robot system completes the task; otherwise, it returns to S2.

9. A search and rescue robot group collaborative task allocation and path planning system, characterized in that: include: A modeling module is used to construct a global product Markov decision process model according to the task requirements corresponding to the search and rescue robot group system task and the current environmental information; An optimization module is used to cyclically select robots based on the MDP model, and use an improved Q-learning algorithm to solve the optimal strategy of each robot in the local MDP model, wherein the optimal strategy refers to the shortest path that ensures the safety of each robot when completing the specified task requirements in the search and rescue process, the task list performed by the robot in the process, and the set of actions performed under each task; A fusion module is used to control the selected robot to move according to the optimal strategy, update the environment estimation weight of each movement of the robot, and fuse the environment estimation results of each robot to obtain an environment fusion result; The output module is used to update the global product MDP according to the environment fusion result and determine whether it enters the final state of the global product MDP. If it does, the calculation is stopped, that is, the robot system completes the task; otherwise, it returns to S2, and the final state is that all robots complete their search and rescue tasks and all position updates in the implementation process.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the search and rescue robot group collaborative task allocation and path planning method as described in any one of claims 1 to 8 are implemented.

Images