Multi-agent unknown environment search and rescue method and system based on deep reinforcement learning

Abstract

The present invention provides a multi-agent unknown environment search and rescue method and system based on deep reinforcement learning, relating to the fields of path planning and unknown environment search and rescue technology. The present invention first obtains a Markov decision model for multi-agent unknown environment search and rescue; then, based on the Markov decision model, uses the QMIX algorithm to obtain the actions of each agent; each agent uses the A* algorithm to plan the optimal path from the current point to the next state target point based on the actions; and the process of the QMIX algorithm determining the actions and the A* algorithm planning the path is continuously repeated until a preset termination condition is met, and the multi-agent unknown environment search and rescue results are output. The present invention effectively improves the efficiency of multi-agent search and rescue in unknown environments.

Description

Technical Field

[0001] The present invention relates to path planning technology and the field of unknown environment search and rescue technology, and in particular to a multi-agent unknown environment search and rescue method and system based on deep reinforcement learning. Background Art

[0002] Emerging with the rapid development of artificial intelligence, rescue agents are a new type of environmental search and rescue technology, distinct from traditional manual rescue. They can provide support in hazardous environments, provide real-time mapping and monitoring of accident sites, and assist in search and rescue operations in unknown environments. However, as the social environment becomes increasingly complex, single agents are no longer able to handle the increasing complexity of real-world problems. This has led to the emergence of multi-agent algorithms and corresponding search and rescue technologies that combine dynamic target selection with collision avoidance strategies. However, how multiple agents can coordinate their search and rescue efforts in complex and unknown environments to locate their targets while avoiding collisions and ensuring their own safety remains a key challenge facing multi-agent search and rescue.

[0003] Currently, there are many path planning and reinforcement learning algorithms, such as: * The algorithm searches based on the target, and by continuously calculating the distance from each vertex to the target vertex, it guides the search queue to approach the target, finding the least vertices to ensure the shortest path; the QMIX algorithm uses a hybrid network to merge the local Q-value functions of single agents, and based on RNN learning map information, it can add global state information assistance during the training and learning process, effectively improving the algorithm performance.

[0004] However, although the above algorithms have achieved good results in navigation tasks such as path planning, there are still the following problems: * Traditional path planning algorithms rely on global task information for path planning. Under conditions of limited information, it is difficult to obtain target points for efficient path planning. The QMIX algorithm does not consider the interaction between agents and cannot perform coordinated search of multiple agents. At the same time, the QMIX algorithm ignores the information provided by mature movement methods in agents, increases the difficulty of training agents, and cannot efficiently realize search and rescue in unknown environments with multiple agents. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a multi-agent unknown environment search and rescue method and system based on deep reinforcement learning, which solves the problem in the existing technology that multi-agents cannot achieve efficient search and rescue in unknown environments.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention proposes a multi-agent unknown environment search and rescue method based on deep reinforcement learning, the method comprising:

[0010] S1. Obtain a Markov decision model based on multi-agent search and rescue in unknown environments;

[0011] S2. Determine the action of each agent based on the Markov decision model using the QMIX algorithm and agent-environment mutual information;

[0012] S3, each agent uses the A* algorithm to plan the optimal path from the current point to the next state target point based on the action;

[0013] S4. Loop through steps S2 and S3 until the preset termination condition is reached and output the multi-agent unknown environment search and rescue results.

[0014] Preferably, the Markov decision model is represented by a sextuple <S, A, T, R, Z, O>;

[0015] Where: S = {S1, S2, ..., SN} represents all possible states of the agent; A = {A1, A2, ..., AN} represents all possible actions of the agent; O = {O1, O2, ..., ON} represents all possible observation values ​​of the agent; R represents the reward value; T represents a set of conditional transition probabilities between states; Z represents the probability distribution of observing O in state S.

[0016] Preferably, the reward value of a single agent at time t is:

[0017]

[0018] in, Represents the agent finding a target victim; Indicates that the agent has encountered an obstacle; Represents the agent-agent mutual information obtained by the interaction between agents; Represents the cost of exercise.

[0019] Preferably, the agent-environment mutual information I(m, a t ) is calculated as follows:

[0020] I(m,a t )=H(m)-H(m|a t )

[0021]

[0022] Among them, I(m,at ) represents the mutual information between the agent and the environment, H(m) is the entropy of the current map; H(m|a t ) indicates that action a is being executed t The expected entropy after collecting sensor observations; p(m i，j ) represents the probability of currently occupying the grid map; m i，j Represents the currently occupied grid map, i represents the number of a single grid unit of the map; j represents the category of the attribute corresponding to each grid unit.

[0023] Preferably, the obtaining of the action of each agent using the QMIX algorithm comprises: selecting the action taken by the agent when the Q value is maximized as the action of the agent to be obtained;

[0024] The calculation formula for maximizing the Q value is:

[0025]

[0026] Among them, Q total (τ, u) represents the global Q value, It represents the maximum Q value when agent i takes a certain action.

[0027] In a second aspect, the present invention further proposes a multi-agent unknown environment search and rescue system based on deep reinforcement learning, the system comprising:

[0028] Markov decision model building module, used to obtain Markov decision models based on multi-agent search and rescue in unknown environments;

[0029] An agent action acquisition module is used to determine the action of each agent based on the Markov decision model using the QMIX algorithm and agent-environment mutual information;

[0030] Path planning module, for each agent to use A based on the action * The algorithm plans the optimal path from the current point to the next state target point;

[0031] The result output module is used to cyclically execute steps S2 and S3 until a preset termination condition is reached and output the multi-agent unknown environment search and rescue results.

[0032] Preferably, the Markov decision model construction module uses a sextuple <S, A, T, R, Z, O> to represent the Markov decision model;

[0033] Where: S = {S1, S2, ..., SN} represents all possible states of the agent; A = {A1, A2, ..., AN} represents all possible actions of the agent; O = {O1, O2, ..., ON} represents all possible observation values ​​of the agent; R represents the reward value; T represents a set of conditional transition probabilities between states; Z represents the probability distribution of observing O in state S.

[0034] Preferably, the reward value of a single agent at time t is:

[0035]

[0036] in, Represents the agent finding a target victim; Indicates that the agent has encountered an obstacle; Represents the agent-agent mutual information obtained by the interaction between agents; Represents the cost of exercise.

[0037] Preferably, the agent-environment mutual information I(m, a t ) is calculated as follows:

[0038] I(m,a t )=H(m)-H(m|a t )

[0039]

[0040] Among them, I(m,a t ) represents the mutual information between the agent and the environment, H(m) is the entropy of the current map; H(m|a t ) indicates that action a is being executed t The expected entropy after collecting sensor observations; p(m i，j ) represents the probability of currently occupying the grid map; m i，j Represents the currently occupied grid map, i represents the number of a single grid unit of the map; j represents the category of the attribute corresponding to each grid unit.

[0041] Preferably, the obtaining of the action of each agent using the QMIX algorithm comprises: selecting the action taken by the agent when the Q value is maximized as the action of the agent to be obtained;

[0042] The calculation formula for maximizing the Q value is:

[0043]

[0044] Among them, Q total (τ, u) represents the global Q value, It represents the maximum Q value when agent i takes a certain action.

[0045] (3) Beneficial effects

[0046] This invention provides a multi-agent unknown environment search and rescue method and system based on deep reinforcement learning. Compared with the existing technology, it has the following advantages:

[0047] 1. The present invention first obtains a Markov decision model based on multi-agent unknown environment search and rescue; then uses the QMIX algorithm based on the Markov decision model to obtain the action of each agent; each agent uses the A* algorithm to plan the optimal path from the current point to the next state target point based on the above action; the process of the QMIX algorithm determining the action and the A* algorithm planning the path is continuously repeated until the preset termination condition is reached and the multi-agent unknown environment search and rescue result is output. The present invention combines the QMIX algorithm and the A* algorithm, and uses QMIX to efficiently search for the target A*. * Providing target points for the planned path accelerates the convergence of the algorithm and improves the efficiency of multi-agent search and rescue in unknown environments.

[0048] 2. The present invention encourages agents to frequently visit interaction points by adding mutual information to the reward function by maximizing mutual information, so that the agent team can disperse the exploration and improve the search efficiency.

[0049] 3. This invention uses the QMIX algorithm to plan the entire multi-agent system. * The algorithm plans the next movement path of each agent to determine the target point, and A * When the target point is known, the algorithm can quickly plan the optimal path from the current position to the target point, combining the QMIX algorithm and A * The combination of the two algorithms complements the advantages of the two algorithms and effectively improves the search and rescue efficiency of multiple agents in unknown environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 This is a flow chart of a multi-agent unknown environment search and rescue method based on deep reinforcement learning of the present invention;

[0052] Figure 2 This is a flowchart of a multi-agent unknown environment search and rescue method based on deep reinforcement learning in an embodiment of the present invention;

[0053] Figure 3 This is a network architecture diagram based on the QMIX model in an embodiment of the present invention;

[0054] Figure 4 QMIX-A in the embodiment of the present invention * Algorithm and A * Comparison chart of algorithm experimental results. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] The embodiments of the present application provide a multi-agent unknown environment search and rescue method and system based on deep reinforcement learning, which solves the problem in the prior art that multi-agents cannot achieve efficient search and rescue in unknown environments, and realizes the purpose of multi-agent collaborative decentralized exploration, thereby improving the search and rescue efficiency of agents on unknown maps.

[0057] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0058] In order to solve the problem of A * Traditional path planning algorithms such as the QMIX algorithm, as well as the QMIX algorithm, when used alone, have the problem of multi-agent search and rescue being unable to achieve efficient multi-agent search and rescue in unknown environments. The technical solution of the present invention first represents the multi-agent search and rescue problem in unknown environments using a Markov decision model; then, based on the Markov decision model, the QMIX algorithm is used to obtain the actions of each agent; then, based on these actions, each agent uses the A* algorithm to plan the optimal path from the current point to the next state target point; when the QMIX algorithm determines the action and the A* algorithm plans the path in a continuous cycle until a preset termination condition is reached, the multi-agent search and rescue result in unknown environments is output. The present invention combines the QMIX algorithm and the A* algorithm to effectively improve the efficiency of multi-agent search and rescue in unknown environments.

[0059] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0060] In the first aspect, the present invention first proposes a multi-agent unknown environment search and rescue method based on deep reinforcement learning, see Figure 1 , the method comprising:

[0061] S1. Obtain a Markov decision model based on multi-agent search and rescue in unknown environments;

[0062] S2. Determine the action of each agent based on the Markov decision model using the QMIX algorithm and agent-environment mutual information;

[0063] S3, each agent uses A based on the action * The algorithm plans the optimal path from the current point to the next state target point;

[0064] S4. Loop through steps S2 and S3 until the preset termination condition is reached and output the multi-agent unknown environment search and rescue results.

[0065] It can be seen that this embodiment first obtains a Markov decision model based on multi-agent unknown environment search and rescue; then uses the QMIX algorithm based on the Markov decision model to obtain the action of each of the agents; each agent uses the A* algorithm to plan the optimal path from the current point to the next state target point based on the above action; the process of the QMIX algorithm determining the action and the A* algorithm planning the path is continuously repeated until the preset termination condition is reached and the multi-agent unknown environment search and rescue result is output. The present invention combines the QMIX algorithm and the A* algorithm, and uses QMIX to efficiently search for the target A*. * Providing target points for the planned path accelerates the convergence of the algorithm and improves the efficiency of multi-agent search and rescue in unknown environments.

[0066] The following is combined with Figure 1-3 , and explanations of specific steps S1-S4 are provided to describe in detail the implementation process of an embodiment of the present invention.

[0067] See also Figure 1-2 In this embodiment, the steps of a multi-agent unknown environment search and rescue method based on deep reinforcement learning are as follows:

[0068] S1. Obtain a Markov decision model based on multi-agent search and rescue in unknown environments.

[0069] A Markov decision model for multi-agent search and rescue in an unknown environment is obtained, and the states, observations, actions, and rewards in the Markov decision model are determined. In determining the rewards, this embodiment uses mutual information to encourage agents to visit interaction points more frequently.

[0070] 1) Construct a sextuple to describe the Markov decision process of multi-agent search and rescue in unknown environments.

[0071] The multi-agent search and rescue problem can be described as: a group of multi-agents conducts partially observable Markov decision process exploration in an unknown environment, avoids collisions with obstacles in the environment, and searches for the victims with the minimum time cost, thereby completing the entire search and rescue goal.

[0072] In this embodiment, a six-tuple <S, A, T, R, Z, O> is used to describe this partially observable Markov decision process. Among them, S = {S1, S2, ..., SN} represents all possible states of the agent; A = {A1, A2, ..., AN} represents all possible actions of the agent; O = {O1, O2, ..., ON} represents all possible observations of the agent; R represents the reward value, and T represents a set of conditional transition probabilities between states. The state and reward of the agent at the current time t are denoted as s t 、r t , the agent selects the current action a according to the strategy t , the environment accepts the action a made by the agent t The environment status will be updated to status s t+1 And give feedback r to the agent t+1 The agent then takes new actions according to the strategy based on the new state and reward, and iterates.

[0073] The multi-agent search and rescue environment consists of a two-dimensional occupancy grid, where each square can be blank, unknown, an obstacle, or a target (i.e., a victim). As an agent explores the environment, its reward depends on the information in the newly discovered grid. The goal of multi-agent collaborative exploration of an unknown environment is to search as much of the entire map as possible within a given timeframe, while avoiding obstacles and finding the target.

[0074] 2) Design reward value.

[0075] To enhance the interaction between agents and the external environment, promote multi-agent collaborative decentralized exploration, and improve the efficiency of search and rescue on unknown maps, a preferred approach in this embodiment is to incorporate mutual information into the reward function, using mutual information to quantify the impact of one agent on the exploration process of other agents. Maximizing the mutual information value will encourage agents to visit interaction points more frequently. Through these points, the agent team can achieve a configuration that is rarely visited by decentralized exploration. Therefore, in this embodiment, the reward of a single agent at time t can be expressed as:

[0076]

[0077] in, Represents the agent finding a target victim, It means that the agent has encountered an obstacle. Represents the agent-agent mutual information obtained by the interaction between agents, Represents the cost of exercise.

[0078] 3) Model the environment of search and rescue missions performed by homogeneous multi-agents.

[0079] Specifically, the state space, action space and observation space of the homogeneous multi-agent search and rescue mission are constructed respectively. Assume that a multi-agent system M = {1, 2, ... N} consists of N agents. The state space S global The entire global environment map is included, and a square with the length of the agent's field of view is extracted from the global map as a local map to represent the state space S. The agent's actions are limited to four possible actions A M ={up, down, left, right}. In this embodiment, the observation information includes the coordinates of the intelligent body (X rob , Y rob ), target (victim) coordinates (X goal , Y goal ), and the obstacle coordinates (X obs , Y obs ).

[0080] S2. Based on the Markov decision model, the action of each agent is determined using the QMIX algorithm and agent-environment mutual information.

[0081] The present invention combines the QMIX algorithm and A * The algorithm is used to perform multi-agent search and rescue in unknown environments. The QMIX algorithm has advantages in searching for targets, can select actions purposefully, and quickly train strategies in an end-to-end manner; while A * The algorithm is an efficient path optimization algorithm that can quickly plan the optimal path from the current position to the target point when the target point is known. By combining these two algorithms, the present invention can quickly select actions and reach the target point, which can greatly improve the search and rescue efficiency of multi-agent systems. Based on this, the strategy trained by the QMIX algorithm is first used to select the actions of each agent, providing the target point for the A* algorithm to plan the path. Specifically:

[0082] When multiple agents perform search and rescue tasks, the goal of each agent is to learn a policy function π i (a i |s i ) to maximize its own expected discounted return, which is represented by Q value, that is, the goal of each agent is to maximize its Q value. Specifically,

[0083]

[0084]

[0085]

[0086] in, is the sum of the reward function and the value function; R i (s, a) represents the reward function, γ is the discount factor, and t∈[0, T] represents the time range; Represents the agent value function, which is an expectation function that predicts the expected future cumulative discounted return at the current moment, and is related to the reward function R i Unlike the immediate reward value given to the agent, the value function has a long-term nature.

[0087] See also Figure 3 , each agent obtains the Q value of each action through agent_network, records the hidden layer of each agent, and uses it as the next input of the GRU cycle layer of agent_network. The action is selected based on the calculated Q value, and the formula for maximizing the Q value is:

[0088]

[0089] where Q total (τ, u) represents the global Q value, It represents the maximum Q value when the agent i takes a certain action, and the action of each agent is determined according to the choice of Q value.

[0090] When the mutual information value is less than or equal to zero, the reward is negative, which discourages the robot from moving to the observed area. To prevent the situation where the agent's actions may collide with obstacles or have zero information gain when taking certain actions when using the QMIX algorithm to select and determine each agent's actions based on the Q value, this embodiment uses the agent-environment mutual information of the agent's interaction with the environment to assist in determining the agent's action selection and determination.

[0091] In information theory, entropy represents the amount of information and is used to describe the uncertainty of a source. The greater the uncertainty, the greater the amount of information and the greater the entropy. Conversely, the smaller the uncertainty, the smaller the amount of information and the smaller the entropy. In this embodiment, Shannon entropy is used as a metric to represent the completeness of our map knowledge, which is described as follows:

[0092]

[0093] Among them, p(m i，j ) represents the probability of the current occupied grid map, that is, the probability of grid i when its grid attribute is j; m i，jRepresents the currently occupied grid map, i represents the number of a single grid unit on the map, j represents the category of the attribute corresponding to each grid unit, and the attributes of the grid unit include blank, unknown, obstacle, and target.

[0094] In order to define the interaction between the agent and the environment, the agent-environment mutual information I(m, a t ) is about action a t The expected information gain, that is, choosing action a t After that, the difference between the amount of grid information within the current robot's field of view and the amount of grid information before the action is selected is calculated, which can be expressed as:

[0095] I(m,a t )=H(m)-H(m|a t )

[0096] Among them, I(m,a t ) represents the agent-environment mutual information of the interaction between the agent and the environment; H(m) is the entropy of the current map, H(m|a t ) is executing action a t The expected entropy after collecting sensor observations.

[0097] S3, each agent uses A based on the action * The algorithm plans the optimal path from the current point to the next state target point.

[0098] After clarifying the actions of each agent, use A * The algorithm plans the path of each agent to the next target point, A * The cost function of the algorithm is:

[0099] F(x, y) = G(x, y) + H(x, y)

[0100] Where F(x, y) represents the cost function for the node (x, y), i.e., the estimated cost from the starting point through the current node to the target node. G(x, y) represents the shortest path from the starting point to the current node; H(x, y) represents the estimated cost of the optimal path from the current node to the target node, calculated using the following formula:

[0101] H(x, y) = |x goal -x node |+|y goal -y node |

[0102] Among them, (x node ,y node ) represents the coordinates of the current node, (x goal ,y goal) represents the coordinates of the target node.

[0103] S, S next , the observation value O of each agent, the action A, reward R, selected action u of each agent, and whether the environment search is completed are stored in the experience pool D.

[0104] When the experience pool accumulates to a certain amount, data is sampled from the experience pool as training samples, and all training samples are used as input values ​​of the neural network to continuously train the above QMIX-A * Algorithm model, batch processing to obtain Q total The whole training process is Q total Towards target Q total During the approximation process, the parameters are updated according to the following loss function:

[0105]

[0106] Among them, r represents the reward function (immediate reward function); max a′ Q total (τ′, a′, s′; θ - ) represents the value function, that is, the expected reward for the next step, which is the maximum value of all actions in the next state after the currently selected action reaches the next state; γ represents the coefficient, which determines how much of the next Q value is included in the current reward; Q total (τ, a, s; θ) represents the output value.

[0107] Update the parameters according to the loss function, select actions according to the new policy function, and continue searching the environment map.

[0108] S4. Loop through steps S2 and S3 until the preset termination condition is reached and output the multi-agent unknown environment search and rescue results.

[0109] The above steps S2 and S3 are executed cyclically, that is, the QMIX algorithm is continuously used to determine the action of each agent, and then the A * The algorithm plans the optimal path from the current point to the next state's target point. The search process terminates when the number of iterations reaches a preset limit. If the search process has terminated but the number of iterations has not reached the limit, the missing data is padded with zeros to ensure data consistency. The number of iterations can be set manually based on actual needs. The search process terminates when the calculated entropy of the map reaches 0 (assuming there are no more disaster victims on the map).

[0110] After the above process, the multi-agent will search all the target points in the unknown environment and obtain the optimal path from the starting point to the target point.

[0111] At this point, the entire process of the multi-agent unknown environment search and rescue method based on deep reinforcement learning of the present invention is completed.

[0112] In order to verify the superiority of the multi-agent unknown environment search and rescue method based on deep reinforcement learning, that is, to prove that the method of the present invention can realize the coordinated decentralized exploration of multiple agents to improve the search and rescue efficiency of agents on unknown maps, the present invention is illustrated by experimental data. Specifically,

[0113] Train a policy in a 2D maze-like environment, using 5663 maps during training. The training scene consists of two agents with a view, obstacles, and targets (victims). The targets are randomly distributed throughout the map. Both agents are initially located at the same point in the map. Obstacles are randomly distributed throughout the map.

[0114] 1) Training QMIX-A * Algorithm model.

[0115] The first step is to model the search and rescue mission environment.

[0116] This experiment provides a 360-degree field of view noiseless distance sensor for the simulated agent. Each map is 640m×480m, the sensor distance is 80m, and the step length of each agent is 40m.

[0117] The second step is to set the experimental evaluation indicators.

[0118] The following performance metrics were determined to measure algorithm performance:

[0119] Search and rescue time: We define the search and rescue time metric as the total time required for a multi-robot system to complete an exploration mission. The timer starts when at least one robot in the system begins exploration and ends when at least one robot has explored a target percentage (e.g., 99%) of the entire terrain. Exploration of the map ends only after the entropy of the map drops to zero.

[0120] Search and rescue cost: The energy consumption of performing the task can be approximated by the energy consumed by the robot movement. Therefore, this scheme defines the search and rescue cost metric as the sum of the distance traveled by each robot in the multi-robot system:

[0121]

[0122] Where n is the number of robots in the system, d i is the moving distance of robot i.

[0123] Search and rescue efficiency: From an economic perspective, search and rescue efficiency can be defined as the ratio between the inputs into operating a mission and the outputs obtained from the mission.

[0124]

[0125] Where n is the number of robots in the system, M is the total search and rescue area

[0126] For example, if the search and rescue efficiency value is 1.5, it means that all robots in the system move 1 meter each time and they discover an average of 1.5 square meters of terrain. Inspired by the benefit-cost ratio analysis in economics, it can be considered that an algorithm with a search and rescue efficiency value greater than or equal to 1 is worth using.

[0127] Map integrity: Map construction is a task closely integrated with search and rescue, so map integrity is also a major issue of concern to researchers. This scheme defines map integrity as the ratio of the total multi-robot search and rescue area M to the actual total map area P.

[0128]

[0129] Where n is the number of robots in the system and M is the total search and rescue area.

[0130] Personnel rescue rate: The multi-robot search and rescue problem considered in this scheme requires the robot system to be able to find the largest number of victims to the greatest extent possible. Therefore, the success rate of searching and rescuing victims is one of the most important indicators to measure the quality of the algorithm. This scheme defines the personnel rescue rate as the ratio of the total number of victims V rescued by the multi-robot system to the actual total number of victims H in the environment.

[0131]

[0132] The third step is to present the experimental process and results.

[0133] By comparing the QMIX-A proposed in this scheme * The algorithm and the traditional path planning algorithm A* algorithm verify that the proposed algorithm has better effectiveness.

[0134] Use the QMIX-A proposed in this scheme respectively * The algorithm and the traditional path planning algorithm A* algorithm were trained on 5663 environmental maps and compared on 30 test maps using search and rescue time, search and rescue cost, map integrity, and personnel rescue rate as evaluation indicators. Figure 4 , through observation, we can see that the QMIX-A proposed in this scheme * The algorithm has more effective performance in terms of search and rescue time, search and rescue cost, map completeness, and personnel search and rescue rate. Therefore, the algorithm proposed in this scheme has higher effectiveness than traditional algorithms, and can effectively help homogeneous multi-robot systems complete searches in unknown environments and rescue more victims.

[0135] Example 2:

[0136] In a second aspect, the present invention further provides a multi-agent unknown environment search and rescue system based on deep reinforcement learning, the system comprising:

[0137] Markov decision model building module, used to obtain Markov decision models based on multi-agent search and rescue in unknown environments;

[0138] An agent action acquisition module is used to determine the action of each agent based on the Markov decision model, the QMIX algorithm and the agent-environment mutual information;

[0139] Path planning module, for each agent to use A based on the action * The algorithm plans the optimal path from the current point to the next state target point;

[0140] The result output module is used to cyclically execute steps S2 and S3 until a preset termination condition is reached and output the multi-agent unknown environment search and rescue results.

[0141] Optionally, the Markov decision model construction module uses a sextuple <S, A, T, R, Z, O> to represent the Markov decision model;

[0142] Where: S = {S1, S2, ..., SN} represents all possible states of the agent; A = {A1, A2, ..., AN} represents all possible actions of the agent; O = {O1, O2, ..., ON} represents all possible observation values ​​of the agent; R represents the reward value; T represents a set of conditional transition probabilities between states; Z represents the probability distribution of observing O in state S.

[0143] Optionally, the reward value for a single agent at time t is:

[0144]

[0145] in, Represents the agent finding a target victim; Indicates that the agent has encountered an obstacle; Represents the agent-agent mutual information obtained by the interaction between agents; Represents the cost of exercise.

[0146] Optionally, the agent-environment mutual information I(m, a t ) is calculated as follows:

[0147] I(m,a t )=H(m)-H(m|a t )

[0148]

[0149] Among them, I(m,a t ) represents the mutual information between the agent and the environment, H(m) is the entropy of the current map; H(m|a t ) indicates that action a is being executed t The expected entropy after collecting sensor observations; p(m i，j ) represents the probability of currently occupying the grid map; m i，j Represents the currently occupied grid map, i represents the number of a single grid unit of the map; j represents the category of the attribute corresponding to each grid unit.

[0150] Optionally, the obtaining of the action of each agent by using the QMIX algorithm includes: selecting the action taken by the agent when the Q value is maximized as the action of the agent to be obtained;

[0151] The calculation formula for maximizing the Q value is:

[0152]

[0153] Among them, Q total (τ, u) represents the global Q value, It represents the maximum Q value when agent i takes a certain action.

[0154] It can be understood that the multi-agent unknown environment search and rescue system based on deep reinforcement learning provided by the embodiment of the present invention corresponds to the above-mentioned multi-agent unknown environment search and rescue method based on deep reinforcement learning. The explanations, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the multi-agent unknown environment search and rescue method based on deep reinforcement learning, and will not be repeated here.

[0155] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0156] 1. The present invention first obtains a Markov decision model based on multi-agent unknown environment search and rescue; then uses the QMIX algorithm based on the Markov decision model to obtain the action of each agent; each agent uses the A* algorithm to plan the optimal path from the current point to the next state target point based on the above action; the process of the QMIX algorithm determining the action and the A* algorithm planning the path is continuously repeated until the preset termination condition is reached and the multi-agent unknown environment search and rescue result is output. The present invention combines the QMIX algorithm and the A* algorithm, and uses QMIX to efficiently search for the target A*. * Providing target points for the planned path accelerates the convergence of the algorithm and improves the efficiency of multi-agent search and rescue in unknown environments.

[0157] 2. The present invention encourages agents to frequently visit interaction points by adding mutual information to the reward function by maximizing mutual information, so that the agent team can disperse the exploration and improve the search efficiency.

[0158] 3. This invention uses the QMIX algorithm to plan the entire multi-agent system. * The algorithm plans the next movement path of each agent to determine the target point, and A * When the target point is known, the algorithm can quickly plan the optimal path from the current position to the target point, combining the QMIX algorithm and A * The combination of the two algorithms complements the advantages of the two algorithms and effectively improves the search and rescue efficiency of multiple agents in unknown environments.

[0159] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0160] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-agent unknown environment search and rescue method based on deep reinforcement learning, characterized in that: The method comprises: S1. Obtain a Markov decision model based on multi-agent search and rescue in unknown environments; S2. Determine the action of each agent based on the Markov decision model using the QMIX algorithm and agent-environment mutual information; S3, each agent uses A based on the action * The algorithm plans the optimal path from the current point to the next state target point; S4, looping through steps S2 and S3 until a preset termination condition is reached and outputting a multi-agent unknown environment search and rescue result; The method of determining the action of each agent by using the QMIX algorithm and agent-environment mutual information includes: When multiple agents perform search and rescue tasks, the goal of each agent is to learn a policy function π i (a i |s i ) to maximize its own expected discounted return, which is expressed as Q value, that is, the goal of each agent is to maximize its Q value; Determine the action of each agent based on the selection of Q value, and provide the target point for path planning for the A* algorithm; When the value of the agent-environment mutual information is less than or equal to zero, the reward obtained is negative, which means that the robot is not encouraged to go to the observed area.

2. The method according to claim 1, wherein Use sextuple<S,A,T,R,Z,O> To represent the Markov decision model; Where: S = {S1, S2, ..., SN} represents all possible states of the agent; A = {A1, A2, ..., AN} represents all possible actions of the agent; O = {O1, O2, ..., ON} represents all possible observation values ​​of the agent; R represents the reward value; T represents a set of conditional transition probabilities between states; Z represents the probability distribution of observing O in state S.

3. The method according to claim 2, wherein The reward value of a single agent at time t is: in, Represents the agent finding a target victim; Indicates that the agent has encountered an obstacle; Represents the agent-agent mutual information obtained by the interaction between agents; Represents the cost of exercise.

4. The method according to claim 1, wherein The agent-environment mutual information I(m,a t ) is calculated as follows: I(m,a t )=H(m)-H(m∣a t ) Among them, I(m,a t ) represents the mutual information between the agent and the environment, H(m) is the entropy of the current map; H(m|a t ) indicates that action a is being executed t The expected entropy after collecting sensor observations; p(m i,j ) represents the probability of currently occupying the grid map; m i,j Represents the currently occupied grid map, i represents the number of a single grid unit of the map; j represents the category of the attribute corresponding to each grid unit.

5. The method according to claim 1, wherein The method of obtaining the action of each agent by using the QMIN algorithm includes: selecting the action taken by the agent when the Q value is maximized as the action of the agent to be obtained; The calculation formula for maximizing the Q value is: Among them, Q total (τ,u) represents the global Q value, It represents the maximum Q value when agent i takes a certain action.

6. A multi-agent unknown environment search and rescue system based on deep reinforcement learning, characterized by: The system comprises: The Markov decision model building module is used to execute S1 and obtain a Markov decision model based on multi-agent unknown environment search and rescue; An agent action acquisition module is used to execute S2, determine the action of each agent based on the Markov decision model using the QMIX algorithm and agent-environment mutual information; Path planning module, used to execute S3, each agent uses A based on the action * The algorithm plans the optimal path from the current point to the next state target point; A result output module is used to execute S4, and then loop through steps S2 and S3 until a preset termination condition is reached and output the multi-agent unknown environment search and rescue result; The method of determining the action of each agent by using the QMIX algorithm and agent-environment mutual information includes: When multiple agents perform search and rescue tasks, the goal of each agent is to learn a policy function π i (a i |s i ) to maximize its own expected discounted return, which is expressed as Q value, that is, the goal of each agent is to maximize its Q value; Determine the action of each agent based on the selection of Q value, and provide the target point for path planning for the A* algorithm; When the value of the agent-environment mutual information is less than or equal to zero, the reward obtained is negative, which means that the robot is not encouraged to go to the observed area.

7. The system according to claim 6, wherein: The Markov decision model building module uses six tuples<S,A,T,R,Z,O> To represent the Markov decision model; Where: S = {S1, S2, ..., SN} represents all possible states of the agent; A = {A1, A2, ..., AN} represents all possible actions of the agent; O = {O1, O2, ..., ON} represents all possible observation values ​​of the agent; R represents the reward value; T represents a set of conditional transition probabilities between states; Z represents the probability distribution of observing O in state S.

8. The system according to claim 7, wherein: The reward value of a single agent at time t is: in, Represents the agent finding a target victim; Indicates that the agent has encountered an obstacle; Represents the agent-agent mutual information obtained by the interaction between agents; Represents the cost of exercise.

9. The system according to claim 6, wherein: The agent-environment mutual information I(m,a t ) is calculated as follows: I(m,a t )=h(m)-H(m∣a t ) Among them, I(m,a t ) represents the mutual information between the agent and the environment, H(m) is the entropy of the current map; H(m|a t ) indicates that action a is being executed t The expected entropy after collecting sensor observations; p(m i,j ) represents the probability of currently occupying the grid map; m i,j Represents the currently occupied grid map, i represents the number of a single grid unit of the map; j represents the category of the attribute corresponding to each grid unit.

10. The system according to claim 6, wherein: The method of obtaining the action of each agent by using the QMIX algorithm includes: selecting the action taken by the agent when the Q value is maximized as the action of the agent to be obtained; The calculation formula for maximizing the Q value is: Among them, Q total (τ,u) represents the global Q value, It represents the maximum Q value when agent i takes a certain action.

Images