Optimization method and system for heterogeneous multi-machine intelligent systems to collaboratively perform search and rescue tasks

Abstract

The present invention provides an optimization method and system for a heterogeneous multi-machine intelligent system to collaboratively perform search and rescue tasks, and relates to the fields of artificial intelligence and emergency search and rescue technology. The present invention models the search and rescue task environment based on the search and rescue characteristics of multiple heterogeneous machine intelligent systems, and determines the action space and state space of each heterogeneous machine intelligent system based on the modeled search and rescue task environment; then establishes a reward system based on the action space and state space of each heterogeneous machine intelligent system; finally, constructs a collaborative search and rescue task model based on the above-mentioned action space, state space, and reward system, and uses a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue task model, and uses the trained model to guide the execution of the search and rescue task. Compared with the existing technology, the present invention has higher coordination and execution efficiency in performing search and rescue tasks in a complex task environment by multiple heterogeneous intelligent agents, and higher stability and accuracy in the selection of search and rescue strategies.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and emergency search and rescue technology, and in particular to an optimization method and system for a heterogeneous multi-machine intelligent system to collaboratively perform search and rescue tasks. Background Art

[0002] Compared to single-machine intelligent systems, collaborative multi-machine intelligent systems offer advantages in robustness to hardware failures, adaptability to environmental changes, and cost-effectiveness. In a collaborative multi-machine intelligent system, each individual intelligent system is assigned different but complementary tasks. This reduces the need for operators to independently control each intelligent system, thereby improving task execution efficiency.

[0003] With the widespread development and practical application of various machine learning algorithms, many scholars have begun to study the use of machine learning algorithms to control multi-machine intelligent systems, making machine intelligent systems more intelligent and autonomous.

[0004] However, when multiple intelligent machine systems perform search and rescue missions in unknown environments, the complex and changing terrain often requires a multi-machine intelligent system search and rescue team composed of heterogeneous rescue machine intelligent systems of varying sizes, motion performance, and sensing capabilities. Therefore, allocating tasks to multiple intelligent machine systems is a complex process. However, existing methods often lack task environment modeling and specific implementation methods for multiple heterogeneous intelligent systems, which can lead to problems with coordination and stability when multiple heterogeneous intelligent systems perform tasks in complex environments. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides an optimization method and system for a heterogeneous multi-machine intelligent system to collaboratively perform search and rescue tasks, which solves the problem of insufficient coordination and stability when multiple heterogeneous intelligent systems perform tasks in the existing technology.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention first proposes an optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform a search and rescue mission, the method comprising:

[0010] Modeling a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determining an action space and a state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment;

[0011] Establishing a reward system based on the action space and state space of each of the heterogeneous machine intelligence systems;

[0012] A collaborative search and rescue mission model is constructed based on the action space, state space, and reward system, and a deep reinforcement learning algorithm is used to train and optimize the collaborative search and rescue mission model.

[0013] Preferably, modeling the search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determining the action space and state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment includes:

[0014] S11. Modeling the search and rescue mission environment into a simulated maze environment map using a grid method;

[0015] S12. Based on the different search and rescue characteristics of the multiple heterogeneous machine intelligent systems, a local simulated maze environment submap of the current location of each heterogeneous machine intelligent system is captured, and all the local simulated maze environment submaps are used as inputs to the neural network, thereby gray-scaling the simulated maze environment map to obtain a two-dimensional grid maze map;

[0016] S13, and determining the action space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment;

[0017] S14. Determine the state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment.

[0018] Preferably, the reward system includes:

[0019] The reward function of any agent i in a multi-machine intelligent system at time t is:

[0020]

[0021] in: It means that the machine agent i finds a victim at time t; Indicates that the machine agent i encounters an obstacle; represents the movement cost of machine agent i; Represents the new map information obtained by machine agent i; g represents the reward for finding the victims; c represents the reward for encountering obstacles; m represents the movement cost; e represents the new map information obtained.

[0022] Preferably, the step of constructing a collaborative search and rescue mission model based on the action space, state space, and reward system, and training and optimizing the collaborative search and rescue mission model using a deep reinforcement learning algorithm includes:

[0023] Based on the action space, state space, and reward system, combined with the multi-machine intelligent system set and time, a collaborative search and rescue mission model is constructed. The collaborative search and rescue mission model uses a Markov decision process hexagram describing the multi-machine intelligent system.<I,S,A,T,r,γ> Represents, where I represents the set of multi-machine intelligent systems; S represents the state space; A represents the action space; T represents time; r represents the reward value in the reward system; γ represents the discount rate;

[0024] The action-value function in the complex state space is calculated by a deep reinforcement learning algorithm to approximate the optimal strategy to train and optimize the collaborative search and rescue mission model; the action-value function is expressed as follows:

[0025] Q(s i ,a i )=Q(s i ,a i )+α[r i +γmaxQ(s′,a′)-Q(s i ,a i )]

[0026] Among them, Q(s i ,a i ) is the machine intelligence system in state s i Next take action a i Q value; s i Indicates the current state; a i represents the current action; α represents the learning rate; a' represents the next action; s' represents the state at the next moment; γ represents the discount rate; r i Represents the reward value; i represents the serial number of the machine agent.

[0027] Preferably, the method further comprises: evaluating the performance of the collaborative search and rescue mission model using a collaborative search and rescue mission model performance evaluation system.

[0028] In a second aspect, the present invention further proposes an optimization system for heterogeneous multi-machine intelligent systems to collaboratively perform search and rescue tasks, the system comprising:

[0029] An action and state space acquisition module is used to model a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determine the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment;

[0030] a reward system acquisition module, configured to establish a reward system based on the action space and state space of each of the heterogeneous machine intelligence systems;

[0031] A model building and optimization module is used to build a collaborative search and rescue mission model based on the action space, state space, and reward system, and to train and optimize the collaborative search and rescue mission model using a deep reinforcement learning algorithm.

[0032] Preferably, the action and state space acquisition module models the search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determines the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment, including:

[0033] S11. Modeling the search and rescue mission environment into a simulated maze environment map using a grid method;

[0034] S12. Based on the different search and rescue characteristics of the multiple heterogeneous machine intelligent systems, a local simulated maze environment submap of the current location of each heterogeneous machine intelligent system is captured, and all the local simulated maze environment submaps are used as inputs to the neural network, thereby gray-scaling the simulated maze environment map to obtain a two-dimensional grid maze map;

[0035] S13, and determining the action space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment;

[0036] S14. Determine the state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment.

[0037] Preferably, the reward system includes:

[0038] The reward function of any agent i in a multi-machine intelligent system at time t is:

[0039]

[0040] in: It means that the machine agent i finds a victim at time t; Indicates that the machine agent i encounters an obstacle; represents the movement cost of machine agent i; Represents the new map information obtained by machine agent i; g represents the reward for finding the victims; c represents the reward for encountering obstacles; m represents the movement cost; e represents the new map information obtained.

[0041] Preferably, the model construction and optimization module constructs a collaborative search and rescue mission model based on the action space, state space, and reward system, and uses a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model, including:

[0042] Based on the action space, state space, and reward system, combined with the multi-machine intelligent system set and time, a collaborative search and rescue mission model is constructed. The collaborative search and rescue mission model uses a Markov decision process hexagram describing the multi-machine intelligent system.<I,S,A,T,r,γ> Represents; where I represents the set of multi-machine intelligent systems; S represents the state space; A represents the action space; T represents time; r represents the reward value in the reward system; γ represents the discount rate;

[0043] The action-value function in the complex state space is calculated by a deep reinforcement learning algorithm to approximate the optimal strategy to train and optimize the collaborative search and rescue mission model; the action-value function is expressed as follows:

[0044] Q(s i ,a i )=Q(s i ,a i )+α[r i +γmaxQ(s',a')-Q(s i ,a i )]

[0045] Among them, Q(s i ,a i ) is the machine intelligence system in state s i Next take action a i Q value; s i Indicates the current state; a i represents the current action; α represents the learning rate; a' represents the next action; s' represents the state at the next moment; γ represents the discount rate; r i Represents the reward value; i represents the serial number of the machine agent.

[0046] Preferably, the system further comprises: a model performance evaluation module for evaluating the performance of the collaborative search and rescue mission model using the collaborative search and rescue mission model performance evaluation system.

[0047] (3) Beneficial effects

[0048] The present invention provides an optimization method and system for a heterogeneous multi-machine intelligent system to collaboratively perform search and rescue missions. Compared with the existing technology, it has the following advantages:

[0049] 1. The present invention models the search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determines the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment; then establishes a reward system based on the action space and state space of each heterogeneous machine intelligence system; finally, based on the above action space, state space, and reward system, constructs a collaborative search and rescue mission model, and uses a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model, and uses the trained model to guide the execution of the search and rescue mission. The present invention effectively solves the problem of insufficient coordination and execution efficiency of multiple heterogeneous intelligent agents performing search and rescue tasks in a complex mission environment, and achieves the purpose of improving the stability and accuracy of search and rescue strategy selection.

[0050] 2. The present invention considers the different search ranges (sensor ranges), search and rescue capabilities, and motion capabilities of each machine intelligence system and sets different action spaces and observation values ​​for different intelligent agents, models the task environment and strategy reward system, and uses the deep reinforcement learning algorithm (DQN) to learn the environment image input. The Double DQN idea is used to solve the instability problem of the DQN algorithm using neural networks to approximate the value function and the low efficiency problem of the traditional Q-learning algorithm, update the value function in a larger state space, and find the optimal search and rescue strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 The present invention is a flowchart of an optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform a search and rescue mission;

[0053] Figure 2 This is a diagram illustrating an embodiment of an optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform a search and rescue mission in an embodiment of the present invention;

[0054] Figure 3 A simulated maze environment map in an embodiment of the present invention;

[0055] Figure 4 This is a pseudo code diagram of the deep reinforcement learning algorithm in an embodiment of the present invention;

[0056] Figure 5 This is a flowchart of an optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform a search and rescue mission when performing model performance evaluation in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] The embodiments of the present application provide an optimization method and system for a heterogeneous multi-machine intelligent system to collaboratively perform search and rescue tasks, thereby solving the problem of insufficient coordination and stability in the execution of tasks by multiple heterogeneous intelligent systems in the prior art, and achieving the purpose of improving the stability and accuracy of search and rescue strategy selection.

[0059] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0060] An important implementation scenario in the field of multi-machine intelligent system search and rescue is that the machine intelligent system is deployed in a completely unknown environment. The search and rescue environment usually has complex and changing terrain features. Each machine intelligent system needs to have different capabilities, such as rescue capabilities, field of view, movement speed, etc. These machine intelligent systems work together to complete the same search and rescue mission. Therefore, it is necessary to consider that each machine intelligent system has different search ranges (sensor ranges), search and rescue capabilities, and movement capabilities; and the process of assigning tasks to different machine intelligent systems is very complicated, and the state space is large. Heterogeneous machine intelligent systems need to avoid collisions with obstacles in the environment through collaborative planning, and search for victims at the lowest time cost, thereby completing the entire search and rescue goal. Traditional modeling methods and Q-leaning algorithms for homogeneous intelligent systems may be difficult to apply. In order to solve this problem, this application proposes the following technical solutions:

[0061] Example 1:

[0062] In the first aspect, the present invention first proposes an optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform search and rescue tasks, see Figure 1-2 , the method comprising:

[0063] S1. Modeling a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determining an action space and a state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment;

[0064] S2. Establishing a reward system based on the action space and state space of each of the heterogeneous machine intelligence systems;

[0065] S3. Construct a collaborative search and rescue mission model based on the action space, state space, and reward system, and use a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model.

[0066] As can be seen, this embodiment models the search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems. Based on the modeled search and rescue mission environment, the action space and state space of each heterogeneous machine intelligence system are determined. A reward system is then established based on the action space and state space of each heterogeneous machine intelligence system. Finally, a collaborative search and rescue mission model is constructed based on these action spaces, state spaces, and reward systems. A deep reinforcement learning algorithm is used to train and optimize the collaborative search and rescue mission model, and the trained model is used to guide the execution of the search and rescue mission. This embodiment effectively addresses the issues of insufficient coordination and execution efficiency among multiple heterogeneous agents performing search and rescue missions in complex mission environments.

[0067] The following is combined with Figure 1-5 , and explanations of specific steps S1-S4 are provided to describe in detail the implementation process of an embodiment of the present invention.

[0068] This embodiment proposes an optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform a search and rescue mission. For a heterogeneous multi-machine intelligent system to perform a search and rescue mission, this embodiment uses a six-tuple<I,S,A,T,r,γ> To describe the Markov decision process of a multi-machine intelligent system, S = {S1, S2, …, SN} represents all possible states of the machine intelligent system, I = {1, 2, …, N} represents the set of multi-machine intelligent systems, and A, T, r, and γ represent the action, time, reward value, and discount rate, respectively. The environment consists of a two-dimensional occupancy grid, where each square can be in the following states: blank, unknown, obstacle, or target (victim) cell. The specific steps are as follows:

[0069] S1. Modeling a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligent systems, and determining an action space and a state space of each of the heterogeneous machine intelligent systems based on the modeled search and rescue mission environment.

[0070] The search and rescue mission environment is modeled based on multiple heterogeneous machine intelligence systems including search range (sensor range), search and rescue capability and motion capability.

[0071] S11. Use the grid method to model the search and rescue mission environment into a simulated maze environment map.

[0072] In this example, a ground-based mobile machine intelligence system and an aerial drone were selected as the heterogeneous machine intelligence system objects to be controlled in the actual research. A 360-degree field-of-view noiseless distance sensor was provided for the simulated machine intelligence system. The occupancy grid of this 360-degree field-of-view noiseless distance sensor is a deterministic frequency occupancy grid, where p = 1 represents a searched cell, p = 0 represents a disaster-affected cell, and p = 0.5 represents an unknown cell.

[0073] In addition, we set the search and rescue mission environment area of ​​multiple heterogeneous machine intelligence systems as a square area with two entrances and exits. The area is 50m long and 50m wide. The entire search and rescue mission environment area is similar to a simulated maze environment map. The specific information of the simulated maze environment map is set as follows: Figure 3 shown.

[0074] S12. Based on the different search and rescue characteristics of multiple heterogeneous machine intelligent systems, capture the local simulated maze environment sub-map of the current location of each heterogeneous machine intelligent system, and use all the local simulated maze environment sub-maps as input to the neural network, thereby gray-scaling the simulated maze environment map to obtain a two-dimensional grid maze map.

[0075] Because the search and rescue mission environment often features complex and changing terrain, a multi-machine intelligent system search and rescue team must be composed of heterogeneous rescue machine intelligent systems of varying sizes, motion performance, and sensor capabilities. For example, a large rescue machine intelligent system is needed to navigate through cluttered rubble, or a small machine intelligent system is required to navigate through confined spaces to locate victims, a common occurrence during search and rescue missions. Therefore, the search and rescue characteristics of multiple heterogeneous machine intelligent systems vary. Therefore, in practice, the search and rescue mission environment must be modeled based on the search and rescue characteristics of the heterogeneous machine intelligent systems, such as their search range (sensor range), search and rescue capabilities, and motion capabilities.

[0076] In this example, we assume that the drone's sensor range is 10m, the step size of each machine intelligence system is 5m, and the sensor range of the ground machine intelligence system is 5m. We use the 5m local submap captured at the machine intelligence system's current position as the input state of the neural network, convert the local simulated maze environment map into a grayscale image, and thus construct a two-dimensional grid maze map. This two-dimensional grid maze map is used as the image input of the reinforcement learning (DQN) algorithm model. In this two-dimensional grid maze map, the grayscale values ​​255, 1, and 127 represent free space, occupied space, and undeveloped space, respectively.

[0077] S13, and determining the action space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment.

[0078] In the two-dimensional space corresponding to the two-dimensional grid maze map, the action space has four types: up, down, left, and right. Therefore, the action space of the ground mobile machine intelligent system is discretized into the above four types, namely A = {up, down, left, right}. In order to meet the autocorrelation constraint conditions, the movement speed of the machine intelligent system is set to the maximum movable speed v1, and the single movement distance d = v 1max ; Aerial machine intelligence systems (such as aerial drones) are more flexible, have complex movement directions, and are faster than ground machine intelligence systems. Therefore, the movement angle of the machine intelligence system is set to θ, the movement speed is v2, and the movement distance d = v 2max , the action space is:

[0079]

[0080] S14. Determine the state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment.

[0081] In the modeled search and rescue mission environment, a Cartesian coordinate system is established. Assume that the position coordinates of the agent i in the environment in the machine intelligence system are (x rob ,y rob ), the location coordinates of the disaster victims are (x goal ,y goal ), the position coordinates of the obstacle are (x obs ,y obs The machine intelligence system uses the obtained environmental information to construct a map and uses the map information as the state information of the machine intelligence system in the environment. This can express any state information of heterogeneous machine intelligence systems in the environment and maintains continuity. The specific observation information coordinate settings of the machine intelligence system in the environment are shown in Table 1 below.

[0082] Table 1 Observation information table of machine intelligence system

[0083]

[0084] S2. Establish a reward system based on the action space and state space of each of the heterogeneous machine intelligence systems.

[0085] Different reward values ​​are set for the search and rescue behaviors of different heterogeneous machine intelligence systems, and a reward system for deep reinforcement learning algorithms is established. Specifically, it includes:

[0086] The goal of a heterogeneous multi-machine intelligent system to collaboratively explore an unknown environment within a given timeframe is to search the entire map as much as possible, avoid obstacles, and find the target. The reward function for agent i at time t in a multi-machine intelligent system can be expressed as:

[0087]

[0088] in:

[0089] Indicates that machine agent i finds a victim;

[0090] Indicates that the machine agent i encounters an obstacle;

[0091] represents the movement cost of machine agent i;

[0092] Represents the new map information obtained by machine agent i.

[0093] The specific reward value settings are shown in Table 2 below.

[0094] Table 2 Reward value table

[0095]

[0096] Movement cost r m In the example, m is used to represent the frequency of a single intelligent system visiting the maze square with coordinates (i, j). m is set to:

[0097]

[0098] S3. Construct a collaborative search and rescue mission model based on the action space, state space, and reward system, and use a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model.

[0099] Based on the action space, state space, and reward system obtained above, combined with the multi-machine intelligent system set and time, a Markov decision process six-tuple is constructed to describe the multi-machine intelligent system.<I,S,A,T,r,γ> , that is, to build a collaborative search and rescue mission model, then use the deep reinforcement learning algorithm - Deep Q-Network (DQN) to interpret the graphical input representation from the search and rescue mission environment, calculate the action value function Q(s,a) in the complex state space, and approximate the optimal policy through continuous training. This deep reinforcement learning-based search and rescue algorithm is evaluated and the model performance is explored. Among them, the deep reinforcement learning algorithm - Deep Q-Network (DQN) is used to calculate the action value function Q in the complex state space and approximate the optimal policy. The model setup process is as follows:

[0100] The convolutional neural network (CNN) module reads the graphical representation of the input state from the search and rescue mission environment, and interprets the graphical representation of the input state from the environment by constructing a deep reinforcement learning policy network Deep Q-Network (DQN). The algorithm is continuously trained to approach the optimal policy and calculate the Q value. The specific process is as follows: Figure 2 shown.

[0101] The DQN algorithm is an improvement based on the Q-leaning algorithm. It can be regarded as a policy network τ with a parameter β. It can calculate the value function Q(s,a) in a larger state and action space. The value function update formula is:

[0102] Q(s i ,a i )=Q(s i ,a i )+α[r i +γmaxQ(s′,a′)-Q(s i ,a i )]

[0103] Among them, Q(s i ,a i ) is the machine intelligence system in state s i Next take action a i The Q value, s represents the current state; a represents the current action; α represents the learning rate; a′ represents the next action; s′ represents the state at the next moment; γ represents the discount rate; r i Represents the reward value.

[0104] The experience replay idea is used to train the collaborative search and rescue mission model to approach the optimal strategy:

[0105] (1) The intelligent system randomly selects actions from the action space by exploring the environment. The exploration strategy can explore the environment well, that is, sample state-action pairs.

[0106] (2) After the experience (memory) pool accumulates to a certain level, samples are randomly selected for training and fitted using a neural network.

[0107] The DQN algorithm typically uses the Bellman equation to minimize the loss function. However, to reduce overestimation of Q values ​​during training and the instability of the neural network's approximation value function, this paper adopts the Double DQN algorithm idea to separate the selection of greedy behavior from the behavior evaluation. That is, the maximum operator in the Q-value function update formula is decoupled into two different operators, expressed as a loss function:

[0108] L DDQN (β)=E[r+γQ(s',maxQ(s',a'|β))-Q(s,a|β) 2 ]

[0109] Among them, a' represents the next action; s' represents the state at the next moment.

[0110] The pseudo code setting of the deep reinforcement learning algorithm (DQN) to realize the search and rescue mission of multiple heterogeneous intelligent systems is as follows: Figure 4 As shown, the implementation process includes:

[0111] (1) Heterogeneous machine intelligence systems explore unknown environments, build experience buffers, and perform sampling;

[0112] (2) Using the convolutional neural network (CNN) module in the DQN policy network to interpret the graphical representation of the search task environment read by the sensors of the heterogeneous machine intelligence system;

[0113] (3) At each time t, the machine intelligence system randomly selects a strategy π θ i:O i ×A i →[0,1] select action a∈A i , calculate the next state through the state transition function T(s'|s,a);

[0114] (4) Heterogeneous machine intelligence systems are based on the reward function r i :S×A i →R gets the reward and gets a new observation value O related to its own state in the next state i :S→O i .

[0115] (5) Based on the joint strategy of different machine intelligence systems π=π1,π2,…,πN, a joint action-value function is generated. and a value function V ext,π (s)=max a Q ext,π (s,a).

[0116] (6) Based on the DQN algorithm loss function, gradient descent is performed to update the value function. The goal of each machine intelligence system is to learn a strategy πi(ai|si) and use the joint strategy to maximize the team expectation.

[0117] Only after the collaborative search and rescue mission model training achieves the expected results can the trained collaborative search and rescue mission model be used to guide the heterogeneous multi-machine intelligent system to collaboratively perform the search and rescue mission. In this embodiment, in order to judge the performance of the deep reinforcement learning-based collaborative search and rescue mission model constructed in this embodiment, the optimization method for the collaborative search and rescue mission of the heterogeneous multi-machine intelligent system further includes:

[0118] S4. Use the collaborative search and rescue mission model performance evaluation system to evaluate the performance of the collaborative search and rescue mission model.

[0119] See also Figure 5In this embodiment, a preferred approach is that the performance evaluation system of the heterogeneous multi-machine intelligent system collaboratively executing the search and rescue mission model includes:

[0120] (1) Search and rescue time: The timer starts when at least one machine intelligence system in the system begins to explore, and ends when at least one machine intelligence system has explored the target percentage (e.g., 99%) of the entire terrain;

[0121] (2) Search and rescue cost: During the search and rescue process, energy consumption is the only factor affected by the search and rescue strategy, and it can be approximated as the energy consumed by the movement of the machine intelligence system. Therefore, the search and rescue cost is defined as the sum of the distances traveled by each machine intelligence system in the multi-machine intelligence system:

[0122]

[0123] Where n represents the number of machine intelligence systems in the system, d i is the moving distance of the machine intelligence system i.

[0124] (3) Search and rescue efficiency: In the case of multi-machine intelligent system search and rescue, the search and rescue cost can be regarded as the input and the searched area as the output. The search and rescue efficiency can be defined as the ratio between the input and output of the operation task:

[0125]

[0126] Where n represents the number of machine intelligence systems in the system, and M is the total search and rescue area.

[0127] (4) Personnel rescue rate: the ratio of the number of disaster victims rescued by the heterogeneous multi-machine intelligent system to the total number of disaster victims in the environment:

[0128]

[0129] Where V represents the number of disaster victims rescued, and H represents the total number of disaster victims.

[0130] After the performance of the collaborative search and rescue mission model is evaluated by the collaborative search and rescue mission model performance evaluation system, if the performance of the collaborative search and rescue mission model meets the expected requirements, the trained collaborative search and rescue mission model can be used to guide the heterogeneous multi-machine intelligent system to collaboratively perform the search and rescue mission. Otherwise, continue to repeat the above model training and performance evaluation steps until the model performance reaches the expected effect.

[0131] At this point, the entire process of the optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform a search and rescue mission in this embodiment is completed.

[0132] Example 2:

[0133] In a second aspect, the present invention further provides an optimization system for heterogeneous multi-machine intelligent systems to collaboratively perform search and rescue tasks, the system comprising:

[0134] An action and state space acquisition module is used to model a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determine the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment;

[0135] a reward system acquisition module, configured to establish a reward system based on the action space and state space of each of the heterogeneous machine intelligence systems;

[0136] A model building and optimization module is used to build a collaborative search and rescue mission model based on the action space, state space, and reward system, and to train and optimize the collaborative search and rescue mission model using a deep reinforcement learning algorithm.

[0137] Optionally, the action and state space acquisition module models the search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determines the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment, including:

[0138] S11. Modeling the search and rescue mission environment into a simulated maze environment map using a grid method;

[0139] S12. Based on the different search and rescue characteristics of the multiple heterogeneous machine intelligent systems, a local simulated maze environment submap of the current location of each heterogeneous machine intelligent system is captured, and all the local simulated maze environment submaps are used as inputs to the neural network, thereby gray-scaling the simulated maze environment map to obtain a two-dimensional grid maze map;

[0140] S13, and determining the action space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment;

[0141] S14. Determine the state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment.

[0142] Optionally, the reward system includes:

[0143] The reward function of any agent i in a multi-machine intelligent system at time t is:

[0144]

[0145] in: It means that the machine agent i finds a victim at time t; Indicates that the machine agent i encounters an obstacle; represents the movement cost of machine agent i; Represents the new map information obtained by machine agent i; g represents the reward for finding the victims; c represents the reward for encountering obstacles; m represents the movement cost; e represents the new map information obtained.

[0146] Optionally, the model construction and optimization module constructs a collaborative search and rescue mission model based on the action space, state space, and reward system, and uses a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model, including:

[0147] Based on the action space, state space, and reward system, combined with the multi-machine intelligent system set and time, a collaborative search and rescue mission model is constructed. The collaborative search and rescue mission model uses a Markov decision process hexagram describing the multi-machine intelligent system.<I,S,A,T,r,γ> Represents; where I represents the set of multi-machine intelligent systems; S represents the state space; A represents the action space; T represents time; r represents the reward value in the reward system; γ represents the discount rate;

[0148] The action-value function in the complex state space is calculated by a deep reinforcement learning algorithm to approximate the optimal strategy to train and optimize the collaborative search and rescue mission model; the action-value function is expressed as follows:

[0149] Q(s i ,a i )=Q(s i ,a i )+α[r i +γmaxQ(s′,a′)-Q(s i ,a i )]

[0150] Among them, Q(s i ,a i ) is the machine intelligence system in state s i Next take action a i Q value; s i Indicates the current state; a i represents the current action; α represents the learning rate; a' represents the next action; s' represents the state at the next moment; γ represents the discount rate; r i Represents the reward value; i represents the serial number of the machine agent.

[0151] Optionally, the system further includes: a model performance evaluation module, configured to evaluate the performance of the collaborative search and rescue mission model using a collaborative search and rescue mission model performance evaluation system.

[0152] It is understandable that the optimization system for heterogeneous multi-machine intelligent systems to collaboratively perform search and rescue tasks provided in the embodiments of the present invention corresponds to the optimization method for heterogeneous multi-machine intelligent systems to collaboratively perform search and rescue tasks mentioned above. The explanations, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the optimization method for heterogeneous multi-machine intelligent systems to collaboratively perform search and rescue tasks, and will not be repeated here.

[0153] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0154] 1. The present invention models the search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determines the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment; then establishes a reward system based on the action space and state space of each heterogeneous machine intelligence system; finally, based on the above action space, state space, and reward system, constructs a collaborative search and rescue mission model, and uses a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model, and uses the trained model to guide the execution of the search and rescue mission. The present invention effectively solves the problem of insufficient coordination and execution efficiency of multiple heterogeneous intelligent agents performing search and rescue tasks in a complex mission environment, and achieves the purpose of improving the stability and accuracy of search and rescue strategy selection.

[0155] 2. The present invention considers the different search ranges (sensor ranges), search and rescue capabilities, and motion capabilities of each machine intelligence system and sets different action spaces and observation values ​​for different intelligent agents, models the task environment and strategy reward system, and uses the deep reinforcement learning algorithm (DQN) to learn the environment image input. The Double DQN idea is used to solve the instability problem of the DQN algorithm using neural networks to approximate the value function and the low efficiency problem of the traditional Q-learning algorithm, update the value function in a larger state space, and find the optimal search and rescue strategy.

[0156] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0157] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An optimization method for a heterogeneous multi-machine intelligent system to collaboratively perform search and rescue tasks, characterized in that: The method comprises: Modeling a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determining an action space and a state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment; Establishing a reward system based on the action space and state space of each of the heterogeneous machine intelligence systems; Constructing a collaborative search and rescue mission model based on the action space, state space, and reward system, and using a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model; The step of constructing a collaborative search and rescue mission model based on the action space, state space, and reward system, and training and optimizing the collaborative search and rescue mission model using a deep reinforcement learning algorithm includes: Based on the action space, state space, and reward system, combined with the multi-machine intelligent system set and time, a collaborative search and rescue mission model is constructed. The collaborative search and rescue mission model uses a Markov decision process hexagram describing the multi-machine intelligent system.<I,S,A,T,r,γ> Represents, where I represents the set of multi-machine intelligent systems; S represents the state space; A represents the action space; T represents time; r represents the reward value in the reward system; γ represents the discount rate; The action-value function in the complex state space is calculated by a deep reinforcement learning algorithm to approximate the optimal strategy to train and optimize the collaborative search and rescue mission model; the action-value function is expressed as follows: Q(s i ,a i )=Q(s i ,a i )+α[r i +γmaxQ(s′,a′)-Q(s i ,a i )] Among them, Q(s i ,a i ) is the machine intelligence system in state s i Next take action a i Q value; s i Indicates the current state; a i represents the current action; α represents the learning rate; a′ represents the next action; s′ represents the state at the next moment; γ represents the discount rate; r i Represents the reward value; i represents the serial number of the machine agent.

2. The method according to claim 1, wherein The step of modeling a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determining an action space and a state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment includes: S11. Modeling the search and rescue mission environment into a simulated maze environment map using a grid method; S12. Based on the different search and rescue characteristics of the multiple heterogeneous machine intelligent systems, a local simulated maze environment submap of the current location of each heterogeneous machine intelligent system is captured, and all the local simulated maze environment submaps are used as inputs to the neural network, thereby gray-scaling the simulated maze environment map to obtain a two-dimensional grid maze map; S13, and determining the action space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment; S14. Determine the state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment.

3. The method according to claim 1, wherein The reward system includes: The reward function of any agent i in a multi-machine intelligent system at time t is: in: It means that the machine agent i finds a victim at time t; Indicates that the machine agent i encounters an obstacle; represents the movement cost of machine agent i; Represents the new map information obtained by machine agent i; g represents the reward for finding the victims; c represents the reward for encountering obstacles; m represents the movement cost; e represents the new map information obtained.

4. The method according to claim 1, wherein The method further includes: evaluating the performance of the collaborative search and rescue mission model using a collaborative search and rescue mission model performance evaluation system.

5. An optimization system for heterogeneous multi-machine intelligent systems to collaboratively perform search and rescue tasks, characterized by: The system comprises: An action and state space acquisition module is used to model a search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determine the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment; a reward system acquisition module, configured to establish a reward system based on the action space and state space of each of the heterogeneous machine intelligence systems; A model building and optimization module, configured to build a collaborative search and rescue mission model based on the action space, state space, and reward system, and to train and optimize the collaborative search and rescue mission model using a deep reinforcement learning algorithm; The model construction and optimization module constructs a collaborative search and rescue mission model based on the action space, state space, and reward system, and uses a deep reinforcement learning algorithm to train and optimize the collaborative search and rescue mission model, including: Based on the action space, state space, and reward system, combined with the multi-machine intelligent system set and time, a collaborative search and rescue mission model is constructed. The collaborative search and rescue mission model uses a Markov decision process hexagram describing the multi-machine intelligent system.<I,S,A,T,r,γ> Represents; where I represents the set of multi-machine intelligent systems; S represents the state space; A represents the action space; T represents time; r represents the reward value in the reward system; γ represents the discount rate; The action-value function in the complex state space is calculated by a deep reinforcement learning algorithm to approximate the optimal strategy to train and optimize the collaborative search and rescue mission model; the action-value function is expressed as follows: Q(s i ,a i )=Q(s i ,a i )+α[r i +γmaxQ(s′,a′)-Q(s i ,a i )] Among them, Q(s i ,a i ) is the machine intelligence system in state s i Next take action a i Q value; s i Indicates the current state; a i represents the current action; α represents the learning rate; a′ represents the next action; s′ represents the state at the next moment; γ represents the discount rate; r i Represents the reward value; i represents the serial number of the machine agent.

6. The system according to claim 5, wherein: The action and state space acquisition module models the search and rescue mission environment based on the search and rescue characteristics of multiple heterogeneous machine intelligence systems, and determines the action space and state space of each heterogeneous machine intelligence system based on the modeled search and rescue mission environment, including: S11. Modeling the search and rescue mission environment into a simulated maze environment map using a grid method; S12. Based on the different search and rescue characteristics of the multiple heterogeneous machine intelligent systems, a local simulated maze environment submap of the current location of each heterogeneous machine intelligent system is captured, and all the local simulated maze environment submaps are used as inputs to the neural network, thereby gray-scaling the simulated maze environment map to obtain a two-dimensional grid maze map; S13, and determining the action space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment; S14. Determine the state space of each of the heterogeneous machine intelligence systems based on the modeled search and rescue mission environment.

7. The system according to claim 5, wherein: The reward system includes: The reward function of any agent i in a multi-machine intelligent system at time t is: in: It means that the machine agent i finds a victim at time t; Indicates that the machine agent i encounters an obstacle; represents the movement cost of machine agent i; Represents the new map information obtained by machine agent i; g represents the reward for finding the victims; c represents the reward for encountering obstacles; m represents the movement cost; e represents the new map information obtained.

8. The system according to claim 5, wherein: The system further comprises: a model performance evaluation module for evaluating the performance of the collaborative search and rescue mission model using the collaborative search and rescue mission model performance evaluation system.

Images