Unmanned aerial vehicle cluster collaborative search rescue algorithm based on reinforcement learning and virtual simulation verification method

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to an unmanned aerial vehicle cluster collaborative search and rescue algorithm based on reinforcement learning and a virtual simulation verification method, which are used for planning unmanned aerial vehicle cluster collaborative search and rescue tasks by combining the requirements of unmanned aerial vehicle cluster search and rescue and adopting an algorithm based on deep reinforcement learning. A virtual simulation verification platform is developed by utilizing Unity, a real unmanned aerial vehicle cluster model and a complex environment are established, and a real-time data transmission function with a reinforcement learning algorithm end is realized, so that the actual execution effect and effectiveness of the designed collaborative search rescue algorithm are verified.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a UAV cluster collaborative search and rescue algorithm based on reinforcement learning and a virtual simulation verification method. Background Art

[0002] Unmanned aerial vehicles (UAVs) refer to unmanned aerial vehicles. Compared with manned aircraft, UAVs have the advantages of low cost, small size, high maneuverability, good concealment, and strong survivability. They are widely used in military and civilian fields such as battlefield reconnaissance, material transportation, weather monitoring, and resource exploration. Quadcopter UAVs are capable of a variety of low-altitude missions due to their simple structure, low production cost, vertical take-off and landing, and fixed-point hovering. They have been widely used in exploration missions in complex environments and have become an international research hotspot.

[0003] As the complexity of the mission environment and the diversity of mission types continue to increase, a single UAV is unable to complete large-scale complex missions due to its limited payload and endurance. In order to solve the various problems existing in a single UAV, the concept of UAV swarms is proposed. UAV swarms are flexible to deploy, and the functions of expensive advanced UAVs can be broken down into small pieces and distributed to multiple low-cost UAVs with simple structures. Different modular UAV swarms can be formed according to different mission characteristics, and various problems can be flexibly solved through the cooperation between various types of UAVs. Each UAV in a UAV swarm takes the completion of the overall mission as the highest priority, while completing its own mission as much as possible. Through the coordination and cooperation between individual UAVs, complex tasks that cannot be completed by a single individual can be completed. Therefore, the research on cooperative control algorithms for UAV swarms has important practical significance.

[0004] However, since large-scale drone swarms are expensive, organizing drone swarm flight tests requires a lot of manpower and material resources, and there are certain risks, the developed swarm collaborative control algorithm cannot be directly tested on a real machine. In order to verify the effectiveness of the algorithm, a set of effective simulation verification technology is needed. Traditional digital simulation technology usually uses the simulation module provided by MATLAB to realize the simulation verification of the drone kinematic model and control algorithm. However, traditional simulation platforms can only perform offline simulation and cannot interact with the algorithm end in real-time simulation. Researchers cannot intuitively judge the actual execution effect of the algorithm. Summary of the invention

[0005] The purpose of the present invention is to provide a UAV swarm collaborative search and rescue algorithm and a virtual simulation verification method based on reinforcement learning, which can make use of modern game development tools such as Unity and Unreal Engine to interact with the algorithm end through external communication and control instructions and use the platform physics engine to automatically solve the UAV motion, thereby realizing the interaction between the UAV swarm and the simulation environment driven by the algorithm.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A UAV swarm collaborative search and rescue algorithm based on reinforcement learning, the algorithm comprises the following steps:

[0008] S1: Through the study of the reinforcement learning UAV swarm collaborative search and rescue path planning algorithm, a UAV swarm collaborative search and rescue game model is established based on the Markov decision process, and constraints are set considering the characteristics of the search and rescue mission; through the process of model establishment-iterative training-online decision-making, the UAV swarm path planning strategy is optimized, and based on the internal information sharing of the UAV swarm, collaborative exploration is carried out in unknown environments;

[0009] In S1, considering that the conditions for drone clusters to perform search and rescue missions in different environments are different, it is necessary to clarify the mission scenario, and the mission scenario setting for drone cluster collaborative search and rescue includes the following steps:

[0010] S11: Assume that there are several static obstacles, several friendly UAVs and a target to be rescued in the mission area, and the mission goal of the friendly UAV is to avoid the obstacles and reach the location of the target to be rescued;

[0011] S12: The environment that the drone needs to explore is a rectangular urban environment with boundaries on all sides. There are buildings scattered in the environment. The drone cannot collide with buildings and other drones during flight. The drone only flies in a two-dimensional plane with a fixed height. Buildings with a height greater than or equal to the flight plane are considered obstacles.

[0012] S13: At the beginning of the mission, several drones were randomly scattered in different areas of the environment to ensure that they would not collide with buildings in the initial state;

[0013] S14: Each drone is equipped with sensors and communication devices to sense the local environmental obstacle information around it and share information within the drone cluster;

[0014] S15: There is a target object in the environment, the position of which is known at the beginning and does not change over time;

[0015] S16: If more than 80% of the drones reach the target location to be rescued within the specified time and hover over the location for more than 3 time steps at the same time, the mission is successful, otherwise the mission fails after the time runs out;

[0016] S17: When the mission is over, the less time it takes to reach the destination, the greater the benefit.

[0017] When the drone is performing a mission, it is subject to its own equipment and safety restrictions, and must meet constraints, including speed constraints, obstacle avoidance constraints, collision avoidance constraints, and boundary constraints.

[0018] Speed ​​Constraints:

[0019] In the exploration mission, the drone has a maximum speed limit due to its own maneuverability, namely:

[0020] ||v i ||≤v max (1)

[0021] Where i is the number of the drone, i∈[1,N], N is the number of drones; v i is the current speed of the drone, v max is the maximum speed of the drone;

[0022] Obstacle avoidance constraints:

[0023] Due to the presence of several static obstacles in the area, for safety reasons, the drone cannot collide with obstacles during flight, that is, the relative distance between the two must be kept within a safe range, namely:

[0024] Δd ik >d min (2)

[0025] Where, Δd ik =||p i -p k ||, represents the distance between the ith UAV and the kth obstacle around it, p i and p k Respectively represent the position of the i-th UAV and the position of the center point of the k-th obstacle; d min Indicates the maximum safety radius of the drone;

[0026] Collision avoidance constraints:

[0027] For safety reasons, drones cannot collide with each other, and the relative distance between them needs to be kept within a safe range, that is:

[0028] Δd ij >d min (3)

[0029] Where, Δd ij =||p i -p j || represents the distance between the i-th UAV and the j-th UAV, i,j∈[1,N], i≠j;

[0030] Boundary constraints:

[0031] To ensure the smooth progress of the exploration mission, the drone cannot exceed the given mission area during flight, that is:

[0032] 0≤p i,ξ ≤d bound,ξ (4)

[0033] Among them, ξ∈[1,ξ max ] represents the motion dimension of the drone, which is used to explore the two-dimensional environment, ξ max =2.d bound,ξ represents the region boundary in the ξ dimension. i,ξ Represents the position of the drone in the ξ dimension.

[0034] In S1, the random game model of the drone swarm search and rescue mission includes a state set S, an observation state set O, an action set A, and a reward function R;

[0035] State set S:

[0036] According to the specific situation of the task, the obstacle position grid map g at time t is b , the path grid map g has been found r , the detected area grid map g a , the target position p t , the position of our drone p i , speed v i and direction d i , as the state quantity s, that is:

[0037] s=(g b ,g r ,g a ,p t ,p i ,v i ,d i )∈S (5)

[0038] Among them, i∈[1,N], N is the number of drones;

[0039] Observation state set O:

[0040] For the i-th self-drone, i∈[1,N], the path grid map g found at time t is r , the detected area grid map g a , the target position p t , its own position p i , speed v i and direction d i , the relative position Δp between the drone and other drones ij As an observation state, that is:

[0041] oi =(g r ,g a ,p t ,p i ,v i ,d i ,Δp ij ) (6)

[0042] Action Set A:

[0043] Discretize the continuous environment into a grid map, and the action space can be discretized from the continuous speed control amount to moving between the grids of the grid map, using the discrete action space based on the drone's own coordinate system;

[0044] Reward function R:

[0045] The ultimate goal is to light up the map and reach the target location within the specified time. A reward and punishment mechanism is established to determine the benefit value obtained by a single-step decision. For a single drone, the benefit value mainly includes exploration benefit and target benefit.

[0046] Explore the benefits:

[0047] Before the drone reaches the rescue target location, since the obstacle information on the map is unknown, it needs to explore the map first to find a path to the target location. The exploration process is goal-oriented. Considering the reward function with more complex obstacle distribution, the shortest path to the target is calculated through breadth-first search. The drone’s behavior is rewarded or punished based on whether it moves along the shortest path, so that the agent can learn the ability of path planning.

[0048] The result returned by the breadth-first search algorithm is a matrix. If a grid value is -1, it means that the grid cannot reach the target point through any path. Otherwise, it indicates the shortest moving distance from the grid to the target point. The exploration reward is:

[0049] r explore =dist[i,j]-dist[i′,j′] (7)

[0050] Among them, dist is the two-dimensional matrix returned by the breadth-first search algorithm, i, j are the original coordinates of the drone, and i′, j′ are the new coordinates of the drone after performing the action;

[0051] Target income:

[0052] When the UAV cannot find a path to the rescue target, the negative distance between the UAV and the target is used as the reward function. When the distance between the UAV and the target is far, a large penalty is imposed, and when the distance between the UAV and the target is close, a small penalty is imposed. The target reward is:

[0053]

[0054] Among them, i target and j target Indicates the location of the target in the grid map;

[0055] Combining equations (7) and (8), the total reward of the drone mission is designed as:

[0056] r=r explore +r target (9).

[0057] In S1, the drone cluster collaborative search and rescue adopts the MASAC algorithm, and the application of the MASAC algorithm is divided into two parts: iterative training and online decision-making;

[0058] Iterative training:

[0059] A reinforcement learning algorithm of centralized training and distributed decision-making is used to establish a decision network and an evaluation network. The decision network determines the discrete action sequence of the drone at the current moment based on the drone's own state information provided by the simulation platform, the grid map information collected during the exploration process, and the known location information of the rescue target. The evaluation network evaluates the quality of the decision results based on the state information and decision information, and adds new interactive experience to the experience database. Through the random experience playback mechanism, the experience in the database is randomly extracted to gradually train the evaluation network and the decision network, and the drone strategy is obtained through multiple iterations.

[0060] Online Decision Making:

[0061] In the simulation platform, the initial positions of the UAV and the target to be rescued are randomly given, the map obstacle distribution is randomly initialized, and the trained decision network is used to generate UAV decision results in real time to complete the UAV cluster motion planning for the search task.

[0062] S2: Develop a virtual simulation environment for drone cluster collaborative search and rescue based on the Unity game engine to provide support for algorithm training and verification, and establish a set of parallel training scenarios and test scenarios. The modules for regular scenarios are the same, while customized modules are required for special scenarios.

[0063] In S2, the conventional scene module includes an object model, a camera module, a grid map module, a random map module and a map lighting module;

[0064] Object model: Unity supports importing general 3D models in 3DS and FBX formats, and uses 3D modeling tools to create and export 3D models. When building large-scale urban scenes, use Unity's City Generator plug-in, which uses blocks as the smallest unit and generates urban scenes of different scales by splicing blocks. Use regular expressions to find the mesh model of the corresponding building from the model library and attach it to the mesh collision body component.

[0065] Camera module: Unity's scenes are created by placing and moving objects in three-dimensional space. When there are multiple cameras, multi-camera display is performed by setting the offsets of different cameras.

[0066] Grid map module: In training or testing scenarios, grid maps are used to express environmental information. The environment is regarded as a two-dimensional plane, which is divided into several grids of equal size with numerical information. Each grid stores the value of the surrounding environment information, and the obstacle information in the environment is simplified into a two-dimensional array. Each element of the array represents the state of a grid area. A value of 0 represents an open space, and a value of 1 represents an obstacle.

[0067] The position of objects in the Unity scene is determined by the world coordinate system, and the position of the object in the world coordinate system is represented by a triplet of the form (x, y, z), where the values ​​of x, y, and z are real numbers. The position of the grid map is determined by the grid coordinate system, and the position in the grid map is represented by a doublet of the form (i, j), where the values ​​of i and j are natural numbers.

[0068] Due to the differences in the above coordinate system representation methods, calculation is required to transform the world coordinate system and the grid coordinate system. The transformation from the world coordinate system to the grid coordinate system is:

[0069]

[0070] Among them, (i obj ,j obj ) is the position of the object in the grid map after transformation, (x obj ,z obj ) is the position of the object in the world coordinate system, (x ref ,z ref ) is the position of the reference point in the world coordinate system, (i ref ,j ref ) is the position of the reference point in the grid map, l is the grid length of the grid map;

[0071] Conversely, the grid coordinate system can be transformed to the world coordinate system by the following formula:

[0072]

[0073] Among them, (x obj ,y obj ,z obj ) is the position of the object in the world coordinate system after transformation, (i obj ,j obj ) is the position of the object in the grid map, (x ref ,z ref ) is the position of the reference point in the world coordinate system, (i ref ,j ref ) is the position of the reference point in the grid map, l is the grid length of the grid map;

[0074] The above transformation processes all involve reference points. Different reference points will result in different coordinate transformation results. In the application, the center coordinates of the map are (x, y, z) = (0, 0, 0), (i, j) = (i max / 2,j max / 2) as a reference point, aligning the center of the world map with the center of the grid map;

[0075] Random map module: During the iterative training of reinforcement learning, maps need to be randomly generated so that the agent can learn new environmental information in each iteration;

[0076] Map lighting module: used to display the historical trajectory of the drone during flight and mark the points where the drone has flown on the map.

[0077] A reinforcement learning-based UAV swarm collaborative search virtual simulation verification method is proposed. The collaborative search and rescue algorithm design in S1 and the virtual simulation environment development in S2 are simulated and verified through a real-time simulation architecture. The overall simulation architecture is constructed and a real-time data interface based on ML-Agents is configured.

[0078] The overall simulation architecture is constructed by using Unity rendering and physical simulation to simulate the UAV flight scene. During training, the environment generates a variety of random scenes and provides observation information of the UAV, including the position, speed, posture and perception data of the surrounding environment. The UAV executes the corresponding action generated by the reinforcement learning algorithm based on the received observation information. The environment calculates the reward signal based on the action and current state of the UAV to guide the learning process of the algorithm. This process continues to cycle until the predetermined training convergence condition is met.

[0079] On the Python reinforcement learning algorithm side, the system receives real-time observation data of the drone from the Unity simulation environment. The data is transmitted through the ML-Agents framework. After receiving the observation information, the data is input into a pre-designed deep neural network, which is the policy network. It is trained to extract features from complex observation data and output the optimal action decision to maximize the cumulative reward. Based on the reward signal obtained from the environment and the feedback of the detection mechanism, the parameters of the policy network are updated through the back-propagation algorithm, and the updated network parameters are reloaded to use the latest model in the next decision-making process.

[0080] The ML-Agents framework enables games and virtual simulation platforms to become environments for training intelligent agents, and consists of a learning environment, a Python interface, and external communications;

[0081] The operations of the learning environment on the intelligent agents in the scene include collecting observation information, executing given actions, and obtaining rewards. The learning environment components can be divided into components of different levels. The highest level is the Academy component. There can only be one Academy component in a scene, which is used to manage the entire learning environment. It is also responsible for judging whether the environment has reached the terminal state and starting a new round of learning. The middle level is the Brain component. There can be multiple Brain components under one Academy component, which is used to manage a certain type of intelligent agent. It is also responsible for processing the observation information passed in by the lower-level intelligent agents and giving action instructions. The bottom level is the Agent component. There can be multiple Agent components under one Brain component, corresponding to each intelligent agent in the scene, which is used to control the behavior of a single intelligent agent.

[0082] The technical effects achieved by the present invention are:

[0083] Traditional UAV path planning algorithms need to plan the path to the target location based on prior map information. The reinforcement learning-based path planning algorithm proposed in this scheme gradually collects map information by interacting with the environment, and can explore effective paths in unknown environments.

[0084] Traditional simulation environments mostly represent the status of drone clusters and the environment by establishing and solving mathematical models, which is not convenient for intuitively analyzing simulation results. This solution uses the Unity engine to develop a virtual simulation environment, automatically realizes the interaction between drone models and the environment through the physics engine of the game development tool, and intuitively demonstrates the flight status of the drone cluster on the screen through the camera module.

[0085] Traditional algorithm model training mostly uses offline data sets and cannot adapt to dynamic and changing environments. The present invention uses the architecture of algorithm computer-real-time data interface-simulation computer to build a simulation platform, realizing two-way high-speed data interaction between the algorithm end and the simulation end, thereby improving the training speed and the final performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a general structure diagram of the simulation verification platform of the embodiment of the present invention;

[0087] Figure 2 It is a schematic diagram of the iterative training link of the algorithm of an embodiment of the present invention;

[0088] Figure 3 It is a schematic diagram of the online decision-making link of the algorithm of the embodiment of the present invention;

[0089] Figure 4 It is a schematic diagram of the "training-testing" parallel scenario module of an embodiment of the present invention;

[0090] Figure 5 is a schematic diagram of an obstacle and an intelligent agent model in a training environment according to an embodiment of the present invention;

[0091] Figure 6 is a schematic diagram of the structure of a drone model in a test scenario according to an embodiment of the present invention;

[0092] Figure 7 is a schematic diagram of an urban environment model in a test scenario of an embodiment of the present invention;

[0093] Figure 8 This is a schematic diagram of a multi-camera display effect according to an embodiment of the present invention;

[0094] Fig. 9 is a flowchart of raster map scanning according to an embodiment of the present invention;

[0095] Fig.10 is a flow chart of random map generation according to an embodiment of the present invention;

[0096] Fig.11 It is a schematic diagram of the simulation training and verification architecture of an embodiment of the present invention. DETAILED DESCRIPTION

[0097] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0098] Embodiment 1:

[0099] like Figure 1-Figure 11As shown in the figure, the overall structure of the simulation verification platform includes three parts: algorithm computer, simulation computer, and real-time data interface. The present invention focuses on the UAV swarm collaborative search and rescue mission, proposes a UAV swarm collaborative search and rescue path planning algorithm based on reinforcement learning, and designs a virtual simulation environment based on Unity for the training and verification requirements of the algorithm. Finally, the algorithm end and the simulation end are connected through the real-time simulation data interface, realizing the three-dimensional visualization demonstration and verification of the UAV swarm collaborative search and rescue mission. The specific implementation process is as follows:

[0100] The UAV swarm collaborative search and rescue algorithm based on reinforcement learning includes the following steps:

[0101] S1: Through the study of the reinforcement learning UAV swarm collaborative search and rescue path planning algorithm, a UAV swarm collaborative search and rescue game model is established based on the Markov decision process, and constraints are set considering the characteristics of the search and rescue mission; through the process of model establishment-iterative training-online decision-making, the UAV swarm path planning strategy is optimized, and based on the internal information sharing of the UAV swarm, collaborative exploration is carried out in unknown environments;

[0102] Drone swarm search and rescue mission scenario settings:

[0103] In S1, considering that the conditions for drone swarms to perform search and rescue missions in different environments are different, it is necessary to clarify the mission scenario. The mission scenario setting for drone swarm collaborative search and rescue includes the following steps:

[0104] S11: Assume that there are several static obstacles, several friendly UAVs and a target to be rescued in the mission area, and the mission goal of the friendly UAV is to avoid the obstacles and reach the location of the target to be rescued;

[0105] S12: The environment that the drone needs to explore is a rectangular urban environment with boundaries on all sides. There are buildings scattered in the environment. The drone cannot collide with buildings and other drones during flight. The drone only flies in a two-dimensional plane with a fixed height. Buildings with a height greater than or equal to the flight plane are considered obstacles.

[0106] S13: At the beginning of the mission, several drones were randomly scattered in different areas of the environment to ensure that they would not collide with buildings in the initial state;

[0107] S14: Each drone is equipped with sensors and communication devices to sense the local environmental obstacle information around it and share information within the drone cluster;

[0108] S15: There is a target object in the environment, the position of which is known at the beginning and does not change over time;

[0109] S16: If more than 80% of the drones reach the target location to be rescued within the specified time and hover over the location for more than 3 time steps at the same time, the mission is successful, otherwise the mission fails after the time runs out;

[0110] S17: When the mission is over, the less time it takes to reach the destination, the greater the benefit.

[0111] In addition, when the UAV is performing its mission, it is subject to its own equipment and safety restrictions, and the UAV must meet constraints, including speed constraints, obstacle avoidance constraints, collision avoidance constraints, and boundary constraints;

[0112] Speed ​​Constraints:

[0113] In the exploration mission, the drone has a maximum speed limit due to its own maneuverability, namely:

[0114] ||v i ||≤v max (1)

[0115] Where i is the number of the drone, i∈[1,N], N is the number of drones; v i is the current speed of the drone, v max is the maximum speed of the drone;

[0116] Obstacle avoidance constraints:

[0117] Due to the presence of several static obstacles in the area, for safety reasons, the drone cannot collide with obstacles during flight, that is, the relative distance between the two must be kept within a safe range, namely:

[0118] Δd ik >d min (2)

[0119] Where, Δd ik =||p i -p k ||, represents the distance between the ith UAV and the kth obstacle around it, p i and p k Respectively represent the position of the i-th UAV and the position of the center point of the k-th obstacle; d min Indicates the maximum safety radius of the drone;

[0120] Collision avoidance constraints:

[0121] For safety reasons, drones cannot collide with each other, and the relative distance between them needs to be kept within a safe range, that is:

[0122] Δd ij >d min (3)

[0123] Where, Δd ij =||p i -p j || represents the distance between the i-th UAV and the j-th UAV, i,j∈[1,N], i≠j;

[0124] Boundary constraints:

[0125] To ensure the smooth progress of the exploration mission, the drone cannot exceed the given mission area during flight, that is:

[0126] 0≤p i,ξ ≤d bound,ξ (4)

[0127] Among them, ξ∈[1,ξ max ] represents the motion dimension of the drone, which is used to explore the two-dimensional environment, ξ max =2.d bound,ξ represents the region boundary in the ξ dimension. i,ξ Represents the position of the drone in the ξ dimension.

[0128] In summary, the goal of the UAV swarm search and rescue mission can be described as: considering the constraints of the UAV's flight speed, obstacle avoidance, collision avoidance, boundaries, etc., using each UAV's own observation information and the UAV swarm shared information, to reach the target location to be rescued within the specified time.

[0129] UAV swarm search and rescue model establishment:

[0130] In S1, the random game model of the UAV swarm search and rescue mission includes the state set S, the observation state set O, the action set A, and the reward function R;

[0131] State set S:

[0132] According to the specific situation of the task, the obstacle position grid map g at time t is b , the path grid map g has been found r , the detected area grid map g a , the target position p t , the position of our drone p i , speed v i and direction d i , as the state quantity s, that is:

[0133] s=(g b ,g r ,g a ,p t ,p i ,v i ,d i )∈S (5)

[0134] Among them, i∈[1,N], N is the number of drones;

[0135] Observation state set O:

[0136] For the i-th self-drone, i∈[1,N], the path grid map g found at time t is r , the detected area grid map g a , the target position p t , its own position p i , speed v i and direction d i , the relative position Δp between the drone and other drones ij As an observation state, that is:

[0137] o i =(g r ,g a ,p t ,p i ,v i ,d i ,Δp ij ) (6)

[0138] Action Set A:

[0139] Discretize the continuous environment into a grid map, and the action space can be discretized from the continuous speed control amount to moving between the grids of the grid map, using the discrete action space based on the drone's own coordinate system; ① turn left (the drone's position remains unchanged, and the direction turns left 90 degrees), ② turn right (the drone's position remains unchanged, and the direction turns right 90 degrees), ③ decelerate forward (the drone's speed decreases by 1, the direction remains unchanged, and the speed moves forward by a number of grids), ④ move forward at a steady speed (the drone's speed remains unchanged, the direction remains unchanged, and the speed moves forward by a number of grids), ⑤ accelerate forward (the drone's speed increases by 1, the direction remains unchanged, and the speed moves forward by a number of grids), ⑥ emergency stop (the drone sets the speed to 0 and the direction remains unchanged);

[0140] Reward function R:

[0141] The ultimate goal is to light up the map and reach the target location within the specified time. A reward and punishment mechanism is established to determine the benefit value obtained by a single-step decision. For a single drone, the benefit value mainly includes exploration benefit and target benefit.

[0142] Explore the benefits:

[0143] Before the drone reaches the rescue target location, since the map obstacle information is unknown, it needs to explore the map first to find a path to the target location. The exploration process is goal-oriented and meaningless exploration cannot be performed. Considering the more complex reward function of obstacle distribution, the shortest path to the target is calculated through breadth-first search, and the drone's behavior is rewarded or punished based on whether the drone moves along the shortest path, so that the agent can learn the ability of path planning;

[0144] Breadth-first search (BFS) is one of the simpler search algorithms. It belongs to the blind search method, and its purpose is to systematically expand and check all nodes in the graph. The shortest moving distance from each point in the grid map to the target is calculated. The process is shown in Algorithm 1:

[0145]

[0146] The basic idea of ​​the BFS algorithm is to start from the target point, add the target point to the queue, then pop out the elements in the queue one by one, and add the elements around it to the queue until the queue is empty. Through rounds of cycles, all grids can be visited and the shortest moving distance to the target point can be calculated;

[0147] The result returned by the breadth-first search (BFS) algorithm is a matrix. If a grid value is -1, it means that the grid cannot reach the target point through any path. Otherwise, it indicates the shortest moving distance from the grid to the target point. The exploration reward is:

[0148] r explore =dist[i,j]-dist[i′,j′] (7)

[0149] Among them, dist is the two-dimensional matrix returned by the breadth-first search algorithm, i, j are the original coordinates of the drone, and i′, j′ are the new coordinates of the drone after performing the action;

[0150] Target income:

[0151] When the drone cannot find a path to the rescue target, the negative distance between the drone and the target is used as the reward function. When the distance between the drone and the target is far, a large penalty is imposed, and when the distance between the drone and the target is close, a small penalty is imposed. At the same time, because the reward is always negative, in order to maximize the total return, this requires the drone to complete the task as soon as possible and end the round to avoid receiving more penalties. The target reward is:

[0152]

[0153] Among them, i target and j target Indicates the location of the target in the grid map;

[0154] Combining equations (7) and (8), the total reward of the drone mission is designed as:

[0155] r=r explore +r target (9).

[0156] UAV swarm collaborative search and rescue algorithm based on MASAC:

[0157] The MASAC algorithm is a widely used multi-agent reinforcement learning algorithm that uses the Actor-Critic network architecture. The Actor-Critic architecture consists of two parts: the Actor and the Critic. The Actor part is the policy network, and the Critic part is the value network. The input of the Actor network is observation, and the output is action. Its optimization goal is to be able to select the best action based on the observation input; the input of the Critic network is state and action, and the output is value. Its optimization goal is to be able to most accurately judge the value of the state-action pair;

[0158] In S1, the drone cluster collaborative search and rescue adopts the MASAC algorithm. The application of the MASAC algorithm is divided into two parts: iterative training and online decision-making;

[0159] Iterative training:

[0160] like Figure 2 As shown in the figure, a centralized training-distributed decision-making reinforcement learning algorithm is adopted to establish a decision network and an evaluation network. The decision network determines the discrete action sequence number of the drone at the current moment based on the drone's own state information provided by the simulation platform, the grid map information collected during the exploration process, and the known location information of the rescue target; the evaluation network evaluates the quality of the decision results based on the state information and decision information, and adds new interactive experience to the experience database. Through the random experience playback mechanism, the experience in the database is randomly extracted to gradually train the evaluation network and the decision network, and the drone strategy is obtained through multiple iterations; the detailed network training process is shown in Algorithm 2:

[0161]

[0162]

[0163] Online Decision Making:

[0164] like Figure 3 As shown in the figure, the initial positions of the UAV and the target to be rescued are randomly given in the simulation platform, the map obstacle distribution is randomly initialized, and the trained decision network is used to generate the UAV decision results in real time to complete the UAV cluster motion planning for the search task; the online decision algorithm is shown in Algorithm 3:

[0165]

[0166] S2: Develop a virtual simulation environment for drone cluster collaborative search and rescue based on the Unity game engine to provide support for algorithm training and verification, and establish a set of parallel training scenarios and test scenarios. The modules for regular scenarios are the same, while customized modules are required for special scenarios.

[0167] Among them, the UAV cluster collaborative search and rescue path planning algorithm designed in S1 only determines the control instructions of the UAV in the form of numbers, which cannot intuitively reflect the execution of the overall search and rescue mission. At the same time, the reinforcement learning algorithm needs to continuously interact with the simulation environment to collect data in order to optimize its own strategy;

[0168] The ultimate goal of the reinforcement learning algorithm is to learn an exploration strategy that is suitable for any unknown map. In order to enhance the generalization performance of the model, it is necessary to collect sample data of multiple maps during the training process, and the simulation environment is required to be able to generate random maps as needed. During the training process, a random map is often discarded after being used only a few times. From the perspective of improving training efficiency, a method of automatically generating maps based on a specific algorithm should be adopted. However, automatically generated maps are often quite different from real-world urban environments, and the mapping and modeling are simple. From the perspective of optimizing visual demonstration, a method of manually constructing maps should be adopted.

[0169] Since the above two map construction methods have their own advantages and disadvantages, the compromise method is to establish a set of parallel training scenes and test scenes to fully utilize the advantages of the two map construction methods. For this set of parallel training scenes and test scenes, most of the scene modules are the same, but the implementation of special functions of the scene requires customized modules. Figure 4 The figure shows the module composition of the "training-testing" parallel scenario, where the Academy and Agent modules will be implemented in the third step.

[0170] In S2, the conventional scene module includes object model, camera module, grid map module, random map module and map lighting module;

[0171] Object model: The training scene does not pay much attention to the refinement and aesthetics of the model, but should focus on the smoothness of the scene operation to ensure that the reinforcement learning algorithm can interact with the environment efficiently. Therefore, the most reasonable method is to use Unity's built-in 3D objects to create training scenes. Figure 5 As shown, various combinations of cubes can be used to represent obstacles, and capsules can be used to represent agents.

[0172] like Figure 6As shown in the figure, for the test environment, due to the need for visual demonstration, the object model cannot be as simple as the training scene. Unity supports importing general 3D models in 3DS and FBX formats, and using 3D modeling tools to create and export 3D models. When building large-scale urban scenes, manually selecting building models is often time-consuming and laborious. When building large-scale urban scenes, use Unity's City Generator plug-in, which uses blocks as the smallest unit and generates urban scenes of different scales by splicing blocks, such as Figure 7 However, although the city scene generated by the City Generator plug-in is beautiful, the building model does not contain a collider, and subsequent map scanning cannot be performed, so it is necessary to add a collider component by yourself. In the city scene generated by the City Generator plug-in, the building models all contain a Mesh Renderer component, but no Mesh Collider component. Therefore, the mesh model of the corresponding building is found from the model library through a regular expression and attached to the Mesh Collider component;

[0173] Camera module: Unity's scenes are created by placing and moving objects in three-dimensional space. Since the observer's screen is a two-dimensional screen, a method is needed to capture the view and "flatten" it for display. This process is completed through the camera; when there are multiple cameras, multi-camera display is performed by setting the offset of different cameras;

[0174] When a camera observes an object, the farther the object is from the viewpoint, the smaller it appears, which is the "perspective effect". However, in some cases, it is desirable that the displayed object does not shrink as the distance increases, which is the "orthogonal effect". In Unity, you can flexibly select the display effect based on the function of the camera. For a camera that follows the movement of the agent and is used to display the field of view, the perspective effect is selected; for a camera that views the global map, the orthogonal effect should be selected.

[0175] The camera can fill the empty space with some objects. By using the skybox, the infinitely far image can be rendered as the sky, thus achieving the effect of simulating the environment. When there are multiple cameras, the offset of different cameras can be set to achieve the effect of multi-camera display.

[0176] After making the above settings, you can observe the images captured by each camera in real time in the Game window in Unity, such as Figure 8 shown.

[0177] Grid map module: In training scenarios or test scenarios, a simple way to express environmental obstacle information is needed so that the reinforcement learning model can better grasp the state of the environment. Grid map is a simple way to express environmental information. It uses grid map to express environmental information. The environment is regarded as a two-dimensional plane, and the plane is divided into several grids of equal size with numerical information. Each grid stores the value of the surrounding environment information, and the obstacle information in the environment is simplified into a two-dimensional array. Each element of the array represents the state of a grid area. The value of 0 represents an open space, and the value of 1 represents an obstacle.

[0178] The position of objects in the Unity scene is determined by the world coordinate system, and the position of the object in the world coordinate system is represented by a triplet of the form (x, y, z), where the values ​​of x, y, and z are real numbers. The position of the grid map is determined by the grid coordinate system, and the position in the grid map is represented by a doublet of the form (i, j), where the values ​​of i and j are natural numbers.

[0179] Due to the differences in the above coordinate system representation methods, calculation is required to transform the world coordinate system and the grid coordinate system. The transformation from the world coordinate system to the grid coordinate system is:

[0180]

[0181] Among them, (i obj ,j obj ) is the position of the object in the grid map after transformation, (x obj ,z obj ) is the position of the object in the world coordinate system, (x ref ,z ref ) is the position of the reference point in the world coordinate system, (i ref ,j ref ) is the position of the reference point in the grid map, l is the grid length of the grid map;

[0182] Conversely, the grid coordinate system can be transformed to the world coordinate system by the following formula:

[0183]

[0184] Among them, (x obj ,y obj ,z obj ) is the position of the object in the world coordinate system after transformation, (i obj ,j obj ) is the position of the object in the grid map, (x ref ,z ref ) is the position of the reference point in the world coordinate system, (i ref ,j ref) is the position of the reference point in the grid map, l is the grid length of the grid map;

[0185] The above transformation processes all involve reference points. Different reference points will result in different coordinate transformation results. In the application, the center coordinates of the map are (x, y, z) = (0, 0, 0), (i, j) = (i max / 2,j max / 2) As a reference point, the center of the world map corresponds to the center of the grid map, thereby maximizing the use of grid information;

[0186] After deriving the transformation relationship between the world coordinate system and the grid coordinate system, the Unity scene map can be discretized and scanned to obtain a concise description of the obstacle information. Fig. 9 As shown, the scanning result of the discretized grid map can be obtained.

[0187] Random map module: During the iterative training of reinforcement learning, if the obstacle information is always fixed when the agent interacts with the environment, then even though the grid map information is input, the agent may not learn the ability to understand the map, but only memorize certain fixed optimal paths. The above problems are likely to lead to insufficient generalization of the algorithm, and a significant drop in performance when encountering new unknown environments. Therefore, during the iterative training of reinforcement learning, it is necessary to randomly generate maps so that the agent can learn new environmental information in each iteration;

[0188] Based on the above requirements, the training scenario needs to be able to randomly generate maps with different obstacle distributions in different rounds of reinforcement learning, so a random map module needs to be implemented. In addition, in order to adapt to the course learning technology of reinforcement learning, the scenario needs to be able to generate maps of different difficulties. Considering that the difficulty of the map mainly depends on the number of obstacles, it is necessary to implement the function of generating a specified number of obstacles.

[0189] like Fig.10 As shown, a random scene map with a variable number of obstacles and a variable search target position can be generated.

[0190] Map lighting module: used to display the historical trajectory of the drone during flight and mark the points where the drone has flown on the map.

[0191] First, create a plane object of the same size as the map as a mask and place it between the camera and the map. Next, change the material of the mask object to transparent. You can control the transparency of the mask by changing the material map. At the beginning of the simulation, cover all maps with black, and restore the grids that the drone passes through at each step to transparent, so that the map can be gradually lit up as the drone flies.

[0192] Embodiment 2:

[0193] like Fig.11 As shown in the figure, the virtual simulation verification method of UAV cluster collaborative search based on reinforcement learning is used to conduct overall simulation training and verification of the collaborative search and rescue algorithm design in S1 and the virtual simulation environment development in S2 through the real-time simulation architecture, and the overall simulation architecture is constructed and the real-time data interface based on ML-Agents is configured.

[0194] Building the overall simulation architecture Through Unity rendering and physical simulation, the UAV flight scene is simulated. During training, the environment generates a variety of random scenes and provides observation information of the UAV, including position, speed, attitude and perception data of the surrounding environment; the UAV executes the corresponding actions generated by the reinforcement learning algorithm based on the received observation information; the environment calculates the reward signal based on the UAV's actions and current state to guide the learning process of the algorithm; this process continues in a loop until the predetermined training convergence conditions are met; the system also sets up a test environment, whose main function is to evaluate the performance of the trained model in unknown or unseen scenes. The test environment also provides observation information of the UAV and accepts the actions performed by the UAV, but the scene is not seen in the training process, so as to effectively evaluate the generalization performance and robustness of the model. The test results will be used to analyze the advantages and disadvantages of the model and provide a basis for subsequent model improvements.

[0195] On the Python reinforcement learning algorithm side, the system receives real-time observation data of the drone from the Unity simulation environment. The data is transmitted through the ML-Agents framework. After receiving the observation information, the data is input into a pre-designed deep neural network, which is the policy network. It is trained to extract features from complex observation data and output the optimal action decision to maximize the cumulative reward. Based on the reward signal obtained from the environment and the feedback of the detection mechanism, the parameters of the policy network are updated through the back-propagation algorithm, and the updated network parameters are reloaded to use the latest model in the next decision-making process.

[0196] The entire system achieves seamless connection between the Unity simulation environment and the Python reinforcement learning algorithm through the ML-Agents framework. The framework provides a standardized communication interface, allowing observation data, action instructions, and reward signals to be efficiently transmitted between the two ends. At the same time, ML-Agents also supports parallel training, which accelerates the convergence of the model.

[0197] The ML-Agents framework enables games and virtual simulation platforms to become environments for training intelligent agents. It consists of a learning environment, a Python interface, and external communications.

[0198] The learning environment component is the core part. The operations of the learning environment on the intelligent agents in the scene include collecting observation information, executing given actions, and obtaining rewards. The learning environment components can be divided into components of different levels. The highest level is the Academy component. There can only be one Academy component in a scene, which is used to manage the entire learning environment. It is also responsible for judging whether the environment has reached the terminal state and starting a new round of learning. The middle level is the Brain component. There can be multiple Brain components under one Academy component, which is used to manage a certain type of intelligent agent. It is also responsible for processing the observation information passed in by the lower-level intelligent agent and giving action instructions. The bottom level is the Agent component. There can be multiple Agent components under one Brain component, corresponding to each intelligent agent in the scene, which is used to control the behavior of a single intelligent agent.

[0199] Academy Component Settings

[0200] The Academy script in ML-Agents only implements some basic functions. To apply it in actual scenarios, you need to write a new derived class that inherits Academy and implement the corresponding functions in it. The functions required by the Academy derived class are as follows:

[0201] Record and manage the training process of reinforcement learning

[0202] First, it is necessary to record the current round number and the number of steps in this round during the reinforcement learning training process; secondly, it is necessary to count the task success rates of the past several rounds in order to evaluate the training effect.

[0203] Determine whether the termination condition has been reached

[0204] In the environment, each agent can only perceive the local state of its vicinity and cannot know the global state. The terminal state may be triggered only when all agents meet certain conditions. Therefore, only the global Academy can determine whether the terminal condition has been reached. In addition, when the agent's strategy is not enough to complete the task, if the task is still not completed after a certain number of steps, the environment needs to be reset to the initial state to start the next round of training.

[0205] In the Academy derived class script, the abstract functions that need to be rewritten are as follows:

[0206] InitializeAcademy()

[0207] It is called when the environment is initialized to set relevant environment parameters, such as the maximum number of steps per round of training.

[0208] AcademyReset()

[0209] Called every time the environment is reset. All parameters that need to be reset before a round of training start are set here, such as resetting the step recorder, calculating the task success rate, scanning the grid map, etc.

[0210] AcademyStep()

[0211] It is called once at each time step, where some environment variables are updated, such as the step recorder. If the global state of the environment meets the termination condition or the step recorder reaches the maximum value, the environment is reset and prepared for the next round of training.

[0212] After writing the Academy derived class script, you need to create an empty object under the Unity root node and mount the Academy derived class script on it, then you can adjust the Academy parameters. In the training environment, set Time Scale to 100, the system will shorten the time step as much as possible to speed up the training; in the test environment, set TimeScale to 1, the system will run at normal speed, which is convenient for real-life display.

[0213] Brain component settings

[0214] Unlike the Academy component, you do not need to write scripts to use the Brain component. You can directly create an empty child object under the Academy and mount the Brain script, and then set the parameters of the Brain.

[0215] First, modify the Brain type, including the following options: External is for communicating with an external algorithm, which is responsible for giving a strategy; Internal is for selecting a strategy through a pre-trained model loaded by Unity; Player is for the operator to control the actions of the agent through external devices such as the mouse and keyboard; Heuristic is for hard-coding the agent in advance, and the agent selects a strategy according to a certain program logic. Because the strategy is given by the Python reinforcement learning algorithm, all Brain types use External.

[0216] Finally, set the state space to continuous and the action space to discrete.

[0217] Agent component settings

[0218] Similar to the Academy component, the Agent script only implements some basic functions. You need to write a new derived class that inherits Agent and implement the corresponding functions in it. The functions required by the Agent derived class are as follows:

[0219] Reset Agent State

[0220] After the environment is reset, the state of the agent also needs to be reset. The agent should reselect the initial position and set the speed to 0.

[0221] Send self-observation information

[0222] At each time step, the algorithm needs to send its own observation information about the environment to the external algorithm.

[0223] Receive and execute actions given by external algorithms

[0224] At each time step, the action given by the external algorithm needs to be executed and the corresponding reward is returned.

[0225] In the Agent derived class script, the abstract functions that need to be rewritten are as follows:

[0226] AgentReset()

[0227] Called in AcademyReset() of Academy to reset the agent state. The agent should randomly choose an initial position in the grid map where there are no obstacles and set the speed to 0.

[0228] CollectObservations()

[0229] It is called once at each time step to send its own observation information of the environment to the external algorithm.

[0230] AgentAction()

[0231] It is called once at each time step to execute the action given by the external algorithm and return the corresponding reward.

[0232] After writing the Agent derived class script, you need to mount it on the intelligent body and specify the corresponding Brain.

[0233] After completing all component settings, package Unity into an EXE executable file, which can be called through a Python script to achieve real-time data interaction between the simulation end and the algorithm end.

[0234] The above are only preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to conventional means in the art unless otherwise specified and limited.

Claims

1. A UAV swarm collaborative search and rescue algorithm based on reinforcement learning, characterized by: The algorithm comprises the following steps: S1: Through the study of the reinforcement learning UAV swarm collaborative search and rescue path planning algorithm, a UAV swarm collaborative search and rescue game model is established based on the Markov decision process, and constraints are set considering the characteristics of the search and rescue mission; through the process of model establishment-iterative training-online decision-making, the UAV swarm path planning strategy is optimized, and based on the internal information sharing of the UAV swarm, collaborative exploration is carried out in unknown environments; S2: Develop a virtual simulation environment for drone cluster collaborative search and rescue based on the Unity game engine to provide support for algorithm training and verification, and establish a set of parallel training scenarios and test scenarios. The modules for regular scenarios are the same, while customized modules are required for special scenarios.

2. The UAV swarm collaborative search and rescue algorithm based on reinforcement learning according to claim 1 is characterized in that: In S1, considering that the conditions for drone clusters to perform search and rescue missions in different environments are different, it is necessary to clarify the mission scenario, and the mission scenario setting for drone cluster collaborative search and rescue includes the following steps: S11: Assume that there are several static obstacles, several friendly UAVs and a target to be rescued in the mission area, and the mission goal of the friendly UAV is to avoid the obstacles and reach the location of the target to be rescued; S12: The environment that the drone needs to explore is a rectangular urban environment with boundaries on all sides. There are buildings scattered in the environment. The drone cannot collide with buildings and other drones during flight. The drone only flies in a two-dimensional plane with a fixed height. Buildings with a height greater than or equal to the flight plane are considered obstacles. S13: At the beginning of the mission, several drones were randomly scattered in different areas of the environment to ensure that they would not collide with buildings in the initial state; S14: Each drone is equipped with sensors and communication devices to sense the local environmental obstacle information around it and share information within the drone cluster; S15: There is a target object in the environment, the position of which is known at the beginning and does not change over time; S16: If more than 80% of the drones reach the target location to be rescued within the specified time and hover over the location for more than 3 time steps at the same time, the mission is successful, otherwise the mission fails after the time runs out; S17: When the mission is over, the less time it takes to reach the destination, the greater the benefit.

3. The UAV swarm collaborative search and rescue algorithm based on reinforcement learning according to claim 2 is characterized in that: When the drone is performing a mission, it is subject to its own equipment and safety restrictions, and must meet constraints, including speed constraints, obstacle avoidance constraints, collision avoidance constraints, and boundary constraints. Speed ​​Constraints: In the exploration mission, the drone has a maximum speed limit due to its own maneuverability, namely: ||v i ||≤v max (1) Where i is the number of the drone, i∈[1,N], N is the number of drones; v i is the current speed of the drone, v max is the maximum speed of the drone; Obstacle avoidance constraints: Due to the presence of several static obstacles in the area, for safety reasons, the drone cannot collide with obstacles during flight, that is, the relative distance between the two must be kept within a safe range, namely: Δd ik >d min (2) Where, Δd ik =||p i -p k ||, represents the distance between the ith UAV and the kth obstacle around it, p i and p k Respectively represent the position of the i-th UAV and the position of the center point of the k-th obstacle; d min Indicates the maximum safety radius of the drone; Collision avoidance constraints: For safety reasons, drones cannot collide with each other, and the relative distance between them needs to be kept within a safe range, that is: Δd ij >d min (3) Where, Δd ij =||p i -p j || represents the distance between the i-th UAV and the j-th UAV, i,j∈[1,N], i≠j; Boundary constraints: To ensure the smooth progress of the exploration mission, the drone cannot exceed the given mission area during flight, that is: 0≤p i,ξ ≤d bound,ξ (4) Among them, ξ∈[1,ξ max ] represents the motion dimension of the drone, which is used to explore the two-dimensional environment, ξ max =2.d bound,ξ represents the region boundary in the ξ dimension. i,ξ Represents the position of the drone in the ξ dimension.

4. The UAV swarm collaborative search and rescue algorithm based on reinforcement learning according to claim 1 is characterized in that: In S1, the random game model of the drone swarm search and rescue mission includes a state set S, an observation state set O, an action set A, and a reward function R; State set S: According to the specific situation of the task, the obstacle position grid map g at time t is b , the path grid map g has been found r , the detected area grid map g a , the target position p t , the position of our drone p i , speed v i and direction d i , as the state quantity s, that is: s=(g b ,g r ,g a ,p t ,p i ,v i ,d i )∈S (5) Among them, i∈[1,N], N is the number of drones; Observation state set O: For the i-th self-drone, i∈[1,N], the path grid map g found at time t is r , the detected area grid map g a , the target position p t , its own position p i , speed v i and direction d i , the relative position Δp between the drone and other drones ij As an observation state, that is: o i =(g r ,g a ,p t ,p i ,v i ,d i ,Δp ij ) (6) Action Set A: Discretize the continuous environment into a grid map, and the action space can be discretized from the continuous speed control amount to moving between the grids of the grid map, using the discrete action space based on the drone's own coordinate system; Reward function R: The ultimate goal is to light up the map and reach the target location within the specified time. A reward and punishment mechanism is established to determine the benefit value obtained by a single-step decision. For a single drone, the benefit value mainly includes exploration benefit and target benefit. Explore the benefits: Before the drone reaches the rescue target location, since the obstacle information on the map is unknown, it needs to explore the map first to find a path to the target location. The exploration process is goal-oriented. Considering the reward function with more complex obstacle distribution, the shortest path to the target is calculated through breadth-first search. The drone’s behavior is rewarded or punished based on whether it moves along the shortest path, so that the agent can learn the ability of path planning. The result returned by the breadth-first search algorithm is a matrix. If a grid value is -1, it means that the grid cannot reach the target point through any path. Otherwise, it indicates the shortest moving distance from the grid to the target point. The exploration reward is: r explore =dist[i,j]-dist[i′,j′] (7) Among them, dist is the two-dimensional matrix returned by the breadth-first search algorithm, i, j are the original coordinates of the drone, and i′, j′ are the new coordinates of the drone after performing the action; Target income: When the UAV cannot find a path to the rescue target, the negative distance between the UAV and the target is used as the reward function. When the distance between the UAV and the target is far, a large penalty is imposed, and when the distance between the UAV and the target is close, a small penalty is imposed. The target reward is: Among them, i target and j target Indicates the location of the target in the grid map; Combining equations (7) and (8), the total reward of the drone mission is designed as: r＝r explore +r target (9)。 5. The UAV swarm collaborative search and rescue algorithm based on reinforcement learning according to claim 1 is characterized in that: In S1, the drone cluster collaborative search and rescue adopts the MASAC algorithm, and the application of the MASAC algorithm is divided into two parts: iterative training and online decision-making; Iterative training: A reinforcement learning algorithm of centralized training and distributed decision-making is used to establish a decision network and an evaluation network. The decision network determines the discrete action sequence of the drone at the current moment based on the drone's own state information provided by the simulation platform, the grid map information collected during the exploration process, and the known location information of the rescue target. The evaluation network evaluates the quality of the decision results based on the state information and decision information, and adds new interactive experience to the experience database. Through the random experience playback mechanism, the experience in the database is randomly extracted to gradually train the evaluation network and the decision network, and the drone strategy is obtained through multiple iterations. Online Decision Making: In the simulation platform, the initial positions of the UAV and the target to be rescued are randomly given, the map obstacle distribution is randomly initialized, and the trained decision network is used to generate UAV decision results in real time to complete the UAV cluster motion planning for the search task.

6. The UAV swarm collaborative search and rescue algorithm based on reinforcement learning according to claim 1 is characterized in that: In S2, the conventional scene module includes an object model, a camera module, a grid map module, a random map module and a map lighting module; Object model: Unity supports importing general 3D models in 3DS and FBX formats, and uses 3D modeling tools to create and export 3D models. When building large-scale urban scenes, use Unity's CityGenerator plug-in, which uses blocks as the smallest unit to generate urban scenes of different scales by splicing blocks. Use regular expressions to find the mesh model of the corresponding building from the model library and attach it to the mesh collision body component. Camera module: Unity's scenes are created by placing and moving objects in three-dimensional space. When there are multiple cameras, multi-camera display is performed by setting the offsets of different cameras. Grid map module: In training or testing scenarios, grid maps are used to express environmental information. The environment is regarded as a two-dimensional plane, which is divided into several grids of equal size with numerical information. Each grid stores the value of the surrounding environment information, and the obstacle information in the environment is simplified into a two-dimensional array. Each element of the array represents the state of a grid area. A value of 0 represents an open space, and a value of 1 represents an obstacle. The position of objects in the Unity scene is determined by the world coordinate system, and the position of the object in the world coordinate system is represented by a triplet of the form (x, y, z), where the values ​​of x, y, and z are real numbers. The position of the grid map is determined by the grid coordinate system, and the position in the grid map is represented by a doublet of the form (i, j), where the values ​​of i and j are natural numbers. Due to the differences in the above coordinate system representation methods, calculation is required to transform the world coordinate system and the grid coordinate system. The transformation from the world coordinate system to the grid coordinate system is: Among them, (i obj ,j obj ) is the position of the object in the grid map after transformation, (x obj ,z obj ) is the position of the object in the world coordinate system, (x ref ,z ref ) is the position of the reference point in the world coordinate system, (i ref ,j ref ) is the position of the reference point in the grid map, l is the grid length of the grid map; Conversely, the grid coordinate system can be transformed to the world coordinate system by the following formula: Among them, (x obj ,y obj ,z obj ) is the position of the object in the world coordinate system after transformation, (i obj ,j obj ) is the position of the object in the grid map, (x ref ,z ref ) is the position of the reference point in the world coordinate system, (i ref ,j ref ) is the position of the reference point in the grid map, l is the grid length of the grid map; The above transformation processes all involve reference points. Different reference points will result in different coordinate transformation results. In the application, the center coordinates of the map are (x, y, z) = (0, 0, 0), (i, j) = (i max / 2,j max / 2) as a reference point, aligning the center of the world map with the center of the grid map; Random map module: During the iterative training of reinforcement learning, maps need to be randomly generated so that the agent can learn new environmental information in each iteration; Map lighting module: used to display the historical trajectory of the drone during flight and mark the points where the drone has flown on the map.

7. A virtual simulation verification method for cooperative search of drone clusters based on reinforcement learning, characterized by: The reinforcement learning-based UAV swarm collaborative search and rescue algorithm described in any one of claims 1 to 6 is used to perform overall simulation training and verification on the collaborative search and rescue algorithm design in S1 and the virtual simulation environment development in S2 through a real-time simulation architecture, and to construct an overall simulation architecture and configure a real-time data interface based on ML-Agents.

8. The UAV swarm collaborative search virtual simulation verification method based on reinforcement learning according to claim 7 is characterized by: The overall simulation architecture is constructed by using Unity rendering and physical simulation to simulate the UAV flight scene. During training, the environment generates a variety of random scenes and provides observation information of the UAV, including the position, speed, posture and perception data of the surrounding environment. The UAV executes the corresponding actions generated by the reinforcement learning algorithm based on the received observation information. The environment calculates the reward signal based on the UAV's action and current state to guide the learning process of the algorithm. This process continues in a cycle until the predetermined training convergence condition is met; On the Python reinforcement learning algorithm side, the system receives real-time observation data of the drone from the Unity simulation environment. The data is transmitted through the ML-Agents framework. After receiving the observation information, the data is input into a pre-designed deep neural network, which is the policy network. It is trained to extract features from complex observation data and output the optimal action decision to maximize the cumulative reward. Based on the reward signal obtained from the environment and the feedback of the detection mechanism, the parameters of the policy network are updated through the back-propagation algorithm, and the updated network parameters are reloaded to use the latest model in the next decision-making process.

9. The UAV swarm collaborative search virtual simulation verification method based on reinforcement learning according to claim 8 is characterized by: The ML-Agents framework enables games and virtual simulation platforms to become environments for training intelligent agents, and consists of a learning environment, a Python interface, and external communications; The operations of the learning environment on the intelligent agents in the scene include collecting observation information, executing given actions, and obtaining rewards. The learning environment components can be divided into components of different levels. The highest level is the Academy component. There can only be one Academy component in a scene, which is used to manage the entire learning environment. It is also responsible for judging whether the environment has reached the terminal state and starting a new round of learning. The middle level is the Brain component. There can be multiple Brain components under one Academy component, which is used to manage a certain type of intelligent agent. It is also responsible for processing the observation information passed in by the lower-level intelligent agents and giving action instructions. The bottom level is the Agent component. There can be multiple Agent components under one Brain component, corresponding to each intelligent agent in the scene, which is used to control the behavior of a single intelligent agent.

Images