Unmanned aerial vehicle cluster autonomous region search method based on reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster autonomous area search method based on reinforcement learning, and the method comprises the steps: enabling an unmanned aerial vehicle cluster to share a respective information deterministic diagram at each moment through building a communication network when executing a target search task in a task area, and obtaining the information deterministic diagram of the unmanned aerial vehicle cluster; inputting the information certainty diagram and the environment information diagram of the unmanned aerial vehicle cluster into a trained unmanned aerial vehicle search network model to obtain the action of each unmanned aerial vehicle at the next moment; the unmanned aerial vehicle executes a search task according to the output action; wherein the to-be-searched task area of the unmanned aerial vehicle and the no-fly area in the task area are subjected to gridding processing, and the environment information graph is constructed based on the cell state; assigning an attribute of information certainty degree to the cells of the task area to obtain an information certainty map of each unmanned aerial vehicle; each unmanned aerial vehicle moves a cell in the task area at each moment, and the information deterministic map is updated once;

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle path planning and maneuver decision-making, and specifically relates to an autonomous area search method for unmanned aerial vehicle clusters based on reinforcement learning, which is suitable for unmanned aerial vehicle clusters to search for unknown mobile targets in a specified area, and can automatically avoid no-fly zones to ensure the flight safety of unmanned aerial vehicles. Background Art

[0002] With the rapid development of artificial intelligence and unmanned systems, drone swarms combined with artificial intelligence technology have broad application prospects. Among them, drone swarms performing coordinated regional search is one of the main applications, showing significant advantages in large-area regional search tasks such as post-disaster rescue, environmental monitoring, military reconnaissance and agricultural census.

[0003] There are two main methods for collaborative search of multiple targets in traditional unknown areas: rule-based collaborative search method and mathematical programming-based collaborative search method. The rule-based scanning search method can cover the mission area well, but does not consider the dynamic nature of the search. When the target state, quantity, location and maneuverability are concerned, the pre-planned trajectory is less applicable. Collaborative search methods based on mathematical programming are divided into global centralized and distributed methods. The global centralized optimization method can better adapt to dynamic environments, but the centralized solution is sensitive to the size of the cluster and cannot meet the real-time requirements of large-scale clusters. The distributed solution effectively reduces the difficulty of solving problems caused by the increase in scale, but it still has a strong dependence on the rules designed in advance, and the planning effect depends to a large extent on whether the designed model is reasonable.

[0004] There are still some problems in the current research in engineering applications: there are few studies on the search for evasive moving targets. Most of the research is based on the search for static targets, and few consider the situation where the target actively avoids the drone. Search performance is related to model accuracy. The search performance of the current research scheme is directly related to the accuracy of the manually designed model. In a dynamic environment, the model accuracy is limited. The communication environment is limited in cluster tasks. The communication environment has a great impact on the method design, and the method design under limited communication conditions should be considered. Dealing with dynamic changes in cluster nodes. The addition or deletion of cluster nodes is a common situation in the execution of tasks. A distributed planning scheme that is insensitive to the number of cluster nodes is required. The use of traditional reinforcement learning methods has poor interpretability and difficult reward function design, which needs to be improved at the network level. Summary of the invention

[0005] The purpose of the present invention is to provide a UAV cluster autonomous area search method based on reinforcement learning to solve the problems existing in the prior art.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for autonomous area search of drone swarm based on reinforcement learning, comprising:

[0008] When the drone swarm performs the target search task in the mission area, it builds a communication network and shares its own information deterministic graph at each moment, thereby obtaining the information deterministic graph of the drone cluster; the information deterministic graph of the drone cluster and the environmental information graph are input into the trained drone search network model to obtain the action of each drone at the next moment; the drone performs the search task according to the output action; wherein:

[0009] Gridding the mission area to be searched by the drone and the no-fly zone therein, and constructing the environmental information map based on the cell state;

[0010] The cells in the mission area are assigned the attribute of information certainty, and the information certainty map of each UAV is obtained; each UAV moves one cell in the mission area at each moment, and the information certainty map is updated at the same time.

[0011] Furthermore, in the gridded task area, each cell has two states, and the state set is:

[0012]

[0013] The cell status Indicates prohibited entry, corresponding to no-fly zones and mission boundaries; Indicates that the cell can be entered; after assigning the cell a state, an environmental information graph is obtained.

[0014] Furthermore, the drone is regarded as a particle in the mission area, and the movement of the drone is discretized; at each moment, the state of the drone is described by position information and heading, and the detection range of the drone is subject to angle and distance restrictions; the detection direction of the drone is set to the current movement direction, and by configuring the size of the cells in the mission area, the detection range of the drone is the current cell and the three cells in front of it;

[0015] After sensing the position of the drone, the target moves to the cell area just searched by the drone to reduce the probability of being discovered; the target is set to have the ability to sense the eight neighboring cell areas around the cell and make maneuvering decisions; the target's speed is set to be lower than the drone's speed, and it performs a maneuver every T0 time; the target will calculate the cell with the largest escape fitness based on the positions of the surrounding drones sensed in the past T0 time, and move to the cell at the T0+1th time; the escape fitness of each cell is calculated as follows:

[0016]

[0017] Where ΔT is the time difference between the last time a drone appeared in the cell and the current time. ΔT = 0 means that there is a drone at the location at the current maneuvering time, and the escape fitness is 0, which means it is not suitable to move to this cell. g is the fitness attenuation factor, g∈(0,1). The escape fitness is the preset minimum fitness p min .

[0018] Furthermore, the specific setting method of the information certainty graph is:

[0019] In the information certainty graph, the attribute of the cell is the information certainty s ifo ∈[0,1], indicating the certainty of the cell state; the longer the cell has been searched by the drone for the last time, the lower the certainty of the cell is, and the more inclined to re-search; its update method is:

[0020]

[0021] Where t is the interval time between the current cell and the last search.

[0022] Furthermore, the drone search model includes a situation prediction part, an action analysis part, a survival decision part and an overall network part, wherein:

[0023] The input of the situation prediction part is the information deterministic graph of the drone cluster. The features are extracted through a 4-layer convolutional neural network, and then analyzed through two fully connected layers to output the prediction of the possible position of the target, that is, the situation prediction information. During the training stage, the situation prediction part also includes an auxiliary analyzer; the auxiliary analyzer compares the situation prediction information with the actual position of the target and feeds the result back to the situation prediction part through the back propagation algorithm.

[0024] The input of the action analysis part is the situation prediction information and information certainty graph output by the situation prediction part. The features are extracted through a 4-layer convolutional neural network, and then analyzed through two fully connected layers to obtain action analysis information.

[0025] The survival decision part takes the environment information graph as input, processes it through two fully connected layers, and outputs the survival decision information;

[0026] The overall network includes a value layer and an action advantage layer. The value layer designs the reward mechanism of the drone, and the action advantage layer determines the value corresponding to different actions of the drone. The input of the overall network is survival decision information and action judgment information, and the output is the value of different actions of each drone. The drone takes the action corresponding to the maximum value as the next move.

[0027] Furthermore, the action space of the drone is set in the action advantage layer, and the flight and steering behaviors of the drone are defined as three actions: turn left after going straight, keep the direction unchanged after going straight, and turn right after going straight; the steering angle is discretized, and the discretized action space is symmetric about 0. The action space has three action indexes. The action space can be represented as an action set, and the discrete action a can be represented as the action index n. a The linear function of is:

[0028]

[0029] Discretize the continuous action space into a set The corresponding action index is n a =0,1,2.

[0030] Furthermore, in the value layer, the rewards of the drones during network training include sparse rewards and dense rewards;

[0031] The sparse reward is divided into two parts: target capture reward and boundary penalty; the target capture reward r cap The drone receives a reward only when it finds the target within its exploration range. At other times, the reward is zero.

[0032]

[0033] Boundary Penalty secure It refers to the negative reward that the drone receives when flying outside the boundary. The purpose is to prevent the drone from flying out of the boundary or entering the no-fly zone. The boundary penalty is defined as:

[0034]

[0035] After the action is completed, a negative reward is given, and the time cost reward is designed as follows:

[0036]

[0037] Where Δt is the time difference from the last time the drone discovered the target. As the time difference increases, the cost becomes greater.

[0038] The dense reward is a probability capture reward. The probability capture reward of the drone swarm is as follows:

[0039]

[0040] in:

[0041]

[0042] In the above formula, n represents the number of drones, M represents the number of targets, k = 1, 2, ..., M; d (i,k)represents the distance between UAV i and target k, is the reward of drone i for target k, d0 is the distance threshold for calculating the reward, and the probability capture reward is not calculated beyond this distance.

[0043] Furthermore, the D3QN method is used to train the drone search network model. Before training, a data set is pre-built, and the samples in the data set are the state quantities S = (S ifo ,S env ) and the real position of each target at the corresponding time; where S ifo The information deterministic graph representing the drone cluster, S env For environmental infographics;

[0044] The samples are input into the UAV search network model, and the situation prediction part is used to obtain the situation prediction information. The information certainty graph and the situation prediction information are input into the action judgment part to obtain the action judgment information. The built environment information graph is input into the survival decision part to obtain the survival decision information. The survival decision information and the action judgment information are used as the input of the overall network. The value layer and the action advantage layer are used to calculate the value of the three actions of each UAV at the next moment. The action with the largest value is selected as the action of the UAV at the next moment. After training until the network converges, a trained UAV search network model is obtained.

[0045] Compared with the prior art, the present invention has the following technical features:

[0046] Aiming at the challenges of high dynamic environment, large-scale cluster, communication load constraint, modeling difficulty and target avoidance maneuver, the present invention provides an autonomous regional search method for unmanned aerial vehicle cluster based on reinforcement learning. The method provides that each node of the unmanned aerial vehicle should execute appropriate maneuver decision at each time step, so as to guide the unmanned aerial vehicle to efficiently complete the regional search task; the neural network structure is designed to make the training have a certain degree of interpretability, reduce the sensitivity to the reward function, and accelerate the convergence of the method; adaptive rules are generated by reinforcement learning, and environmental characteristics are fitted by machine learning to solve the problem of accuracy limitation of traditional manual modeling; a regional revisit mechanism is realized by using a dynamic information determination degree graph to solve the difficulty of dynamic target search; a distributed autonomous planning scheme is adopted to break through the limit on the number of clusters and solve the problem of sensitivity of the planning method to the number of clusters; a method involving only local information interaction is designed, which can complete the task without global information and solve the problem of limited communication distance of single units. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of map information;

[0048] Figure 2 It is a schematic diagram of information certainty;

[0049] Figure 3 is an example of an information certainty graph in one embodiment of the present invention; wherein (a) represents the information certainty of the area near the drone, (b) represents the nearby no-fly zone, and (c) and (d) are state layers corresponding to (a) and (b), respectively;

[0050] Figure 4 It is the drone search network model structure;

[0051] Figure 5 It is an implementation flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0052] Referring to the accompanying drawings, the present invention provides a method for autonomous area search of a drone cluster based on reinforcement learning, comprising the following steps:

[0053] Step 1: Grid the mission area to be searched by the drone and the no-fly zone therein, and construct an environmental information map based on the cell state.

[0054] Set the task area Area to (xmin, ymin, xmax, ymax), where xmin, ymin, xmax, and ymax are the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate, and maximum vertical coordinate of the task area, respectively. Grid the task area, such as Figure 1 As shown in the figure, the area is divided into M×N cells, and all the cells are the mission area; the cell coordinate set of the no-fly zone {(x, y)|No-fly zone} represents the position coordinates of the no-fly zone after gridding; in order to ensure that the UAV does not fly out of the mission area when performing reconnaissance missions, on the basis of the original mission area Area, two cells are expanded outward on each side as the mission boundary.

[0055] In the gridded task area, each cell has two states, and the state set is:

[0056]

[0057] The cell status Indicates that entry is prohibited (corresponding to no-fly zones and mission boundaries), Indicates that the cell can be entered; after assigning the cell a state, an environmental information graph is obtained.

[0058] Step 2: Build the motion model of the UAV and the target in the mission area.

[0059] The UAV is regarded as a particle in the mission area, and its motion is discretized. At each moment K, the state of the UAV is described by its position information and heading, and the detection range of the UAV is subject to angle and distance limitations. The detection direction of the UAV is set to the current motion direction, and the size of the cells in the mission area is configured so that the detection range of the UAV is the current cell and the three cells in front of it.

[0060] The target often adopts some maneuvering strategies to avoid being detected as much as possible; in this scheme, the target adopts a generalized avoidance strategy during training, that is, after sensing the position of the drone, the target moves to the cell area just searched by the drone to reduce the probability of being discovered; the target is set to have the ability to sense and make maneuvering decisions in the eight neighboring cell areas around the cell where it is located; the target speed is set to be lower than the drone speed, and it performs a maneuver every T0 time step; the target will calculate the cell with the largest escape fitness based on the positions of the surrounding drones sensed in the past T0 time steps, and move to this cell at the T0+1th time; the escape fitness of each cell is calculated as follows:

[0061]

[0062] Where ΔT is the time difference between the last time a drone appeared in the cell and the current time. ΔT = 0 means that there is a drone at the location at the current maneuvering time, the escape fitness is 0, and it is not suitable to move to this cell. g is the fitness decay factor, g∈(0,1). As ΔT increases, the fitness decays. If a drone has never appeared in the cell, the escape fitness is the preset minimum fitness p. min .

[0063] Practical use shows that although the real maneuvering strategy of the UAV cannot be obtained, the use of this strategy in training has good performance and can cope with various target maneuvering behaviors.

[0064] Step 3: Based on the map information, an information deterministic graph is designed to describe the interaction between the historical search path of the drone cluster and the environment.

[0065] Assign the attribute of information certainty to the cells in the mission area, and obtain the information certainty map of each UAV; record that each UAV moves one cell in the mission area at each moment, and update the information certainty map at the same time.

[0066] In the information certainty graph, the attribute of the cell is the information certainty s ifo ∈[0,1], indicating the certainty of the cell state; the longer the cell has been since it was last searched by the drone, the lower the certainty of the cell is, and the more inclined it is to search again; Figure 2As shown in the figure, the darker the color, the higher the certainty. As the time from the last search increases, the color gradually becomes lighter, indicating that the uncertainty of the corresponding position increases. Indicates that the cell has just completed the search task and the state of the cell is completely determined. The longer the time interval from the last search, the greater the uncertainty. The update method is:

[0067]

[0068] Where t is the interval time between the current cell and the last search. When the cell is not revisited for more than 100 steps, the uncertainty is considered to be the largest.

[0069] Step 4: Design the communication interaction update mechanism of the drone cluster.

[0070] There is a maximum limit to the communication distance of drones. Drones beyond the communication distance cannot exchange information. With ad hoc networking technology, even if the communication range of a single drone is exceeded, as long as they are in the same communication network, they can exchange information through multiple intermediate nodes to ensure the latest information.

[0071] Therefore, in this scheme, drone clusters are used to build a communication network. Each drone needs to maintain a routing table to share information according to the communicable drone nodes in the table; the rules for establishing a routing table with communication distance constraints are as follows:

[0072] L=C1∪C2∪…∪C n

[0073] Among them C i Represents the list of neighboring nodes within the communication range of UAV i, i,j = 1, 2, ..., n, where n is the number of UAVs.

[0074] C i ={j|d(i,j)≤d max}

[0075]

[0076] Where d(i,j) is the distance between UAV i and UAV j, d max is the maximum communication distance of the UAV, (x i ,y i )、(x j ,y j ) represent the coordinates of the cells where UAV i and UAV j are currently located.

[0077] The information content of the interaction between drones is the information deterministic graph of each drone. The information deterministic graph of drones in the communication network is updated at every moment. The information interaction update mechanism is as follows:

[0078]

[0079] Among them, S ifo Information deterministic graph representing a drone swarm, Represents the information deterministic graph of UAV i at the current moment; sends the information deterministic graph of the UAV cluster to all UAVs in the communication network to complete the information interaction between UAVs.

[0080] Step 5: Build a drone search network model, train the network model, and deploy it to each drone.

[0081] The network model of the present invention includes a situation prediction part, an action analysis part, a survival decision part and an overall network part, wherein:

[0082] The input of the situation prediction part is the information deterministic graph S of the drone cluster ifo , features are extracted through a 4-layer convolutional neural network, and then analyzed through two fully connected layers to output a prediction of the possible location of the target, namely, situation prediction information; during the training stage, the situation prediction part also includes an auxiliary analyzer; the auxiliary analyzer compares the situation prediction information with the actual location of the target and feeds the results back to the situation prediction part through the back propagation algorithm to support the training process.

[0083] The input of the action analysis part is the situation prediction information and information certainty graph output by the situation prediction part. The features are extracted through a 4-layer convolutional neural network and then analyzed through two fully connected layers to obtain the action analysis information. The function of this part is to give the action analysis information features for the current predicted situation and past actions, that is, the prediction of the information certainty graph of the drone swarm at the next moment, so as to determine the tendency of subsequent actions.

[0084] Among them, the Prelu activation function is set after the first fully connected layer of the situation prediction part and the action analysis part, and after the two fully connected layers of the survival decision part.

[0085] The survival decision part takes the environmental information graph as input, processes it through two fully connected layers, and outputs survival decision information; this module is used to ensure the survival status of the drone and avoid exceeding the boundary or entering the no-fly zone.

[0086] The overall network includes a value layer and an action advantage layer. The input of the overall network is survival decision information and action judgment information, and the output is the value of different actions of each drone. The drone takes the action corresponding to the maximum value as the action for the next move. The overall network is trained using the D3QN (Deep Double Dueling Q-Network) method.

[0087] (1) Define the state quantity S of the drone cluster = (S ifo ,S env ), where S ifo is the information deterministic graph of the updated drone cluster, reflecting the interaction between the cluster and the environment, S env It is an environmental information map, including information such as boundaries and no-fly zones.

[0088] Figure 3 It represents the information certainty and no-fly zone constraints near the drone at the current moment. The 5*5 area around the drone represents the state quantity that the drone can intercept. Based on the direction of the drone's flight, the state layers of the two channels are generated as follows: Figure 3 As shown in (c) and (d).

[0089] (2) Action Advantage Layer

[0090] The action space of the drone is set in the action advantage layer, and the flight and steering behaviors of the drone are defined as three actions: turn left after going straight, keep the direction unchanged after going straight, and turn right after going straight. The steering angle is discretized, and the discretized action space is symmetric about 0. The action space has three action indexes. The action space can be represented as an action set, and the discrete action a can be represented as the action index n. a The linear function of is:

[0091]

[0092] Discretize the continuous action space into a set The corresponding action index is n a =0,1,2, when action index 0 is selected, the corresponding drone action is to go straight and then turn left.

[0093] (3) Value layer

[0094] In the value layer, the rewards of the drone during network training include sparse rewards and dense rewards.

[0095] The sparse reward is divided into two parts: target capture reward and boundary penalty; the target capture reward r cap The drone receives a reward only when it finds the target within its exploration range. Otherwise, the reward is zero. It can be expressed as:

[0096]

[0097] Boundary Penalty secure It refers to the negative reward that the drone receives when flying outside the boundary. The purpose is to prevent the drone from flying out of the boundary or entering the no-fly zone. The boundary penalty is defined as:

[0098]

[0099] After the action is completed, a negative reward is given, and the time cost reward is designed as follows:

[0100]

[0101] Among them, Δt is the time difference from the last time the UAV discovered the target. As the time difference increases, the cost becomes greater, which encourages the UAV to find the target quickly.

[0102] The dense reward is a probability capture reward. The probability capture reward of the drone swarm is as follows:

[0103]

[0104] in:

[0105]

[0106] In the above formula, n represents the number of drones, M represents the number of targets, k = 1, 2, ..., M; d (i,k) represents the distance between UAV i and target k, is the reward of drone i for target k, d0 is the distance threshold for calculating the reward, and the probability capture reward is not calculated beyond this distance.

[0107] (4) UAV search network model training

[0108] The D3QN (Deep Double Dueling Q-Network) method is used to train the drone search network model. A data set is pre-built before training. The samples in the data set are the state quantities S = (S ifo ,S env ) and the actual position of each target at the corresponding moment.

[0109] The samples are input into the UAV search network model, and the situation prediction part is used to obtain the situation prediction information. The information certainty graph and the situation prediction information are input into the action judgment part to obtain the action judgment information. The built environment information graph is input into the survival decision part to obtain the survival decision information. The survival decision information and the action judgment information are used as the input of the overall network. The value layer and the action advantage layer are used to calculate the value (reward value) of the three actions of each UAV at the next moment. The action with the largest value is selected as the action of the UAV at the next moment. After training until the network converges, a trained UAV search network model is obtained.

[0110] Step 6: When the drone swarm performs the target search mission in the mission area, it builds a communication network to share its own information determinism graph at each moment, thereby obtaining the information determinism graph of the drone cluster; the information determinism graph and the environmental information graph of the drone cluster are input into the trained drone search network model to obtain the action of each drone at the next moment; the drone performs the search mission according to the output action, and if the target is found, the target information is sent back to the command center, or the target is marked or destroyed, until the drone runs out of energy and needs to return or the mission is terminated.

[0111] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for autonomous area search of drone clusters based on reinforcement learning, characterized in that: include: When the drone swarm performs the target search task in the mission area, it builds a communication network and shares its own information deterministic graph at each moment, thereby obtaining the information deterministic graph of the drone cluster; the information deterministic graph of the drone cluster and the environmental information graph are input into the trained drone search network model to obtain the action of each drone at the next moment; the drone performs the search task according to the output action; wherein: Gridding the mission area to be searched by the drone and the no-fly zone therein, and constructing the environmental information map based on the cell state; The cells in the mission area are assigned the attribute of information certainty, and the information certainty map of each UAV is obtained; each UAV moves one cell in the mission area at each moment, and the information certainty map is updated at the same time.

2. The autonomous area search method for drone clusters based on reinforcement learning according to claim 1 is characterized in that: In the gridded task area, each cell has two states, and the state set is: The cell status Indicates prohibited entry, corresponding to no-fly zones and mission boundaries; Indicates that the cell can be entered; after assigning the cell a state, an environmental information graph is obtained.

3. The autonomous area search method for drone clusters based on reinforcement learning according to claim 1 is characterized in that: The drone is regarded as a particle in the mission area, and the movement of the drone is discretized; at each moment, the state of the drone is described by position information and heading, and the detection range of the drone is subject to angle and distance restrictions; the detection direction of the drone is set to the current movement direction, and by configuring the size of the cells in the mission area, the detection range of the drone is the current cell and the three cells in front of it; After sensing the position of the drone, the target moves to the cell area just searched by the drone to reduce the probability of being discovered; the target is set to have the ability to sense the eight neighboring cell areas around the cell and make maneuvering decisions; the target's speed is set to be lower than the drone's speed, and it performs a maneuver every T0 time; The target will calculate the cell with the largest escape fitness based on the positions of the surrounding drones sensed in the past T0 moments, and move to this cell at the T0+1th moment; the escape fitness of each cell is calculated as follows: Where ΔT is the time difference between the last time a drone appeared in the cell and the current time. ΔT = 0 means that there is a drone at the location at the current maneuvering time, and the escape fitness is 0, which means it is not suitable to move to this cell. g is the fitness attenuation factor, g∈(0,1). The escape fitness is the preset minimum fitness p min .

4. The autonomous area search method for drone swarms based on reinforcement learning according to claim 1 is characterized in that: The specific setting method of the information certainty graph is: In the information certainty graph, the attribute of the cell is the information certainty s ifo ∈[0,1], indicating the certainty of the cell state; the longer the cell has been searched by the drone for the last time, the lower the certainty of the cell is, and the more inclined to re-search; its update method is: Where t is the interval time between the current cell and the last search.

5. The autonomous area search method for drone clusters based on reinforcement learning according to claim 1 is characterized in that: The UAV search model includes a situation prediction part, an action analysis part, a survival decision part, and an overall network part, wherein: The input of the situation prediction part is the information deterministic graph of the drone cluster. The features are extracted through a 4-layer convolutional neural network, and then analyzed through two fully connected layers to output the prediction of the possible position of the target, that is, the situation prediction information. During the training stage, the situation prediction part also includes an auxiliary analyzer; the auxiliary analyzer compares the situation prediction information with the actual position of the target and feeds the result back to the situation prediction part through the back propagation algorithm. The input of the action analysis part is the situation prediction information and information certainty graph output by the situation prediction part. The features are extracted through a 4-layer convolutional neural network, and then analyzed through two fully connected layers to obtain action analysis information. The survival decision part takes the environment information graph as input, processes it through two fully connected layers, and outputs the survival decision information; The overall network includes a value layer and an action advantage layer. The value layer designs the reward mechanism of the drone, and the action advantage layer determines the value corresponding to different actions of the drone. The input of the overall network is survival decision information and action judgment information, and the output is the value of different actions of each drone. The drone takes the action corresponding to the maximum value as the next move.

6. The autonomous area search method for drone swarms based on reinforcement learning according to claim 1 is characterized in that: The action space of the drone is set in the action advantage layer, and the flight and turning behaviors of the drone are defined as three actions: turn left after going straight, keep the direction unchanged after going straight, and turn right after going straight; The steering angle is discretized, and the discretized action space is symmetric about 0. The action space has three action indexes. The action space can be represented as an action set. The discrete action a can be represented as about the action index n. a The linear function of is: Discretize the continuous action space into a set The corresponding action index is n a =0,1,2.

7. The autonomous area search method for drone swarms based on reinforcement learning according to claim 1 is characterized in that: In the value layer, the rewards of the drone during network training include sparse rewards and dense rewards; The sparse reward is divided into two parts: target capture reward and boundary penalty; the target capture reward r cap The drone receives a reward only when it finds the target within its exploration range. At other times, the reward is zero. Boundary Penalty secure It refers to the negative reward that the drone receives when flying outside the boundary. The purpose is to prevent the drone from flying out of the boundary or entering the no-fly zone. The boundary penalty is defined as: After the action is completed, a negative reward is given, and the time cost reward is designed as follows: Where Δt is the time difference from the last time the drone discovered the target. As the time difference increases, the cost becomes greater. The dense reward is a probability capture reward. The probability capture reward of the drone swarm is as follows: in: In the above formula, n represents the number of drones, M represents the number of targets, k = 1, 2, ..., M; d (i,k) represents the distance between UAV i and target k, is the reward of drone i for target k, d0 is the distance threshold for calculating the reward, and the probability capture reward is not calculated beyond this distance.

8. The autonomous area search method for drone swarms based on reinforcement learning according to claim 1 is characterized in that: The D3QN method is used to train the drone search network model. A data set is pre-built before training. The samples in the data set are the state quantities S = (S ifo ,S env ) and the real position of each target at the corresponding time; where S ifo The information deterministic graph representing the drone cluster, S env For environmental infographics; Input the sample into the UAV search network model, use the situation prediction part to obtain the situation prediction information, input the information certainty graph and the situation prediction information into the action judgment part to obtain the action judgment information; Input the built environment information graph into the survival decision part to obtain the survival decision information; The survival decision information and action judgment information are taken as the input of the overall network. The value layer and action advantage layer are used to calculate the value of the three actions of each drone at the next moment. The action with the largest value is selected as the action of the drone at the next moment. After training until the network converges, a trained drone search network model is obtained.

Images