A pheromone-inspired autonomous coverage method for multiple unmanned aerial vehicles

Abstract

This invention relates to a pheromone-inspired multi-UAV autonomous coverage method, comprising the following steps: Step S1, constructing a coverage scene map of multi-UAV autonomously collecting data on natural disaster areas; Step S2, constructing a multi-UAV path pheromone map based on a pheromone aggregation model and marking the multi-UAV paths; Step S3, defining a state function, action function, and reward function according to the multi-UAV autonomous coverage task, establishing a neural network-based multi-UAV autonomous coverage model, and training it; Step S4, performing multi-UAV autonomous coverage using the trained multi-UAV autonomous coverage model. Compared with existing technologies, this invention has the advantages of high efficiency and energy saving, high coverage, and wide applicability.

Description

Technical Field

[0001] This invention relates to the field of autonomous control technology for unmanned aerial vehicles (UAVs), and in particular to a pheromone-inspired autonomous coverage method for multiple UAVs. Background Technology

[0002] Traditional static area coverage methods, once deployed with static sensor nodes, are difficult to redeploy if coverage gaps or malfunctions occur. In contrast, dynamic area coverage offers greater flexibility and is therefore widely used in target detection, structural health monitoring, and disaster relief.

[0003] In dynamic area coverage, the operability, flexibility, and robustness of drones allow them to dynamically adjust their positions according to environmental changes, greatly improving the performance of mobile sensor networks. Dynamic coverage algorithms control drones to collect data from IoT devices or wireless sensors, achieving precise coverage of target areas. Furthermore, the cost advantages and rapid deployment capabilities of drone wireless communication systems make them more suitable for sudden events or missions with limited durations, such as emergency communication restoration after natural disasters that damage communication infrastructure, thereby enhancing the coverage and performance of communication networks in emergency communications.

[0004] Control algorithms for dynamic coverage of unmanned aerial vehicles (UAVs) are generally divided into two categories: non-self-organizing control algorithms and self-organizing control algorithms. Non-self-organizing dynamic coverage algorithms optimize the target area by modeling the configuration space to reduce redundant coverage, thus achieving efficient coverage. However, this non-self-organizing control method has low robustness; in emergency situations, UAV malfunctions or excessive distance between UAVs can lead to coverage gaps. Self-organizing control algorithms give UAVs strong robustness and good autonomy; they can solve the problem of uneven coverage in the target area in emergency situations through local information interaction. Intelligent deep reinforcement learning is considered a key technology for autonomous control of UAVs because it can handle complex state spaces and environments well. However, traditional RL methods, such as Q-Learning or policy gradient methods, are not suitable for dynamic coverage environments with multiple UAVs. This is because during training, the policy of each UAV changes continuously, resulting in the UAVs always being in a dynamic environment, making it difficult for the UAV policies to converge.

[0005] Providing long-term, effective dynamic area coverage is a significant challenge. Because UAVs are typically small to medium-sized, battery-powered devices with limited communication range, flight time, and effective payload capacity, the dynamic coverage process requires energy-efficient operation to extend network lifespan. Furthermore, UAVs operate in dynamic, open, and unpredictable environments; resource sharing can effectively reduce redundant coverage and unnecessary energy consumption by UAVs. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a pheromone-inspired autonomous coverage method for multiple unmanned aerial vehicles, which has the advantages of high efficiency and energy saving, high coverage, and wide applicability.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] According to a first aspect of the present invention, a pheromone-inspired multi-UAV autonomous coverage method is provided, the method comprising the following steps:

[0009] Step S1: Construct a coverage scene map of multiple drones autonomously collecting data on natural disaster areas;

[0010] Step S2: Based on the pheromone aggregation model, construct a multi-UAV path pheromone map and mark the multi-UAV paths;

[0011] Step S3: Based on the multi-UAV autonomous coverage task, define the state function, action function and reward function, establish a neural network-based multi-UAV autonomous coverage model and train it;

[0012] Step S4: Use the trained multi-UAV autonomous coverage model to perform multi-UAV autonomous coverage.

[0013] Preferably, the coverage scene map for autonomous collection of natural disaster area data by multiple UAVs in step S1 includes the locations of UAVs, coverage targets, and obstacles, specifically:

[0014] Let n = 1, 2, ..., N represent the numbers of each drone, and N be the total number of drones. Then the set of all drones can be represented as:

[0015] Define k = 1, 2, ..., K to represent the numbers of each disaster source in the task area, and K to represent the total number of disaster sources. The set of all disaster sources in a natural disaster area is represented as:

[0016] The coverage area is defined as a two-dimensional bounded region with length L and width W. This two-dimensional region is divided into several cells and contains several randomly generated obstacles.

[0017] Preferably, step S2 includes the following sub-steps:

[0018] Step S21: Initialization process of pheromone map;

[0019] Step S22: The aggregation process of pheromone maps.

[0020] Preferably, the pheromone initialization function for the drone's starting position is:

[0021] P e,0 =P ((x,y), (1)

[0022] Where (x,y) represents the starting position of the UAV, P e,0 This indicates the pheromone concentration during the initialization of the drone in the mission area.

[0023] Preferably, the pheromone map aggregation process specifically includes:

[0024] When the drone moves to cell i, the total amount of pheromones at that location is represented as:

[0025]

[0026] in, Let n be the amount of pheromone released by the k-th drone at cell i; n represents the total number of drones within the drone's sensing range r at cell i at time (t-1, t).

[0027] The evaporation of the trace pheromone in the i-th cell at time (t, t+1) is expressed as:

[0028]

[0029] Where η is the pheromone evaporation coefficient, η∈(0,1);

[0030] The total pheromone content in the i-th cell at time t+1 is expressed as:

[0031]

[0032] Where η is the pheromone evaporation coefficient, η∈(0,1), Increase the pheromone level of cell i at time t+1;

[0033] The value of trace pheromones is limited to [P] min ,P max Within this scope, the local pheromone update rules and the global pheromone update rules are integrated as follows:

[0034]

[0035] Preferably, the state function, action function, and reward function in step S3 are as follows:

[0036] State function: Contains the trace pheromone concentration in the cells of the task area. The remaining energy consumption of the nth UAV and the coverage status of disaster sources At time t, the state function is expressed as:

[0037]

[0038] Action function: contains the drone's flight direction Flight distance of drones At time t, the action function is expressed as:

[0039]

[0040] Reward function: includes rewards obtained by drones covering disaster sources. Rewards for hitting obstacles And rewards for overlapping coverage of a certain area At time t, the reward function is expressed as:

[0041]

[0042] Preferably, the multi-UAV autonomous coverage model based on neural networks in step S3 specifically comprises: the neural network including an action network and an evaluation network; specifically, the input to the action network is the state function of each UAV. The output is the action function for each drone. The input to the evaluation network is the joint state s(t) and the joint action a(t), and the output is the Q value, denoted as Q(s,a).

[0043] Preferably, the training process of the neural network-based multi-UAV autonomous coverage model is as follows:

[0044] 1) Randomly initialize the evaluation network Q(s,a|θ) Q ) and action network π(s|θ π ), and the corresponding weight parameter θ Q and θ π ;

[0045] 2) From the experience replay buffer A batch of samples was randomly selected from the samples, and the sample size was... One of them is (s) j ,a j ,r j ,s ′j ), where j is the sample index;

[0046] 3) Using the minimum loss function Updated drone evaluation network:

[0047]

[0048] Among them, y jThis is calculated from the Target Q network of the j-th drone. This indicates that under the joint state function s, actions a1,…,a are taken using policy π. N ) expected discount reward;

[0049] 4) Update the target network parameters corresponding to the action network and evaluation network:

[0050]

[0051] Where the parameter τ represents the learning rate, These are the target network parameters.

[0052] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.

[0053] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] 1) This invention utilizes a multi-UAV path pheromone map to effectively avoid overlapping coverage of the mission area in a short period of time, thereby minimizing the energy consumption of the UAV.

[0056] 2) The reward function designed for autonomous coverage tasks by multiple UAVs in this invention effectively guides multiple UAVs to collaboratively complete the data collection problem in natural disaster areas and solves the problem of dynamic instability in reinforcement learning of multiple UAVs;

[0057] 3) The multi-UAV autonomous coverage method proposed in this invention has the advantages of high efficiency and energy saving, high coverage and wide applicability, thereby improving the efficiency of multi-UAV disaster relief. Attached Figure Description

[0058] Figure 1 This is a schematic flowchart of the method of the present invention;

[0059] Figure 2 This is a scene diagram illustrating the autonomous data collection by multiple drones in natural disaster areas according to the present invention.

[0060] Figure 3 This is a schematic diagram of the pheromone map of the present invention;

[0061] Figure 4 This is a schematic diagram of the network structure of the present invention;

[0062] Figure 5This is a schematic diagram of the Actor network in the application scenario of this invention in natural disasters;

[0063] Figure 6 This is a schematic diagram of the Critic network in the application scenario of the present invention in natural disasters. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0065] Example

[0066] like Figure 1 As shown, this embodiment takes the autonomous collection of natural disaster area data by multiple UAVs as the research background, and is based on a pheromone-inspired autonomous coverage method for multiple UAVs. This method includes the following steps:

[0067] Step S1: Establish a multi-UAV autonomous coverage schematic diagram, including the locations of coverage targets and obstacles, and then construct a coverage scenario map of multi-UAV autonomously collecting data on natural disaster areas, such as... Figure 2 As shown, it specifically includes:

[0068] 1) Define n = 1, 2, ..., N as the ID numbers of each drone, and N as the total number of drones. Then the set of all drones can be represented as:

[0069] 2) Define k = 1, 2, ..., K as the number of each disaster source in the task area, and K as the total number of disaster sources. Then the set of all disaster sources in the natural disaster area can be represented as:

[0070] 3) Define the coverage area as a two-dimensional bounded region with length L and width W. This two-dimensional region is divided into smaller cells and contains several randomly generated obstacles.

[0071] Step S2: Based on the pheromone mechanism, design a pheromone aggregation model, construct a multi-UAV path pheromone map, and mark the multi-UAV paths. This includes the following sub-steps:

[0072] 1) Initialization process of pheromone map:

[0073] Ants use pheromones to guide other ants' foraging routes. By defining track pheromones to record the drone's travel path, this mechanism can effectively avoid repeated visits by the drone in a short period of time. Compared with exhaustive search algorithms, it can reduce the search difficulty and improve global coverage.

[0074] The pheromone initialization function for the drone's starting position is shown below:

[0075] P e,0 =P ((x,y),0) (1)

[0076] Where (x,y) represents the starting position of the UAV, P e,0 This indicates the pheromone concentration during the initialization of the drone in the mission area.

[0077] 2) The aggregation process of pheromone maps:

[0078] First, each drone can sense the concentration of trace pheromones within a certain range.

[0079] Consider any cell filled with trace pheromones as a repulsive force in the local environment, exerting a repulsive effect on nearby drones, thus effectively observing the local state space. In this case, the drone needs to select the area with the lowest trace pheromone concentration to perform its actions, indicating that the area is unvisited or rarely visited. When the drone moves to cell i, the total pheromone concentration at that location can be expressed as:

[0080]

[0081] in, Let be the amount of pheromone released by the k-th drone at cell i; n represents the total number of drones within cell i at time (t-1, t), with a radius of r.

[0082] Then, the natural process of trace pheromone aggregation includes evaporation, which means that the deposited pheromone will disappear over time. This also satisfies the requirement for the drone to collect disaster source data again after a period of time. The amount of trace pheromone evaporating in the i-th cell at time (t, t+1) can be expressed as:

[0083]

[0084] Where η is the pheromone evaporation coefficient, η∈(0,1).

[0085] Finally, the total pheromone content in the i-th cell at time t+1 can be expressed as:

[0086]

[0087] Where η is the pheromone evaporation coefficient, η∈(0,1), The pheromone increment value for cell i at time t+1.

[0088] To avoid the algorithm getting trapped in local optima and to prevent the trace pheromone from becoming negative or fluctuating excessively, the value of the trace pheromone is restricted to [P]. min ,P max Within [ ]. While increasing the search space, the algorithm maintains a fast convergence speed, and the local pheromone update rule is integrated with the global pheromone update rule as follows:

[0089]

[0090] Step S3: Based on the multi-UAV autonomous coverage task, define the state function, action function and reward function, establish a neural network-based multi-UAV autonomous coverage model and train it;

[0091] The expressions for the state function, action function, and reward function are as follows:

[0092] ① State function: Contains the trace pheromone concentration in the cells of the task area. The remaining energy consumption of the nth UAV and the coverage status of disaster sources At time t, the state function can be expressed as:

[0093]

[0094] Among them, when When, it indicates that the disaster source k has been covered; if This indicates that the source of the disaster has not been covered.

[0095] ② Action function: includes the drone's flight direction Flight distance of drones

[0096] The expression for the action function at time t is:

[0097]

[0098] in, Indicates the flight direction of drone n; This represents the flight distance of drone n. If... This indicates that the drone is hovering; if This indicates that the drone flew the maximum distance allowed by its energy.

[0099] ③ Reward function: Includes the reward obtained by drones covering disaster sources. Rewards for hitting obstacles And rewards for overlapping coverage of a certain area Three parts.

[0100] Specifically, if the disaster source is within the coverage area r of the drone, the nth UAV will receive a reward. V k This represents the value of k disaster sources within the coverage area r of the drone;

[0101] When a drone collides with an obstacle or boundary, the nth UAV receives a reward. R represents the total number of collisions with obstacles or boundaries.

[0102] When a drone selects a direction, if it chooses the area with the highest pheromones, it means that the area has been visited by multiple drones, and the nth UAV receives a reward.

[0103] Therefore, at time t, the reward function can be expressed as:

[0104]

[0105] The neural network is then used to update its parameters, resulting in a neural network-based autonomous coverage model for multiple unmanned aerial vehicles (UAVs).

[0106] like Figure 4 As shown, this network structure consists of two parts: an Actor network and a Critic network. Figure 5 As shown, the input to the Actor network is the state function of each drone. The output is the action function for each drone. like Figure 6 As shown, the input to the Critic network is the joint state s(t) and the joint action a(t), and the output is the Q value, denoted as Q(s,a0).

[0107] The training process of a neural network is as follows:

[0108] 1) Randomly initialize the Critic network Q(s,a|Q) Q ) and Actor network π(s|θ π and weighting parameter θ Q and θ π At the same time, initialize the experience replay buffer.

[0109] 2) Calculate the state s(t), action a(t), and reward function r(t) of N UAVs at time t, and calculate the state s at the next time step. ′ And store it in the experience replay buffer.

[0110] 3) From the experience replay buffer A batch of samples was randomly selected from the samples, and the sample size was... One of them is (s) j ,a j ,r j ,s ′j ), where j is the sample index;

[0111] 4) Using the minimum loss function Update the drone's Critic network:

[0112]

[0113] Where y j This is calculated from the Target Q network of the j-th drone. This indicates that under the joint state function s, actions a1,…,a are taken using policy π. N ) expected discount reward.

[0114] 5) Update the target network parameters corresponding to the Actor network and Critic network:

[0115]

[0116] Where the parameter τ represents the learning rate, Indicates the network parameters of the Target policy. This represents the network parameters of the Target policy.

[0117] Step S4: Use the trained multi-UAV autonomous coverage model to perform multi-UAV autonomous coverage.

[0118] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0119] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The processing unit executes the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S4 by any other suitable means (e.g., by means of firmware).

[0121] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0122] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-UAV autonomous coverage method based on pheromone heuristic, characterized in that, The method includes the following steps: Step S1: Construct a coverage scene map of multiple drones autonomously collecting data on natural disaster areas; Step S2: Based on the pheromone aggregation model, construct a multi-UAV path pheromone map and mark the multi-UAV paths; Step S3: Based on the multi-UAV autonomous coverage task, define the state function, action function and reward function, establish a neural network-based multi-UAV autonomous coverage model and train it; The state function, action function, and reward function are respectively: State function: pheromone concentration of the trace information contained in the task area cell , the remaining energy consumption of the aerial vehicle and the coverage state of the disaster source At the moment, the state function is expressed as: (6) Action function: contains the flight direction of the drone and the flight distance of the drone At time, the action function is represented as: (7) Reward function: including reward for the drone covering the disaster source , reward for hitting an obstacle , and reward for overlapping coverage of an area , at time, the reward function is expressed as: + (8) Among them, the reward for overlapping a certain area Negatively correlated with the total amount of pheromones in the cell; Step S4: Use the trained multi-UAV autonomous coverage model to perform multi-UAV autonomous coverage.

2. The pheromone-inspired multi-UAV autonomous coverage method according to claim 1, characterized in that, The coverage scene map of the natural disaster area data collected autonomously by multiple UAVs in step S1 includes the locations of the UAVs, the coverage targets, and obstacles, specifically: definition This indicates the serial number of each drone. Let be the total number of drones, then the set of all drones is represented as: ; definition This indicates the number of each disaster source in the mission area. The total number of disaster sources is represented as the set of all disaster sources in a natural disaster area: ; The coverage area is defined as having a length of Width is A two-dimensional bounded region, which is divided into several cells and contains several randomly generated obstacles.

3. The pheromone-inspired multi-UAV autonomous coverage method according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S21: Initialization process of pheromone map; Step S22: The aggregation process of pheromone maps.

4. The pheromone-inspired multi-UAV autonomous coverage method according to claim 3, characterized in that, The pheromone initialization function for the drone's starting position is: (1) in,( x,y This indicates the starting position of the drone. This indicates the pheromone concentration during the initialization of the drone in the mission area.

5. The pheromone-inspired multi-UAV autonomous coverage method according to claim 4, characterized in that, The aggregation process of the pheromone map is as follows: When the drone moves into the cell At that time, the total amount of pheromones at that location is expressed as: (2) in, For the first Drones in the cell The amount of pheromones released at that location; express At any moment, in the cell At this location, the radius is the drone's sensing range. The total number of drones within the area; No. The trace pheromones in each cell The amount of evaporation at any given time is expressed as: (3) in, The pheromone evaporation coefficient. ; Time of the first The total pheromone content of each cell is expressed as: (4) in, The pheromone evaporation coefficient. , For cells The increase in pheromone levels at time t+1; The value of trace pheromones is limited to... Within the system, the local pheromone update rules and the global pheromone update rules are integrated as follows: (5)。 6. The pheromone-inspired multi-UAV autonomous coverage method according to claim 1, characterized in that, The multi-UAV autonomous coverage model based on neural networks in step S3 specifically includes an action network and an evaluation network. Specifically, the input to the action network is the state function of each UAV. The output is the action function for each drone. The input to the evaluation network is the joint state. and joint actions The output is Q Value, denoted as .

7. A pheromone-inspired multi-UAV autonomous coverage method according to claim 6, characterized in that, The training process of the neural network-based multi-UAV autonomous coverage model is as follows: 1) Randomly initialize the evaluation network and Action Network ), and the corresponding weight parameters. and ; 2) From the experience replay buffer A batch of samples was randomly selected from the samples, and the sample size was... One of them is ,in For sample index; 3) Using the minimum loss function Updated drone evaluation network: (9) in, It is the first The Target Q network calculations for the drone show that... Indicates the joint state function Use strategy Take action Expected discount rewards; 4) Update the target network parameters corresponding to the action network and evaluation network: (10) Where the parameter τ represents the learning rate, These are the target network parameters.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images