Data acquisition method and device of ground automatic monitoring station based on unmanned aerial vehicle cluster and deep learning algorithm

By adopting the Actor-Critic structure and Transformer network in the MADDPG model, the network structure is simplified, and the problem of slow convergence of drone cluster data acquisition algorithm is solved, the timeliness and efficiency of data acquisition is improved, and the cooperation of drone clusters is enhanced ability.

CN119067160APending Publication Date: 2024-12-03ZHENGZHOU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411260746.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing multi-agent reinforcement learning algorithm MADDPG has greatly increased training time when the agent increases and is difficult to converge, affecting the efficiency of data acquisition of drone clusters at the ground automatic monitoring station.

Method used

The MADDPG model adopts the Actor-Critic structure, each agent has Actor, Critic, Select_Actor and Select_Actor_target networks. All agents share Actor_target and Critic_target networks. The Transformer network is used to simplify the network structure to reduce state space and model scale and improve convergence speed.

Benefits of technology

It improves the timeliness of data acquisition and task completion efficiency of the automatic monitoring station of the drone cluster on the ground, reduces energy consumption, enhances the cooperation capabilities of the drone cluster, and solves the problem of slow algorithm convergence.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067160A_ABST
    Figure CN119067160A_ABST
Patent Text Reader

Abstract

The invention provides a data acquisition method and device for a ground automatic monitoring station based on an unmanned aerial vehicle cluster and a deep learning algorithm. The method comprises the following steps: establishing a model for acquiring data of the ground automatic monitoring station by the unmanned aerial vehicle cluster; building an unmanned aerial vehicle cluster collection mobile model; an MADDPG model is constructed; the MADDPG model is trained; and applying the award value after convergence, the coverage rate and the service ground user time obtained after training to an actual scene as required. When the data of the ground automatic monitoring station are collected, a self-organizing network is deployed by using the unmanned aerial vehicle cluster, so that the energy consumption is reduced while the collected data of the ground automatic monitoring station has higher timeliness. In the aspect of scene simulation, a more practical scene in which the unmanned cluster cooperatively completes data acquisition of the ground automatic monitoring station is considered, and more attention is paid to cooperative task completion of the unmanned aerial vehicle cluster and the requirement of the unmanned aerial vehicle cluster for data AoI in the task execution process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deployment of environmental ground automatic monitoring stations, and in particular to a data collection method and device for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm. Background Art

[0002] As people's awareness of environmental protection increases, they attach more and more importance to obtaining accurate and timely environmental data. Timely and effective collection of data from ground automatic monitoring stations deployed in the environment can not only understand the ecological environment information, but also help to formulate effective environmental management strategies. Existing ecological indicator monitoring is mostly manual data collection at field sites, which consumes huge manpower and material resources and affects the timeliness of data. Field collection is also dangerous. With the rapid development of Internet of Things technology and artificial intelligence, unmanned equipment can replace manual perception of environmental information for autonomous decision-making and task planning. Using drone clusters to collect data from ground automatic monitoring stations in ecological areas not only saves labor costs but also improves data collection efficiency. Finally, the drone cluster transmits the collected data to the server via satellite.

[0003] Deep reinforcement learning has been widely used when considering the control of drone swarms. In drone swarm scheduling schemes, a single drone is usually regarded as an agent. MADDPG (Multi-Agent DeepDeterministic Policy Gradient) is a multi-agent reinforcement learning algorithm. It is an extension of the DDPG (DeepDeterministic Policy Gradient) algorithm and is specifically used to solve multi-agent collaborative decision-making problems. MADDPG can achieve coordination and cooperation between agents in a multi-agent environment, and has more advantages in dealing with multi-agent collaboration and competition environments. However, with the increase in agents, the state space and model size will also increase, the training time will be greatly increased and it will be difficult to converge.

[0004] Therefore, a solution needs to be provided to address the problem of slow convergence of the MADDPG algorithm due to the increase in intelligent agents. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies in the prior art and to provide a data collection method and device for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm.

[0006] In a first aspect, the present invention provides a MADDPG model, wherein the internal network of each agent acting as an agent is an Actor-Critic structure;

[0007] Each agent has four networks: Actor, Critic, Select_Actor, and Select_Actor_target. All agents share the Actor_target network and Critic_target network; Critic and Critic_target use the Transformer network;

[0008] The MADDPG model is trained by the following steps:

[0009] ① Based on the initial parameters of the agent, form the environmental state space S of each agent and the observation value set O of the agent;

[0010] At time t, there is an environmental state S t , the agent's observation state of the environment O n,t ;

[0011] For agent i, Select_Actor obtains the selection strategy m based on the reward function and the strategy function, and obtains the observed state O after the agent is selected based on the selection strategy m. m,n,t , determine action a according to the reward function and strategy function i,t , get the new environment S t+1 And the reward for this round r t ;

[0012] Update the environment and (S t , O n,t , O m,n,t , a i,t , r t , S t+1 ) is stored as a sample in the experience playback area K;

[0013] ②Agent performs network update:

[0014] Randomly select x groups of samples from the experience replay area K and transform the agent's observation state O of the environment n,t Input into Select_Actor network to get selection strategy m;

[0015] The observed state O after the agent selection m,n,t Input into the Actor network to obtain the action strategy a of the agent cluster i,t , and then select strategy m and action strategy a i,t Input into the Critic_target network, get the reward value, and update the network parameters of Critic_target;

[0016] ③ Repeat steps ① and ②, and continue training in the environment built in step ①. Stop training when the set number of rounds is reached.

[0017] In a second aspect, the present invention provides a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm, comprising the following steps:

[0018] (1) Build a drone cluster to collect ground automatic monitoring station data model

[0019] In the three-dimensional range of the target area, M ground automatic monitoring station target points are set as to be collected, where w m (x m ,y m , 0) is the three-dimensional coordinate of the ground automatic monitoring station m, d m is the amount of data to be collected by the ground automatic monitoring station m; suppose that the UAV cluster performing the task has a total of N UAVs, H is the flight altitude of the UAV, and W i (x i ,y i , H) represents the three-dimensional position of the i-th UAV during flight; V max is the maximum moving speed of the UAV; D is the maximum communication distance between UAVs; r is the minimum anti-collision distance between UAVs; E i The energy of the drone i; SNR im is the connection signal-to-noise ratio threshold between the ground automatic monitoring station m and the UAV i;

[0020] (2) Build a drone cluster collection mobile model

[0021] Define the drone's environmental state space S:

[0022] The UAV is used as an agent. The agent interacts with the environment sequentially in T discrete time steps. At each time slice t, the agent's environmental state is S t ; Environmental status S t Including speed v n,t =[v x,n,t , v y,n,t ]、Position W i,t =[x i,t ,y i,t , H], energy E i,t and the position, velocity, and energy state of other drones;

[0023] Define the drone's observation space O:

[0024] The observation value of each UAV includes the location of other UAVs and the location of the ground automatic monitoring station and the amount of stored data. The observation value of the nth UAV is represented as O n,t =[w 1,t , w 2,t , w 3,t , …w M,t , d1,t , d 2,t , d 3,t , …d M,t , W 1,t , W 1,t , …W N,t ], the observation value of the ground automatic monitoring station m is selected as O m,n,t =[w m,t ,d m,t , W 1,t , W 1,t , …W N,t ];

[0025] Define the drone’s action space A:

[0026] At time t, after each drone obtains an observation, each agent uses strategy Π n,t Determine its own actions independently. The action of the nth drone is a n,t =[v x,n,t , v y,n,t ]; The drone updates its position W after Δt time i,t' =(x i,t +v x,n,t ×Δt,y i,t +v y,n,t ×Δt), and calculate its energy consumption, where t'=t+Δt;

[0027] Define the reward function of drone n at time T:

[0028]

[0029] Among them, E n,i is the energy consumed by drone n at the i-th moment, eAoI d,t is the eAoI of the ground automatic monitoring station m where the UAV n collects data at the tth time slot;

[0030] eAoI is a data value metric. d,t It is a data volume weighted exponential freshness metric;

[0031] The eAoI of the ground automatic monitoring station m at time t is expressed as: eAoI d,t =F(AoI d,t ).data d , where data d Indicates the data volume of the ground automatic monitoring station d, AoI d,t represents the time from the sensor generating data to the current time t; F(t) represents the natural exponential function: F(t) = αe -βt , where α is the initial value of the attenuation function and β is the attenuation constant;

[0032] The reward value of the drone swarm is expressed as:

[0033]

[0034] (3) Building the MADDPG model

[0035] Build the overall environment of the MADDPG model described in claim 1, input the location of the ground automatic monitoring station, and make the ground automatic monitoring station generate data according to the Poisson distribution model, and also input the number, speed, energy and starting position of the drones;

[0036] (4) Training the MADDPG model described in claim 1

[0037] ① Based on the initial parameters of the UAV, the environmental state space S of each UAV and the observation value set O of the UAV are formed;

[0038] At time t, there is an environmental state S t , the agent's observation state of the environment O n,t ;

[0039] For drone i, Select_Actor obtains the selected ground automatic monitoring station m according to the reward function and the strategy function, and obtains the observation state O of the drone after selection according to the selected ground automatic monitoring station m. m,n,t , determine action a according to the reward function and strategy function i,t , get the new environment S t+1 And the reward for this round r t ;

[0040] Update the environment and (S t , O n,t , O m,n,t , a i,t , r t , S t+1 ) is stored in the experience playback area K;

[0041] ②Use the drone as a proxy to update the network:

[0042] Randomly select x groups of samples from the experience playback area K and transform the drone’s observation state of the environment O n,t Input into the Select_Actor network to get the selected ground automatic monitoring station m of the drone cluster;

[0043] The observation status after selecting the ground automatic monitoring station is O m,n,t Input into the Actor network to obtain the action strategy a of the drone cluster i,t , and then m and action strategy a i,tInput into the Critic_target network, get the reward value, and update the network parameters of Critic_target;

[0044] ③ Repeat steps ① and ②, and continue training in the environment built in step ①. Stop training when the set number of rounds is reached;

[0045] (5) The converged reward value and data energy ratio obtained after the training are applied to the actual scenario as needed.

[0046] Based on the above, in step (1), the specific settings for each parameter are as follows:

[0047] Can the data from the ground automatic monitoring station be collected by the drone? The signal-to-noise ratio (SNR) from the ground automatic monitoring station m to the drone i im To determine, the calculation formula is:

[0048]

[0049] in, is the transmission power of UAV i, is the path loss between UAV i and ground automatic monitoring station m, N gw represents stable Gaussian white noise;

[0050] The specific calculation of energy loss is as follows:

[0051] I. Air-to-Air Communication Model

[0052] The communication between drones uses air-to-air channels, and the path loss between drone i and drone j is modeled as the propagation loss in free space, that is:

[0053]

[0054] Among them, d ij is the Euclidean distance between UAV i and UAV j, f a is the carrier frequency of the air-to-air channel, c is the speed of light;

[0055] II. Air-to-ground communication model

[0056] The air-to-ground channel is divided into two parts: line-of-sight (LOS) link and non-line-of-sight (NLOS) link. The path loss between UAV i and ground automatic monitoring station m is expressed as:

[0057]

[0058] Where p(LoS,θ im ) is the elevation angle θ between UAV i and user m im The probability of LoS occurring at and The path losses for line-of-sight LoS and non-line-of-sight NLoS links, respectively;

[0059] p(LoS,θ im ) is calculated as follows:

[0060]

[0061] Among them, a and b are constant landscape parameters reflecting rural and urban environments;

[0062] Path loss and It is expressed as:

[0063]

[0064] Among them, d im is the Euclidean distance between UAV i and ground automatic monitoring station m, f g is the carrier frequency of the air-to-ground channel, ηLoS and ηNLoS are the excess path losses under LOS and NLOS, respectively, depending on the environment.

[0065] Based on the above, in step (2), the calculation of each important data is as follows:

[0066] When defining the action space A of the drone, the energy consumption of the drone is related to the moving distance. Assuming that the power of the drones is the same, the energy consumption is expressed as W n,t+1 -W n ;

[0067] When defining the reward function of the drone, it is necessary to determine whether the drone is connected to the ground automatic monitoring station; the connectivity matrix function of the drone is:

[0068]

[0069] Among them, SNR ij represents the signal-to-noise ratio of wireless transmission from drone i to j. ij ≥SNR a can communicate with UAV j only when SNR im ≥SNR g Only when UAV i can serve the ground automatic monitoring station m.

[0070] In a third aspect, the present invention provides a data acquisition device for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm, comprising:

[0071] one or more processors;

[0072] a memory for storing one or more programs,

[0073] When the one or more programs are executed by the one or more processors, the one or more processors execute the steps of a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm.

[0074] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, characterized in that when the program is executed by a processor, the steps of a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm are implemented.

[0075] The present invention has outstanding substantive features and significant progress compared to the prior art. Specifically, the present invention has the following beneficial effects:

[0076] 1. When collecting data from ground automatic monitoring stations, the present invention considers using drone clusters to deploy self-organizing networks, so that the collected ground automatic monitoring station data has higher timeliness while reducing energy consumption, thereby improving the efficiency of task completion.

[0077] 2. In terms of simulation scenarios, the present invention considers a more practical scenario in which unmanned clusters collaborate to complete data collection at ground automatic monitoring stations, pays more attention to the cooperation of drone clusters to complete tasks, and pays more attention to the requirements of drone clusters for data AoI in the process of executing tasks. The multi-agent deep deterministic policy gradient algorithm is improved. To address the problem of slow convergence of the algorithm, its internal network structure is simplified, and a small amount of Critic networks are used to evaluate the Actor network, which reduces the network scale, speeds up the convergence speed, and improves the cooperation ability of drone swarms, so that the collected ground automatic monitoring station data has higher timeliness while reducing energy consumption and improving the efficiency of task completion.

[0078] 3. The present invention designs and improves MADDPG based on the concept of Transformer, and constructs four networks, Actor, Critic, Select_Actor and Select_Actor_target, for each agent. All agents share the Actor_target network and the Critic_target network, which improves the cooperative collection of ground automatic monitoring station data by drone groups and reduces the network scale. The Critic and Critic_target use the Transformer network architecture to improve the evaluation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Flow chart of the method of Example 1.

[0080] Figure 2 This is the internal structure diagram of the intelligent agent. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] Example 1

[0083] like Figure 1 As shown, this embodiment proposes a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm, comprising the following steps:

[0084] (1) Build a drone cluster to collect ground automatic monitoring station data model

[0085] In the three-dimensional range of the target area, M ground automatic monitoring station target points are set as to be collected, where w m (x m ,y m , 0) is the three-dimensional coordinate of the ground automatic monitoring station m, d m is the amount of data to be collected by the ground automatic monitoring station m; suppose that the UAV cluster performing the task has a total of N UAVs, H is the flight altitude of the UAV, and W i (x i ,y i , H) represents the three-dimensional position of the i-th UAV during flight; V max is the maximum moving speed of the UAV; D is the maximum communication distance between UAVs; r is the minimum anti-collision distance between UAVs; E i The energy of the drone i; SNR im is the connection signal-to-noise ratio threshold between the ground automatic monitoring station m and the UAV i.

[0086] The specific settings for each parameter are as follows:

[0087] Can the data from the ground automatic monitoring station be collected by the drone? The signal-to-noise ratio (SNR) from the ground automatic monitoring station m to the drone i im To determine, the calculation formula is:

[0088]

[0089] in, is the transmission power of UAV i, is the path loss between UAV i and ground automatic monitoring station m, N gw Represents a steady white Gaussian noise.

[0090] The specific calculation of energy loss is as follows:

[0091] I. Air-to-Air Communication Model

[0092] The communication between drones uses air-to-air channels, and the path loss between drone i and drone j is modeled as the propagation loss in free space, that is:

[0093]

[0094] Among them, d ij is the Euclidean distance between UAV i and UAV j, f a is the carrier frequency of the air-to-air channel, c is the speed of light;

[0095] II. Air-to-ground communication model

[0096] The air-to-ground channel is divided into two parts: line-of-sight (LOS) link and non-line-of-sight (NLOS) link. The path loss between UAV i and ground automatic monitoring station m is expressed as:

[0097]

[0098] Where p(LoS,θ im ) is the elevation angle θ between UAV i and user m im The probability of LoS occurring at and The path losses for line-of-sight LoS and non-line-of-sight NLoS links, respectively;

[0099] p(LoS,θ im ) is calculated as follows:

[0100]

[0101] Among them, a and b are constant landscape parameters reflecting rural and urban environments;

[0102] Path loss and It is expressed as:

[0103]

[0104] Among them, d im is the Euclidean distance between UAV i and ground automatic monitoring station m, f g is the carrier frequency of the air-to-ground channel, ηLoS and ηNLoS are the excess path losses under LOS and NLOS, respectively, depending on the environment.

[0105] (2) Build a drone cluster collection mobile model

[0106] Define the drone's environmental state space S:

[0107] The UAV is used as an agent. The agent interacts with the environment sequentially in T discrete time steps. At each time slice t, the agent's environmental state is S t ; Environmental status S t Including speed v n,t =[v x,n,t , v y,n,t ]、Position W i,t =[x i,t ,y i,t , H], energy E i,t and the position, velocity, and energy status of other drones.

[0108] Define the drone's observation space O:

[0109] The observation value of each UAV includes the location of other UAVs and the location of the ground automatic monitoring station and the amount of stored data. The observation value of the nth UAV is represented as O n,t =[w 1,t , w 2,t , w 3,t , …w M,t , d 1,t , d 2,t , d 3,t , …d M,t , W 1,t , W 1,t , …W N,t ], the observation value of the ground automatic monitoring station m is selected as O m,n,t =[w m,t ,d m,t , W 1,t , W 1,t , …W N,t ].

[0110] Define the drone’s action space A:

[0111] At time t, after each drone obtains an observation, each agent uses strategy Π n,t Determine its own actions independently. The action of the nth drone is a n,t =[v x,n,t , v y,n,t ]; The drone updates its position W after Δt time i,t' =(x i,t +v x,n,t ×Δt,y i,t +v y,n,t ×Δt), and calculate its energy consumption, where t'=t+Δt.

[0112] Define the reward function of drone n at time T:

[0113]

[0114] Among them, E n,i is the energy consumed by drone n at the i-th moment, eAoI d,t is the eAoI of the ground automatic monitoring station m where the UAV n collects data in the tth time slot.

[0115] The reward value of the drone swarm is expressed as:

[0116]

[0117] r n,T is the reward value of drone n at time T.

[0118] The calculation of each important data is as follows:

[0119] When defining the action space A of the drone, the energy consumption of the drone is related to the moving distance. Assuming that the power of the drones is the same, the energy consumption is expressed as W n,t+1 -W n ;

[0120] AoI (Age of Information) is a metric for measuring the timeliness of data collection. AoI is defined as the time from when the sensor generates the latest information to when the drone uses the information. However, AoI only captures the freshness of the data without considering the amount of data. Therefore, a new data value metric eAoI (exponential-AoI) is proposed. The eAoI of the ground automatic monitoring station m at time t is expressed as: eAoI d,t =F(AoI d,t ).data d , where data d represents the amount of data at monitoring station d, AoI d,t represents the time from the sensor generating data to the current time t; F(t) represents the natural exponential function: F(t) = αe -βt , where α is the initial value of the decay function and β is the decay constant. In short, eAoI d,t It is a data volume-weighted exponential freshness metric.

[0121] When defining the reward function of the drone, it is necessary to determine whether the drone is connected to the ground automatic monitoring station; the connectivity matrix function of the drone is:

[0122]

[0123] Among them, SNR ij represents the signal-to-noise ratio of wireless transmission from drone i to j. ij ≥SNR acan communicate with UAV j only when SNR im ≥SNR g Only when , drone i can serve user m.

[0124] (3) Building the MADDPG model

[0125] Build the overall environment of the MADDPG model, input the location of the ground automatic monitoring station, and make the ground automatic monitoring station generate data according to the Poisson distribution model, and also input the number, speed, energy and starting position of the drones; Figure 2 As shown, the internal network of each intelligent agent is an Actor-Critic structure; each agent has four networks: Actor, Critic, Select_Actor and Select_Actor_target, and all agents share the Actor_target network and the Critic_target network.

[0126] (4) Training the MADDPG model

[0127] ① Based on the initial parameters of the UAV, the environmental state space S of each UAV and the observation value set O of the UAV are formed;

[0128] At time t, there is an environmental state S t , the agent's observation state of the environment O n,t ;

[0129] For drone i, Select_Actor obtains the selected ground automatic monitoring station m according to the reward function and the strategy function, and obtains the observation state O of the drone after selection according to the selected ground automatic monitoring station m. m,n,t , determine action a according to the reward function and strategy function i,t , get the new environment S t+1 And the reward for this round r t ;

[0130] Update the environment and (S t , O n,t , O m,n,t , a i,t , r t , S t+1 ) is stored in the experience replay area K.

[0131] ②Use the drone as a proxy to update the network:

[0132] Randomly select x groups of samples from the experience playback area K and transform the drone’s observation state of the environment O n,t Input into the Select_Actor network to get the selected ground automatic monitoring station m of the drone cluster;

[0133] The observation status after selecting the ground automatic monitoring station is O m,n,t Input into the Actor network to obtain the action strategy a of the drone cluster i,t , and then m and action strategy a i,t Input into the Critic_target network, get the reward value, and update the network parameters of Critic_target.

[0134] ③ Repeat steps ① and ②, and continue training in the environment built in step ①. Stop training when the set number of rounds is reached.

[0135] (5) The converged reward value and data energy ratio obtained after the training are applied to the actual scenario as needed.

[0136] In actual use, the pseudo code of the MADDPG model algorithm is:

[0137] initialization:

[0138] For each agent i:

[0139] Initialize actor network select_π i ,π i and critic network Q i , and their target network select_π' i π' i and Q' i .

[0140] Initialize the experience replay buffer D.

[0141] Repeat (for each episode):

[0142] Initialize the environment state S

[0143] Repeat (for each step):

[0144] For each agent i:

[0145] According to the current strategy select_π i and state S observes O i , get the status after selecting the ground automatic monitoring station o i

[0146] According to the current strategy π i and the selected state S observes o i , select action a i

[0147] Execute all the agent's actions [a1,...,aN ], observe the new state S' and reward R

[0148] For each agent i, transform (t = (O i ,o i ,a i ,R i ,O' i ))Store in D

[0149] For each agent i:

[0150] Randomly sample a batch of transformations from D

[0151] For each sample conversion:

[0152] Calculate the target Q value using the target network

[0153] Update critic network Q i , minimize the loss: L = (Q i (o i ,a i )-Q) 2

[0154] Update actor network select_π i ,π i Using Policy Gradients

[0155] For each agent i:

[0156] Soft update target network parameters: π' i ←τπ i +(1-τ)π' i

[0157] select_π' i ←τselect_π i +(1-τ)select_π' i

[0158] Q' i ←τQ i +(1-τ)Q' i

[0159] Until the environment ends this episode

[0160] Repeat until the termination condition is met.

[0161] Example 2

[0162] This embodiment provides a data collection device for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm, including:

[0163] one or more processors;

[0164] a memory for storing one or more programs,

[0165] When the one or more programs are executed by the one or more processors, the one or more processors execute the steps of a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm as described in Example 1.

[0166] Example 3

[0167] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm as described in Example 1.

[0168] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer non-transient readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0169] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A MADDPG model, Features: The internal network of each agent is an Actor-Critic structure; Each agent has four networks: Actor, Critic, Select_Actor, and Select_Actor_target. All agents share the Actor_target network and Critic_target network; Critic and Critic_target use the Transformer network; The MADDPG model is trained by the following steps: ① Based on the initial parameters of the agent, form the environmental state space S of each agent and the observation value set O of the agent; At time t, there is an environmental state S t , the agent's observation state of the environment O n,t ; For agent i, Select_Actor obtains the selection strategy m based on the reward function and the strategy function, and obtains the observed state O after the agent is selected based on the selection strategy m. m,n,t , determine action a according to the reward function and strategy function i,t , get the new environment S t+1 And the reward for this round r t ; Update the environment and (S t , O n,t , O m,n,t , a i,t , r t , S t+1 ) is stored as a sample in the experience playback area K; ②Agent performs network update: Randomly select x groups of samples from the experience replay area K and transform the agent's observation state O of the environment n,t Input into Select_Actor network to get selection strategy m; The observed state O after the agent selection m,n,t Input into the Actor network to obtain the action strategy a of the agent cluster i,t , and then select strategy m and action strategy a i,t Input into the Critic_target network, get the reward value, and update the network parameters of Critic_target; ③ Repeat steps ① and ②, and continue training in the environment built in step ①. Stop training when the set number of rounds is reached.

2. A data collection method for ground automatic monitoring stations based on drone clusters and deep learning algorithms, It is characterized in that The following steps are involved: (1) Build a drone cluster to collect ground automatic monitoring station data model In the three-dimensional range of the target area, M ground automatic monitoring station target points are set as to be collected, where w m (x m ,y m , 0) is the three-dimensional coordinate of the ground automatic monitoring station m, d m is the amount of data to be collected by the ground automatic monitoring station m; suppose that the UAV cluster performing the task has a total of N UAVs, H is the flight altitude of the UAV, and W i (x i ,y i , H) represents the three-dimensional position of the i-th UAV during flight; V max is the maximum moving speed of the UAV; D is the maximum communication distance between UAVs; r is the minimum anti-collision distance between UAVs; E i The energy of the drone i; SNR im is the connection signal-to-noise ratio threshold between the ground automatic monitoring station m and the UAV i; (2) Build a drone cluster collection mobile model Define the drone's environmental state space S: The UAV is used as an agent. The agent interacts with the environment sequentially in T discrete time steps. At each time slice t, the agent's environmental state is S t ; Environmental status S t Including speed v n,t =[v x,n,t , v y,n,t ]、Position W i,t =[x i,t ,y i,t , H], energy E i,t and the position, velocity, and energy state of other drones; Define the drone's observation space O: The observation value of each UAV includes the location of other UAVs and the location of the ground automatic monitoring station and the amount of stored data. The observation value of the nth UAV is represented as O n,t =[w 1,t , w 2,t , w 3,t , …w M,t , d 1,t , d 2,t , d 3,t , …d M,t , W 1,t , W 1,t , …W N,t ], the observation value of the ground automatic monitoring station m is selected as O m,n,t =[w m,t ,d m,t , W 1,t , W 1,t , …W N,t ]; Define the drone’s action space A: At time t, after each drone obtains an observation, each agent uses the strategy ∏ n,t Determine its own actions independently. The action of the nth drone is a n,t =[v x,n,t , v y,n,t ]; The drone updates its position W after Δt time i,t' =(x i,t +v x,n,t ×Δt,y i,t +v y,n,t ×Δt), and calculate its energy consumption, where t'=t+Δt; Define the reward function of drone n at time T: Among them, E n,i is the energy consumed by drone n at the i-th moment, eAoI d,t is the eAoI of the ground automatic monitoring station m where the UAV n collects data at the tth time slot; eAoI is a data value metric. d,t It is a data volume weighted exponential freshness metric; The eAoI of the ground automatic monitoring station m at time t is expressed as: eAoI d,t =F(AoI d,t ).data d , where data d Indicates the data volume of the ground automatic monitoring station d, AoI d,t represents the time from the sensor generating data to the current time t; F(t) represents the natural exponential function: F(t) = αe -βt , where α is the initial value of the attenuation function and β is the attenuation constant; The reward value of the drone swarm is expressed as: (3) Building the MADDPG model Build the overall environment of the MADDPG model described in claim 1, input the location of the ground automatic monitoring station, and make the ground automatic monitoring station generate data according to the Poisson distribution model, and also input the number, speed, energy and starting position of the drones; (4) Training the MADDPG model described in claim 1 ① Based on the initial parameters of the UAV, the environmental state space S of each UAV and the observation value set O of the UAV are formed; At time t, there is an environmental state S t , the agent's observation state of the environment O n,t ; For drone i, Select_Actor obtains the selected ground automatic monitoring station m according to the reward function and the strategy function, and obtains the observation state O of the drone after selection according to the selected ground automatic monitoring station m. m,n,t , determine action a according to the reward function and strategy function i,t , get the new environment S t+1 And the reward for this round r t ; Update the environment and (S t , O n,t , O m,n,t , a i,t , r t , S t+1 ) is stored in the experience playback area K; ②Use the drone as a proxy to update the network: Randomly select x groups of samples from the experience playback area K and transform the drone’s observation state of the environment O n,t Input into the Select_Actor network to get the selected ground automatic monitoring station m of the drone cluster; The observation status after selecting the ground automatic monitoring station is O m,n,t Input into the Actor network to obtain the action strategy a of the drone cluster i,t , and then m and action strategy a i,t Input into the Critic_target network, get the reward value, and update the network parameters of Critic_target; ③ Repeat steps ① and ②, and continue training in the environment built in step ①. Stop training when the set number of rounds is reached; (5) The converged reward value and data energy ratio obtained after the training are applied to the actual scenario as needed.

3. The data collection method of the ground automatic monitoring station based on the drone cluster and deep learning algorithm according to claim 2, It is characterized in that In step (1), the specific settings for each parameter are as follows: Can the data from the ground automatic monitoring station be collected by the drone? The signal-to-noise ratio (SNR) from the ground automatic monitoring station m to the drone i im To determine, the calculation formula is: in, is the transmission power of UAV i, is the path loss between UAV i and ground automatic monitoring station m, N gw represents stable Gaussian white noise; The specific calculation of energy loss is as follows: I. Air-to-Air Communication Model The communication between drones uses air-to-air channels, and the path loss between drone i and drone j is modeled as the propagation loss in free space, that is: Among them, d ij is the Euclidean distance between UAV i and UAV j, f a is the carrier frequency of the air-to-air channel, c is the speed of light; II. Air-to-ground communication model The air-to-ground channel is divided into two parts: line-of-sight (LOS) link and non-line-of-sight (NLOS) link. The path loss between UAV i and ground automatic monitoring station m is expressed as: Where p(LoS,θ im ) is the elevation angle θ between UAV i and user m im The probability of LoS occurring at and The path losses for line-of-sight LoS and non-line-of-sight NLoS links, respectively; p(LoS,θ im ) is calculated as follows: Among them, a and b are constant landscape parameters reflecting rural and urban environments; Path loss and It is expressed as: Among them, d im is the Euclidean distance between UAV i and ground automatic monitoring station m, f g is the carrier frequency of the air-to-ground channel, ηLoS and ηNLoS are the excess path losses under LOS and NLOS, respectively, depending on the environment.

4. The data collection method of the ground automatic monitoring station based on the drone cluster and deep learning algorithm according to claim 3, It is characterized in that In step (2), the calculation of each important data is as follows: When defining the action space A of the drone, the energy consumption of the drone is related to the moving distance. Assuming that the power of the drones is the same, the energy consumption is expressed as W n,t+1 -W n ; When defining the reward function of the drone, it is necessary to determine whether the drone is connected to the ground automatic monitoring station; the connectivity matrix function of the drone is: Among them, SNR ij represents the signal-to-noise ratio of wireless transmission from drone i to j. ij ≥SNR a can communicate with UAV j only when SNR im ≥SNR g Only when UAV i can serve the ground automatic monitoring station m.

5. A data acquisition device for ground automatic monitoring stations based on drone clusters and deep learning algorithms, It is characterized in that include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the steps of a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, It is characterized in that When the program is executed by a processor, the steps of a data collection method for a ground automatic monitoring station based on a drone cluster and a deep learning algorithm as described in any one of claims 1 to 4 are implemented.