UAV swarm intelligence perception scheduling method, system, equipment and medium

Abstract

This invention provides a method, system, device, and medium for swarm sensing and scheduling of unmanned aerial vehicles (UAVs), relating to the fields of swarm sensing and reinforcement learning. The method comprises: Step S1: modeling the UAV scheduling problem as a constrained cooperative Markov game, designing state, action, reward, and loss functions; Step S2: designing a multi-task, multi-agent reinforcement learning framework that uses a common strategy to handle similar UAV sensing tasks, reducing the computational and sample complexity of the training strategy; and Step S3: proposing a primal-dual multi-agent reinforcement learning training algorithm that alternates between improving overall sensing benefit and reducing constraint violations for each task. This invention effectively combines movement, sensing, and charging decisions, maximizing the long-term sensing benefit of UAVs within the constraints of a charging budget.

Description

Technical Field

[0001] The present invention relates to the technical field of swarm perception and reinforcement learning, and specifically to a method and system for swarm perception scheduling of unmanned aerial vehicles (UAVs) based on multi-task constrained multi-agent reinforcement learning, and more particularly to a method, system, device and medium for swarm perception scheduling of UAVs. Background Art

[0002] Today, unmanned aerial vehicles (UAVs) equipped with specialized sensors (such as visual cameras, lidars, and thermal imagers) have been widely used to perform various perception tasks (such as event security, disaster relief, and industrial inspections). In such tasks, the locations where events (such as accidents and failures) occur are usually regarded as points of interest (POIs), which are often difficult to predict and need to be perceived in a timely manner. Compared with humans and land vehicles, drones for crowd sensing have the following advantages. On the one hand, drones can be deployed in dangerous or remote areas that are inaccessible to humans and land vehicles, such as floods and mountainous areas. On the other hand, due to their flexible maneuverability, drones are more efficient than humans in performing wide-area sensing tasks, such as power line inspection and wildlife monitoring.

[0003] In drone perception systems, perception results and charging costs are two major concerns. To jointly optimize these two objectives, drone scheduling is crucial, as it determines the drone's trajectory, thereby affecting its energy consumption and the coverage of POIs. Furthermore, perception requires the drone's sensors (such as cameras and lidar) to remain active and pre-process the sensor data, which, like movement, consumes energy. Therefore, whether or not perception is performed will affect the drone's operating time and ultimately its future data collection potential. When the battery is low, the drone needs to recharge at a charging station, which incurs charging costs for the drone platform. However, the charging time of a drone may be long, resulting in some POIs not being able to be perceived in a timely manner. Therefore, for drones, when to recharge and whether to continue charging after a period of charging requires foresight.

[0004] This paper aims to design a mechanism to help drones make joint movement, perception, and charging decisions, optimizing the perception results while satisfying the charging budget constraint. This paper proposes a novel multi-agent reinforcement learning (MARL) method that treats each drone as an agent and trains a distributed policy for each agent. However, designing such a multi-agent reinforcement learning method faces the following two challenges. The first challenge is that drone systems in the real world often face a variety of perception tasks, such as monitoring wildlife in nature reserves, detecting emergencies of major events, and inspecting defects in power lines. Traditional methods learn a separate policy for each perception task, which is computationally expensive and sample-inefficient. Another challenge is how to optimize long-term perception results while ensuring that the policy satisfies the charging budget constraint. Traditional methods often only optimize the perception results of a single decision or fail to consider the charging budget constraint. Summary of the Invention

[0005] In response to the deficiencies in the prior art, the present invention provides a method, system, device and medium for swarm intelligence perception and scheduling of drones.

[0006] According to the present invention, a method, system, device and medium for swarm intelligence perception and scheduling of drones are provided, and the scheme is as follows:

[0007] In a first aspect, a method for swarm intelligence perception and scheduling of drones is provided, the method comprising:

[0008] Step S1: Model the UAV scheduling problem as a constrained cooperative Markov game. Design the state, action, reward, and loss functions to solve an optimal scheduling strategy that maximizes the long-term perception benefits of the UAVs under the constraints of the platform charging budget.

[0009] Step S2: Design a multi-task multi-agent reinforcement learning framework to represent the neural network structure of the optimal strategy. The framework can handle a large number of similar drone perception tasks and reduce the computational complexity and sample complexity of the training strategy.

[0010] Step S3: Propose a primal-dual multi-agent reinforcement learning training algorithm that alternates between improving the overall perception gain and reducing the constraint violation of each task;

[0011] Step S4: Deploy the trained strategy to the drone; the drone inputs global information and local observations into the strategy, and the strategy outputs the drone's next movement direction and distance, as well as whether to collect sensory data and charge during movement; the drone executes the results of the strategy output and repeats this process until the entire decision-making process is completed.

[0012] Preferably, the step S1 includes:

[0013] Step S1.1: Consider each UAV i in the UAV perception system as an agent in a constrained cooperative Markov game, and each agent has its own strategy π i , use N to represent the set of all intelligent agents;

[0014] Step S1.2: At the beginning of each time slot t, state s t The time slot index t, the platform remaining budget and the global information g t Composition, global information g t Contains the current location of POI, drone and charging station;

[0015] Step S1.3: At the beginning of each time slot t, the agents that are not in the charging state can take actions. The set of these agents is denoted as N t , the action of each agent Including the direction and distance of movement, whether to perceive, whether to charge, the joint action of the agent in time period t is recorded as

[0016] Step S1.4: At the beginning of each time slot t, each agent i receives an observation Among them, g t is global information, is a local information vector containing the residual energy of agent i and the positions of nearby obstacles that agent i can reach in a time slot. The joint observation of the agent in time period t is expressed as

[0017] Step S1.5: Policy π for each agent i i Assign agent i to a given observation at time slot t Take every action Probability The joint strategy of all agents is expressed as π = [π i ] i∈N ;

[0018] Step S1.6: At the end of each time slot t, each agent i receives an immediate reward is the amount of data perceived by agent i at time slot t; the immediate cost received is the charging cost of the agent in time slot t. The platform has a budget B for system charging cost for each task k. k ;

[0019] Step S1.7: Solve a general scheduling policy π that jointly makes movement, sensing, and charging decisions to maximize the sum of the expected sensing benefits of all tasks and satisfy the charging budget constraint of each task, that is,

[0020]

[0021]

[0022] in, represents the system's expected cumulative perceived benefit; represents the system's expected cumulative charging cost, and K represents the set of all tasks.

[0023] Preferably, step S2 includes:

[0024] Step S2.1: In the multi-task multi-agent reinforcement learning framework, each agent i has a policy network π i , a rewarded reviewer network and a loss critic function The training process involves interacting with each task k times to collect experience, and then using this experience to update the parameters of the policy and critic networks;

[0025] Design a parallel sampling architecture that replicates the policy networks of all agents for each task k and performs the experience collection phase for all tasks in parallel;

[0026] Step S2.2: At the beginning of each time slot t, the policy network of each agent i sends the global information g t and the local information vector As input;

[0027] Discretize the perception area into G grids, g t It consists of the number of POIs, UAVs, and charging stations in each grid in G, g t It is a tensor and input into the convolution layer, and then the spatial features are transformed into The output is:

[0028]

[0029] Where f is the ReLU activation function, W1 is the weight matrix of the MLP layer, and W c is the convolution kernel, * indicates the convolution operation;

[0030] The policy network of agent i converts the local information vector as input and feed it into a series of MLP layers to obtain a single feature

[0031]

[0032] Among them, W2 and W3 are the weight matrices of the MLP layer;

[0033] Secondly, and concatenate and further feed into the MLP layer to obtain the intermediate representation for:

[0034]

[0035] Among them, W4 is the weight matrix of the MLP layer, || represents the connection operation; A mask representing the action;

[0036] Finally, the mask vector is fed into the Softmax layer to generate i Generate action distribution on:

[0037]

[0038] Among them, W5 is the weight matrix of the MLP layer, It means element-wise addition;

[0039] Step S2.3: Using the global state s as the input to the critic network for each agent i, the last layer of the reward critic network converts the intermediate representation The state-value function that maps to a scalar reward is:

[0040]

[0041] where W6 is the weight matrix of the MLP layer; similarly, the last layer of the loss critic network maps The state value function of the estimated loss is:

[0042]

[0043] Among them, W7 is the weight matrix of the MLP layer.

[0044] Preferably, step S3 includes:

[0045] Step S3.1: Use the Lagrange multiplier method to transform the original constrained optimization problem into the following max-min problem:

[0046]

[0047] Among them, θ is the parameter of the strategy π, which is the original variable; is a non-negative dual variable;

[0048] To solve the max-min problem, an iterative algorithm is designed that alternates between gradient ascent steps on θ and gradient descent steps on λ.

[0049] Step S3.2: Update the dual variables;

[0050] In each cycle h, the dual variable is updated by projected gradient descent as:

[0051]

[0052] where η k is the step size, Γ[·] is the projection of each dual variable to [0, λ max ] projection operator within the range;

[0053] Step S3.3: Update strategy parameters;

[0054] In each cycle h, given the updated dual variable λ h , by stochastic gradient ascent h The parameters of θ h Update to θ h+1 :

[0055]

[0056] Among them, ρ is the parameter update step size, is the gradient of the objective function with respect to the policy:

[0057]

[0058] Among them, D k represents the set of samples, b k Indicates D k One of the samples in , and:

[0059]

[0060] in, and denote the reward- and cost-related advantage functions of agent j∈N respectively:

[0061]

[0062]

[0063] Step S3.4: Update the commentator network parameters;

[0064] In each cycle h, the parameters of the agent’s reward critic network are adjusted from Update to As shown in the following formula:

[0065]

[0066] in, is the reward state value function of agent i;

[0067] Similarly, the parameters of the critic network are calculated by minimizing the agent cost from ξ h Updated to h+1 :

[0068]

[0069] in, is the loss state value function of agent i.

[0070] Preferably, step S4 includes:

[0071] Step S4.1: Deploy the trained policy to the drone;

[0072] Step S4.2: At each decision time t, each UAV i receives the global information g from the platform t and your own local observations of the environment Input to strategy π i In the strategy, the next moment’s moving direction of the drone is output and distance And whether to collect sensory data and charge during movement;

[0073] Step S4.3: The drone executes the output of the strategy and moves in the direction Moving distance Repeat this process until the entire decision-making process is completed.

[0074] In a second aspect, a drone swarm intelligence perception and scheduling system is provided, the system comprising:

[0075] Module M1: Model the drone scheduling problem as a constrained cooperative Markov game. Design the state, action, reward, and loss functions to solve an optimal scheduling strategy that maximizes the long-term perception benefits of drones under the constraints of the platform charging budget.

[0076] Module M2: Design a multi-task multi-agent reinforcement learning framework to represent the neural network structure of the optimal policy. The framework can handle a large number of similar drone perception tasks and reduce the computational complexity and sample complexity of the training policy.

[0077] Module M3: Proposes a primal-dual multi-agent reinforcement learning training algorithm that alternates between improving the overall perception gain and reducing the constraint violations of each task;

[0078] Module M4: Deploy the trained strategy to the drone; the drone inputs global information and local observations into the strategy, and the strategy outputs the drone's next movement direction and distance, as well as whether to collect sensory data and charge during movement; the drone executes the results of the strategy output and repeats this process until the entire decision-making process is completed.

[0079] Preferably, the module M1 includes:

[0080] Module M1.1: Consider each drone i in the drone perception system as an agent in a constrained cooperative Markov game, each with its own strategy π i , use N to represent the set of all intelligent agents;

[0081] Module M1.2: At the beginning of each time slot t, state s t The time slot index t, the platform remaining budget and the global information g t Composition, global information g t Contains the current location of POI, drone and charging station;

[0082] Module M1.3: At the beginning of each time slot t, agents that are not in the charging state can take actions. The set of these agents is denoted as N t , the action of each agent Including the direction and distance of movement, whether to perceive, whether to charge, the joint action of the agent in time period t is recorded as

[0083] Module M1.4: At the beginning of each time slot t, each agent i receives an observation Among them, g t is global information, is a local information vector containing the residual energy of agent i and the positions of nearby obstacles that agent i can reach in a time slot. The joint observation of the agent in time period t is expressed as

[0084] Module M1.5: Policy π for each agent i i Assign agent i to a given observation at time slot t Take every action Probability The joint strategy of all agents is expressed as π = [π i ] i∈N ;

[0085] Module M1.6: At the end of each time slot t, each agent i receives an immediate reward is the amount of data perceived by agent i at time slot t; the immediate cost received is the charging cost of the agent in time slot t. The platform has a budget B for system charging cost for each task k. k ;

[0086] Module M1.7: Solve a general scheduling policy π that jointly makes movement, sensing, and charging decisions to maximize the sum of the expected sensing benefits of all tasks and satisfy the charging budget constraint of each task, that is,

[0087]

[0088]

[0089] in, represents the system's expected cumulative perceived benefit; represents the system's expected cumulative charging cost, and K represents the set of all tasks.

[0090] Preferably, the module M2 includes:

[0091] Module M2.1: In a multi-task multi-agent reinforcement learning framework, each agent i has a policy network π i , a rewarded reviewer network and a loss critic function The training process involves interacting with each task k times to collect experience, and then using this experience to update the parameters of the policy and critic networks;

[0092] Design a parallel sampling architecture that replicates the policy networks of all agents for each task k and performs the experience collection phase for all tasks simultaneously;

[0093] Module M2.2: At the beginning of each time slot t, the policy network of each agent i sends the global information g t and the local information vector As input;

[0094] Discretize the perception area into G grids, g t It consists of the number of POIs, UAVs, and charging stations in each grid in G, g t It is a tensor and input into the convolution layer, and then the spatial features are transformed into The output is:

[0095]

[0096] Where f is the ReLU activation function, W1 is the weight matrix of the MLP layer, and W c is the convolution kernel, * indicates the convolution operation;

[0097] The policy network of agent i converts the local information vector as input and feed it into a series of MLP layers to obtain a single feature

[0098]

[0099] Among them, W2 and W3 are the weight matrices of the MLP layer;

[0100] Secondly, and concatenate and further feed into the MLP layer to obtain the intermediate representation for:

[0101]

[0102] Among them, W4 is the weight matrix of the MLP layer, || represents the connection operation; A mask representing the action;

[0103] Finally, the mask vector is fed into the Softmax layer to generate i Generate action distribution on:

[0104]

[0105] Among them, W5 is the weight matrix of the MLP layer, It means element-wise addition;

[0106] Module M2.3: Use the global state s as the input of the review network of each agent i, and reward the last layer of the reviewer network to convert the intermediate representation The state-value function that maps to a scalar reward is:

[0107]

[0108] where W6 is the weight matrix of the MLP layer; similarly, the last layer of the loss critic network maps The state value function of the estimated loss is:

[0109]

[0110] Among them, W7 is the weight matrix of the MLP layer;

[0111] The module M3 includes:

[0112] Module M3.1: Use the Lagrange multiplier method to transform the original constrained optimization problem into the following max-min problem:

[0113]

[0114] Among them, θ is the parameter of the strategy π, which is the original variable; is a non-negative dual variable;

[0115] To solve the max-min problem, an iterative algorithm is designed that alternates between gradient ascent steps on θ and gradient descent steps on λ.

[0116] Module M3.2: Update dual variables;

[0117] In each cycle h, the dual variable is updated by projected gradient descent as:

[0118]

[0119] where η k is the step size, Γ[·] is the projection of each dual variable to [0, λ max ] projection operator within the range;

[0120] Module M3.3: Update strategy parameters;

[0121] In each cycle h, given the updated dual variable λ h , by stochastic gradient ascent h The parameters of θ h Update to θ h+1 :

[0122]

[0123] Among them, ρ is the parameter update step size, is the gradient of the objective function with respect to the policy:

[0124]

[0125] Among them, D k represents the set of samples, b k Indicates D k One of the samples in , and:

[0126]

[0127] in, and Respectively represent agents The advantage function related to the reward and cost is:

[0128]

[0129]

[0130] Module M3.4: Update commentator network parameters;

[0131] In each cycle h, the parameters of the agent’s reward critic network are adjusted from φ to h Update to φ h+1 , as shown below:

[0132]

[0133] in, is the reward state value function of agent i;

[0134] Similarly, the parameters of the critic network are calculated by minimizing the agent cost from ξ h Updated to h+1 :

[0135]

[0136] in, is the loss state value function of agent i;

[0137] The module M4 includes:

[0138] Module M4.1: Deploy the trained policy to the drone;

[0139] Module M4.2: At each decision time t, each UAV i receives the global information g from the platform t and your own local observations of the environment Input to strategy π i In the strategy, the next moment’s moving direction of the drone is output and distance And whether to collect sensory data and charge during movement;

[0140] Module M4.3: The results of the drone's execution strategy output, heading in the direction Moving distance Repeat this process until the entire decision-making process is completed.

[0141] In a third aspect, a computer-readable storage medium storing a computer program is provided, which, when executed by a processor, implements the steps of the drone swarm intelligence perception scheduling method.

[0142] In a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the drone swarm intelligence perception and scheduling method are implemented.

[0143] Compared with the prior art, the present invention has the following beneficial effects:

[0144] 1. Through multi-task multi-agent reinforcement learning, a universal scheduling strategy is designed to handle similar drone perception tasks, reducing the computational complexity and sample complexity of the training strategy.

[0145] 2. Using a primal-dual optimization algorithm, we alternate between improving overall perception and reducing constraint violations for each task, achieving the optimal strategy for constrained cooperative Markov games.

[0146] 3. The present invention has a reasonable structure, is easy to use, and can overcome the defects of the prior art.

[0147] Other beneficial effects of the present invention will be explained through the introduction of specific technical features and technical solutions in the specific implementation methods. Those skilled in the art should be able to understand the beneficial technical effects brought about by the introduction of these technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0148] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0149] Figure 1 is an example diagram of a policy network structure in an embodiment of the present invention;

[0150] Figure 2 This is an example diagram of a drone swarm perception scenario in an embodiment of the present invention. DETAILED DESCRIPTION

[0151] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0152] The embodiment of the present invention provides a method for swarm intelligence perception and scheduling of drones, referring to Figure 1 and Figure 2 As shown, the method specifically includes the following contents:

[0153] Step S1: Model the UAV scheduling problem as a constrained cooperative Markov game. Design the state, action, reward, and loss functions. The goal is to solve an optimal scheduling strategy to maximize the long-term perception benefits of the UAV under the constraint of the platform charging budget.

[0154] Specifically, step S1 includes:

[0155] Step S1.1: Consider each UAV i in the UAV perception system as an agent in a constrained cooperative Markov game, and each agent has its own strategy π i , and use N to represent the set of all intelligent agents.

[0156] Step S1.2: At the beginning of each time slot t, state s t The time slot index t, the platform remaining budget and the global information g t Composition, global information g t Contains the current location of POI, drone, and charging station.

[0157] Step S1.3: At the beginning of each time slot t, the agents that are not in the charging state can take actions. The set of these agents is denoted as N t , the action of each agent Including the direction and distance of movement, whether to perceive, whether to charge, the joint action of the agent in time period t is recorded as

[0158] Step S1.4: At the beginning of each time slot t, each agent i receives an observation Among them, g t is global information, is a local information vector containing the residual energy of agent i and the positions of nearby obstacles that agent i can reach in a time slot. The joint observation of the agent in time period t is expressed as

[0159] Step S1.5: Policy π for each agent i i Assign agent i to a given observation at time slot t Take every action Probability The joint strategy of all agents is expressed as π = [π i ] i∈N .

[0160] Step S1.6: At the end of each time slot t, each agent i receives an immediate reward is the amount of data perceived by agent i at time slot t; the immediate cost received is the charging cost of the agent in time slot t. The platform has a budget B for system charging cost for each task k. k .

[0161] Step S1.7: The goal is to solve a general scheduling policy π that jointly makes movement, sensing, and charging decisions, maximizes the sum of the expected sensing benefits of all tasks, and satisfies the charging budget constraint of each task, that is,

[0162]

[0163]

[0164] in, represents the system's expected cumulative perceived benefit; represents the system's expected cumulative charging cost, and K represents the set of all tasks.

[0165] Step S2: Design a multi-task multi-agent reinforcement learning framework to represent the neural network structure of the optimal strategy. This framework can use a general strategy to handle similar drone perception tasks, reducing the computational complexity and sample complexity of the training strategy.

[0166] The step S2 specifically includes:

[0167] Step S2.1: In the multi-task multi-agent reinforcement learning framework, each agent i has a policy network π i , a rewarded reviewer network and a loss critic function The training process involves interacting with each task k times to collect experience, which is then used to update the parameters of the policy and critic networks.

[0168] To improve the efficiency of interactions between agents and multiple tasks, we design a parallel sampling architecture that replicates the policy networks of all agents for each task k and performs the experience collection phase for all tasks simultaneously.

[0169] Step S2.2: The architecture of the policy network of each agent i is as shown in the attached Figure 1 At the beginning of each time slot t, each agent i’s participant network will send the global information g t and the local information vector as input.

[0170] In order to facilitate the feature extraction of the neural network, we discretize the perception area into G grids, g t It consists of the number of POIs, UAVs, and charging stations in each grid in G, g t It is a tensor and input into the convolution layer, and then the spatial features are transformed into The output is:

[0171]

[0172] Where f is the ReLU activation function, W1 is the weight matrix of the MLP layer, and W c is the convolution kernel, * indicates the convolution operation;

[0173] In addition, the participant network of agent i transforms the local information vector as input and feed it into a series of MLP layers to obtain a single feature

[0174]

[0175] Among them, W2 and W3 are the weight matrices of the MLP layer;

[0176] Secondly, and concatenate and further feed into the MLP layer to obtain the intermediate representation for:

[0177]

[0178] Among them, W4 is the weight matrix of the MLP layer, and || represents the connection operation. We use A mask representing the action.

[0179] Finally, the mask vector is fed into the Softmax layer to generate i Generate action distribution on:

[0180]

[0181] Among them, W5 is the weight matrix of the MLP layer, Represents element-wise addition.

[0182] Step S2.3: The critic network is only used to reduce variance during the training phase. The structures of the actor network and the critic network are identical except for the last layer and the input. Since we adopt the centralized training and decentralized execution (CTDE) paradigm, the critic network can access the global state during training and thus better estimate the state-value function. Therefore, we use the global state s as the input of the critic network for each agent i. The last layer of the reward critic network converts the intermediate representation The state-value function that maps to a scalar reward is:

[0183]

[0184] where W6 is the weight matrix of the MLP layer; similarly, the last layer of the loss critic network maps The state value function of the estimated loss is:

[0185]

[0186] Among them, W7 is the weight matrix of the MLP layer.

[0187] Step S3: Propose a primal-dual multi-agent reinforcement learning training algorithm that alternates between improving the overall perception gain and reducing the constraint violation of each task.

[0188] Specifically, step S3 includes:

[0189] Step S3.1: Use the Lagrange multiplier method to transform the original constrained optimization problem into the following max-min problem:

[0190]

[0191] Among them, θ is the parameter of the strategy π, which is the original variable; is a non-negative dual variable;

[0192] To solve the max-min problem, we design an iterative algorithm that alternates between gradient ascent steps on θ and gradient descent steps on λ.

[0193] Step S3.2: Update the dual variables.

[0194] In each cycle h, the dual variable is updated by projected gradient descent as:

[0195]

[0196] where η k is the step size, Γ[·] is the projection of each dual variable to [0, λ max ] range of projection operators.

[0197] Step S3.3: Update the policy parameters.

[0198] In each cycle h, given the updated dual variable λ h , by stochastic gradient ascent h The parameters of θ h Update to θ h+1 :

[0199]

[0200] Among them, ρ is the parameter update step size, is the gradient of the objective function with respect to the policy:

[0201]

[0202] Among them, D k represents the set of samples, b k Indicates D k One of the samples in , and:

[0203]

[0204] in, and Respectively represent agents The advantage function related to the reward and cost is:

[0205]

[0206]

[0207] Step S3.4: Update the commentator network parameters.

[0208] In each cycle h, the parameters of the agent’s reward critic network are adjusted from φ to h Update to φ h+1 , as shown below:

[0209]

[0210] in, is the reward state value function of agent i.

[0211] Similarly, the parameters of the critic network are calculated by minimizing the agent cost from ξ h Updated to h+1 :

[0212]

[0213] in, is the loss state value function of agent i.

[0214] Step S4: Deploy the trained policy to the drone. The drone inputs global information and local observations into the policy, which then outputs the drone's next movement direction and distance, as well as whether to collect sensory data and recharge during movement. The drone executes the policy's output, repeating this process until the entire decision-making process is complete.

[0215] Specifically, step S4 includes:

[0216] Step S4.1: Deploy the trained policy to the drone;

[0217] Step S4.2: At each decision time t, each UAV i receives the global information g from the platform t and your own local observations of the environment Input to strategy π i In the strategy, the next moment’s moving direction of the drone is output and distance And whether to collect sensory data and charge during movement;

[0218] Step S4.3: The drone executes the output of the strategy and moves in the direction Moving distance Repeat this process until the entire decision-making process is completed.

[0219] This example focuses on the practical problem of drone swarm perception and proposes a multi-task, constrained multi-agent reinforcement learning approach to make joint drone movement, perception, and charging decisions. First, the drone scheduling problem is modeled as a constrained cooperative Markov game. State, action, reward, and loss functions are designed. The goal is for drones to collaboratively optimize long-term perception benefits while satisfying the platform's charging budget. Second, a multi-task, multi-agent reinforcement learning framework is designed to handle a large number of similar drone perception tasks with a universal strategy, reducing the computational and sample complexity of the training strategy. Finally, a primal-dual multi-agent reinforcement learning training algorithm is proposed, which alternates between improving overall perception benefits and reducing constraint violations for each task. Experiments on a New York City emergency data set fully demonstrate the effectiveness of this example. In terms of system perception benefits and system charging costs, this example significantly outperforms the best existing drone scheduling methods. Although this example is designed for drone swarm perception scenarios, the proposed multi-task, constrained multi-agent reinforcement learning approach can also be applied to a variety of other scheduling tasks.

[0220] The present invention also provides a drone swarm intelligence perception scheduling system, which can be implemented by executing the process steps of the drone swarm intelligence perception scheduling method. That is, those skilled in the art can understand the drone swarm intelligence perception scheduling method as a preferred implementation of the drone swarm intelligence perception scheduling system.

[0221] Module M1: Model the drone scheduling problem as a constrained cooperative Markov game, and design the state, action, reward, and loss functions.

[0222] Module M1 includes:

[0223] Module M1.1: Consider each drone i in the drone perception system as an agent in a constrained cooperative Markov game, each with its own strategy π i , use N to represent the set of all intelligent agents;

[0224] Module M1.2: At the beginning of each time slot t, state s t The time slot index t, the platform remaining budget and the global information g t Composition, global information g t Contains the current location of POI, drone and charging station;

[0225] Module M1.3: At the beginning of each time slot t, agents that are not in the charging state can take actions. The set of these agents is denoted as N t , the action of each agent Including the direction and distance of movement, whether to perceive, whether to charge, the joint action of the agent in time period t is recorded as

[0226] Module M1.4: At the beginning of each time slot t, each agent i receives an observation Among them, g t is global information, is a local information vector containing the residual energy of agent i and the positions of nearby obstacles that agent i can reach in a time slot. The joint observation of the agent in time period t is expressed as

[0227] Module M1.5: Policy π for each agent i i Assign agent i to a given observation at time slot t Take every action Probability The joint strategy of all agents is expressed as π = [π i ] i∈N ;

[0228] Module M1.6: At the end of each time slot t, each agent i receives an immediate reward is the amount of data perceived by agent i at time slot t; the immediate cost received is the charging cost of the agent in time slot t. The platform has a budget B for system charging cost for each task k. k ;

[0229] Module M1.7: The goal is to solve a general scheduling policy π that jointly makes mobility, sensing, and charging decisions, maximizes the sum of the expected sensing benefits of all tasks, and satisfies the charging budget constraint of each task, that is,

[0230]

[0231]

[0232] in, represents the system's expected cumulative perceived benefit; represents the system's expected cumulative charging cost, and K represents the set of all tasks.

[0233] Module M2: Design a multi-task multi-agent reinforcement learning framework to use a common strategy to handle similar drone perception tasks and reduce the computational complexity and sample complexity of the training strategy.

[0234] Module M2 includes:

[0235] Module M2.1: In a multi-task multi-agent reinforcement learning framework, each agent i has a policy network π i , a rewarded reviewer network and a loss critic function The training process involves interacting with each task k times to collect experience, and then using this experience to update the parameters of the policy and critic networks;

[0236] Design a parallel sampling architecture that replicates the policy networks of all agents for each task k and performs the experience collection phase for all tasks simultaneously;

[0237] Module M2.2: At the beginning of each time slot t, the policy network of each agent i sends the global information g t and the local information vector As input;

[0238] Discretize the perception area into G grids, g t It consists of the number of POIs, UAVs, and charging stations in each grid in G, g t It is a tensor and input into the convolution layer, and then the spatial features are transformed into The output is:

[0239]

[0240] Where f is the ReLU activation function, W1 is the weight matrix of the MLP layer, and W c is the convolution kernel, * indicates the convolution operation;

[0241] The policy network of agent i converts the local information vector as input and feed it into a series of MLP layers to obtain a single feature

[0242]

[0243] Among them, W2 and W3 are the weight matrices of the MLP layer;

[0244] Secondly, and concatenate and further feed into the MLP layer to obtain the intermediate representation for:

[0245]

[0246] Among them, W4 is the weight matrix of the MLP layer, || represents the connection operation; A mask representing the action;

[0247] Finally, the mask vector is fed into the Softmax layer to generate i Generate action distribution on:

[0248]

[0249] Among them, W5 is the weight matrix of the MLP layer, It means element-wise addition;

[0250] Module M2.3: Use the global state s as the input of the review network of each agent i, and reward the last layer of the reviewer network to convert the intermediate representation The state-value function that maps to a scalar reward is:

[0251]

[0252] where W6 is the weight matrix of the MLP layer; similarly, the last layer of the loss critic network maps The state value function of the estimated loss is:

[0253]

[0254] Among them, W7 is the weight matrix of the MLP layer.

[0255] Module M3: Propose a primal-dual multi-agent reinforcement learning training algorithm that alternates between improving the overall perception gain and reducing the constraint violation of each task.

[0256] Module M3 includes:

[0257] Module M3.1: Use the Lagrange multiplier method to transform the original constrained optimization problem into the following max-min problem:

[0258]

[0259] Among them, θ is the parameter of the strategy π, which is the original variable; is a non-negative dual variable;

[0260] To solve the max-min problem, an iterative algorithm is designed that alternates between gradient ascent steps on θ and gradient descent steps on λ.

[0261] Module M3.2: Update dual variables;

[0262] In each cycle h, the dual variable is updated by projected gradient descent as:

[0263]

[0264] where η k is the step size, Γ[·] is the projection of each dual variable to [0, λ max ] projection operator within the range;

[0265] Module M3.3: Update strategy parameters;

[0266] In each cycle h, given the updated dual variable λ h , by stochastic gradient ascent h The parameters of θ h Update to θ h+1 :

[0267]

[0268] Among them, ρ is the parameter update step size, is the gradient of the objective function with respect to the policy:

[0269]

[0270] Among them, D k represents the set of samples, b k Indicates D k One of the samples in , and:

[0271]

[0272] in, and Respectively represent agents The advantage function related to the reward and cost is:

[0273]

[0274]

[0275] Module M3.4: Update commentator network parameters;

[0276] In each cycle h, the parameters of the agent’s reward critic network are adjusted from φ to h Update to φ h+1 , as shown below:

[0277]

[0278] in, is the reward state value function of agent i;

[0279] Similarly, the parameters of the critic network are calculated by minimizing the agent cost from ξ h Updated to h+1 :

[0280]

[0281] in, is the loss state value function of agent i.

[0282] Embodiments of the present invention provide a method, system, device, and medium for swarm intelligence perception scheduling of drones, which adopt a multi-task parallel training framework. It can effectively jointly formulate movement, perception, and charging decisions, and maximize the long-term perception benefits of drones under the constraints of the charging budget.

[0283] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0284] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for swarm intelligence perception and scheduling of drones, characterized in that: include: Step S1: Model the UAV scheduling problem as a constrained cooperative Markov game. Design the state, action, reward, and loss functions to solve an optimal scheduling strategy that maximizes the long-term perception benefits of the UAVs under the constraints of the platform charging budget. Step S2: Design a multi-task multi-agent reinforcement learning framework to represent the neural network structure of the optimal strategy. The framework can handle a large number of similar drone perception tasks and reduce the computational complexity and sample complexity of the training strategy. Step S3: Propose a primal-dual multi-agent reinforcement learning training algorithm that alternates between improving the overall perception gain and reducing the constraint violation of each task; Step S4: Deploy the trained policy to the drone. The drone inputs global information and local observations into the policy, which then outputs the drone's next movement direction and distance, as well as whether to collect sensory data and charge during movement. The drone executes the policy output, and this process repeats until the entire decision-making process is complete. The step S3 comprises: Step S3.1: Use the Lagrange multiplier method to transform the original constrained optimization problem into the following max-min problem: Among them, θ is the parameter of strategy Π, which is the original variable; is a non-negative dual variable; To solve the max-min problem, an iterative algorithm is designed that alternates between gradient ascent steps on θ and gradient descent steps on λ. Step S3.2: Update the dual variables; In each cycle h, the dual variable is updated by projected gradient descent as: where η k is the step size, Γ[·] is the projection of each dual variable to [0,λ max ] projection operator within the range; Step S3.3: Update strategy parameters; In each cycle h, given the updated dual variable λ h , by stochastic gradient ascent h The parameters of θ h Update to θ h+1 : Among them, ρ is the parameter update step size, is the gradient of the objective function with respect to the policy: Among them, D k represents the set of samples, b k Indicates D k One of the samples in , and: in, and denote the reward- and cost-related advantage functions of agent j∈N respectively: Step S3.4: Update the commentator network parameters; In each cycle h, the parameters of the agent’s reward critic network are adjusted from Update to As shown in the following formula: in, is the reward state value function of agent i; Similarly, the parameters of the critic network are calculated by minimizing the agent cost from ξ h Updated to h+1 : in, is the loss state value function of agent i.

2. The drone swarm intelligence perception scheduling method according to claim 1 is characterized in that: The step S1 comprises: Step S1.1: Consider each drone i in the drone perception system as an agent in a constrained cooperative Markov game, and each agent has its own strategy Π i , use N to represent the set of all intelligent agents; Step S1.2: At the beginning of each time slot t, state s t The time slot index t, the platform remaining budget and the global information g t Composition, global information g t Contains the current location of POI, drone and charging station; Step S1.3: At the beginning of each time slot t, the agents that are not in the charging state can take actions. The set of these agents is denoted as N t , the action of each agent Including the direction and distance of movement, whether to perceive, whether to charge, the joint action of the agent in time period t is recorded as Step S1.4: At the beginning of each time slot t, each agent i receives an observation Among them, g t is global information, is a local information vector containing the residual energy of agent i and the positions of nearby obstacles that agent i can reach in a time slot. The joint observation of the agent in time period t is expressed as Step S1.5: Strategy Π for each agent i i Assign agent i to a given observation at time slot t Take every action Probability The joint strategy of all agents is expressed as Π = [Π i ] i∈N ; Step S1.6: At the end of each time slot t, each agent i receives an immediate reward is the amount of data perceived by agent i at time slot t; the immediate cost received is the charging cost of the agent in time slot t. The platform has a budget B for system charging cost for each task k. k ; Step S1.7: Solve a general scheduling policy Π that jointly makes movement, sensing, and charging decisions to maximize the sum of the expected sensing benefits of all tasks and satisfy the charging budget constraint of each task, that is, in, represents the system's expected cumulative perceived benefit; represents the system's expected cumulative charging cost, and K represents the set of all tasks.

3. The UAV swarm intelligence perception scheduling method according to claim 1 is characterized in that: The step S2 comprises: Step S2.1: In the multi-task multi-agent reinforcement learning framework, each agent i has a policy network Π i , a rewarded reviewer network and a loss critic function The training process involves interacting with each task k times to collect experience, and then using this experience to update the parameters of the policy and critic networks; Design a parallel sampling architecture that replicates the policy networks of all agents for each task k and performs the experience collection phase for all tasks in parallel; Step S2.2: At the beginning of each time slot t, the policy network of each agent i sends the global information g t and the local information vector As input; Discretize the perception area into G grids, g t It consists of the number of POIs, UAVs, and charging stations in each grid in G, g t It is a tensor and input into the convolution layer, and then the spatial features are transformed into The output is: Where f is the ReLU activation function, W1 is the weight matrix of the MLP layer, and W c is the convolution kernel, * indicates the convolution operation; The policy network of agent i converts the local information vector as input and feed it into a series of MLP layers to obtain a single feature Among them, W2 and W3 are the weight matrices of the MLP layer; Secondly, and concatenate and further feed into the MLP layer to obtain the intermediate representation for: Among them, W4 is the weight matrix of the MLP layer, / / represents the connection operation; A mask representing the action; Finally, the mask vector is fed into the Softmax layer to generate i Generate action distribution on: Among them, W5 is the weight matrix of the MLP layer, It means element-wise addition; Step S2.3: Using the global state s as the input to the critic network for each agent i, the last layer of the reward critic network converts the intermediate representation The state-value function that maps to a scalar reward is: where W6 is the weight matrix of the MLP layer; similarly, the last layer of the loss critic network maps The state value function of the estimated loss is: Among them, W7 is the weight matrix of the MLP layer.

4. The UAV swarm intelligence perception scheduling method according to claim 1 is characterized in that: The step S4 comprises: Step S4.1: Deploy the trained policy to the drone; Step S4.2: At each decision time t, each UAV i receives the global information g from the platform t and your own local observations of the environment Input to strategy Π i In the strategy, the next moment’s moving direction of the drone is output and distance And whether to collect sensory data and charge during movement; Step S4.3: The drone executes the output of the strategy and moves in the direction Moving distance Repeat this process until the entire decision-making process is completed.

Images