UAV control method, system, electronic device and storage medium

Abstract

This application discloses a drone control method, system, electronic device, and storage medium, all related to the technical field of drone communication technology. The drone control method includes: establishing a problem model and constructing a constrained Markov decision problem based on the problem model; modeling the constrained Markov decision problem as a target model; obtaining a target algorithm by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; using the target algorithm to train multiple drone agents, and using the trained agents to perform trajectory planning and / or communication scheduling for the drone. This application enables drone control that adapts to dynamic characteristics and distributed control requirements.

Description

Technical Field

[0001] The present application relates to the field of drone communication technology, and in particular to a drone control method, system, electronic device, and storage medium. Background Art

[0002] To fully leverage the advantages of drone-assisted communication technology, research on the joint optimization of drone trajectories and communications has garnered widespread attention in academia. In drone-to-ground communication service scenarios, the quality of communication service for ground nodes is influenced by both the drone's trajectory and the communication strategy. The drone's trajectory determines the relative position and distance between the drone and ground nodes, which in turn affects the quality of the air-ground channel. The communication strategy further influences the transmission quality between the drone and ground nodes by rationally adjusting factors such as communication resource allocation, the relationship between ground nodes and drones, and the order in which drones dispatch communications to ground nodes. These factors are interdependent and tightly coupled, collectively determining the overall communication performance of the system. Therefore, research on the joint optimization of drone trajectories and communications is crucial for improving the performance of drone-to-ground communication systems.

[0003] The specific manifestations of the joint optimization problem of UAV trajectories and communications vary across different scenarios, but these problems are essentially multivariable, non-convex optimization problems with complex optimization objectives and constraints, making them difficult to solve using conventional convex optimization tools. To address this, numerous studies have proposed optimization methods based on traditional mathematical tools. These methods solve the optimization problem offline based on static optimization models and prior environmental information, providing theoretically optimal or near-optimal solutions for the joint optimization of UAV trajectories and communications. However, these methods face a number of limitations in practical applications. First, in UAV-to-ground communication systems, environmental factors such as the deployment location and number of communication nodes are time-varying, making it difficult for offline-designed UAV trajectory and communication strategies to meet performance requirements in dynamic environments. Second, traditional optimization methods typically have high computational complexity, making real-time updating of optimization variables in dynamic environments extremely difficult. Most of these methods rely on centralized solutions based on global environmental information, ignoring the distributed control requirements of multi-UAV scenarios.

[0004] Therefore, how to achieve UAV control that adapts to dynamic characteristics and distributed control requirements is a technical problem that technicians in this field currently need to solve. Summary of the Invention

[0005] The purpose of this application is to provide a drone control method, system, electronic device and storage medium that can realize drone control that adapts to dynamic characteristics and distributed control requirements.

[0006] To solve the above technical problems, the present application provides a drone control method, comprising:

[0007] Establishing a problem model and constructing a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent multiple drones collecting data from multiple ground nodes and offloading the collected data to a ground data center;

[0008] Modeling the constrained Markov decision problem as a target model; wherein the target model is a distributed locally observable Markov decision process model;

[0009] A target algorithm is obtained by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model and is used to solve an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and has permutation invariance, and the second network layer is a network layer based on a fully connected layer and has permutation equivariance;

[0010] The target algorithm is used to train the intelligent agents of the plurality of drones, and the trained intelligent agents are used to perform trajectory planning and / or communication scheduling on the drones.

[0011] Optionally, constructing a constrained Markov decision problem based on the problem model includes:

[0012] Determining environmental state characteristics according to the problem model; wherein the environmental state characteristics include the current location information of the UAV, the location information of the current ground node, the location information of the current ground data center, the amount of data to be uploaded by the current ground node, and the total amount of stored data of the UAV;

[0013] Determine a probabilistic line-of-sight model based on the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path;

[0014] Determining the trajectory variables and binary scheduling variables of the UAV according to the problem model; wherein the binary scheduling variables are used to describe the communication status between the ground node and the ground data center;

[0015] Determining an average information age function of the ground nodes according to the problem model;

[0016] The constrained Markov decision problem is constructed based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

[0017] Optionally, modeling the constrained Markov decision problem as a target model includes:

[0018] Performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity; wherein the feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node;

[0019] Key elements are defined using the feature vector of the drone entity and the feature vector of the ground node entity, and the target model is built based on the key elements; wherein the key elements include state, observation, action and reward.

[0020] Optionally, before obtaining the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, the method further includes:

[0021] Introducing a first super-network shared by all input elements into a single fully connected layer to form the first network layer; wherein the first super-network takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features of the single input element and sum them to obtain a network output;

[0022] A second super-network shared by all output elements and a first network layer for processing the overall input of the output elements are introduced into a single fully connected layer to form a second network layer; wherein the second super-network takes the input element corresponding to the single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply with the output of the first network layer to obtain the network output.

[0023] Optionally, obtaining the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm includes:

[0024] An input layer is constructed based on a fully connected layer and two first network layers; wherein the input of the input layer is all features in the local observation of the drone, and the output of the input layer is a single embedded feature merged by a summation operation;

[0025] An intermediate layer is constructed based on the gated recurrent unit in the multi-agent reinforcement learning algorithm; wherein the input of the intermediate layer is the single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot;

[0026] An output layer is constructed based on a fully connected layer and a second network layer; wherein the input of the output layer is the output of the intermediate layer, and the output of the output layer is the reward values ​​of all actions in the action space and the optimal action corresponding to the maximum reward value;

[0027] Building a hyperlocal value network comprising the input layer, the intermediate layer, and the output layer; wherein the hyperlocal value network is used to fit the local value function of the intelligent agent of the drone;

[0028] Building a hyperhybrid network for combining the local rewards of all the agents into a global reward value;

[0029] The target algorithm is obtained by combining the hyperlocal value network of each agent and the hypermixed network.

[0030] Optionally, the constructing of a super-hybrid network for combining the local reward values ​​of all the agents into a global reward value includes:

[0031] The hyperhybrid network is constructed using a fully connected input layer and a fully connected output layer; wherein the hyperhybrid network is used to combine the local reward values ​​of all the agents into a global reward value;

[0032] Among them, the input of the fully connected input layer is the local reward value of all intelligent agents, and the output of the fully connected input layer is a single embedded feature; the input of the fully connected output layer is the single embedded feature output by the input layer, and the output of the fully connected output layer is the global reward value.

[0033] Optionally, also include:

[0034] Determining a weight matrix and a bias vector of the fully connected input layer;

[0035] The weight matrix and bias vector of the fully connected output layer are determined.

[0036] Optionally, using the target algorithm to train multiple intelligent agents of the drones includes:

[0037] The target algorithm is used to perform centralized offline training on the intelligent agents of the plurality of drones, so that the intelligent agents of the drones update the parameters of the hyperlocal value network using experience samples; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

[0038] Optionally, after performing centralized offline training on the intelligent agents of the plurality of drones using the target algorithm, the method further includes:

[0039] Deploying the trained intelligent agent to the corresponding UAV; wherein the trained intelligent agent includes a hyperlocal value network with updated parameters;

[0040] The strategy migration operation is performed on the intelligent agent in the UAV by means of model reloading.

[0041] The present application also provides a drone control system, the system comprising:

[0042] A problem construction module is used to establish a problem model and construct a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that multiple drones collect data from multiple ground nodes and offload the collected data to a ground data center;

[0043] A model building module, configured to model the constrained Markov decision problem as a target model; wherein the target model is a distributed locally observable Markov decision process model;

[0044] An algorithm building module, configured to obtain a target algorithm by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model and is configured to solve an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance;

[0045] A control module is used to train the intelligent agents of the plurality of drones using the target algorithm, and to perform trajectory planning and / or communication scheduling for the drones using the trained intelligent agents.

[0046] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned drone control method when calling the computer program in the memory.

[0047] The present application also provides a storage medium, which stores computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, the steps of the above-mentioned drone control method are implemented.

[0048] This application provides a method for controlling unmanned aerial vehicles (UAVs). This method first establishes a problem model for representing multiple UAVs collecting data from multiple ground nodes and offloading the collected data to a ground data center. Based on the problem model, a constrained Markov decision problem is constructed, modeled as a distributed locally observable Markov decision process (Dec-POMDP). The Dec-POMDP model allows each UAV agent to make decisions based on local information while taking into account the presence and activities of other agents, helping to address uncertainty in multi-agent systems and achieve distributed control. This application introduces the first network layer (HPI) and the second network layer (HPE) into the multi-agent reinforcement learning algorithm QMIX to obtain a target algorithm. Because the first network layer (HPI) is based on a fully connected layer and is permutation invariant, and the second network layer (HPE) is based on a fully connected layer and is permutation equivariant, the target algorithm obtained in this application can solve agent policies that adapt to changes in the number of entities. This application uses the target algorithm to train multiple UAV agents, so that the trained agents can make decisions based on local information to adapt to dynamic characteristics and distributed control requirements. This application uses the Dec-POMDP model and target algorithm to enable trained agents to adapt to various dynamic environmental factors and distributed control requirements during drone control. This application also provides a drone control system, electronic device, and storage medium that exhibit the aforementioned beneficial effects, which will not be further elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 A flowchart of a drone control method provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of a scalable multi-agent reinforcement learning method provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of another scalable multi-agent reinforcement learning method provided in an embodiment of the present application;

[0053] Figure 4 Schematic diagram of convergence curves of average information age during training for a HyperQMIX algorithm and a first set of comparison algorithms provided in an embodiment of the present application;

[0054] Figure 5Schematic diagram of convergence curves of average information age during training for a HyperQMIX algorithm and a second set of comparison algorithms provided in an embodiment of the present application;

[0055] Figure 6 Schematic diagram of convergence curves of average information age during training for a HyperQMIX algorithm and a third set of comparative algorithms provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a drone trajectory generated by a strategy learned by a HyperQMIX algorithm provided in an embodiment of the present application during the first period of a round;

[0057] Figure 8 A schematic diagram of a drone trajectory generated by a strategy learned by a HyperQMIX algorithm provided in an embodiment of the present application during the second period of a round;

[0058] Figure 9 A schematic diagram of a drone trajectory generated by a strategy learned by a HyperQMIX algorithm provided in an embodiment of the present application during the third period of a round;

[0059] Figure 10 A schematic diagram of convergence curves of multiple algorithms in a simulation experiment provided in an embodiment of the present application;

[0060] Figure 11 A schematic diagram of convergence curves of multiple algorithms in another simulation experiment provided in an embodiment of the present application;

[0061] Figure 12 A schematic diagram of a convergence curve of an ablation experiment in a simulation experiment provided in an embodiment of the present application;

[0062] Figure 13 A schematic diagram comparing the average information age of different algorithms under different numbers of ground nodes provided in an embodiment of the present application;

[0063] Figure 14 A schematic diagram showing a comparison of the average information age of different algorithms under different numbers of drones provided in an embodiment of the present application;

[0064] Figure 15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0065] Figure 16 A schematic diagram of the structure of a drone control system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] See below Figure 1 , Figure 1 This is a flowchart of a drone control method provided in an embodiment of the present application.

[0068] Specific steps may include:

[0069] S101: Establish a problem model, and construct a constrained Markov decision problem based on the problem model;

[0070] Among them, this embodiment can be applied to electronic devices with intelligent body training functions. This embodiment can establish a problem model based on relevant information of drones, ground nodes and ground data centers. The problem model is used to represent that multiple drones collect data from multiple ground nodes and unload the collected data to a ground data center.

[0071] Ground nodes are devices or sensor nodes located on the ground that collect various data and communicate with drones. Ground data centers are facilities located on the ground that process, store, and manage data collected by drones or other sensors.

[0072] Based on the problem model described above, a constrained Markov decision problem can be constructed based on the problem model. This process abstracts the actual problem into a mathematical model to address how an intelligent agent can make the optimal decision under specific constraints. The constrained Markov decision problem extends the traditional Markov decision process by incorporating constraints such as energy consumption and time limits. By constructing a constrained Markov decision problem, strategies can be designed for intelligent agents, such as drones, to maximize cumulative rewards while complying with constraints, enabling efficient data collection and offloading.

[0073] S102: Modeling the constrained Markov decision problem as a target model;

[0074] The target model is a distributed locally observable Markov decision process model, also known as the Dec-POMDP (Decentralized Partially Observable Markov Decision Process) model. Modeling constrained Markov decision problems as a Dec-POMDP model is a method for taking local information and uncertainty into account in multi-agent systems. The Dec-POMDP model allows each agent to make decisions based on local observations while taking into account the presence and activities of other agents to achieve distributed control and collaborative work. The Dec-POMDP model helps to handle uncertainty in multi-agent systems and achieve efficient distributed decision-making.

[0075] S103: Obtaining a target algorithm by introducing the first network layer and the second network layer into a multi-agent reinforcement learning algorithm;

[0076] Among them, the target algorithm is an algorithm designed based on the target model, which is used to solve the intelligent agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on the fully connected (FC) layer and has permutation invariance, and the second network layer is a network layer based on the fully connected layer and has permutation equivariance.

[0077] The first network layer is also called the HPI layer (Hypernetwork-based Permutation Invariant, adapted to variable-length inputs), and the second network layer is also called the HPE layer (Hypernetwork-based Permutation Equivariant, adapted to variable-length outputs). This step introduces the HPI and HPE layers into the QMIX algorithm (a hybrid-based multi-agent reinforcement learning algorithm) to obtain the target algorithm (i.e., the HyperQMIX algorithm, an enhanced multi-agent reinforcement learning algorithm). This step can be preceded by pre-building the HPI and HPE layers.

[0078] Specifically, this embodiment introduces an automatic weight generation mechanism based on the FC layer to obtain the HPI layer. The HPI layer dynamically adjusts the weight matrix to adapt to changes in the number of input elements, satisfying permutation invariance. That is, if the order of the input elements is changed, the function output does not change. This application names the neural network module that undergoes this improvement to the FC layer network structure as the HPI layer.

[0079] This embodiment also introduces an automatic weight generation mechanism based on the FC layer, establishing a one-to-one correspondence between input and output elements, resulting in the HPE layer. This mechanism ensures that the weight matrix can be adjusted accordingly with changes in the number of input and output elements, satisfying the permutation equivariance property, that is, changing the order of the input elements will result in the same change in the order of the output elements. This application names the neural network module that undergoes this improvement to the FC layer network structure the HPE layer.

[0080] This embodiment introduces the HPI and HPE layers into the multi-agent reinforcement learning algorithm QMIX to obtain the target algorithm HyperQMIX. The HyperQMIX algorithm is designed based on the Dec-POMDP model and is used to solve agent policies that adapt to changes in the number of entities. Specifically, this embodiment can introduce the HPI and HPE layers into the QMIX algorithm based on the Dec-POMDP model to obtain the HyperOMIX algorithm. The HyperQMIX algorithm is an improvement on the QMIX algorithm, which can solve agent policies that adapt to changes in the number of entities. The improvement lies in the introduction of the HPI and HPE layers into the QMIX algorithm.

[0081] The HyperQMIX algorithm combines the characteristics of the HPI layer, HPE layer and QMIX algorithm, and can solve the agent strategy that adapts to the changes in the number of entities, thereby achieving efficient data collection and offloading in a multi-agent environment.

[0082] S104: Using the target algorithm to train the intelligent agents of the plurality of drones, and using the trained intelligent agents to perform trajectory planning and / or communication scheduling for the drones.

[0083] This embodiment uses the HyperQMIX target algorithm to train multiple drone agents, enabling them to make decisions tailored to their dynamic characteristics and distributed control requirements. Through training, the agents learn effective trajectory planning and communication scheduling strategies, enabling efficient data collection and offloading in a multi-agent environment.

[0084] This embodiment first establishes a problem model for multiple drones collecting data from multiple ground nodes and offloading the collected data to a ground data center. Based on this problem model, a constrained Markov decision problem is constructed, modeled as a Dec-POMDP model. The Dec-POMDP model allows each drone agent to make decisions based on local information while taking into account the presence and activities of other agents, helping to address uncertainty in multi-agent systems and achieve distributed control. This embodiment introduces the HPI layer and HPE layer into the QMIX algorithm to obtain the HyperQMIX algorithm. Because the HPI layer is a permutation-invariant network layer based on the FC layer, and the HPE layer is a permutation-equivariant network layer based on the FC layer, the resulting HyperQMIX algorithm can solve agent policies that adapt to changes in the number of entities. This embodiment uses the HyperQMIX algorithm to train multiple drone agents, allowing the trained agents to make decisions based on local information to adapt to dynamic characteristics and distributed control requirements. This embodiment uses the Dec-POMDP model and the HyperQMIX algorithm to enable the trained intelligent agent to adapt to various dynamic environmental factors and distributed control requirements during the process of controlling the drone.

[0085] As for Figure 1 For further description of the corresponding embodiment, the constrained Markov decision problem can be constructed in the following way:

[0086] Step A1: Determine environmental state characteristics based on the problem model; wherein the environmental state characteristics include the current location information of the UAV, the location information of the current ground node, the location information of the current ground data center, the amount of data to be uploaded by the current ground node, and the total amount of stored data of the UAV;

[0087] Step A2: determining a probabilistic line-of-sight model based on the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path;

[0088] Step A3: Determine the trajectory variable and binary scheduling variable of the UAV according to the problem model; wherein the binary scheduling variable is used to describe the communication status between the ground node and the ground data center;

[0089] Step A4: determining the average information age function of the ground nodes according to the problem model;

[0090] The average information age function is a function that calculates the average information age of ground nodes. Information age is an important indicator of information freshness, and maintaining a low information age is crucial for the timeliness of real-time update systems. The information age of each data packet refers to the length of time that has elapsed from its generation to the current moment. Each data packet can only provide information for subsequent decision-making after being successfully received by the data center. Therefore, the information age of each ground node is defined as the age of the most recent data packet received from that ground node by the data center.

[0091] Step A5: Constructing the constrained Markov decision problem based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

[0092] Based on the environmental state characteristics, probabilistic line-of-sight model, UAV trajectory variables, binary scheduling variables, and average information age function, this embodiment can comprehensively consider the UAV's mobility characteristics, communication constraints, and information generation time requirements, thereby designing a constrained Markov decision problem for the UAV while satisfying the constraints to achieve efficient data collection and offloading.

[0093] As for Figure 1 For further introduction to the corresponding embodiment, this embodiment can construct the constrained Markov decision problem into a target model Dec-POMDP in the form of entity decomposition. The specific process is as follows:

[0094] Step B1: performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity;

[0095] The feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node;

[0096] Step B2: defining key elements using the feature vector of the drone entity and the feature vector of the ground node entity;

[0097] The key elements include state, observation, action and reward;

[0098] Step B3: Build the target model Dec-POMDP based on the key elements.

[0099] The constrained Markov decision problem is decomposed into entities, resulting in drone entities as intelligent entities and ground node entities as non-intelligent entities. Key elements such as state, observation, action, and reward are defined using the feature vectors of the drone and ground node entities. Based on these key elements, this embodiment constructs a Dec-POMDP model. This model allows each drone agent to make decisions based on local information while taking into account the presence and activities of other agents, thereby achieving distributed control and collaborative work.

[0100] As for Figure 1 For further description of the corresponding embodiment, before obtaining the HyperQMIX algorithm by introducing the HPI layer and the HPE layer into the QMIX algorithm, the HPI layer and the HPE layer may be formed in the following manner:

[0101] This embodiment can introduce a first hypernetwork HN shared by all input elements in a single FC layer. in (⋅) forms the HPI layer; wherein, the first hypernetwork takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features of the single input element and sum them to obtain the network output.

[0102] This embodiment can introduce a second hypernetwork HN shared by all output elements in a single FC layer. out (⋅) and an HPI layer for processing the overall input of the output elements, together forming an HPE layer; wherein the second super network takes the input element corresponding to the single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply the output of the HPI layer to obtain the network output.

[0103] As for Figure 1 For further description of the corresponding embodiment, the HPI layer and the HPE layer can be introduced into the QMIX algorithm to obtain the HyperQMIX algorithm in the following manner:

[0104] Step C1: Build the input layer based on one FC layer and two HPI layers;

[0105] The input layer takes in all features from the drone's local observations, and its output is a single embedded feature, combined via a summation operation. This step builds the input layer using a FC layer and two permutation-invariant HPI layers. The input layer receives all features from the drone's local observations, including but not limited to the current location, the amount of data to be unloaded from the buffer, and the scheduling relationship with the data center during the previous time slot. Among these features, drone n itself is processed via a FC layer, while the feature sets of neighboring drones and ground nodes within the perception range are processed by two HPI layers, respectively. The HPI layers can process inputs of varying permutations and produce the same output, enabling the input layer to efficiently process entity sets. The outputs of the FC layer and the two HPI layers are combined via a summation operation to form a single embedded feature. This feature represents key information from the drone's local observations and provides input for subsequent decision-making.

[0106] Step C2: Build the middle layer based on the GRU structure in the QMIX algorithm;

[0107] The input of the intermediate layer is the single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot.

[0108] Specifically, this step builds an intermediate layer based on the GRU (Gated Recurrent Unit) structure from the QMIX algorithm. This intermediate layer receives a single embedded feature from the input layer and the hidden state from the previous time slot as input. The GRU structure is capable of processing sequential data and preserving long-term dependencies. During this process, the single embedded feature is input into the GRU structure along with the hidden state from the previous time slot. The GRU structure updates the hidden state through a gating mechanism to reflect the state changes in the current time slot. After processing, the intermediate layer outputs the hidden state for the current time slot, providing updated state information for subsequent decision-making.

[0109] Step C3: Build an output layer based on an FC layer and an HPE layer;

[0110] The input of the output layer is the output of the intermediate layer, and the output of the output layer is the Q value (i.e., reward value) of all actions in the action space and the optimal action corresponding to the maximum Q value (i.e., maximum reward value);

[0111] Specifically, this step builds an output layer based on a FC layer and a permutation-equivariant HPE layer. The FC layer in the output layer receives the current time slot hidden state from the intermediate layer as input, while the HPE layer receives the features of all neighboring ground nodes and the current time slot hidden state from the intermediate layer as input. The HPE layer can process inputs with different permutations and produce outputs related to the permutation. This enables the output layer to generate Q values ​​for all actions in the action space. Finally, the output layer outputs these Q values, which provide a basis for the decision-making process and guide the drone agent to select the optimal action.

[0112] Step C4: building a hyperlocal Q network including the input layer, the intermediate layer, and the output layer;

[0113] The hyperlocal Q-network (also known as the hyperlocal value network) is used to fit the local Q-function (also known as the local value function) of the drone's agent. The input layer of the hyperlocal Q-network consists of a FC layer and two permutation-invariant HPI layers for processing entity sets. The intermediate layers of the hyperlocal Q-network are implemented based on the GRU structure for processing sequential data. The output layer of the hyperlocal Q-network consists of a FC layer and a permutation-equivariant HPI layer for generating the Q-values ​​of actions. The goal of the hyperlocal Q-network is to learn a local Q-function that predicts the value of each possible action based on the agent's local observations, thereby guiding the agent to make the optimal decision. The hyperlocal Q-network is a neural network architecture for multi-agent reinforcement learning. It solves decision-making problems in complex environments by processing local information and distributed decision-making. In this way, each agent can make decisions independently while taking into account the presence and activities of other agents, achieving efficient data collection and offloading. The hyperlocal Q-network learns the local Q-function of the drone agent in an end-to-end manner, predicting the value of each possible action in a given state and guiding the agent to make the optimal decision.

[0114] This embodiment uses the HyperQMIX algorithm to introduce the HPI layer and the HPE layer into the local Q network and the global hybrid network, respectively, thereby forming an improved hyper-local Q network and a hyper-hybrid network.

[0115] Step C5: building a super-hybrid network for combining the local Q-values ​​(i.e., local reward values) of all the agents into a global Q-value (i.e., global reward value);

[0116] Specifically, this embodiment can use an FC input layer (i.e., fully connected input layer) and an FC output layer (i.e., fully connected output layer) to build the super hybrid network; wherein, the super hybrid network is used to combine the local Q values ​​of all the intelligent agents into a global Q value; wherein, the input of the FC input layer is the local Q values ​​of all intelligent agents, and the output of the FC input layer is a single embedded feature; the input of the FC output layer is the single embedded feature output by the input layer, and the output of the FC output layer is the global Q value.

[0117] The hyperhybrid network consists of an FC input layer and an FC output layer, connected by a weight matrix and an activation function. The weight matrix is ​​adjusted during training to optimize the performance of the entire network. During training, the network adjusts its weights using a backpropagation algorithm to minimize the difference between the predicted and actual global Q values.

[0118] The hyperhybrid network is a neural network architecture for multi-agent reinforcement learning, used to combine the local Q-values ​​of all agents into a single global Q-value. The hyperhybrid network consists of an FC input layer and an FC output layer. The FC input layer receives the local Q-values ​​of all agents as input and converts the input into a single embedded feature vector through a nonlinear transformation. The FC output layer receives this embedded feature vector as input and outputs a scalar value, the global Q-value. In this way, the hyperhybrid network can effectively combine the decision information of multiple agents, providing a solution to the distributed decision-making problem in multi-agent systems.

[0119] Step C6: Combining the hyperlocal Q network of each agent and the hypermixed network to obtain the HyperQMIX algorithm.

[0120] This step combines each agent's hyperlocal Q-network with a hypermixed network, forming the HyperQMIX algorithm. In the HyperQMIX algorithm, each agent has a hyperlocal Q-network that predicts the Q-value of its action based on its local observations. These local Q-values ​​are then fed into the hypermixed network, which merges all the local Q-values ​​into a single global Q-value. This approach enables the HyperQMIX algorithm to effectively handle local information and distributed decision-making in multi-agent systems.

[0121] As a further introduction to the above embodiment, the weight matrix and bias vector of the FC input layer, as well as the weight matrix and bias vector of the FC output layer, may also be determined. The specific process is as follows:

[0122] Step D1: Concatenate the embedded features and drone features into an HPE layer and generate the weight matrix of the FC input layer through the absolute value activation function;

[0123] Among them, the embedded feature is an embedded feature obtained by summing the first processing result and the second processing result, the first processing result is the result obtained by using the HPI layer to process the feature vector of the drone entity, and the second processing result is the result obtained by using the HPI layer to process the feature vector of the ground node entity; the drone feature is the feature of the drone entity.

[0124] Step D2: Using the HPI module to process entity features in the environment state, and generating the weight matrix of the FC output layer through the absolute value activation function;

[0125] Step D3: The embedded features and drone features are concatenated and input into an HPE layer and the bias vector of the FC input layer is generated through the linear rectification function ReLU (Rectified Linear Unit);

[0126] Step D4: Use the HPI module to process the entity features in the environment state, and generate the bias vector of the FC output layer through the linear rectification function ReLU.

[0127] As for Figure 1 Further introduction to the corresponding embodiment can be performed on the intelligent agents of the multiple drones in the following manner: the intelligent agents of the multiple drones are centrally trained offline using the HyperQMIX algorithm so that the intelligent agents of the drones update the parameters of the hyperlocal Q network using experience samples; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

[0128] After using the HyperQMIX algorithm to perform centralized offline training on the intelligent agents of multiple drones, this embodiment can also deploy the trained intelligent agents to the corresponding drones; wherein the trained intelligent agents include a hyperlocal Q network with updated parameters; and perform strategy migration operations on the intelligent agents in the drones by means of model reloading.

[0129] The process described in the above embodiment is explained below through an embodiment in actual application.

[0130] Among the current solutions to the joint optimization problem of drone trajectories and communications, firstly, in the drone-to-ground communication system, environmental factors such as the deployment location and number of communication nodes have time-varying characteristics, which makes it difficult for the offline designed drone trajectory and communication strategies to meet the performance requirements in a dynamic environment. Secondly, traditional optimization methods usually have high computational complexity, which brings great difficulties to the real-time update of optimization variables in a dynamic environment. Finally, most of these methods rely on global environmental information for centralized solution, ignoring the distributed control requirements in multi-drone scenarios. To this end, the embodiments of the present application disclose a scalable multi-agent reinforcement learning method, device, equipment and medium, which can design a new drone trajectory and communication joint optimization solution for the drone trajectory and communication joint optimization problem, adapting to the dynamic characteristics and distributed control requirements of the actual system.

[0131] See Figure 2 , Figure 2 A flowchart of a scalable multi-agent reinforcement learning method provided in an embodiment of the present application specifically includes the following steps:

[0132] Step S201: Establish a problem model for collecting data from a number of ground nodes for multiple UAVs and unloading the data to a ground data center, and construct a constrained Markov decision problem based on the problem model.

[0133] In this embodiment, the problem model of multiple drones collecting data from a number of ground nodes and unloading the data to a ground data center is specifically as follows: Consider a scenario where multiple drones collect data from a number of ground nodes and unload the data to a ground data center, that is, N drones with limited data storage collect data from a number of ground nodes and unload the data to a ground data center. Collect data from multiple ground node devices and unload all data to a single ground data center. Represents the collection of all ground node devices. The location of each ground node device can be Indicates that , the location of the ground data center can be used Indicates that the drone is at a fixed altitude. Fly at a constant speed V. Divide time into multiple time slots, the length of each time slot is . Define the set To describe different time slots, so It represents the total number of time slots required for the UAV to collect all the data in a mission. In each time slot t, the position of the UAV is , and a minimum distance must be maintained between the two drones In the probabilistic line-of-sight model, the straight-line distance from ground node m to UAV n at time slot t is , the elevation angle is , based on these variables, the line-of-sight link probability can be expressed as , the probability of non-line-of-sight link can be expressed as , a and b are parameters related to the specific environment. The channel gain from UAV n to ground node m is expressed as , Indicates the channel gain in the line-of-sight situation, Represents the channel gain in non-line-of-sight conditions; similar methods can be used to calculate the channel gain from drone n to the data center. In order to describe the scheduling of UAV n to ground node m , that is, each UAV can only perform one operation in each time slot (schedule a ground node to collect data or unload data to the data center), and each ground node can only be scheduled by one UAV at most, so the binary scheduling variable Should meet , In order to determine the communication range for data collection and data offloading respectively, it is assumed that all drones communicate in the same frequency band, a directional antenna model is used to limit the energy of transmission and reception to a specific angle, and a binary indicator variable is introduced. To represent the coverage between the UAV and the ground node or data center, Indicates that the ground node m is within the coverage area of ​​UAV n, otherwise The transmission power of the ground node is set to , the communication bandwidth for transmitting data to the UAV is In order to reduce the energy consumption of ground nodes, it is stipulated that ground nodes only generate new perception data when they are dispatched by UAVs, and immediately upload the data to UAVs. The amount of data generated by each ground node each time is fixed at When the buffer of drone n reaches the upper limit of capacity, the drone will no longer collect new data until there is enough space in the buffer, so Constraints must be met , if represents a conditional judgment function. To ensure the continuity and integrity of data transmission, if the ground node fails to completely upload its data to the drone within a time slot, the scheduling relationship between the ground node and the drone must remain unchanged until all data is completely uploaded. Similarly, if the drone fails to completely unload its data to the data center within a time slot, the scheduling relationship between the drone and the data center must also remain unchanged until all data is completely unloaded. Therefore, Constraints must also be met ; The goal of this embodiment is to minimize the average information age of the system by optimizing the trajectory of the drone and the communication scheduling with the ground nodes. In other words, the problem model in this embodiment can provide a scalable multi-agent reinforcement learning method with the goal of minimizing the average information age of the system under the above constraints, so that the strategies learned by the agents can adapt to changes in the number of entities such as drones and ground nodes.

[0134] The process of constructing a constrained Markov decision problem based on the problem model includes the following steps E1 to E5:

[0135] Step E1: Constructing an environmental state feature including the current location information of the UAV, the location information of the current ground node, the location information of the current data center, the amount of data to be uploaded by the current ground node and the total amount of stored data of the UAV; wherein the environmental state feature The relationship is: ; Where n represents the number of the current drone, m represents the number of the current node, and t represents the number of the current time slot. and Represents the set of all UAV features and ground node features, UAV features ,in Indicates the current time slot position information of the UAV, represent the horizontal coordinate, vertical coordinate, and height of the UAV in the spatial coordinate system, respectively, and , is a constant, Indicates the total amount of data to be unloaded in the drone buffer, is the upper limit of the drone buffer zone capacity, represents the amount of data unloaded by UAV n at time slot t-1, represents the amount of data collected by drone n at time slot t-1, Indicates the scheduling relationship between the UAV and the data center in the previous time slot, Indicates whether the UAV has scheduled any ground node in the last time slot, and the characteristics of the ground node , represents the position of the ground node, where , Indicates the amount of data to be uploaded by the ground node, a binary indicator Indicates whether the ground node m generates a new data packet, fixed value Indicates the amount of data generated by the ground node each time, represents the amount of data uploaded by ground node m in time slot t-1, represents the information age of ground node m, represents the time when the data center last received a data packet from ground node m. Indicates whether node m was scheduled by any UAV in the previous time slot.

[0136] Step E2: Construct a probabilistic line-of-sight model that takes both line-of-sight and non-line-of-sight propagation paths into account; wherein the relational expression of the probabilistic line-of-sight model is: Where, and represent the probability of line-of-sight and non-line-of-sight links respectively, where a and b are parameters related to the specific environment. represents the elevation angle from ground node m to drone n, represents the straight-line distance from ground node m to drone n, Represents the norm.

[0137] Step E3: Construct a binary scheduling variable including the trajectory variable of the drone and the communication status of the ground node and the data center; wherein the drone trajectory variable and binary scheduling variables The relationship is: , Where, represents the set of all UAVs, ground nodes, and time slots. Each data collection task consists of T time slots.

[0138] Step E4: Design the average information age function of the ground nodes; wherein the average information age function The relationship is: Where, represents the average information age of ground node m during the entire data collection duration; Indicates the information age of the ground node.

[0139] Step E5: Construct a constrained Markov decision problem based on the environmental state characteristics, the probabilistic sight distance model, the trajectory variables and the binary scheduling variables, and the average information age function; wherein the goal of the constrained Markov decision problem is to solve the problem of minimizing the average information age of the system. .

[0140] Step S202: The problem is constructed as a distributed locally observable Markov decision process model Dec-POMDP by using entity decomposition.

[0141] In this embodiment, the process of constructing the problem into a distributed locally observable Markov decision process model Dec-POMDP by using the entity decomposition method specifically includes the following steps F1 to F3:

[0142] Step F1: Establish two time-varying sets to represent neighboring drones and ground nodes within the drone’s sensing range 、 ;

[0143] Step F2: The UAV’s action space is divided into two parts. The first part is all the scheduling actions related to the ground nodes. The second part is the movement and data unloading actions that are not related to specific entities. ; Wherein, the relationship formula of the scheduling action is: , the relationship between the movement action and the data offloading action is: Where, represents the action of selecting ground node m for data collection, is a finite set that discretizes the continuous direction space. Indicates the direction the drone is heading The action of flying, Indicates the action of the drone unloading data to the data center.

[0144] Step F3: Define the reward function based on the Dec-POMDP model ; Wherein, the reward function The relationship is: Where, represents the number of collisions caused by all drones in time slot t, and is the scaling factor.

[0145] Step S203: Design two types of neural network modules, the HPI layer and the HPE layer, and introduce them into the QMIX algorithm to form the HyperQMIX algorithm.

[0146] In this embodiment, the FC layer network is improved by designing an HPI layer that adapts to variable-length inputs and satisfies permutation invariance, and an HPE layer that adapts to variable-length outputs and satisfies permutation equivariance. This allows the improved hypernetwork to effectively handle changes in state dimensions while establishing a one-to-one correspondence between drone features and the input layer weight matrix, ensuring that the network model is unaffected by the number of entities. This step can be used to design a HyperQMIX algorithm for the constrained Markov decision problem by introducing two types of neural network modules, the HPI layer and the HPE layer, into the QMIX algorithm.

[0147] This embodiment establishes a problem model for collecting data from a number of ground nodes for multiple drones and offloading the data to a ground data center, constructs a constrained Markov decision problem based on the problem model, and constructs the problem as a distributed locally observable Markov decision process model Dec-POMDP using entity decomposition. The algorithm then introduces two types of neural network modules, the HPI layer and the HPE layer, into the QMIX algorithm to form the HyperQMIX algorithm. Thus, the present application first establishes a problem model for collecting data from a number of ground nodes for multiple drones and offloading the data to a ground data center, constructs a constrained Markov decision problem based on the problem model, and models the problem as Dec-POMDP. It then designs and introduces two types of neural network modules, the HPI layer and the HPE layer, into the QMIX algorithm, and proposes a scalable HyperQMIX algorithm that achieves permutation invariance and permutation equivariance in hyperlocal Q networks and hypermixed networks, making the network structure independent of the number of entities and improving the sample efficiency of the algorithm for fixed-scale tasks.

[0148] See Figure 3 , Figure 3 This is a flowchart of another scalable multi-agent reinforcement learning method provided in an embodiment of the present application. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution, specifically including the following steps:

[0149] Step S301: Establish a problem model for collecting data from a number of ground nodes for multiple UAVs and unloading the data to a ground data center, and construct a constrained Markov decision problem based on the problem model.

[0150] Step S302: The problem is constructed as a distributed locally observable Markov decision process model Dec-POMDP by using entity decomposition.

[0151] The constrained Markov decision problem entity is decomposed into the UAV as the intelligent entity and the ground node as the non-intelligent entity. The feature vector of the UAV entity is and the feature vector of the ground node Contains all environmental information related to it;

[0152] Use the above entity characteristics to define the key elements in Dec-POMDP: state , observation ,action and rewards , a distributed locally observable Markov decision process model Dec-POMDP was built;

[0153] Among them, the relationship between the drone entity feature vector is: , the ground node entity feature vector relationship is: , the relationship between the system state is: , the comprehensive observation The relationship is: , the relationship between the scheduling action is: , the relationship between the movement action and the data offloading action is: The reward function The relationship is: ;

[0154] Where, Indicates the location of the data center, Represents the set of all drone features, Represents the set of all ground node features, represents the feature set of UAV n itself, represents the feature set of neighboring drones, Represents the feature set of ground nodes, represents the action of selecting ground node m for data collection, is a finite set that discretizes the continuous direction space. Indicates the direction the drone is heading The action of flying, Indicates the action of the drone unloading data to the data center, represents the number of collisions caused by all drones in time slot t, and is the scaling factor.

[0155] Step S303: Building a scalable network based on the HPI layer and the HPE layer.

[0156] In this embodiment, in order to make the dimension of the weight matrix change synchronously with the number of input elements, a hypernetwork consisting of a single FC layer is built at the HPI layer. Yields each input element The corresponding weight matrix , the weight matrix extracts features from each input element, and then obtains the final network output by summing Build a super network based on the FC layer at the HPE layer Generate a weight matrix corresponding to each output element , The input element corresponding to each output element As input, in order to make the HPE layer adapt to the input of dynamic dimensions, an HPI layer is introduced Processing the overall input of the network , then each output element , a one-to-one correspondence is established between input elements and output elements. Represents the total number of input elements, and j represents the jth input element.

[0157] Step S304: Build a drone agent responsible for fitting local function Hyperlocal network; among them, represents the local observation of the UAV, represents the optimal action selected by the drone.

[0158] In this embodiment, the input layer uses one FC layer and two HPI layers to perceive all the features of the drone's local observation as input, and merges them into a single embedded feature through summation. As output, this process is represented as: ; 、 and Respectively represent the embedding functions implemented by the above network structure; the middle layer uses the GRU structure in the QMIX algorithm to further process the single embedding features output by the above input layer, and the GRU stores the hidden state of each time slot t , which is updated based on the embedded features and the hidden state of the previous time slot: , Represents the function implemented by the GRU layer; the output layer uses an FC layer and an HPE layer to take the hidden state output by the above intermediate layer as input and output the Q value of all entity-independent actions: , ,in, represents the output function implemented by the FC layer, represents the output function implemented by the HPE layer; combined with all Q values, that is, Determine the optimal action and its corresponding value.

[0159] Step S305: Build a super hybrid network that combines the local Q values ​​of all agents into a global Q value.

[0160] In this embodiment, the local Q values ​​of all agents are used as input to an FC input layer to generate an embedded feature ,in, and represent the weight matrix and bias vector of the input layer respectively, represents the input activation function, Represents the vector composed of all local Q values; then the above input layer output embedding features are used as the input of an FC output layer to generate a global Q value: ,in, and is the weight matrix and bias vector of the output layer, Represents the output activation function. The input of the FC input layer is the local Q value of all agents, and its output is a single embedded feature. The input of the FC output layer is the single embedded feature output by the above input layer , whose output is the global Q value.

[0161] The weight matrix and bias vector of the FC input layer and FC output layer are generated as follows:

[0162] The improved super network that generates the input layer weight matrix in the super hybrid network is denoted as , Use two HPI layers to process the features of drones and ground nodes respectively, and then merge the results into an embedded feature through summation. , the embedded features and drone features are concatenated and input into an HPE layer, and the weight matrix of each drone is generated through the absolute value activation function ,in, and represents the output function implemented by the HPI layer, represents the output function implemented by the HPE layer, is the absolute value activation function; the improved super network that produces the output layer weight matrix in the super hybrid network is recorded as The HPI module is used to process the entity features in the environment state, and the bias vector of each is generated by the absolute value activation function. ,in, and Denotes the embedding function implemented by the HPI layer. The bias vectors for the input and output layers in the superhybrid network are generated in a similar manner, but are not required to be non-negative. Therefore, the ReLU function is used instead of the absolute value function in the corresponding improved supernetwork. This embodiment allows for end-to-end training of all the network parameters constructed above.

[0163] Step S306: Use the HyperQMIX algorithm to train and update the parameters of the network constructed above.

[0164] In this embodiment, the Bellman equation corresponding to the global Q value is used to construct the loss function ,in, represents the global target Q value, θ represents the set of all learnable network parameters in HyperQMIX, θ −Denotes the target network parameters, which are periodically copied from θ. In this embodiment, the target network parameters can also be periodically updated by copying from the set of learnable network parameters. represents the coefficient of the desired operation on the data set D, represents the global Q value of the network output with parameter θ, represents the action that the drone chooses to perform at time t, represents the discount rate, represents the reward value at time t, Indicates status.

[0165] In the process of building a deep reinforcement learning framework based on the HyperQMIX algorithm for the constrained Markov decision problem, the following operations can also be performed:

[0166] Centralized offline training, where the UAV agent collects experience samples by interacting with the environment and uses these experience samples to update network parameters;

[0167] Distributed online execution, after the above training is completed, the trained hyperlocal Q network is deployed on each drone;

[0168] Inter-task strategy transfer, through model reloading, transfers the agent collaboration mechanism learned in the source task and other semantic information that helps new task decision-making to the new target task.

[0169] This embodiment addresses the joint optimization of drone trajectories and communication scheduling with ground nodes. It introduces a HPI layer, which adapts to variable-length inputs and satisfies permutation invariance, and an HPE layer, which adapts to variable-length outputs and satisfies permutation equivariance, to the local and hybrid networks of the QMIX algorithm in deep reinforcement learning. The HPI and HPE layer modules significantly enhance the model's representational capabilities, enabling hypernetwork parameters to be shared across different input elements, ensuring that the number of network parameters remains constant despite changes in input and output dimensions. Furthermore, the parallel processing of the computational process prevents exponential computational complexity from increasing with the number of elements. The introduction of the HPI and HPE modules ensures scalability of the newly formed HyperQMIX algorithm, enabling knowledge transfer between agents at different scales. This ensures that the network structure is unaffected by changes in the number of neighboring drones and ground nodes within the drone's perception range. This establishes permutation invariance and permutation equivariance between observations and actions. Furthermore, by leveraging prior environmental information within the network structure, the algorithm's training stability and sample efficiency are improved.

[0170] In order to verify the effectiveness of this embodiment, the effects of this application will be further explained below in conjunction with simulation experiments.

[0171] In addition to the HyperQMIX method proposed in this example, five other comparative methods were used in the simulation experiments: 1. Greedy (based on information age); 2. QMIX (based on deep learning); 3. DeepSets-QMIX (based on hybrid multi-agent reinforcement learning); 4. Attention-QMIX (based on hybrid multi-agent reinforcement learning); and 5. GNN-QMIX (based on hybrid multi-agent reinforcement learning). All deep reinforcement learning-related experimental code was implemented in Pytorch, an open-source deep learning framework for machine learning and deep learning.

[0172] In the simulation experiment, the wireless channel model between the UAV and the ground node is a probabilistic line-of-sight model, so the channel power gain at time slot t is It can be modeled as:

[0173] ;

[0174] in, is the carrier center frequency, α represents the path loss exponent, is the additional path loss coefficient in each case, represents the straight-line distance from ground node m to drone n, and c represents the speed of light. Therefore, the final channel gain is calculated as the expected value of line-of-sight and non-line-of-sight conditions. It can be modeled as:

[0175] ;

[0176] Among them, LoS represents the line-of-sight propagation path, and NLoS represents the non-line-of-sight propagation path. Assuming that all drones communicate in the same frequency band, the transmission power of the ground node is set to , the communication bandwidth for transmitting data to the UAV is , then the signal-to-interference-noise ratio of the communication link between ground node m and UAV n in time slot t can be expressed as:

[0177] ;

[0178] In the above formula and Represents different ground nodes, and Indicates different drones, and Respectively represent the coverage between the UAV and different ground nodes, and They represent the scheduling of different UAVs to different ground nodes. and They represent the channel gains from the UAV to different ground nodes.

[0179] in, represents the power spectral density of additive white Gaussian noise. Therefore, the maximum amount of data collected by drone n from ground node m in time slot t is It can be calculated as:

[0180] ;

[0181] Similarly, the transmission power of the UAV is , the communication bandwidth for offloading data to the data center is , the signal-to-interference-noise ratio between drone n and the data center can be obtained And the maximum amount of data that drone n offloads to the data center Therefore, in time slot t, the total amount of data uploaded by ground node m is , the total amount of data collected by drone n And the total amount of data unloaded by drone n They are:

[0182] ;

[0183] ;

[0184] ;

[0185] The information age of the ground node is , if the data center receives a new data packet from ground node m, will be reset, otherwise The average information age of ground node m during the entire data collection duration is It can be calculated by the following formula:

[0186] ;

[0187] Therefore, the average information of all ground nodes Age:

[0188] ;

[0189] It is worth noting that all the models assumed here are only used for simulation experiments. In the specific implementation of this embodiment, the UAV agent does not require any prior knowledge about the system model.

[0190] One segment is equivalent to the completion of a mission. In each segment, the initial positions of the UAV and the ground node are randomly generated within a 2000m×2000m area. The relevant parameters of the UAV are set as follows: , , The communication related parameters are set as follows: up =B off =1MHz, N0=-169dBm / Hz. For data collection, this application assumes , , , communication range for drone data collection , communication range for data offloading , the drone’s perception range of neighboring drones / sensors The channel gain related parameters are set to: a=9.61, b=0.15, , , During training, the exploration probability ϵ is the value of the first 10 6 The probability of the training process linearly decays from 1 to 0.05 over 5000 time steps and remains constant throughout the training process. The discount factor γ is set to 0.99. The Adam optimizer is used to update network parameters with a learning rate of 0.0005. The experience replay pool stores samples from the most recent 5000 training rounds, and each network update uses samples from 32 rounds. The total training process lasts for 105 rounds, with each round consisting of 150 time steps. Unless otherwise specified, the number of drones is 4 and the number of ground nodes is 64.

[0191] See Figure 4 、 Figure 5 and Figure 6 , Figure 4 Schematic diagram of convergence curves of the average information age of a HyperQMIX algorithm and a first group of comparative algorithms during training provided in an embodiment of the present application. Figure 5 Schematic diagram of the convergence curve of the average information age of a HyperQMIX algorithm and a second set of comparative algorithms during the training process provided in an embodiment of the present application. Figure 6 Schematic diagram of the convergence curve of the average information age of a HyperQMIX algorithm provided in an embodiment of the present application and a third group of comparative algorithms during the training process. Figure 4 、 Figure 5 and Figure 6 The horizontal axis represents the number of training rounds, and the vertical axis represents the average information age (s). The figure shows the positions corresponding to 1600 training rounds. Figure 4The algorithm in this paper is the HyperQMIX algorithm, the first algorithm is the greedy algorithm Greedy based on information age, and the second algorithm is the DeepSets-QMIX algorithm. Figure 5 The first algorithm is the HyperQMIX algorithm, the third algorithm is the QMIX algorithm, and the fourth algorithm is the GNN-QMIX algorithm. Figure 6 The first algorithm is the HyperQMIX algorithm, and the fifth algorithm is the Attention-QMIX algorithm. All multi-agent reinforcement learning algorithms perform better than the greedy algorithm. The average information age achieved by the greedy algorithm after convergence is about 650s, which is 56.6% higher than the best-performing HyperQMIX algorithm, and even 8.8% higher than the worst-performing QMIX algorithm. The final average information age of HyperQMIX is 415s, which is 11.3% lower than the second-best performing GNN-QMIX and 30.7% lower than the worst-performing QMIX. These results verify the significant advantage of HyperQMIX in reducing the average information age. Secondly, HyperQMIX also performs well in terms of convergence speed and sample efficiency, reducing computing resources and the need to generate experience samples. By Figures 4 to 6 HyperQMIX surpassed the convergence performance of all baseline algorithms in just approximately 1,600 training rounds. In contrast, GNN-QMIX required approximately 105 rounds to reach a similar level of performance. Ultimately, HyperQMIX converged after approximately 4×104 training rounds, while the other algorithms required over 8×104 training rounds to reach near convergence. HyperQMIX's performance advantage is primarily due to its effective utilization of permutation invariance and permutation equivariance in multi-agent systems, as well as its use of a hypernetwork to dynamically generate embedding layer parameters, significantly improving its representational capabilities and thus achieving better performance.

[0192] See Figure 7 、 Figure 8 and Figure 9 , Figure 7 This is a schematic diagram of the trajectory of a drone generated by the strategy learned by the HyperQMIX algorithm in the first period of a round in a simulation experiment provided by an embodiment of the present application. Figure 8 A schematic diagram of the trajectory of a drone generated by the strategy learned by the HyperQMIX algorithm in the second period of a round in a simulation experiment provided by an embodiment of the present application; Figure 9This figure illustrates the trajectory of drones generated during the third period of a round using the strategy learned by the HyperQMIX algorithm in a simulation experiment provided by an embodiment of the present application. In these trajectory diagrams, x and y represent the horizontal and vertical coordinates, measured in meters. The arrows in the dashed box U1 represent the trajectory of drone 1, the arrows in the dashed box U2 represent the trajectory of drone 2, the arrows in the dashed box U3 represent the trajectory of drone 3, and the arrows in the dashed box U4 represent the trajectory of drone 4. Squares represent data centers, and triangles represent ground nodes. Through trajectory design, the four drones collaborate to serve all ground nodes to improve data collection efficiency. During each time period, the drones collect data in different areas, maintaining appropriate distances to ensure coverage of more ground nodes while avoiding collisions and inter-link interference. Each drone flexibly switches between movement, data collection, and data offloading. Circles represent the locations where the drones collect data at each time. These locations are mostly located around ground nodes, indicating that the drones have learned to optimize the air-ground channel through trajectory design, improving data transmission efficiency. Stars indicate locations where drones offload data. It can be seen that data offloading typically occurs when the drones move from one ground node to another. Once within the data center's communication range, drones immediately offload their collected data, preventing the continued increase in information age. Furthermore, due to the constant changes in observed information within their perception range as they move, drones tend to fly to areas with a higher average information age and fewer drones, rather than consistently serving the same ground nodes in the same area. These results further validate the effectiveness of HyperQMIX in policy learning.

[0193] See Figure 10 and Figure 11 , Figure 10 This is a schematic diagram of the convergence curves of multiple algorithms in a simulation experiment provided in an embodiment of the present application. Figure 11 This is a schematic diagram of the convergence curves of multiple algorithms in another simulation experiment provided in an embodiment of the present application. Figure 10 and Figure 11 The horizontal axis is the number of training rounds and the vertical axis is the average information age (s). Figure 10Convergence curves are presented for the first transfer learning algorithm, ZT-HyperQMIX; the second transfer learning algorithm, TL-HyperQMIX; our algorithm (HyperQMIX algorithm); and the third algorithm, QMIX. ZT-HyperQMIX (Zero-Shot Transfer-based HyperQMIX) represents HyperQMIX based on zero-shot transfer. TL-HyperQMIX (Transfer Learning-based HyperQMIX) represents HyperQMIX based on transfer learning. It can be seen that model reloading achieves better initial performance on the target task. When training HyperQMIX from scratch, the initial average information age is 741 seconds. After using the pretrained model, the initial average information age of ZT-HyperQMIX and TL-HyperQMIX is reduced to approximately 480 seconds, which is only 8.8% higher than the result after convergence of HyperQMIX. Furthermore, this result is 24.6% lower than the final average information age of QMIX, validating the effectiveness of pre-trained models in handling tasks of varying scales. By further fine-tuning the pre-trained model, TL-HyperQMIX quickly achieved performance comparable to training from scratch. These results demonstrate HyperQMIX's excellent knowledge transfer capabilities.

[0194] See Figure 12 , Figure 12 A schematic diagram of a convergence curve of an ablation experiment in a simulation experiment provided in an embodiment of the present application. Figure 12The horizontal axis represents the number of training rounds, and the vertical axis represents the average information age (s). The figure shows the positions at 3400 and 17000 training rounds. The figure also shows the convergence curves for the network using our HyperQMIX algorithm, the network using the third algorithm rQMIX, the first reference network without HPE, the second reference network without HLQN, and the third reference network without HMN. "with HPE" represents a network in which only all HPE layers in HyperQMIX are replaced with FC layers, "with HLQN" represents a network in which only the hyperlocal Q network in HyperQMIX is replaced with the standard local Q network in QMIX, and "with HMN" represents a network in which only the hyperhybrid network in HyperQMIX is replaced with the standard hybrid Q network in QMIX. It can be seen that HyperQMIX, with the HPE layer removed, outperforms algorithms using other permutation-invariant models, such as DeepSets, Attention, and GNNs. This is because the hypernetwork-based weight generation mechanism in the HPI layer enhances the network's representational capabilities, leading to improved performance. Compared with the standard HyperQMIX, after removing the HPE layer, the average information age of the algorithm finally increased from 415s to 485s, an increase of 17%. This change reflects the performance loss caused by removing the HPE layer. After removing the hyperlocal Q network, the algorithm 4 The algorithm converged quickly within 10 training rounds, but its average information age after convergence reached 540 seconds, 30% higher than HyperQMIX. This demonstrates that the hyperlocal Q network, with its HPI and HPE layers, significantly outperforms the standard local Q network in finding the optimal policy. Removing the hypermixing network resulted in slower convergence, indicating that the hypermixing network primarily contributes to the convergence speed.

[0195] See Figure 13 and Figure 14 , Figure 13 This is a schematic diagram comparing the average information age of different algorithms under different numbers of ground nodes provided by the embodiment of the present application. Figure 14 A schematic diagram comparing the average information age of different algorithms under different numbers of drones provided in an embodiment of the present application. Figure 13 The horizontal axis is the number of ground nodes, the vertical axis is the average information age (s), M represents the number of ground nodes, and N represents the number of UAVs; Figure 14The horizontal axis represents the number of drones, and the vertical axis represents the average information age (s). The algorithms used for comparison include the transfer learning algorithm ZT-HyperQMIX, the transfer learning algorithm TL-HyperQMIX, the HyperQMIX algorithm, and the QMIX algorithm. In the figure, ① represents the HyperQMIX algorithm, ② represents the QMIX algorithm, ③ represents the transfer learning algorithm ZT-HyperQMIX, and ④ represents the transfer learning algorithm TL-HyperQMIX. Figure 13 The performance of different algorithms in terms of average information age under different numbers of ground nodes in the simulation experiment is shown; Figure 14 The performance of different algorithms in terms of mean information age (MIA) for different numbers of UAVs in simulation experiments is shown. As the number of ground nodes increases, the QMIX algorithm struggles to cope with the exponential growth of the state-action space, leading to a gradual decline in performance and an increase in mean information age. In contrast, HyperQMIX, utilizing its improved hyperlocal Q-network and hyperhybrid network models, effectively handles the explosive growth of the state-action space and exhibits superior scalability. Consequently, HyperQMIX significantly outperforms QMIX in all scenarios, and this advantage becomes more pronounced with increasing numbers of ground nodes. The performance gap between HyperQMIX and QMIX widens as the number of UAVs increases. For example, with only one UAV, HyperQMIX achieves a 7.3% performance improvement over QMIX. This advantage increases to 58% when the number of UAVs reaches 16, further validating HyperQMIX's scalability.

[0196] Based on the above embodiments, the present application provides a real-time drone trajectory optimization method that minimizes the task completion time under energy constraints for a single drone-assisted IoT data collection system. The present application first constructs a restricted Markov constraint process based on the original problem. Subsequently, the proposed method can update the original domain policy network parameters and the dual domain Lagrangian penalty coefficients in turn, and finally enables the drone agent to learn an adaptive strategy that can generate the optimal trajectory in real time under different device location layouts. Simulation results show that the method proposed in this application has improved the average task completion time and computational complexity compared to traditional optimization algorithms and other deep reinforcement learning algorithms, and has great practical significance.

[0197] The present application provides a scalable multi-agent reinforcement learning device for a trajectory planning device for collecting data from a drone, the device comprising:

[0198] A problem building module is used to establish a problem model for collecting data from a number of ground nodes for multiple drones and offloading the data to a ground data center, and to construct a constrained Markov decision problem based on the problem model; a framework building module is used to build a deep reinforcement learning framework based on the SAC algorithm for the constrained Markov decision problem;

[0199] The model building module models the problem as Dec-POMDP using entity decomposition;

[0200] Framework building module, designing two types of neural network modules, HPI layer and HPE layer, and introducing them into local Q network and global hybrid network to build a super local Q network and super hybrid network framework that meets permutation invariance and permutation equivariance;

[0201] Algorithm design module, designs a scalable multi-agent strategy learning algorithm based on HyperQMIX.

[0202] This embodiment establishes a problem model for collecting data from a number of ground nodes for multiple drones and offloading this data to a ground data center, and constructs a constrained Markov decision problem based on the problem model. The problem is modeled as a Dec-POMDP using entity decomposition. Two types of neural network modules, the HPI layer and the HPE layer, are designed and introduced into the local Q network and the global hybrid network to form a hyperlocal Q network and a hyperhybrid network that satisfy permutation invariance and permutation equivariance. A scalable HyperQMIX algorithm based on the QMIX algorithm is proposed. Thus, the present application first establishes a problem model for collecting data from a number of ground nodes for multiple drones and offloading this data to a ground data center, and constructs a constrained Markov decision problem based on the problem model. The problem is modeled as a Dec-POMDP. Then, two types of neural network modules, the HPI layer and the HPE layer, are designed and introduced into the QMIX algorithm. The scalable HyperQMIX algorithm achieves permutation invariance and permutation equivariance in the hyperlocal Q network and the hyperhybrid network, making the network structure independent of the number of entities and improving the sample efficiency of the algorithm for fixed-scale tasks.

[0203] See Figure 15 , Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the steps provided in the above embodiment.

[0204] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0205] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0206] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0207] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, enabling the processor 21 to calculate and process the massive amount of data 223 in the memory 22. It can be run under Windows, Unix, Linux, or other operating systems. In addition to including computer programs capable of implementing the trajectory planning method for collecting data from a drone, as disclosed in any of the aforementioned embodiments, the computer programs 222 may also include computer programs capable of performing other specific tasks. Data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0208] Furthermore, embodiments of the present application disclose a storage medium storing a computer program that, when executed, implements the steps provided in the above embodiments. The storage medium may include a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0209] See Figure 16 , Figure 16 This is a schematic diagram of the structure of a drone control system provided in an embodiment of the present application. The system may include:

[0210] A problem construction module is used to establish a problem model and construct a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that multiple drones collect data from multiple ground nodes and offload the collected data to a ground data center;

[0211] A model building module, configured to model the constrained Markov decision problem as a target model; wherein the target model is a distributed locally observable Markov decision process model;

[0212] An algorithm building module, configured to obtain a target algorithm by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model and is configured to solve an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance;

[0213] A control module is used to train the intelligent agents of the plurality of drones using the target algorithm, and to perform trajectory planning and / or communication scheduling for the drones using the trained intelligent agents.

[0214] This embodiment first establishes a problem model for multiple drones collecting data from multiple ground nodes and offloading the collected data to a ground data center. Based on this problem model, a constrained Markov decision problem is constructed, modeled as a Dec-POMDP model. The Dec-POMDP model allows each drone agent to make decisions based on local information while taking into account the presence and activities of other agents, helping to address uncertainty in multi-agent systems and achieve distributed control. This embodiment introduces the HPI layer and HPE layer into the QMIX algorithm to obtain the HyperQMIX algorithm. Because the HPI layer is a permutation-invariant network layer based on the FC layer, and the HPE layer is a permutation-equivariant network layer based on the FC layer, the resulting HyperQMIX algorithm can solve agent policies that adapt to changes in the number of entities. This embodiment uses the HyperQMIX algorithm to train multiple drone agents, allowing the trained agents to make decisions based on local information to adapt to dynamic characteristics and distributed control requirements. This embodiment uses the Dec-POMDP model and the HyperQMIX algorithm to enable the trained intelligent agent to adapt to various dynamic environmental factors and distributed control requirements during the process of controlling the drone.

[0215] Furthermore, the process of constructing a constrained Markov decision problem based on the problem model by the problem construction module includes: determining environmental state characteristics according to the problem model; wherein the environmental state characteristics include the current position information of the UAV, the position information of the current ground node, the position information of the current ground data center, the amount of data to be uploaded of the current ground node and the total amount of stored data of the UAV; determining a probabilistic line-of-sight model according to the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path; determining the trajectory variables and binary scheduling variables of the UAV according to the problem model; wherein the binary scheduling variables are used to describe the communication status between the ground node and the ground data center; determining the average information age function of the ground node according to the problem model; constructing the constrained Markov decision problem based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

[0216] Furthermore, the process of the model building module modeling the constrained Markov decision problem as a target model includes: performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity; wherein, the feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node; using the feature vector of the drone entity and the feature vector of the ground node entity to define key elements, and building the target model based on the key elements; wherein, the key elements include state, observation, action and reward.

[0217] Furthermore, it also includes:

[0218] A network layer construction module is used to introduce a first super network shared by all input elements into a single fully connected layer to form the first network layer before obtaining the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm; wherein the first super network takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features of the single input element and sum them to obtain the network output; and is also used to introduce a second super network shared by all output elements and a first network layer for processing the overall input of the output elements into a single fully connected layer to form a second network layer; wherein the second super network takes the input element corresponding to the single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply with the output of the first network layer to obtain the network output.

[0219] Furthermore, the algorithm construction module obtains the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, including: building an input layer based on a fully connected layer and two first network layers; wherein the input of the input layer is all features of the local observation of the drone, and the output of the input layer is a single embedded feature merged by a summation operation; building an intermediate layer based on the gated recurrent unit in the multi-agent reinforcement learning algorithm; wherein the input of the intermediate layer is the single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot; building an output layer based on a fully connected layer and a second network layer; wherein the input of the output layer is the output of the intermediate layer, and the output of the output layer is the reward values ​​of all actions in the action space and the optimal action corresponding to the maximum reward value; building a super local value network including the input layer, the intermediate layer, and the output layer; wherein the super local value network is used to fit the local value function of the drone's agent; building a super hybrid network for combining the local reward values ​​of all the agents into a global reward value; and combining the super local value network of each agent and the super hybrid network to obtain the target algorithm.

[0220] Furthermore, the process of the algorithm construction module building a super-hybrid network for combining the local reward values ​​of all the agents into a global reward value includes: using a fully connected input layer and a fully connected output layer to build the super-hybrid network; wherein, the super-hybrid network is used to combine the local reward values ​​of all the agents into a global reward value; wherein, the input of the fully connected input layer is the local reward value of all the agents, and the output of the fully connected input layer is a single embedded feature; the input of the fully connected output layer is a single embedded feature output by the input layer, and the output of the fully connected output layer is the global reward value.

[0221] Furthermore, it also includes:

[0222] A setting module is used to determine the weight matrix and bias vector of the fully connected input layer; and is also used to determine the weight matrix and bias vector of the fully connected output layer.

[0223] Furthermore, the process of the control module using the target algorithm to train the intelligent agents of the multiple drones includes: using the target algorithm to perform centralized offline training on the intelligent agents of the multiple drones, so that the intelligent agents of the drones use experience samples to update the parameters of the hyperlocal value network; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

[0224] Furthermore, it also includes:

[0225] A strategy migration module is used to deploy the trained agents to the corresponding drones after centralized offline training of the agents of multiple drones using the target algorithm; wherein the trained agents include a hyperlocal value network with updated parameters; and is also used to perform strategy migration operations on the agents in the drones by means of model reloading.

[0226] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and will not be repeated here.

[0227] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0228] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0229] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0230] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0231] Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A drone control method, characterized in that: include: Establishing a problem model and constructing a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent multiple drones collecting data from multiple ground nodes and offloading the collected data to a ground data center; Modeling the constrained Markov decision problem as a target model; wherein the target model is a distributed locally observable Markov decision process model; A target algorithm is obtained by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model and is used to solve an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and has permutation invariance, and the second network layer is a network layer based on a fully connected layer and has permutation equivariance; Using the target algorithm to train the intelligent agents of the plurality of drones, and using the trained intelligent agents to perform trajectory planning and / or communication scheduling for the drones; The constrained Markov decision problem is modeled as a target model, including: Performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity; wherein the feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node; Key elements are defined using the feature vector of the drone entity and the feature vector of the ground node entity, and the target model is built based on the key elements; wherein the key elements include state, observation, action and reward.

2. The drone control method according to claim 1, characterized in that: Constructing a constrained Markov decision problem based on the problem model includes: Determining environmental state characteristics according to the problem model; wherein the environmental state characteristics include the current location information of the UAV, the location information of the current ground node, the location information of the current ground data center, the amount of data to be uploaded by the current ground node, and the total amount of stored data of the UAV; Determine a probabilistic line-of-sight model based on the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path; Determining the trajectory variables and binary scheduling variables of the UAV according to the problem model; wherein the binary scheduling variables are used to describe the communication status between the ground node and the ground data center; Determining an average information age function of the ground nodes according to the problem model; The constrained Markov decision problem is constructed based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

3. The drone control method according to claim 1, characterized in that: Before obtaining the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, it also includes: Introducing a first super-network shared by all input elements into a single fully connected layer to form the first network layer; wherein the first super-network takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features of the single input element and sum them to obtain a network output; A second super-network shared by all output elements and a first network layer for processing the overall input of the output elements are introduced into a single fully connected layer to form a second network layer; wherein, the second super-network takes the input element corresponding to the single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply with the output of the first network layer to obtain the network output.

4. The drone control method according to claim 1, characterized in that: The target algorithm is obtained by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, including: An input layer is constructed based on a fully connected layer and two first network layers; wherein the input of the input layer is all features in the local observation of the drone, and the output of the input layer is a single embedded feature merged by a summation operation; An intermediate layer is constructed based on the gated recurrent unit in the multi-agent reinforcement learning algorithm; wherein the input of the intermediate layer is the single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot; An output layer is constructed based on a fully connected layer and a second network layer; wherein the input of the output layer is the output of the intermediate layer, and the output of the output layer is the reward values ​​of all actions in the action space and the optimal action corresponding to the maximum reward value; Building a hyperlocal value network comprising the input layer, the intermediate layer, and the output layer; wherein the hyperlocal value network is used to fit the local value function of the intelligent agent of the drone; Building a hyperhybrid network for combining the local rewards of all the agents into a global reward value; The target algorithm is obtained by combining the hyperlocal value network of each agent and the hypermixed network.

5. The drone control method according to claim 4, characterized in that: The hypermixing network for combining the local reward values ​​of all the agents into a global reward value comprises: The hyperhybrid network is constructed using a fully connected input layer and a fully connected output layer; wherein the hyperhybrid network is used to combine the local reward values ​​of all the agents into a global reward value; Among them, the input of the fully connected input layer is the local reward value of all intelligent agents, and the output of the fully connected input layer is a single embedded feature; the input of the fully connected output layer is the single embedded feature output by the input layer, and the output of the fully connected output layer is the global reward value.

6. The drone control method according to claim 5, characterized in that: Also includes: Determining a weight matrix and a bias vector of the fully connected input layer; The weight matrix and bias vector of the fully connected output layer are determined.

7. The drone control method according to claim 4, characterized in that: Training the intelligent agents of the plurality of drones using the target algorithm includes: The target algorithm is used to perform centralized offline training on the intelligent agents of the plurality of drones, so that the intelligent agents of the drones update the parameters of the hyperlocal value network using experience samples; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

8. The drone control method according to claim 7, characterized in that: After performing centralized offline training on the intelligent agents of the plurality of drones using the target algorithm, the method further includes: Deploying the trained intelligent agent to the corresponding UAV; wherein the trained intelligent agent includes a hyperlocal value network with updated parameters; The strategy migration operation is performed on the intelligent agent in the UAV by means of model reloading.

9. A UAV control system, characterized in that: include: A problem construction module is used to establish a problem model and construct a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that multiple drones collect data from multiple ground nodes and offload the collected data to a ground data center; A model building module, configured to model the constrained Markov decision problem as a target model; wherein the target model is a distributed locally observable Markov decision process model; An algorithm building module, configured to obtain a target algorithm by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model and is configured to solve an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance; a control module, configured to train the intelligent agents of the plurality of drones using the target algorithm, and perform trajectory planning and / or communication scheduling for the drones using the trained intelligent agents; Among them, the process of the model building module modeling the constrained Markov decision problem as a target model includes: performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity; wherein the feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node; using the feature vector of the drone entity and the feature vector of the ground node entity to define key elements, and building the target model based on the key elements; wherein the key elements include state, observation, action and reward.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the drone control method according to any one of claims 1 to 8 are implemented.

11. A storage medium, characterized in that: The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the drone control method according to any one of claims 1 to 8.

Images