Unmanned aerial vehicle control method and system, electronic equipment and storage medium

Abstract

The invention discloses an unmanned aerial vehicle control method and system, electronic equipment and a storage medium, and belongs to the technical field of unmanned aerial vehicle communication technologies. The unmanned aerial vehicle control method comprises the following steps: establishing a problem model, and constructing a constraint Markov decision problem based on the problem model; modeling the constraint Markov decision problem into a target model; a target algorithm is obtained by introducing the first network layer and the second network layer into a multi-agent reinforcement learning algorithm; and training the agents of the plurality of unmanned aerial vehicles by using the target algorithm, and performing trajectory planning and / or communication scheduling on the unmanned aerial vehicles by using the trained agents. According to the invention, unmanned aerial vehicle control adapting to dynamic characteristics and distributed control requirements can be realized.

Description

Technical Field

[0001] The present application relates to the field of drone communication technology, and in particular to a drone control method, system, electronic device and storage medium. Background Art

[0002] In order to give full play to the advantages of UAV-assisted communication technology, research on the joint optimization of UAV trajectory and communication has received widespread attention in the academic community. In the UAV-to-ground communication service scenario, the communication service quality of the ground node is jointly affected by the UAV trajectory and communication strategy. The UAV trajectory determines the relative position and distance between the UAV and the ground node, which in turn affects the quality of the air-ground channel. The communication strategy further affects the communication transmission quality between the UAV and the ground node by reasonably adjusting factors such as the allocation of communication resources, the association between the ground node and the UAV, and the communication scheduling order of the UAV to the ground node. These factors are interdependent and tightly coupled, and jointly determine the overall communication performance of the system. Therefore, studying the joint optimization of UAV trajectory and communication is of great significance to improving the performance of the UAV-to-ground communication system.

[0003] The specific manifestations of the joint optimization of UAV trajectory and communication in different scenarios are different, but these problems are essentially multivariable non-convex optimization problems with complex optimization objectives and constraints, which are difficult to solve using conventional convex optimization tools. To this end, a large number of studies have proposed optimization methods based on traditional mathematical tools. These methods solve the optimization problem offline based on static optimization models and environmental prior information, providing theoretically optimal or approximately optimal solutions for the joint optimization of UAV trajectory and communication. However, these methods face a series of limitations in practical applications. First, in the UAV-to-ground communication system, environmental factors such as the deployment location and number of communication nodes have time-varying characteristics, which makes it difficult for the offline designed UAV trajectory and communication strategies to meet the performance requirements in a dynamic environment. Second, traditional optimization methods usually have high computational complexity, which brings great difficulties to the real-time update of optimization variables in a dynamic environment. Most of these methods rely on global environmental information for centralized solution, ignoring the distributed control requirements in multi-UAV scenarios.

[0004] Therefore, how to achieve UAV control that adapts to dynamic characteristics and distributed control requirements is a technical problem that technical personnel in this field currently need to solve. Summary of the invention

[0005] The purpose of this application is to provide a drone control method, system, electronic device and storage medium, which can realize drone control that adapts to dynamic characteristics and distributed control requirements.

[0006] In order to solve the above technical problems, the present application provides a drone control method, comprising:

[0007] Establishing a problem model, and constructing a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that multiple drones collect data from multiple ground nodes and unload the collected data to a ground data center;

[0008] Modeling the constrained Markov decision problem as a target model; wherein the target model is a distributed locally observable Markov decision process model;

[0009] A target algorithm is obtained by introducing the first network layer and the second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model, and is used to solve an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance;

[0010] The target algorithm is used to train the intelligent agents of the plurality of drones, and the trained intelligent agents are used to perform trajectory planning and / or communication scheduling for the drones.

[0011] Optionally, constructing a constrained Markov decision problem based on the problem model includes:

[0012] Determine the environmental state characteristics according to the problem model; wherein the environmental state characteristics include the current location information of the UAV, the location information of the current ground node, the location information of the current ground data center, the amount of data to be uploaded by the current ground node, and the total amount of stored data of the UAV;

[0013] Determine a probabilistic line-of-sight model according to the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path;

[0014] Determining the trajectory variable and the binary scheduling variable of the UAV according to the problem model; wherein the binary scheduling variable is used to describe the communication status between the ground node and the ground data center;

[0015] Determine the average information age function of the ground nodes according to the problem model;

[0016] The constrained Markov decision problem is constructed based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

[0017] Optionally, the constrained Markov decision problem is modeled as a target model, including:

[0018] Performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity; wherein the feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node;

[0019] Key elements are defined using the feature vector of the drone entity and the feature vector of the ground node entity, and the target model is built based on the key elements; wherein the key elements include state, observation, action and reward.

[0020] Optionally, before obtaining the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, the method further includes:

[0021] Introducing a first super network shared by all input elements into a single fully connected layer to form the first network layer; wherein the first super network takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features from the single input element and sum them to obtain a network output;

[0022] A second super network shared by all output elements and a first network layer for processing the overall input of the output elements are introduced into a single fully connected layer to form a second network layer; wherein the second super network takes the input element corresponding to the single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply the output of the first network layer to obtain the network output.

[0023] Optionally, obtaining the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm includes:

[0024] An input layer is constructed based on a fully connected layer and two first network layers; wherein the input of the input layer is all features in the local observation of the UAV, and the output of the input layer is a single embedded feature merged by a summation operation;

[0025] An intermediate layer is built based on the gated recurrent unit in the multi-agent reinforcement learning algorithm; wherein the input of the intermediate layer is a single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot;

[0026] An output layer is constructed based on a fully connected layer and a second network layer; wherein the input of the output layer is the output of the intermediate layer, and the output of the output layer is the reward values ​​of all actions in the action space and the optimal action corresponding to the maximum reward value;

[0027] Building a hyperlocal value network including the input layer, the intermediate layer and the output layer; wherein the hyperlocal value network is used to fit the local value function of the intelligent agent of the drone;

[0028] Building a hypermixing network for combining the local rewards of all the agents into a global reward value;

[0029] The target algorithm is obtained by combining the hyperlocal value network of each agent and the hypermixed network.

[0030] Optionally, the building of a super hybrid network for combining the local reward values ​​of all the agents into a global reward value includes:

[0031] The super-hybrid network is constructed using a fully connected input layer and a fully connected output layer; wherein the super-hybrid network is used to combine the local reward values ​​of all the agents into a global reward value;

[0032] Among them, the input of the fully connected input layer is the local reward value of all intelligent agents, and the output of the fully connected input layer is a single embedded feature; the input of the fully connected output layer is a single embedded feature output by the input layer, and the output of the fully connected output layer is a global reward value.

[0033] Optionally, also include:

[0034] Determining a weight matrix and a bias vector of the fully connected input layer;

[0035] The weight matrix and bias vector of the fully connected output layer are determined.

[0036] Optionally, using the target algorithm to train the intelligent agents of the plurality of drones includes:

[0037] The target algorithm is used to perform centralized offline training on the intelligent agents of the plurality of drones, so that the intelligent agents of the drones update the parameters of the hyperlocal value network using experience samples; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

[0038] Optionally, after the target algorithm is used to perform centralized offline training on the intelligent agents of the plurality of drones, the method further includes:

[0039] Deploy the trained intelligent agent to the corresponding drone; wherein the trained intelligent agent includes a hyperlocal value network with updated parameters;

[0040] The strategy transfer operation is performed on the intelligent agent in the drone by means of model reloading.

[0041] The present application also provides a drone control system, the system comprising:

[0042] A problem construction module, used to establish a problem model, and to construct a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that a plurality of drones collect data from a plurality of ground nodes and unload the collected data to a ground data center;

[0043] A model building module, used to model the constrained Markov decision problem as a target model; wherein the target model is a distributed local observable Markov decision process model;

[0044] An algorithm building module, used for obtaining a target algorithm by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model, and is used for solving an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance;

[0045] The control module is used to train the intelligent agents of the plurality of drones using the target algorithm, and perform trajectory planning and / or communication scheduling for the drones using the trained intelligent agents.

[0046] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned drone control method when calling the computer program in the memory.

[0047] The present application also provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the above-mentioned drone control method are implemented.

[0048] The present application provides a method for controlling a drone. The method first establishes a problem model for representing that multiple drones collect data from multiple ground nodes and unload the collected data to a ground data center, and constructs a constrained Markov decision problem based on the problem model, and models the constrained Markov decision problem as a distributed local observable Markov decision process model Dec-POMDP. The Dec-POMDP model allows each drone agent to make decisions based on local information, while taking into account the existence and activities of other agents, which helps to deal with uncertainties in multi-agent systems and achieve distributed control. The present application introduces the first network layer HPI and the second network layer HPE into the multi-agent reinforcement learning algorithm QMIX to obtain a target algorithm. Since the first network layer HPI is a network layer based on a fully connected layer and has permutation invariance, and the second network layer HPE is a network layer based on a fully connected layer and has permutation equivariance, the target algorithm obtained in the present application can solve the agent strategy that adapts to changes in the number of entities. The present application uses the target algorithm to train the agents of multiple drones so that the trained agents can make decisions based on local information to adapt to dynamic characteristics and distributed control requirements. This application uses the Dec-POMDP model and the target algorithm to enable the trained agent to adapt to various dynamic environmental factors and distributed control requirements in the process of controlling the drone. This application also provides a drone control system, electronic equipment and storage medium, which have the above-mentioned beneficial effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A flowchart of a drone control method provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of a scalable multi-agent reinforcement learning method provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of another scalable multi-agent reinforcement learning method provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of convergence curves of average information age of a HyperQMIX algorithm and a first group of comparison algorithms during training provided in an embodiment of the present application;

[0054] Figure 5A schematic diagram of convergence curves of average information age of a HyperQMIX algorithm and a second group of comparison algorithms during training provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of convergence curves of average information age of a HyperQMIX algorithm and a third group of comparative algorithms during training provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a drone trajectory generated by a strategy learned by a HyperQMIX algorithm provided in an embodiment of the present application in the first period of a round;

[0057] Figure 8 A schematic diagram of a drone trajectory generated in the second period of a round by a strategy learned by a HyperQMIX algorithm provided in an embodiment of the present application;

[0058] Fig. 9 A schematic diagram of a drone trajectory generated by a strategy learned by a HyperQMIX algorithm provided in an embodiment of the present application in the third period of a round;

[0059] Fig.10 A schematic diagram of convergence curves of multiple algorithms in a simulation experiment provided in an embodiment of the present application;

[0060] Fig.11 A schematic diagram of convergence curves of multiple algorithms in another simulation experiment provided in an embodiment of the present application;

[0061] Fig.12 A schematic diagram of a convergence curve of an ablation experiment in a simulation experiment provided in an embodiment of the present application;

[0062] Fig.13 A schematic diagram showing a comparison of average information ages of different algorithms under different numbers of ground nodes provided in an embodiment of the present application;

[0063] Fig.14 A schematic diagram showing a comparison of average information ages of different algorithms under different numbers of drones provided in an embodiment of the present application;

[0064] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0065] Fig.16 A schematic diagram of the structure of a drone control system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] See below Figure 1 , Figure 1 A flowchart of a drone control method provided in an embodiment of the present application.

[0068] Specific steps may include:

[0069] S101: Establish a problem model, and construct a constrained Markov decision problem based on the problem model;

[0070] Among them, this embodiment can be applied to electronic devices with intelligent agent training functions. This embodiment can establish a problem model based on relevant information of drones, ground nodes and ground data centers. The problem model is used to represent that multiple drones collect data from multiple ground nodes and unload the collected data to a ground data center.

[0071] Ground Nodes refer to devices or sensor nodes located on the ground. Ground Nodes can collect various data and communicate with drones. Ground Data Center refers to facilities located on the ground that are used to process, store and manage data collected by drones or other sensors.

[0072] Based on the above problem model, a constrained Markov decision problem can be constructed based on the problem model. Constructing a constrained Markov decision problem based on the problem model is the process of abstracting the actual problem into a mathematical model to solve the problem of how the intelligent agent can make the best decision under specific constraints. The constrained Markov decision problem expands the traditional Markov decision process and adds constraints such as energy consumption and time limits. By constructing a constrained Markov decision problem, strategies can be designed for intelligent agents such as drones to maximize cumulative rewards while complying with constraints, thereby achieving efficient data collection and unloading.

[0073] S102: Modeling the constrained Markov decision problem as a target model;

[0074] The target model is a distributed locally observable Markov decision process model, which is also called the Dec-POMDP (Decentralized Partially Observable MarkovDecision Process) model. Modeling the constrained Markov decision problem as a Dec-POMDP model is a method to take local information and uncertainty into account in a multi-agent system. The Dec-POMDP model allows each agent to make decisions based on local observations while taking into account the existence and activities of other agents to achieve distributed control and collaborative work. The Dec-POMDP model helps to deal with uncertainty in multi-agent systems and achieve efficient distributed decision-making.

[0075] S103: obtaining a target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm;

[0076] Among them, the target algorithm is an algorithm designed based on the target model, which is used to solve the intelligent agent strategy that adapts to the change in the number of entities; the first network layer is a network layer based on the fully connected (FC) layer and has permutation invariance, and the second network layer is a network layer based on the fully connected layer and has permutation equivariance.

[0077] The first network layer is also called the HPI layer (Hypernetwork-based Permutation Invariant, a hyper permutation invariant layer that adapts to variable-length input), and the second network layer is also called the HPE layer (Hypernetwork-based Permutation Equivariant, a hyper permutation equivariant layer that adapts to variable-length output). This step introduces the HPI layer and the HPE layer into the QMIX algorithm (a hybrid-based multi-agent reinforcement learning algorithm) to obtain the target algorithm (i.e., the HyperQMIX algorithm, an enhanced multi-agent reinforcement learning algorithm). Before this step, there may be an operation of pre-building the HPI layer and the HPE layer.

[0078] Specifically, this embodiment can introduce an automatic weight generation mechanism on the basis of the FC layer to obtain the HPI layer. The HPI layer realizes the dynamic adjustment of the weight matrix to adapt to the change in the number of input elements and satisfies the permutation invariance, that is, if the order of the input elements is changed, the function output will not change. This application names the neural network module after the above-mentioned improvement of the FC layer network structure as the HPI layer.

[0079] This embodiment can also introduce an automatic weight generation mechanism based on the FC layer, establish a one-to-one correspondence between input and output elements, and obtain an HPE layer. The above mechanism ensures that the weight matrix can be adjusted accordingly with the change in the number of input and output elements, and satisfies the permutation equivariance, that is, if the order of input elements is changed, the order of output elements will be changed in the same way. This application names the neural network module after this improvement of the FC layer network structure as the HPE layer.

[0080] This embodiment introduces the HPI layer and the HPE layer into the multi-agent reinforcement learning algorithm QMIX to obtain the target algorithm HyperQMIX. The above-mentioned HyperQMIX algorithm is an algorithm designed based on the Dec-POMDP model, which is used to solve the agent strategy that adapts to the change in the number of entities. Specifically, this embodiment can introduce the HPI layer and the HPE layer into the QMIX algorithm on the basis of the Dec-POMDP model to obtain the HyperOMIX algorithm. The HyperQMIX algorithm is a new algorithm that can solve the agent strategy that adapts to the change in the number of entities after improving the QMIX algorithm. The improvement point is to introduce the HPI layer and the HPE layer into the QMIX algorithm.

[0081] The HyperQMIX algorithm combines the characteristics of the HPI layer, HPE layer and QMIX algorithm, and can solve the agent strategy that adapts to the changes in the number of entities, thereby achieving efficient data collection and offloading in a multi-agent environment.

[0082] S104: Using the target algorithm to train the intelligent agents of the plurality of drones, and using the trained intelligent agents to perform trajectory planning and / or communication scheduling for the drones.

[0083] Among them, this embodiment uses the target algorithm HyperQMIX to train multiple UAV agents, so that the UAV can make decisions that adapt to dynamic characteristics and distributed control requirements. Through training, the agent can learn effective trajectory planning and communication scheduling strategies, thereby achieving efficient data collection and offloading in a multi-agent environment.

[0084] In this embodiment, a problem model is first established to represent that multiple drones collect data from multiple ground nodes and unload the collected data to a ground data center, and a constrained Markov decision problem is constructed based on the problem model, and the constrained Markov decision problem is modeled as a Dec-POMDP model. The Dec-POMDP model allows each drone agent to make decisions based on local information, while taking into account the existence and activities of other agents, which helps to deal with uncertainties in multi-agent systems and realize distributed control. In this embodiment, the HPI layer and the HPE layer are introduced into the QMIX algorithm to obtain the HyperQMIX algorithm. Since the HPI layer is a network layer based on the FC layer and has permutation invariance, and the HPE layer is a network layer based on the FC layer and has permutation equivariance, the HyperQMIX algorithm obtained in this embodiment can solve the agent strategy that adapts to the change in the number of entities. In this embodiment, the HyperQMIX algorithm is used to train the agents of multiple drones so that the trained agents can make decisions based on local information to adapt to dynamic characteristics and distributed control requirements. This embodiment uses the Dec-POMDP model and the HyperQMIX algorithm to enable the trained intelligent agent to adapt to various dynamic environmental factors and distributed control requirements during the process of controlling the drone.

[0085] As for Figure 1 For further description of the corresponding embodiment, the constrained Markov decision problem can be constructed in the following way:

[0086] Step A1: determining the environmental state characteristics according to the problem model; wherein the environmental state characteristics include the current location information of the UAV, the location information of the current ground node, the location information of the current ground data center, the amount of data to be uploaded of the current ground node, and the total amount of stored data of the UAV;

[0087] Step A2: determining a probabilistic line-of-sight model according to the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path;

[0088] Step A3: determining the trajectory variable and the binary scheduling variable of the UAV according to the problem model; wherein the binary scheduling variable is used to describe the communication status between the ground node and the ground data center;

[0089] Step A4: determining the average information age function of the ground node according to the problem model;

[0090] The average information age function refers to: a function that calculates the average information age of ground nodes. Information age is an important indicator for measuring the freshness of information. Maintaining a small information age is crucial for the timeliness of real-time update systems. The information age of each data packet refers to the length of time from the moment it is generated to the current moment. Each data packet can only provide information for subsequent decisions after it is successfully received by the data center. Therefore, the information age of each ground node is defined as the information age of the latest data packet received by the data center from the ground node.

[0091] Step A5: constructing the constrained Markov decision problem based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

[0092] Based on the factors of environmental state characteristics, probabilistic line-of-sight model, UAV trajectory variables, binary scheduling variables and average information age function, this embodiment can comprehensively consider the mobility characteristics, communication constraints and information generation time requirements of the UAV, so as to design a constrained Markov decision problem for the UAV under the constraints to achieve efficient data collection and offloading.

[0093] As for Figure 1 For further introduction to the corresponding embodiment, the present embodiment can construct the constrained Markov decision problem as a target model Dec-POMDP in the form of entity decomposition. The specific process is as follows:

[0094] Step B1: performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity;

[0095] The feature vector of the drone entity includes environmental information related to the drone, and the feature vector of the ground node entity includes environmental information related to the ground node;

[0096] Step B2: defining key elements using the feature vector of the drone entity and the feature vector of the ground node entity;

[0097] The key elements include state, observation, action and reward;

[0098] Step B3: Building the target model Dec-POMDP based on the key elements.

[0099] The constrained Markov decision problem is entity decomposed to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity. The key elements such as state, observation, action and reward are defined using the feature vector of the drone entity and the feature vector of the ground node entity. Based on these key elements, the present embodiment builds a Dec-POMDP model, which allows each drone agent to make decisions based on local information while taking into account the existence and activities of other agents to achieve distributed control and collaborative work.

[0100] As for Figure 1 For further description of the corresponding embodiment, before obtaining the HyperQMIX algorithm by introducing the HPI layer and the HPE layer into the QMIX algorithm, the HPI layer and the HPE layer may also be formed in the following manner:

[0101] This embodiment can introduce a first hypernetwork HN shared by all input elements in a single FC layer. in (⋅) forms the HPI layer; wherein the first hypernetwork takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features of the single input element and sum them to obtain the network output.

[0102] This embodiment can introduce a second hypernetwork HN shared by all output elements in a single FC layer. out (⋅) and an HPI layer for processing the overall input of the output elements, together forming an HPE layer; wherein the second super network takes the input element corresponding to a single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply the output of the HPI layer to obtain the network output.

[0103] As for Figure 1 For further description of the corresponding embodiment, the HPI layer and the HPE layer can be introduced into the QMIX algorithm to obtain the HyperQMIX algorithm in the following manner:

[0104] Step C1: Build the input layer based on one FC layer and two HPI layers;

[0105] Among them, the input of the input layer is all the features in the local observation of the drone, and the output of the input layer is a single embedded feature merged by a summation operation. This step builds the input layer based on an FC layer and two HPI layers with permutation invariance. The input layer receives all the features of the local observation of the drone, including but not limited to the current position, the total amount of data to be unloaded in the buffer, the scheduling relationship between the previous time slot and the data center, etc. Among these features, the features of the drone n itself are processed by an FC layer, and the feature sets of neighboring drones and ground nodes within the perception range are processed by two HPI layers respectively. The HPI layer can process inputs of different arrangements and produce the same output, which enables the input layer to effectively process entity sets. The outputs of an FC layer and two HPI layers are merged into a single embedded feature through a summation operation. This feature represents the key information of the local observation of the drone and provides input for the subsequent decision-making process.

[0106] Step C2: Build the middle layer based on the GRU structure in the QMIX algorithm;

[0107] The input of the intermediate layer is the single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot.

[0108] Specifically, this step builds an intermediate layer based on the GRU (Gated Recurrent Unit) structure in the QMIX algorithm. The intermediate layer receives a single embedded feature from the input layer and the hidden state of the previous time slot as input. The GRU structure is able to process sequence data and retain long-term dependency information. In this process, a single embedded feature is input into the GRU structure together with the hidden state of the previous time slot, and the GRU structure updates the hidden state through a gating mechanism to reflect the state change of the current time slot. After processing, the intermediate layer outputs the hidden state of the current time slot, providing updated state information for the subsequent decision-making process.

[0109] Step C3: Building an output layer based on an FC layer and an HPE layer;

[0110] The input of the output layer is the output of the intermediate layer, and the output of the output layer is the Q value (i.e., reward value) of all actions in the action space and the optimal action corresponding to the maximum Q value (i.e., maximum reward value);

[0111] Specifically, this step builds an output layer based on an FC layer and an HPE layer with permutation equivariance. The FC layer in the output layer receives the current time slot hidden layer state from the middle layer as input, and the HPE layer receives the features of all neighboring ground nodes and the current time slot hidden layer state from the middle layer as input. The HPE layer can process inputs of different arrangements and produce outputs related to the arrangement, which enables the output layer to generate Q values ​​for all actions in the action space. Finally, the output layer outputs these Q values ​​to provide a basis for the decision-making process and guide the UAV agent to select the optimal action.

[0112] Step C4: constructing a hyperlocal Q network including the input layer, the intermediate layer and the output layer;

[0113] The hyperlocal Q network (also known as the hyperlocal value network) is used to fit the local Q function (also known as the local value function) of the UAV agent; the input layer of the hyperlocal Q network includes an FC layer and two HPI layers with permutation invariance for processing entity sets. The intermediate layer of the hyperlocal Q network is implemented based on the GRU structure for processing sequence data. The output layer of the hyperlocal Q network includes an FC layer and an HPE layer with permutation equivariance for generating the Q value of the action. The goal of the hyperlocal Q network is to learn a local Q function that can predict the value of each possible action based on the local observation information of the agent, thereby guiding the agent to make the best decision. The hyperlocal Q network is a neural network architecture for multi-agent reinforcement learning. The hyperlocal Q network solves decision-making problems in complex environments by processing local information and distributed decision-making. In this way, each agent can make decisions independently while taking into account the existence and activities of other agents to achieve efficient data collection and unloading. The hyperlocal Q network learns the local Q function of the UAV agent in an end-to-end manner, thereby predicting the value of each possible action in a given state and guiding the agent to make the best decision.

[0114] This embodiment uses the HyperQMIX algorithm to introduce the HPI layer and the HPE layer into the local Q network and the global hybrid network respectively, thereby forming an improved super-local Q network and a super-hybrid network.

[0115] Step C5: constructing a super-hybrid network for combining the local Q-values ​​(i.e., local reward values) of all the agents into a global Q-value (i.e., global reward value);

[0116] Specifically, this embodiment can use an FC input layer (i.e., fully connected input layer) and an FC output layer (i.e., fully connected output layer) to build the super hybrid network; wherein the super hybrid network is used to combine the local Q values ​​of all the agents into a global Q value; wherein the input of the FC input layer is the local Q values ​​of all the agents, and the output of the FC input layer is a single embedded feature; the input of the FC output layer is a single embedded feature output by the input layer, and the output of the FC output layer is a global Q value.

[0117] The super-hybrid network consists of an FC input layer and an FC output layer, which can be connected through a weight matrix and an activation function. The weight matrix will be adjusted according to the training process to optimize the performance of the entire network. During the training process, the network adjusts its weights through the back-propagation algorithm to minimize the difference between the predicted global Q value and the actual global Q value.

[0118] The hyperhybrid network is a neural network architecture for implementing multi-agent reinforcement learning, which is used to merge the local Q-values ​​of all agents into a global Q-value. The hyperhybrid network consists of an FC input layer and an FC output layer. The FC input layer receives the local Q-values ​​of all agents as input and converts the input into a single embedded feature vector through a nonlinear transformation. The FC output layer receives this embedded feature vector as input and outputs a scalar value, the global Q-value. In this way, the hyperhybrid network can effectively combine the decision information of multiple agents and provide a solution to the distributed decision-making problem in multi-agent systems.

[0119] Step C6: The hyperlocal Q network of each agent and the hypermixed network are combined to obtain the HyperQMIX algorithm.

[0120] This step combines the hyperlocal Q-network of each agent with the hypermixed network to form the HyperQMIX algorithm. In the HyperQMIX algorithm, each agent has a hyperlocal Q-network that predicts the Q-value of the action based on its local observation information. These local Q-values ​​are then input into the hypermixed network, which merges all the local Q-values ​​into a global Q-value. In this way, the HyperQMIX algorithm can effectively handle local information and distributed decision-making problems in multi-agent systems.

[0121] As a further introduction to the above embodiment, the weight matrix and bias vector of the FC input layer and the weight matrix and bias vector of the FC output layer may also be determined. The specific process is as follows:

[0122] Step D1: concatenate the embedded features and the drone features into an HPE layer and generate the weight matrix of the FC input layer through the absolute value activation function;

[0123] Among them, the embedded feature is an embedded feature obtained by summing the first processing result and the second processing result, the first processing result is the result obtained by using the HPI layer to process the feature vector of the drone entity, and the second processing result is the result obtained by using the HPI layer to process the feature vector of the ground node entity; the drone feature is the feature of the drone entity.

[0124] Step D2: Processing entity features in the environment state using the HPI module, and generating a weight matrix of the FC output layer through an absolute value activation function;

[0125] Step D3: concatenate the embedded features and the drone features and input them into an HPE layer and generate the bias vector of the FC input layer through the linear rectification function ReLU (Rectified Linear Unit);

[0126] Step D4: Use the HPI module to process entity features in the environment state, and generate the bias vector of the FC output layer through the linear rectification function ReLU.

[0127] As for Figure 1 For further introduction to the corresponding embodiment, the intelligent agents of the plurality of drones can be trained in the following manner: the intelligent agents of the plurality of drones are centrally trained offline using the HyperQMIX algorithm so that the intelligent agents of the drones update the parameters of the hyper-local Q network using experience samples; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

[0128] After using the HyperQMIX algorithm to perform centralized offline training on the intelligent agents of multiple drones, this embodiment can also deploy the trained intelligent agents to the corresponding drones; wherein the trained intelligent agents include a hyper-local Q network with updated parameters; and perform strategy migration operations on the intelligent agents in the drones by means of model overloading.

[0129] The process described in the above embodiment is explained below through an embodiment in actual application.

[0130] In the current solutions to the joint optimization problem of drone trajectories and communications, firstly, in the drone-to-ground communication system, environmental factors such as the deployment location and number of communication nodes have time-varying characteristics, which makes it difficult for the offline designed drone trajectory and communication strategies to meet the performance requirements in a dynamic environment. Secondly, traditional optimization methods usually have a high computational complexity, which brings great difficulties to the real-time update of optimization variables in a dynamic environment. Finally, most of these methods rely on global environmental information for centralized solution, ignoring the distributed control requirements in multi-drone scenarios. To this end, the embodiments of the present application disclose a scalable multi-agent reinforcement learning method, device, equipment and medium, which can design a new drone trajectory and communication joint optimization solution for the drone trajectory and communication joint optimization problem, adapting to the dynamic characteristics and distributed control requirements of the actual system.

[0131] See also Figure 2 , Figure 2 A flowchart of a scalable multi-agent reinforcement learning method provided in an embodiment of the present application specifically includes the following steps:

[0132] Step S201: Establish a problem model for collecting data from a number of ground nodes for multiple UAVs and unloading the data to a ground data center, and construct a constrained Markov decision problem based on the problem model.

[0133] In this embodiment, the problem model of multiple drones collecting data from a number of ground nodes and unloading the data to a ground data center is specifically as follows: Consider a scenario where multiple drones collect data from a number of ground nodes and unload the data to a ground data center, that is, N drones with limited data storage collect data from a number of ground nodes. Collect data from multiple ground node devices and offload all data to a single ground data center. Represents the set of all ground node devices. The location of each ground node device can be used Indicates that , the location of the ground data center can be used Indicates that the drone is at a fixed altitude Fly at a constant speed V. Divide time into multiple time slots, the length of each time slot is . Define the collection To describe different time slots, represents the total number of time slots required for the drone to collect all the data in a mission. In each time slot t, the position of the drone is , and the two drones must maintain a minimum distance between them In the probabilistic line-of-sight model, the straight-line distance from ground node m to drone n at time slot t is , the elevation angle is According to these variables, the line-of-sight link probability can be expressed as , the probability of non-line-of-sight link can be expressed as , a and b are parameters related to the specific environment. The channel gain from UAV n to ground node m is expressed as , represents the channel gain in the line-of-sight case, Represents the channel gain in non-line-of-sight conditions; similarly, a similar method can be used to calculate the channel gain from drone n to the data center In order to describe the scheduling of UAV n to ground node m , that is, each UAV can only perform one operation in each time slot (schedule a ground node to collect data or unload data to the data center), and each ground node can only be scheduled by one UAV at most, so the binary scheduling variable Should meet , In order to determine the communication range for data collection and data offloading respectively, it is assumed that all drones communicate in the same frequency band, a directional antenna model is adopted to limit the energy of transmission and reception to a specific angle, and a binary indicator variable is introduced To represent the coverage between the drone and the ground node or data center, Indicates that ground node m is within the coverage area of ​​drone n, otherwise The transmission power of the ground node is set to , the communication bandwidth for transmitting data to the UAV is In order to reduce the energy consumption of ground nodes, it is stipulated that ground nodes only generate new perception data when they are dispatched by UAVs, and immediately upload the data to UAVs, and the amount of data generated by each ground node each time is fixed to When the buffer of drone n reaches the upper limit of capacity, the drone will no longer collect new data until there is enough space in the buffer, so Constraints must be met , if represents a conditional judgment function. To ensure the continuity and integrity of data transmission, if the ground node fails to completely upload its data to the drone within a time slot, the scheduling relationship between the ground node and the drone must remain unchanged until all data is completely uploaded. Similarly, if the drone fails to completely unload its data to the data center within a time slot, the scheduling relationship between the drone and the data center must also remain unchanged until all data is completely unloaded. Therefore, Constraints must also be met ; The goal of this embodiment is to minimize the average information age of the system by optimizing the trajectory of the drone and the communication scheduling to the ground nodes. That is, the problem model in this embodiment can provide a scalable multi-agent reinforcement learning method with the goal of minimizing the average information age of the system under the above constraints, so that the strategies learned by the agents can adapt to changes in the number of entities such as drones and ground nodes.

[0134] The process of constructing a constrained Markov decision problem based on the problem model includes the following steps E1 to E5:

[0135] Step E1: constructing an environmental state feature including the current location information of the drone, the location information of the current ground node, the location information of the current data center, the amount of data to be uploaded by the current ground node, and the total amount of stored data of the drone; wherein the environmental state feature The relationship is: ; In the formula, n represents the number of the current drone, m represents the number of the current node, and t represents the number of the current time slot. and Represents the set of all UAV features and ground node features, UAV features ,in represents the current time slot position information of the drone, represent the horizontal coordinate, vertical coordinate and height of the UAV in the spatial coordinate system respectively, and , is a constant, represents the total amount of data to be unloaded in the drone buffer, is the upper limit of the drone buffer zone capacity, represents the amount of data unloaded by UAV n at time slot t-1, represents the amount of data collected by drone n at time slot t-1, represents the scheduling relationship between the drone and the data center in the previous time slot, Indicates whether the drone has scheduled any ground nodes in the previous time slot, and the characteristics of the ground nodes , represents the position of the ground node, where , Indicates the amount of data to be uploaded by the ground node, a binary indicator Indicates whether the ground node m generates a new data packet, fixed value Indicates the amount of data generated by the ground node each time, represents the amount of data uploaded by ground node m in time slot t-1, represents the information age of ground node m, represents the time when the data packet from ground node m was last received by the data center. Indicates whether node m was scheduled by any UAV in the previous time slot.

[0136] Step E2: construct a probabilistic line-of-sight model that takes into account both line-of-sight propagation paths and non-line-of-sight propagation paths; wherein the relational expression of the probabilistic line-of-sight model is: ; In the formula, and They represent the probability of line-of-sight and non-line-of-sight links appearing, respectively, where a and b are parameters related to the specific environment. represents the elevation angle from ground node m to drone n, represents the straight-line distance from ground node m to drone n, Represents the norm.

[0137] Step E3: Construct a binary scheduling variable including the trajectory variable of the drone and the communication status of the ground node and the data center; wherein the drone trajectory variable and binary scheduling variables The relationship is: , ; In the formula, represents the set of all UAVs, ground nodes, and time slots. Each data collection task consists of T time slots.

[0138] Step E4: Design the average information age function of the ground node; wherein the average information age function The relationship is: ; In the formula, represents the average information age of ground node m during the entire data collection duration; Represents the information age of the ground node.

[0139] Step E5: construct a constrained Markov decision problem based on the environmental state characteristics, the probabilistic sight distance model, the trajectory variables and the binary scheduling variables, and the average information age function; wherein the goal of the constrained Markov decision problem is to solve the problem of minimizing the average information age of the system .

[0140] Step S202: The problem is constructed as a distributed locally observable Markov decision process model Dec-POMDP by means of entity decomposition.

[0141] In this embodiment, the process of constructing the problem into a distributed local observable Markov decision process model Dec-POMDP by entity decomposition specifically includes the following steps F1 to F3:

[0142] Step F1: Establish two time-varying sets to represent neighboring drones and ground nodes within the drone’s sensing range , ;

[0143] Step F2: The UAV’s action space is divided into two parts. The first part is all the scheduling actions related to the ground nodes. The second part is the movement actions and data unloading actions that are not related to specific entities. ; Wherein, the relationship formula of the scheduling action is: , the relationship between the moving action and the data offloading action is: ; In the formula, represents the action of selecting ground node m for data collection, is a finite set that discretizes the continuous direction space. Indicates the direction the drone is heading The action of flying, Indicates the action of the drone unloading data to the data center.

[0144] Step F3: Define reward function based on Dec-POMDP model ; Wherein, the reward function The relationship is: ; In the formula, represents the number of collisions caused by all drones in time slot t, and is the scaling factor.

[0145] Step S203: design two types of neural network modules, HPI layer and HPE layer, and introduce them into the QMIX algorithm to form the HyperQMIX algorithm.

[0146] In this embodiment, the FC layer network is improved, and an HPI layer that adapts to variable-length input and satisfies permutation invariance and an HPE layer that adapts to variable-length output and satisfies permutation equivariance are designed, so that the improved hypernetwork can effectively handle changes in state dimensions and establish a one-to-one correspondence between drone features and input layer weight matrices, ensuring that the network model is not affected by the number of entities. This step can design a HyperQMIX algorithm that introduces two types of neural network modules, the HPI layer and the HPE layer, into the QMIX algorithm for the constrained Markov decision problem.

[0147] This embodiment establishes a problem model for collecting data from a number of ground nodes for multiple drones and unloading the data to a ground data center, and constructs a constrained Markov decision problem based on the problem model; for the constrained Markov decision problem, the problem is constructed as a distributed local observable Markov decision process model Dec-POMDP by entity decomposition; and the two types of neural network modules, HPI layer and HPE layer, are designed to be introduced into the QMIX algorithm to form the HyperQMIX algorithm. It can be seen that the present application first establishes a problem model for collecting data from a number of ground nodes for multiple drones and unloading the data to a ground data center, and constructs a constrained Markov decision problem based on the problem model, and models the problem as Dec-POMDP, and then designs and introduces two types of neural network modules, HPI layer and HPE layer, into the QMIX algorithm, and proposes a scalable HyperQMIX algorithm, which realizes the permutation invariance and permutation equivariance in the super-local Q network and the super-hybrid network, so that the network structure is independent of the number of entities, and at the same time improves the sample efficiency of the algorithm in fixed-scale tasks.

[0148] See also Figure 3 , Figure 3 This is a flowchart of another scalable multi-agent reinforcement learning method provided in an embodiment of the present application. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution, and specifically includes the following steps:

[0149] Step S301: Establish a problem model for collecting data from a number of ground nodes for multiple drones and unloading the data to a ground data center, and construct a constrained Markov decision problem based on the problem model.

[0150] Step S302: construct the problem into a distributed locally observable Markov decision process model Dec-POMDP by entity decomposition.

[0151] The constrained Markov decision problem entity is decomposed into the UAV as an intelligent entity and the ground node as a non-intelligent entity. The feature vector of the UAV entity is and the feature vector of the ground node Contains all environmental information related to it;

[0152] Use the above entity characteristics to define the key elements in Dec-POMDP: state , Observation ,action and rewards , a distributed locally observable Markov decision process model Dec-POMDP was built;

[0153] Among them, the relationship between the drone entity feature vector is: , the ground node entity feature vector relationship is: , the relationship between the system state is: , the comprehensive observation The relationship is: , the relationship between the scheduling action is: , the relationship between the moving action and the data offloading action is: ; The reward function The relationship is: ;

[0154] In the formula, Indicates the location of the data center, Represents the set of all drone features, Represents the set of all ground node features, represents the feature set of drone n itself, represents the feature set of neighboring drones, represents the feature set of ground nodes, represents the action of selecting ground node m for data collection, is a finite set that discretizes the continuous direction space. Indicates the direction the drone is heading The action of flying, represents the action of the drone unloading data to the data center, represents the number of collisions caused by all drones in time slot t, and is the scaling factor.

[0155] Step S303: Building a scalable network based on the HPI layer and the HPE layer.

[0156] In this embodiment, in order to make the dimension of the weight matrix change synchronously with the number of input elements, a hypernetwork consisting of a single FC layer is built at the HPI layer. Generates each input element The corresponding weight matrix , the weight matrix extracts features from each input element, and then sums them up to get the final network output ; Build a super network based on the FC layer at the HPE layer Generate a weight matrix corresponding to each output element , The input element corresponding to each output element As input, in order to adapt the HPE layer to the input of dynamic dimensions, an HPI layer is introduced Processing the overall input of the network , then each output element , a one-to-one correspondence between input elements and output elements is established. Represents the total number of input elements, and j represents the jth input element.

[0157] Step S304: Build a drone agent responsible for fitting Local function The super local network; among them, represents the local observation of the UAV, Represents the optimal action selected by the drone.

[0158] In this embodiment, the input layer uses one FC layer and two HPI layers to perceive all the features of the local observation of the drone as input, and merges them into a single embedded feature through summation operation. As output, this process is represented as: ; , and They represent the embedding functions implemented by the above network structure respectively; the middle layer uses the GRU structure in the QMIX algorithm to further process the single embedding features output by the above input layer, and the GRU stores the hidden state of each time slot t , which is updated based on the embedded features and the hidden state of the previous time slot: , Represents the function implemented by the GRU layer; the output layer uses an FC layer and an HPE layer to take the hidden state output by the above intermediate layer as input and output the Q value of all entity-independent actions: , ,in, represents the output function implemented by the FC layer, represents the output function implemented by the HPE layer; combined with all Q values, that is, Determine the best action and its corresponding value.

[0159] Step S305: Build a super hybrid network responsible for combining the local Q values ​​of all agents into a global Q value.

[0160] In this embodiment, the local Q values ​​of all agents are used as input to an FC input layer to generate an embedded feature ,in, and Represent the weight matrix and bias vector of the input layer respectively, represents the input activation function, Represents the vector composed of all local Q values; then the above input layer output embedding features are used as the input of a FC output layer to generate a global Q value: ,in, and is the weight matrix and bias vector of the output layer, Represents the output activation function. The input of the FC input layer is the local Q value of all agents, and its output is a single embedded feature. The input of the FC output layer is the single embedded feature output by the above input layer , whose output is the global Q value.

[0161] The weight matrix and bias vector of the FC input layer and the FC output layer are generated as follows:

[0162] The improved super network that generates the input layer weight matrix in the super hybrid network is denoted as , Two HPI layers are used to process the features of drones and ground nodes respectively, and then the results are combined into one embedded feature through summation. , the embedded features and drone features are concatenated and input into an HPE layer, and the weight matrix of each drone is generated through the absolute value activation function ,in, and represents the output function implemented by the HPI layer, represents the output function implemented by the HPE layer, is the absolute value activation function; the improved super network that produces the output layer weight matrix in the super hybrid network is recorded as The HPI module is used to process the entity features in the environment state, and the bias vector of each is generated through the absolute value activation function. ,in, and Denotes the embedding function implemented by the HPI layer. The bias vectors of the input layer and the output layer in the super-hybrid network are generated in a similar manner, but are not required to be non-negative, so the ReLU function is used instead of the absolute value function in the corresponding improved super network. This embodiment can train all the network parameters built above end-to-end.

[0163] Step S306: Use the HyperQMIX algorithm to train and update the parameters of the above-built network.

[0164] In this embodiment, the Bellman equation corresponding to the global Q value is used to construct the loss function ,in, represents the global target Q value, θ represents the set of all learnable network parameters in HyperQMIX, and θ −represents the target network parameters, which are regularly copied from θ. In this embodiment, the target network parameters can also be regularly updated by copying from the learnable network parameter set. represents the coefficient of the desired operation on the data set D, represents the global Q value of the network output with parameter θ, represents the action that the drone chooses to perform at time t, represents the discount rate, represents the reward value at time t, Indicates status.

[0165] In the process of building a deep reinforcement learning framework based on the HyperQMIX algorithm for the constrained Markov decision problem, the following operations can also be performed:

[0166] Centralized offline training, where the UAV agent collects experience samples by interacting with the environment and uses these experience samples to update network parameters;

[0167] Distributed online execution, after the above training is completed, the trained hyper-local Q network is deployed on each drone;

[0168] Inter-task strategy transfer, through model overloading, transfers the agent collaboration mechanism learned in the source task and other semantic information that helps new task decision-making to the new target task.

[0169] This embodiment aims at the joint optimization problem of the trajectory of the UAV and the communication scheduling of the ground nodes, and designs and introduces the HPI layer that adapts to variable-length input and satisfies the permutation invariance and the HPE layer that adapts to variable-length output and satisfies the permutation equivariance for the local network and hybrid network of the QMIX algorithm in the deep reinforcement learning technology. The design of the HPI layer and HPE layer modules significantly enhances the representation ability of the model, realizes the sharing of the parameters of the hypernetwork between different input elements, ensures that the number of network parameters will not change due to the change of the input and output dimensions, and at the same time, the parallel processing of its calculation process also avoids the exponential growth of the calculation complexity with the change of the number of elements; the introduction of the HPI layer and HPE layer modules makes the newly formed HyperQMIX algorithm meet the scalability, realizes the knowledge transfer of the intelligent agent between tasks of different scales, that is, the network structure is not affected by the change of the number of neighboring UAVs and ground nodes within the perception range of the UAV, establishes the permutation invariance and permutation equivariance relationship between observation and action, and improves the training stability and sample efficiency of the algorithm by making full use of the prior environmental information in the network structure.

[0170] In order to verify the effectiveness of this embodiment, the effect of this application will be further explained in combination with simulation experiments.

[0171] In addition to the HyperQMIX method proposed in this embodiment, five comparison methods are also used in the simulation experiment: 1. Greedy algorithm based on information age; 2. QMIX algorithm; 3. DeepSets-QMIX algorithm (deep set-based hybrid multi-agent reinforcement learning algorithm); 4. Attention-QMIX algorithm (attention mechanism-based hybrid multi-agent reinforcement learning algorithm); 5. GNN-QMIX algorithm (graph neural network-based hybrid multi-agent reinforcement learning algorithm). All experimental codes related to deep reinforcement learning are implemented based on Pytorch, an open source deep learning framework for machine learning and deep learning.

[0172] In the simulation experiment, the wireless channel model between the UAV and the ground node is a probabilistic line-of-sight model, so the channel power gain at time slot t is It can be modeled as:

[0173] ;

[0174] in, is the carrier center frequency, α represents the path loss exponent, is the additional path loss coefficient in each case, represents the straight-line distance from ground node m to drone n, and c represents the speed of light. Therefore, the final channel gain is calculated as the expected value of line-of-sight and non-line-of-sight conditions. It can be modeled as:

[0175] ;

[0176] Among them, LoS represents the line-of-sight propagation path, and NLoS represents the non-line-of-sight propagation path. Assuming that all drones communicate in the same frequency band, the transmission power of the ground node is set to , the communication bandwidth for transmitting data to the UAV is , then the signal-to-interference-to-noise ratio of the communication link between ground node m and UAV n in time slot t can be expressed as:

[0177] ;

[0178] In the above formula and Represents different ground nodes, and Indicates different drones, and Respectively represent the coverage between the UAV and different ground nodes, and They respectively represent the dispatching of different UAVs to different ground nodes. and They represent the channel gains from the UAV to different ground nodes respectively.

[0179] in, represents the power spectral density of additive white Gaussian noise. Therefore, the maximum amount of data collected by drone n from ground node m in time slot t is It can be calculated as:

[0180] ;

[0181] Similarly, the transmission power of the drone is , the communication bandwidth for offloading data to the data center is , the signal-to-interference-to-noise ratio between drone n and the data center can be obtained and the maximum amount of data that drone n offloads to the data center Therefore, in time slot t, the total amount of data uploaded by ground node m is , the total amount of data collected by drone n And the total amount of data unloaded by drone n They are:

[0182] ;

[0183] ;

[0184] ;

[0185] The information age of the ground node is , if the data center receives a new data packet from ground node m, will be reset, otherwise will increase at a constant rate. The average information age of ground node m during the entire data collection duration is It can be calculated by the following formula:

[0186] ;

[0187] Therefore, the average information of all ground nodes Age:

[0188] ;

[0189] It is worth noting that all the models assumed here are only used for simulation experiments. In the specific implementation of this embodiment, the UAV agent does not need any prior knowledge about the system model.

[0190] One segment is equivalent to the completion of a mission. In each segment, the initial positions of the UAV and the ground node are randomly generated in an area of ​​2000m×2000m. The relevant parameters of the UAV are set as follows: , , The communication related parameters are set as follows: up =B off =1MHz, N0=-169dBm / Hz. For data collection, this application assumes , , , communication range for drone data collection , communication range for data offloading , the drone’s perception range of neighboring drones / sensors The channel gain related parameters are set to: a=9.61, b=0.15, , , During training, the exploration probability ϵ is calculated in the first 10 6 The discount factor γ is set to 0.99. The Adam optimizer is used to update the network parameters, and the learning rate is set to 0.0005. The experience replay pool stores samples from the last 5000 training rounds, and each network update uses samples from 32 rounds. The overall training process lasts 105 rounds, each round contains 150 time steps. Unless otherwise specified, the number of drones is 4 and the number of ground nodes is 64.

[0191] See also Figure 4 , Figure 5 and Figure 6 , Figure 4 A schematic diagram of convergence curves of average information age of a HyperQMIX algorithm and a first group of comparison algorithms during training provided in an embodiment of the present application, Figure 5 A schematic diagram of convergence curves of average information age of a HyperQMIX algorithm and a second group of comparison algorithms during training provided in an embodiment of the present application, Figure 6 A schematic diagram of the convergence curve of the average information age of a HyperQMIX algorithm and a third group of comparison algorithms during the training process provided in an embodiment of the present application. Figure 4 , Figure 5 and Figure 6 The horizontal axis represents the number of training rounds, and the vertical axis represents the average information age (s). The figure shows the positions corresponding to 1600 training rounds. Figure 4The algorithm in this paper is the HyperQMIX algorithm, the first algorithm is the greedy algorithm Greedy based on information age, and the second algorithm is the DeepSets-QMIX algorithm. Figure 5 The first algorithm is the HyperQMIX algorithm, the third algorithm is the QMIX algorithm, and the fourth algorithm is the GNN-QMIX algorithm. Figure 6 The first algorithm is the HyperQMIX algorithm, and the fifth algorithm is the Attention-QMIX algorithm. The performance of all multi-agent reinforcement learning algorithms is better than that of the greedy algorithm. The average information age reached by the greedy algorithm after convergence is about 650s, which is 56.6% higher than the best performing HyperQMIX algorithm, and even 8.8% higher than the worst performing QMIX algorithm. The final average information age of HyperQMIX is 415s, which is 11.3% lower than the second best performing GNN-QMIX and 30.7% lower than the worst performing QMIX. These results verify the significant advantage of HyperQMIX in reducing the average information age. Secondly, HyperQMIX also performs well in terms of convergence speed and sample efficiency, reducing computing resources and the need to generate experience samples. By Figures 4 to 6 It can be seen that HyperQMIX surpassed the convergence performance of all benchmark algorithms in only about 1,600 training rounds. In contrast, GNN-QMIX requires about 105 rounds to reach a similar performance level. In the end, HyperQMIX converged at about 4×104 training rounds, while other algorithms required more than 8×104 training rounds to approach convergence. The performance advantage achieved by HyperQMIX is mainly due to the effective use of permutation invariance and permutation equivariance in multi-agent systems, as well as its use of hypernetworks to dynamically generate embedding layer parameters, which significantly improves its representation capabilities and achieves better performance.

[0192] See also Figure 7 , Figure 8 and Fig. 9 , Figure 7 A schematic diagram of the trajectory of a drone generated in the first period of a round by a strategy learned by the HyperQMIX algorithm in a simulation experiment provided in an embodiment of the present application, Figure 8 A schematic diagram of a drone trajectory generated in the second period of a round by a strategy learned by the HyperQMIX algorithm in a simulation experiment provided in an embodiment of the present application; Fig. 9A schematic diagram of the drone trajectory generated by the strategy learned by the HyperQMIX algorithm in a simulation experiment provided by an embodiment of the present application in the third period of a round. In these trajectory diagrams, x and y represent horizontal and vertical coordinates, in meters (m), the arrow in the dotted box U1 represents the moving trajectory of drone 1, the arrow in the dotted box U2 represents the moving trajectory of drone 2, the arrow in the dotted box U3 represents the moving trajectory of drone 3, and the arrow in the dotted box U4 represents the moving trajectory of drone 4. The square represents the data center, and the triangle represents the ground node. The four drones serve all ground nodes through trajectory design to improve the efficiency of data collection. In each time period, the drone collects data in different areas, maintains an appropriate distance, ensures that more ground nodes are covered, and avoids collisions and interference between links. Each drone flexibly switches between the three processes of movement, data collection, and data unloading. The circle represents the location of the drone's data collection each time. Most of these locations are located around the ground node, indicating that the drone has learned to optimize the air-to-ground channel through trajectory design to improve the efficiency of data transmission. The five-pointed star represents the location of drone data unloading. It can be seen that, in general, data unloading occurs when the drone moves from one ground node position to another. Once entering the communication range of the data center, the drone will immediately unload the collected data, which can avoid the continuous increase of information age. In addition, due to the continuous change of observation information within the perception range during the movement of the drone, the drone tends to fly to areas with a larger average information age and fewer drones, rather than always serving the same ground nodes in the same area. The above results further verify the effectiveness of HyperQMIX in policy learning.

[0193] See also Fig.10 and Fig.11 , Fig.10 This is a schematic diagram of convergence curves of multiple algorithms in a simulation experiment provided in an embodiment of the present application. Fig.11 A schematic diagram of convergence curves of multiple algorithms in another simulation experiment provided in an embodiment of the present application. Fig.10 and Fig.11 The horizontal axis is the number of training rounds and the vertical axis is the average information age (s). Fig.10The convergence curves of the first transfer learning algorithm ZT-HyperQMIX, the second transfer learning algorithm TL-HyperQMIX, the present algorithm (HyperQMIX algorithm), and the third algorithm QMIX are shown. ZT-HyperQMIX (Zero-Shot Transfer-based HyperQMIX) represents HyperQMIX based on zero-shot transfer. TL-HyperQMIX (Transfer Learning-based HyperQMIX) represents HyperQMIX based on transfer learning. It can be seen that by reloading the model, the target task obtains better initial performance. When HyperQMIX is trained from scratch, the initial average information age is 741 s. After using the pre-trained model, the initial average information age of ZT-HyperQMIX and TL-HyperQMIX is reduced to about 480 s, which is only 8.8% higher than the result after HyperQMIX converges. In addition, this result is 24.6% lower than the final average information age of QMIX, verifying the effectiveness of the pre-trained model in handling tasks of different scales. TL-HyperQMIX quickly achieved a performance level comparable to training from scratch by further fine-tuning the pre-trained model. These results demonstrate the excellent knowledge transfer capabilities of HyperQMIX.

[0194] See also Fig.12 , Fig.12 A schematic diagram of a convergence curve of an ablation experiment in a simulation experiment provided in an embodiment of the present application. Fig.12The horizontal axis is the number of training rounds, and the vertical axis is the average information age (s). The figure shows the positions of 3400 training rounds and 17000 training rounds. The figure shows the convergence curves of the network of this algorithm HyperQMIX, the network of the third algorithm rQMIX, the first reference network w / oHPE, the second reference network w / oHLQN, and the third reference network w / oHMN. w / oHPE means that only all HPE layers in HyperQMIX are replaced with FC layers, w / oHLQN means that only the hyperlocal Q network in HyperQMIX is replaced with the standard local Q network in QMIX, and w / oHMN means that only the hypermixed network in HyperQMIX is replaced with the standard hybrid Q network in QMIX. It can be seen that the performance of HyperQMIX with the HPE layer removed is better than that of algorithms using other permutation invariant models (such as DeepSets, Attention, and GNN). This is because the hypernetwork-based weight generation mechanism in the HPI layer enhances the representation ability of the network, thereby bringing performance improvement. Compared with the standard HyperQMIX, after removing the HPE layer, the average information age of the algorithm finally increased from 415s to 485s, an increase of 17%. This change reflects the performance loss caused by removing the HPE layer. After removing the hyperlocal Q network, the algorithm 4 The algorithm converged quickly within 1 training round, but its average information age after convergence reached 540s, which is 30% higher than HyperQMIX. This shows that the hyperlocal Q network with the HPI layer and HPE layer is significantly better than the standard local Q network in solving the optimal strategy. After removing the hypermixing network, the algorithm showed a slower convergence speed, indicating that the hypermixing network mainly plays a role in improving the convergence speed.

[0195] See also Fig.13 and Fig.14 , Fig.13 A schematic diagram showing a comparison of average information ages of different algorithms under different numbers of ground nodes provided in an embodiment of the present application. Fig.14 A schematic diagram of comparing the average information age of different algorithms under different numbers of drones provided in an embodiment of the present application. Fig.13 The horizontal axis is the number of ground nodes, the vertical axis is the average information age (s), M represents the number of ground nodes, and N represents the number of drones; Fig.14The horizontal axis is the number of drones, and the vertical axis is the average information age (s). The algorithms used for comparison include the transfer learning algorithm ZT-HyperQMIX, the transfer learning algorithm TL-HyperQMIX, the HyperQMIX algorithm, and the QMIX algorithm. In the figure, ① represents the HyperQMIX algorithm, ② represents the QMIX algorithm, ③ represents the transfer learning algorithm ZT-HyperQMIX, and ④ represents the transfer learning algorithm TL-HyperQMIX. Fig.13 The performance of different algorithms in terms of average information age under different numbers of ground nodes in the simulation experiment is shown; Fig.14 The performance of different algorithms in terms of average information age under different numbers of UAVs in the simulation experiment is shown. It can be seen that with the increase in the number of ground nodes, the QMIX algorithm has difficulty coping with the exponential growth of the state-action space, and the performance gradually decreases, resulting in an increase in the average information age. In contrast, HyperQMIX effectively handles the explosive growth of the state-action space by using its improved hyperlocal Q network and hypermixed network model, and has better scalability. Therefore, HyperQMIX performs significantly better than QMIX in all scenarios, and this advantage becomes more obvious as the number of ground nodes increases. As the number of UAVs increases, the performance gap between HyperQMIX and QMIX gradually widens. For example, with only one UAV, HyperQMIX performs 7.3% better than QMIX. When the number of UAVs increases to 16, this advantage increases to 58%, further verifying the scalability of HyperQMIX.

[0196] Based on the above embodiments, the present application provides a real-time drone trajectory optimization method that minimizes the task completion time under energy constraints for a single drone-assisted IoT data collection system. The present application first constructs a restricted Markov constraint process based on the original problem. Subsequently, the proposed method can update the original domain policy network parameters and the dual domain Lagrangian penalty coefficients in turn, and finally enables the drone agent to learn an adaptive strategy that can generate the optimal trajectory in real time under different device location layouts. The simulation results show that the method proposed in this application has improved the average task completion time and computational complexity compared to traditional optimization algorithms and other deep reinforcement learning algorithms, and has great practical significance.

[0197] The embodiment of the present application provides a trajectory planning device for collecting data from a drone, a scalable multi-agent reinforcement learning device, and the device includes:

[0198] A problem building module, used to establish a problem model for collecting data from a number of ground nodes for multiple drones and unloading the data to a ground data center, and to build a constrained Markov decision problem based on the problem model; a framework building module, used to build a deep reinforcement learning framework based on the SAC algorithm for the constrained Markov decision problem;

[0199] The model building module models the problem as Dec-POMDP using entity decomposition;

[0200] Framework building module, designing two types of neural network modules, HPI layer and HPE layer, introducing them into local Q network and global hybrid network to build a super local Q network and super hybrid network framework that meets permutation invariance and permutation equivariance;

[0201] Algorithm design module, designs a scalable multi-agent strategy learning algorithm based on HyperQMIX.

[0202] This embodiment establishes a problem model for collecting data from a number of ground nodes for multiple drones and unloading the data to a ground data center, and constructs a constrained Markov decision problem based on the problem model; the problem is modeled as Dec-POMDP in the form of entity decomposition; two types of neural network modules, HPI layer and HPE layer, are designed, and introduced into the local Q network and the global hybrid network to form a super-local Q network and a super-hybrid network that satisfy permutation invariance and permutation equivariance, and an extensible HyperQMIX algorithm based on the QMIX algorithm is proposed. It can be seen that the present application first establishes a problem model for collecting data from a number of ground nodes for multiple drones and unloading the data to a ground data center, and constructs a constrained Markov decision problem based on the problem model, and models the problem as Dec-POMDP, and then designs and introduces two types of neural network modules, HPI layer and HPE layer, into the QMIX algorithm, and proposes an extensible HyperQMIX algorithm, which realizes the permutation invariance and permutation equivariance in the super-local Q network and the super-hybrid network, making the network structure independent of the number of entities, and at the same time improving the sample efficiency of the algorithm in fixed-scale tasks.

[0203] See also Fig.15 , Fig.15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the steps provided in the above embodiment.

[0204] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0205] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0206] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0207] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the trajectory planning method of the drone data collection executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data received by the electronic device and transmitted from an external device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0208] Furthermore, the embodiment of the present application also discloses a storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0209] See also Fig.16 , Fig.16 A schematic diagram of the structure of a drone control system provided in an embodiment of the present application, the system may include:

[0210] A problem construction module, used to establish a problem model, and to construct a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that a plurality of drones collect data from a plurality of ground nodes and unload the collected data to a ground data center;

[0211] A model building module, used to model the constrained Markov decision problem as a target model; wherein the target model is a distributed local observable Markov decision process model;

[0212] An algorithm building module, used for obtaining a target algorithm by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model, and is used for solving an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance;

[0213] The control module is used to train the intelligent agents of the plurality of drones using the target algorithm, and perform trajectory planning and / or communication scheduling for the drones using the trained intelligent agents.

[0214] In this embodiment, a problem model is first established to represent that multiple drones collect data from multiple ground nodes and unload the collected data to a ground data center, and a constrained Markov decision problem is constructed based on the problem model, and the constrained Markov decision problem is modeled as a Dec-POMDP model. The Dec-POMDP model allows each drone agent to make decisions based on local information, while taking into account the existence and activities of other agents, which helps to deal with uncertainties in multi-agent systems and realize distributed control. In this embodiment, the HPI layer and the HPE layer are introduced into the QMIX algorithm to obtain the HyperQMIX algorithm. Since the HPI layer is a network layer based on the FC layer and has permutation invariance, and the HPE layer is a network layer based on the FC layer and has permutation equivariance, the HyperQMIX algorithm obtained in this embodiment can solve the agent strategy that adapts to the change in the number of entities. In this embodiment, the HyperQMIX algorithm is used to train the agents of multiple drones so that the trained agents can make decisions based on local information to adapt to dynamic characteristics and distributed control requirements. This embodiment uses the Dec-POMDP model and the HyperQMIX algorithm to enable the trained intelligent agent to adapt to various dynamic environmental factors and distributed control requirements during the process of controlling the drone.

[0215] Furthermore, the process of constructing a constrained Markov decision problem based on the problem model by the problem construction module includes: determining environmental state characteristics according to the problem model; wherein the environmental state characteristics include the current location information of the UAV, the location information of the current ground node, the location information of the current ground data center, the amount of data to be uploaded of the current ground node and the total amount of stored data of the UAV; determining a probabilistic line-of-sight model according to the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path; determining the trajectory variables and binary scheduling variables of the UAV according to the problem model; wherein the binary scheduling variables are used to describe the communication status between the ground node and the ground data center; determining the average information age function of the ground node according to the problem model; constructing the constrained Markov decision problem based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

[0216] Furthermore, the process of the model building module modeling the constrained Markov decision problem as a target model includes: performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity; wherein the feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node; defining key elements using the feature vector of the drone entity and the feature vector of the ground node entity, and building the target model based on the key elements; wherein the key elements include state, observation, action and reward.

[0217] Furthermore, it also includes:

[0218] A network layer construction module is used to introduce a first super network shared by all input elements into a single fully connected layer to form the first network layer before obtaining a target algorithm by introducing the first network layer and the second network layer into a multi-agent reinforcement learning algorithm; wherein the first super network takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features of the single input element and sum them to obtain the network output; and is also used to introduce a second super network shared by all output elements and a first network layer for processing the overall input of the output elements into a single fully connected layer to form a second network layer; wherein the second super network takes the input element corresponding to the single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply the output of the first network layer to obtain the network output.

[0219] Furthermore, the algorithm construction module obtains the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, including: building an input layer based on a fully connected layer and two first network layers; wherein the input of the input layer is all the features of the local observation of the drone, and the output of the input layer is a single embedded feature merged by summing operation; building an intermediate layer based on the gated recurrent unit in the multi-agent reinforcement learning algorithm; wherein the input of the intermediate layer is the single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot; building an output layer based on a fully connected layer and a second network layer; wherein the input of the output layer is the output of the intermediate layer, and the output of the output layer is the reward value of all actions in the action space and the optimal action corresponding to the maximum reward value; building a super local value network including the input layer, the intermediate layer and the output layer; wherein the super local value network is used to fit the local value function of the drone's agent; building a super hybrid network for combining the local reward values ​​of all the agents into a global reward value; and combining the super local value network of each agent and the super hybrid network to obtain the target algorithm.

[0220] Furthermore, the process of the algorithm building module building a super-hybrid network for combining the local reward values ​​of all the agents into a global reward value includes: using a fully connected input layer and a fully connected output layer to build the super-hybrid network; wherein the super-hybrid network is used to combine the local reward values ​​of all the agents into a global reward value; wherein the input of the fully connected input layer is the local reward value of all the agents, and the output of the fully connected input layer is a single embedded feature; the input of the fully connected output layer is a single embedded feature output by the input layer, and the output of the fully connected output layer is the global reward value.

[0221] Furthermore, it also includes:

[0222] A setting module is used to determine the weight matrix and bias vector of the fully connected input layer; and is also used to determine the weight matrix and bias vector of the fully connected output layer.

[0223] Furthermore, the process in which the control module uses the target algorithm to train the intelligent agents of the multiple drones includes: using the target algorithm to perform centralized offline training on the intelligent agents of the multiple drones, so that the intelligent agents of the drones use experience samples to update the parameters of the hyper-local value network; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

[0224] Furthermore, it also includes:

[0225] A strategy migration module is used to deploy the trained agents to the corresponding drones after centralized offline training of the agents of multiple drones using the target algorithm; wherein the trained agents include a hyperlocal value network with updated parameters; and is also used to perform strategy migration operations on the agents in the drones by means of model overloading.

[0226] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, which will not be repeated here.

[0227] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0228] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0229] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0230] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0231] Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A drone control method, characterized in that: include: Establishing a problem model, and constructing a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that multiple drones collect data from multiple ground nodes and unload the collected data to a ground data center; Modeling the constrained Markov decision problem as a target model; wherein the target model is a distributed locally observable Markov decision process model; A target algorithm is obtained by introducing the first network layer and the second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model, and is used to solve an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance; The target algorithm is used to train the intelligent agents of the plurality of drones, and the trained intelligent agents are used to perform trajectory planning and / or communication scheduling for the drones.

2. The drone control method according to claim 1, characterized in that: Constructing a constrained Markov decision problem based on the problem model includes: Determine the environmental state characteristics according to the problem model; wherein the environmental state characteristics include the current location information of the UAV, the location information of the current ground node, the location information of the current ground data center, the amount of data to be uploaded by the current ground node, and the total amount of stored data of the UAV; Determine a probabilistic line-of-sight model according to the problem model; wherein the probabilistic line-of-sight model includes a line-of-sight propagation path and a non-line-of-sight propagation path; Determining the trajectory variable and the binary scheduling variable of the UAV according to the problem model; wherein the binary scheduling variable is used to describe the communication status between the ground node and the ground data center; Determining the average information age function of the ground nodes according to the problem model; The constrained Markov decision problem is constructed based on the environmental state characteristics, the probabilistic line-of-sight model, the trajectory variables of the UAV, the binary scheduling variables, and the average information age function.

3. The drone control method according to claim 1, characterized in that: Modeling the constrained Markov decision problem as a target model includes: Performing entity decomposition on the constrained Markov decision problem to obtain a drone entity as an intelligent entity and a ground node entity as a non-intelligent entity; wherein the feature vector of the drone entity contains environmental information related to the drone, and the feature vector of the ground node entity contains environmental information related to the ground node; Key elements are defined using the feature vector of the drone entity and the feature vector of the ground node entity, and the target model is built based on the key elements; wherein the key elements include state, observation, action and reward.

4. The drone control method according to claim 1, characterized in that: Before obtaining the target algorithm by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, it also includes: Introducing a first super network shared by all input elements into a single fully connected layer to form the first network layer; wherein the first super network takes a single input element as input to generate a first weight matrix corresponding to each input element, and the first weight matrix is ​​used to extract features from the single input element and sum them to obtain a network output; A second super network shared by all output elements and a first network layer for processing the overall input of the output elements are introduced into a single fully connected layer to form a second network layer; wherein the second super network takes the input element corresponding to the single output element as input to generate a second weight matrix corresponding to each output element, and the second weight matrix is ​​used to multiply the output of the first network layer to obtain the network output.

5. The drone control method according to claim 1, characterized in that: The target algorithm is obtained by introducing the first network layer and the second network layer into the multi-agent reinforcement learning algorithm, including: An input layer is constructed based on a fully connected layer and two first network layers; wherein the input of the input layer is all features in the local observation of the UAV, and the output of the input layer is a single embedded feature merged by a summation operation; An intermediate layer is built based on the gated recurrent unit in the multi-agent reinforcement learning algorithm; wherein the input of the intermediate layer is a single embedded feature output by the input layer and the hidden layer state of the previous time slot, and the output of the intermediate layer is the hidden layer state of the current time slot; An output layer is constructed based on a fully connected layer and a second network layer; wherein the input of the output layer is the output of the intermediate layer, and the output of the output layer is the reward values ​​of all actions in the action space and the optimal action corresponding to the maximum reward value; Constructing a hyperlocal value network including the input layer, the intermediate layer, and the output layer; wherein the hyperlocal value network is used to fit the local value function of the intelligent agent of the drone; Building a hypermixing network for combining the local rewards of all the agents into a global reward value; The target algorithm is obtained by combining the hyperlocal value network of each agent and the hypermixed network.

6. The drone control method according to claim 5, characterized in that: The super-hybrid network for combining the local reward values ​​of all the agents into a global reward value comprises: The super-hybrid network is constructed using a fully connected input layer and a fully connected output layer; wherein the super-hybrid network is used to combine the local reward values ​​of all the agents into a global reward value; Among them, the input of the fully connected input layer is the local reward value of all intelligent agents, and the output of the fully connected input layer is a single embedded feature; the input of the fully connected output layer is a single embedded feature output by the input layer, and the output of the fully connected output layer is a global reward value.

7. The drone control method according to claim 6, characterized in that: Also includes: Determining a weight matrix and a bias vector of the fully connected input layer; The weight matrix and bias vector of the fully connected output layer are determined.

8. The drone control method according to claim 5, characterized in that: Using the target algorithm to train the intelligent agents of the plurality of drones comprises: The target algorithm is used to perform centralized offline training on the intelligent agents of the plurality of drones, so that the intelligent agents of the drones update the parameters of the hyperlocal value network using experience samples; wherein the experience samples are experience samples collected by the intelligent agents of the drones through interaction with the environment.

9. The drone control method according to claim 8, characterized in that: After the target algorithm is used to perform centralized offline training on the intelligent agents of the plurality of drones, the method further includes: Deploy the trained intelligent agent to the corresponding drone; wherein the trained intelligent agent includes a hyperlocal value network with updated parameters; The strategy transfer operation is performed on the intelligent agent in the UAV by means of model reloading.

10. A drone control system, characterized in that: include: A problem construction module, used to establish a problem model, and to construct a constrained Markov decision problem based on the problem model; wherein the problem model is used to represent that a plurality of drones collect data from a plurality of ground nodes and unload the collected data to a ground data center; A model building module, used to model the constrained Markov decision problem as a target model; wherein the target model is a distributed local observable Markov decision process model; An algorithm building module, used for obtaining a target algorithm by introducing a first network layer and a second network layer into a multi-agent reinforcement learning algorithm; wherein the target algorithm is an algorithm designed based on a target model, and is used for solving an agent strategy that adapts to changes in the number of entities; the first network layer is a network layer based on a fully connected layer and having permutation invariance, and the second network layer is a network layer based on a fully connected layer and having permutation equivariance; The control module is used to train the intelligent agents of the plurality of drones using the target algorithm, and perform trajectory planning and / or communication scheduling for the drones using the trained intelligent agents.

11. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the drone control method according to any one of claims 1 to 9 are implemented.

12. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the drone control method according to any one of claims 1 to 9 are implemented.

Images