A multi-unmanned aerial vehicle scheduling method based on hierarchical multi-agent reinforcement learning

Abstract

The application discloses a multi-unmanned aerial vehicle scheduling method based on layered multi-agent reinforcement learning, and establishes a layered reinforcement learning framework to guide multi-unmanned aerial vehicle scheduling, wherein the layered reinforcement learning framework comprises instruction generation of an upper manager and instruction learning of lower unmanned aerial vehicles; the upper manager collects visual field information of the lower unmanned aerial vehicles and gives a strategy in a longer time scale to the lower unmanned aerial vehicles; the lower unmanned aerial vehicles automatically learn an instruction set sent by the upper manager by using a subsequent state feature, and execute specific decisions according to the instruction set sent by the upper manager and current visual field observation. The above method can realize cooperation of the multi-unmanned aerial vehicles, increase generalization ability to different environments and different unmanned aerial vehicle formation compositions, and enhance scalability of the algorithm, and can be applied to multi-unmanned aerial vehicle tasks such as temporary communication service provided by the unmanned aerial vehicles, unmanned aerial vehicle Internet of Things construction, mobile edge computing and multi-unmanned aerial vehicle tracking.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a multi-UAV scheduling method based on hierarchical multi-agent reinforcement learning. Background Technology

[0002] In recent years, Unmanned Aerial Vehicle (UAV) technology has developed rapidly and has been widely used in civilian fields such as agricultural monitoring, search and rescue, and logistics. However, individual UAVs are limited by flight distance and payload, and their ability to respond to emergencies is also limited. In contrast, multi-UAV systems offer better performance, more flexible applications, and stronger fault response capabilities. As the cost of UAVs decreases, the application of multi-UAV collaborative missions is becoming more widespread, and the need for multi-UAV collaborative planning is becoming increasingly urgent. In multi-UAV deployments, an effective algorithm is needed for task scheduling and to achieve multi-UAV collaboration. Multi-UAV missions in different scenarios share common attributes and can be modeled uniformly. Users scattered across a map require certain services from UAVs. During multi-UAV operation, new service demands may constantly arise. When users are within the service range of a UAV, the UAV can consume certain resources (including computing resources, communication resources, etc.) and time to provide services. During mission execution, UAVs may encounter malfunctions or run out of power, and other UAVs may join the team midway through the mission. During the mission, it is necessary to achieve goals such as meeting user needs, improving service quality, and reducing UAV energy consumption; therefore, collaboration between UAVs is essential. Under this modeling, the multi-UAV scheduling problem can be reduced to the multi-traveling salesman problem, which is an NP-hard combinatorial optimization problem.

[0003] With the development of reinforcement learning (RL), researchers are attempting to apply this technology to the multi-UAV domain. Reinforcement learning policies obtain feedback rewards through interaction with the environment, requiring no prior knowledge of the environment and learning an effective policy from scratch. Furthermore, reinforcement learning is a real-time decision-making method, advantageous in scenarios requiring timely response and rapid adaptation. Multi-agent reinforcement learning, following a centralized training and distributed execution model, better suits the limitations of local observation and the need for global coordination in multi-UAV applications. Multi-agent reinforcement learning has broad research prospects and value in the problem of multi-UAV cooperative scheduling. Current multi-agent reinforcement learning algorithms typically assume a fixed number of UAVs, but in practical applications, the composition of UAV formations is likely to change during task execution, potentially weakening the collaborative ability of the reinforcement learning policy. Simultaneously, the scalability of multi-agent reinforcement learning algorithms is insufficient, typically limited to learning cooperative patterns of 3-8 agents, making it difficult to apply to large-scale UAV formations. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-UAV scheduling method based on hierarchical multi-agent reinforcement learning. This method can achieve multi-UAV cooperation while increasing the generalization ability to different environments and different UAV formations, and enhancing the scalability of the algorithm.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A multi-UAV scheduling method based on hierarchical multi-agent reinforcement learning, the method comprising:

[0007] Step 1: Establish a hierarchical reinforcement learning framework to guide the scheduling of multiple UAVs. The hierarchical reinforcement learning framework includes the generation of instructions by the upper-level manager and the learning of instructions by the lower-level UAVs.

[0008] Step 2: Upper-level managers collect the field-of-view information of lower-level drones and provide strategies for the lower-level drones on a longer time scale.

[0009] Step 3: The lower-level UAV automatically learns the instruction set sent by the upper-level manager by utilizing the subsequent state characteristics, and executes specific decisions based on the instruction set sent by the upper-level manager and the current field of view observation.

[0010] As can be seen from the technical solution provided by the present invention, the above method can realize multi-drone collaboration, while increasing the generalization ability to different environments and different drone formations, enhancing the scalability of the algorithm, and can be applied to multi-drone tasks such as drones providing temporary communication services, drone IoT construction, mobile edge computing, and multi-drone tracking. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the multi-UAV scheduling method based on hierarchical multi-agent reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, and do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0014] like Figure 1 The diagram shows a flowchart of a multi-UAV scheduling method based on hierarchical multi-agent reinforcement learning provided by an embodiment of the present invention. The method includes:

[0015] Step 1: Establish a hierarchical reinforcement learning framework to guide the scheduling of multiple UAVs. The hierarchical reinforcement learning framework includes the generation of instructions by the upper-level manager and the learning of instructions by the lower-level UAVs.

[0016] In this step, in multi-agent reinforcement learning, the multi-UAV scheduling problem is defined as a partially observable Markov decision process (POMDP), denoted by M = (N, S, O, A, R, γ, P), where N = {1, ..., n} is the set of agents, i.e., the set of individual UAVs; in specific implementations, the number of individual UAVs does not need to be fixed and can increase or decrease within a certain range; S is the global state space, which cannot be directly obtained by the individual UAVs; O = X i∈N O i It is the joint observation space of individual drones, where O i Let X represent the observation space of the i-th agent. i∈N O i The global state S is represented by the Cartesian product of the observation spaces of each agent. In implementation, it is usually approximated by the concatenation of the local observations of all agents; A = X i∈N A i It is the action space of individual drones, where A i Let X represent the action space of the i-th agent. i∈N A i Represents the Cartesian product of the action spaces of each agent; It is the reward function, where Let represent a one-dimensional real space; γ∈[0,1] is the reward discount factor. If the value of γ is close to 0, the observable Markov decision process POMDP focuses on the current reward value, and if the value of γ is close to 1, it focuses on the future reward value; P: S×A×S→[0,1] is the state transition probability function.

[0017] In the setting of Partially Observable Markov Decision Processes (POMDPs), the task execution process is divided into several discrete time steps; in a certain time step t, the agent (i.e., the drone) receives environmental observations s. t ∈O=X i∈N O i The action a is determined based on the agent's policy π(a|s). t ∈A=X i∈N A i And receive the reward r(s) at the current time step. t a t Simultaneously, the environment transitions to the next state s based on current state observations, agent actions, and the state transition function P. t+1 Repeat the above process until the termination state is reached or the time step reaches the upper limit.

[0018] The task of multi-agent reinforcement is to learn a policy π(a|s) and determine actions in different states to maximize the expected cumulative reward, expressed as:

[0019]

[0020] in, This indicates the expectation, and (s) t a t The probability distribution of ) follows the strategy π(a|s);

[0021] The reward function R adopts the Quality of Service (QoS) of the individual drone, that is, the reinforcement learning objective is to maximize the cumulative QoS;

[0022] In the hierarchical reinforcement learning framework, the upper-level manager receives an environmental reward r(s, a), which is the reward in the POMDP, and provides policy guidance to the lower-level drones. Let the instruction be z, and the reward received by the lower-level drone is r(s, a) + r. in (s, z, a) includes two parts: environmental reward and intrinsic reward. The intrinsic reward r in (s, z, a) are given internally by the multi-agent system and are used to guide the lower-level drones to follow the instructions z of the upper-level manager;

[0023] Let the strategy of upper-level managers be π. H Upper management strategy π H The upper-level manager receives state observations s as input and outputs instructions z to the lower-level drone; the goal of reinforcement learning is to adjust π given the current lower-level policy. H To maximize the sum of accumulated upper-level rewards Q H The expression is:

[0024]

[0025] Let the lower-level drone strategy be π. L Lower-level drone strategy π L The UAV receives state observations s and upper-layer instructions z as inputs and outputs the UAV action a; the goal of the lower-layer UAV reinforcement learning is: given the current upper-layer policy, adjust π L To maximize the sum of accumulated lower-level rewards Q L The expression is:

[0026]

[0027] Reinforcement learning algorithms were applied to different levels to obtain policy π. L , π H .

[0028] Step 2: Upper-level managers collect the field-of-view information of lower-level drones and provide strategies for the lower-level drones on a longer time scale.

[0029] In this step, upper-level managers output discretized instructions, corresponding to different collaborative behavior patterns, i.e., π. H (s): S→{0,1} m , where m is the dimension of the instruction vector; specifically, it means: the upper-level manager's strategy π H Taking the environment state as input, output an m-dimensional instruction, where each dimension of the instruction is 0 or 1;

[0030] To enable the upper layer to set different strategies for different scenarios, this invention draws on the transfer learning method, analogizing the switching of cooperation modes in different scenarios to task transfer, and adds a successor representation (SR) module to the upper layer to make the generalization ability of the upper layer decision stronger.

[0031] In the subsequent representation, the reward function r(s, a) is decomposed into the product of the environmental dynamics φ and the weight w, i.e.:

[0032] r(s, a) = φ(s, a) T w(2)

[0033] Let ψ(s, a) = ∑ t γ t If φ(s, a), then the sum of cumulative rewards Q(s, a) can be written in the following form:

[0034]

[0035] Adding a successor representation at the upper level, we have r(s, a) = φ(s, a) T w, Q H(s, z) = ψ(φ, z) T w, φ(s, a) represent the single-step environmental change from the perspective of the specific action at the lower level, and ψ(φ, z) represent the long-term environmental change from the perspective of the action at the upper level, i.e., the subsequent representation, which is a further abstract representation of the environment based on φ(s, a). Therefore, the calculation of ψ(φ, z) depends on φ(s, a).

[0036] Upper management comprises three parts: upper management strategy π H The single-step environmental change φ(s, a) and reward weight w, and the subsequent representation ψ(φ, z); where the upper-level instruction π H The latter three are represented by a neural network, and their specific parameters are learned through gradient descent and do not need to be defined manually. In the learning process of single-step environmental change φ(s, a), reward weight w, and subsequent representation ψ(φ, z), the data used for global state observation s, local state observation o, lower-level action a, and upper-level instruction z all come from the dataset collected during training.

[0037] To ensure that the single-step environmental change φ(s, a) contains complete environmental information, it is necessary to be able to decode the original environmental state s from φ(s, a). Therefore, an auxiliary network D(φ) is introduced during the training phase to decode the original environmental state s from the single-step environmental change φ(s, a). The goal of gradient descent is to minimize the state reconstruction loss, and the loss functions L(φ) and L(D) are shown in Equation (4):

[0038] L(φ)=L(D)=||D(φ(s,a))-s|| 2 (4)

[0039] Where φ and D are regarded as the encoder and decoder of the environment state s; D(φ(s,a)) means that encoding and decoding are performed first, which is called reconstruction. Minimizing the state reconstruction loss means that the reconstruction result is the closest to the original state input s.

[0040] The training method represented by formula (4) references the autoencoder (AE) algorithm, and can ultimately learn an effective encoder φ;

[0041] According to the definition of the successor representation ψ(φ, z), the training objective of the weight w is to minimize the reconstruction loss of the reward function r(s, a), and the loss function L(w) is expressed as:

[0042] L(w)=||φ(s,a) T wr(s, a)|| 2 (5)

[0043] The successor representation ψ(φ, z) is the discounted cumulative sum of the single-step environmental changes φ(s, a). Therefore, the training objective of the successor representation ψ(φ, z) is to minimize the temporal difference loss L(ψ), expressed as:

[0044]

[0045] in, This indicates the use of a ψ(φ, z) network, but the gradient of this operation is not included in the training process; s′ represents the successor state of state s; z′ represents the upper-level instruction corresponding to step s′.

[0046] The goal of upper-level reinforcement learning is to maximize the sum of cumulative rewards, Q. H (s, z) = ψ(φ, z) T w, given state observations s, and the upper-level manager's strategy π. H The calculation method is as follows:

[0047] π H (s)=max z ψ(φ(s), z) T w (7)

[0048] In practical calculations, since ψ, φ, and w are represented by neural networks and are all continuously differentiable functions, the maximum value in formula (7) can be obtained using Newton's method. However, Newton's method takes longer to calculate than using a fixed π. H The function is a function, but the upper-level managers are more concerned with the spatial and temporal scales than the lower-level managers, so they believe that the latency caused by this part of the time consumption is acceptable.

[0049] φ and ψ are independent of the composition of the lower-level drones and depend only on the environment, which remains constant during mission execution. When the composition of the lower-level drone fleet changes, the upper-level manager can quickly adapt to the changes by updating the weights w.

[0050] φ and ψ are learned during training and fixed during execution; the weights w are learned during training and will continue to be updated during execution to cope with changes in the composition of the lower-level drone fleet during execution.

[0051] Step 3: The lower-level UAV automatically learns the instruction set sent by the upper-level manager by utilizing the subsequent state characteristics, and executes specific decisions based on the instruction set sent by the upper-level manager and the current field of view observation.

[0052] In this step, to ensure that the lower-level drones ultimately behave in accordance with the instructions given by the upper-level manager, intrinsic rewards are needed to guide the lower-level drones to follow the instructions. Since the upper-level manager's instructions do not have manually defined semantics, the expected performance of the lower-level drones in following the instructions is that they exhibit different behavioral patterns under different instructions. Specifically, we hope that the behavioral trajectories of the lower-level drones under various instructions will produce a clustering effect in the latent space, and the specific class definitions are given by the instructions generated by the upper level, with each instruction corresponding to a class in the latent space.

[0053] In the process of the lower-level UAV automatically learning the instruction set sent by the upper-level manager using subsequent state features, an auxiliary network, denoted as encoder E, is introduced during the training phase. L Used to transmit the observation trajectory (o) of the lower-level UAV in k steps. t , ..., o t+k The image is projected into the latent space, i.e., the encoder E. L The received input is the observation trajectory (o) of the lower-level UAV in k steps. t , ..., o t+k The auxiliary network is trained using the cross-entropy loss function L(E), which outputs the projection z of the observed trajectory in the latent space. L ) is represented as:

[0054] L(E L =CrossEntropy(E L (z|o t , ..., o t+k ), z) (8)

[0055] Encoder E L The latent space dimension of the projected lower-level UAV observation trajectory is the same as the dimension m of the instruction set sent by the upper-level manager. Through cross-entropy loss training, the encoder output is made to closely approximate the distribution of instructions z in the dataset. Therefore, the encoder output is also represented by z. However, the latent space corresponding to the encoder output is a continuous space, where the value of each dimension can be interpreted as the probability that the corresponding instruction for the observation trajectory has a value of 1, unlike the discrete instruction set space. The encoder E... L Provide the intrinsic reward for the lower-level drones during training:

[0056]

[0057] That is, the observation trajectory of the lower-level UAV for steps t to (t+k) corresponds to the instruction set z sent by the upper-level manager, where:

[0058] In the introduced encoder E L When completely accurate, i.e., encoder E L To minimize equation (8), since the instruction set z∈{0,1}m If the observation trajectory of the lower-level UAV (o t , ..., o t+k If it fully conforms to instruction set z, then the formula softmax(E) is used. L (z|o t , ..., o t+k Let z be the approximation;

[0059] Otherwise, use the formula ||softmax(E) L (z|o t , ..., o t+k ))-z|| 2 Encoder E L The degree of incompatibility between the output and the instruction set z.

[0060] Therefore, the hierarchical reinforcement learning introduced in this embodiment of the invention provides hierarchical command from macro to micro levels, enhancing global coordination among agents. Under the hierarchical reinforcement learning framework, the upper layer makes decisions from the perspective of macro observation and long-term reward, and conveys the goal to multiple lower layers, namely individual drones; the lower layers combine the instructions given by the upper layer with their own field of vision observations to derive the final action to be executed.

[0061] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0062] The following example illustrates the method described in this embodiment of the invention. In this example, the multi-drone task is set to provide temporary communication services in a disaster relief scenario. The global state S is the location of all people who need communication (hereinafter referred to as drone service users) in the application rescue scenario, as well as the current coordinates and speeds of all drones, which cannot be known in advance. The agent observation O is the coordinates and speed of each drone, as well as other drones and users to be served within the field of vision. The agent action A is the speed direction and speed magnitude of its own drone. The reward function R consists of local rewards and global rewards. If the current drone is providing communication services to a user, it receives a +1 reward; if it is providing services to the same user simultaneously with other drones, resulting in redundancy, the reward value is +1 divided by the redundancy ratio; if it is idle, it receives a -1 reward; if it completes service for a user, it receives a +5 reward.

[0063] The global reward consists of two parts: the number of users covered by the drone and a penalty for users not receiving service. The penalty for a single user is proportional to the waiting time for service. The goal of multi-agent reinforcement learning is to optimize the global reward, but the global reward can only be obtained in a simulated environment. In a real environment, consider using only local rewards.

[0064] In this example, the process of a drone providing temporary base station service is simplified to the drone hovering over the people in need of service for a period of time. In the simulated environment, users requesting communication services are distributed throughout the map, and their coordinates are unknown to the drone agent. Communication requests are only received and user coordinates are determined after the drone enters the corresponding communication range. Each user requires a certain duration of communication service. Once the service duration is met, the user disappears from the map, and the simulation of that part of the rescue process ends. Communication service users may appear midway through a simulation, representing newly discovered people in need of rescue.

[0065] This example uses a Multi-UAV Multi-Service OpenAI Gym simulation environment as the training environment. This environment has the following characteristics: obstacles exist that the drones need to avoid; each drone may run out of power; new drones may be added during operation, and existing drones may leave the team due to power depletion or collisions with obstacles. The QMIX algorithm is used as the basic multi-agent reinforcement learning algorithm, and the example is divided into two parts: a training phase and an execution phase. The operation process is described below.

[0066] Assume the policy parameter π of the lower-level drone L The subsequent representation parameters ψ, φ, and w of the environment maintained by the upper-level managers have all been learned during the training phase. The execution phase process is as follows:

[0067] 1. The lower-level UAVs 1, 2, ..., n each update their own observations o1, ..., o2 based on the current environment. n .in, It includes its own coordinates, velocity, and field of view image.

[0068] 2. Lower-level drones 1, 2, ..., n each attempt to receive a set of instructions z sent from the upper-level administrator. The received content in this step may be empty; if empty, the previously received instruction is retained. If a drone has never received an instruction from the upper level, the next step uses an empty instruction (0, ..., 0) ∈ {0, 1}. m run.

[0069] 3. The lower-level UAVs 1, 2, ..., n each observe o according to their own observations. i The final action is calculated using the upper-level instruction set z. Perform the action and proceed to step 1.

[0070] 4. After repeating steps 1-3k times, the lower-level UAV will display the local observations at the current time step. i and the local reward r iSend to the upper-level administrator. There may be a delay in the message sending process; the handling method will be detailed in the next step. Steps 1-4 do not require drones to wait or synchronize; instead, they are executed in a distributed manner.

[0071] 5. The upper-level manager receives observations and rewards sent by the lower-level drones, and stitches together the local observations to form an approximate global state s = (o1, ..., o2). n The message reception process begins when the first observation from a lower-level drone is received and ends when all observations from lower-level drones are received or when the waiting time limit is exceeded. If no observation from a lower-level drone j is received by the end of the process, the last received observation is used instead. If no new state of a drone is received within 2×k steps, the drone is considered to have left the queue, and its state is no longer maintained.

[0072] 6. If the composition of the lower-level drone fleet has changed compared to the last time the upper-level instructions were calculated, the weight w of the environmental dynamic reward is updated according to formula (5).

[0073] 7. The upper-level manager calculates the new instruction z = π according to formula (7). H (s) sends a new instruction set z to the lower-level UAV and jumps to step 1.

[0074] 8. Repeat steps 1-7 until the task is completed.

[0075] The training phase requires the lower-level UAV to interact with the environment to collect training data. Some processes are the same as or similar to the execution phase. Since the training phase only takes place in a simulated environment, communication latency is not considered. Let the dataset be D, and the training phase process is as follows:

[0076] 1. During the execution phase, steps 1-3 of the process will be executed, where (s = {o1, ..., o...} n}, a = {a1, ..., a n}, r, s′={o′1,...,o′ n}, a′={a′1,...,a′ n Store in temporary dataset D T .

[0077] 2. After repeating step 1k times, the lower-level UAV will record the current observation o i The data is sent to the upper-level administrator; the upper-level administrator receives the observations sent by the lower-level drones and splices them together to form an approximate global state s = (o1, ..., o2). n ).

[0078] 3. The upper-level manager updates the weight w of the environmental dynamic reward according to formula (5).

[0079] 4. The upper-level manager uses the updated weight w to calculate the new instruction z′=π according to formula (5). H (s) sends new instructions to the lower-level drones.

[0080] 5. The temporary dataset D at this point... T All content, along with the old instruction z and the current environment weight w, are stored as a single data entry in dataset D. The temporary dataset D is then cleared. T Jump to step 1.

[0081] 6. After repeating steps 1-5p times, sample b data points from dataset D and use the sampled data to update the parameters of each module.

[0082] 7. Calculate the intrinsic reward r obtained by the lower-level UAV for each data point according to formula (9). in .

[0083] 8. Update the lower-level trajectory encoder E according to formula (8). L parameter.

[0084] 9. Use r+r in As the lower-level reward value, the policy parameter π of the lower-level drone is updated according to the QMIX algorithm. L .

[0085] 10. According to formulas (4) and (6), update the upper-level manager instructions and subsequent representation parameters ψ and φ, and jump to step 1.

[0086] 11. After repeating steps 1-10q times, use the weights w in the dataset and the updated ψ and φ to recalculate the corresponding upper-level instructions for all entries in dataset D, update dataset D, and jump to step 1.

[0087] 12. Repeat steps 1-11 until the strategy parameter π of the lower-level UAV is reached. L The subsequent representation parameters ψ and φ of the upper-level managers converge.

[0088] In summary, the method described in the embodiments of the present invention has the following advantages:

[0089] (1) No prior knowledge of the environment is required, and no heuristic functions need to be written manually.

[0090] (2) Upper-level managers can automatically learn the instruction set without manually defining it; lower-level drones can learn how to follow instructions without manually writing the corresponding behavioral characteristics.

[0091] (3) For different team compositions, or in the event of a drone suddenly becoming damaged and unable to work, upper management can switch strategies and adjust the collaboration mode between drones so that the mission is not affected.

[0092] (4) Through hierarchical multi-agent reinforcement learning, it can cope with the changes in UAV formation during the task execution process and has stronger robustness and generalization ability; at the same time, hierarchical reinforcement learning also enhances the scalability of multi-agent reinforcement learning algorithms.

[0093] Furthermore, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0094] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A multi-unmanned aerial vehicle scheduling method based on hierarchical multi-agent reinforcement learning, characterized in that, The method comprises: Step 1, establishing a hierarchical reinforcement learning framework to guide multi-unmanned aerial vehicle scheduling, the hierarchical reinforcement learning framework comprising instruction generation of an upper manager and instruction learning of lower unmanned aerial vehicles; In step 1, in multi-agent reinforcement learning, the multi-UAV scheduling problem is defined as a partially observable Markov decision process (POMDP) denoted by M = (N, S, O, A, R, γ, P), where N = {1,..., n} is the set of agents, i.e., the set of UAV individuals; S is the global state space, which cannot be directly obtained by the UAV individuals; O = X i∈N O i is the joint observation space of the UAV individuals, where O i denotes the observation space of the i-th agent, X i∈N O i denotes the Cartesian product of the individual observation spaces; A = X i∈N A i is the joint action space of the UAV individuals, where A i denotes the action space of the i-th agent, X i∈N A i denotes the Cartesian product of the individual action spaces; R: S x A → R is the reward function, where denotes the one-dimensional real number space; γ ∈ [0, 1] is the reward discount factor, with a value close to 0 indicating that the POMDP focuses on current rewards, and a value close to 1 indicating that it focuses on future rewards; P: S x A x S → [0, 1] is the state transition probability function. In the setting of a partially observable Markov decision process (POMDP), the task execution process is divided into a number of discrete time steps; at a time step t, the agent receives an observation s t ∈ O = X i∈N O i from the environment, decides an action a t ∈ A = X i∈N A i according to its policy π(a|s), and receives a reward r(s t , a t ) for the current time step; meanwhile, the environment transitions to a next state s t+1 according to the current state observation, the agent action, and a state transition function P; the above process is repeated until a termination state is reached or a time step limit is reached. The multi-agent reinforcement learning task is to learn a strategy π(a|s) to determine actions in different states to maximize the expected cumulative reward J, expressed as: where denotes the expectation, and (s t ,a t ) follows the policy π(a|s). The reward function R adopts the quality of service QoS of the unmanned aerial vehicle individual, that is, the reinforcement learning goal is to maximize the cumulative QoS; Under the hierarchical reinforcement learning framework, the upper manager obtains the environment reward r(s, a), i.e. the reward in POMDP, and gives the policy guidance to the lower UAV, denoted as z; the lower UAV obtains the reward r(s, a) + r(s, z, a) in (s, z, a) includes two parts of environment reward and intrinsic reward r(s, z, a), and the intrinsic reward r(s, z, a) is given by the multi-agent system internally, for guiding the lower UAV to follow the instruction z of the upper manager; in (s, z, a) includes two parts of environment reward and intrinsic reward r(s, z, a), and the intrinsic reward r(s, z, a) is given by the multi-agent system internally, for guiding the lower UAV to follow the instruction z of the upper manager; Let the upper manager policy be π H Let the upper manager policy be π H Receive state observation s as input, and output instruction z to the lower drone; the goal of upper manager reinforcement learning is to adjust π H To maximize the sum of cumulative upper rewards Q H The expression is: Let the lower UAV policy be π L , the lower UAV policy π L Receive state observation s and upper-level instruction z as input, and output the action a of the UAV; the goal of the lower UAV reinforcement learning is to adjust π L to maximize the sum of cumulative lower-level rewards Q L , which is expressed as: Different levels apply reinforcement learning algorithm respectively, and get strategy π L , π H ; Step 2, the upper manager collects the visual field information of the lower unmanned aerial vehicles, and gives the lower unmanned aerial vehicles a strategy in a longer time scale; Step 3, the lower unmanned aerial vehicles automatically learn the instruction set sent by the upper manager by using the successor state feature, and execute specific decisions according to the instruction set sent by the upper manager and the current visual field observation.

2. The multi-UAV scheduling method based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, In step 2, the upper manager outputs the discretized instruction, corresponding to different cooperative behavior patterns, i.e. π H (s) : S → {0, 1} m where m is the dimension of the instruction vector; the specific meaning is: the upper manager strategy π H The environment state is input, and the m-dimensional instruction is output, each dimension of the instruction is 0 or 1; In the successor representation, the reward function r(s,a) is decomposed into the product of the environment dynamics φ and the weight w, that is: r(s, a) = φ(s, a) T w (2) Let ψ(s, a) = ∑ t γ t φ(s, a), then the sum of the cumulative rewards, Q(s, a), is written as follows: After the upper layer is added, the successor representation is r(s, a) = φ(s, a) T W, Q H (s, z) = ψ(φ, z) T W, φ(s, a) represents the single-step environmental change of the lower layer specific action perspective, and ψ(φ, z) represents the long-term environmental change of the upper layer action perspective, that is, the successor representation, which is a further abstract representation of the environment based on φ(s, a). Therefore, the calculation of ψ(φ, z) depends on φ(s, a); The upper manager includes three parts: upper manager policy π H , single-step environment change φ(s, a) and reward weight w, and successor representation ψ(φ, z); In order to ensure that the single-step environment change φ(s,a) contains complete environment information, it is necessary to decode the original environment state s from φ(s,a), and thus an auxiliary network D(φ) in the training stage is introduced to decode the original environment state s from the single-step environment change φ(s,a), and the gradient descent goal is to minimize the state reconstruction loss, the loss function L(φ), L(D) is shown in formula (4): L(φ) = L(D) = ||D(φ(s,a)) - s|| 2 (4) Wherein, φ, D are regarded as the encoder and decoder of the environment state s; D(φ(s,a)) indicates that the encoding is performed first and then the decoding, which is called reconstruction, and the minimization of the state reconstruction loss means that the reconstruction result is expected to be closest to the original state input s; According to the definition of the successor representation ψ(φ,z), the training goal of the weight w is to minimize the reward function r(s,a) reconstruction loss, and the loss function L(w) is expressed as: L(w) = ||φ(s, a) - w||2 T w-r(s, a)| 2 (5) The successor representation ψ(φ,z) is the discounted cumulative sum of the single-step environment change φ(s,a), and therefore the training goal of the successor representation ψ(φ,z) is to minimize the time difference loss L(ψ), which is expressed as: wherein, denotes the use of the ψ(φ, z) network, but the gradient of the operation here is not taken into account during training; s' denotes the successor state of state s; z' denotes the upper-level instruction corresponding to the step of s'; The objective of the upper manager reinforcement learning is to maximize the sum of Q H (s, z) = ψ(φ, z) T The upper manager policy π H is calculated as π H (s) = max z ψ(φ(s),z) T w (7) When the lower unmanned aerial vehicle team changes, the upper manager can quickly adapt to the change through the update of the weight w.

3. The method of claim 1, wherein, In step 3, in the process that the lower-layer UAV automatically learns the instruction set sent by the upper-layer manager by using the subsequent state features, an auxiliary network in a training stage is introduced, denoted as an encoder E L , which is used to project the observation trajectory (o t ,..., o t+k ) of the lower-layer UAV for k steps into a hidden space, that is, the encoder E L receives the input of the observation trajectory (o t ,..., o t+k ) of the lower-layer UAV for k steps, and outputs the projection z of the observation trajectory in the hidden space; the auxiliary network is trained by using a cross-entropy loss function, and a loss function L(E L ) is represented as: L(E L )=CrossEntropy(E L (z|o t ,...,o t+k ),z) (8) Encoder E L The dimension of the hidden space of the lower-layer UAV observation trajectory projection is the same as the dimension m of the instruction set sent by the upper-layer manager; an encoder E is used L The intrinsic reward of the lower-layer UAV during training is given, which is represented as: That is, the observation trajectory of the lower unmanned aerial vehicle t~(t+k) steps corresponds to the instruction set z sent by the upper manager, wherein: At the introduced encoder E L completely accurate, i.e. the encoder E L Minimizing the formula (8), since the instruction set z∈{0,1} m , if the observation trajectory (o t ,..., o t+k ) of the lower-layer UAV completely conforms to the instruction set z, it is expressed by the formula softmax(E L (z|o t ,..., o t+k ))≈z; Otherwise, use the formula ||softmax(E L (Z|o t ,..., o t+k ))-z|| 2 to represent the degree of inconsistency of the output of the encoder E L with the instruction set z.

Images