A multi-agent reinforcement learning method for multi-unmanned aerial vehicle cooperative resource optimization

Abstract

The present application relates to the technical field of unmanned aerial vehicle cooperative control, and particularly relates to a multi-agent reinforcement learning method for multi-unmanned aerial vehicle cooperative resource optimization, comprising the following steps: modeling a multi-agent joint optimization problem as a partially observable Markov decision model, and converting a joint optimization target into a multi-agent optimal strategy search problem; constructing a hybrid network cooperative decision framework based on adaptive information filtering to output a local action value of each agent; in the process of fusing the local action values of the agents by the hybrid network of the cooperative decision framework, introducing a state-dependent correction term to obtain a global joint action value function; and iteratively training through an experience replay and target network mechanism. The present application enables unmanned aerial vehicles to overcome environmental non-stationarity and observation limitations by enhancing neighborhood information interaction, achieve joint optimization of flight trajectories and task allocation, and improve the performance of unmanned aerial vehicle-assisted computing power networks.

Description

Technical Field

[0001] This invention relates to the field of UAV cooperative control technology, and in particular to a multi-agent reinforcement learning method for optimizing resource cooperation among multiple UAVs. Background Technology

[0002] With the deep integration of computing networks and the low-altitude economy, drones, due to their high mobility and flexible deployment capabilities, are widely used as mobile aerial computing nodes to access distributed computing systems. By incorporating lightweight communication and computing components, drones can effectively compensate for the coverage shortcomings of fixed ground nodes in complex terrains, enabling dynamic deployment and service extension of network nodes. In multi-drone collaborative scenarios, drones can act as local computing nodes to handle lightweight tasks or as relay nodes to offload tasks to ground-based edge computing nodes. However, due to the highly dynamic flight trajectories of drones, limited onboard computing resources, and the time-varying load of ground base stations, using fixed flight routes or lacking reasonable task allocation decisions can easily lead to resource contention and load imbalance. Therefore, how to coordinate the flight trajectories and task allocation decisions of multiple drones to optimize the overall computing resource allocation of the system has become a key technical problem that urgently needs to be solved.

[0003] To address the aforementioned problems, traditional static planning methods rely on precise mathematical models and pre-defined routes. However, in practical applications, the highly dynamic environment makes it difficult to obtain prior information, construct precise mathematical models, and adapt to real-time changes in network topology and load conditions. In contrast, Deep Reinforcement Learning (DRL), with its online learning and adaptive decision-making capabilities, can iteratively approach the optimal policy in dynamic, high-dimensional environments. Furthermore, while distributed deep reinforcement learning improves system scalability, the independent decision-making of each agent makes it susceptible to environmental non-stationarity, leading to learning instability and difficulty in achieving global collaborative optimality. Therefore, a multi-agent reinforcement learning method based on a "Centralized Training with Decentralized Execution" (CTDE) architecture has been proposed, attempting to combine the advantages of global coordination and distributed execution.

[0004] However, existing CTDE methods still have significant limitations in practical applications. First, during the execution phase, agents can only rely on local observation information to make decisions, unable to know the real-time state and intentions of other agents in the neighborhood, lacking an effective information exchange mechanism, which severely restricts the efficiency of multi-agent collaboration. Second, although existing communication-assisted multi-agent reinforcement learning methods attempt to introduce inter-agent communication, they are often limited by the incompleteness of local observations and the redundancy of communication content, easily leading to value decomposition bias and decision conflicts, which in turn reduces the accuracy of task allocation and convergence speed. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a multi-agent reinforcement learning method for resource optimization in multi-UAV collaborative operations. This method enables UAVs to overcome environmental non-stationarity and observation limitations by enhancing neighborhood information interaction, thereby achieving joint optimization of flight trajectory and task allocation and improving the performance of UAV-assisted computing networks.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A multi-agent reinforcement learning method for resource optimization in multi-UAV cooperative operation includes the following steps:

[0008] S1. The multi-agent joint optimization problem is modeled as a partially observable Markov decision model, and the joint optimization objective is transformed into a multi-agent optimal policy search problem.

[0009] S2. Construct a hybrid network collaborative decision-making framework based on adaptive information filtering. Through a temporal coding module, a policy prior module, and an information fusion module, obtain the agent's local temporal information vector, policy preference representation vector, and neighborhood information vector. Through an adaptive information fusion mechanism, weighted filtering of the neighborhood information vector is performed and fused with the policy preference representation vector to output the local action value of each agent.

[0010] S3. In the process of fusing the local action values ​​of each agent in the hybrid network of the collaborative decision-making framework, a state-dependent correction term is introduced to make a state-related correction to the global joint value in order to obtain the global joint action value function.

[0011] S4. The collaborative decision-making framework is iteratively trained through experience replay and target network mechanism, and the algorithm is converged through soft parameter update to obtain the optimal decision-making strategy for multi-agent collaboration.

[0012] Furthermore, in step S1, the construction of the partially observable Markov decision model includes defining a local observation space, defining a joint action space, and defining a shared reward function;

[0013] The defined local observation space includes the agent's own position, velocity, task data volume, location information of neighboring drones and base stations, link transmission rate, and computing load ratio of each node.

[0014] The joint action space is defined by the UAV's heading angle, normalized flight speed, and task allocation strategy.

[0015] The shared reward function is defined based on task allocation success rate, drone load fairness, base station load fairness, task processing latency, and drone collision penalty, and the shared reward function is:

[0016]

[0017] The system's complete simulation time is divided into several time slots, using... express, For a certain time slot; For the system in time slots Task assignment success rate; For the system in time slots The drone payload fairness index; For the system in time slots The computing load fairness index is a value that approaches 1 when the cumulative load of all drones or base stations is equal, indicating a high degree of load fairness. For the system in time slots Total task latency; As a penalty for collision, when the drone When a collision occurs, Otherwise, it is 0.

[0018] Furthermore, all drone agents share the same optimization objective, and maximize the cumulative team reward as follows:

[0019]

[0020] in, As a discount factor, For shared reward functions; The total number of time slots for one complete simulation; The total number of drones; For a single drone intelligent agent;

[0021] Furthermore, the multi-agent optimal policy search problem is the optimal policy under the MARL framework. Search question:

[0022]

[0023] in, For strategy; For time slots The reward function for a single agent within the system.

[0024] Further, in step S2, the timing coding module encodes the historical observation sequence of the agent through a gated loop unit to generate a hidden state containing timing information, and maps the hidden state into an exchangeable information vector through a multilayer perceptron and sends it to the neighboring agent.

[0025] The policy prior module and the temporal coding module share the GRU network, and the policy prior module generates a policy preference representation vector independent of neighborhood information based on local observations, which serves as the prior experience of the agent's own decision preferences.

[0026] The information fusion module calculates the weight coefficients of the neighborhood information vector through an attention mechanism, aggregates neighborhood information of different importance levels, and then concatenates the aggregated neighborhood information vector with the local hidden state and the policy preference representation vector before inputting it into the local Q network to generate the local action value of each agent.

[0027] Further, in step S2, the neighborhood information vector weights of each agent are calculated using a multi-head attention mechanism. The neighborhood information is then weighted and aggregated according to these weights to obtain a filtered collaborative representation. The attention weights are calculated as follows:

[0028]

[0029] in, For intelligent agents The query vector; For neighborhood intelligent agents The key vector; The dimension of the attention vector; It is the neighborhood set of the agent.

[0030] Further, in step S3, the hybrid network receives the local action values ​​and global states of all agents as inputs, and fuses them through a nonlinear transformation that satisfies monotonicity constraints; at the same time, a state-dependent residual branch is introduced, and the state-dependent residual branch takes the global state as input, extracts nonlinear features through a multi-layer MLP, outputs a scalar residual term, and adds the output of the hybrid network to the scalar residual term to obtain the final global joint action value.

[0031] Furthermore, in step S3, the value of the global joint action is:

[0032]

[0033] in, For intelligent agents in time slots The state space; For intelligent agents in time slots The action; Output of a monotonic mixed network; This refers to the state residual term.

[0034] Furthermore, in step S4, the iterative training steps include:

[0035] S4.1 Initialize the environment, network parameters, and experience replay buffer;

[0036] S4.2 In each training round, the agent selects actions according to the current policy and greedy algorithm, interacts with the environment, and collects sample data asynchronously and stores it into the experience replay buffer;

[0037] When the sample size of the experience replay buffer in S4.3 exceeds the preset threshold, a small batch of samples is randomly sampled for centralized training.

[0038] S4.4 Calculate the temporal difference objective value and loss function, and update the main network parameters through the Adam optimizer;

[0039] S4.5 The target network parameters are slowly updated using a soft update strategy to ensure the stability of the training process;

[0040] S4.6 Repeat the above steps until the algorithm converges to obtain the optimal decision-making strategy for multi-agent collaboration.

[0041] Further, in step S4.4, for each sample in the batch, the time-series difference target value is calculated:

[0042]

[0043] in, For shared reward functions; Discount factor; These are the target network parameters. These are the system global state and joint actions for the next time slot, respectively.

[0044] The deviation between the current joint action value and the temporal difference target value is calculated using the mean squared error to obtain the loss function, which is:

[0045]

[0046] in, These are samples randomly sampled from the buffer. This is valuable for current joint operations.

[0047] The beneficial effects of this invention are:

[0048] This invention constructs a hybrid network collaborative decision-making framework with adaptive information filtering. It introduces an adaptive information filtering and selection mechanism into multi-UAV collaborative decision-making, enabling each agent to weighted aggregate neighborhood information based on an attention mechanism under limited local observation conditions. This significantly enhances the perception of the global situation, effectively alleviating the information exchange bottleneck under a centralized training-distributed execution architecture and improving global perception under local observation. An independent policy prior module generates a policy preference representation vector independent of neighborhood information. In the early stages of training, when neighborhood communication information is unstable and message noise is high, this effectively compensates for the negative impact of random initialization of policy network parameters, accelerating training convergence, improving algorithm learning efficiency, and mitigating information noise interference in the early training phase. A state-dependent residual branch with global state as input is introduced into the hybrid network. Nonlinear features are extracted through a multi-layer MLP, and the output scalar residual term performs state-related corrections on the global value, explicitly capturing the inherent value baseline of the environmental state. This overcomes the limitations of the function expression space imposed by the monotonicity constraint of the traditional QMIX algorithm, improving the fitting accuracy and gradient propagation efficiency of the value function in complex environments. The information weights of each neighboring agent are calculated based on the multi-head attention mechanism. The neighboring information is aggregated in a differentiated weighted manner according to different importance levels, which effectively filters out low-value or conflicting communication information and reduces bandwidth requirements and decision conflict risks. Attached Figure Description

[0049] Figure 1 This is a flowchart of a preferred embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram illustrating the working principle of a collaborative decision-making framework according to a preferred embodiment of the present invention.

[0051] Figure 3 This is an algorithm flowchart of a preferred embodiment of the collaborative decision-making framework of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] It should be noted that when a component is described as "fixed to" another component, it can be directly on the other component or may have a component in between. When a component is considered "connected to" another component, it can be directly connected to the other component or may have a component in between. When a component is considered "set on" another component, it can be directly set on the other component or may have a component in between. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0055] Please also see Figures 1 to 3 A preferred embodiment of the present invention provides a multi-agent reinforcement learning method for resource optimization in multi-UAV cooperative operation, comprising the following steps:

[0056] S1. The multi-agent joint optimization problem is modeled as a partially observable Markov decision model, and the joint optimization objective is transformed into a multi-agent optimal policy search problem.

[0057] In step S1, the construction of the partially observable Markov decision model includes defining the local observation space, defining the joint action space, and defining the shared reward function.

[0058] This embodiment models the joint problem of multi-UAV trajectory optimization, task allocation, and computing resource allocation optimization as a partially observable Markov decision model, and realizes the collaborative decision-making process of UAV swarms under local observation constraints. First, the environment and the states of each agent are initialized, and the local observation space is defined. Joint Action Space and shared reward function In each time slot Each agent, based on its current local observations Generate Actions After execution, update the environment status and calculate the immediate reward. By using experience replay and target network mechanisms, the optimal collaborative strategy is learned iteratively, and finally a joint optimization strategy that takes into account task allocation success rate, load balancing and task latency is output.

[0059] The local observation space is defined as including the agent's own position, velocity, task data volume, location information of neighboring drones and base stations, link transmission rate, and computing load ratio of each node.

[0060] In this embodiment, the intelligent agent Local observation state space The signal range of drones is limited; this embodiment considers the distance from the [missing information]. The drone is within effective communication range The nearest Each base station and The information of each drone is represented by formula (1):

[0061]

[0062] in, , Local observation space includes drones and ground base stations 3D coordinates and 3D coordinates of neighboring drones and neighboring base stations and ; The amount of data for the task; This refers to the link transmission rate; and To ensure a fair load ratio between drones and base stations.

[0063] The joint action space is defined by the UAV's heading angle, normalized flight speed, and task allocation strategy.

[0064] In this embodiment, an intelligent agent is designed. Action space , represents the set of all actions that the agent can choose to perform, as expressed by formula (2):

[0065]

[0066] Among them, heading angle This indicates the drone's flight direction in the horizontal plane; the drone's actual flight speed is the normalized velocity. Maximum flight speed of drones The product; This indicates the allocation decision of drones, if Then it means the first Each drone will assign tasks to nearby base stations for execution. This indicates that the drone has chosen to perform its mission locally.

[0067] In multi-UAV collaborative scenarios, the base station, acting as an edge computing node in the computing network, provides limited computing resources to the UAVs. The UAV task allocation success rate reflects the effective utilization of computing resources, the load fairness index characterizes the balance of computing resource allocation, and latency measures the response time of computing services. Therefore, this paper aims to jointly optimize the average task allocation success rate, maximize the UAV onboard computing load fairness index and the base station computing load fairness index, and minimize the total task latency. A shared reward function is defined based on the task allocation success rate, UAV load fairness, base station load fairness, task processing latency, and UAV collision penalty. The shared reward function is:

[0068]

[0069] The system's complete simulation time is divided into several time slots, using... express, For a certain time slot; For the system in time slots Task assignment success rate; For the system in time slots The drone payload fairness index; For the system in time slots The computing load fairness index is a value that approaches 1 when the cumulative load of all drones or base stations is equal, indicating a high degree of load fairness. For the system in time slots Total task latency; As a penalty for collision, when the drone When a collision occurs, Otherwise, it is 0.

[0070] In this embodiment, in the time slot , No. A drone agent, based on its own local observations Choose by strategy Decision action Once all the actions of the intelligent agents have been completed, the drone intelligent agent... Receive rewards Meanwhile, the system status changed from Transfer to For collaborative tasks, all drone agents share the same optimization objective: maximizing cumulative team rewards.

[0071] All drone agents share the same optimization objective, and the cumulative team reward is maximized as follows:

[0072]

[0073] in, As a discount factor, For shared reward functions; The total number of time slots for one complete simulation; The total number of drones; For a single drone intelligent agent.

[0074] In this embodiment, the original optimization problem is transformed into an optimal strategy under the MARL framework. The search problem is a multi-agent optimal policy search problem, specifically an optimal policy search problem within the MARL framework. Search question:

[0075]

[0076] in, For strategy; For time slots The reward function for a single agent within the system.

[0077] S2. Construct a hybrid network collaborative decision-making framework based on adaptive information filtering. Through a temporal coding module, a policy prior module, and an information fusion module, obtain the agent's local temporal information vector, policy preference representation vector, and neighborhood information vector. Through an adaptive information fusion mechanism, the neighborhood information vector is weighted and filtered, and fused with the policy preference representation vector to output the local action value of each agent.

[0078] In step S2, the timing coding module encodes the agent's historical observation sequence through a gated loop unit to generate a hidden state containing timing information, and maps the hidden state into a commutative information vector through a multilayer perceptron and sends it to the neighboring agent.

[0079] The policy prior module and the temporal coding module share the GRU network, and the policy prior module generates a policy preference representation vector based on local observations that is independent of neighborhood information, as the prior experience of the agent's own decision preferences.

[0080] The information fusion module calculates the weight coefficients of the neighborhood information vector through the attention mechanism, aggregates neighborhood information of different importance levels, and then concatenates the aggregated neighborhood information vector with the local hidden state and policy preference representation vector before inputting it into the local Q network to generate the local action value of each agent.

[0081] like Figure 2 As shown, the principles of the collaborative decision-making framework include:

[0082] S2.1 The time-series coding module will use local observations As input, the output of the GRU is in the time slot. Hidden state Then, information vectors are generated through MLP. It is sent to neighboring intelligent agents for collaborative decision-making.

[0083] In this embodiment, each agent extracts local features based on its own local observation information. Taking the first... Taking a drone intelligent agent as an example, its current local observation The input timing encoding module encodes the agent's historical and current observations using a GRU to obtain the hidden state. Based on the hidden state Generate information vectors for communication:

[0084]

[0085] in, For information encoding networks, For intelligent agents The information vector sent to the neighboring agents in the current time slot.

[0086] S2.2 Characterizes the agent's independent decision-making preferences through a policy prior module. This is achieved by sharing local observations with the GRU in the encoding module. As input, the output agent is the intelligent agent. Prior policy representation vector The policy prior module constructs an endogenous decision independent of neighborhood information for each agent as prior experience for the agent's own policy preferences. In the initial training phase, due to the random initialization of the policy network parameters of each agent, the messages generated and transmitted contain a large amount of noise and have high information entropy. Therefore, the policy prior module is introduced to compensate for the negative impact and accelerate the training process.

[0087] S2.3 Merging the neighborhood range through the information fusion module In-situ agent information vector With local information vector Form a set Then As input, neighborhood information is weighted and aggregated using an attention mechanism. Finally, the weighted and aggregated neighborhood information is combined with... Perform concatenation and output the local action value function. The global joint value function is obtained based on the hybrid network output and state residual terms.

[0088] An attention mechanism is used to filter and select information from neighboring agents. The agent determines its information based on its hidden state. Generate a query vector, and generate key and value vectors based on the neighborhood information vector:

[0089]

[0090]

[0091]

[0092] in, , , is a learnable parameter matrix. For drones Within communication range Other agents within. Then calculate the first... An agent interacts with neighboring agents. Attention weights:

[0093] In step S2, the neighborhood information vector weights of each agent are calculated using a multi-head attention mechanism. The neighborhood information is then weighted and aggregated according to these weights to obtain the filtered collaborative representation. The attention weights are calculated as follows:

[0094]

[0095] in, For intelligent agents The query vector; For neighborhood intelligent agents The key vector; Using the attention vector dimension, and then weighting and aggregating the neighborhood information according to the attention weights, we obtain the filtered neighborhood collaborative representation. ; The neighborhood set of the agent; For the first An agent interacts with neighboring agents. Attention weights.

[0096] The policy prior module generates a local policy representation based on the agent's own hidden state. This is used to characterize the agent's local decision preferences when it does not rely on neighborhood information. The local hidden state is obtained. Strategy representation and filtered neighborhood information Then, the three are concatenated and input into the Q network to obtain the first... The local action value of an agent under its current action:

[0097]

[0098] in, Represents a local action value network. These are the local network parameters. To ensure consistency between the optimal individual actions and the globally optimal joint actions, the hybrid network needs to satisfy the following monotonicity constraints:

[0099]

[0100] Under this constraint, the output of the hybrid network is represented as follows:

[0101]

[0102] in, For hybrid networks, These are mixed network parameters.

[0103] S3. In the process of fusing the local action values ​​of each agent in the hybrid network of the collaborative decision-making framework, a state-dependent correction term is introduced to make state-related corrections to the global joint value in order to obtain the global joint action value function.

[0104] In step S3, the hybrid network receives the local action values ​​and global states of all agents as inputs and fuses them through a nonlinear transformation that satisfies monotonicity constraints. At the same time, a state-dependent residual branch is introduced, which takes the global state as input, extracts nonlinear features through a multi-layer MLP, and outputs scalar residual terms. The hybrid network output is added to the scalar residual terms to obtain the final global joint action value.

[0105] In this embodiment, a state residual branch is introduced to receive the global state. Nonlinear feature extraction is performed using a multi-layer fully connected neural network, outputting a scalar residual term. This is used to display the inherent value baseline of capturing environmental states, thereby enhancing the expressive power of the value function and compensating for the limitations of monotonicity constraints on the function space. The final output is the global joint value function. It is obtained by adding the output of the hybrid network to the state residual term.

[0106] In this embodiment, a state residual branch is introduced into the hybrid network to perform state-related corrections on the global joint value. The state residual branch is based on the global state. As input, it is constructed from an MLP. Let the intermediate hidden representation of the residual branch be:

[0107]

[0108]

[0109] The state residual term can be expressed as:

[0110]

[0111] in, This represents the trainable parameters in the residual branch. Represents a non-linear activation function. This indicates the number of network layers in the residual branch. This is a scalar state correction term.

[0112] In step S3, the global joint action value is:

[0113]

[0114] in, For intelligent agents in time slots The state space; For intelligent agents in time slots The action; Output of a monotonic mixed network; This refers to the state residual term.

[0115] S4. The collaborative decision-making framework is iteratively trained through experience replay and target network mechanisms, and the algorithm is converged through soft parameter updates to obtain the optimal decision-making strategy for multi-agent collaboration.

[0116] In step S4, the iterative training steps include:

[0117] S4.1 Initialize the environment, network parameters, and experience playback buffer.

[0118] S4.2 In each training round, the agent selects actions based on the current policy and a greedy algorithm, interacts with the environment, and collects sample data asynchronously and stores it in the experience replay buffer.

[0119] S4.3 When the sample size of the experience replay buffer exceeds the preset threshold, a small batch of samples is randomly sampled for centralized training.

[0120] S4.4 Calculate the temporal difference objective value and loss function, and update the main network parameters through the Adam optimizer.

[0121] In step S4.4, for each sample in the batch, the time-series difference target value is calculated:

[0122]

[0123] in, For shared reward functions; Discount factor; These are the target network parameters. These are the system global state and joint actions for the next time slot, respectively.

[0124] The deviation between the current joint action value and the temporal difference target value is calculated using the mean squared error, and the loss function is obtained as follows:

[0125]

[0126] in, These are samples randomly sampled from the buffer. This is valuable for current joint operations.

[0127] This embodiment utilizes the Adam optimizer to backpropagate the loss function while simultaneously updating the network parameters:

[0128]

[0129] in, Indicates the learning rate. This represents the gradient of the loss function with respect to the parameters of the main network.

[0130] S4.5 The target network parameters are slowly updated using a soft update strategy to ensure the stability of the training process.

[0131] In step S4.5, the target network parameters are updated using a soft update strategy to obtain... .

[0132]

[0133] in, To meet The constant.

[0134] S4.6 Repeat the above steps until the algorithm converges to obtain the optimal decision-making strategy for multi-agent collaboration.

[0135] In this embodiment, the multi-UAV cooperative task allocation and resource optimization problem is modeled as a distributed partially observable Markov decision process. Considering the constraints of local observation and limited communication range of UAVs, a local observation space is defined, including information such as position, speed, task load, link quality, and base station load. A joint action space is defined, including heading angle, speed, and task execution mode. A shared reward function is designed that integrates task allocation success rate, dual-end load fairness, task latency, and collision penalty.

[0136] Design a communication-enhanced multi-agent reinforcement learning framework. This framework allows UAV agents to filter encoded information with neighboring agents within communication range. Each agent uses a GRU to temporally encode historical and current observations, generating an information vector for broadcast. An attention mechanism is introduced to adaptively filter received neighbor information, calculating the importance weights of different neighbors for the current decision, and weighted aggregation to obtain a collaborative representation, thereby enhancing the agent's ability to perceive the global situation.

[0137] A QMIX algorithm based on information filtering, AIFQMIX, is proposed as the core decision-making algorithm. This algorithm comprises three modules: a temporal coding module for extracting temporal features of observations; a policy prior module for generating local decision preferences independent of neighborhood information to mitigate the impact of information noise in the initial training phase; and an information fusion module for concatenating the local hidden state, prior policy, and filtered neighborhood information, outputting the local action value via an MLP. The hybrid network fuses the local values ​​of each agent while satisfying monotonicity constraints, and introduces a residual branch with the global state as input, outputting a scalar correction term to obtain a more accurate estimate of the global joint action value.

[0138] Compared with existing technologies, this embodiment introduces an adaptive information filtering and selection mechanism in multi-UAV collaborative decision-making, significantly improving the global perception capability under local observation conditions. By introducing state-dependent residual terms into the hybrid network, it overcomes the monotonicity limitation of the traditional QMIX method, improves the fitting accuracy of the value function in complex environments, and can effectively improve the task allocation success rate and load balancing of multi-UAV collaborative systems while reducing task processing latency.

Claims

1. A multi-agent reinforcement learning method for resource optimization in multi-UAV collaborative operation, characterized in that, Includes the following steps: S1. The multi-agent joint optimization problem is modeled as a partially observable Markov decision model, and the joint optimization objective is transformed into a multi-agent optimal policy search problem. S2. Construct a hybrid network collaborative decision-making framework based on adaptive information filtering. Through a temporal coding module, a policy prior module, and an information fusion module, obtain the agent's local temporal information vector, policy preference representation vector, and neighborhood information vector. Through an adaptive information fusion mechanism, weighted filtering of the neighborhood information vector is performed and fused with the policy preference representation vector to output the local action value of each agent. S3. In the process of fusing the local action values ​​of each agent in the hybrid network of the collaborative decision-making framework, a state-dependent correction term is introduced to make a state-related correction to the global joint value in order to obtain the global joint action value function. S4. The collaborative decision-making framework is iteratively trained through experience replay and target network mechanism, and the algorithm is converged through soft parameter update to obtain the optimal decision-making strategy for multi-agent collaboration.

2. The multi-agent reinforcement learning method for multi-UAV cooperative resource optimization according to claim 1, characterized in that: In step S1, the construction of the partially observable Markov decision model includes defining the local observation space, defining the joint action space, and defining the shared reward function; The defined local observation space includes the agent's own position, velocity, task data volume, location information of neighboring drones and base stations, link transmission rate, and computing load ratio of each node. The joint action space is defined by the UAV's heading angle, normalized flight speed, and task allocation strategy. The shared reward function is defined based on task allocation success rate, drone load fairness, base station load fairness, task processing latency, and drone collision penalty, and the shared reward function is: The system's complete simulation time is divided into several time slots, using... express, For a certain time slot; For the system in time slots Task assignment success rate; For the system in time slots The drone payload fairness index; For the system in time slots The computing load fairness index is a value that approaches 1 when the cumulative load of all drones or base stations is equal, indicating a high degree of load fairness. For the system in time slots Total task latency; As a collision penalty, when the drone When a collision occurs, Otherwise, it is 0.

3. The multi-agent reinforcement learning method for multi-UAV cooperative resource optimization according to claim 2, characterized in that: All drone agents share the same optimization objective, and the cumulative team reward is maximized as follows: in, As a discount factor, For shared reward functions; The total number of time slots for one complete simulation; The total number of drones; For a single drone intelligent agent.

4. The multi-agent reinforcement learning method for resource optimization of multiple UAVs according to claim 3, characterized in that: The multi-agent optimal policy search problem is the optimal policy under the MARL framework. Search question: in, For strategy; For time slots The reward function for a single agent within the system.

5. The multi-agent reinforcement learning method for resource optimization of multiple UAVs according to claim 1, characterized in that: In step S2, the timing coding module encodes the historical observation sequence of the agent through a gated loop unit to generate a hidden state containing timing information, and maps the hidden state into a commutative information vector through a multilayer perceptron and sends it to the neighboring agent. The policy prior module and the temporal coding module share the GRU network, and the policy prior module generates a policy preference representation vector independent of neighborhood information based on local observations, which serves as the prior experience of the agent's own decision preferences. The information fusion module calculates the weight coefficients of the neighborhood information vector through an attention mechanism, aggregates neighborhood information of different importance levels, and then concatenates the aggregated neighborhood information vector with the local hidden state and the policy preference representation vector before inputting it into the local Q network to generate the local action value of each agent.

6. A multi-agent reinforcement learning method for resource optimization of multiple UAVs according to claim 5, characterized in that: In step S2, the neighborhood information vector weights of each agent are calculated using a multi-head attention mechanism. The neighborhood information is then weighted and aggregated according to these weights to obtain a filtered collaborative representation. The attention weights are calculated as follows: in, For intelligent agents The query vector; For neighborhood intelligent agents The key vector; The dimension of the attention vector; It is the neighborhood set of the agent.

7. A multi-agent reinforcement learning method for resource optimization of multiple unmanned aerial vehicles (UAVs) according to claim 1, characterized in that: In step S3, the hybrid network receives the local action values ​​and global states of all agents as inputs and fuses them through a nonlinear transformation that satisfies monotonicity constraints. At the same time, a state-dependent residual branch is introduced, and the state-dependent residual branch takes the global state as input, extracts nonlinear features through a multi-layer MLP, outputs a scalar residual term, and adds the output of the hybrid network to the scalar residual term to obtain the final global joint action value.

8. A multi-agent reinforcement learning method for resource optimization of multiple UAVs according to claim 5, characterized in that: In step S3, the value of the global joint action is: in, For intelligent agents in time slots The state space; For intelligent agents in time slots The action; Output of a monotonic mixed network; This refers to the state residual term.

9. A multi-agent reinforcement learning method for resource optimization of multiple UAVs according to claim 1, characterized in that: In step S4, the iterative training steps include: S4.1 Initialize the environment, network parameters, and experience replay buffer; S4.2 In each training round, the agent selects actions according to the current policy and greedy algorithm, interacts with the environment, and collects sample data asynchronously and stores it into the experience replay buffer; When the sample size of the experience replay buffer in S4.3 exceeds the preset threshold, a small batch of samples is randomly sampled for centralized training. S4.4 Calculate the temporal difference objective value and loss function, and update the main network parameters through the Adam optimizer; S4.5 The target network parameters are slowly updated using a soft update strategy to ensure the stability of the training process; S4.6 Repeat the above steps until the algorithm converges to obtain the optimal decision-making strategy for multi-agent collaboration.

10. A multi-agent reinforcement learning method for resource optimization of multiple unmanned aerial vehicles (UAVs) according to claim 1, characterized in that: In step S4.4, for each sample in the batch, the time-series difference target value is calculated: in, For shared reward functions; Discount factor; These are the target network parameters. These are the system global state and joint actions for the next time slot, respectively. The deviation between the current joint action value and the temporal difference target value is calculated using the mean squared error to obtain the loss function, which is: in, These are samples randomly sampled from the buffer. This is valuable for current joint operations.

Images