A method and system for coordinating control of a group of unmanned aerial vehicles based on communication information completion

Abstract

The application provides a kind of unmanned aerial vehicle group coordination control method and system based on communication information completion, information weight is calculated based on local observation value and the observation information of other unmanned aerial vehicles through information level weight network, and then the weighted information of unmanned aerial vehicle is obtained, and global state is generated through adaptive generation network.Then, based on global state and local action value function, the local action value function of unmanned aerial vehicle is integrated through hybrid network based on global action value function.In distributed execution phase, unmanned aerial vehicle obtains local action value function through decoder based on Transformer according to local observation value and information weight, and utilizes strategy network to make coordination decision.Compared with prior art, the application realizes information completion through generative adversarial network in training phase, and only relies on local information to make decision in execution phase, which effectively reduces the computing overhead.The method improves the cooperation and decision-making ability of unmanned aerial vehicle group, and has good scalability and application prospect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicle cooperative control, and particularly relates to a method and system for coordinating control of a group of unmanned aerial vehicles based on communication information completion. BACKGROUND

[0002] In recent years, multi-unmanned aerial vehicle reinforcement learning has attracted extensive attention due to its potential for modeling and solving complex cooperative tasks in real-world applications. Unlike single-unmanned aerial vehicle reinforcement learning, multi-unmanned aerial vehicle reinforcement learning faces the challenge of promoting unmanned aerial vehicle cooperation to achieve common goals. The increased cooperation dimension brings many difficulties, as both centralized and distributed methods cannot effectively cope with the non-stationarity and scalability problems generated by concurrent learning of joint strategies as the number of unmanned aerial vehicles increases. Existing multi-unmanned aerial vehicle reinforcement learning algorithms mainly rely on the Centralized Training with Decentralized Execution (CTDE) paradigm. However, this paradigm has certain limitations. In centralized training, the assumption that unmanned aerial vehicles can access the global state is idealistic, while in decentralized execution, the decision of an unmanned aerial vehicle can only rely on its own local observation, which may affect its ability to make decisions and cooperate.

[0003] To improve the decision-making and cooperation ability of unmanned aerial vehicles, more attention has been focused on research into integrating communication mechanisms into the Multi-Agent Reinforcement Learning (MARL) framework. By allowing unmanned aerial vehicles to exchange information such as local observations, beliefs, or intentions, the inference of the global state and coordinated decision-making can be promoted. However, existing methods face three main problems. First, the communication model accuracy of existing methods is not high. These models usually define the communication mode in advance, and communication is carried out through broadcasting or peer-to-peer mechanisms. These methods explicitly define the timing of unmanned aerial vehicle communication and the objects that need to be communicated, but do not delve into the information level. Second, many methods have low computational efficiency. The communication model and strategy network can only be jointly learned through sparse reinforcement learning rewards, and the communication module cannot be separated from the training process, resulting in reduced training efficiency. Finally, communication is usually limited to the decentralized execution phase. In the centralized training process, these methods still rely on access to the global state, and fail to bridge the gap between training and execution through communication. Therefore, there is currently a lack of a plug-in multi-unmanned aerial vehicle reinforcement learning method that can achieve information-level modeling to solve these problems. SUMMARY

[0004] The purpose of the present application is to overcome the deficiencies of the prior art and provide a method and system for coordinating control of a group of unmanned aerial vehicles based on communication information completion.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] This invention provides a method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion, comprising the following steps:

[0007] Step S1: Each UAV acquires local observations from the environment, including environmental information that a single UAV can perceive;

[0008] Step S2: Centralized training phase. Based on the local observations, information weights are obtained through an information-level weighted network. Based on the information weights, weighted information for each UAV is obtained. Based on the weighted information for each UAV, a global state is generated through an adaptive generative network. Based on the global state and the local action value functions of the UAVs, the local action value functions of the UAVs are integrated through a hybrid network to obtain a global action value function. The local action value function is calculated by each UAV using an independent Q-network based on local observations. Based on the global action value function, the UAV swarm interacts with the environment to obtain rewards. Based on the rewards, the parameters of the hybrid network are updated using a first loss function. Based on the information weights, local observations, and global state, the parameters of the adaptive generative network and the information-level weighted network are updated using a generative adversarial network loss function.

[0009] Step S3: Reacquire the information weights based on the updated information-level weight network, and send the acquired information weights to each UAV;

[0010] Step S4: Distributed execution phase. Each UAV obtains its local action value function based on local observations and information weights through a Transformer-based decoder. Based on the local action value function, a policy network is used to coordinate and make decisions to guide the UAV to perform corresponding actions.

[0011] Furthermore, the step of obtaining information weights through an information-level weighted network based on the local observations specifically includes:

[0012] The local observations of UAV i and information obtained from other drones Perform position encoding to convert it into a vector representation containing position information;

[0013] The location-encoded vector is repeatedly processed and converted into the input format of the attention mechanism, including the query, key, and value, and the local observations of UAV i. The location encoding vector is repeatedly processed by a multilayer perceptron (MLP) to generate the query vector. Information obtained by other drones After processing, a key vector is generated. Sum value vector

[0014] Calculate query vector and key vector The attention score is obtained by taking the dot product of the values. The attention score is then normalized using the softmax function to obtain the information weights between each drone. The calculation formula is as follows:

[0015]

[0016] Among them, M t Let represent the set of local observations of all UAVs at time t. Let be the information weight of drone i at time t, d be the dimension of the query vector and the key vector, σ be the ReLU activation function, and m be the number of times the process is repeated.

[0017] Furthermore, based on the information weights, the weighted information of each UAV is obtained using the following formula:

[0018]

[0019] in, The weighted information of drone i at time t. Let i be the information weight of drone i at time t. This represents the complete information acquired by drone i at time t. Let ∈ represent the set of local observations of all drones other than drone i at time t. t It is white noise. It follows a normal distribution, and I is the identity matrix.

[0020] Furthermore, the adaptive generation network includes an information completion network and a global discriminator network. The information completion network is connected to the information weight network. The information completion network adopts a U-Net structure and follows an encoder-decoder structure.

[0021] Furthermore, the generation of the global state based on the weighted information of each UAV through an adaptive generative network specifically includes:

[0022] Weighted information of each drone By downsampling, the downsampled input data is used to complete the encoder and decoder of the network, and the global state is output:

[0023]

[0024] Among them, s t The global state at time t is generated to complete the information in the network, where n is the number of drones in the drone swarm.

[0025] Furthermore, the step of integrating the local action value functions of the UAV based on the global state and the UAV's local action value functions through a hybrid network to obtain the global action value function specifically includes:

[0026] All drones learn through a hybrid network Q tot (τ,a,m,s;θ), utilizing the additivity structure of the QMIX network, the weights of the local action value functions are determined using the global state, and the local action value functions of each UAV are synthesized into a global action value function through the determined weights of the local action value functions.

[0027] Furthermore, the first loss function is:

[0028]

[0029] in, Let n be the number of drones, and let n be the first loss function. Let Q be the target Q value of drone i. tot (τ t ,α t ,m t ,s t ;θ) is the Q-value of the hybrid network at time t, τ t Let α be the historical observation value at time t. t Let m be the action at time t. t For information about the drone swarm at time t, s t Let be the global state at time t, θ be the parameters of the hybrid network, r be the reward obtained by the UAV swarm interacting with the environment, and γ be the discount factor. This represents the action a that maximizes the value. t+1 Q tot value.

[0030] Furthermore, the loss function of the generative adversarial network is:

[0031]

[0032] in, Let $\mathbf{a}$ be the second loss function, $\mathbf{a}$ be the Euclidean norm, and $\mathbf{a}$ be the value of $\mathbf{a}$. t For actual local observation, O(s) t Let ) represent the local observations predicted based on the global state, D be the global discriminator network in the adaptive generative network, G be the information completion network in the adaptive generative network, E represent the expected value, o~O represent o being sampled from distribution O, s~S represent s being sampled from distribution S, and s t Let D(s) be the global state at time t. t To transfer the global state s tThe global state s output by the input global discriminator network t The probability of whether it is true, o t Let be the local observation value at time t, and α be the weight hyperparameter.

[0033] Furthermore, the step of obtaining the local motion value function of the UAV based on the local observations and information weights through a Transformer-based decoder specifically includes:

[0034] Based on local observations and information weights, weighted information of the UAV is obtained. The obtained weighted information and the UAV's hidden state at time step t-1 are fed into the embedding layer. The embedding layer converts the input data into a continuous vector representation. The output of the embedding layer is used as the input of the Transformer-based decoder. The Transformer-based decoder processes the input through a self-attention mechanism, extracts useful features, and outputs the UAV's hidden state at time step t and its local action value function.

[0035] Another aspect of the present invention provides a pluggable multi-UAV reinforcement learning system for communication information completion, comprising:

[0036] Information-level weighted networks are used to process the local observations of each UAV and the information of other UAVs to generate information weights.

[0037] An adaptive generative network includes an information completion network and a global discriminator network. The information completion network is connected to an information-level weight network and generates a global state through an encoder-decoder structure. The global discriminator network is used to determine the authenticity of the global state.

[0038] Hybrid networks are used to integrate the local motion value functions of various UAVs and generate global motion value functions through an additivity structure.

[0039] A Transformer-based decoder is used to calculate the local motion value function of the UAV based on local observations and information weights during the distributed execution phase.

[0040] A policy network is used to generate the final action based on local action-value functions and coordination decisions;

[0041] The loss function module includes a first loss function and a generative adversarial network loss function, which are used to update the parameters of the hybrid network, the adaptive generative network, and the information-level weighted network.

[0042] The communication module is used to transmit weighted information and decision-making information between different drones;

[0043] The environment interaction module is used to simulate the interaction process between the drone and the environment and to collect reward signals.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] (1) Complete communication information and reduce the system’s dependence on accessing the global state: For multi-UAV coordinated control, a pluggable module is used to combine the generator network with the multi-UAV reinforcement learning communication module, allowing UAVs to generate the global state using weighted local observations and make coordinated decisions using these states and the learned communication weights, thereby reducing the system’s dependence on accessing the global state.

[0046] (2) Information-level communication modeling is achieved, reducing the computational requirements associated with joint training of communication and policy networks: pluggable modules can be pre-trained and seamlessly integrated into various MARL algorithms. By fusing generative model loss functions, the training of communication modules is decoupled from sparse reinforcement learning rewards, thereby improving the efficiency of the policy network learning process.

[0047] (3) The pluggable multi-UAV reinforcement learning method for communication information completion of the present invention designs a flexible structure to separate communication and policy training, thereby improving computational efficiency and effectively bridging the gap between centralized training and distributed execution.

[0048] (4) This invention introduces an information-level weighted network during the centralized training phase. Based on the local observations of each UAV and the observation information of other UAVs, it calculates the information weights between each UAV, thereby effectively capturing the information correlation between UAVs. This technique enhances the information exchange and collaboration capabilities between UAVs and overcomes the problem of low accuracy in communication models in traditional methods.

[0049] (5) By introducing generative adversarial networks, this invention can process global information during the centralized training phase, while relying only on local observations and communication information during the distributed execution phase, thereby effectively narrowing the gap between training and execution. This technique improves the system's flexibility and scalability, and overcomes the limitation of centralized training relying on global information in traditional methods. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the pluggable module in this invention;

[0051] Figure 2 This is a schematic diagram of an adaptive generative network;

[0052] Figure 3 This is a system relationship diagram of pluggable modules and methods;

[0053] Figure 4A flowchart illustrating an example of a pluggable multi-UAV reinforcement learning method for communication information completion. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] Example 1:

[0056] This embodiment provides a multi-UAV reinforcement learning method for communication information completion, applied to multi-UAV collaboration. The method specifically includes the following steps:

[0057] Step 1: The UAV acquires local observations of the environment;

[0058] Step 2: Centralized training phase. Based on the local observations, information weights are obtained through an information-level weighted network. Based on the information weights, weighted information for each UAV is obtained. Based on the weighted information for each UAV, a global state is generated through an adaptive generative network. Based on the global state and the local action value functions of the UAVs, the local action value functions of the UAVs are integrated through a hybrid network to obtain a global action value function. The local action value function is calculated by each UAV using an independent Q-network based on its local observations. Based on the global action value function, the UAV swarm interacts with the environment to obtain rewards. Based on the rewards, the parameters of the hybrid network are updated using a first loss function. Based on the information weights, local observations, and global state, the parameters of the adaptive generative network and the information-level weighted network are updated using a generative adversarial network loss function.

[0059] Step 3: Re-acquire the information weights based on the updated information-level weight network, and send the acquired information weights to each UAV;

[0060] Step 4: Distributed execution phase. Based on the local observations and information weights, the local action value function of the UAV is obtained through a Transformer-based decoder. Based on the UAV's policy network, coordinated decision-making is performed using weighted information.

[0061] As a preferred technical solution, the pluggable module of the present invention consists of four parts, specifically, as follows: Figure 1As shown, it includes an information-level weight network for generating information weights, an adaptive generation network for generating global states, a Transformer-based decoder for integrating information and extracting features, and a hybrid network for generating global action value functions.

[0062] As a preferred technical solution, the local observations are provided by UAVs. Specifically, at each time step t, each UAV receives a local observation from the observation function.

[0063] As a preferred technical solution, the information-level weighted network has an attention mechanism. Specifically, it uses the local observations of each UAV as the query vector and the local observations of all UAVs as the key and value.

[0064] As a preferred technical solution, the process of obtaining information weights through an information-level weighted network specifically includes encoding the location of the local observations and then calculating the information-level weights through a dot product attention mechanism.

[0065] As a preferred technical solution, the process of generating the global state through an adaptive generative network specifically includes inputting weighted information into an information completion network, performing information completion modeling through U-Net, and then outputting the generated global state; further, inputting the generated global state into a global discriminator network, fusing it through a cascaded layer using a sigmoid activation function, and outputting the true probability of the global state to evaluate the performance of the information completion network.

[0066] As a preferred technical solution, the aforementioned coordination decision-making process specifically includes the policy network updating the policy through an action value function.

[0067] This invention designs a pluggable multi-UAV reinforcement learning module unit that simultaneously includes functions such as communication, information completion, decision-making, and adaptive weights for communication information completion. The module supports seamless integration into multi-UAV reinforcement learning networks, providing support for the collaborative control of multi-UAV systems and enhancing UAV communication and decision-making.

[0068] like Figure 1 The pluggable multi-UAV reinforcement learning module of the present invention includes an information-level weight network, an adaptive generative network, a Transformer-based decoder, a hybrid network, UAVs, and the environment.

[0069] Information-level weight network 2 is used to receive local observations from the UAV through the network's communication component, generate information-level weights during the centralized training phase, and input these weights into the adaptive generative network. The information-level weight network also obtains weights from the generative model through the network's communication component, performs weighted extraction of information during the distributed execution phase, filters redundant information, and inputs the weighted extracted information into the UAV's policy network.

[0070] The communication information input to the information-level weighted network consists of local observations from each UAV. After feature extraction, these local observations can be represented as a one-dimensional vector of length l, denoted as... in Representing the observed values, in a fully communicative scenario, the complete information acquired by each UAV at time t is defined as... The symbol - represents all other drones except drone i. The weighted information includes the communication time, target, and content, as well as the weight. The formula for calculating the weighted information is shown in formula (1):

[0071]

[0072] in, The weighted information of drone i at time t. Let i be the information weight of drone i at time t. This represents the complete information acquired by drone i at time t. Let ∈ represent the set of local observations of all drones other than drone i at time t. t It is white noise. The distribution follows a normal pattern, and I is the identity matrix. The information-level weighted network is jointly trained with the adaptive generative network during the centralized training phase, and their parameters are updated together.

[0073] The Adaptive Generative Network (AGN) receives local observations from the UAV through its communication component. An information-level weight network assists in generating information-level weights, which are then jointly trained to complete the communication information and output the generated global state. This global state serves as input to the hybrid network component, enhancing the expressive power of the global action-value function. The AGN comprises n input channels and one output channel, integrating the UAV's local observations into the communication process, ensuring scalability. During the centralized training phase, the AGN uses MSE loss and GAN loss to update its parameters.

[0074] The hybrid network aggregates the local action value functions of individual UAVs into a global action value function to guide the overall strategy iteration of the multi-UAV system. The input to the hybrid network is the local action value function of each UAV and the global state output from the adaptive generative network. The output is the global action value function, which evaluates the overall value of the multi-UAV system taking a certain cooperative action in a given state. The UAV swarm interacts with the environment based on the global action value function and receives rewards, updating the hybrid network parameters based on these rewards.

[0075] like Figure 2 The adaptive generative network module of the present invention includes an information completion network and a global discriminator network, wherein the information completion network is connected to an information-level weight network.

[0076] The information completion network receives local observations from the UAV and weighted information generated by the information-level weighted network through the network's communication component, completes the information, and generates the global state. The information completion network consists of repeated one-dimensional convolutional residual blocks, uses U-Net for information completion modeling, and follows an encoder-decoder structure. It reduces memory usage and computation time by downsampling before further information processing, and then upsampling to restore the output to the length of the global state.

[0077] The role of the global discriminator network is to distinguish whether the global state is real or generated from local observations. Based on a one-dimensional convolutional neural network, the global state is compressed into a compact feature vector. The network's output is fused through a connection layer that uses a sigmoid activation function to predict a continuous value between 0 and 1, representing the probability that the global state corresponds to the true situation rather than being a result of information completion.

[0078] Specifically, the steps for obtaining information weights through the information-level weighted network are as follows: Input information, i.e., the local observations of the UAV, is obtained through the communication part of the pluggable module; the input information is position-encoded, converting the input data into a vector representation containing position information; the position-encoded vector is repeatedly processed to convert it into the input format of the attention mechanism (including query, key, and value); the attention mechanism generates attention weights by calculating dot product attention; where the query of the attention mechanism is the local observation. The calculation formula is shown in formula (2):

[0079]

[0080] in, Let t be the i-th query at time t, MLP be a multilayer perceptron, and m be the number of times the query is repeated.

[0081] The local observations of all UAVs are used as keys and values, and the calculation formula is shown in formula (3):

[0082]

[0083] Among them, M t Let represent the set of local observations of all UAVs at time t.

[0084] The formula for calculating the information level weight is shown in formula (4):

[0085]

[0086] Where d is the dimension of the query vector and the key vector, σ is the ReLU activation function, and softmax is a normalization method. Let be the information weight of drone i at time t.

[0087] Specifically, the steps for generating the global state through the adaptive generative network are as follows: Input information is obtained from the communication part of the pluggable module, including local observations and information-level weights of the UAV; before further processing, preprocessing is performed by downsampling to reduce memory usage and improve computational efficiency; the preprocessed information is then upsampled to restore the length of the global state; the weighted information is input into the information completion network to achieve information completion, and the generated global state is sent to the global discriminator network. The formula for calculating the weighted information is shown in formula (1), and the output global state is mathematically defined as:

[0088]

[0089] Among them, s t The global state at time t is generated to complete the information in the network, where n is the number of drones in the drone swarm.

[0090] Specifically, the steps to determine whether the global state is real are as follows: input information, including the generated global state, into the communication part of the adaptive network; extract local features from the input information through a one-dimensional convolutional neural network, and then perform pooling to compress the global state into a compact feature vector; the output is fused through a cascaded layer using the sigmoid activation function, and the output global state corresponds to the probability of the real situation rather than being generated by information completion, thereby evaluating the performance of the adaptive generative network, and improving the performance of the adaptive generative network through generative adversarial training.

[0091] Specifically, the steps to obtain an approximate global action value function involve inputting information from the communication section of the pluggable module, including the local action value function of each UAV and the generated global state; during the centralized training phase, all UAVs learn from the hybrid network Q. tot(τ,a,m,s;θ), utilizing the additivity structure of the QMIX network, the weights of the local action-value functions are determined using the global state. These weights are then used to synthesize the UAV's local action-value functions into a global action-value function Q. tot (s,a), where the parameter θ is updated by minimizing the expected temporal difference (TD) error:

[0092]

[0093] in, For TD error, Let Q be the target Q value. tot (τ t ,α t ,m t ,s t ;θ) is the Q-value of the hybrid network at time t, τ t Let α be the historical observation value at time t. t Let m be the action at time t. t For information about the drone swarm at time t, s t Let t be the global state at time t.

[0094] The calculation process is shown in formula (7):

[0095]

[0096] Where, θ - Here are the parameters of the target hybrid network, r is the reward obtained by performing the action, and γ is the discount factor. This represents the action a that maximizes the value. t+1 Q tot value.

[0097] Specifically, the steps for obtaining an approximate local action value function are as follows: inputting information from the communication section of the pluggable module, including local observations of the UAV; during the distributed execution phase, the UAV learns a Q-network Q. i (τ,a,m,s), thus estimating the action value function Q. i (s,a).

[0098] Specifically, this invention uses two loss functions in the process of simultaneously training the adaptive generative network and the information-level weighted network and updating their parameters: the MSE loss for stability and the GAN loss for enhancing the realism of the results. Combining these two loss functions allows the generative network to be trained adversarially against the discriminator network, and the optimization method becomes:

[0099]

[0100] in, Let $\mathbf{a}$ be the second loss function, $\mathbf{a}$ be the Euclidean norm, and $\mathbf{a}$ be the value of $\mathbf{a}$. t For actual local observation, O(s) t Let ) represent the local observations predicted based on the global state, D be the global discriminator network in the adaptive generative network, G be the information completion network in the adaptive generative network, E represent the expected value, o~O represent o being sampled from distribution O, s~S represent s being sampled from distribution S, and s t Let D(s) be the global state at time t. t To transfer the global state s t The global state s output by the input global discriminator network t The probability of whether it is true, o t Let be the local observation value at time t, and α be the weight hyperparameter.

[0101] Example 2:

[0102] like Figure 3 , Figure 4 The multi-rotor drone is a highly stable and expandable six-rotor industrial drone with an all-carbon fiber fuselage. This embodiment will use this multi-rotor drone as the drone and implement it based on the technical solution of this invention.

[0103] The pluggable multi-drone reinforcement learning module distinguishes between two task execution scenarios for multi-rotor drones based on different execution strategies, providing support for full-process logical analysis and diagnosis. The specific details are as follows:

[0104] Scenario 1: When a multi-rotor drone needs to perform a task that allows access to the global state.

[0105] Scenario 2: When a multi-rotor drone needs to perform a task in which the global state is inaccessible.

[0106] like Figure 4 As shown in Figure a, this is a flowchart illustrating a task that a multi-rotor UAV needs to perform to access the global state. Figure 4 Figure b is a flowchart illustrating a task performed by a multi-rotor UAV that requires access to an inaccessible global state. When the multi-rotor UAV faces situation 1, the specific steps for performing the relevant task using pluggable modules and multi-UAV reinforcement learning methods include:

[0107] Step S101: Build an integrated equipment platform including a UAV control system, and deploy sensors and a pluggable multi-UAV reinforcement learning network module for communication information completion.

[0108] Step S102: The UAV control system issues collaborative task instructions for the multi-rotor UAV swarm to the multi-UAV system;

[0109] Step S103: Input the mission commands of the UAV control system and the data captured by the sensors into the pluggable module through the wireless receiving device, and let the pluggable module analyze the data;

[0110] Step S104: Process all input information through an information-level weighted network; input the processed data and the global state of the system into the UAV's policy network, determine the current actions of the multi-rotor UAV cluster according to the policy function, and send the action policy to the UAV control system.

[0111] Step S105: The UAV control system receives the action strategy issued by the pluggable module and evaluates and analyzes it. If the current action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV control system, the multi-rotor UAV swarm is allowed to execute the current action by default. If the current action of the multi-rotor UAV swarm does not meet the task command requirements issued by the UAV control system, the UAV control system will send a command to stop the current action of the multi-rotor UAV swarm and require the reinforcement learning network module to collect all data again and repeat steps S3 to S5 until the action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV swarm control system, and then stop the process.

[0112] When a multi-rotor drone faces situation 2, the specific steps for performing the relevant task using pluggable modules and multi-drone reinforcement learning methods include:

[0113] Step S201: Build an integrated equipment platform including a UAV control system, and deploy sensors and a pluggable multi-UAV reinforcement learning network module for communication information completion.

[0114] Step S202: The UAV control system issues collaborative task instructions for the multi-rotor UAV swarm to the multi-UAV system;

[0115] Step S203: Input the mission commands of the UAV control system and the data captured by the sensors into the pluggable module through the wireless receiving device, and let the pluggable module analyze the current situation;

[0116] Step S204: Process all input information through the information-level weight network, and pass the processed data and information-level weights as input to the adaptive generation network; complete the information through the adaptive generation network and generate the global state of the current multi-rotor UAV cluster, pass the generated global state and input information as input to the UAV's policy network, determine the current actions of the multi-rotor UAV cluster according to the policy function, and send the action policy to the UAV control system.

[0117] Step S205: The UAV control system receives the action strategy issued by the pluggable module and evaluates and analyzes it. If the current action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV control system, the multi-rotor UAV swarm is allowed to execute the current action by default. If the current action of the multi-rotor UAV swarm does not meet the task command requirements issued by the UAV control system, the UAV control system will send a command to stop the current action of the multi-rotor UAV swarm and require the reinforcement learning network module to collect all data again and repeat steps S3 to S5 until the action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV swarm control system, and then stop the process.

[0118] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion, characterized in that, Includes the following steps: Step S1: Each UAV acquires local observation values ​​from the environment, which are obtained in real time through sensors installed on the UAV. Step S2: Centralized training phase. Based on the local observations, information weights are obtained through an information-level weighted network. Based on the information weights, weighted information for each UAV is obtained. Based on the weighted information for each UAV, a global state is generated through an adaptive generative network. Based on the global state and the local action value functions of the UAVs, the local action value functions of the UAVs are integrated through a hybrid network to obtain a global action value function. The local action value function is calculated by each UAV using an independent Q-network based on local observations. Based on the global action value function, the UAV swarm interacts with the environment to obtain rewards. Based on the rewards, the parameters of the hybrid network are updated using a first loss function. Based on the information weights, local observations, and global state, the parameters of the adaptive generative network and the information-level weighted network are updated using a generative adversarial network loss function. Step S3: Reacquire the information weights based on the updated information-level weight network, and send the acquired information weights to each UAV; Step S4: Distributed execution phase. Each UAV obtains its local action value function based on local observations and information weights through a Transformer-based decoder. Based on the local action value function, a policy network is used to coordinate and make decisions to guide the UAV swarm to perform corresponding actions.

2. The method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion according to claim 1, characterized in that, The step of obtaining information weights based on the local observations through an information-level weighted network specifically includes: The local observations of UAV i and information obtained from other drones Perform position encoding to convert it into a vector representation containing position information; The location-encoded vector is repeatedly processed and converted into the input format of the attention mechanism, including the query, key, and value, and the local observations of UAV i. The location encoding vector is repeatedly processed by a multilayer perceptron (MLP) to generate the query vector. Information obtained by other drones After processing, a key vector is generated. Sum value vector Calculate query vector and key vector The attention score is obtained by taking the dot product of the values. The attention score is then normalized using the softmax function to obtain the information weights between each drone. The calculation formula is as follows: Among them, M t Let represent the set of local observations of all UAVs at time t. Let be the information weight of drone i at time t, d be the dimension of the query vector and the key vector, σ be the ReLU activation function, and m be the number of times the process is repeated.

3. The method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion according to claim 1, characterized in that, The weighted information of each UAV is obtained based on the information weights, using the following formula: in, The weighted information of drone i at time t. Let i be the information weight of drone i at time t. This represents the complete information acquired by drone i at time t. Let ∈ represent the set of local observations of all drones other than drone i at time t. t It is white noise. It follows a normal distribution, and I is the identity matrix.

4. The method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion according to claim 1, characterized in that, The adaptive generative network includes an information completion network and a global discriminator network. The information completion network is connected to the information weight network. The information completion network adopts a U-Net structure and follows an encoder-decoder structure.

5. A method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion as described in claim 1 or 4, characterized in that, The process of generating a global state based on weighted information from each UAV through an adaptive generative network specifically includes: Weighted information of each drone By downsampling, the downsampled input data is used to complete the encoder and decoder of the network, and the global state is output: Among them, s t The global state at time t is generated to complete the information in the network, where n is the number of drones in the drone swarm.

6. The method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion according to claim 1, characterized in that, The process of integrating the UAV's local action value functions based on the global state and the UAV's local action value functions through a hybrid network to obtain the global action value function specifically includes: All drones learn through a hybrid network Q tot (τ,a,m,s;θ), utilizing the additivity structure of the QMIX network, the weights of the local action value functions are determined using the global state, and the local action value functions of each UAV are synthesized into a global action value function through the determined weights of the local action value functions.

7. The method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion according to claim 1, characterized in that, The first loss function is: in, Let n be the number of drones, and let n be the first loss function. Let Q be the target Q value of drone i. tot (τ t ,α t ,m t ,s t ;θ) is the Q-value of the hybrid network at time t, τ t Let α be the historical observation value at time t. t Let m be the action at time t. t For information about the drone swarm at time t, s t Let be the global state at time t, θ be the parameters of the hybrid network, r be the reward obtained by the UAV swarm interacting with the environment, and γ be the discount factor. This represents the action a that maximizes the value. t+1 Q tot value.

8. The method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion according to claim 1, characterized in that, The loss function of the generative adversarial network is: in, Let $\mathbf{a}$ be the second loss function, $\mathbf{a}$ be the Euclidean norm, and $\mathbf{a}$ be the value of $\mathbf{a}$. t For actual local observation, O(s) t Let ) represent the local observations predicted based on the global state, D be the global discriminator network in the adaptive generative network, G be the information completion network in the adaptive generative network, E represent the expected value, o~O represent o being sampled from distribution O, s~S represent s being sampled from distribution S, and s t Let D(s) be the global state at time t. t To transfer the global state s t The global state s output by the input global discriminator network t The probability of whether it is true, o t Let be the local observation value at time t, and α be the weight hyperparameter.

9. The method for coordinated control of unmanned aerial vehicle (UAV) swarms based on communication information completion according to claim 1, characterized in that, The process of obtaining the local motion value function of the UAV based on the local observations and information weights through a Transformer-based decoder specifically includes: Based on local observations and information weights, weighted information of the UAV is obtained. The obtained weighted information and the UAV's hidden state at time step t-1 are fed into the embedding layer. The embedding layer converts the input data into a continuous vector representation. The output of the embedding layer is used as the input of the Transformer-based decoder. The Transformer-based decoder processes the input through a self-attention mechanism, extracts useful features, and outputs the UAV's hidden state at time step t and its local action value function.

10. A coordinated control system for unmanned aerial vehicle (UAV) swarms based on communication information completion, characterized in that, include: Information-level weighted networks are used to process the local observations of each UAV and the information of other UAVs to generate information weights. An adaptive generative network includes an information completion network and a global discriminator network. The information completion network is connected to an information-level weight network and generates a global state through an encoder-decoder structure. The global discriminator network is used to determine the authenticity of the global state. Hybrid networks are used to integrate the local motion value functions of various UAVs and generate global motion value functions through an additivity structure. A Transformer-based decoder is used to calculate the local motion value function of the UAV based on local observations and information weights during the distributed execution phase. A policy network is used to generate the final action based on local action-value functions and coordination decisions; The loss function module includes a first loss function and a generative adversarial network loss function, which are used to update the parameters of the hybrid network, the adaptive generative network, and the information-level weighted network. The communication module is used to transmit weighted information and decision-making information between different drones; The environment interaction module is used to simulate the interaction process between the drone and the environment and to collect reward signals.

Images