Resilience recovery method, system and device of unmanned aerial vehicle cluster network and storage medium

Abstract

The application discloses a kind of unmanned aerial vehicle cluster network resilience recovery method, system, equipment and storage medium, it is related to unmanned system and intelligent network technical field, including: when detecting that unmanned aerial vehicle cluster is split into multiple unconnected subnets, determine the remaining unmanned aerial vehicle set and collect current network state information;The information is input into the reinforcement learning decision algorithm based on adaptive data enhancement and playback mechanism and / or attention mechanism training, the three-dimensional trajectory adjustment amount of each unmanned aerial vehicle is optimized jointly with transmit power, generates cooperative recovery action instruction and is issued to execute.The strategy effectively overcomes the problems such as low sample efficiency and weak cooperation ability caused by agent symmetry and environmental dynamics in traditional reinforcement learning, simultaneously reduces the energy consumption of movement and communication in high-dimensional continuous action space, speeds up the recovery speed, and improves the robustness and connectivity of the network topology after recovery.

Description

Technical Field

[0001] This application relates to the fields of unmanned systems and intelligent network technology, and in particular to methods, systems, devices and storage media for resilient recovery of unmanned aerial vehicle (UAV) swarm networks. Background Technology

[0002] Currently, with the widespread application of unmanned aerial vehicle (UAV) swarms in fields such as emergency communications, their network resilience—the system's ability to maintain core functions and recover quickly when subjected to attacks, failures, or disturbances—has become a key research focus. When UAV swarm networks perform missions in complex denied environments, they are highly vulnerable to physical, electronic, or cyberattacks, leading to node damage, link interruptions, and system performance degradation. Therefore, researching intelligent recovery methods after attacks is crucial for ensuring mission success.

[0003] Traditional network resilience recovery methods primarily rely on mathematical models and optimization theory. One type of work focuses on repairing the network topology, while another emphasizes the management of communication resources. These methods typically model the problem as mixed-integer programming, convex optimization, or specific heuristic algorithms.

[0004] However, existing resilience optimization methods mainly face the following problems: Traditional optimization algorithms (such as greedy algorithms and game theory algorithms) are often computationally complex and difficult to obtain real-time, efficient optimal solutions when dealing with high-dimensional, nonlinear, and dynamic joint optimization problems of UAV trajectory and power. Existing reinforcement learning methods, when applied to such multi-agent collaborative decision-making scenarios, suffer from low efficiency in learning from experience samples, insufficient multi-agent collaborative capabilities, and poor model generalization, resulting in slow network recovery speed, high energy consumption, and weak topological robustness after recovery.

[0005] Therefore, existing technologies have failed to effectively address how to achieve rapid, collaborative, and adaptive resilient recovery in a high-dimensional continuous action space after a drone swarm network is attacked and splits, while simultaneously overcoming the core problem of low sample efficiency, weak collaborative ability, and poor generalization of traditional reinforcement learning methods in multi-agent scenarios.

[0006] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0007] The main objective of this application is to provide a method, system, device, and storage medium for resilient recovery of unmanned aerial vehicle (UAV) swarm networks, aiming to solve the technical problem of how to achieve fast, energy-efficient, and topologically robust collaborative resilience recovery after a UAV swarm network splits, and to improve the sample efficiency and collaborative capabilities of multi-agent reinforcement learning.

[0008] To achieve the above objectives, this application proposes a resilient recovery method for unmanned aerial vehicle (UAV) swarm networks, the resilient recovery method for UAV swarm networks comprising: When the drone swarm network is detected to have split into multiple disconnected subnets, the remaining drone set consisting of the currently undamaged drones is determined, and the current network status information is collected through the remaining drone set; The current network state information is input into a pre-trained reinforcement learning decision algorithm. The pre-trained reinforcement learning decision algorithm is used to jointly optimize the three-dimensional trajectory adjustment and transmission power of each remaining UAV and generate corresponding collaborative recovery action commands. The pre-trained reinforcement learning decision algorithm is trained based on adaptive data augmentation and adaptive playback mechanisms and / or attention mechanisms. The coordinated recovery action command is sent to the corresponding remaining drones to perform corresponding movement and transmission power adjustment operations in order to restore the full connectivity of the drone swarm network.

[0009] In one embodiment, the preset pre-trained reinforcement learning decision algorithm includes a single-agent SAC algorithm or a multi-agent SAC algorithm. Before the step of jointly optimizing the 3D trajectory adjustment and transmission power of each remaining UAV using the pre-trained reinforcement learning decision algorithm, the following steps are also included: The neural network model corresponding to the single agent SAC algorithm is trained using an adaptive data augmentation and adaptive replay mechanism to obtain a trained single agent SAC model. The neural network model corresponding to the multi-agent SAC algorithm is trained using an attention mechanism to obtain a trained multi-agent SAC model.

[0010] In one embodiment, the step of training the neural network model corresponding to the single-agent SAC algorithm using an adaptive data augmentation and adaptive replay mechanism to obtain a trained single-agent SAC model includes: Obtain the global status information of the remaining drones in their current time slots; The global state information is input into the corresponding initial neural network model to obtain the global state information, reward value and termination flag for the next time slot, and form an empirical tuple. Store the experience tuples into the experience replay area; When sampling experience tuples from the experience replay area, the experience tuples are adaptively weighted by combining the time-first replay strategy and the time-difference error-first strategy, and the initial neural network model is updated based on the sampling results until convergence, thus obtaining the trained single-agent SAC model.

[0011] In one embodiment, the step of adaptively weighting the empirical tuple by combining the time-first playback strategy and the time-difference error-first strategy includes: In the experience replay area, a replay priority is assigned to each experience tuple, and the replay priority is determined by the timestamp generated by each experience tuple and the corresponding time difference error. The sampling probability of each empirical tuple is calculated based on the playback priority, and the corresponding importance sampling weight is generated based on the sampling probability. The empirical tuples are subjected to non-uniform probability sampling and bias correction based on the playback priority and the importance sampling weight.

[0012] In one embodiment, the step of training the neural network model corresponding to the multi-agent SAC algorithm using an attention mechanism to obtain a trained multi-agent SAC model includes: Encode the local observations of any remaining UAV into a query vector, and encode the observations and actions of all remaining UAVs other than the aforementioned remaining UAV into key vectors and value vectors, respectively. Based on the query vector and each key vector, the attention score is calculated, and the attention score is normalized to obtain the attention weights corresponding to all remaining drones except for any of the remaining drones. The value vector is weighted and summed according to the attention weights to generate an aggregated context representation; The aggregated context representation, the local observations of any remaining drone, and the actions of any remaining drone are input into the value network corresponding to any remaining drone to estimate the action value of any remaining drone performing the action. The neural network model corresponding to the multi-agent SAC algorithm is iteratively optimized using the action value to obtain a trained multi-agent SAC model. Among them, the value networks corresponding to all remaining drones share the same attention mechanism during training.

[0013] In one embodiment, the step of jointly optimizing the 3D trajectory adjustment and transmission power of each remaining UAV using the pre-trained reinforcement learning decision algorithm and generating corresponding cooperative recovery action commands includes: The current network state information is input into the pre-trained reinforcement learning decision algorithm, which outputs the three-dimensional trajectory adjustment amount and transmission power of each remaining UAV based on a preset objective function, as the collaborative recovery action command.

[0014] In one embodiment, the preset objective function is constructed based on a weighted sum of resilience assessment indicators, which include: UAV mobile energy consumption, UAV signal transmission energy consumption, network connection recovery time, and system connection stability characterized by the maximum eigenvector centrality of each UAV node in the UAV swarm network.

[0015] Furthermore, to achieve the above objectives, this application also proposes a resilient recovery system for unmanned aerial vehicle (UAV) swarm networks, the resilient recovery system for UAV swarm networks comprising: The status acquisition module is used to determine the remaining set of drones that are currently not damaged when the drone cluster network is detected to have split into multiple disconnected subnets, and to collect the current network status information through the remaining set of drones; The collaborative decision-making module is used to input the current network state information into a pre-trained reinforcement learning decision-making algorithm, and to jointly optimize the three-dimensional trajectory adjustment amount and transmission power of each remaining UAV through the pre-trained reinforcement learning decision-making algorithm, and generate corresponding collaborative recovery action commands. The pre-trained reinforcement learning decision-making algorithm is trained based on adaptive data augmentation and adaptive playback mechanism and / or attention mechanism. The command issuance model is used to send the collaborative recovery action command to the corresponding remaining drones to perform corresponding movement and transmission power adjustment operations in order to restore the full connectivity of the drone swarm network.

[0016] Furthermore, to achieve the above objectives, this application also proposes a resilient recovery device for a drone swarm network, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the resilient recovery method for a drone swarm network as described above.

[0017] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the resilient recovery method for unmanned aerial vehicle swarm networks as described above.

[0018] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the resilient recovery method for unmanned aerial vehicle (UAV) swarm networks as described above.

[0019] This application proposes a method, system, device, and storage medium for resilient recovery of a drone swarm network. The method includes: when the drone swarm network is detected to have split into multiple disconnected subnets, determining the remaining drone set consisting of currently undamaged drones, and collecting current network state information through the remaining drone set; inputting the current network state information into a pre-trained reinforcement learning decision algorithm, and jointly optimizing the 3D trajectory adjustment amount and transmission power of each remaining drone through the pre-trained reinforcement learning decision algorithm, and generating corresponding collaborative recovery action commands, wherein the reinforcement learning decision algorithm is trained based on an adaptive data augmentation and playback enhancement scheme and / or attention mechanism; and sending the collaborative recovery action commands to the corresponding remaining drones to perform corresponding movement and transmission power adjustment operations to restore the full connectivity of the drone swarm network. This solution effectively addresses the core problems of low sample efficiency and insufficient collaborative ability in unmanned swarm network recovery decisions caused by agent identity symmetry and environmental dynamism, by employing a reinforcement learning decision algorithm based on adaptive data augmentation and replay enhancement schemes and / or attention mechanisms. Furthermore, by applying the pre-trained reinforcement learning decision algorithm to the joint optimization of the 3D trajectory adjustment and transmission power of each remaining UAV, it can simultaneously reduce mobile and communication energy consumption, shorten network recovery time, and improve the robustness of the recovered topology in a high-dimensional continuous action space. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating an embodiment of the resilient recovery method for unmanned aerial vehicle (UAV) swarm networks provided in this application. Figure 2 This is a schematic diagram of a drone swarm network provided in Embodiment 1 of this application; Figure 3 This is a schematic diagram of the drone cluster receiving damage and resilience recovery stage according to Embodiment 1 of this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the resilient recovery method for unmanned aerial vehicle (UAV) swarm networks provided in this application. Figure 5This is a flowchart illustrating Embodiment 3 of the resilient recovery method for unmanned aerial vehicle (UAV) swarm networks provided in this application. Figure 6 This is a flowchart illustrating Embodiment 4 of the resilient recovery method for unmanned aerial vehicle (UAV) swarm networks provided in this application. Figure 7 This is a schematic diagram of the shared attention mechanism module structure provided in Embodiment 4 of this application; Figure 8 This is a schematic diagram of the module structure of the resilient recovery system for a drone swarm network according to an embodiment of this application; Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the resilient recovery method of the UAV swarm network in this embodiment of the application.

[0023] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] The main solution of this application embodiment is: when the drone cluster network is detected to have split into multiple disconnected subnets, the remaining drone set consisting of the currently undamaged drones is determined, and the current network status information is collected through the remaining drone set; The current network state information is input into a pre-trained reinforcement learning decision algorithm. The algorithm jointly optimizes the 3D trajectory adjustment and transmission power of each remaining UAV and generates corresponding collaborative recovery action commands. The pre-trained reinforcement learning decision algorithm is trained based on adaptive data augmentation and adaptive playback mechanisms and / or attention mechanisms. The collaborative recovery action commands are sent to the corresponding remaining UAVs to perform corresponding movement and transmission power adjustment operations to restore the full connectivity of the UAV swarm network.

[0027] In this embodiment, for ease of description, the following description will focus on the resilient recovery system of the UAV swarm network.

[0028] Because current drone swarms are vulnerable to physical, electronic, or cyberattacks during missions, leading to node damage, link interruptions, and system performance degradation, severely impacting mission completion, enhancing the resilience of drone swarm networks—that is, the system's ability to quickly restore connectivity and maintain core functions after an attack—is crucial.

[0029] Existing resilience optimization methods mainly face the following problems: Traditional optimization algorithms (such as greedy algorithms and game theory algorithms) are often computationally complex and struggle to obtain real-time, efficient optimal solutions when dealing with high-dimensional, nonlinear, and dynamic joint optimization problems of UAV trajectory and power. Existing reinforcement learning methods, when applied to such multi-agent collaborative decision-making scenarios, suffer from low efficiency in learning from experience samples, insufficient multi-agent collaborative capabilities, and poor model generalization, resulting in slow network recovery speed, high energy consumption, and weak topological robustness after recovery.

[0030] This application provides a solution that, in terms of performance optimization, effectively addresses the core problems of low sample efficiency and insufficient collaborative ability in the recovery decision-making of unmanned swarm networks by proposing an adaptive data and replay enhancement scheme and an attention-based multi-agent cooperative architecture. Regarding algorithmic efficiency, the designed training mechanism significantly improves the convergence speed and stability of policy learning, enabling it to efficiently handle high-dimensional continuous trajectory and power joint optimization problems. In terms of practicality and scalability, the adopted centralized training and distributed execution paradigm allows the system to make rapid and collaborative recovery decisions based solely on local observation information from the UAV during online deployment, demonstrating strong adaptability to dynamic random attack scenarios and good engineering implementation potential.

[0031] It should be noted that the executing entity of this embodiment can be a computing device with data processing, network communication and program execution capabilities, such as a ground control station, an airborne computing unit or a server; the following uses a ground control station as an example to describe this embodiment and the following embodiments.

[0032] Based on this, embodiments of this application provide a method for resilient recovery of unmanned aerial vehicle (UAV) swarm networks, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the resilient recovery method for unmanned aerial vehicle (UAV) swarm networks according to this application.

[0033] In this embodiment, the resilient recovery method for the drone swarm network includes steps S10 to S30: Step S10: When it is detected that the drone cluster network has split into multiple disconnected subnets, determine the remaining drone set consisting of the currently undamaged drones, and collect the current network status information through the remaining drone set; In this embodiment, the drone swarm network consists of N drones, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of a drone swarm network. The drone index is: Establish an XYZ three-dimensional coordinate system.

[0034] At time slot t, the three-dimensional coordinates of the i-th UAV are: The transmission power for sending signals to other drones is In time slot t, the i-th drone and the... For two drones to establish a two-way communication link, the following conditions must be met: the power of the signals received by both drones from the other must be greater than a reception threshold. :

[0035] Its meaning is: only when drones Received drone Received power of transmitted signal Not less than the preset receiving threshold And drones Received drone Received power of transmitted signal Not less than Only then can a stable two-way communication link be established between the two.

[0036] Two drones with a valid two-way communication link are marked as each other's neighbors. In the initial state, the drone swarm system is unattacked, all drones are interconnected to form a network, and each drone can transmit data to any other drone node via multi-hop links.

[0037] The communication channel between UAVs is modeled as an air-to-air (A2A) communication link. In time slot t, The drone received The signal power of the drone is:

[0038] in, This represents the drone at time t. Received drone The received power of the transmitted signal; and These are the constant antenna gains for receiving and transmitting UAVs, respectively. This is a large-scale fading effect. This is a small-scale fading effect; Indicates the first The transmit power of the drone at time t.

[0039] Since the system does not consider ground obstacles, the large-scale fading effect is expressed as:

[0040] in, This is the path loss coefficient. The frequency of the signal electromagnetic wave. At the speed of light, This represents the Euclidean distance between the two drones.

[0041] The small-scale fading effect is modeled as follows:

[0042] in, This represents the Euclidean distance between the two drones. This represents the variance of shadow fading. Let K denote the zeroth-order modified Bessel function, and K denote the Rice factor. Indicates the reference distance.

[0043] Therefore, the received signal power is related to the relative distance between the two UAVs and the transmitting power of the sending UAV. Thus, the condition for establishing a two-way communication link (CLEC) between the two UAVs is:

[0044] in, This represents the instantaneous value of the small-scale fading effect. , These represent the launch power of the two drones, respectively.

[0045] In real-world applications, unmanned swarm networks are vulnerable to physical, electronic, and cyberattacks. Individual or combined attacks can damage the system, causing failures in system nodes and links. Therefore, in this embodiment, several drones are damaged after being attacked, and the damaged nodes are detected by the system through their neighbors. The attack causes the network to lose full connectivity, dispersing it into multiple disconnected subnets. At this point, the system identifies the currently undamaged drones and groups them into a remaining drone set. It then collects current network state information from this set, including the location coordinates, transmission power, adjacency relationships, and link quality of each drone, to inform subsequent recovery planning by a pre-trained reinforcement learning decision-making algorithm.

[0046] Through the above steps, real-time perception and modeling of the split network state are achieved, providing the necessary input for joint optimization in a high-dimensional continuous action space.

[0047] Step S20: Input the current network state information into the pre-trained reinforcement learning decision algorithm, and jointly optimize the three-dimensional trajectory adjustment amount and transmission power of each remaining UAV through the pre-trained reinforcement learning decision algorithm, and generate corresponding cooperative recovery action commands; It should be noted that the pre-trained reinforcement learning decision-making algorithm includes the single-agent SAC algorithm or the multi-agent SAC algorithm. The Single Agent SAC algorithm treats the entire drone swarm as a single agent, with a centralized policy network outputting the joint actions of all remaining drones based on the global state. The multi-agent SAC algorithm configures an independent agent for each remaining UAV, optimizes the strategy using global information during the centralized training phase, and generates actions based solely on local observations during the distributed execution phase.

[0048] It is understandable that due to the large scale, high dimensionality of the action space, and strong environmental dynamics of drone swarms, traditional optimization methods or unimproved reinforcement learning strategies can easily lead to problems such as low sample efficiency, difficulty in policy convergence, and inconsistent collaborative behavior. Therefore, by executing step S20, and introducing adaptive data augmentation and adaptive replay mechanisms (for single agents) or shared attention mechanisms (for multi-agents), the training instability caused by state space symmetry and non-stationarity is effectively alleviated, thereby achieving efficient and robust collaborative recovery decision-making in a high-dimensional continuous action space. In one feasible embodiment, step S20 may include step S21: Step S21: Input the current network state information into the pre-trained reinforcement learning decision algorithm, and the pre-trained reinforcement learning decision algorithm outputs the three-dimensional trajectory adjustment amount and transmission power of each remaining UAV based on the preset objective function, as the cooperative recovery action command; In this embodiment, the system concatenates the global states (including position, power, and adjacency relationships) of all remaining UAVs into a single input vector, which is then input into a pre-trained reinforcement learning decision algorithm. The reinforcement learning decision algorithm calculates the joint action vector of all UAVs based on a preset objective function.

[0049] When the pre-trained reinforcement learning decision algorithm is the multi-agent SAC algorithm, each remaining UAV only inputs its local observations (such as its own coordinates, neighbor connection status, and remaining energy) into its corresponding pre-trained policy network to independently generate local action commands.

[0050] Furthermore, in order to achieve efficient, balanced, and resilient network reconstruction under limited energy resources, this embodiment uses the weighted sum of resilience assessment indicators as the preset objective function when jointly optimizing the three-dimensional trajectory adjustment and transmission power of each remaining UAV through the pre-trained reinforcement learning decision algorithm. The resilience assessment indicators include UAV mobile energy consumption, UAV signal transmission energy consumption, network connection recovery time, and system connection stability.

[0051] Specifically, such as Figure 3As shown, Figure 3 This diagram illustrates the resilience recovery phase of a drone swarm after receiving damaged drones. The system receives the set of remaining drones as follows: In each time slot In this context, the remaining drones can be considered as an undirected graph. , For the nodes of the graph, Let be the edges of the graph. Let be the coordinate topology matrix of all remaining UAVs. The maximum connected component of the graph, i.e., the number of subnets in the UAV swarm network, is . .

[0052] Among them, the adjacency matrix of an undirected graph When drones and When there is a bidirectional communication link ,otherwise The degree of a node is defined as: .

[0053] In the initial state, when the unmanned swarm network is unattacked, all drones are interconnected, forming a fully connected communication network. Each drone can transmit data to any other drone node via multi-hop links. Modeling the unmanned swarm network using an undirected graph, the maximum number of unconnected connected subnets (i.e., connected components) in the system is determined. .

[0054] When the system is subjected to an unpredictable attack, several drone nodes are damaged, which is detected by neighboring nodes. This study considers the general case where node damage affects the connectivity of the cluster network. After being attacked and damaged, the system is divided into multiple unconnected subnets. .

[0055] After being damaged by an attack, the system enters a resilience recovery phase: trajectory planning and power allocation are performed for the remaining undisturbed UAVs to restore full system connectivity as quickly as possible and reactivate the system. .

[0056] You can use depth-first search or breadth-first search to find the set of nodes for all connected components:

[0057] Where R[t] represents the set of all remaining UAVs at time slot t, Let j represent the j-th connected component in time slot t, where j and k represent the indices of the connected components.

[0058] To study the resilience optimization problem of restoring full connectivity to a drone swarm after unpredictable attacks leading to node damage and network fragmentation, this embodiment discretizes the continuous recovery process: dividing time into several equal-length time slots and making the following reasonable assumptions: (1) Each UAV flies at a constant speed along a straight line within a single time slot; (2) Its transmission power remains constant within this time slot; (3) The attack only occurs once at the initial moment, and there is no additional damage afterward.

[0059] In this scenario, the system's resilience is comprehensively evaluated based on the following four indicators: Energy consumption of drone movement (related to three-dimensional displacement); UAV signal transmission energy consumption (related to transmission power and time slot length). Network connection recovery time (number of time slots required to go from a split to full connectivity); The system's connectivity stability is characterized by the maximum value of the eigenvector centrality (EC).

[0060] In this context, EC measures the relative importance of a node in the network, which depends not only on its own number of connections but also on the importance of its neighboring nodes. Consider the adjacency matrix of the graph. The largest eigenvalue is Its corresponding feature vector is Then each component in this eigenvector Represents a node Eigenvector centrality:

[0061] Meanwhile, considering the limited energy of each drone in the cluster, it is necessary to avoid individual nodes running out of energy prematurely due to excessive recovery tasks. Therefore, this embodiment adopts a min-max optimization objective: the time-slot-wise weighted sum of "mobile energy consumption + transmission energy consumption + EC value" for each drone during the recovery process is used as its individual comprehensive cost. The system objective is to minimize the maximum value of the individual costs of all drones, i.e.:

[0062] Among them, (1) is the two-way communication link constraint, (2) and (3) are the unmanned cluster networking connectivity constraint, i.e., the number of subnets constraint, (4) is the maximum moving distance constraint of the UAV in each time slot, (5) is the UAV collision avoidance constraint, (6) is the UAV transmission power allocation constraint, (7) is the UAV maximum energy constraint, and (8) is the scene boundary constraint.

[0063] Through the above steps, a fast, energy-efficient, and topologically robust collaborative recovery of a split UAV swarm network is achieved in a high-dimensional continuous action space. This effectively balances recovery efficiency, energy fairness, and system resilience, and significantly improves the self-healing capability and mission continuity of large-scale UAV swarms in dynamic denial environments.

[0064] Step S30: Send the collaborative recovery action command to the corresponding remaining drones and perform corresponding movement and transmission power adjustment operations to restore the full connectivity of the drone swarm network.

[0065] In this embodiment, after receiving the coordinated recovery action command, each remaining UAV moves a corresponding distance along the specified direction within the current time slot and adjusts its transmission power to the target value. After execution, the system reassesses the communication link status, updates the adjacency relationship, and determines whether full connectivity has been restored. If not, it enters the next recovery cycle and repeats steps S10–S30 until the network rebuilds global connectivity or the maximum recovery time limit is reached.

[0066] The above-described embodiments employ a pre-trained reinforcement learning decision-making algorithm based on adaptive data augmentation and playback enhancement schemes and / or attention mechanisms. This effectively addresses the core issues of low sample efficiency and insufficient collaborative capabilities in traditional reinforcement learning for unmanned swarm network recovery decisions due to agent identity symmetry and environmental dynamism. Furthermore, by applying the pre-trained reinforcement learning decision-making algorithm to the joint optimization of the three-dimensional trajectory adjustment and transmission power of each remaining UAV, it is possible to simultaneously reduce mobility and communication energy consumption, shorten network recovery time, and improve the robustness of the recovered topology in a high-dimensional continuous action space.

[0067] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Before step S20, the resilient recovery method for the drone swarm network further includes steps S01-S02: Step S01: The neural network model corresponding to the single agent SAC algorithm is trained using an adaptive data augmentation and adaptive replay mechanism to obtain a trained single agent SAC model. Step S02: The neural network model corresponding to the multi-agent SAC algorithm is trained using an attention mechanism to obtain the trained multi-agent SAC model.

[0068] In one embodiment, due to the large scale of the UAV swarm, the high-dimensional continuity of the state-action space, and the fact that the optimization objective is a non-convex, nonlinear, sequential decision-making problem with a min-max structure, traditional methods are difficult to solve. Therefore, this embodiment is based on the deep reinforcement learning SAC (Soft Actor-Critic) algorithm framework and introduces an adaptive data and replay enhancement scheme (ADRA) to improve sample efficiency and policy robustness.

[0069] Specifically, the following designs are included: (1) State space: It is composed of the three-dimensional coordinate matrix of all remaining UAVs, the remaining energy vector and the adjacency matrix (taking the upper triangular part without the diagonal and flattening it); (2) Action space: includes the three-dimensional displacement adjustment and transmission power allocation of each UAV, which is constrained by the maximum travel distance, power range and airborne energy; (3) Reward function: comprehensively reflects the resilience recovery target, including: Negative term: The maximum value of "mobile energy consumption + transmit energy consumption + eigenvector centrality" for each machine in the current time slot; Positive items: Jain Fairness Index (promotes load balancing), Effective Bidirectional Link Count (reflects connectivity); Relaxation term: The sum of center distances between disconnected subnets (replacing the number of discrete subnets to alleviate reward sparsity); Penalty: A large negative reward is given and the current turn is terminated upon collision or crossing the boundary. (4) Adaptive training mechanism: Data augmentation: By leveraging the node symmetry of the drone ensemble, equivalent experience tuples are generated through index permutation of states and actions, thereby expanding the diversity of training data; Adaptive replay: When sampling, it integrates recent experience priority (emphasizing data generated by new strategies) and TD error priority (focusing on samples with high learning value), and uses importance sampling to correct distribution bias, which accelerates convergence and avoids overfitting.

[0070] In another embodiment, to address the practical constraints such as limited communication and incomplete local observations in large-scale clusters, this embodiment adopts the multi-agent SAC (MASAC) algorithm with a centralized training with decentralized execution (CTDE) paradigm, and further introduces an attention mechanism to enhance the agent's dynamic perception of key neighbors.

[0071] Specifically, the following designs are included: (1) Local observation space: the observation of each UAV i in time slot t It consists of three parts: Self-3D coordinate vector ; Remaining launch energy ; Connection vector , where the j-th dimension is 1 if and only if there is a valid bidirectional communication link between UAV i and j.

[0072] (2) Action space: Each UAV independently outputs its three-dimensional displacement adjustment and transmission power allocation, subject to movement distance, power limit and energy constraints.

[0073] (3) Hybrid reward function: Combining local and global information, it is defined as: Local terms: Individual energy consumption and eigenvector centrality; Fairness metric: Jain index (global); Connectivity terms: number of subnets, sum of center distances between subnets (global); Robustness terms: self-degree value (local), collision / boundary violation penalty (local).

[0074] (4) Attention-enhanced CTDE architecture: Each agent has its own independent policy network (input local observations) ) and the dual-Q value network; Value network input: Local observations from all agents With action It also dynamically weights neighbor information through a multi-head attention mechanism and automatically identifies the collaborative nodes that have the greatest impact on the current decision. Target network and temperature parameters: Designed according to the SAC standard, supporting entropy regularization and soft updates; Training objective: Maximize the sum of long-term discount reward and policy entropy. The loss function includes importance sampling correction and minimization of TD error.

[0075] This design effectively mitigates the non-stationarity problem caused by partial observability in multi-agent environments, enabling agents not only to learn environmental dynamics but also to efficiently model the behavioral intentions of other agents. Compared to traditional MASAC, the introduction of an attention mechanism significantly improves the scalability and collaborative efficiency of the policy in large-scale, highly dynamic scenarios, making it particularly suitable for resilient recovery tasks involving frequent node failures and rapid topology changes.

[0076] The above-described embodiments employ adaptive data augmentation and adaptive replay mechanisms to train the neural network model corresponding to the single-agent SAC algorithm, resulting in a trained single-agent SAC model. Similarly, an attention mechanism is used to train the neural network model corresponding to the multi-agent SAC algorithm, resulting in a trained multi-agent SAC model. This approach effectively balances the recovery efficiency, energy fairness, and topological robustness of large-scale UAV swarms after damage. On one hand, the single-agent approach enhances sample utilization through data augmentation and accelerates convergence through adaptive replay, making it suitable for centralized decision-making scenarios. On the other hand, the multi-agent approach dynamically focuses on key neighbors using the attention mechanism, achieving efficient collaboration within the CTDE framework. This significantly enhances the scalability and resilience of the strategy under locally limited observation conditions, thus providing a flexible and efficient resilient recovery solution for UAV swarm networks of different sizes and communication constraints.

[0077] Based on the second embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the second embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Further defining step S01, the resilient recovery method for the drone swarm network also includes steps A01 to A04: Step A01: Obtain the global status information of the remaining UAVs in the current time slot; It should be noted that the global state information includes, but is not limited to: the three-dimensional coordinates, remaining energy, communication adjacency relationships of each remaining UAV, and graph features reflecting the network topology.

[0078] Step A02: Input the global state information into the corresponding initial neural network model to obtain the global state information, reward value and termination flag for the next time slot, and form an empirical tuple; It should be noted that the reward value refers to a scalar feedback signal calculated based on the current network state, the action performed, and the recovery effect. It is used to measure the contribution of the action to the resilience recovery goal. Its specific forms include energy consumption penalty items, fairness reward items, connectivity improvement items, and collision / boundary violation penalty items. The termination flag is a Boolean variable used to indicate whether the current recovery cycle has ended. It is set to true when any of the following conditions are met: (1) the drone swarm network restores full connectivity; (2) the maximum number of recovery time slots is reached; (3) a drone collision or flying out of the scene boundary occurs.

[0079] In this embodiment, the global state information is input into the corresponding initial neural network model, which outputs a collaborative recovery action (including the 3D trajectory adjustment and transmission power of each UAV) and interacts with the environment to obtain the global state information, reward value, and termination flag at the next moment, thereby forming an empirical tuple. And store it in the experience replay area.

[0080] Specifically, let the symmetric group of the finite integer set I[t] formed by the remaining unmanned aerial vehicles be... The number of permutation elements in a symmetric group is Each element is a permutation map from the set to itself. , is represented as:

[0081] in, This indicates the new index position of the original i-th UAV after the replacement.

[0082] Then, taking the permutation elements from the symmetric group, we perform a synchronous mapping between the original state and the action to obtain the mapped state:

[0083] in, This represents the coordinates of the original i-th UAV's position vector under the new index after the permutation; Let A[t] be the upper triangular expansion vector of the adjacency matrix A[t] after undergoing the same permutation; This represents the value of the remaining energy of the i-th drone under the new index after the permutation.

[0084] Mapped actions:

[0085] in, , , These represent the three-dimensional displacement adjustment of the i-th UAV in time slot t; Adjust its transmission power.

[0086] Since system performance depends only on the relative topology rather than node labels, the mapped reward function satisfies:

[0087] Termination mark:

[0088] Therefore, each enhanced experience tuple At most, it can be generated through permutation mapping. tuples These augmented tuples have the same physical meaning but different node label orders, and can be used to augment training data and improve policy generalization ability.

[0089] Step A03: Store the experience tuple in the experience replay area; In this embodiment, additional data is randomly generated each time based on the mapping. tuples, and take the tuples Added to the experience replay area:

[0090] Number of tuples placed in the experience replay area It adaptively decreases as the training cycle increases. The decay coefficient is beneficial for decorrelation in the experience replay region during later training stages. .

[0091] Step A04: When sampling experience tuples from the experience replay area, the experience tuples are adaptively weighted by combining the time-first replay strategy and the time-difference error-first strategy, and the initial neural network model is updated based on the sampling results until convergence, thus obtaining the trained single-agent SAC model.

[0092] It should be noted that the time-priority replay strategy refers to assigning sampling priority to experience tuples based on their generation timestamps, with later-generated experience tuples receiving higher priority. This strategy is based on the observation that during reinforcement learning training, the policy network is continuously updated, and recently generated experiences better reflect the current policy's capabilities and the dynamic characteristics of the environment. Especially in the later stages of training, utilizing such experiences helps improve the stability of policy updates and sample efficiency.

[0093] The Temporal Difference Error Prioritized Experience Replay strategy refers to assigning priority based on the magnitude of the temporal difference error (TD-error) corresponding to the experience tuple. The larger the TD-error, the higher the uncertainty or the greater the information content of the experience in estimating the value function, and therefore it should be replayed more frequently to accelerate learning.

[0094] Since the policy changes continuously during training, experience tuples generated later in time better reflect the learning and updating status of the policy. Utilizing these experience tuples more extensively in the mid-to-late stages of training contributes to the stability and efficiency of the training. To address this issue, this embodiment proposes an adaptive replay enhancement scheme, including a recent experience enhancement module and a priority experience replay module.

[0095] Specifically, in the recent experience enhancement module, enhanced experience tuples generated later are sampled at a higher frequency. Let K be the total number of experience tuple sampling and learning operations (i.e., the number of time slots in the entire training cycle), and let the tuple capacity of the experience replay region be... In the k-th sampling update, instead of randomly sampling from the entire experience replay region, samples are taken from the one with the latest generation time. A set of enhanced empirical tuples Medium sampling:

[0096] in, For hyperparameters, To control the sampling weight of the most recently generated empirical tuple, when The sampling time is uniform.

[0097] Let T be the total number of time steps in the training process. and for Initial and final values, Annealing during training:

[0098] In the early stages of training, the agent learns slowly. A larger value with a smaller value for k more closely approximates uniform sampling from the entire experience replay region to fully utilize richer past experience samples. As the training period increases, the agent learns faster. Smaller and larger k gradually exclude more old data and learn more strategies from more recently generated experience tuples.

[0099] In the finite experience replay model, the temporal difference (TD) error index is used to characterize the importance of experience tuples to the current policy learning and the replay priority. In the reinforcement learning SAC algorithm, the value network considers experience tuples... The timing difference error is:

[0100] in, Represents enhanced experience tuples The next state in; This represents the reward at the current moment; Indicates the discount factor; Indicates temperature parameter; Represents enhanced experience tuples The current state; Represents enhanced experience tuples The current action, Q( , () represents the current predicted value. ( , ) represents the target network output.

[0101] During training, the value network is continuously updated, and the temporal difference error of each empirical tuple changes accordingly, requiring dynamic adjustment of its replay priority. However, for computational efficiency, usually only the sampled tuples are updated, rather than a full refresh. This may lead to tuples with low initial error being ignored for a long time, or overfitting to noisy and early high-error samples.

[0102] Therefore, when a new empirical tuple is added to the empirical replay area, since there is no known temporal difference error, its replay priority is set to the highest to ensure that all empirical tuples are sampled and learned by the model at least once. Simultaneously, a balance is struck between greedy sampling and completely random sampling of high-replay-priority empirical tuples, ensuring that the sampling probability of an empirical tuple is positively correlated with its replay priority. This defines the empirical tuple. The sampling probability is:

[0103] In the formula, The set of empirical tuples in the current sampling batch. The original priority of the i-th empirical tuple. This is the priority index.

[0104] Among them, empirical tuples The playback priority is , For timing difference error, Even empirical tuples with very small positive values, ensuring zero timing difference error, have a probability of being sampled. (Parameter) The extent to which replay priorities are used has been determined. The priority experience replay module is combined with the aforementioned recent experience enhancement module, within the set... Calculate playback priority and sampling empirical tuples within the range.

[0105] In the experiment, a sum tree was used to efficiently manage experience tuples and their replay priorities: leaf nodes store the priority of each tuple, and internal nodes store the sum of the priorities of their child nodes. During sampling, the range will be... The sample is divided into multiple segments, the number of which is the same as the batch size. A value is then uniformly sampled from each segment. Finally, the indices of the empirical tuples corresponding to these playback priorities are retrieved from the summation tree. This reduces the time complexity of sampling empirical tuples and updating playback priorities from... Reduced to .

[0106] During training, experience tuples are sampled from the experience replay region, and action values ​​are estimated through a value network. If random uniform sampling is used, the sampling distribution is consistent with the expected distribution, resulting in an unbiased estimate of the action value, which eventually converges to an unbiased solution. However, the priority experience replay method samples non-uniformly based on the replay priority of sample tuples, introducing bias and altering the original data distribution, leading to an inconsistency between the sampling distribution and the expected distribution. Therefore, even if the distributions of the policy and environment state are fixed, the converged solution will still be biased. To address this issue, this embodiment calculates the importance sampling weights based on the sampling probability. :

[0107] In the formula, N represents the total number of experience tuples in the experience replay area, and P(i) represents the number of experience tuples. Sampling probability, This represents the hyperparameter that controls the smoothing of the weights.

[0108] Importance sampling weights Introducing this into the update loss function of the value network reduces the weight of high-priority samples, thereby reducing or eliminating bias:

[0109] In the formula, Indicates the period from the experience replay area Mid-sampled state-action pairs take the expectation. Let y[t] represent the importance sampling weight, s[t] represent the target value, a[t] represent the state, and a[t] represent the action.

[0110] In the implementation, the importance sampling weights are normalized. To maximize its value to 1, thus avoiding gradient explosion. When the parameter This can completely offset the bias of non-uniform sampling, and the parameters will be used during training. By gradually annealing from 0 to 1, the system prioritizes the effect of experience playback in the early stage of training, while gradually remapping the non-uniform sampling distribution back to the original distribution in the middle and later stages of training to eliminate the bias in the results.

[0111] The above-described method obtains the global state information of the remaining UAVs in the current time slot; the global state information is input into the corresponding initial neural network model to obtain the global state information, reward value, and termination flag of the next time slot, and forms an experience tuple; the experience tuple is stored in the experience replay area; when sampling experience tuples from the experience replay area, the experience tuples are adaptively weighted and sampled by combining the time-series priority replay strategy and the time-series difference error priority strategy, and the initial neural network model is updated based on the sampling results until convergence, resulting in a trained single-agent SAC model. This significantly improves sample utilization efficiency and policy generalization ability, effectively alleviates the sparsity problem of experience replay while ensuring semantic consistency of recovery actions, accelerates policy convergence, and enhances the robustness of the model to changes in node arrangement, thereby achieving efficient, balanced, and resilient recovery of UAV swarm networks.

[0112] Based on the second embodiment of this application, in the fourth embodiment of this application, the content that is the same as or similar to the fourth embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 Further specifying step S02, the resilient recovery method for the drone swarm network also includes steps B01 to B05: Step B01: Encode the local observations of any remaining UAV into a query vector, and encode the observations and actions of all remaining UAVs other than the aforementioned remaining UAV into key vectors and value vectors, respectively. In this embodiment, the attention mechanism from Transformer is employed. Instead of treating all input equally, it focuses on the most important parts when processing information. The core idea is to allow the model to dynamically focus on the parts of the input data most relevant to the current task. The attention mechanism comprises three key components: the query is the target item for which the output needs to be calculated; the key is the identifier of the input item, used to calculate its relevance to the query; and the value is the actual content corresponding to the input item identifier.

[0113] Step B02: Calculate the attention score based on the query vector and each key vector, and normalize the attention score to obtain the attention weights corresponding to all remaining drones except for any of the remaining drones. In this embodiment, the attention score is first calculated, and the scaled dot product attention score is used to represent the attention score of the first digit. The query and the first The similarity of the keys is such that a higher score indicates greater relevance.

[0114] In the formula, This is the transpose of the key vector. This is a scaling factor used to prevent the gradient from vanishing due to an excessively large dot product. For the i-th query vector, Let be the i-th key vector.

[0115] Attention weights are calculated based on attention scores, and then converted into weights using the sftmax function, representing the importance of each input item.

[0116] Step B03: The value vector is weighted and summed according to the attention weights to generate an aggregated context representation; In this embodiment, the final output is a weighted sum of all values, with the weights determined by the degree of matching between the query and the key, i.e., the attention weights, resulting in the final attention output:

[0117] in, For attention weights, Let j be the value vector.

[0118] In practice, Query, Key, and Value are each learnable networks. , , The attention output is represented as:

[0119] in, For the transpose of the key, This is the scaling factor.

[0120] Multi-head attention is used to compute multiple sets of attention in parallel and then concatenate the results. Each attention head can independently focus on different dimensions of information, enabling multi-view modeling and avoiding overfitting from a single attention head. The information aggregated from each head can be concatenated into a richer contextual representation.

[0121] Step B04: The aggregated context representation, the local observation of any remaining UAV, and the action of any remaining UAV are input together into the value network corresponding to any remaining UAV to estimate the action value of any remaining UAV performing the action.

[0122] The MASAC-AM algorithm used in this embodiment enables the central value network of each agent to dynamically adjust its attention to other agents that have a greater impact on its own training at different training time points, instead of simply considering all agents at all time points with equal weight, thereby improving the performance in the multi-UAV complex cooperative control resilience optimization problem studied in this project.

[0123] MASAC-AM uses an attention mechanism to allow each agent to selectively focus on information from other agents to train its own value network. Each agent queries other agents for their observations and actions, incorporating this information into its own action value estimation. Unlike computer vision and natural language processing, MASAC-AM does not require making any assumptions about the input observations and actions. The action value function still takes into account the observations of all agents globally. and action value In addition to intelligent agents The set of all intelligent agents other than [other agents] is represented as and use Index the collection. The Query, Key, and Value settings in the MASAC-AM attention mechanism are:

[0124] in For intelligent agents Its own embedding, This involves embedding for all other agents. An attention mechanism module is shared across the value networks of all agents; the structure of the shared attention mechanism module is as follows: Figure 7 As shown.

[0125] After using the shared attention mechanism module, the agent's value network no longer receives information from all agents globally with equal weight. Instead, it aggregates information from other agents using the attention mechanism module, helping the agent find information that is more important for its own decision-making. The action value function estimated by the value network of the shared attention mechanism is:

[0126] in, For network parameters, It is a two-layer multi-layer perceptron (MLP). For single-layer MLP embedding functions, For other intelligent agents to intelligent agents Contribution value:

[0127] in For intelligent agents The embedding, encoded using an embedding function, is processed by a globally shared learnable matrix. Perform a linear transformation. Leaky ReLU is a non-linear activation function, an improved version of ReLU to address the neuron failure issue in the negative input region and improve gradient flow.

[0128] Attention weight Using bilinear mapping to transform the agent Embedded With intelligent agents Embedded The comparison is equivalent to a query-key-value module, which extracts three elements: selection, key, and value. The similarity value between the embeddings of the two agents is then passed to the softmax function.

[0129] Among them, the query matrix Will Transform it into an extraction selection, i.e., query, key matrix Will It is converted into a key, and after matching the value, it is scaled to prevent gradient vanishing.

[0130] This embodiment uses multiple attention heads, each using a separate set of parameters. Each independently generates all other agents' relationships with the agent. The aggregated contributions are obtained by concatenating the aggregated contributions of all attention heads into a single vector. Each attention head can learn a different aggregated weight parameter.

[0131] This embodiment also considers that while UAV agents share state characteristics, individual rewards vary significantly. Therefore, the encoding parameters for query, key, and value in the attention mechanism are set to be globally shared to improve learning efficiency. Within the CTDE framework, both the value network and the target network of MASAC-AM use the observations and actions of all agents as input and employ a shared attention mechanism. The value network update loss function is:

[0132] In the formula, Let be the target value of the i-th agent in time slot t. To determine the expected value of the samples in the experience replay region B, This represents the current value network's prediction of the value of an action.

[0133] Value network The target value is:

[0134] In the formula, Represents the parameters of the value network. To obtain the expected value of the samples in the experience replay area B, As a discount factor, Output to the target value network. For temperature parameters, Let i be the action sampled by agent i at time t+1. Let represent the local environmental information that agent i can observe at time t+1.

[0135] In the shared attention mechanism MASAC-AM, a dominance function is used instead of the action value function to improve the efficiency and stability of policy gradients. The dominance function measures the gain of a single action relative to the average action value in the current state. This effectively reduces policy update variance and, when other agent behaviors are fixed, more accurately determines whether the selected action brings additional benefits, thus helping the attention mechanism focus on actions more important for policy improvement. The dominance function is defined as:

[0136] in, Baseline state value function for multi-agent systems:

[0137] Among them, actions Sampling is performed from the current policy network of all agents, rather than from the experience replay region. The loss function for updating the MASAC-AM policy network is:

[0138] In the formula, For policy network parameters, For local observation and noise Take the expected value. For the policy function, Ad is the action output function of the policy network. Let i be the advantage function of agent i.

[0139] Step B05: Use the action value to iteratively optimize the neural network model corresponding to the multi-agent SAC algorithm to obtain a trained multi-agent SAC model.

[0140] In this embodiment, the loss function of the value network is constructed based on the action value and target value estimated in step B04, and the value network parameters of all agents are updated by gradient descent. At the same time, the parameters of the value network are periodically soft-updated to the corresponding target value network to stabilize the training process. Secondly, a loss function for the policy network is constructed based on the advantage function, and the reparameterization technique is used to optimize the policy network parameters end-to-end. Meanwhile, the parameters of the value network are periodically soft-updated to the corresponding target value network to stabilize the training process; The above optimization process is repeated during the centralized training phase until the preset convergence conditions are met—including but not limited to: the number of training rounds reaches the upper limit, the average reward tends to stabilize, or the network recovery success rate exceeds the threshold for several consecutive cycles, and finally a well-trained multi-agent SAC model is obtained.

[0141] The value network and policy network are both integrated with a globally shared multi-head attention mechanism module, which enables each agent to dynamically focus on the neighbor information that has the greatest influence on its own decision during the training process, thereby achieving efficient joint optimization of cooperative trajectory and power in a high-dimensional continuous action space.

[0142] The method described above encodes the local observations of any remaining UAV into a query vector, and encodes the observations and actions of all remaining UAVs (excluding any remaining UAV) into key vectors and value vectors, respectively. Attention scores are calculated based on the query vectors and key vectors, and these scores are normalized to obtain the attention weights for each of the remaining UAVs (excluding any remaining UAV). The value vectors are weighted and summed according to the attention weights to generate an aggregated context representation. The aggregated context representation, the local observations of any remaining UAV, and the actions of any remaining UAV are input into the value network corresponding to any remaining UAV to estimate the action value of the action performed by the remaining UAV. The action value is then used to iteratively optimize the neural network model corresponding to the multi-agent SAC algorithm, resulting in a trained multi-agent SAC model. This model enables each agent to dynamically identify and focus on the most influential collaborative nodes during training, effectively integrating key neighbor information and significantly improving the collaborative perception capability, policy robustness, and recovery efficiency of the multi-UAV system in complex scenarios such as partial observability, topological time-varying conditions, and node failures, while also considering computational scalability and communication feasibility.

[0143] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the resilient recovery method of the UAV swarm network of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0144] This application also provides a resilient recovery system for unmanned aerial vehicle (UAV) swarm networks. Please refer to [reference needed]. Figure 8 The resilient recovery system for the drone swarm network includes: The status acquisition module 10 is used to determine the remaining set of drones that are currently not damaged when the drone cluster network is detected to have split into multiple disconnected subnets, and to collect the current network status information through the remaining set of drones. The collaborative decision-making module 20 is used to input the current network state information into a pre-trained reinforcement learning decision-making algorithm, and to jointly optimize the three-dimensional trajectory adjustment amount and transmission power of each remaining UAV through the pre-trained reinforcement learning decision-making algorithm, and generate corresponding collaborative recovery action commands. The instruction issuing model 30 is used to send the collaborative recovery action instruction to the corresponding remaining drones to perform corresponding movement and transmission power adjustment operations in order to restore the full connectivity of the drone swarm network.

[0145] The resilient recovery system for UAV swarm networks provided in this application, employing the resilient recovery method for UAV swarm networks described in the above embodiments, can solve the technical problem of how to achieve rapid, energy-efficient, and topologically robust collaborative resilience recovery after a UAV swarm network splits, and improve the sample efficiency and collaborative capabilities of multi-agent reinforcement learning. Compared with the prior art, the beneficial effects of the resilient recovery system for UAV swarm networks provided in this application are the same as those of the resilient recovery method for UAV swarm networks provided in the above embodiments, and other technical features of the resilient recovery system for UAV swarm networks are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0146] This application provides a resilient recovery device for a drone swarm network. The resilient recovery device for a drone swarm network includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the resilient recovery method for a drone swarm network in the first embodiment described above.

[0147] The following is for reference. Figure 9This document illustrates a structural schematic diagram of a resilient recovery device suitable for implementing the drone swarm network embodiments of this application. The resilient recovery device for the drone swarm network in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 9 The resilient recovery device for the drone swarm network shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0148] like Figure 9 As shown, the resilient recovery device for a drone swarm network may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the resilient recovery device for the drone swarm network. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the resilient recovery device of the UAV swarm network to communicate wirelessly or wiredly with other devices to exchange data. Although a resilient recovery device for a UAV swarm network with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0149] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0150] The resilient recovery device for UAV swarm networks provided in this application, employing the resilient recovery method for UAV swarm networks described in the above embodiments, solves the technical problem of how to achieve rapid, energy-efficient, and topologically robust collaborative resilience recovery after a UAV swarm network splits, and improves the sample efficiency and collaborative capabilities of multi-agent reinforcement learning. Compared with the prior art, the beneficial effects of the resilient recovery device for UAV swarm networks provided in this application are the same as those of the resilient recovery method for UAV swarm networks provided in the above embodiments, and other technical features of this resilient recovery device for UAV swarm networks are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0151] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0153] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the resilient recovery method for the UAV swarm network in the above embodiments.

[0154] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0155] The aforementioned computer-readable storage medium may be included in the resilient recovery device of the UAV swarm network; or it may exist independently and not be assembled into the resilient recovery device of the UAV swarm network.

[0156] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the resilient recovery device of the UAV swarm network, the resilient recovery device of the UAV swarm network: when it detects that the UAV swarm network has split into multiple disconnected subnets, it determines the set of remaining UAVs that are currently undamaged, and collects current network state information through the set of remaining UAVs; it inputs the current network state information into a pre-trained reinforcement learning decision algorithm, and performs joint optimization of the three-dimensional trajectory adjustment amount and transmission power of each remaining UAV through the pre-trained reinforcement learning decision algorithm, and generates corresponding collaborative recovery action commands, wherein the reinforcement learning decision algorithm is trained based on an adaptive data augmentation and playback enhancement scheme and / or attention mechanism; it sends the collaborative recovery action commands to the corresponding remaining UAVs, and performs corresponding movement and transmission power adjustment operations to restore the full connectivity of the UAV swarm network.

[0157] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0160] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the resilient recovery method for the above-described UAV swarm network. This solves the technical problem of how to achieve fast, energy-efficient, and topologically robust cooperative resilience recovery after a UAV swarm network splits, and improve the sample efficiency and cooperative capability of multi-agent reinforcement learning. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the resilient recovery method for the UAV swarm network provided in the above embodiments, and will not be repeated here.

[0161] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the resilient recovery method for unmanned aerial vehicle (UAV) swarm networks described above.

[0162] The computer program product provided in this application solves the technical problem of how to achieve fast, energy-efficient, and topologically robust cooperative resilience recovery after a UAV swarm network splits, and improve the sample efficiency and cooperative ability of multi-agent reinforcement learning. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the UAV swarm network resilience recovery method provided in the above embodiments, and will not be elaborated upon here.

[0163] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for resilient recovery of unmanned aerial vehicle (UAV) swarm networks, characterized in that, The drone swarm network consists of multiple drones, and any two drones can establish a two-way communication link within a preset communication distance. The resilience recovery method of the drone swarm network includes: When the drone swarm network is detected to have split into multiple disconnected subnets, the remaining drone set consisting of the currently undamaged drones is determined, and the current network status information is collected through the remaining drone set; The current network state information is input into a pre-trained reinforcement learning decision algorithm. The pre-trained reinforcement learning decision algorithm jointly optimizes the 3D trajectory adjustment and transmission power of each remaining UAV and generates corresponding cooperative recovery action commands. The pre-trained reinforcement learning decision algorithm includes a single-agent SAC algorithm or a multi-agent SAC algorithm. The pre-trained reinforcement learning decision algorithm uses adaptive data augmentation and adaptive replay mechanisms to train the neural network model corresponding to the single-agent SAC algorithm to obtain a trained single-agent SAC model; and uses an attention mechanism to train the neural network model corresponding to the multi-agent SAC algorithm to obtain a trained multi-agent SAC model. The coordinated recovery action command is sent to the corresponding remaining drones to perform corresponding movement and transmission power adjustment operations in order to restore the full connectivity of the drone swarm network.

2. The resilient recovery method for unmanned aerial vehicle (UAV) swarm networks as described in claim 1, characterized in that, The steps of training the neural network model corresponding to the single-agent SAC algorithm using adaptive data augmentation and adaptive replay mechanisms to obtain the trained single-agent SAC model include: Obtain the global status information of the remaining drones in their current time slots; The global state information is input into the corresponding initial neural network model to obtain the global state information, reward value and termination flag for the next time slot, and form an empirical tuple. Store the experience tuples into the experience replay area; When sampling experience tuples from the experience replay area, the experience tuples are adaptively weighted by combining the time-first replay strategy and the time-difference error-first strategy, and the initial neural network model is updated based on the sampling results until convergence, thus obtaining the trained single-agent SAC model.

3. The resilient recovery method for unmanned aerial vehicle (UAV) swarm networks as described in claim 2, characterized in that, The step of adaptively weighting the empirical tuple by combining the time-first playback strategy and the time-difference error-first strategy includes: In the experience replay area, a replay priority is assigned to each experience tuple, and the replay priority is determined by the timestamp generated by each experience tuple and the corresponding time difference error. The sampling probability of each empirical tuple is calculated based on the playback priority, and the corresponding importance sampling weight is generated based on the sampling probability. The empirical tuples are subjected to non-uniform probability sampling and bias correction based on the playback priority and the importance sampling weight.

4. The resilient recovery method for unmanned aerial vehicle (UAV) swarm networks as described in claim 1, characterized in that, The step of training the neural network model corresponding to the multi-agent SAC algorithm using an attention mechanism to obtain the trained multi-agent SAC model includes: Encode the local observations of any remaining UAV into a query vector, and encode the observations and actions of all remaining UAVs other than the aforementioned remaining UAV into key vectors and value vectors, respectively. Based on the query vector and each key vector, the attention score is calculated, and the attention score is normalized to obtain the attention weights corresponding to all remaining drones except for any of the remaining drones. The value vector is weighted and summed according to the attention weights to generate an aggregated context representation; The aggregated context representation, the local observations of any remaining drone, and the actions of any remaining drone are input into the value network corresponding to any remaining drone to estimate the action value of any remaining drone performing the action. The neural network model corresponding to the multi-agent SAC algorithm is iteratively optimized using the action value to obtain a trained multi-agent SAC model. Among them, the value networks corresponding to all remaining drones share the same attention mechanism during training.

5. The resilient recovery method for unmanned aerial vehicle (UAV) swarm networks as described in claim 1, characterized in that, The step of jointly optimizing the 3D trajectory adjustment and transmission power of each remaining UAV using the pre-trained reinforcement learning decision algorithm, and generating corresponding cooperative recovery action commands, includes: The current network state information is input into the pre-trained reinforcement learning decision algorithm, which outputs the three-dimensional trajectory adjustment amount and transmission power of each remaining UAV based on a preset objective function, as the collaborative recovery action command.

6. The resilient recovery method for unmanned aerial vehicle (UAV) swarm networks as described in claim 5, characterized in that, The preset objective function is constructed based on a weighted sum of resilience assessment indicators, which include: UAV mobile energy consumption, UAV signal transmission energy consumption, network connection recovery time, and system connection stability characterized by the maximum eigenvector centrality of each UAV node in the UAV swarm network.

7. A resilient recovery system for unmanned aerial vehicle (UAV) swarm networks, characterized in that, The resilient recovery system for the drone swarm network includes: The status acquisition module is used to determine the remaining set of drones that are currently not damaged when the drone cluster network is detected to have split into multiple disconnected subnets, and to collect the current network status information through the remaining set of drones; The collaborative decision-making module is used to input the current network state information into a pre-trained reinforcement learning decision-making algorithm. The pre-trained algorithm jointly optimizes the 3D trajectory adjustment and transmission power of each remaining UAV and generates corresponding collaborative recovery action commands. The pre-trained reinforcement learning decision-making algorithm includes a single-agent SAC algorithm or a multi-agent SAC algorithm. The pre-trained algorithm uses adaptive data augmentation and adaptive replay mechanisms to train the neural network model corresponding to the single-agent SAC algorithm, obtaining a trained single-agent SAC model; and uses an attention mechanism to train the neural network model corresponding to the multi-agent SAC algorithm, obtaining a trained multi-agent SAC model. The command issuance model is used to send the collaborative recovery action command to the corresponding remaining drones to perform corresponding movement and transmission power adjustment operations in order to restore the full connectivity of the drone swarm network.

8. A resilience recovery device for unmanned aerial vehicle (UAV) swarm networks, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the resilient recovery method for a drone swarm network as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the resilient recovery method for a drone swarm network as described in any one of claims 1 to 6.

Images