A control method for swarm drone systems based on DPPO deep reinforcement learning

Abstract

The present invention relates to an intelligent control method for an unmanned aerial vehicle (UAV) system, and in particular to a control method for a swarm UAV system based on DPPO deep reinforcement learning. The method constructs a three-tiered structure of cluster-formation-single-machine, comprehensively analyzes the collaborative relationships between formations within a cluster and between UAVs within a formation, and forms a cross-tier dynamic control system. Using a deep neural network model, multi-dimensional situational information is extracted, including mission area characteristics, formation coverage characteristics, UAV location characteristics, and damage area characteristics, providing a comprehensive decision-making basis for the generation of dynamic control strategies. The present invention realizes the generation and real-time adjustment of optimal control strategies for swarm UAV systems in high-dimensional state space, enhances the system's adaptability and robustness in complex environments, and significantly improves the task execution efficiency and stability of swarm UAV systems.

Description

Technical Field

[0001] The present invention relates to an intelligent control method for an unmanned aerial vehicle (UAV) system, and in particular to a control method for a swarm UAV system based on DPPO deep reinforcement learning. Background Art

[0002] In recent years, swarm drone systems have been widely used in military reconnaissance, environmental monitoring, logistics, and other fields. By enabling the coordinated operation of multiple drones, swarm systems can perform complex tasks over a larger area, demonstrating advantages that cannot be achieved by a single drone. However, in practical applications, swarm drone systems often face complex and changing environments, including dynamic changes in mission objectives, environmental uncertainty, and external interference or attacks. This places higher demands on swarm system control and mission execution.

[0003] Traditional drone swarm control methods are usually based on preset rules or heuristic algorithms, such as centralized control, distributed-based collaborative control, etc. These methods can achieve collaborative work between drones in the swarm to a certain extent, but often have the following limitations: 1. Local optimality rather than global optimality: Traditional methods usually only focus on the optimal control strategy within a local range and lack global optimization of the entire swarm system. For multi-level drone swarm systems, optimization of a single level may lead to a decrease in overall performance. 2. Lack of adaptability: Preset rules and heuristic algorithms are difficult to cope with the dynamic changes of complex environments and tasks, and cannot be adaptively adjusted according to real-time situation information. They are prone to failure when the environment changes or external interference occurs. 3. Insufficient real-time performance: Traditional control methods often require a long calculation and adjustment time, which makes it difficult to meet the real-time control requirements of drone swarms in complex dynamic environments.

[0004] With the development of artificial intelligence and deep learning technologies, reinforcement learning (RL) has shown great potential in solving complex dynamic decision-making problems. Deep reinforcement learning (DRL) builds interactive learning strategies between agents and their environments, enabling optimal decision-making in high-dimensional state spaces. This approach provides a new approach for the dynamic control of drone swarms. However, traditional deep reinforcement learning methods also face challenges in the application of drone swarm systems: 1. High-dimensional state space: Drone swarm systems typically consist of a large number of drones, resulting in a high-dimensional and complex state space. Traditional deep reinforcement learning methods struggle to effectively learn and optimize policies in such a high-dimensional state space. 2. Multi-level coordination and dynamic adjustment: Drone swarm systems involve dynamic coordination and adjustment at three levels: cluster, formation, and individual drones. Existing methods struggle to achieve optimal control strategies simultaneously at all three levels. 3. Real-time response and policy generation: In practical applications, drone swarms must generate control policies in real time based on mission requirements and environmental changes, placing high demands on the algorithm's real-time performance and responsiveness. Summary of the Invention

[0005] To address these issues, this paper proposes a control method for swarm drone systems based on distributed proximal policy optimization (DPPO) deep reinforcement learning. This method constructs a three-tiered structure: cluster-formation-individual drone. It comprehensively analyzes the collaborative relationships between formations within a swarm and between drones within a formation, forming a cross-tiered dynamic control system. Using a deep neural network model, it extracts multidimensional situational information, including mission area characteristics, formation coverage characteristics, drone location characteristics, and damage zone characteristics, providing a comprehensive decision-making basis for generating dynamic control strategies.

[0006] To generate control strategies, a reinforcement learning algorithm based on DPPO is employed. This algorithm uses a dominant actor-critic policy gradient algorithm combined with experience replay technology for asynchronous updates, enabling real-time optimization of strategies. Furthermore, a federated learning mechanism, comprised of three types of agents: a master agent, a master explorer, and a coalition of explorers, enables distributed learning and training to enhance the global optimality and diversity of strategies. Ultimately, as the drone swarm executes its mission, the generated control strategy adjusts the formation topology and drone path planning in real time, ensuring system efficiency and continuous mission execution.

[0007] The introduction of this method realizes the generation and real-time adjustment of optimal control strategies for swarm UAV systems in high-dimensional state space, enhances the system's adaptability and robustness in complex environments, and significantly improves the mission execution efficiency and stability of swarm UAV systems.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0009] A control method for a swarm drone system based on DPPO deep reinforcement learning, the method comprising the following steps:

[0010] 1) Construction of a three-level structure for swarm drone systems: Build a three-level structure of clusters, formations, and individual drones. Analyze the collaborative relationships and topological configuration characteristics between formations within a swarm and between drones within a formation to form a three-level dynamic control system.

[0011] 2) Multi-dimensional Situational Awareness and Feature Extraction: Design a deep neural network model, using the ResNet module to process mission area features, formation coverage features, drone location features, and damage area features. Utilize the LSTM module to extract temporal features of high-dimensional information, and use the Attention mechanism to focus on interactions within the formation and across layers to obtain multi-dimensional situational information.

[0012] 3) Control strategy generation based on distributed proximal policy optimization (DPPO): Applying the superior Actor-Critic policy gradient algorithm, combined with experience replay technology for asynchronous updates, maximizing the expected policy reward, and using TD(λ), V-trace, and UPGO algorithms to train and update the neural network, generating a dynamic control strategy for the swarm UAV system;

[0013] 4) Design and training of alliance learning mechanisms: By designing alliance games and virtual self-learning mechanisms, including three types of agents: the master agent, the master explorer, and the alliance explorer, distributed learning training is carried out to optimize the control strategy of the swarm drone system;

[0014] 5) Dynamic control strategy execution: During the execution of a UAV swarm mission, the formation topology and UAV path planning are adjusted in real time based on the generated control strategy to ensure the continuity of mission execution and the optimization of system performance.

[0015] Preferably, the cluster-formation-standalone three-layer structure in step 1) includes:

[0016] 1.1) Analyze the mission coordination characteristics of UAV swarms and build coordination relationship models within and between formations;

[0017] 1.2) Establish a three-level topology model for the UAV swarm system, including the swarm as a whole, the formation layer, and the single-machine layer, to support dynamic control and adjustment between different layers.

[0018] Preferably, the multidimensional situation feature extraction in step 2) includes:

[0019] 2.1) Represent the task area features as a matrix to describe the coverage of the task area in the relative coordinate system;

[0020] 2.2) Extract the formation’s coverage features to describe the formation’s coverage effect within the mission area;

[0021] 2.3) Design a drone position feature matrix to describe the current location of the drone in the swarm;

[0022] 2.4) Describe the characteristics of the destruction area, including the location of the destruction center, the destruction radius, and the status of the affected drones.

[0023] Preferably, the generation of the DPPO-based control strategy in step 3) includes:

[0024] 3.1) Leveraging the Actor-Critic Policy Gradient algorithm, a powerful tool in deep reinforcement learning, combined with experience replay technology, we asynchronously update the control strategy of the drone swarm.

[0025] 3.2) Maximize the policy reward function, consider the efficiency and robustness of the UAV swarm mission execution, and optimize the control strategy;

[0026] 3.3) Apply TD(λ), V-trace and UPGO algorithms to train deep neural networks to achieve optimal updates of the UAV dynamic control strategy.

[0027] Preferably, the alliance learning mechanism in step 4) includes:

[0028] 4.1) Design alliance games, randomly generate multi-objective tasks and destruction scenarios, and the main agent will play against opponents in the alliance archive;

[0029] 4.2) Applying a virtual self-learning mechanism, select opponents with high win rates from the alliance archive to play against, improving the control strategy of the main agent;

[0030] 4.3) Design of three types of agents, including the master agent, master explorer, and coalition explorer, to collaboratively learn and optimize strategies.

[0031] Preferably, the execution of the dynamic control strategy in step 5) includes:

[0032] 5.1) Dynamically control the formation topology of the drone swarm and adjust it in real time based on mission requirements and environmental changes;

[0033] 5.2) Based on the generated control strategy, the drone path is planned in real time to optimize the mission execution efficiency;

[0034] 5.3) In the event of local damage, mission performance can be restored and continued by redeploying the UAV formation.

[0035] By adopting the above-mentioned technical solution, the present invention realizes the global optimal control, real-time strategy generation, multi-level collaboration, and distributed learning of the swarm drone system in a high-dimensional situation space through a swarm drone system control method based on DPPO deep reinforcement learning. This greatly improves the system's task execution efficiency, stability, and adaptability in complex environments, demonstrating significant technical advantages and broad application prospects. This is specifically reflected in the following aspects:

[0036] 1. Globally Optimal Multi-Level Collaborative Control: This invention achieves global optimal control of a swarm drone system by constructing a three-tiered structure: cluster, formation, and individual drones. This approach not only focuses on coordination within a single formation but also considers dynamic adjustments between formations and the entire swarm, ensuring optimal coordination at all levels. Compared to traditional local optimization methods, this global control strategy significantly improves the overall mission execution efficiency of the swarm drone system.

[0037] 2. Efficient Multi-Dimensional Situational Awareness and Feature Extraction: By designing a deep neural network model based on ResNet, LSTM, and the Attention mechanism, this invention can efficiently extract multi-dimensional situational features in a high-dimensional state space, including mission area features, formation coverage features, drone location features, and damage area features. This comprehensive situational awareness and feature extraction provides accurate and rich information for the generation of dynamic control strategies, improving the precision and effectiveness of strategic decisions.

[0038] 3. Real-time Dynamic Control Strategy Generation: Utilizing a DPPO-based reinforcement learning algorithm model, this invention generates and optimizes control strategies in real time as a drone swarm executes its mission. By asynchronously updating the neural network model, it rapidly responds to environmental changes and mission requirements, enabling real-time control and adjustment of the drone swarm. This real-time capability ensures the system can promptly adjust mission deployment and topology in complex and dynamic environments, maintaining continuity and efficiency in mission execution.

[0039] 4. Enhanced Adaptability and Robustness: This invention improves the adaptability and robustness of drone swarms in complex environments through real-time perception and feature extraction of multi-dimensional situational information. The system generates adaptive dynamic control strategies to address environmental changes and localized damage that may occur during mission execution, enabling the drone swarm to self-adjust and recover, thereby ensuring mission performance and stability.

[0040] 5. Distributed Policy Learning and Optimization: By introducing a federated learning mechanism, including a master agent, a master explorer, and federated explorers, this invention achieves distributed policy learning and optimization. The distributed learning mechanism expands the policy search space, improves policy diversity and global optimality, enables control strategies to adapt to the needs of different tasks and environmental conditions, and enhances system execution efficiency.

[0041] 6. Improving the Mission Execution Efficiency of Drone Swarms: In practical applications, this invention significantly improves the efficiency and flexibility of mission execution by dynamically adjusting the formation topology and path planning of drone swarms. Even in the event of localized system damage, the system can promptly reconfigure and adjust to restore mission performance and ensure continuous mission execution. This capability is particularly effective in multi-target missions and complex environments, improving the overall mission completion efficiency of drone swarms.

[0042] 7. Policy Training Efficiency and Real-Time Performance: Leveraging the advantages of the DPPO reinforcement learning algorithm's Actor-Critic Policy Gradient, Experience Replay, TD(λ), V-trace, and UPGO techniques, this paper achieves efficient training and real-time generation of control policies. By maximizing the policy reward function, it ensures rapid policy convergence during training and maintains high performance in dynamic environments, achieving breakthroughs in real-time performance and task execution efficiency.

[0043] 8. Ability to cope with complex and dynamic environments: This invention enables the drone swarm system to make efficient decisions in complex and dynamic environments through real-time strategy generation and execution. Utilizing multi-dimensional situational information extracted by deep neural networks, the system can flexibly adjust strategies during mission execution, ensuring that the drone swarm maintains high levels of mission performance despite varying environmental changes and mission requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Dynamically reconstruct the decision-making framework for cluster systems.

[0045] Figure 2 It is the DPPO reinforcement learning algorithm model.

[0046] Figure 3 Learning algorithm models for the DPPO alliance.

[0047] Figure 4 Dynamically reconstruct a case diagram for a swarm drone system.

[0048] Figure 5 Dynamically reconstruct the Agent-Environment architecture across multiple levels for cluster systems.

[0049] Figure 6 Reward curves for different methods.

[0050] Figure 7 This is an analysis diagram of various reconstruction strategies for different situations. DETAILED DESCRIPTION

[0051] The present invention targets typical problems such as operation management in cluster system operation and maintenance, and focuses on a representative frontier problem of this type of problem - the decision problem of "dynamic reconstruction" of cluster systems. It analyzes the task coordination and topological configuration characteristics of a three-level spatiotemporal dynamic cluster system. During the execution of tasks, the cluster system is usually constructed in a three-level form of cluster-cluster-node. Based on the needs of task coordination, it is necessary to focus on the topological configuration characteristics between clusters within the cluster and between nodes within the cluster. In addition, the cluster system may perform tasks in some complex environments, and clusters and nodes in the cluster may fail and be damaged at any time. For emergencies such as local damage, the cluster system can restore cluster performance to a certain extent to continue to complete the established tasks through two-level dynamic reconstruction: dynamic reconstruction between clusters within the cluster and dynamic reconstruction between nodes within the cluster. How to reconstruct the cluster topology configuration from the three levels of cluster-cluster-node towards task collaboration to complete the current task is an important research work. Existing research usually focuses on the reconstruction within each cluster, but in practice, an effective reconstruction strategy needs to include the three levels of cluster-cluster-node. Each node has the same overall task goal as other nodes in the cluster. Therefore, it is necessary to dynamically reconstruct the cluster within the cluster and the cluster within the node, and consider the cross-level dynamic reconstruction deployment of the cluster system task collaboration and topology configuration.

[0052] In order to solve the dynamic reconstruction decision problem of the three-level spatiotemporal dynamic cluster system mentioned above, considering the task coordination and topological configuration characteristics of the cluster system during task execution, a dynamic reconstruction decision method based on Distributed Proximal Policy Optimization (DPPO) deep reinforcement learning is studied, such as Figure 1-Figure 3The proposed DPPO-based method consists of three parts: a cluster system dynamic reconstruction decision-making architecture, a DPPO reinforcement learning algorithm model, and a DPPO alliance learning model. First, a two-level dynamic reconstruction decision-making architecture for the cluster system is constructed. This architecture comprehensively analyzes cluster system information and high-dimensional situational information about complex environments. A deep neural network is designed, and features are extracted using ResNet and LSTM. The Attention mechanism is used to focus on cross-level interactions within and between clusters. The Reinforce principle is then applied to estimate the value of reconstruction actions under two-level reconstruction to support cross-level dynamic reconstruction strategy optimization. Based on the above two-level dynamic reconstruction, the reinforcement learning algorithm model and alliance learning model of DPPO are further studied. The reinforcement learning algorithm model needs to apply the dominant actor-critic policy gradient algorithm and perform asynchronous updates based on the experience replay technology. By optimizing the strategy, the expected reward of the reconstruction strategy is maximized. Then, TD(𝝀), V-trace and UPGO are used to train and update the neural network, pushing the strategy to a trajectory that is better than the average return. In addition, to realize the distributed learning and training of DPPO, an alliance learning model is designed, and a virtual self-learning mechanism with alliance games and priority considerations is defined. Three types of agents are used for learning and training to achieve dynamic reconstruction strategy optimization. Based on the proposed DPPO dynamic reconstruction decision algorithm, the present invention conducts case analysis on the dynamic reconstruction of a multi-formation drone cluster under the scenario of local destruction. The case analysis results show that the proposed dynamic reconstruction decision algorithm can provide an efficient dynamic reconstruction solution in the topological configuration redeployment of multi-formation clusters for task collaboration, realizing the redeployment of drone clusters under local destruction to continue to perform the current cluster task.

[0053] 1.1 Description of the Cluster System Dynamic Reconfiguration Problem

[0054] 1.1.1 Cluster System Tasks, Destruction, and Reconstruction

[0055] First, we analyze the cluster-cluster-node three-level spatiotemporal dynamic cluster system tasks, destruction and reconstruction, considering the multi-objective tasks including M There are different task areas, and the task area space is described as follows:

[0056] Ω={A1, A2,…,A m ,…,A M} (1.1)

[0057] The mission area A m , m∈{1, 2, ..., M} is an irregular polygonal area.

[0058] For this multi-target mission, a team consisting of MIf a cluster S is composed of clusters, then the cluster set C of the cluster S can be expressed as:

[0059] C={c1, c2,…,c m ,…,c M} (1.2)

[0060] in c m Indicates dispatch to the mission area A m Clusters; consider clusters c m By N m nodes, then the cluster c m Expressed as:

[0061] C m ={n (m , 1) , n (m , 2) ,…, n (m , n) ,…, n (m , Nm)} (1.3)

[0062] where n∈1,2,…,N m} represents node n (m , n) In cluster c m The dynamic number in N m Cluster c m The total number of nodes depends on the size of the task area A m Sure.

[0063] During the execution of the multi-objective task, the cluster system may be locally damaged, resulting in the destruction of some nodes in one or more clusters. Taking the impact of this damage as input, each node is considered to have two states: normal state and fault state. When a node is damaged, it enters the fault state. The scope of the local damage is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value is randomly generated.

[0064] Under localized damage, the dynamic reconfiguration of a cluster system is considered an autonomous behavior, adapting to environmental changes to continue executing the current task. When a cluster is affected by dynamic changes during task execution, a corresponding dynamic reconfiguration strategy is needed to restore the cluster system's task performance and redeploy its topology, thereby ensuring task continuity. When localized damage occurs, a dynamic reconfiguration algorithm observes the cluster system's damage and generates a reconfiguration strategy that is communicated to each node. The cluster system then performs dynamic reconfiguration between nodes within the cluster and between clusters within the cluster.

[0065] For the τ moment of the dynamic reconstruction process of the cluster system, the cluster balance degree ε at the τ moment is defined as b (τ) represents the mean square error of the redundancy values ​​of all nodes that can work normally in each cluster during the task execution phase, as shown below:

[0066] (1.4)

[0067] Where M represents the total number of clusters in cluster S, N m Represents cluster c m The total number of nodes, ε mn (τ) represents the node n at time τ (m , n) The sum of the overlapping areas of the coverage area of ​​​​the node and the coverage areas of other nodes in the same cluster within the task area, It represents the average redundancy value of all nodes in each cluster at time τ.

[0068] 1.1.2 Dynamic Reconstruction Strategy Optimization Problem

[0069] Based on the aforementioned task coordination and topological configuration characteristics, and considering the local damage scenario during the cluster system mission, we construct a cluster system dynamic reconfiguration decision (DRD) problem. This problem describes the damage center and radius, describes the cluster failure scale, determines the threshold for reconfigurable nodes in each cluster, determines the maximum movement distance of reconfigurable nodes, stipulates the minimum coverage of the task area, describes the cluster balance, and determines the reconstruction time threshold. Based on this, the optimization objective, constraints, and control variables of the DRD problem are defined as follows:

[0070] (1.5)

[0071] where ε m Represents cluster c m The degree of balance, ε m min Represents cluster c m The balance threshold, N m moveRepresents cluster c m The total number of reconfigurable nodes in m normal Represents cluster c m The total number of nodes in normal state, d represents the distance between two nodes in normal state, d min It represents the minimum allowed distance, that is, the safe distance between nodes, τ tot represents the total time of the reconstruction process, τ thr Reconstruction time threshold.

[0072] 1.2 DPPO-based algorithm principle

[0073] For this type of three-level spatiotemporal dynamic cluster system, the dynamic reconfiguration decision-making problem, considering task coordination and topological configuration characteristics, requires a focus on the topological configuration characteristics of clusters within the cluster and between nodes within the cluster, based on the requirements of task coordination. A comprehensive analysis of cross-level dynamic reconfiguration is then conducted. A DPPO approach is proposed to address this problem. DPPO does not simply select and evaluate reconfiguration actions for individual nodes at the individual level. Instead, it comprehensively considers the three levels of cluster, cluster, and node that an effective reconfiguration strategy must encompass, analyzing the overall task objectives shared by each node with other nodes in the cluster. This requires a two-level dynamic reconfiguration decision-making architecture for the cluster system, comprehensively analyzing cluster system information and high-dimensional situational information in complex environments, and designing a deep neural network. Building on this two-level dynamic reconfiguration, a reinforcement learning algorithm model for DPPO is further developed, maximizing the expected reward of the reconfiguration strategy through optimization. Therefore, a dynamic reconfiguration decision-making method based on DPPO is designed to consider two levels of dynamic reconfiguration: intra-cluster reconfiguration and intra-cluster reconfiguration. This allows for cross-level dynamic reconfiguration deployment for cluster system task coordination and topological configuration.

[0074] The deep reinforcement learning idea based on DPPO that this invention intends to adopt includes two core ideas: strategy iteration with cross-level dynamic reconstruction and cross-level dynamic reconstruction based on DPPO.

[0075] 1.2.1 Strategy Iteration for Dynamic Reconstruction Across Layers

[0076] The cross-level dynamic reconstruction strategy iteration is to dynamically reconstruct the interaction between the agent and the simulation environment, such as Figure 5As shown. In the Agent-Environment architecture for dynamic cross-level reconstruction of cluster systems considering task collaboration and topological configuration, when the agent interacts with the environment, each agent still follows the goal of reinforcement learning, that is, to maximize the cumulative rewards that can be obtained, and at this time, the change in the global state of the environment is related to the joint actions of all agents. Therefore, in the process of agent strategy learning, the impact of joint actions needs to be considered. In the process of formulating dynamic reconstruction strategies, considering that the multi-formation cluster system has an excessively large decision space in the task area set above, and the formations in the cluster need to be as intelligent as possible in the current environment to complete the dynamic reconstruction process of the formation in the face of sudden destruction in a timely manner, a multi-agent reinforcement learning algorithm is selected to complete the generation of the dynamic reconstruction strategy of the multi-formation cluster system, and a part of the systems that meet the requirements in the multi-formation cluster system are selected as agents according to certain rules, and then the Agent-Environment architecture for dynamic cross-level reconstruction of the multi-formation cluster system is designed, as shown in Figure 5 shown.

[0077] The clusters in the damaged cluster system are divided into two parts according to whether they meet the task coverage constraint. Nodes are selected from the clusters that meet the task coverage constraint. In the clusters that meet the task coverage constraint, the intact systems are selected and their cluster balance is calculated. These systems are sorted from large to small according to their elasticity importance. When the cluster does not meet the task coverage constraint, the selection is stopped. n The first node is the node that will participate in the reconstruction.

[0078] The action space 𝒜 is the set of all node numbers when the cluster system is not damaged. Since we hope that the loss of balance is as small as possible, the reward function R It is a function of the inverse of the cluster balance.

[0079] 1.2.2 Cross-level dynamic reconstruction algorithm model based on DPPO

[0080] In order to solve the problem of dynamic reconstruction strategy generation of cluster systems, during the execution of tasks by the cluster system, dynamic reconstruction within the cluster and between clusters is realized in the face of local destruction scenarios, and the cluster performance is restored to a certain extent to continue to complete the established tasks. The present invention proposes a dynamic reconstruction method based on DPPO. DPPO not only selects reconstruction actions for a single node and evaluates its action value, but also comprehensively considers the three levels of cluster-cluster-node that an effective reconstruction strategy needs to include, analyzes the overall task goals that each node has with other nodes in the cluster, and needs to build a two-level dynamic reconstruction architecture for the cluster system. The problems analyzed by traditional reinforcement learning methods are often limited to the scale of action space and sample space. In general, tasks that are closer to the actual situation are often more complex and need to consider large-scale and continuous state space and action space. When the input data is images or sounds, it often has high dimensions, which are difficult for traditional reinforcement learning to handle. Deep reinforcement learning is to combine the high-dimensional input of deep learning with reinforcement learning. The proposed DPPO-based method is as follows. Figure 1-3 As shown, it includes three parts: cluster system dynamic reconstruction decision architecture, DPPO reinforcement learning algorithm model, and DPPO alliance learning model. First, it is necessary to build a two-level dynamic reconstruction decision architecture for the cluster system, comprehensively analyze the cluster system information and the high-dimensional situation information of the complex environment, and apply the Reinforce mechanism to estimate the reconstruction action value under the two-level reconstruction to support cross-level dynamic reconstruction strategy optimization. On the basis of the above two-level dynamic reconstruction, the reinforcement learning algorithm model and alliance learning model of DPPO are further studied. The generation network of the deviation of the reinforcement learning agent training does not have an absolute value activation function. The final deviation is generated by a two-layer super network with ReLU nonlinearity. The final loss function is:

[0081] (1.6)

[0082] in b represents the number of samples sampled from the experience pool, , θ¯ is the parameter of the target network, so the TD error can be expressed as:

[0083] (1.7)

[0084] where Q tot (target) represents the maximum value Q obtained from all actions in state s' tot According to the IGM condition, the input is the maximum action value of each agent in this state; Q tot (evaluate): status s In the case of tot ; rIndicates the reward value under the current strategy, which refers to the balance in this invention.

[0085] 1.3 Dynamic Reconfiguration Decision Algorithm Considering Task Collaboration and Topological Configuration

[0086] Considering the distributed functional collaboration of multiple clusters in the cluster, which leads to cross-level interaction in the reconstruction space, a comprehensive analysis of the two-level dynamic reconstruction within the cluster and between clusters is conducted, and a dynamic reconstruction decision-making method based on DPPO is studied. First, a cross-level dynamic reconstruction decision-making architecture for the cluster system is constructed, and then reinforcement learning and alliance learning models are designed based on DPPO to realize reconstruction strategy optimization.

[0087] 1.3.1 Cluster System Dynamically Reconstructs Decision-Making Architecture Across Layers

[0088] Build a cluster system that dynamically reconstructs decision-making architecture across levels, such as Figure 1 As shown in the figure, the multi-dimensional situation information of the three-level spatiotemporal dynamic cluster system is analyzed, and then a deep neural network model is designed to extract multi-dimensional situation features and evaluate the value of reconstruction actions. Based on the completion of the cross-level dynamic reconstruction decision object, the reinforcement learning model based on DPPO and the alliance learning model are further studied to train the neural network. In the proposed cross-level dynamic reconstruction decision architecture, a dynamic reconstruction agent is designed to use DNN to extract the multi-dimensional situation features of the cluster system, focusing on the cross-level interactions within the cluster and between clusters, and then selecting reconstruction actions for each node in the cluster. The selection of reconstruction actions can be expressed as:

[0089] (1.8)

[0090] Among them S t Represents the dynamic reconstruction process t The current multidimensional situation tensor, M t-1 Represents a cluster mobility feature set, which consists of the historical mobility features of the cluster {mov t-1 ,mov t-2 ,mov t-3}.

[0091] The historical mobility characteristics of the cluster are necessary because the mobility trends of all nodes in the cluster cannot be fully observed from the current mobility status alone. The mobility characteristics of the nodes selected by the cluster at time t can be described as mov t |[S t , M (t-1) ]=[loc t init ,loc t final ], where loc t init and loc tfinal Represents the node location feature matrix, which is described by the I×J map matrix. Each element in the node location feature matrix is ​​associated with a coordinate. If the element is 1, the associated coordinate is the selected target location. t init and loc t final There is only one element in it that is equal to 1, and the other elements are equal to 0. Node position feature matrix loc t init represents the initial position of the reconstruction action at time t, and the node position feature matrix loc t final Indicates the end point of the same reconstruction action. In addition, if t <1, then loc t init and loc t final are all zero matrices. From the nature of the defined reconstruction action, this problem is integer programming using binary variables. For each node, the node position feature matrix loc is t final Any element in can be used as its current strategy, then the number of its strategies is 2 I×J ; For the entire cluster, the total number of strategies is Therefore, the dynamic decision-making problem studied is an NP-hard problem. The entire dynamic reconstruction process includes task dispatching, local destruction generation, reconstruction action generation, and cluster balance recovery assessment. Based on the above decision-making architecture, the DPPO reinforcement learning algorithm is further designed to support cross-level dynamic reconstruction strategy optimization.

[0092] (1) Multi-dimensional situation feature extraction

[0093] In the cross-level dynamic reconstruction decision-making architecture of the constructed cluster system, we must first pay attention to the cluster system information and the multi-dimensional situation information of the complex environment, including environmental feature information such as destruction characteristics and task characteristics, as well as cluster feature information such as cluster node location characteristics and business coverage characteristics, and further convert the above cluster system feature information and complex environment feature information into mathematical language. The present invention converts these feature information into the related I×J map matrix to mathematically describe it. After determining the relative coordinate system according to the geographical location of the specific task area, the I×J map matrix is ​​determined by the selected relative coordinate system. For example, it can be defined as a 256×256 image matrix to describe the current map information. The matrix processes a 256×256 relative coordinate system into a 256×256 image grid, and each element in the matrix corresponds to a specific small square grid in the relative coordinate system. Therefore, the task area feature matrix S at time t of the dynamic reconstruction process tmission It can be expressed as:

[0094] (1.9)

[0095] The matrix element s ij m (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s ij m (t)=1 means covered, s ij m (t) = 0 represents no coverage; the service coverage characteristic matrix S at time t in the dynamic reconstruction process t cover Expressed as:

[0096] (1.10)

[0097] The matrix element s ij c (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the business area of ​​the cluster node, s ij c (t) = 1 means it is covered, s ij c (t) = 0 means not covered; the node position feature matrix S at time t in the dynamic reconstruction process t location Expressed as:

[0098] (1.11)

[0099] The matrix elements Indicates whether there is any node n in the small square grid represented by (i, j) in the relative coordinate system at time t (m , n) , =1 means existence, =0 means it does not exist; the characteristic matrix S of the damaged area at time t in the dynamic reconstruction process t damage Expressed as:

[0100] (1.12)

[0101] The matrix element s ij d (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the destruction area, s ij d (t)=1 means covered, sij d (t)=0 means not covered.

[0102] Based on the above description, the multidimensional situation tensor S at time t in the dynamic reconstruction process in formula (1.8) is t It can be expressed as:

[0103] S t =[ S t mission , S t damage , S t location , S t cover ](1.13).

[0104] (2) Reconstruction strategy generation

[0105] During dynamic reconstruction t At this moment, the Agent is dynamically reconstructed to move the feature set M in the cluster system t-1 ={mov t-1 ,mov t-2 ,mov t-3} and the multidimensional situation tensor S t As input, and output the reconstruction action act at the current moment t｜mn During the entire dynamic reconstruction process, a series of dynamic reconstruction actions are generated by the dynamic reconstruction agent. All nodes in the cluster system execute the dispatched reconstruction actions in sequence or simultaneously to complete the redeployment of the cluster topology configuration, so as to improve the cluster balance and restore it to a performance level that can continue to execute the current task. The above dynamic reconstruction action set can be expressed as:

[0106] (1.14)

[0107] Among them t｜mn Indicates cluster node n (m , n) During dynamic reconstruction t The reconstruction action performed at the moment, and the reconstruction action is defined as , which means cluster node n (m，n) During dynamic reconstruction t The position of the moment in the relative coordinate system from the previous moment Move to the current position , the defined reconstruction action act t｜mn According to the cluster node n described in formula (1.8) (m , n) The moving feature mov t |[S t , Mt ] is transformed; further cluster c m The dynamic reconstruction action set is expressed as:

[0108] (1.15).

[0109] Based on the above description of the input and output of the dynamic reconstruction agent, a DNN model f is designed for the dynamic reconstruction agent. θ , to generate reconstruction actions, apply Res Net to extract features of cluster system information and environmental information, apply LSTM to process the temporal features of high-dimensional information, apply Attention mechanism to focus on task area and node position, apply Reinforce with Baseline principle to estimate the action value of the entire reconstruction process, and use value network as its baseline function only when training the neural network. The designed DNN model for dynamic reconstruction of cluster system is a multi-input and multi-output neural network. As a reconstruction strategy generation model, the network parameters θ0 are randomly initialized. The DNN in the dynamic reconstruction agent uses a ResNet module composed of 5 convolutional layers with ReLU functions to process the multi-dimensional situation input feature matrix, including the task area feature matrix S t mission , business coverage feature matrix S t cover , node position feature matrix S t location , damage area characteristic matrix S t damage , cluster system mobile feature set M t-1 ={mov t-1 ,mov t-2 ,mov t-3}. After the destruction information and task information are decoded by their respective ResNet modules, they are input into the scene kernel composed of residual MLP modules for processing, while the business coverage information, historical reconstruction action information and cluster node location information are decoded by their respective ResNet modules and input into the cluster kernel composed of LSTM modules for processing. The outputs of the scene kernel and the cluster kernel serve as part of the input of the value network. The other part of the input of the value network is the baseline feature of the entire dynamic reconstruction process. This feature tensor is composed of the reconstruction action set of the entire dynamic reconstruction process. The value network evaluates the cluster balance loss of the entire dynamic reconstruction process and is only used when training the parameters of the entire DNN neural network. In the dynamic reconstruction process, after the reconstruction agent uses the DNN model to input the above-mentioned multi-dimensional situation features, it first outputs an I×J probability matrix through an Attention module to select S t locationAs the object for performing the reconstruction action at this moment, the Attention module inputs the processing information of the scene kernel and the cluster kernel, and inputs the reconstruction action features and cluster node position features through the skip connection of the residual mechanism. The ResNet module that previously processed the task information generates a task information input feature through the residual connection, and embeds the I×J probability matrix output by the above Attention module into it, and then inputs it to a new Attention module for processing, and then outputs an I×J probability matrix to select S t mission The task area in the dataset is used as the target task area. The ResNet module that previously processed the cluster node position information generates a pole position information input feature through a residual connection. The feature information describing the target task area is embedded in an MLP module and input into a Deconv residual module. The module then outputs an I×J probability matrix to select the target position of the reconstructed action.

[0110] The designed DNN model includes convolutional modules, residual modules, batch normalization modules, ReLU function modules, MLP modules, and Attention modules. The input multi-dimensional situation features are processed by multiple deeply stacked residual modules. The middle layer of the deep neural network is constructed by stacking residual modules, including the following structure:

[0111] a) Situation feature processing module:

[0112] ① A convolutional layer consisting of 128 3×3 filters with a stride of 1;

[0113] ②Batch normalization layer;

[0114] ③ReLU function layer;

[0115] ④ A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0116] ⑤Batch normalization layer;

[0117] ⑥Addition module: Add the input feature tensor of the residual module to the output feature tensor of the previous layer;

[0118] ⑦ReLU function layer;

[0119] The scenario core consists of a three-layer MLP, the cluster core consists of an LSTM layer, and the output includes a value module, a reconstruction node module, a target task area module, and a target location module:

[0120] b) Value Module:

[0121] ① A convolutional layer consisting of two 1×1 filters with a stride of 1;

[0122] ② A linear fully connected layer with a size of 256;

[0123] ③ReLU function layer;

[0124] ④Linear fully connected layer;

[0125] ⑤Sigmoid function layer;

[0126] c) Refactor the node module:

[0127] ① A convolutional layer consisting of two 1×1 filters with a stride of 1;

[0128] ② A linear fully connected layer with a size of 256;

[0129] ③ReLU function layer;

[0130] ①Multi-head Attention layer;

[0131] ②ReLU function layer;

[0132] ③Fully connected output layer: outputs a feature vector of size I×J, corresponding to the selection of S t location The probability of hitting a certain UAV p The logit number of ;

[0133] d) Target mission area module:

[0134] ① A convolutional layer consisting of two 1×1 filters with a stride of 1;

[0135] ② A linear fully connected layer with a size of 256;

[0136] ③ReLU function layer;

[0137] ④Multi-head Attention layer;

[0138] ⑤ReLU function layer;

[0139] ⑥Fully connected output layer: outputs a feature vector of size I×J, corresponding to the selection of S t mission The probability of a task area being the target task area p The logit number of ;

[0140] e) Target location module:

[0141] ① A convolutional layer consisting of 128 3×3 filters with a stride of 1;

[0142] ②Batch normalization layer;

[0143] ③ReLU function layer;

[0144] ④ A linear fully connected layer with a size of 256;

[0145] ⑤ReLU function layer;

[0146] ⑥Fully connected output layer: Outputs a feature vector of size I×J, corresponding to the probability of selecting the target position of the reconstructed action p The logit number of ;

[0147] The network parameters of the above DNN model are θ , initial network parameters θ 0 is obtained by random generation.

[0148] 1.3.2 Dynamic Reconfiguration Strategy Optimization Based on DPPO

[0149] Based on the cross-level dynamic reconstruction decision-making architecture of the cluster system constructed in the previous section, we design a reinforcement learning algorithm based on DPPO and a federated learning algorithm model to train the DNN model in the dynamic reconstruction agent to achieve dynamic reconstruction strategy optimization.

[0150] (1) Reinforcement learning algorithm based on DPPO

[0151] Based on the cross-level dynamic reconstruction decision architecture in the previous section, after completing each dynamic reconstruction process through simulation, a set of multi-dimensional situation data and corresponding reconstruction strategy data can be generated. Then, it is necessary to evaluate the dynamic reconstruction process, calculate the cluster balance after the reconstruction, and the degree of loss of cluster balance during the entire reconstruction process, so as to comprehensively evaluate the entire dynamic reconstruction process and design a reward function based on this. For each dynamic reconstruction action act t｜mn Once the reconstruction action is completed, it is considered that the mobile node starts to perform the current task at the new location, and then the service coverage feature matrix S can be updated. t cover According to formula (1.4), the current cluster balance can be calculated. The goal of the cluster system is to improve its cluster balance as efficiently as possible. Therefore, the reward function should be defined for cluster balance and reconstruction time as follows:

[0152] (1.16)

[0153] where R t Represents the reward value at time t in the dynamic reconstruction process, τ t+ζ represents the cumulative reconstruction time at the time (t+ζ) of the dynamic reconstruction process, τ t+ζ-1 represents the cumulative reconstruction time at the time (t+ζ-1) of the dynamic reconstruction process, ε bthr represents the cluster system balance threshold, δ represents the discount factor of the reward function, τ T Represents the cumulative reconstruction time (TTFR) to complete the entire reconstruction process. Based on the reward function and the dataset of input and output feature information of the dynamic reconstruction agent during the reconstruction process, a reinforcement learning algorithm is designed, involving technologies such as policy gradient, experience replay, TD(𝝀), V-trace and UPGO, such as Figure 2 shown.

[0154] Reward the dynamic reconstruction strategy and train the parameters of the DNN model through the sample data set θ i For the next policy iteration process i +1, generate new initial multi-dimensional situation feature information, and then use the dynamic reconstruction process to θ i The DNN model as a parameter generates a new reconstruction strategy. For the first policy iteration process, the dynamic reconstruction process uses the parameters θ The DNN model is randomly initialized with 0, which aims to simultaneously approximate the value function of the reconstruction strategy and multiple policy functions describing the reconstruction actions. The DNN is trained during the policy iteration process of dynamic reconstruction. When performing the dynamic reconstruction process, the reconstruction agent selects a series of reconstruction actions to generate a complete strategy. The reconstruction agent needs to output three probability matrices for each selected reconstruction action, and the reward R of the reconstruction action is used as the output. t The key idea of ​​the reinforcement learning algorithm is to iteratively perform a dynamic reconstruction process during policy iteration: training the DNN parameters so that the reconstruction probabilities and values ​​predicted by the neural network are closer to the reconstruction policy and reward. By maximizing the similarity between the DNN-predicted probability of the reconstructed action and the reconstructed policy, the policy gradient can be expressed as:

[0155] (1.17)

[0156] (1.18)

[0157] where r(s t , a t )express t The reward value of the reconstructed action at each moment, s t express t The state at the moment, a t express t The reconstruction action at the moment, V(s t ) represents the average value, A(s t , a t) represents the action advantage value; asynchronous update based on experience replay technology, by optimizing the strategy π θ The expected reward J(π θ ) is maximized, which can be expressed as:

[0158] (1.19)

[0159] In addition, the DNN is adjusted to minimize the gap between the predicted value and the reward value of the reconstruction strategy. TD(𝝀), V-trace and UPGO training are applied to update the neural network to push the strategy to a trajectory better than the average return. When the dynamic reconstruction process is performed in the next policy iteration, the dynamic reconstruction agent uses the new DNN parameters to better search for the optimal reconstruction action.

[0160] (2) DPPO-based federated learning model

[0161] In order to realize the distributed learning and training of DPPO, a coalition learning model is designed, coalition games and a virtual self-learning mechanism considering priorities are defined, and three types of agents are used for learning and training to achieve dynamic reconstruction strategy optimization, such as Figure 3 shown.

[0162] Game definition: Randomly generated n match Multi-objective tasks and destruction, a single game consists of two agents generating reconstruction strategies for a task, and the one with the smaller loss of cluster balancing ability wins. n match Game win rate

[0163] Prioritized Virtual Self-Learning (PFSP): Virtual Self-Learning (SP) archives the alliance at regular intervals and randomly and evenly selects opponents from the alliance to play against the agent being trained. Prioritization refers to giving priority to opponents with a higher win rate in the alliance archive.

[0164] The three types of agents are the master agent, master explorer, and alliance explorer. The master agent: selects opponents from the alliance archive based on PFSP to play against, and regularly saves the game. The master explorer: plays against the master agent of the current iteration and reinitializes regularly. The alliance explorer: defeats all opponents in the alliance with a certain win rate, saves the game to the alliance, and reinitializes regularly otherwise.

[0165] 2.3 Dynamic Reconstruction Case Analysis

[0166] The present invention takes the dynamic reconstruction of the task coordination and topological configuration of the swarm UAV system as an example (such as Figure 4As shown in the figure, the proposed method is used to generate a reconstruction strategy and verify the effectiveness of the method. For cluster systems with formation characteristics, they are usually organized into multiple formations during mission execution, and it is necessary to focus on the dynamic coordination characteristics within the formation and between formations. During the mission of a multi-formation cluster system, in response to emergencies such as local damage, the multi-formation cluster system can restore the cluster performance to a certain extent through two-level dynamic reconstruction of internal formation functions and inter-formation functions to continue to complete the established mission. For this type of dynamic reconstruction decision-making problem under the condition of functional coordination of multi-formation cluster systems, the present invention solves the problem based on the proposed DPPO method and conducts relevant discussions and analyses.

[0167] 2.3.1 Drone Swarm Case Study

[0168] To study the dynamic reconfiguration process of drone swarms, this paper establishes a method for evaluating the operational destruction and recovery performance of drone swarms by analyzing the drone swarm system and mission processes. Furthermore, based on the proposed DPPO dynamic reconfiguration decision-making method, a reconfiguration strategy is generated to complete the swarm formation configuration redeployment. First, the composition, control structure, and mission execution process of the drone swarm system are analyzed to confirm the control method and mission execution objectives of the drone swarm system. Second, the mission execution results of the drone swarm are evaluated based on the type and objectives of the drone swarm mission. Finally, the resilience and resilience importance of the drone swarm during the reconfiguration process are defined through performance evaluation indicators, completing the establishment of a comprehensive evaluation method for the drone swarm reconfiguration process and results.

[0169] Generally speaking, the UAV subsystem, ground station subsystem, communication link, and ground maintenance and support subsystem are the main components of an intelligent UAV swarm system. The UAV subsystem (UAV platform), as the core component of the swarm system, is the primary vehicle for the intelligent UAV swarm to execute its missions. It primarily consists of an intelligent unit, flight control and navigation system, power unit, payload, and airframe. The ground station subsystem primarily comprises a communication link ground terminal, a command and control system, an intelligence processing station, and a take-off and landing control system. Its primary function is to manage UAV take-off and landing, as well as command and control. The communication link is a crucial component of the intelligent UAV swarm, forming a complex "machine-station-chain" system for information exchange and coordinated control. It serves as the information transmission medium for the ground station subsystem. The communication link contains information such as communication protocols and information transmission methods. This information enables data download and command upload. Furthermore, the maintenance and support subsystem, which performs maintenance and support for critical equipment within each subsystem, is not considered or analyzed in this study.

[0170] When a drone swarm is performing a mission, each drone in each formation must be capable of autonomous flight and must work collaboratively with other drones in the formation. To achieve this collaborative work, the control and information relationships between each entity in the drone formation must be determined both physically and logically. To achieve this, the drone swarm must ensure that all drones can participate in the mission, while maintaining a certain distance between them and forming a relatively stable formation structure. While ensuring this stability, in actual combat environments, the formation may also need to be dynamically adjusted based on the actual situation and specific mission requirements.

[0171] 2.3.2 UAV Swarm Mission and Damage Analysis

[0172] First, the task of the UAV cluster is analyzed. The case of this invention takes the multi-target joint reconnaissance task of the UAV cluster as an example. Considering the multi-target reconnaissance task includes M Different reconnaissance mission areas, each of which is represented by an irregular polygonal area. For the multi-target joint reconnaissance mission facing the UAV cluster, a team of M The UAV cluster S is composed of formations, and the formation set of the cluster S is represented as C. c m Indicates dispatch to a certain reconnaissance mission area A m Consider each UAV formation c m By N m UAVs, then the UAV formation c m It can be represented as a formation UAV set, where n∈{1, 2, ..., N m} indicates that the UAV is in a drone formation c m The dynamic number in N m Forming a drone fleet c m The total number of UAVs is based on the size of the reconnaissance mission area A m OK. The black dot icon represents a UAV. The blue semi-transparent circular area of ​​each UAV represents the UAV's reconnaissance area. Multiple UAVs form a formation in a certain reconnaissance mission area.

[0173] In summary, taking the joint reconnaissance mission of drone clusters as a case study, the dispatch of joint reconnaissance missions at three levels, namely cluster, formation and single machine, is analyzed. The type of cluster mission depends to a certain extent on the intelligence level of the drone cluster. Generally speaking, a drone cluster can carry various types of mission payloads. The present invention focuses on the reconnaissance mission for research and analysis. Any drone in the cluster can independently complete the reconnaissance mission within a certain area. The scope of the reconnaissance mission is a number of circular areas with a certain point as the center and a certain length as the radius. The task coordination goal of the drone cluster is to make the redundancy of each mission area relatively balanced under the premise of completing full coverage within the mission area. The dynamic reconstruction of intelligent drones is an autonomous behavior that adapts to dynamic changes in the environment and restores the execution of tasks. Specifically, when the cluster is affected by dynamic changes such as external attacks or internal failures during the execution of the mission, the system can, under the action of internal control and reconstruction strategies, perform autonomous formation reconstruction with the goal of restoring global mission performance to ensure continuous execution of the mission. Combined with the above definition, the internal element space transfer is realized within and between each formation of the cluster. Specifically, during the execution of the task, each drone in the cluster moves from the initial position to different task areas in accordance with the global target optimization direction, and performs various tasks in the task area in the form of formation. The drone cluster is divided into multiple formations for collaborative reconnaissance within the area. The mission scene is set as a two-dimensional map plane. In order to facilitate the description of the movement process of the drone, the mission area is represented by a square grid. It is stipulated that the movement of the drone in the process of executing the task is from the center of a square grid to the center of another square grid. Different formations perform tasks in the partitions of the overall mission area. The range of reconnaissance tasks that can be completed by a drone is a circle with a radius of r, with the location of the drone as the center. uav In order to ensure the cluster balance of the UAV cluster in performing reconnaissance missions, when the UAV formation performs reconnaissance missions in a certain mission area, its formation topology configuration needs to consider the redundancy deployment of reconnaissance capabilities.

[0174] During the multi-target joint reconnaissance mission described above, a drone swarm may suffer localized damage, resulting in the destruction of some UAVs in one or more formations. In this localized damage scenario, the multi-dimensional operational situational information of a drone swarm primarily includes drone cluster information and environmental information. It also requires the identification of the attributes and functions of each drone, as shown in Table 1. During the multi-target joint reconnaissance mission, there are certain dependencies between the individual UAVs and their components. The interactive relationships between these various types of situational information allow each drone to operate relatively independently while also collaborating with one another, collaboratively describing the functionality and composition of the entire system. Environmental information, as the subject of damage, interacts with drone cluster information after damage occurs, transmitting the corresponding damage data within the swarm. Formation-level information, as a mission execution unit, is considered a whole and interacts with other formations and environmental information through internal whole-machine-level information. Within the formation-level information, multiple whole-machine-level information communicates with each other to achieve state changes. Furthermore, each whole-machine-level information contains functional-layer information representing different functions. These functional-layer information interacts with whole-machine-level information to update the parameters of the whole-machine-level information. When a drone swarm is performing its mission, it is inevitable that it will be affected by some factors and thus damaged. Only when the specific state and data of the drone swarm after the damage is known can the reconstruction strategy be formulated. Therefore, the destruction modeling of the drone swarm is of great significance for the research and formulation of the reconstruction strategy of the drone swarm.

[0175] Table 1 Operation and maintenance situation information under the situation of UAV swarm destruction

[0176]

[0177] In the drone cluster model of the present invention, a hierarchical modeling method is adopted to divide the entire drone cluster system into three levels for modeling, namely functional layer information, whole machine layer information and formation layer information. Among them, the drone functional layer information is taken as the bottom-level information, and its state is set to two states, namely normal state and destruction state. The drone whole machine layer information is the upper-level information of the drone functional layer information, and its state is determined by the functional layer information. The destruction of different functional layer information will cause the whole machine layer information to produce multiple states. The causes of destruction usually include small-scale precision fire strikes, electromagnetic wave interference, large-scale fire strikes, etc. This type of destruction will cause all UAVs in a certain area to be destroyed at the same time.

[0178] For the UAV swarm under the above-mentioned local destruction situation, during the execution of the above-mentioned multi-target joint reconnaissance mission, some UAVs in one or more formations in the UAV swarm are destroyed. Taking the impact of this destruction as input, consider that each UAV has two states: normal state and fault state. When a UAV is destroyed, it enters the fault state. The range of local destruction is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value of is randomly generated. Based on the generated local destruction range, the destruction situation of the drone cluster is analyzed.

[0179] In addition, in order to ensure the accuracy of subsequent modeling and strategy generation, it is necessary to make the arrival mode of UAV swarm destruction as consistent as possible with the actual destruction behavior during the destruction modeling process. In the process of modeling the arrival mode of destruction, the main consideration is to analyze the number of destructions during the simulation process and the time point of each destruction. The number of local destructions is modeled using the non-homogeneous Poisson process method. It is assumed that the intensity of the extreme environmental event that causes large-scale destruction during the model operation is λ S , then the number of random destructions of environmental information is k s The probability is given by the following formula:

[0180] (1.20)

[0181] where λ S (t) is the mean number of damage occurrences within the total mission execution time (0, t]. After obtaining the number of damage occurrences, the time of damage occurrence is determined in the same way as random damage. The coordinates of the damage center are determined by random sampling on the mission area plane, and the damage radius is determined based on the input parameters of the macro-environmental information. For local damage situations, in most cases, they are caused by external attacks, and the way they occur is determined by the preset parameters of the environmental information. Taking into account the randomness and uncertainty of damage, the present invention abstracts the damage into a random area with a center point, and uses a deterministic regional damage model to represent the impact of local damage on the drone cluster. The regional damage caused by local damage is abstracted into a circular area randomly placed on the physical network plane, and all drones intersecting with this area are considered to have suffered damage.

[0182] 2.3.3 Analysis of Dynamic Reconfiguration of UAV Clusters

[0183] Under the aforementioned localized destruction scenario, the dynamic reconfiguration of a UAV swarm is considered an autonomous behavior that can adapt to environmental changes to continue executing its current mission. When a UAV swarm is affected by dynamic changes during mission execution, a corresponding dynamic reconfiguration strategy must be generated to restore the swarm's reconnaissance mission performance and redeploy its topological configuration, thereby ensuring the continuity of the current multi-target joint reconnaissance mission. When localized destruction occurs, the proposed DPPO dynamic reconfiguration decision algorithm observes the destruction of the swarm and generates a reconfiguration strategy that is transmitted to each UAV. The swarm then performs dynamic reconfiguration between UAVs within the formation and between formations within the swarm. A multi-target joint reconnaissance mission consisting of seven irregular polygonal reconnaissance mission areas is randomly generated. The entire relative coordinate system map is represented as a 512×512 positive-direction grid image. The reconnaissance range of each UAV in the swarm is represented by a blue circular image. The reconnaissance area of ​​each formation must cover the corresponding white irregular polygonal mission area. The destruction state is randomly generated, with the destruction center being a randomly sampled point on the mission area and the destruction area being a randomly generated irregular polygon. The multi-target joint reconnaissance mission analyzed includes 7 mission areas. The drone cluster dispatches 7 formations, with a total of 124 drones. Table 2 shows the initial position information and UAV formation number information of the multi-formation drone cluster. The reconstruction action of each UAV in the reconstruction strategy is generated based on formula (1.8) and is determined by its initial position and terminal position, as shown in Table 2. The visualization simulation study of the drone cluster reconstruction is carried out using Python. The numerical simulation is described from the perspective of the optimal configuration of the drone cluster. The effectiveness of the proposed DPPO dynamic reconstruction decision method is verified by the reconstruction results in different scenarios. The above-mentioned drone cluster is for a quadcopter drone cluster. In addition, the proposed method is also applicable to other types of drone clusters. In addition, for the cluster consisting of 124 drones in the above 7 formations, each position on the 512×512 relative coordinate system map can be used as the choice of the terminal position of the reconstruction action of each drone. Based on the properties of the decision variables of the dynamic reconstruction decision problem, it can be seen that the total number of strategies is 2 (512×512×124) =∞.

[0184] Table 2 UAV cluster formation configuration reconstruction information

[0185]

[0186] After this dynamic process, the drone swarm completes its topological redeployment. Assuming all drones in the swarm move at 20 km / h (though the speed can be adjusted based on actual conditions), the swarm's performance exhibits a fluctuating upward trend during the dynamic reconfiguration process.

[0187] 2.3.4 Discussion and Analysis

[0188] The purpose of solving the dynamic reconfiguration decision problem of drone swarms is to generate a feasible optimization strategy. In the discussion and analysis part, an extended analysis is conducted on the performance of the proposed DPPO algorithm and the impact of various factors on the dynamic reconfiguration problem.

[0189] (1) Different decision-making algorithms

[0190] This paper evaluates the performance of the proposed DPPO method. Furthermore, the case study applies the DQN method and cooperative game (CG) approach to analyze the dynamic reconfiguration decision-making problem of a drone swarm. The algorithm is trained on a server consisting of an Intel i97980XE CPU and four RTX2080TI-A11G GPUs. The DPPO and DQN networks are trained during the dynamic reconfiguration process. During training, as described in the methods section, each simulation consists of a randomly generated multi-target joint reconnaissance mission and a localized destruction scenario, from which the algorithm generates a corresponding dynamic reconfiguration strategy.

[0191] The present invention uses the following method to evaluate the performance of each algorithm: after running every 100 episodes, the algorithm pauses and uses the current training state agent to generate reconstruction strategies for 10 test cases. The test cases remain unchanged throughout the training process. During the entire training process, each method generates reconstruction strategies for 10 test cases, calculates the cluster balance and then calculates the reward value. The average of the 10 reward values ​​is calculated as the average reward for the current test. Figure 6 The figure depicts the average reward curves of the three methods throughout the training and learning process. Since the 10 cases in the test case set are fixed, the average reward of the CG method is a constant value, so the reward curve of the CG method is a straight line. The shading around each reward curve represents the standard deviation of the 10 reward values. Throughout the training process, both the DPPO method and the DQN method executed 100,000 episodes. The reward curves of both methods show a fluctuating upward trend. In the first 17,000 episodes, the DQN method showed a faster growth trend than the DPPO method. However, after 20,000 episodes, DPPO grew to a higher reward value position than DQN. In the later stages of the training process, DPPO was significantly better than the other two methods in terms of dynamic reconstruction decision performance.

[0192] (2) Alliance learning analysis

[0193] In addition, in the learning and training process of the proposed DPPO dynamic reconstruction decision algorithm, in order to realize the distributed learning and training of DPPO, a coalition learning model for DPPO is designed. Based on the designed coalition game and the virtual self-learning mechanism considering priority, the main agent, main explorer, and coalition explorer are used for learning and training. The DPPO data used for comparative analysis are all main agent data. The main agent is the core agent of the three types of agents in the designed coalition learning. The main explorer and coalition explorer are both auxiliary agents, which support the main agent to perform better training in the entire coalition learning training, and compare and analyze the winning rates of the three types of agents in the coalition learning process.

[0194] (3) Case studies of different situations

[0195] For different multi-target joint reconnaissance missions and drone swarm sizes, the proposed DPPO reconstruction strategy was applied to the multi-formation drone swarm information from the two case studies (Tables 3 and 4) by randomly generating corresponding localized destruction scenarios. The two case studies differed in the geographic distribution of all mission areas, the sizes of individual mission areas, the overall size of the drone formations (126 and 129 drones, respectively), the swarm configurations, and the destruction states. After the reconstruction process, the redeployment of the formation configurations demonstrated that the proposed DPPO method exhibited stable performance for dynamic reconstruction decision-making under different missions and destruction scenarios. This is because the multi-target joint reconnaissance missions and localized destruction scenarios were randomly generated during the dynamic reconstruction agent training and learning process.

[0196] Table 3 Multi-formation UAV cluster information (discussion and analysis case 1)

[0197]

[0198] Table 4 Multi-formation UAV cluster information (discussion and analysis case 2)

[0199]

[0200] In addition, an extended analysis of the optimal dynamic reconstruction strategy is conducted to determine the impact of different algorithms on the optimal strategy. For the above two cases under different situations, while generating dynamic reconstruction strategies based on the proposed DPPO deep reinforcement learning method, the DQN method and the CG method are also applied to generate reconstruction strategies for the above two cases under different situations. Figure 7 As shown in Figure 2, the dynamic reconstruction strategy given by the DPPO method achieves better cluster balance ε than the other two methods. band less reconstruction time TTFR. The proposed DPPO method achieves a better dynamic reconstruction strategy solution. The DQN method and CG method may lead to local optimality. For example, during the dynamic reconstruction and redeployment process, multiple drones may spend more time executing the deployed formation configuration reconstruction strategy.

[0201] This invention is based on the proposed DPPO dynamic reconfiguration decision algorithm. In its case study, the invention demonstrates dynamic reconfiguration in a multi-formation UAV swarm under localized destruction scenarios. The case studies analyze diverse scenarios, including joint reconnaissance missions, swarm sizes, and destruction scenarios. While generating dynamic reconfiguration strategies based on the proposed DPPO deep reinforcement learning method, the invention also applies various methods to generate reconfiguration strategies for these scenarios. Simulations of dynamic reconfiguration processes under various strategies are performed on a UAV swarm reconnaissance mission. Compared to other strategies, the dynamic reconfiguration strategy proposed by the proposed DPPO method achieves higher cluster balance within a shorter reconfiguration time. Furthermore, while maintaining optimal solution quality, the proposed DPPO method also maintains a relatively high solution speed among various methods, and its solution speed remains relatively stable with increasing task size, swarm size, and destruction scale. In summary, the case study results demonstrate that the proposed DPPO dynamic reconfiguration decision algorithm provides an efficient dynamic reconfiguration solution for topological redeployment of multi-formation swarms for task collaboration, enabling the redeployment of UAV swarms under localized destruction to resume the current swarm mission.

Claims

1. A control method for a swarm drone system based on DPPO deep reinforcement learning. During the execution of a task by a swarm drone, the formation topology and drone path planning are adjusted in real time based on the generated control method. The method is characterized by: The method comprises the following steps: 1) Construction of a three-tiered structure for swarm drone systems: Build a three-tiered structure of swarm, formation, and single drone. Analyze the collaborative relationships and topological configuration characteristics between formations within a swarm and between drones within a formation to form a three-tiered dynamic control system. The establishment of this three-tiered structure includes the following steps: 1.1) Analyze the tasks, destruction and reconstruction of the three-level spatiotemporal dynamic cluster system of cluster-cluster-node, considering the multi-objective tasks including M There are different task areas, and the task area space is described as follows: （1.1） The mission area A m , m∈{1, 2, ..., M} is an irregular polygonal region; 1.2) For this multi-target mission, a team consisting of M Clusters , then the cluster Clusters It can be expressed as: （1.2） in c m Indicates dispatch to the mission area A m Clusters; consider clusters c m Depend on N m nodes, then the cluster c m Expressed as: C m ={n (m，1) ,n (m，2) ,…, n (m，n) ,…, n (m，Nm) }(1.3) in n ∈{ 1，2，...，N m } represents node n (m，n) In cluster c m The dynamic number in N m Cluster c m The total number of nodes depends on the size of the task area A m Sure; 1.3) During the execution of the multi-objective task, the cluster system may be locally damaged, resulting in the destruction of some nodes in one or more clusters. Taking the impact of this damage as input, each node is considered to have two states: normal state and fault state. When a node is damaged, it enters the fault state. The scope of the local damage is represented by a point ( i d ，j d ) as the center, with r d is a circle with a radius of ( i d ，j d )and r d The value of is randomly generated; 2) Multi-dimensional Situational Awareness and Feature Extraction: Design a deep neural network model, using the ResNet module to process mission area features, formation coverage features, drone location features, and damage area features. Utilize the LSTM module to extract temporal features of high-dimensional information, and use the Attention mechanism to focus on interactions within the formation and across layers to obtain multi-dimensional situational information. 3) Control strategy generation based on distributed proximal policy optimization (DPPO): Applying the superior Actor-Critic policy gradient algorithm, combined with experience replay technology for asynchronous updates, maximizing the expected policy reward, and using TD(λ), V-trace, and UPGO algorithms to train and update the neural network, generating a dynamic control strategy for the swarm UAV system; 4) Design and training of alliance learning mechanisms: By designing alliance games and virtual self-learning mechanisms, including three types of agents: the master agent, the master explorer, and the alliance explorer, distributed learning training is carried out to optimize the control strategy of the swarm drone system; 5) Dynamic control strategy execution: During the UAV swarm mission execution, the formation topology and UAV path planning are adjusted in real time based on the generated control strategy to ensure the continuity of mission execution and the optimization of system performance; When a cluster is affected by dynamic changes during task execution, it is necessary to generate a corresponding dynamic reconstruction strategy to achieve cluster system task performance recovery and topology redeployment, thereby ensuring the continuity of task execution; when local damage occurs, the dynamic reconstruction algorithm observes the damage situation of the cluster system and generates a reconstruction strategy to transmit to each node. The cluster system then performs dynamic reconstruction between nodes within the cluster and between clusters within the cluster; Dynamic Reconfiguration of Cluster Systems τ Moment, definition τ Cluster balance at any moment ε b ( τ ), which represents the mean square error of the redundancy values ​​of all nodes that can work normally in each cluster during the task execution phase, as shown below: （1.4） in M Represents a cluster The total number of clusters in N m Represents clusters c m The total number of nodes, express Time node n (m，n) The sum of the overlapping areas of the coverage area of ​​​​the node and the coverage areas of other nodes in the same cluster within the task area, express The average redundancy value of all nodes in each cluster at the moment; Based on the characteristics of task coordination and topological configuration, the local damage scenario during the cluster system mission is considered, and a cluster system dynamic reconstruction decision problem is constructed. The damage center and radius are described, the cluster failure scale is described, the reconfigurable node threshold of each cluster is determined, the maximum moving distance of the reconfigurable node is determined, the minimum coverage rate of the task area is specified, the cluster balance is described, and the reconstruction time threshold is determined. The optimization objectives, constraints, and control variables of the cluster system dynamic reconstruction decision problem are as follows: （1.5） in ε m Represents clusters c m The balance, ε m min Represents clusters c m The balance threshold, N m move Represents clusters c m The total number of reconfigurable nodes in N m normal Represents clusters c m The total number of nodes in normal state, d Represents the distance between two normal nodes, d min It represents the minimum allowed distance, that is, the safe distance between nodes. represents the total time of the reconstruction process, Indicates the reconstruction time threshold.

2. The control method of the swarm drone system according to claim 1, characterized in that: In step 2), the three-level structure of cluster-formation-single machine is reconstructed dynamically by designing a DNN to extract the multi-dimensional situational features of the cluster system, focusing on the cross-level interactions within the cluster and between clusters, and then selecting a reconstruction action for each node in the cluster. The selection of the reconstruction action is expressed as: （1.8） in S t Represents the dynamic reconstruction process t The multi-dimensional situation tensor at the moment, Represents a cluster mobility feature set, consisting of the historical mobility features of the cluster { mov t-1 ,mov t-2 ,mov t-3 }; t The mobility characteristics of the nodes selected by the cluster at a given moment are described as follows: ， in loc t init and loc t final Represents the node position feature matrix, through I×J Map matrix to describe.

3. The control method of the swarm drone system according to claim 1, characterized in that: The multi-dimensional situation feature extraction in step 2) includes: 2.1) Represent the task area features as a matrix to describe the coverage of the task area in the relative coordinate system; 2.2) Extract the formation’s coverage features to describe the formation’s coverage effect within the mission area; 2.3) Design a drone position feature matrix to describe the current location of the drone in the swarm; 2.4) Describe the characteristics of the destruction area, including the location of the destruction center, the destruction radius, and the status of the affected drones.

4. The control method of the swarm drone system according to claim 3, characterized in that: In step 2), the multi-dimensional situation feature information is converted into relevant I×J The map matrix is ​​used to describe it mathematically. After determining the relative coordinate system based on the geographical location of the specific mission area, I×J The map matrix is ​​determined by the selected relative coordinate system; the task area feature matrix at time t in the dynamic reconstruction process S t mission It can be expressed as: （1.9） The matrix elements s ij m ( t )express t In the relative coordinate system ( i, j ) is covered by the task area, s ij m ( t )=1 means it is covered. s ij m ( t )=0 means not covered; Dynamic reconstruction process t Business coverage feature matrix at the moment S t cover Expressed as: （1.10） The matrix elements s ij c ( t ) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the business area of ​​the cluster node. s ij c ( t )=1 means it is covered. s ij c ( t )=0 means not covered; Node position feature matrix at time t in the dynamic reconstruction process S t location Expressed as: （1.11） The matrix elements Indicates the relative coordinate system at time t ( i, j ) whether there is any node n in the small square grid represented by (m，n) , =1 means existence, =0 means it does not exist; The characteristic matrix of the damaged area at time t in the dynamic reconstruction process S t damage Expressed as: （1.12） The matrix elements s ij d ( t ) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the destruction area. s ij d ( t )=1 means it is covered. s ij d ( t )=0 means not covered; Based on the above description, the multidimensional situation tensor of the dynamic reconstruction process at time t in formula (1.8) is S t It can be expressed as: （1.13）； in, S t mission is the task area feature matrix, S t cover is the business coverage feature matrix, S t location is the node position feature matrix, S t damage is the destruction area characteristic matrix; During dynamic reconstruction t At this moment, the Agent is dynamically reconstructed to move the feature set of the cluster system M t-1= { mov t-1 ,mov t-2 ， mov t-3 } and multidimensional situation tensor S t As input, and output the reconstruction action at the current moment act t｜mn During the entire dynamic reconstruction process, a series of dynamic reconstruction actions are generated by the dynamic reconstruction agent. All nodes in the cluster system execute the dispatched reconstruction actions in sequence or simultaneously to complete the redeployment of the cluster topology configuration, so as to improve the cluster balance and restore it to a performance level that can continue to execute the current task. The dynamic reconstruction action set is expressed as: （1.14） in act t｜mn Indicates cluster node n (m，n) During dynamic reconstruction t The reconstruction action performed at the moment, and the reconstruction action is defined as , which means cluster node n (m，n) During dynamic reconstruction t The position of the moment in the relative coordinate system from the previous moment Move to the current position , the defined reconstruction action act t｜mn According to the cluster node n described in formula (1.8) (m，n) Mobile features mov t |[ S t ，M t ] is transformed; further clustering c m The dynamic reconstruction action set is expressed as: （1.15）。 5. The control method of the swarm drone system according to claim 1, characterized in that: The federated learning mechanism described in step 4) includes: 4.1) Design alliance games, randomly generate multi-objective tasks and destruction scenarios, and the main agent will play against opponents in the alliance archive; 4.2) Applying a virtual self-learning mechanism, select opponents with high win rates from the alliance archive to play against, improving the control strategy of the main agent; 4.3) Design of three types of agents, including the master agent, master explorer, and coalition explorer, to collaboratively learn and optimize strategies.

6. The control method of the swarm drone system according to claim 1, characterized in that: The execution of the dynamic control strategy in step 5) includes: 5.1) Dynamically control the formation topology of the drone swarm and adjust it in real time based on mission requirements and environmental changes; 5.2) Based on the generated control strategy, the drone path is planned in real time to optimize the mission execution efficiency; 5.3) In the event of local damage, mission performance can be restored and continued by redeploying the UAV formation.

Images