Control method of cluster unmanned aerial vehicle system based on DPPO deep reinforcement learning

Through the DPPO deep reinforcement learning method, the cluster-format-stand-stand-machine three-level structure is constructed, multi-dimensional situation information is extracted and alliance learning mechanism is designed, which solves the problem of insufficient global optimality and adaptability in the traditional method, and realizes the drone cluster. Efficient and stable task execution in complex environments.

CN119002518AActive Publication Date: 2024-11-22HANGZHOU INTERNATIONAL INNOVATION INSTITUTE OF BEIHANG UNIVERSITY

Patent Information

Application Number
CN202411479607.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-23
Publication Date
2024-11-22
Estimated Expiration
2044-10-23

AI Technical Summary

Technical Problem

Traditional drone cluster control methods are difficult to achieve global optimal control in complex and changing environments, lack adaptability and real-time, and are difficult to cope with high-dimensional state space and multi-level coordinated adjustments.

Method used

Using deep reinforcement learning method based on DPPO, a cluster-format-stand-stand-machine three-level structure is constructed, multi-dimensional situation information is extracted through deep neural networks, combined with Actor-Critic policy gradient algorithm and empirical playback technology, a alliance learning mechanism is designed to realize dynamic control strategy Generation and real-time optimization.

Benefits of technology

The global optimal control of the drone cluster in high-dimensional state space is realized, the system's adaptability and robustness are enhanced, and the task execution efficiency and stability are significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002518A_ABST
    Figure CN119002518A_ABST
Patent Text Reader

Abstract

The invention relates to an intelligent control method of an unmanned aerial vehicle system, in particular to a control method of a cluster unmanned aerial vehicle system based on DPPO deep reinforcement learning. According to the method, a cluster-formation-single machine three-level structure is constructed, the cooperative relation between the formations in the cluster and the cooperative relation between the unmanned aerial vehicles in the formations are comprehensively analyzed, and a cross-level dynamic control system is formed. A deep neural network model is utilized to extract multi-dimensional situation information including task area features, formation coverage features, unmanned aerial vehicle position features and damage area features, and a comprehensive decision basis is provided for generation of a dynamic control strategy. According to the invention, the optimal control strategy generation and real-time adjustment of the cluster unmanned aerial vehicle system in a high-dimensional state space are realized, the adaptability and robustness of the system in a complex environment are enhanced, and the task execution efficiency and stability of the cluster unmanned aerial vehicle system are significantly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an intelligent control method for an unmanned aerial vehicle system, and in particular to a control method for a clustered unmanned aerial vehicle system based on DPPO deep reinforcement learning. Background Art

[0002] In recent years, swarm UAV systems have been widely used in military reconnaissance, environmental monitoring, logistics and transportation, etc. Through the collaborative work of multiple UAVs, swarm systems can perform complex tasks in a larger space, showing advantages that a single UAV cannot achieve. However, in practical applications, swarm UAV systems usually face complex and changing environments, including dynamic changes in mission objectives, environmental uncertainties, and external interference or attacks, which puts higher requirements on the control and mission execution of swarm systems.

[0003] Traditional UAV swarm control methods are usually based on preset rules or heuristic algorithms, such as centralized control, distributed-based collaborative control, etc. These methods can achieve collaborative work between UAVs in the swarm to a certain extent, but often have the following limitations: 1. Local optimality rather than global optimality: Traditional methods usually only focus on the optimal control strategy within a local range, and lack global optimization of the entire swarm system. For multi-level UAV swarm systems, optimization of a single level may lead to a decrease in overall performance. 2. Lack of adaptability: Preset rules and heuristic algorithms are difficult to cope with the dynamic changes of complex environments and tasks, and cannot be adaptively adjusted according to real-time situation information, and are prone to failure when the environment changes or external interference occurs. 3. Insufficient real-time performance: Traditional control methods often require a long calculation and adjustment time, and it is difficult to meet the real-time control requirements of UAV swarms in complex dynamic environments.

[0004] With the development of artificial intelligence and deep learning technology, reinforcement learning (RL) has shown great potential in solving complex dynamic decision-making problems. Deep reinforcement learning (DRL) can achieve optimal decision-making in high-dimensional state space by constructing interactive learning strategies between agents and the environment, providing a new solution for the dynamic control of drone swarms. However, traditional deep reinforcement learning methods also face challenges in the application of drone swarm systems: 1. High-dimensional state space: drone swarm systems are usually composed of a large number of drones, and their state space has high dimensions and complexity. Traditional deep reinforcement learning methods are difficult to effectively learn and optimize strategies in such high-dimensional state space. 2. Multi-level coordination and dynamic adjustment: drone swarm systems involve dynamic coordination and adjustment of three levels: cluster-formation-single machine. Existing methods are difficult to achieve optimal control strategies at these three levels at the same time. 3. Real-time response and strategy generation: In practical applications, drone swarms need to generate control strategies in real time according to task requirements and environmental changes, which places high demands on the real-time performance and response speed of the algorithm. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a cluster UAV system control method based on distributed proximal policy optimization (DPPO) deep reinforcement learning. This method constructs a three-level structure of cluster-formation-single machine, comprehensively analyzes the collaborative relationship between formations within the cluster and between UAVs within the formation, and forms a cross-level dynamic control system. Using a deep neural network model, multi-dimensional situation information is extracted, including mission area characteristics, formation coverage characteristics, UAV location characteristics, and damage area characteristics, providing a comprehensive decision-making basis for the generation of dynamic control strategies.

[0006] In terms of control strategy generation, a reinforcement learning algorithm based on DPPO is adopted, and asynchronous updates are performed through the dominant Actor-Critic policy gradient algorithm combined with experience replay technology to achieve real-time optimization of the strategy. At the same time, by designing an alliance learning mechanism, including three types of agents: the main agent, the main explorer, and the alliance explorer, distributed learning training is carried out to improve the global optimality and diversity of the strategy. Finally, in the process of the drone cluster performing tasks, the formation topology structure and drone path planning are adjusted in real time based on the generated control strategy to ensure the efficiency of the system and the continuity of task execution.

[0007] The introduction of this method realizes the generation and real-time adjustment of optimal control strategies for swarm UAV systems in high-dimensional state space, enhances the adaptability and robustness of the system in complex environments, and significantly improves the mission execution efficiency and stability of swarm UAV systems.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions: A control method for a swarm drone system based on DPPO deep reinforcement learning, the method comprising the following steps: 1) Construction of the three-level structure of swarm drone system: Construct a three-level structure of cluster-formation-single machine, analyze the coordination relationship and topological configuration characteristics between formations within the cluster and between drones within the formation, and form a three-level dynamic control system; 2) Multi-dimensional situational awareness and feature extraction: Design a deep neural network model, use the ResNet module to process mission area features, formation coverage features, drone location features, and damage area features, use the LSTM module to extract the temporal features of high-dimensional information, and use the Attention mechanism to focus on the interactions within the formation and across levels to obtain multi-dimensional situational information; 3) Control strategy generation based on distributed proximal strategy optimization DPPO: Apply the advantage of Actor-Critic policy gradient algorithm, combined with experience replay technology for asynchronous update, maximize the expected policy reward, use TD(λ), V-trace and UPGO algorithms to train and update the neural network, and generate the dynamic control strategy of the swarm UAV system; 4) Design and training of alliance learning mechanism: By designing alliance games and virtual self-learning mechanisms, including three types of agents, namely the main agent, the main explorer and the alliance explorer, distributed learning training is carried out to optimize the control strategy of the swarm drone system; 5) Dynamic control strategy execution: During the execution of a mission by a drone swarm, the formation topology and drone path planning are adjusted in real time based on the generated control strategy to ensure the continuity of mission execution and optimization of system performance.

[0009] Preferably, the cluster-formation-standalone three-level structure in step 1) includes: 1.1) Analyze the task coordination characteristics of UAV swarms and build coordination relationship models within and between formations; 1.2) Establish a three-level topological structure model of the UAV swarm system, including the swarm as a whole, the formation layer, and the single-machine layer, to support dynamic control and adjustment between different levels.

[0010] Preferably, the multidimensional situation feature extraction in step 2) comprises: 2.1) Represent the task area features as a matrix to describe the coverage of the task area in the relative coordinate system; 2.2) Extract the coverage features of the formation and describe the coverage effect of the formation in the mission area; 2.3) Design the drone position feature matrix to describe the current location of the drone in the cluster; 2.4) Describe the characteristics of the destruction area, including the location of the destruction center, the destruction radius, and the status of the affected drones.

[0011] Preferably, the generation of the DPPO-based control strategy in step 3) includes: 3.1) Using the Actor-Critic policy gradient algorithm in deep reinforcement learning and combining it with experience replay technology, the control strategy of the drone cluster is updated asynchronously; 3.2) Maximize the policy reward function, consider the efficiency and robustness of the UAV swarm task execution, and optimize the control strategy; 3.3) Apply TD(λ), V-trace and UPGO algorithms to train deep neural networks to achieve optimal updates of the UAV dynamic control strategy.

[0012] Preferably, the alliance learning mechanism in step 4) includes: 4.1) Design alliance games, randomly generate multi-objective tasks and destruction scenarios, and the main agent plays against opponents in the alliance archive; 4.2) Apply the virtual self-learning mechanism to select opponents with high winning rates in the alliance archive to play against and improve the control strategy of the main agent; 4.3) Design of three types of agents, including the main agent, the main explorer and the coalition explorer, to collaboratively learn and optimize strategies.

[0013] Preferably, the execution of the dynamic control strategy in step 5) includes: 5.1) Dynamically control the formation topology of the drone swarm and adjust it in real time according to mission requirements and environmental changes; 5.2) According to the generated control strategy, the UAV path is planned in real time to optimize the task execution efficiency; 5.3) In the event of local damage, the mission performance can be restored and continued by redeploying the UAV formation.

[0014] The present invention adopts the above-mentioned technical solution, and realizes the global optimal control, real-time strategy generation, multi-level collaboration and distributed learning of the drone swarm system in high-dimensional situation space through the swarm drone system control method based on DPPO deep reinforcement learning, which greatly improves the task execution efficiency, stability and adaptability of the system in complex environments, showing significant technical advantages and broad application prospects. Specifically, it is manifested in the following aspects: 1. Globally optimal multi-level collaborative control: The present invention realizes global optimization control of the UAV cluster system by constructing a three-level structure of cluster-formation-single machine. This method not only focuses on the coordination within a single formation, but also considers the dynamic adjustment between formations and the cluster as a whole, ensuring the optimal coordination of the system at different levels. Compared with traditional local optimization methods, this global perspective control strategy significantly improves the overall task execution efficiency of the UAV cluster.

[0015] 2. Efficient multi-dimensional situational awareness and feature extraction: By designing a deep neural network model based on ResNet, LSTM and Attention mechanism, the present invention can efficiently extract multi-dimensional situational features in high-dimensional state space, including mission area features, formation coverage features, drone location features and damage area features. This comprehensive situational awareness and feature extraction provides accurate and rich information for the generation of dynamic control strategies, improving the accuracy and effectiveness of strategic decision-making.

[0016] 3. Real-time dynamic control strategy generation: Using the DPPO-based reinforcement learning algorithm model, the present invention can generate and optimize control strategies in real time during the execution of tasks by drone clusters. By asynchronously updating the neural network model, it can quickly respond to environmental changes and task requirements, and achieve real-time control and adjustment of drone clusters. This real-time performance ensures that the system can adjust task deployment and topology in a complex and dynamic environment in a timely manner, maintaining the continuity and efficiency of task execution.

[0017] 4. Enhanced adaptability and robustness: The present invention improves the adaptability and robustness of drone clusters in complex environments through real-time perception and feature extraction of multi-dimensional situation information. In response to environmental changes and local damage that may occur during mission execution, the system can generate adaptive dynamic control strategies to achieve self-adjustment and recovery of drone clusters, thereby ensuring the mission performance and stability of the system.

[0018] 5. Distributed strategy learning and optimization: By introducing the alliance learning mechanism, including the master agent, the master explorer and the alliance explorer, the present invention realizes distributed strategy learning and optimization. The distributed learning mechanism expands the strategy search space, improves the diversity and global optimality of the strategy, enables the control strategy to adapt to the needs of different tasks and environmental conditions, and enhances the execution efficiency of the system.

[0019] 6. Improve the efficiency of task execution of drone clusters: In practical applications, the present invention significantly improves the efficiency and flexibility of task execution by dynamically adjusting the formation topology and path planning of drone clusters. When the system suffers local damage, it can be reconstructed and adjusted in time to restore task performance and ensure continuous execution of tasks. This capability is particularly prominent in multi-target tasks and complex environments, improving the overall task completion efficiency of drone clusters.

[0020] 7. Strategy training efficiency and real-time performance: By utilizing the advantages of Actor-Critic strategy gradient, experience replay, TD(λ), V-trace and UPGO technologies in the DPPO reinforcement learning algorithm, the present invention realizes efficient training and real-time generation of control strategies. By maximizing the strategy reward function, it ensures rapid convergence of the strategy during training and maintains high performance in a dynamic environment, achieving a breakthrough in the system's real-time performance and task execution efficiency.

[0021] 8. Ability to cope with complex dynamic environments: The present invention enables the drone cluster system to make efficient decisions in complex dynamic environments through real-time strategy generation and execution. Using the multi-dimensional situation information extracted by the deep neural network, the system can flexibly adjust the strategy during the task execution process to ensure that the drone cluster maintains a high level of mission performance in different environmental changes and mission requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Dynamically reconstruct the decision-making framework for cluster systems.

[0023] Figure 2 It is the DPPO reinforcement learning algorithm model.

[0024] Figure 3 Learning algorithm model for DPPO alliance.

[0025] Figure 4 Dynamically reconstruct a case diagram for a swarm drone system.

[0026] Figure 5 Dynamically reconstruct the Agent-Environment architecture across levels for cluster systems.

[0027] Figure 6 Reward curves for different methods.

[0028] Figure 7 This is an analysis diagram of various reconstruction strategies for different situations. DETAILED DESCRIPTION

[0029] The present invention targets typical problems such as operation management in cluster system operation and maintenance, focuses on the representative frontier problem of this type of problem - the decision problem of "dynamic reconstruction" of cluster systems, analyzes the task coordination and topological configuration characteristics of three-level spatiotemporal dynamic cluster systems, and during the execution of tasks by cluster systems, they are usually constructed in the form of cluster-cluster-node. It is necessary to focus on the topological configuration characteristics between clusters within the cluster and between nodes within the cluster based on the needs of task coordination. In addition, cluster systems may perform tasks in some complex environments, and clusters and nodes in the cluster may fail and be damaged at any time. For emergencies such as local damage, the cluster system can restore cluster performance to a certain extent through two-level dynamic reconstruction, namely, dynamic reconstruction between clusters within the cluster and dynamic reconstruction between nodes within the cluster, so as to continue to complete the given tasks. How to reconstruct the cluster topology configuration from the three levels of cluster-cluster-node towards task collaboration to complete the current task is an important research work. Existing research usually focuses on the reconstruction within each cluster, but in practice an effective reconstruction strategy needs to include the three levels of cluster-cluster-node. Each node has the same overall task goal as other nodes in the cluster. Therefore, it is necessary to perform two-level dynamic reconstruction for cluster internal reconstruction and cluster internal reconstruction, and consider cross-level dynamic reconstruction deployment of cluster system task collaboration and topology configuration.

[0030] In order to solve the dynamic reconstruction decision problem of the above three-level spatiotemporal dynamic cluster system, the task coordination and topological configuration characteristics of the cluster system during task execution are considered, and a dynamic reconstruction decision method based on Distributed Proximal Policy Optimization (DPPO) deep reinforcement learning is studied. Figure 1-Figure 3As shown in the figure. The proposed DPPO-based method includes three parts: cluster system dynamic reconstruction decision architecture, DPPO reinforcement learning algorithm model, and DPPO alliance learning model. First, it is necessary to build a two-level dynamic reconstruction decision architecture for the cluster system, comprehensively analyze the cluster system information and the high-dimensional situation information of the complex environment, design a deep neural network, apply ResNet, LSTM, etc. for feature extraction, use the Attention mechanism to focus on the cross-level interaction within the cluster and between clusters, and use the Reinforce principle to estimate the reconstruction action value under the two-level reconstruction to support the optimization of cross-level dynamic reconstruction strategy. On the basis of the above two-level dynamic reconstruction, the reinforcement learning algorithm model and alliance learning model of DPPO are further studied. The reinforcement learning algorithm model needs to apply the policy gradient algorithm of the dominant actor-critic, and perform asynchronous updates based on the experience replay technology. By optimizing the strategy, the expectation of the reward of the reconstruction strategy is maximized, and then TD(𝝀), V-trace and UPGO are used to train and update the neural network to push the strategy to a trajectory better than the average return; in addition, in order to realize the distributed learning and training of DPPO, an alliance learning model is designed, and the alliance game and the virtual self-learning mechanism considering the priority are defined. Three types of agents are used for learning and training to realize the optimization of dynamic reconstruction strategy. Based on the proposed DPPO dynamic reconstruction decision algorithm, the present invention verifies the dynamic reconstruction of multi-formation drone clusters in the case analysis part for the dynamic reconstruction of the local destruction scenario. The case analysis results show that the proposed dynamic reconstruction decision algorithm can provide an efficient dynamic reconstruction solution in the topological configuration redeployment work of multi-formation clusters for task collaboration, and realizes the redeployment of drone clusters under local destruction to continue to perform the current cluster task.

[0031] 1.1 Description of the cluster system dynamic reconstruction problem 1.1.1 Cluster system tasks, destruction and reconstruction First, we analyze the tasks, destruction and reconstruction of the three-level spatiotemporal dynamic cluster system of cluster-cluster-node, and consider the multi-objective tasks including M There are different task areas, and the task area space is described as follows: Ω={A1, A2,…,A m ,…,A M} (1.1) The mission area A m ,m∈{1,2,...,M} is an irregular polygonal area.

[0032] For this multi-target mission, a team consisting of M If a cluster S is composed of clusters, then the cluster set C of the cluster S can be expressed as: C={ c1, c2,…,cm ,…,c M} (1.2) in c m Indicates dispatch to mission area A m clusters; consider clusters c m By N m nodes, then the cluster c m It is expressed as: C m ={n (m,1) ,n (m,2) ,…, n (m,n) ,…, n (m,Nm)} (1.3) where n∈1,2,…,N m} represents node n (m,n) In cluster c m The dynamic number in N m Cluster c m The total number of nodes, depending on the size of the task area A m Sure.

[0033] During the execution of the multi-objective task, the cluster system may be locally damaged, resulting in the destruction of some nodes in one or more clusters. Taking the impact of this damage as input, each node is considered to have two states: normal state and fault state. When a node is damaged, it enters the fault state. The scope of the local damage is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value is randomly generated.

[0034] Under the situation of local damage, the dynamic reconstruction of the cluster system is considered to be an autonomous behavior that can adapt to environmental changes to continue to perform the current task. When the cluster is affected by dynamic changes during task execution, it is necessary to generate a corresponding dynamic reconstruction strategy to achieve cluster system task performance recovery and topology redeployment, thereby ensuring the continuity of task execution. When local damage occurs, the dynamic reconstruction algorithm observes the damage situation of the cluster system and generates a reconstruction strategy to communicate to each node. The cluster system then performs dynamic reconstruction between nodes within the cluster and between clusters within the cluster.

[0035] For the time τ of the dynamic reconstruction process of the cluster system, define the cluster balance degree ε at time τ b (τ) represents the mean square error of the redundancy values ​​of all nodes that can work normally in each cluster during the task execution phase, as shown below: (1.4) Where M represents the total number of clusters in cluster S, N m Represents cluster c m The total number of nodes, ε mn (τ) represents the node n at time τ (m,n) The sum of the overlapping areas of the coverage area of ​​​​the node and the coverage areas of other nodes in the same cluster within the task area, Represents the average redundancy value of all nodes in each cluster at time τ.

[0036] 1.1.2 Dynamic Reconstruction Strategy Optimization Problem In view of the above-mentioned task coordination and topological configuration characteristics, the local damage scenario during the cluster system task is considered, and the cluster system dynamic reconfiguration decision (DRD) problem is constructed to describe the damage center and radius, describe the cluster failure scale, determine the threshold of reconfigurable nodes in each cluster, determine the maximum moving distance of reconfigurable nodes, stipulate the minimum coverage of the task area, describe the cluster balance, and determine the reconstruction time threshold. Based on this, the optimization objectives, constraints and control variables of the DRD problem are defined as follows: (1.5) where ε m Represents cluster c m The degree of balance, ε m min Represents cluster c m The balance threshold, N m move Represents cluster c m The total number of reconfigurable nodes in m normal Represents cluster c m The total number of nodes in the normal state, d represents the distance between two nodes in the normal state, d min It represents the minimum allowed distance, that is, the safe distance between nodes, τ tot represents the total time of the reconstruction process, τ thr Indicates the reconstruction time threshold.

[0037] 1.2 Principle of DPPO-based algorithm For this type of three-level spatiotemporal dynamic cluster system, the dynamic reconstruction decision problem under the conditions of task coordination and topological configuration characteristics needs to be considered based on the needs of task coordination. The topological configuration characteristics between clusters within the cluster and between nodes within the cluster need to be considered, and the cross-level dynamic reconstruction needs to be comprehensively analyzed. The DPPO method is proposed to solve this problem. DPPO is not a simple individual level for a single node to select the reconstruction action and evaluate its action value, but through comprehensive consideration of an effective reconstruction strategy to include the three levels of cluster-cluster-node, analyze the overall task goals that each node has with other nodes in the cluster, and build a two-level dynamic reconstruction decision architecture for the cluster system. Comprehensively analyze the cluster system information and complex environment high-dimensional situation information, design a deep neural network, and further study the reinforcement learning algorithm model of DPPO on the basis of the above two-level dynamic reconstruction. Through the optimization strategy, the expectation of the reconstruction strategy reward is maximized. Therefore, a dynamic reconstruction decision method is designed based on DPPO to consider the two-level dynamic reconstruction of the internal reconstruction of the cluster and the internal reconstruction of the cluster, and then perform cross-level dynamic reconstruction deployment for the task coordination and topological configuration of the cluster system.

[0038] The deep reinforcement learning idea based on DPPO that the present invention intends to adopt includes two core ideas: strategy iteration of cross-level dynamic reconstruction and cross-level dynamic reconstruction based on DPPO.

[0039] 1.2.1 Strategy Iteration for Cross-Level Dynamic Reconstruction The cross-level dynamic reconstruction strategy iteration is to dynamically reconstruct the interaction between the agent and the simulation environment, such as Figure 5 As shown. In the Agent-Environment architecture of cross-level dynamic reconstruction of cluster systems considering task collaboration and topological configuration, when the agent interacts with the environment, each agent still follows the goal of reinforcement learning, that is, to maximize the cumulative rewards that can be obtained. At this time, the change in the global state of the environment is related to the joint actions of all agents. Therefore, in the process of agent strategy learning, the impact of joint actions needs to be considered. In the process of formulating dynamic reconstruction strategies, considering that the multi-formation cluster system has too large a decision space in the task area set above, and the formations in the cluster need to be as intelligent as possible in the current environment to complete the dynamic reconstruction process of the formation in the face of sudden damage in a timely manner, a multi-agent reinforcement learning algorithm is used to complete the generation of dynamic reconstruction strategies for multi-formation cluster systems, and a part of the systems that meet the requirements in the multi-formation cluster system are selected as agents according to certain rules, and then the Agent-Environment architecture of cross-level dynamic reconstruction of multi-formation cluster systems is designed, as shown in Figure 5 shown.

[0040] The clusters in the damaged cluster system are divided into two parts according to whether they meet the task coverage constraint. Nodes are selected from the clusters that meet the task coverage constraint. In the clusters that meet the task coverage constraint, the intact systems are selected, their cluster balance is calculated, and these systems are sorted from large to small according to their elasticity importance. When the cluster does not meet the task coverage constraint, the selection is stopped. n The nodes are the nodes that will participate in the reconstruction.

[0041] The action space 𝒜 is the set of all node numbers when the cluster system is not damaged. Since we hope that the loss of balance is as small as possible, the reward function R It is a function of the inverse of cluster balance.

[0042] 1.2.2 Cross-level dynamic reconstruction algorithm model based on DPPO In order to solve the problem of generating dynamic reconstruction strategies for cluster systems, during the execution of tasks by the cluster system, dynamic reconstruction within the cluster and between clusters is realized in the face of local damage scenarios, and the cluster performance is restored to a certain extent to continue to complete the established tasks. The present invention proposes a dynamic reconstruction method based on DPPO. DPPO not only selects reconstruction actions for a single node and evaluates its action value, but also comprehensively considers the three levels of cluster-cluster-node that an effective reconstruction strategy needs to include, analyzes the overall task goals that each node has with other nodes in the cluster, and needs to build a two-level dynamic reconstruction architecture for the cluster system. The problems analyzed by traditional reinforcement learning methods are often limited to the scale of action space and sample space. In general, tasks that are closer to the actual situation are often more complex and require consideration of large-scale and continuous state space and action space. When the input data is images or sounds, it often has high dimensions, which are difficult to handle with traditional reinforcement learning. Deep reinforcement learning combines the high-dimensional input of deep learning with reinforcement learning. The proposed DPPO-based method is as follows: Figure 1-3 As shown in the figure, it includes three parts: dynamic reconstruction decision architecture of cluster system, DPPO reinforcement learning algorithm model, and DPPO alliance learning model. First, it is necessary to build a two-level dynamic reconstruction decision architecture for the cluster system, comprehensively analyze the cluster system information and high-dimensional situation information of complex environments, and apply the Reinforce mechanism to estimate the reconstruction action value under two-level reconstruction to support cross-level dynamic reconstruction strategy optimization. On the basis of the above two-level dynamic reconstruction, the reinforcement learning algorithm model and alliance learning model of DPPO are further studied. The generation network of the deviation of the reinforcement learning agent training does not have an absolute value activation function, and the final deviation is generated by a two-layer super network with ReLU nonlinearity. The final loss function is: (1.6) in brepresents the number of samples sampled from the experience pool, , θ¯ is the parameter of the target network, so the TD error can be expressed as: (1.7) Where Q tot (target) represents the maximum value Q obtained from all actions in state s' tot According to the IGM condition, the input is the maximum action value of each agent in this state; Q tot (evaluate): status s In the case of tot ; r It indicates the reward value under the current strategy, which refers to the balance degree in this invention.

[0043] 1.3 Dynamic Reconfiguration Decision Algorithm Considering Task Coordination and Topological Configuration Considering the distributed functional coordination of multiple clusters in the cluster, which leads to cross-level interaction in the reconstruction space, this paper comprehensively analyzes the two-level dynamic reconstruction within the cluster and between clusters, and studies the dynamic reconstruction decision method based on DPPO. First, a cross-level dynamic reconstruction decision architecture of the cluster system is constructed, and then reinforcement learning and federation learning models are designed based on DPPO to realize the reconstruction strategy optimization.

[0044] 1.3.1 Cluster system dynamically reconstructs decision-making architecture across levels Build a cluster system that dynamically reconstructs decision-making architecture across levels, such as Figure 1 As shown in the figure, the multi-dimensional situation information of the three-level spatiotemporal dynamic cluster system is analyzed, and then a deep neural network model is designed to extract multi-dimensional situation features and evaluate the value of reconstruction actions. On the basis of completing the cross-level dynamic reconstruction decision object, the reinforcement learning model based on DPPO and the alliance learning model training neural network are further studied. In the proposed cross-level dynamic reconstruction decision architecture, the dynamic reconstruction agent is designed to use DNN to extract the multi-dimensional situation features of the cluster system, focusing on the cross-level interactions within the cluster and between clusters, and then selecting reconstruction actions for each node in the cluster. The selection of reconstruction actions can be expressed as: (1.8) Where S t Represents the dynamic reconstruction process t The multidimensional situation tensor at the moment, M t-1 Represents a cluster mobility feature set, which consists of the historical mobility features of the cluster {mov t-1 ,mov t-2 ,mov t-3}.

[0045] The historical mobility characteristics of the cluster are necessary because the mobility trends of all nodes in the cluster cannot be fully observed only from the current mobility status. The mobility characteristics of the nodes selected by the cluster at time t can be described as mov t |[S t ,M (t-1) ]=[loc t init ,loc t final ], where loc t init and loc t final Represents the node location feature matrix, which is described by the I×J map matrix. Each element in the node location feature matrix is ​​associated with a coordinate. If the element is 1, the associated coordinate is the selected target location. t init and loc t final There is only one element in it that is equal to 1, and the other elements are equal to 0. The node position feature matrix loc t init represents the initial position of the reconstruction action at time t, and the node position feature matrix loc t final represents the end point of the same reconstruction action. In addition, if t <1, then loc t init and loc t final are all zero matrices. From the nature of the defined reconstruction action, this problem is integer programming with binary variables. For each node, the node position feature matrix loc t final Any element in can be used as its current strategy, then the number of its strategies is 2 I×J ; For the entire cluster, the total number of strategies is , so the dynamic decision-making problem studied is an NP-hard problem. The entire dynamic reconstruction process includes task dispatch and local destruction generation, reconstruction action generation, and cluster balance recovery evaluation. Based on the above decision-making architecture, the DPPO reinforcement learning algorithm is further designed to support cross-level dynamic reconstruction strategy optimization.

[0046] (1) Multi-dimensional situation feature extraction In the constructed cross-level dynamic reconstruction decision-making architecture of the cluster system, we first need to pay attention to the cluster system information and the multi-dimensional situation information of the complex environment, including environmental feature information such as destruction characteristics and task characteristics, as well as cluster feature information such as cluster node location characteristics and business coverage characteristics, and further convert the above cluster system feature information and complex environment feature information into mathematical language. The present invention converts these feature information into the related I×J map matrix to mathematically describe it. After determining the relative coordinate system according to the geographical location of the specific task area, the I×J map matrix is ​​determined by the selected relative coordinate system. For example, it can be defined as a 256×256 image matrix to describe the current map information. The matrix processes a 256×256 relative coordinate system into a 256×256 image grid, and each element in the matrix corresponds to a specific small square grid in the relative coordinate system. Therefore, the task area feature matrix S at time t of the dynamic reconstruction process t mission It can be expressed as: (1.9) The matrix element s ij m (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s ij m (t) = 1 means it is covered, s ij m (t) = 0 represents no coverage; the service coverage characteristic matrix S at time t of the dynamic reconstruction process t cover It is expressed as: (1.10) The matrix element s ij c (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the business area of ​​the cluster node, s ij c (t) = 1 means covered, s ij c (t) = 0 means not covered; the node position feature matrix S at time t in the dynamic reconstruction process t location It is expressed as: (1.11) The matrix element s ij l (t) indicates whether there is any node n in the small square grid represented by (i, j) in the relative coordinate system at time t (m,n) , s ij l(t) = 1 means there is s ij l (t) = 0 means it does not exist; the characteristic matrix S of the damaged area at time t in the dynamic reconstruction process t damage It is expressed as: (1.12) The matrix element s ij d (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s ij d (t) = 1 means it is covered, s ij d (t) = 0 means not covered.

[0047] Based on the above description, the multidimensional situation tensor S at time t in the dynamic reconstruction process in formula (1.8) t It can be expressed as: S t =[ S t mission , S t damage , S t location , S t cover ](1.13).

[0048] (2) Reconstruction strategy generation In the dynamic reconstruction process t At this moment, the Agent is dynamically reconstructed to move the feature set M in the cluster system t-1 ={mov t-1 ,mov t-2 ,mov t-3} and the multidimensional situation tensor S t As input, and output the reconstruction action act at the current moment t|mn During the entire dynamic reconstruction process, a series of dynamic reconstruction actions are generated by the dynamic reconstruction agent. All nodes in the cluster system execute the dispatched reconstruction actions in sequence or simultaneously to complete the redeployment of the cluster topology configuration, so as to improve the cluster balance and restore it to a performance level that can continue to execute the current task. The above dynamic reconstruction action set can be expressed as: (1.14) Among them act t|mn Indicates cluster node n (m,n) In the dynamic reconstruction process t The reconstruction action performed at the moment, and the reconstruction action can be defined as act t|mn =[sl i”j” (t-1), s l ij (t)], which means cluster node n (m,n) In the dynamic reconstruction process t The position s of the previous moment in the relative coordinate system l i”j” (t-1) Move to the current position s l ij (t), the defined reconstruction action act t|mn According to the cluster node n described in formula (1.8) (m,n) The moving feature mov t |[S t ,M t ] is transformed; further cluster c m The dynamic reconstruction action set is expressed as: (1.15).

[0049] Based on the above description of the input and output of the dynamic reconstruction agent, a DNN model f is designed for the dynamic reconstruction agent. θ , to generate reconstruction actions, apply Res Net to extract features from cluster system information and environmental information, apply LSTM to process the temporal features of high-dimensional information, apply Attention mechanism to focus on task area and node position, apply Reinforce with Baseline principle to estimate the action value of the entire reconstruction process, and use value network as its baseline function only when training neural network. The designed DNN model for dynamic reconstruction of cluster system is a multi-input and multi-output neural network. As the reconstruction strategy generation model, the network parameters θ0 are randomly initialized. The DNN in the dynamic reconstruction agent uses a ResNet module composed of 5 convolutional layers with ReLU function to process the multi-dimensional situation input feature matrix, including the task area feature matrix S t mission , business coverage feature matrix S t cover , node position feature matrix S t location , damage area characteristic matrix S t damage , cluster system mobile feature set M t-1 ={mov t-1 ,mov t-2 ,mov t-3}. After the destruction information and task information are decoded by their respective ResNet modules, they are input into the scene kernel composed of residual MLP modules for processing. The business coverage information, historical reconstruction action information and cluster node location information are decoded by their respective ResNet modules and input into the cluster kernel composed of LSTM modules for processing. The output of the scene kernel and the cluster kernel is used as part of the value network input. The other part of the value network input is the Baseline feature of the entire dynamic reconstruction process. This feature tensor is composed of the reconstruction action set of the entire dynamic reconstruction process. The value network evaluates the cluster balance loss of the entire dynamic reconstruction process and is only used when training the parameters of the entire DNN neural network. In the dynamic reconstruction process, after the reconstruction Agent uses the DNN model to input the above-mentioned multi-dimensional situation features, it first outputs an I×J probability matrix through an Attention module to select S t location As the object for performing the reconstruction action at this moment, the Attention module inputs the processing information of the scene kernel and the cluster kernel, and inputs the reconstruction action features and cluster node position features through the skip connection of the residual mechanism. The ResNet module that previously processed the task information generates a task information input feature through the residual connection, and embeds the I×J probability matrix output by the above Attention module into it, and then inputs it to a new Attention module for processing, and then outputs an I×J probability matrix to select S t mission A task area in is taken as the target task area. The ResNet module that previously processed the cluster node position information generates a pole position information input feature through a residual connection, and embeds the feature information describing the target task area through an MLP module, inputs it to a Deconv residual module, and then outputs an I×J probability matrix to select the target position of the reconstruction action.

[0050] The designed DNN model includes convolution module, residual module, batch normalization module, ReLU function module, MLP module, Attention module, etc. The input multi-dimensional situation features are processed by multiple deeply stacked residual modules. The middle layer of the deep neural network is constructed by stacking residual modules, including the following structure: a) Situation feature processing module: ① A convolutional layer consisting of 128 3×3 filters with a stride of 1; ②Batch normalization layer; ③ReLU function layer; ④ A convolutional layer consisting of 256 3×3 filters with a stride of 1; ⑤Batch normalization layer; ⑥Addition module: add the input feature tensor of the residual module to the output feature tensor of the previous layer; ⑦ReLU function layer; The scenario kernel consists of a three-layer MLP, the cluster kernel consists of an LSTM layer, and the output includes a value module, a reconstruction node module, a target task area module, and a target location module: b) Value Module: ① A convolutional layer consisting of 2 1×1 filters with a stride of 1; ② A linear fully connected layer with a size of 256; ③ReLU function layer; ④Linear fully connected layer; ⑤Sigmoid function layer; c) Refactor node modules: ① A convolutional layer consisting of 2 1×1 filters with a stride of 1; ② A linear fully connected layer with a size of 256; ③ReLU function layer; ①Multi-head Attention layer; ②ReLU function layer; ③Fully connected output layer: outputs a feature vector of size I×J, corresponding to the selection of S t location The probability of a certain UAV p The logit of ; d) Target mission area module: ① A convolutional layer consisting of 2 1×1 filters with a stride of 1; ② A linear fully connected layer with a size of 256; ③ReLU function layer; ④Multi-head Attention layer; ⑤ReLU function layer; ⑥Fully connected output layer: outputs a feature vector of size I×J, corresponding to the selection of S t mission The probability of a task area being the target task area p The logit of ; e) Target location module: ① A convolutional layer consisting of 128 3×3 filters with a stride of 1; ②Batch normalization layer; ③ReLU function layer; ④ A linear fully connected layer with a size of 256; ⑤ReLU function layer; ⑥Fully connected output layer: Outputs a feature vector of size I×J, corresponding to the probability of selecting the target position of the reconstructed action p The logit of ; The network parameters of the above DNN model are θ , initial network parameters θ 0 is generated randomly.

[0051] 1.3.2 Dynamic Reconstruction Strategy Optimization Based on DPPO Based on the cross-level dynamic reconstruction decision-making architecture of the cluster system constructed in the previous section, a reinforcement learning algorithm based on DPPO and a federated learning algorithm model are designed to train the DNN model in the dynamic reconstruction Agent to achieve dynamic reconstruction strategy optimization.

[0052] (1) Reinforcement learning algorithm based on DPPO Based on the cross-level dynamic reconstruction decision architecture in the previous section, after completing each dynamic reconstruction process through simulation, a set of multi-dimensional situation data and corresponding reconstruction strategy data can be generated. Then, it is necessary to evaluate the dynamic reconstruction process, calculate the cluster balance after the reconstruction, and the loss of cluster balance during the entire reconstruction process, so as to comprehensively evaluate the entire dynamic reconstruction process and design a reward function on this basis. For each dynamic reconstruction action act t|mn Once the reconstruction action is completed, it is considered that the moving node starts to perform the current task at the new location, and then the service coverage feature matrix S can be updated. t cover , according to formula (1.4), the current cluster balance can be calculated. The goal of the cluster system is to improve its cluster balance as efficiently as possible. Therefore, the reward function should be defined for cluster balance and reconstruction time as follows: (1.16) Where R t represents the reward value at time t in the dynamic reconstruction process, τ t+ζ represents the cumulative reconstruction time at the time (t+ζ) of the dynamic reconstruction process, τ t+ζ-1 represents the cumulative reconstruction time at the time (t+ζ-1) of the dynamic reconstruction process, ε b thr represents the cluster system balance threshold, δ represents the discount factor of the reward function, τ T Represents the cumulative reconstruction time (Time to finish reconfiguration, TTFR) to complete the entire reconstruction process. Based on the reward function and the data set of the input and output feature information of the dynamic reconstruction agent during the reconstruction process, a reinforcement learning algorithm is designed, involving technologies such as policy gradient, experience replay, TD(𝝀), V-trace and UPGO, such as Figure 2 shown.

[0053] Reward the dynamic reconstruction strategy and train the parameters of the DNN model through the sample data set θ i For the next policy iteration process i +1, generate new initial multi-dimensional situation feature information, and then use the dynamic reconstruction process θ i The DNN model as a parameter generates a new reconstruction strategy. For the first policy iteration process, the dynamic reconstruction process uses the parameters θ 0 randomly initialized DNN model, which aims to approximate the value function of the reconstruction strategy and multiple policy functions describing the reconstruction actions at the same time. The DNN is trained in the policy iteration process of dynamic reconstruction. When executing the dynamic reconstruction process, the reconstruction agent selects a series of reconstruction actions to generate a complete strategy. The reconstruction agent needs to output three probability matrices for each reconstruction action selected, and at the same time use the reward R of the reconstruction action as the reward. t As a value label. The main idea of ​​the reinforcement learning algorithm is to iteratively perform the dynamic reconstruction process during the policy iteration process: training the DNN parameters so that the reconstruction probability and value predicted by the neural network are closer to the reconstruction policy and reward. By maximizing the similarity between the probability of the reconstructed action predicted by the DNN and the reconstruction policy, the policy gradient can be expressed as: (1.17) (1.18) Where r(s t ,a t )express t The reward value of the reconstructed action at each moment, s t express t The state at the moment, a t express t The reconstruction action at the moment, V(s t ) represents the average value, A(s t ,a t ) represents the action advantage value; based on the experience replay technology, asynchronous update is performed by optimizing the strategy π θ The expected reward J(π θ ) is maximized, which can be expressed as: (1.19) In addition, the DNN is adjusted to minimize the gap between the predicted value and the reward value of the reconstruction strategy. TD(𝝀), V-trace and UPGO training are applied to update the neural network to push the strategy to a trajectory better than the average return. When the dynamic reconstruction process is performed in the next policy iteration, the dynamic reconstruction agent uses the new DNN parameters to better search for the optimal reconstruction action.

[0054] (2) Alliance learning model based on DPPO In order to realize the distributed learning and training of DPPO, a coalition learning model is designed, coalition games and virtual self-learning mechanisms considering priorities are defined, and three types of agents are used for learning and training to achieve dynamic reconstruction strategy optimization, such as Figure 3 shown.

[0055] Game definition: randomly generated n match In a single game, two agents generate reconstruction strategies for a task. The one with the smaller loss of cluster balance ability wins. n match Game Win Rate Priority-based virtual self-learning (PFSP): Virtual self-learning (SP) archives the alliance at regular intervals and randomly and evenly selects opponents from the alliance to play against the agent being trained; priority refers to giving priority to opponents with a higher winning rate in the alliance archive.

[0056] The three types of agents include the main agent, the main explorer, and the alliance explorer. The main agent: selects opponents in the alliance archive based on PFSP to play against, and archives regularly; the main explorer: plays against the main agent of the current iteration, and reinitializes regularly; the alliance explorer: defeats all opponents in the alliance with a certain winning rate, and archives to the alliance, otherwise it reinitializes regularly.

[0057] 2.3 Dynamic Reconstruction Case Analysis The present invention takes into account the dynamic reconstruction of the task coordination and topological configuration of the swarm UAV system as an example (such as Figure 4 As shown in the figure, the proposed method is used to generate a reconstruction strategy and verify the effectiveness of the method. For cluster systems with formation characteristics, they are usually built in multiple formations during mission execution, and it is necessary to focus on the dynamic coordination characteristics within the formation and between formations. During the mission of the multi-formation cluster system, in response to emergencies such as local damage, the multi-formation cluster system can restore the cluster performance to a certain extent through two-level dynamic reconstruction of internal formation functions and inter-formation functions to continue to complete the established tasks. For this type of dynamic reconstruction decision-making problem under the condition of functional coordination of multi-formation cluster systems, the present invention solves the problem based on the proposed DPPO method and conducts relevant discussions and analyses.

[0058] 2.3.1 Drone Swarm Case Study In order to study the dynamic reconstruction process of drone clusters, this paper will establish a method for evaluating the destruction and recovery performance of drone cluster operations by analyzing the drone cluster system and the task process, and then generate a reconstruction strategy based on the proposed DPPO dynamic reconstruction decision method to complete the cluster formation configuration redeployment. First, the composition, control structure and task execution process of the drone cluster system are analyzed to confirm the control method and task execution objectives of the drone cluster system; secondly, according to the type and goal of the drone cluster task execution, the task execution results of the drone cluster are evaluated; finally, the elasticity and elasticity importance of the drone cluster in the reconstruction process are defined through the performance evaluation index of the drone cluster, and the establishment of a comprehensive evaluation method for the process and results of drone cluster reconstruction is completed.

[0059] Generally speaking, the UAV subsystem, ground station subsystem, communication link and ground maintenance and support subsystem are the main components of the intelligent UAV cluster system. Among them, the UAV subsystem (UAV platform) as the main part of the cluster system is the main carrier for the intelligent UAV cluster to perform tasks, mainly composed of intelligent units, flight control and navigation systems, power units, mission payloads and fuselages. The ground station subsystem is mainly composed of communication link ground terminals, command and control systems, intelligence processing stations and take-off and landing control systems. Its main function is to perform UAV take-off and landing and command and control. Among them, the communication link is an important part of the "machine-station-chain" complex system that makes the intelligent UAV cluster an information interaction and collaborative control. It is the information transmission medium of the ground station subsystem. The communication link contains information such as communication protocols and information transmission methods. By using this information, data download and command upload can be realized. In addition, the system that carries out maintenance and support work for important equipment in each subsystem is the maintenance and support subsystem, which is not considered and analyzed in the research process of this invention.

[0060] When a drone swarm is performing a mission, each drone in each formation must have the ability to fly autonomously and needs to work in coordination with other drones in the formation. In order to achieve such coordination, it is necessary to determine the control and information relationships between each entity in the drone formation in both physical and logical aspects. In order to achieve the above relationship determination, the drone swarm needs to ensure that all drones can participate in the mission process during the mission execution, and all drones need to maintain a certain distance and form a relatively stable formation structure. While ensuring such stability, in the actual combat environment, it may be necessary to dynamically adjust the formation according to the actual situation and specific mission requirements.

[0061] 2.3.2 Analysis of UAV swarm missions and damage First, the task of the drone cluster is analyzed. The case of this invention takes the multi-target joint reconnaissance task of the drone cluster as an example. Considering the multi-target reconnaissance task includes M Different reconnaissance mission areas, each of which is represented by an irregular polygonal area. For multi-target joint reconnaissance missions for drone clusters, a team of M The drone cluster S consists of formations, and the formation set of the cluster S is represented by C. c m Indicates dispatch to a certain reconnaissance mission area A m Consider each drone formation c m By N m UAVs, then the UAV formation c m It can be represented as a formation UAV set, where n∈{1,2,...,N m} indicates that the UAV is in a drone formation c m The dynamic number in N m Forming a drone fleet c m The total number of UAVs is based on the size of the reconnaissance mission area A m OK. The black dot icon represents the UAV, and the blue semi-transparent circular area of ​​each UAV represents the reconnaissance area of ​​the UAV. Multiple UAVs form a formation in a certain reconnaissance mission area.

[0062] In summary, taking the joint reconnaissance mission of drone clusters as a case, the dispatch of joint reconnaissance missions at three levels of cluster-formation-single machine is analyzed. The type of cluster mission depends to a certain extent on the intelligence level of the drone cluster. Generally speaking, drone clusters can carry various types of mission payloads. The present invention focuses on the reconnaissance mission for research and analysis. Any drone in the cluster can independently complete the reconnaissance mission within a certain area. The scope of the reconnaissance mission is a number of circular areas with a certain point as the center and a certain length as the radius. The task coordination goal of the drone cluster is to make the redundant capabilities of each mission area relatively balanced on the premise of completing full coverage within the mission area. The dynamic reconstruction of intelligent drones is an autonomous behavior that adapts to dynamic changes in the environment and restores the execution of tasks. Specifically, when the cluster is affected by dynamic changes such as external attacks or internal failures during the execution of the mission, the system can, under the action of internal control and reconstruction strategies, with the goal of restoring global mission performance, perform autonomous reconstruction of the formation to ensure continuous execution of the mission. Combined with the above definition, the internal element space transfer is realized within and between each formation of the cluster. Specifically, during the task execution process, each drone in the cluster moves from the initial position to different task areas in accordance with the global target optimization direction, and performs various tasks in the task area in the form of formation. The drone cluster is divided into multiple formations for coordinated reconnaissance in the area. The mission scene is set as a two-dimensional map plane. In order to facilitate the description of the movement process of the drone, the mission area is represented by a square grid. It is stipulated that the movement of the drone during the execution of the task is to move from the center of a square grid to the center of another square grid. Different formations perform tasks in the partitions of the overall mission area. The range of reconnaissance tasks that a drone can complete is a circle with a radius of r, centered on the location of the drone. uav In order to ensure the cluster balance of the drone cluster in performing reconnaissance tasks, when the drone formation performs a reconnaissance mission in a certain mission area, its formation topology configuration needs to consider the redundancy deployment of the reconnaissance capability.

[0063] During the execution of the above multi-target joint reconnaissance mission, the drone cluster may be locally damaged, resulting in the destruction of some UAVs in one or more formations. In the case of local damage, the multi-dimensional operation and maintenance situation information of the drone cluster mainly includes drone cluster information and environmental information, and it is necessary to determine the attribute information and functions of each drone, as shown in Table 1. In the process of executing the multi-target joint reconnaissance mission, there is a certain correlation between the various UAVs and the components of the UAVs. Among them, the interactive relationship between various situation information, each drone will cooperate with each other while operating relatively independently, and jointly describe the functions and composition of the entire system. As the subject of damage, environmental information will interact with drone cluster information after the damage occurs to transmit the corresponding damage data to the cluster. The formation layer information is regarded as a whole as a task execution unit, and interacts with other formations and environmental information through the internal whole machine layer information. Within the formation layer information, multiple whole machine layer information completes the state change through communication. At the same time, each whole machine layer information has functional layer information representing different functions. The functional layer information completes the parameter update of the whole machine layer information by interacting with the whole machine layer information. The drone swarm will inevitably be affected by some factors and destroyed during the execution of the mission. Only when the specific state and data of the drone swarm after the destruction are known can the reconstruction strategy be formulated. Therefore, the destruction modeling of the drone swarm is of great significance for the formulation of the reconstruction strategy of the drone swarm.

[0064] Table 1 Operation and maintenance situation information under the situation of drone swarm destruction

[0065] In the drone cluster model of the present invention, a hierarchical modeling method is adopted to divide the entire drone cluster system into three levels for modeling, namely, functional layer information, whole machine layer information and formation layer information. Among them, the drone functional layer information is taken as the bottom-level information, and its state is set to two states, namely normal state and damage state. The drone whole machine layer information is the upper-level information of the drone functional layer information, and its state is determined by the functional layer information. The damage of different functional layer information will cause the whole machine layer information to produce multiple states. The causes of damage usually include small-scale precision fire strikes, electromagnetic wave interference, large-scale fire strikes, etc. This form of damage will cause all UAVs in a certain area to be destroyed at the same time.

[0066] For the above-mentioned UAV swarm under the local destruction situation, during the execution of the above-mentioned multi-target joint reconnaissance mission, some UAVs in one or more formations in the UAV swarm are destroyed. Taking the impact of this destruction as input, consider that each UAV has two states: normal state and fault state. When the UAV is destroyed, it enters the fault state. The scope of local destruction is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value of is randomly generated. Based on the generated local destruction range, the destruction situation of the drone cluster is analyzed.

[0067] In addition, in order to ensure the accuracy of subsequent modeling and strategy generation, it is necessary to make the arrival mode of drone cluster destruction as consistent as possible with the actual destruction behavior during the destruction modeling process. In the process of modeling the arrival mode of destruction, the main consideration is to analyze the number of destructions in the simulation process and the time point of each destruction. The non-homogeneous Poisson process method is used to model the number of local destructions, assuming that the intensity of extreme environmental events that cause large-scale destruction during the model operation is λ S , then the number of random destructions of environmental information is k s The probability is given by the following formula: (1.20) where λ S (t) is the mean of the number of times the damage occurs within the total duration of the mission execution (0, t]. After obtaining the number of times the damage occurs, the time of the damage is determined in the same way as the random damage. The coordinates of the damage center are determined by random sampling on the mission area plane, and the damage radius is determined according to the input parameters of the macro-environmental information. For the local damage situation, it is mostly caused by external attacks, and its generation method is determined by the preset parameters of the environmental information. Considering the randomness and uncertainty of the damage, the present invention abstracts the damage into a random area with a center point, uses a deterministic regional damage model to represent the impact of local damage on the drone cluster, and abstracts the regional damage caused by the local damage into a circular area randomly placed on the physical network plane, and all drones intersecting with this area are regarded as damaged.

[0068] 2.3.3 Analysis of dynamic reconstruction of drone swarms Under the above local destruction situation, the dynamic reconstruction of the UAV cluster is considered to be an autonomous behavior that can adapt to environmental changes to continue to perform the current task. When the UAV cluster is affected by dynamic changes during the task execution, it is necessary to generate a corresponding dynamic reconstruction strategy to achieve the recovery of the UAV cluster reconnaissance task performance and the redeployment of the topological configuration, so as to ensure the continuity of the current multi-target joint reconnaissance task. When local destruction occurs, the proposed DPPO dynamic reconstruction decision algorithm observes the destruction situation of the UAV cluster and generates a reconstruction strategy to each UAV. The UAV cluster then performs dynamic reconstruction between UAVs within the formation and between formations within the cluster. A multi-target joint reconnaissance task consisting of seven irregular polygonal reconnaissance task areas is randomly generated. The entire relative coordinate system map is represented as a 512×512 positive direction grid image. The reconnaissance range of each UAV in the UAV cluster is represented by a blue circular image. The reconnaissance area of ​​each formation needs to cover the corresponding white irregular polygonal task area. The destruction state is randomly generated, the destruction center is a random sampling point on the task area, and the destruction area is a randomly generated irregular polygon. The multi-target joint reconnaissance mission analyzed includes 7 mission areas. The drone cluster dispatched 7 formations, with a total of 124 drones. Table 2 shows the initial position information and UAV formation number information of the multi-formation drone cluster. The reconstruction action of each UAV in the reconstruction strategy is generated based on formula (1.8) and is determined by its initial position and terminal position, as shown in Table 2; Python is used to conduct a visual simulation study on the reconstruction of the drone cluster. The numerical simulation is described from the perspective of the optimal configuration of the drone cluster. The reconstruction results in different scenarios verify the effectiveness of the proposed DPPO dynamic reconstruction decision method. The above-mentioned drone cluster is for a quadcopter drone cluster. In addition, the proposed method is also applicable to other types of drone clusters. In addition, for the above-mentioned 7 formations consisting of a total of 124 drones, each position on the 512×512 relative coordinate system map can be used as a choice of the terminal position of each drone reconstruction action. Based on the properties of the decision variables of the dynamic reconstruction decision problem, it can be seen that the total number of strategies is 2 (512×512×124) =∞.

[0069] Table 2 UAV swarm formation configuration reconstruction information

[0070] After the above dynamic process, the drone swarm completes the redeployment of the topological configuration. Assuming that all drones in the cluster move at a speed of 20 km / h, of course, the speed of the drone can also be set according to the actual situation. During the dynamic reconstruction process, the performance of the cluster shows a fluctuating upward trend.

[0071] 2.3.4 Discussion and Analysis The purpose of solving the dynamic reconstruction decision problem of drone swarms is to generate a feasible optimization strategy. In the discussion and analysis part, an extended analysis is conducted on the performance of the proposed DPPO algorithm and the impact of various factors on the dynamic reconstruction problem.

[0072] (1) Different decision-making algorithms This paper evaluates the performance of the proposed DPPO method. In addition, the case part also applies the DQN method and the cooperative game (CG) method to analyze the dynamic reconstruction decision problem of the studied drone cluster. The learning and training of the algorithm uses a server composed of an Intel i97980XE CPU and four RTX2080TI-A11G GPUs to train the DPPO network and the DQN network during the dynamic reconstruction process. During the training process, as described in the method section, each simulation will randomly generate a multi-target joint reconnaissance task and a local destruction situation, and then the algorithm will generate the corresponding dynamic reconstruction strategy.

[0073] The present invention uses the following method to evaluate the performance of each algorithm: pause after running every 100 episodes, use the current training state agent to generate reconstruction strategies for 10 test cases, and the test cases remain unchanged during the entire training and learning process. During the entire training and learning process, each method generates reconstruction strategies for 10 cases in the test case, calculates the cluster balance and then calculates the reward value, and calculates the average of the 10 reward values ​​as the average reward of the current test. Figure 6 Describe the average reward curves of the three methods throughout the training and learning process. Since the 10 cases in the test case set are fixed, the average reward of the CG method is a constant value, so the reward curve of the CG method is a straight line. The shadow around each reward curve represents the standard deviation of the 10 reward values. During the entire training process, both the DPPO method and the DQN method executed 100,000 episodes. The reward curves of both methods show a fluctuating upward trend. In the initial 17,000 episodes, the DQN method showed a faster growth trend than the DPPO method. However, DPPO grew to a higher reward value position than DQN after 20,000 episodes, and DPPO was significantly stronger than the other two methods in terms of dynamic reconstruction decision performance in the later stage of the training process.

[0074] (2) Alliance learning analysis In addition, in the learning and training process of the proposed DPPO dynamic reconstruction decision algorithm, in order to realize the distributed learning and training of DPPO, a coalition learning model for DPPO is designed. Based on the designed coalition game and the virtual self-learning mechanism with priority considerations, the main agent, main explorer, and coalition explorer are used for learning and training. The DPPO data used for comparative analysis are all the main agent data. The main agent is the core agent of the three types of agents in the designed coalition learning. The main explorer and coalition explorer are both auxiliary agents, which support the main agent for better training in the entire coalition learning training, and compare and analyze the winning rates of the three types of agents in the coalition learning process.

[0075] (3) Case studies of different situations For different multi-target joint reconnaissance tasks and UAV cluster sizes, the corresponding local destruction situations are randomly generated for the multi-formation UAV cluster related information of the two discussion and analysis cases in Tables 3 and 4, and the DPPO reconstruction strategy is executed. The geographical distribution of all mission areas in the two groups of discussion and analysis cases is different, and the size of each mission area is different. The overall size of the UAV formations in the two groups of cases is 126 and 129 UAVs respectively, and the formation configuration of the cluster is different. In addition, their respective destruction states are different. After the reconstruction process, the redeployment results of the formation configuration show that the proposed DPPO method shows stable performance for dynamic reconstruction decision problems under different tasks and different destruction situations. This is because the multi-target joint reconnaissance tasks and local destruction situations are randomly generated during the dynamic reconstruction agent training and learning process.

[0076] Table 3 Multi-formation UAV cluster information (discussion and analysis case 1)

[0077] Table 4 Multi-formation UAV cluster information (discussion and analysis case 2)

[0078] In addition, an extended analysis of the optimal dynamic reconstruction strategy is conducted to determine the impact of different algorithms on the optimal strategy. For the above two cases under different situations, the dynamic reconstruction strategy is generated based on the proposed DPPO deep reinforcement learning method. The DQN method and the CG method are also used to generate reconstruction strategies for the above two cases under different situations. Figure 7 As shown in Figure 2, the dynamic reconstruction strategy given by the DPPO method achieves better cluster balance ε than the other two methods. band less reconstruction time TTFR. The proposed DPPO method achieves a better dynamic reconstruction strategy solution, while the DQN method and the CG method may lead to local optimality, for example, during the dynamic reconstruction and redeployment process, multiple drones may spend more time executing the deployed formation configuration reconstruction strategy.

[0079] The present invention is based on the proposed DPPO dynamic reconstruction decision algorithm. In the case analysis part, the present invention verifies the dynamic reconstruction of multi-formation UAV clusters under local destruction scenarios, analyzes cases under various situations such as different joint reconnaissance tasks, different cluster sizes, and different destruction situations, and generates dynamic reconstruction strategies based on the proposed DPPO deep reinforcement learning method. At the same time, multiple methods are applied to generate reconstruction strategies for dynamic reconstruction cases of different situations. Based on the simulation analysis of the UAV cluster reconnaissance task case, the dynamic reconstruction process under multiple strategies is executed. Compared with other strategies, the dynamic reconstruction strategy given by the proposed DPPO method can achieve higher cluster balance in a shorter reconstruction time during the reconstruction of the UAV cluster formation. In addition, while maintaining the best solution quality, the proposed DPPO method can also maintain a more efficient solution speed among multiple methods, and the solution speed shows a relatively stable state with the increase of task scale, cluster scale and destruction scale. In summary, the case analysis results show that the proposed DPPO dynamic reconstruction decision algorithm can provide an efficient dynamic reconstruction solution in the topological configuration redeployment work of multi-formation clusters for task collaboration, and realizes the redeployment of UAV clusters under local destruction to continue to perform the current cluster task.

Claims

1. A control method for a swarm drone system based on DPPO deep reinforcement learning, characterized in that: The method comprises the following steps: 1) Construction of the three-level structure of swarm drone system: Construct a three-level structure of cluster-formation-single machine, analyze the coordination relationship and topological configuration characteristics between formations within the cluster and between drones within the formation, and form a three-level dynamic control system; 2) Multi-dimensional situational awareness and feature extraction: Design a deep neural network model, use the ResNet module to process mission area features, formation coverage features, drone location features, and damage area features, use the LSTM module to extract the temporal features of high-dimensional information, and use the Attention mechanism to focus on the interactions within the formation and across levels to obtain multi-dimensional situational information; 3) Control strategy generation based on distributed proximal strategy optimization DPPO: Apply the advantage of Actor-Critic policy gradient algorithm, combined with experience replay technology for asynchronous update, maximize the expected policy reward, use TD(λ), V-trace and UPGO algorithms to train and update the neural network, and generate the dynamic control strategy of the swarm UAV system; 4) Design and training of alliance learning mechanism: By designing alliance games and virtual self-learning mechanisms, including three types of agents, namely the main agent, the main explorer and the alliance explorer, distributed learning training is carried out to optimize the control strategy of the swarm drone system; 5) Dynamic control strategy execution: During the execution of a mission by a drone swarm, the formation topology and drone path planning are adjusted in real time based on the generated control strategy to ensure the continuity of mission execution and optimization of system performance.

2. The control method of the swarm drone system according to claim 1, characterized in that: The three-level structure of cluster-formation-standalone in step 1) includes: 1.1) Analyze the task coordination characteristics of UAV swarms and build coordination relationship models within and between formations; 1.2) Establish a three-level topological structure model of the UAV swarm system, including the swarm as a whole, the formation layer, and the single-machine layer, to support dynamic control and adjustment between different levels.

3. The control method of the swarm drone system according to claim 2, characterized in that: The establishment of the cluster-formation-single machine three-level structure in step 1) specifically includes the following steps: 1.1) Analyze the tasks, destruction and reconstruction of the three-level spatiotemporal dynamic cluster system of cluster-cluster-node, considering multi-objective tasks including M There are different task areas, and the task area space is described as follows: Ω={A1, A2,…,A m ,…,A M }(1.1) The mission area A m , m∈{1,2,...,M} is an irregular polygonal region; 1.2) For this multi-target mission, a team consisting of M If a cluster S is composed of clusters, then the cluster set C of the cluster S can be expressed as: C={ c1, c2,…,c m ,…,c M }(1.2) in c m Indicates dispatch to mission area A m clusters; consider clusters c m By N m nodes, then the cluster c m It is expressed as: C m ={n (m,1) ,n (m,2) ,…, n (m,n) ,…, n (m,Nm) } (1.3) where n∈{1,2,...,N m } represents node n (m,n) In cluster c m The dynamic number in N m is cluster c m The total number of nodes is based on the size of the task area A m Sure; 1.3) During the execution of the multi-objective task, the cluster system may be locally damaged, resulting in the destruction of some nodes in one or more clusters. Taking the impact of this damage as input, each node is considered to have two states: normal state and fault state. When a node is damaged, it enters the fault state. The scope of the local damage is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value of is randomly generated; 1.4) When the cluster is affected by dynamic changes during task execution, it is necessary to generate corresponding dynamic reconstruction strategies to achieve cluster system task performance recovery and topology redeployment, thereby ensuring the continuity of task execution; When local damage occurs, the dynamic reconstruction algorithm observes the damage situation of the cluster system and generates a reconstruction strategy to transmit to each node. The cluster system then performs dynamic reconstruction between nodes within the cluster and between clusters within the cluster. For the time τ of the dynamic reconstruction process of the cluster system, define the cluster balance degree ε at time τ b (τ) represents the mean square error of the redundancy values ​​of all nodes that can work normally in each cluster during the task execution phase, as shown below: (1.4) Where M represents the total number of clusters in cluster S, N m Represents cluster c m The total number of nodes, ε mn (τ) represents the node n at time τ (m,n) The sum of the overlapping areas of the coverage area of ​​​​the node and the coverage areas of other nodes in the same cluster within the task area, Represents the average redundancy value of all nodes in each cluster at time τ; 1.5) In view of the above-mentioned task coordination and topological configuration characteristics, the local damage scenario during the cluster system task is considered, and the cluster system dynamic reconstruction decision problem is constructed. The damage center and radius are described, the cluster failure scale is described, the threshold of reconfigurable nodes in each cluster is determined, the maximum moving distance of reconfigurable nodes is determined, the minimum coverage rate of the task area is specified, the cluster balance is described, and the reconstruction time threshold is determined. The optimization objectives, constraints and control variables based on the cluster system dynamic reconstruction decision problem are as follows: (1.5) where ε m Represents cluster c m The degree of balance, ε m min Represents cluster c m The balance threshold, N m move Represents cluster c m The total number of reconfigurable nodes in m normal Represents cluster c m The total number of nodes in the normal state, d represents the distance between two nodes in the normal state, d min It represents the minimum allowed distance, that is, the safe distance between nodes, τ tot represents the total time of the reconstruction process, τ thr Indicates the reconstruction time threshold.

4. The control method of the swarm drone system according to claim 1, characterized in that: In step 2), the three-level structure of cluster-formation-standalone is designed to use DNN to extract the multi-dimensional situation characteristics of the cluster system by designing a dynamic reconstruction agent, focusing on the cross-level interactions within the cluster and between clusters, and then selecting reconstruction actions for each node in the cluster. The selection of reconstruction actions is expressed as: (1.8) Where S t Represents the dynamic reconstruction process t The multidimensional situation tensor at the moment, M t-1 Represents a cluster mobility feature set, which consists of the historical mobility features of the cluster {mov t-1 ,mov t-2 ,mov t-3 }; t The mobile characteristics of the nodes selected by the cluster at the moment are described as , where loc t init and loc t final Represents the node position feature matrix, which is described by an I×J matrix.

5. The control method of the swarm drone system according to claim 1, characterized in that: The multi-dimensional situation feature extraction in step 2) includes: 2.1) Represent the task area features as a matrix to describe the coverage of the task area in the relative coordinate system; 2.2) Extract the coverage features of the formation and describe the coverage effect of the formation in the mission area; 2.3) Design the drone position feature matrix to describe the current location of the drone in the cluster; 2.4) Describe the characteristics of the destruction area, including the location of the destruction center, the destruction radius, and the status of the affected drones.

6. The control method of the swarm drone system according to claim 5, characterized in that: In step 2), the multi-dimensional situation feature information is converted into an I×J map matrix related to it to describe it mathematically. After the relative coordinate system is determined according to the geographical location of the specific mission area, the I×J map matrix is ​​determined by the selected relative coordinate system; the dynamic reconstruction process is performed on the mission area feature matrix S at the moment t t mission It can be expressed as: (1.9) The matrix element s ij m (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s ij m (t) = 1 means it is covered, s ij m (t) = 0 represents no coverage; the service coverage characteristic matrix S at time t of the dynamic reconstruction process t cover It is expressed as: (1.10) The matrix element s ij c (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the business area of ​​the cluster node, s ij c (t) = 1 means covered, s ij c (t) = 0 means not covered; the node position feature matrix S at time t in the dynamic reconstruction process t location It is expressed as: (1.11) The matrix element s ij l (t) indicates whether there is any node n in the small square grid represented by (i, j) in the relative coordinate system at time t (m,n) , s ij l (t) = 1 means there is s ij l (t) = 0 means it does not exist; the characteristic matrix S of the damaged area at time t in the dynamic reconstruction process t damage It is expressed as: (1.12) The matrix element s ij d (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s ij d (t) = 1 means it is covered, s ij d (t) = 0 means not covered; Based on the above description, the multidimensional situation tensor S at time t in the dynamic reconstruction process in formula (1.8) t It can be expressed as: S t =[ S t mission , S t damage , S t location , S t cover ](1.13); Among them, S t mission is the task area feature matrix, S t cover is the service coverage feature matrix, S t location is the node position feature matrix, S t damage is the destruction area characteristic matrix; In the dynamic reconstruction process t At this moment, the Agent is dynamically reconstructed to move the feature set M in the cluster system t-1= {mov t-1 ,mov t-2 ,mov t-3 } and the multidimensional situation tensor S t As input, and output the reconstruction action act at the current moment t|mn ; During the entire dynamic reconstruction process, a series of dynamic reconstruction actions are generated by the dynamic reconstruction agent. All nodes in the cluster system execute the dispatched reconstruction actions in sequence or simultaneously to complete the redeployment of the cluster topology configuration, so as to improve the cluster balance and restore it to a performance level that can continue to execute the current task; the dynamic reconstruction action set is expressed as: (1.14) Among them act t|mn Indicates cluster node n (m,n) In the dynamic reconstruction process t The reconstruction action performed at the moment, and the reconstruction action can be defined as act t|mn =[s l i”j” (t-1), s l ij (t)], which means cluster node n (m,n) In the dynamic reconstruction process t The position s of the previous moment in the relative coordinate system l i”j” (t-1) Move to the current position s l ij (t), the defined reconstruction action act t|mn According to the cluster node n described in formula (1.8) (m,n) The moving feature mov t |[S t ,M t ] is transformed; further cluster c m The dynamic reconstruction action set is expressed as: (1.15)。 7. The control method of the swarm drone system according to claim 1, characterized in that: The generation of the control strategy based on DPPO in step 3) includes: 3.1) Using the Actor-Critic policy gradient algorithm in deep reinforcement learning and combining it with experience replay technology, the control strategy of the drone swarm is updated asynchronously; 3.2) Maximize the policy reward function, consider the efficiency and robustness of the UAV swarm task execution, and optimize the control strategy; 3.3) Apply TD(λ), V-trace and UPGO algorithms to train deep neural networks to achieve optimal updates of the UAV dynamic control strategy.

8. The control method of the swarm drone system according to claim 1, characterized in that: In step 3), for each dynamic reconstruction action act t|mn Once the reconstruction action is completed, it is considered that the moving node starts to perform the current task at the new location, and then the service coverage feature matrix S can be updated. t cover , according to formula (1.4), the current cluster balance is calculated. The goal of the cluster system is to improve its cluster balance as efficiently as possible; The reward function should be defined for cluster balance and reconstruction time as follows: (1.16) Where R t represents the reward value at time t in the dynamic reconstruction process, τ t+ζ represents the cumulative reconstruction time at the time (t+ζ) of the dynamic reconstruction process, τ t+ζ-1 represents the cumulative reconstruction time at the time (t+ζ-1) of the dynamic reconstruction process, ε b thr represents the cluster system balance threshold, δ represents the discount factor of the reward function, τ T Indicates the cumulative reconstruction time to complete the entire reconstruction process; Reward the dynamic reconstruction strategy and train the parameters of the DNN model through the sample data set θ i ; For the next policy iteration process i +1, generate new initial multi-dimensional situation feature information, and then use the dynamic reconstruction process θ i The DNN model as a parameter generates a new reconstruction strategy; for the first policy iteration process, the dynamic reconstruction process uses the parameters θ The DNN model is randomly initialized with 0. The model aims to approximate the value function of the reconstruction strategy and multiple policy functions describing the reconstruction actions at the same time. The DNN is trained in the policy iteration process of dynamic reconstruction. When executing the dynamic reconstruction process, the reconstruction agent selects a series of reconstruction actions to generate a complete strategy. The reconstruction agent needs to output three probability matrices for each selected reconstruction action, and at the same time use the reward R of the reconstruction action as the output. t As a value label; The reinforcement learning algorithm iteratively performs a dynamic reconstruction process during the policy iteration process: training the DNN parameters so that the reconstruction probability and value predicted by the neural network are closer to the reconstruction strategy and reward; by maximizing the similarity between the reconstruction action probability predicted by the DNN and the reconstruction strategy, the policy gradient is expressed as: (1.17) (1.18) Where r(s t ,a t )express t The reward value of the reconstructed action at each moment, s t express t The state at the moment, a t express t The reconstruction action at the moment, V(s t ) represents the average value, A(s t ,a t ) represents the action advantage value; based on the experience replay technology, asynchronous update is performed by optimizing the strategy π θ The expected reward J(π θ ) is maximized, expressed as: (1.19) Adjust the DNN to minimize the gap between the predicted value and the reward value of the reconstruction strategy, apply TD(𝝀), V-trace and UPGO training to update the neural network, and push the strategy to a trajectory better than the average return. When performing the dynamic reconstruction process in the next policy iteration, the dynamic reconstruction agent uses the new DNN parameters to better search for the optimal reconstruction action.

9. The control method of the swarm drone system according to claim 1, characterized in that: The alliance learning mechanism described in step 4) includes: 4.1) Design alliance games, randomly generate multi-target tasks and destruction scenarios, and the main agent plays against opponents in the alliance archive; 4.2) Apply the virtual self-learning mechanism to select opponents with high winning rates in the alliance archive to play against and improve the control strategy of the main agent; 4.3) Design of three types of agents, including the main agent, the main explorer and the coalition explorer, to collaboratively learn and optimize strategies.

10. The control method of the swarm drone system according to claim 1, characterized in that: The execution of the dynamic control strategy in step 5) includes: 5.1) Dynamically control the formation topology of the drone swarm and adjust it in real time according to mission requirements and environmental changes; 5.2) According to the generated control strategy, the UAV path is planned in real time to optimize the task execution efficiency; 5.3) In the event of local damage, the mission performance can be restored and continued by redeploying the UAV formation.

Citation Information

Patent Citations

  • Complex network local damage control method based on deep reinforcement learning

    CN112183777A

  • Unmanned aerial vehicle cluster efficient communication method based on multi-agent reinforcement learning

    CN113286275A

  • Unmanned aerial vehicle cooperative control training method and system based on multi-agent reinforcement learning

    CN113900445A

  • Unmanned aerial vehicle swarm ground command and control system based on multi-stage formation

    CN114415732A

  • Cooperative attack and defense confrontation decision-making method for unmanned aerial vehicle cluster

    CN115454136A

Cited By

  • Unmanned aerial vehicle cluster optimal path planning method and system

    CN119597020A

  • Multi-level collaborative decision-making method and device

    CN119847183A

  • Unmanned aerial vehicle cluster intelligent control system

    CN119987430A

  • An intelligent control system for UAV swarms

    CN119987430B

  • DPPO reinforcement learning-based dredging mechanical arm intelligent control method and system

    CN120170720A