A dynamic reconstruction decision-making method for cluster systems based on DPPO deep reinforcement learning

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning. The present invention combines a deep neural network model, distributed learning and a multi-agent reinforcement learning method. The method can extract multidimensional situational features in a high-dimensional state space, generate dynamic reconstruction strategies across levels, and achieve global optimization of strategies through alliance learning. The system's task execution efficiency, robustness and adaptability in complex environments are greatly improved, and it has significant technical advantages and broad application prospects.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning. Background Art

[0002] Cluster systems are widely used in modern applications, especially in the fields of drone clusters, sensor networks, computer clusters, etc. They can improve the overall performance and reliability of the system through the collaborative work of multiple nodes. However, in actual operation, cluster systems often face complex and changing environments, including dynamic changes in task requirements, changes in the topology between nodes, and interference or damage from the external environment. These factors put forward higher requirements on the task execution and topology of cluster systems, making dynamic reconstruction a necessary technology to ensure that cluster systems can adapt to environmental changes and continue to perform tasks.

[0003] Traditional cluster system reconstruction methods usually focus on local optimization, such as only reconstructing nodes within the cluster, ignoring cross-level global coordination. This method often fails to achieve optimal reconstruction results when faced with complex dynamic environments and multi-objective tasks. In addition, most existing methods rely on preset rules or heuristic algorithms, lack intelligent adaptive capabilities, and are difficult to cope with real-time changes and uncertainties in cluster system operation.

[0004] In recent years, deep reinforcement learning (DRL) has demonstrated its powerful ability in solving complex dynamic decision-making problems. It achieves optimal decision-making in uncertain and high-dimensional state spaces by building interactive learning strategies between agents and the environment. However, in the dynamic reconstruction problem of cluster systems, traditional DRL algorithms have the following technical problems: 1. High-dimensional state space: Cluster systems are usually composed of a large number of nodes, and their state space has high dimensions and complexity. Traditional DRL methods are difficult to effectively learn and optimize strategies in high-dimensional state spaces. 2. Cross-level dynamic reconstruction: The dynamic reconstruction of cluster systems not only involves the reconstruction of nodes within the cluster, but also includes the dynamic coordination between different clusters within the cluster. A global strategy optimization method that can cross levels is required. 3. Real-time response and optimization: In practical applications, cluster systems need to respond and adjust topological structures in real time under dynamic environments, which places high demands on the computational efficiency and real-time performance of the algorithm. 4. Strategy distributed learning: The dynamic reconstruction of cluster systems under different tasks and environments requires a distributed learning mechanism to achieve a wider range of strategy search and optimization. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a dynamic reconstruction decision method based on distributed proximal policy optimization (DPPO) deep reinforcement learning. As an advanced deep reinforcement learning algorithm, DPPO can achieve efficient strategy learning and optimization in multi-agent systems. By combining deep neural network models, distributed learning and multi-agent reinforcement learning methods, this method can extract multi-dimensional situation features in high-dimensional state space, generate dynamic reconstruction strategies across levels, and achieve global optimization of strategies through alliance learning.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0007] A cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning, characterized in that the method comprises the following steps:

[0008] 1) Construct a dynamic reconstruction decision framework for cluster systems: Analyze the task coordination and topological configuration characteristics of the three-level spatiotemporal dynamic cluster system, construct a three-level dynamic reconstruction decision framework of cluster-cluster-node, and analyze the topological configuration characteristics between clusters within the cluster and between nodes within the cluster;

[0009] 2) Multi-dimensional situation feature extraction: Design a deep neural network model, use the ResNet module to process the task area features, business coverage features, node location features, and damage area features, apply the LSTM module to extract the temporal features of high-dimensional information, and use the Attention mechanism to focus on the internal and cross-level interactions of the cluster to extract multi-dimensional situation features;

[0010] 3) DPPO-based reinforcement learning algorithm model: Apply the policy gradient algorithm of the dominant actor-critic and combine it with the experience replay technology for asynchronous update. By maximizing the expected policy reward, the neural network is updated using TDλ), V-trace and UPGO training to achieve dynamic reconstruction strategy optimization.

[0011] 4) Design of alliance learning model: Design alliance games and virtual self-learning mechanisms, including three types of agents: master agent, master explorer, and alliance explorer, conduct distributed learning training, and optimize dynamic reconstruction strategies;

[0012] 5) Dynamic reconstruction strategy generation: In the dynamic reconstruction process, the dynamic reconstruction agent is used to generate a reconstruction action set based on the multi-dimensional situation information of the cluster system, and redeploy the cluster system topology configuration to restore the task performance.

[0013] Preferably, the cluster system dynamic reconstruction decision architecture in step 1) comprehensively analyzes cluster system information and high-dimensional situation information of complex environments, and supports cross-level dynamic reconstruction strategy optimization based on task collaboration requirements.

[0014] Preferably, the multi-dimensional situation feature extraction in step 2) includes:

[0015] 1) Task area feature extraction: The task area features are expressed as a matrix to describe the task area coverage in the relative coordinate system;

[0016] 2) Service coverage feature extraction: describes the service coverage of cluster nodes, expressed as a service coverage feature matrix;

[0017] 3) Node position feature extraction: Design a node position feature matrix to represent the current position of each node in the cluster;

[0018] 4) Damage area feature extraction: describes the scope and impact of local damage, including the location and radius of the damage center.

[0019] Preferably, the deep neural network model in step 2) comprises:

[0020] 1) ResNet module: used to process mission area features, service coverage features, and damage area features, and extract multi-dimensional feature information;

[0021] 2) LSTM module: used to extract the time series features of high-dimensional information and analyze the dynamic changes of cluster systems;

[0022] 3) Attention mechanism: used to focus on the interactions within and between clusters, and support dynamic reconstruction across levels.

[0023] Preferably, the reinforcement learning algorithm model in step 3) is optimized by a policy gradient reward function, and the reward function includes:

[0024] 1) Cluster balance: represents the mean square error of the node redundancy values ​​in each cluster during the task execution phase;

[0025] 2) Reconstruction time: the cumulative time of the dynamic reconstruction process, as one of the optimization objectives;

[0026] 3) Strategy optimization: Through experience replay and policy gradient algorithm, TD (λ), V-trace and UPGO techniques are used to optimize the reconstruction strategy.

[0027] Preferably, the DPPO alliance learning model includes:

[0028] 1) Alliance game: Multi-target tasks and destruction scenarios are randomly generated, and the main agent plays against opponents in the alliance archive;

[0029] 2) Virtual self-learning mechanism: select opponents with high winning rate in the alliance archive to play games and promote the training of the main agent;

[0030] 3) Three types of agents: including the main agent, the main explorer and the alliance explorer, which work together to optimize the strategy.

[0031] Preferably, the dynamic reconstruction strategy generation includes:

[0032] 1) Dynamic reconstruction action generation: The dynamic reconstruction agent uses a deep neural network model to input multi-dimensional situation features and generate a series of reconstruction actions;

[0033] 2) Topology redeployment: Execute the dispatched reconstruction action, redeploy the topology of the cluster system, and improve the cluster balance.

[0034] The present invention adopts the above technical solution and introduces the deep reinforcement learning algorithm based on DPPO to achieve cross-level optimization, multi-dimensional feature extraction, real-time strategy generation and distributed strategy learning of dynamic reconstruction of cluster system, which greatly improves the task execution efficiency, robustness and adaptability of the system in complex environments, and has significant technical advantages and broad application prospects. Specifically, it is embodied in:

[0035] 1. Cross-level global optimization: This invention constructs a three-level dynamic reconstruction decision architecture of cluster-cluster-node, comprehensively analyzes the topological structure characteristics within the cluster system and within the cluster, and supports cross-level dynamic reconstruction strategy optimization. Compared with the traditional method of only optimizing within the cluster, this invention performs task coordination and topological configuration analysis from a global perspective, realizes collaborative optimization between different levels, and significantly improves the overall performance of the system and task execution efficiency.

[0036] 2. Efficient multi-dimensional situation feature extraction: A deep neural network model, including ResNet, LSTM and Attention mechanism, is designed to efficiently extract task area features, business coverage features, node location features and damage area features. These models can effectively capture the multi-dimensional situation information of the cluster system in the high-dimensional state space, provide a more accurate decision-making basis for the generation of dynamic reconstruction strategies, and improve the accuracy and execution effect of the reconstruction strategy.

[0037] 3. Real-time dynamic reconstruction strategy generation: Using the DPPO-based reinforcement learning algorithm model, the present invention can generate and optimize dynamic reconstruction strategies in real time. By training the deep neural network model in an asynchronous update mode, it can quickly respond to environmental changes and task requirements of the cluster system and provide real-time reconstruction decisions. This real-time performance ensures that the cluster system can be quickly adjusted and redeployed in a complex environment, improving the continuity and stability of task execution.

[0038] 4. Distributed strategy learning and optimization: By designing the DPPO alliance learning model, adopting the alliance game and virtual self-learning mechanism, the present invention realizes distributed strategy learning and optimization. Three types of agents (main agent, main explorer, and alliance explorer) collaborate to optimize the strategy, effectively improving the diversity and global optimality of the strategy. The distributed learning method enhances the breadth and depth of strategy search, and provides strong support for the adaptive reconstruction of cluster systems in different tasks and environments.

[0039] 5. Enhance the robustness and adaptability of cluster systems: This invention generates a highly adaptable dynamic reconstruction strategy for local damage and dynamic environmental changes that cluster systems may encounter during mission execution, and realizes the redeployment of cluster system topology configuration. By maximizing the strategy reward function, including indicators such as cluster balance and reconstruction time, the performance recovery capability of the cluster system under local damage conditions is effectively improved, and the robustness and adaptability of the system are enhanced.

[0040] 6. Efficient strategy training and optimization: By utilizing the advantages of Actor-Critic strategy gradient, experience replay, TD(λ), V-trace and UPGO technologies in the DPPO reinforcement learning algorithm, rapid training and optimization of dynamic reconstruction strategies are achieved. By maximizing the strategy gradient reward function, the global optimality of cluster system performance recovery and task execution efficiency is ensured, ensuring rapid convergence of strategies and optimal performance.

[0041] 7. High robustness and stability of the strategy: By converting multi-dimensional situation feature information into an I×J map matrix for mathematical description, a comprehensive analysis of features such as mission area, business coverage, node location, and damage area is achieved. During the dynamic reconstruction process, the multi-dimensional situation information extracted by the DNN model provides high robustness and stability for the generation of reconstruction strategies, ensuring the adaptability of the cluster system in different environments.

[0042] 8. Efficient task execution in complex environments: The DPPO deep reinforcement learning method of the present invention achieves efficient task execution in the complex environment of cluster systems. Through dynamic reconstruction strategy generation and execution, the cluster system can adaptively adjust the topology structure during task execution, restore and maintain a high level of task performance, and provide an effective solution for cluster system operation and maintenance in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Dynamically reconstruct the decision-making framework for cluster systems.

[0044] Figure 2 It is the DPPO reinforcement learning algorithm model.

[0045] Figure 3Learning algorithm model for DPPO alliance.

[0046] Figure 4 Dynamically reconstruct a case diagram for a swarm drone system.

[0047] Figure 5 Dynamically reconstruct the Agent-Environment architecture across levels for cluster systems.

[0048] Figure 6 Reward curves for different methods.

[0049] Figure 7 This is an analysis diagram of various reconstruction strategies for different situations. DETAILED DESCRIPTION

[0050] The present invention targets typical problems such as operation management in cluster system operation and maintenance, focuses on the representative frontier problem of this type of problem - the decision problem of "dynamic reconstruction" of cluster systems, analyzes the task coordination and topological configuration characteristics of three-level spatiotemporal dynamic cluster systems, and during the execution of tasks by cluster systems, they are usually constructed in the form of cluster-cluster-node. It is necessary to focus on the topological configuration characteristics between clusters within the cluster and between nodes within the cluster based on the needs of task coordination. In addition, cluster systems may perform tasks in some complex environments, and clusters and nodes in the cluster may fail and be damaged at any time. For emergencies such as local damage, the cluster system can restore cluster performance to a certain extent through two-level dynamic reconstruction, namely, dynamic reconstruction between clusters within the cluster and dynamic reconstruction between nodes within the cluster, so as to continue to complete the given tasks. How to reconstruct the cluster topology configuration from the three levels of cluster-cluster-node towards task collaboration to complete the current task is an important research work. Existing research usually focuses on the reconstruction within each cluster, but in practice an effective reconstruction strategy needs to include the three levels of cluster-cluster-node. Each node has the same overall task goal as other nodes in the cluster. Therefore, it is necessary to perform two-level dynamic reconstruction for cluster internal reconstruction and cluster internal reconstruction, and consider cross-level dynamic reconstruction deployment of cluster system task collaboration and topology configuration.

[0051] In order to solve the dynamic reconstruction decision problem of the above three-level spatiotemporal dynamic cluster system, the task coordination and topological configuration characteristics of the cluster system during task execution are considered, and a dynamic reconstruction decision method based on Distributed Proximal Policy Optimization (DPPO) deep reinforcement learning is studied. Figure 1-Figure 3As shown in the figure. The proposed DPPO-based method includes three parts: cluster system dynamic reconstruction decision architecture, DPPO reinforcement learning algorithm model, and DPPO alliance learning model. First, it is necessary to build a two-level dynamic reconstruction decision architecture for the cluster system, comprehensively analyze the cluster system information and the high-dimensional situation information of the complex environment, design a deep neural network, apply ResNet, LSTM, etc. for feature extraction, use the Attention mechanism to focus on the cross-level interaction within the cluster and between clusters, and use the Reinforce principle to estimate the reconstruction action value under the two-level reconstruction to support the optimization of cross-level dynamic reconstruction strategy. On the basis of the above two-level dynamic reconstruction, the reinforcement learning algorithm model and alliance learning model of DPPO are further studied. The reinforcement learning algorithm model needs to apply the policy gradient algorithm of the dominant actor-critic, and perform asynchronous updates based on the experience replay technology. By optimizing the strategy, the expectation of the reward of the reconstruction strategy is maximized, and then TD(𝝀), V-trace and UPGO are used to train and update the neural network to push the strategy to a trajectory better than the average return; in addition, in order to realize the distributed learning and training of DPPO, an alliance learning model is designed, and the alliance game and the virtual self-learning mechanism considering the priority are defined. Three types of agents are used for learning and training to realize the optimization of dynamic reconstruction strategy. Based on the proposed DPPO dynamic reconstruction decision algorithm, the present invention verifies the dynamic reconstruction of multi-formation drone clusters in the case analysis part for the dynamic reconstruction of the local destruction scenario. The case analysis results show that the proposed dynamic reconstruction decision algorithm can provide an efficient dynamic reconstruction solution in the topological configuration redeployment work of multi-formation clusters for task collaboration, and realizes the redeployment of drone clusters under local destruction to continue to perform the current cluster task.

[0052] 1.1 Description of the cluster system dynamic reconstruction problem

[0053] 1.1.1 Cluster system tasks, destruction and reconstruction

[0054] First, the task, destruction and reconstruction of the three-level spatiotemporal dynamic cluster system of cluster-cluster-node are analyzed. Considering that the multi-objective task contains M different task areas, the task area space is described as follows:

[0055] Ω={A1, A2,…,A m ,…,A M} (1.1)

[0056] Mission area A m ,m∈{1,2,...,M} is an irregular polygonal area.

[0057] For this multi-objective task, a cluster S consisting of M clusters is dispatched, and the cluster set C of the cluster S can be expressed as:

[0058] C={ c1, c2,…,c m ,…,c M} (1.2)

[0059] where c m Indicates dispatch to mission area A m clusters; consider cluster c m By N m nodes, then cluster c m It is expressed as:

[0060] C m ={node (m,1) ,node (m,2) ,…, node (m,n) ,…, node (m,Nm)} (1.3)

[0061] n∈{1,2,…,N m} represents node (m,n) In cluster c m The dynamic number in N m Cluster c m The total number of nodes, depending on the size of the task area A m Sure.

[0062] During the execution of the multi-objective task, the cluster system may be partially damaged, resulting in the destruction of some nodes in one or more clusters. Taking the impact of this damage as input, consider that each node has two states: normal state and fault state. When a node is damaged, it enters the fault state. The scope of the local damage is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value is randomly generated.

[0063] Under the situation of local damage, the dynamic reconstruction of the cluster system is considered to be an autonomous behavior that can adapt to environmental changes to continue to perform the current task. When the cluster is affected by dynamic changes during task execution, it is necessary to generate a corresponding dynamic reconstruction strategy to achieve cluster system task performance recovery and topology redeployment, thereby ensuring the continuity of task execution. When local damage occurs, the dynamic reconstruction algorithm observes the damage situation of the cluster system and generates a reconstruction strategy to communicate to each node. The cluster system then performs dynamic reconstruction between nodes within the cluster and between clusters within the cluster.

[0064] For the time τ of the dynamic reconstruction process of the cluster system, define the cluster balance degree ε at time τ b (τ) represents the mean square error of the redundancy values ​​of all nodes that can work normally in each cluster during the task execution phase, as shown below:

[0065] (1.4)

[0066] Where M represents the total number of clusters in cluster S, N m Represents cluster c m The total number of nodes, ε mn (τ) represents the node at time τ (m,n) The sum of the overlapping areas of the coverage area of ​​the node and the coverage areas of other nodes in the same cluster within the task area Represents the average redundancy value of all nodes in each cluster at time τ.

[0067] 1.1.2 Dynamic Reconstruction Strategy Optimization Problem

[0068] In view of the above-mentioned task coordination and topological configuration characteristics, the local damage scenario during the cluster system task is considered, and the cluster system dynamic reconfiguration decision (DRD) problem is constructed to describe the damage center and radius, describe the cluster failure scale, determine the threshold of reconfigurable nodes in each cluster, determine the maximum moving distance of reconfigurable nodes, stipulate the minimum coverage of the task area, describe the cluster balance, and determine the reconstruction time threshold. Based on this, the optimization objectives, constraints and control variables of the DRD problem are defined as follows:

[0069] (1.5)

[0070] where ε m Represents cluster c m The degree of balance, ε m min Represents cluster c m The balance threshold, N m move Represents cluster c m The total number of reconfigurable nodes in m normal Represents cluster c m The total number of nodes in the normal state, d represents the distance between two nodes in the normal state, d min It represents the minimum allowed distance, that is, the safe distance between nodes, τ tot represents the total time of the reconstruction process, τ thr Indicates the reconstruction time threshold.

[0071] 1.2 Principle of DPPO-based algorithm

[0072] For this type of three-level spatiotemporal dynamic cluster system, the dynamic reconstruction decision problem under the conditions of task coordination and topological configuration characteristics needs to be considered based on the needs of task coordination. The topological configuration characteristics between clusters within the cluster and between nodes within the cluster need to be considered, and the cross-level dynamic reconstruction needs to be comprehensively analyzed. The DPPO method is proposed to solve this problem. DPPO is not a simple individual level for a single node to select the reconstruction action and evaluate its action value, but through comprehensive consideration of an effective reconstruction strategy to include the three levels of cluster-cluster-node, analyze the overall task goals that each node has with other nodes in the cluster, and build a two-level dynamic reconstruction decision architecture for the cluster system. Comprehensively analyze the cluster system information and complex environment high-dimensional situation information, design a deep neural network, and further study the reinforcement learning algorithm model of DPPO on the basis of the above two-level dynamic reconstruction. Through the optimization strategy, the expectation of the reconstruction strategy reward is maximized. Therefore, a dynamic reconstruction decision method is designed based on DPPO to consider the two-level dynamic reconstruction of the internal reconstruction of the cluster and the internal reconstruction of the cluster, and then perform cross-level dynamic reconstruction deployment for the task coordination and topological configuration of the cluster system.

[0073] The deep reinforcement learning idea based on DPPO that the present invention intends to adopt includes two core ideas: strategy iteration of cross-level dynamic reconstruction and cross-level dynamic reconstruction based on DPPO.

[0074] 1.2.1 Strategy Iteration for Cross-Level Dynamic Reconstruction

[0075] The cross-level dynamic reconstruction strategy iteration is to dynamically reconstruct the interaction between the agent and the simulation environment, such as Figure 5 As shown. In the Agent-Environment architecture of cross-level dynamic reconstruction of cluster systems considering task collaboration and topological configuration, when the agent interacts with the environment, each agent still follows the goal of reinforcement learning, that is, to maximize the cumulative rewards that can be obtained. At this time, the change in the global state of the environment is related to the joint actions of all agents. Therefore, in the process of agent strategy learning, the impact of joint actions needs to be considered. In the process of formulating dynamic reconstruction strategies, considering that the multi-formation cluster system has too large a decision space in the task area set above, and the formations in the cluster need to be as intelligent as possible in the current environment to complete the dynamic reconstruction process of the formation in the face of sudden damage in a timely manner, a multi-agent reinforcement learning algorithm is used to complete the generation of dynamic reconstruction strategies for multi-formation cluster systems, and a part of the systems that meet the requirements in the multi-formation cluster system are selected as agents according to certain rules, and then the Agent-Environment architecture of cross-level dynamic reconstruction of multi-formation cluster systems is designed, as shown in Figure 5shown.

[0076] The clusters in the damaged cluster system are divided into two parts according to whether they meet the task coverage constraint. Nodes are selected from the clusters that meet the task coverage constraint. In the clusters that meet the task coverage constraint, the intact systems are selected, their cluster balance is calculated, and these systems are sorted from large to small according to their elasticity importance. When the cluster does not meet the task coverage constraint, the selection is stopped. n The nodes are the nodes that will participate in the reconstruction.

[0077] The action space 𝒜 is the set of all node numbers when the cluster system is not damaged. Since we hope that the loss of balance is as small as possible, the reward function R It is a function of the inverse of cluster balance.

[0078] 1.2.2 Cross-level dynamic reconstruction algorithm model based on DPPO

[0079] In order to solve the problem of generating dynamic reconstruction strategies for cluster systems, during the execution of tasks by the cluster system, dynamic reconstruction within the cluster and between clusters is realized in the face of local damage scenarios, and the cluster performance is restored to a certain extent to continue to complete the established tasks. The present invention proposes a dynamic reconstruction method based on DPPO. DPPO not only selects reconstruction actions for a single node and evaluates its action value, but also comprehensively considers the three levels of cluster-cluster-node that an effective reconstruction strategy needs to include, analyzes the overall task goals that each node has with other nodes in the cluster, and needs to build a two-level dynamic reconstruction architecture for the cluster system. The problems analyzed by traditional reinforcement learning methods are often limited to the scale of action space and sample space. In general, tasks that are closer to the actual situation are often more complex and require consideration of large-scale and continuous state space and action space. When the input data is images or sounds, it often has high dimensions, which are difficult to handle with traditional reinforcement learning. Deep reinforcement learning combines the high-dimensional input of deep learning with reinforcement learning. The proposed DPPO-based method is as follows: Figure 1-3 As shown in the figure, it includes three parts: dynamic reconstruction decision architecture of cluster system, DPPO reinforcement learning algorithm model, and DPPO alliance learning model. First, it is necessary to build a two-level dynamic reconstruction decision architecture for the cluster system, comprehensively analyze the cluster system information and high-dimensional situation information of complex environments, and apply the Reinforce mechanism to estimate the reconstruction action value under two-level reconstruction to support cross-level dynamic reconstruction strategy optimization. On the basis of the above two-level dynamic reconstruction, the reinforcement learning algorithm model and alliance learning model of DPPO are further studied. The generation network of the deviation of the reinforcement learning agent training does not have an absolute value activation function, and the final deviation is generated by a two-layer super network with ReLU nonlinearity. The final loss function is:

[0080] (1.6)

[0081] in b Indicates the number of samples sampled from the experience pool , θ¯ is the parameter of the target network, so the TD error can be expressed as:

[0082] (1.7)

[0083] Where Q tot (target) represents the maximum value Q obtained from all actions in state s' tot According to the IGM condition, the input is the maximum action value of each agent in this state; Q tot (evaluate): status s In the case of tot ; r It indicates the reward value under the current strategy, which refers to the balance degree in this invention.

[0084] 1.3 Dynamic Reconfiguration Decision Algorithm Considering Task Coordination and Topological Configuration

[0085] Considering the distributed functional coordination of multiple clusters in the cluster, which leads to cross-level interaction in the reconstruction space, a two-level dynamic reconstruction within the cluster and between clusters is comprehensively analyzed, and the dynamic reconstruction decision method based on DPPO is studied. First, a cross-level dynamic reconstruction decision architecture of the cluster system is constructed, and then reinforcement learning and federation learning models are designed based on DPPO to realize the reconstruction strategy optimization.

[0086] 1.3.1 Cluster system dynamically reconstructs decision-making architecture across levels

[0087] Build a cluster system that dynamically reconstructs decision-making architecture across levels, such as Figure 1 As shown in the figure, the multi-dimensional situation information of the three-level spatiotemporal dynamic cluster system is analyzed, and then a deep neural network model is designed to extract multi-dimensional situation features and evaluate the value of reconstruction actions. On the basis of completing the cross-level dynamic reconstruction decision object, the reinforcement learning model based on DPPO and the alliance learning model training neural network are further studied. In the proposed cross-level dynamic reconstruction decision architecture, the dynamic reconstruction agent is designed to use DNN to extract the multi-dimensional situation features of the cluster system, focusing on the cross-level interactions within the cluster and between clusters, and then selecting reconstruction actions for each node in the cluster. The selection of reconstruction actions can be expressed as:

[0088] (1.8)

[0089] Where S t Represents the dynamic reconstruction processt The multidimensional situation tensor at the moment, M t-1 Represents a cluster mobility feature set, which consists of the historical mobility features of the cluster {mov t-1 ,mov t-2 ,mov t-3}.

[0090] The historical mobility characteristics of the cluster are necessary because the mobility trends of all nodes in the cluster cannot be fully observed only from the current mobility status. The mobility characteristics of the nodes selected by the cluster at time t can be described as mov t |[S t ,M (t-1) ]=[loc t init ,loc t final ], where loc t init and loc t final Represents the node location feature matrix, which is described by the I×J map matrix. Each element in the node location feature matrix is ​​associated with a coordinate. If the element is 1, the associated coordinate is the selected target location. t init and loc t final There is only one element in it that is equal to 1, and the other elements are equal to 0. The node position feature matrix loc t init represents the initial position of the reconstruction action at time t, and the node position feature matrix loc t final represents the end point of the same reconstruction action. In addition, if t <1, then loc t init and loc t final are all zero matrices. From the nature of the defined reconstruction action, this problem is integer programming with binary variables. For each node, the node position feature matrix loc t final Any element in can be used as its current strategy, then the number of its strategies is 2 I×J ; For the entire cluster, the total number of strategies is , so the dynamic decision-making problem studied is an NP-hard problem. The entire dynamic reconstruction process includes task dispatch and local destruction generation, reconstruction action generation, and cluster balance recovery evaluation. Based on the above decision-making architecture, the DPPO reinforcement learning algorithm is further designed to support cross-level dynamic reconstruction strategy optimization.

[0091] (1) Multi-dimensional situation feature extraction

[0092] In the constructed cross-level dynamic reconstruction decision-making architecture of the cluster system, we first need to pay attention to the cluster system information and the multi-dimensional situation information of the complex environment, including environmental feature information such as destruction characteristics and task characteristics, as well as cluster feature information such as cluster node location characteristics and business coverage characteristics, and further convert the above cluster system feature information and complex environment feature information into mathematical language. The present invention converts these feature information into the related I×J map matrix to mathematically describe it. After determining the relative coordinate system according to the geographical location of the specific task area, the I×J map matrix is ​​determined by the selected relative coordinate system. For example, it can be defined as a 256×56 image matrix to describe the current map information. The matrix processes a 256×256 relative coordinate system into a 256×256 image grid, and each element in the matrix corresponds to a specific small square grid in the relative coordinate system. Therefore, the task area feature matrix S at time t in the dynamic reconstruction process t mission It can be expressed as:

[0093] (1.9)

[0094] The matrix element s ij m (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s ij m (t) = 1 means it is covered, s ij m (t) = 0 represents no coverage; the service coverage characteristic matrix S at time t of the dynamic reconstruction process t cover It is expressed as:

[0095] (1.10)

[0096] The matrix element s ij c (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the business area of ​​the cluster node, s ij c (t) = 1 means covered, s ij c (t) = 0 means not covered; the node position feature matrix S at time t in the dynamic reconstruction process t location It is expressed as:

[0097] (1.11)

[0098] The matrix element s ij l (t) indicates whether there is any node n in the small square grid represented by (i, j) in the relative coordinate system at time t (m,n) , s ij l (t) = 1 means there is s ij l (t) = 0 means it does not exist; the characteristic matrix S of the damaged area at time t in the dynamic reconstruction process t damage It is expressed as:

[0099] (1.12)

[0100] The matrix element s ij d (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s ij d (t) = 1 means it is covered, s ij d (t) = 0 means not covered.

[0101] Based on the above description, the multidimensional situation tensor S at time t in the dynamic reconstruction process in formula (1.8) t It can be expressed as:

[0102] S t =[ S t mission , S t damage , S t location , S t cover ](1.13).

[0103] (2) Reconstruction strategy generation

[0104] In the dynamic reconstruction process t At this moment, the Agent is dynamically reconstructed to move the feature set M in the cluster system t-1 ={mov t-1 ,mov t-2 ,mov t-3} and the multidimensional situation tensor S t As input, and output the reconstruction action act at the current moment t｜mnDuring the entire dynamic reconstruction process, a series of dynamic reconstruction actions are generated by the dynamic reconstruction agent. All nodes in the cluster system execute the dispatched reconstruction actions in sequence or simultaneously to complete the redeployment of the cluster topology configuration, so as to improve the cluster balance and restore it to a performance level that can continue to execute the current task. The above dynamic reconstruction action set can be expressed as:

[0105] (1.14)

[0106] Among them act t｜mn Indicates cluster node n (m,n) In the dynamic reconstruction process t The reconstruction action performed at the moment, and the reconstruction action can be defined as act t｜mn =[s l i”j” (t-1), s l ij (t)], which means cluster node n (m,n) In the dynamic reconstruction process t The position s of the previous moment in the relative coordinate system l i”j” (t-1) Move to the current position s l ij (t), the defined reconstruction action act t｜mn According to the cluster node n described in formula (1.8) (m,n) The moving feature mov t |[S t ,M t ] is transformed; further cluster c m The dynamic reconstruction action set is expressed as:

[0107] (1.15).

[0108] Based on the above description of the input and output of the dynamic reconstruction agent, a DNN model f is designed for the dynamic reconstruction agent. θ , to generate reconstruction actions, apply Res Net to extract features from cluster system information and environmental information, apply LSTM to process the temporal features of high-dimensional information, apply Attention mechanism to focus on task area and node location, apply Reinforce with Baseline principle to estimate the action value of the entire reconstruction process, and use value network as its baseline function only when training neural network. The designed DNN model for dynamic reconstruction of cluster system is a multi-input and multi-output neural network. As a reconstruction strategy generation model, the network parameters are randomly initialized. θ 0The DNN in the dynamic reconstruction agent uses a ResNet module composed of five convolutional layers with ReLU functions to process the multi-dimensional situation input feature matrix, including the task area feature matrix S t mission , business coverage feature matrix S t cover , node position feature matrix S t location , damage area characteristic matrix S t damage , cluster system mobile feature set M t-1 ={mov t-1 ,mov t-2 ,mov t-3}. After the destruction information and task information are decoded by their respective ResNet modules, they are input into the scene kernel composed of residual MLP modules for processing. The business coverage information, historical reconstruction action information and cluster node location information are decoded by their respective ResNet modules and input into the cluster kernel composed of LSTM modules for processing. The output of the scene kernel and the cluster kernel is used as part of the value network input. The other part of the value network input is the Baseline feature of the entire dynamic reconstruction process. This feature tensor is composed of the reconstruction action set of the entire dynamic reconstruction process. The value network evaluates the cluster balance loss of the entire dynamic reconstruction process and is only used when training the parameters of the entire DNN neural network. In the dynamic reconstruction process, after the reconstruction Agent uses the DNN model to input the above-mentioned multi-dimensional situation features, it first outputs an I×J probability matrix through an Attention module to select S t location As the object for performing the reconstruction action at this moment, the Attention module inputs the processing information of the scene kernel and the cluster kernel, and inputs the reconstruction action features and cluster node position features through the skip connection of the residual mechanism. The ResNet module that previously processed the task information generates a task information input feature through the residual connection, and embeds the I×J probability matrix output by the above Attention module into it, and then inputs it to a new Attention module for processing, and then outputs an I×J probability matrix to select S t mission A task area in is taken as the target task area. The ResNet module that previously processed the cluster node position information generates a pole position information input feature through a residual connection, and embeds the feature information describing the target task area through an MLP module, inputs it to a Deconv residual module, and then outputs an I×J probability matrix to select the target position of the reconstruction action.

[0109] The designed DNN model includes convolution module, residual module, batch normalization module, ReLU function module, MLP module, Attention module, etc. The input multi-dimensional situation features are processed by multiple deeply stacked residual modules. The middle layer of the deep neural network is constructed by stacking residual modules, including the following structure:

[0110] a) Situation feature processing module:

[0111] ① A convolutional layer consisting of 128 3×3 filters with a stride of 1;

[0112] ②Batch normalization layer;

[0113] ③ReLU function layer;

[0114] ④ A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0115] ⑤Batch normalization layer;

[0116] ⑥Addition module: add the input feature tensor of the residual module to the output feature tensor of the previous layer;

[0117] ⑦ReLU function layer;

[0118] The scenario kernel consists of a three-layer MLP, the cluster kernel consists of an LSTM layer, and the output includes a value module, a reconstruction node module, a target task area module, and a target location module:

[0119] b) Value Module:

[0120] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0121] ② A linear fully connected layer with a size of 256;

[0122] ③ReLU function layer;

[0123] ④Linear fully connected layer;

[0124] ⑤Sigmoid function layer;

[0125] c) Refactor node modules:

[0126] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0127] ② A linear fully connected layer with a size of 256;

[0128] ③ReLU function layer;

[0129] ①Multi-head Attention layer;

[0130] ②ReLU function layer;

[0131] ③Fully connected output layer: outputs a feature vector of size I×J, corresponding to the selection of S t location The probability of a certain UAV p The logarithm of ;

[0132] d) Target mission area module:

[0133] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0134] ② A linear fully connected layer with a size of 256;

[0135] ③ReLU function layer;

[0136] ④Multi-head Attention layer;

[0137] ⑤ReLU function layer;

[0138] ⑥Fully connected output layer: outputs a feature vector of size I×J, corresponding to the selection of S t mission The probability of a task area being the target task area p The logit of ;

[0139] e) Target location module:

[0140] ① A convolutional layer consisting of 128 3×3 filters with a stride of 1;

[0141] ②Batch normalization layer;

[0142] ③ReLU function layer;

[0143] ④ A linear fully connected layer with a size of 256;

[0144] ⑤ReLU function layer;

[0145] ⑥Fully connected output layer: Outputs a feature vector of size I×J, corresponding to the probability of selecting the target position of the reconstructed action p The logarithm of ;

[0146] The network parameters of the above DNN model are θ , initial network parameters θ 0 is generated randomly.

[0147] 1.3.2 Dynamic Reconstruction Strategy Optimization Based on DPPO

[0148] Based on the cross-level dynamic reconstruction decision-making architecture of the cluster system constructed in the previous section, a reinforcement learning algorithm based on DPPO and a federated learning algorithm model are designed to train the DNN model in the dynamic reconstruction Agent to achieve dynamic reconstruction strategy optimization.

[0149] (1) Reinforcement learning algorithm based on DPPO

[0150] Based on the cross-level dynamic reconstruction decision architecture in the previous section, after completing each dynamic reconstruction process through simulation, a set of multi-dimensional situation data and corresponding reconstruction strategy data can be generated. Then, it is necessary to evaluate the dynamic reconstruction process, calculate the cluster balance after the reconstruction, and the loss of cluster balance during the entire reconstruction process, so as to comprehensively evaluate the entire dynamic reconstruction process and design a reward function on this basis. For each dynamic reconstruction action act t｜mn Once the reconstruction action is completed, it is considered that the moving node starts to perform the current task at the new location, and then the service coverage feature matrix S can be updated. t cover , according to formula (1.4), the current cluster balance can be calculated. The goal of the cluster system is to improve its cluster balance as efficiently as possible. Therefore, the reward function should be defined for cluster balance and reconstruction time as follows:

[0151] (1.16)

[0152] Where R t represents the reward value at time t in the dynamic reconstruction process, τ t+ζ represents the cumulative reconstruction time at the time (t+ζ) of the dynamic reconstruction process, τ t+ζ-1 represents the cumulative reconstruction time at the time (t+ζ-1) of the dynamic reconstruction process, ε b thr represents the cluster system balance threshold, δ represents the discount factor of the reward function, τ T Represents the cumulative reconstruction time (Time to finish reconfiguration, TTFR) to complete the entire reconstruction process. Based on the reward function and the data set of the input and output feature information of the dynamic reconstruction agent during the reconstruction process, a reinforcement learning algorithm is designed, involving technologies such as policy gradient, experience replay, TD(𝝀), V-trace and UPGO, such as Figure 2 shown.

[0153] Reward the dynamic reconstruction strategy and train the parameters of the DNN model through the sample data set θ i For the next policy iteration process i +1, generate new initial multi-dimensional situation feature information, and then use the dynamic reconstruction process θ i The DNN model as a parameter generates a new reconstruction strategy. For the first policy iteration process, the dynamic reconstruction process uses the parameters θ 0 randomly initialized DNN model, which aims to approximate the value function of the reconstruction strategy and multiple policy functions describing the reconstruction actions at the same time. The DNN is trained in the policy iteration process of dynamic reconstruction. When executing the dynamic reconstruction process, the reconstruction agent selects a series of reconstruction actions to generate a complete strategy. The reconstruction agent needs to output three probability matrices for each reconstruction action selected, and at the same time use the reward R of the reconstruction action as the reward. t As a value label. The main idea of ​​the reinforcement learning algorithm is to iteratively perform the dynamic reconstruction process during the policy iteration process: training the DNN parameters so that the reconstruction probability and value predicted by the neural network are closer to the reconstruction policy and reward. By maximizing the similarity between the probability of the reconstructed action predicted by the DNN and the reconstruction policy, the policy gradient can be expressed as:

[0154] (1.17)

[0155] (1.18)

[0156] Where r(s t ,a t )express t The reward value of the reconstructed action at each moment, s t express t The state at the moment, a t express t The reconstruction action at the moment, V(s t ) represents the average value, A(s t ,a t ) represents the action advantage value; based on the experience replay technology, asynchronous update is performed by optimizing the strategy π θ The expected reward J(π θ ) is maximized, which can be expressed as:

[0157] (1.19)

[0158] In addition, the DNN is adjusted to minimize the gap between the predicted value and the reward value of the reconstruction strategy. TD(𝝀), V-trace and UPGO training are applied to update the neural network to push the strategy to a trajectory better than the average return. When the dynamic reconstruction process is performed in the next policy iteration, the dynamic reconstruction agent uses the new DNN parameters to better search for the optimal reconstruction action.

[0159] (2) Alliance learning model based on DPPO

[0160] In order to realize the distributed learning and training of DPPO, a coalition learning model is designed, coalition games and virtual self-learning mechanisms considering priorities are defined, and three types of agents are used for learning and training to achieve dynamic reconstruction strategy optimization, such as Figure 3 shown.

[0161] Game definition: randomly generated n match In a single game, two agents generate reconstruction strategies for a task. The one with the smaller loss of cluster balance ability wins. n match Game Win Rate

[0162] Priority-based virtual self-learning (PFSP): Virtual self-learning (SP) archives the alliance at regular intervals and randomly and evenly selects opponents from the alliance to play against the agent being trained; priority refers to giving priority to opponents with a higher winning rate in the alliance archive.

[0163] The three types of agents include the main agent, the main explorer, and the alliance explorer. The main agent: selects opponents in the alliance archive based on PFSP to play against, and archives regularly; the main explorer: plays against the main agent of the current iteration, and reinitializes regularly; the alliance explorer: defeats all opponents in the alliance with a certain winning rate, and archives to the alliance, otherwise it reinitializes regularly.

[0164] 1.4 Dynamic Reconstruction Case Analysis

[0165] The present invention takes into account the dynamic reconstruction of the task coordination and topological configuration of the swarm UAV system as an example (such as Figure 4 As shown in the figure, the proposed method is used to generate a reconstruction strategy and verify the effectiveness of the method. For cluster systems with formation characteristics, they are usually built in multiple formations during mission execution, and it is necessary to focus on the dynamic coordination characteristics within the formation and between formations. During the mission of the multi-formation cluster system, in response to emergencies such as local damage, the multi-formation cluster system can restore the cluster performance to a certain extent through two-level dynamic reconstruction of internal formation functions and inter-formation functions to continue to complete the established tasks. For this type of dynamic reconstruction decision-making problem under the condition of functional coordination of multi-formation cluster systems, the present invention solves the problem based on the proposed DPPO method and conducts relevant discussions and analyses.

[0166] 1.4.1 Drone Swarm Case Study

[0167] In order to study the dynamic reconstruction process of drone clusters, this paper will establish a method for evaluating the destruction and recovery performance of drone cluster operations by analyzing the drone cluster system and the task process, and then generate a reconstruction strategy based on the proposed DPPO dynamic reconstruction decision method to complete the cluster formation configuration redeployment. First, the composition, control structure and task execution process of the drone cluster system are analyzed to confirm the control method and task execution objectives of the drone cluster system; secondly, according to the type and goal of the drone cluster task execution, the task execution results of the drone cluster are evaluated; finally, the elasticity and elasticity importance of the drone cluster in the reconstruction process are defined through the performance evaluation index of the drone cluster, and the establishment of a comprehensive evaluation method for the process and results of drone cluster reconstruction is completed.

[0168] Generally speaking, the UAV subsystem, ground station subsystem, communication link and ground maintenance and support subsystem are the main components of the intelligent UAV cluster system. Among them, the UAV subsystem (UAV platform) as the main part of the cluster system is the main carrier for the intelligent UAV cluster to perform tasks, mainly composed of intelligent units, flight control and navigation systems, power units, mission payloads and fuselages. The ground station subsystem is mainly composed of communication link ground terminals, command and control systems, intelligence processing stations and take-off and landing control systems. Its main function is to perform UAV take-off and landing and command and control. Among them, the communication link is an important part of the "machine-station-chain" complex system that makes the intelligent UAV cluster an information interaction and collaborative control. It is the information transmission medium of the ground station subsystem. The communication link contains information such as communication protocols and information transmission methods. By using this information, data download and command upload can be realized. In addition, the system that carries out maintenance and support work for important equipment in each subsystem is the maintenance and support subsystem, which is not considered and analyzed in the research process of this invention.

[0169] When a drone swarm is performing a mission, each drone in each formation must have the ability to fly autonomously and needs to work in coordination with other drones in the formation. In order to achieve such coordination, it is necessary to determine the control and information relationships between each entity in the drone formation in both physical and logical aspects. In order to achieve the above relationship determination, the drone swarm needs to ensure that all drones can participate in the mission process during the mission execution, and all drones need to maintain a certain distance and form a relatively stable formation structure. While ensuring such stability, in the actual combat environment, it may be necessary to dynamically adjust the formation according to the actual situation and specific mission requirements.

[0170] 1.4.2 Analysis of UAV swarm missions and damage

[0171] First, the task of the drone cluster is analyzed. The case of this invention takes the multi-target joint reconnaissance task of the drone cluster as an example. Considering the multi-target reconnaissance task includes M Different reconnaissance mission areas, each of which is represented by an irregular polygonal area. For multi-target joint reconnaissance missions for drone clusters, a team of M The drone cluster S consists of formations, and the formation set of the cluster S is represented by C. c m Indicates dispatch to a certain reconnaissance mission area A m Consider each drone formation c m By N m UAVs, then the UAV formation c m It can be represented as a formation UAV set, where n∈{1,2,...,N m} indicates that the UAV is in a drone formation c m The dynamic number in N m Forming a drone fleet c m The total number of UAVs is based on the size of the reconnaissance mission area A m OK. The black dot icon represents the UAV, and the blue semi-transparent circular area of ​​each UAV represents the reconnaissance area of ​​the UAV. Multiple UAVs form a formation in a certain reconnaissance mission area.

[0172] In summary, taking the joint reconnaissance mission of drone clusters as a case, the dispatch of joint reconnaissance missions at three levels of cluster-formation-single machine is analyzed. The type of cluster mission depends to a certain extent on the intelligence level of the drone cluster. Generally speaking, drone clusters can carry various types of mission payloads. The present invention focuses on the reconnaissance mission for research and analysis. Any drone in the cluster can independently complete the reconnaissance mission within a certain area. The scope of the reconnaissance mission is a number of circular areas with a certain point as the center and a certain length as the radius. The task coordination goal of the drone cluster is to make the redundant capabilities of each mission area relatively balanced on the premise of completing full coverage within the mission area. The dynamic reconstruction of intelligent drones is an autonomous behavior that adapts to dynamic changes in the environment and restores the execution of tasks. Specifically, when the cluster is affected by dynamic changes such as external attacks or internal failures during the execution of the mission, the system can, under the action of internal control and reconstruction strategies, with the goal of restoring global mission performance, perform autonomous reconstruction of the formation to ensure continuous execution of the mission. Combined with the above definition, the internal element space transfer is realized within and between each formation of the cluster. Specifically, during the task execution process, each drone in the cluster moves from the initial position to different task areas in accordance with the global target optimization direction, and performs various tasks in the task area in the form of formation. The drone cluster is divided into multiple formations for coordinated reconnaissance in the area. The mission scene is set as a two-dimensional map plane. In order to facilitate the description of the movement process of the drone, the mission area is represented by a square grid. It is stipulated that the movement of the drone during the execution of the task is to move from the center of a square grid to the center of another square grid. Different formations perform tasks in the partitions of the overall mission area. The range of reconnaissance tasks that a drone can complete is a circle with a radius of r, centered on the location of the drone. uav In order to ensure the cluster balance of the drone cluster in performing reconnaissance tasks, when the drone formation performs a reconnaissance mission in a certain mission area, its formation topology configuration needs to consider the redundancy deployment of the reconnaissance capability.

[0173] During the execution of the above multi-target joint reconnaissance mission, the drone cluster may be locally damaged, resulting in the destruction of some UAVs in one or more formations. In the case of local damage, the multi-dimensional operation and maintenance situation information of the drone cluster mainly includes drone cluster information and environmental information, and it is necessary to determine the attribute information and functions of each drone, as shown in Table 1. In the process of executing the multi-target joint reconnaissance mission, there is a certain correlation between the various UAVs and the components of the UAVs. Among them, the interactive relationship between various situation information, each drone will cooperate with each other while operating relatively independently, and jointly describe the functions and composition of the entire system. As the subject of damage, environmental information will interact with drone cluster information after the damage occurs to transmit the corresponding damage data to the cluster. The formation layer information is regarded as a whole as a task execution unit, and interacts with other formations and environmental information through the internal whole machine layer information. Within the formation layer information, multiple whole machine layer information completes the state change through communication. At the same time, each whole machine layer information has functional layer information representing different functions. The functional layer information completes the parameter update of the whole machine layer information by interacting with the whole machine layer information. The drone swarm will inevitably be affected by some factors and destroyed during the execution of the mission. Only when the specific state and data of the drone swarm after the destruction are known can the reconstruction strategy be formulated. Therefore, the destruction modeling of the drone swarm is of great significance for the formulation of the reconstruction strategy of the drone swarm.

[0174] Table 1 Operation and maintenance situation information under the situation of drone swarm destruction

[0175]

[0176] In the drone cluster model of the present invention, a hierarchical modeling method is adopted to divide the entire drone cluster system into three levels for modeling, namely, functional layer information, whole machine layer information and formation layer information. Among them, the drone functional layer information is taken as the bottom-level information, and its state is set to two states, namely normal state and damage state. The drone whole machine layer information is the upper-level information of the drone functional layer information, and its state is determined by the functional layer information. The damage of different functional layer information will cause the whole machine layer information to produce multiple states. The causes of damage usually include small-scale precision fire strikes, electromagnetic wave interference, large-scale fire strikes, etc. This form of damage will cause all UAVs in a certain area to be destroyed at the same time.

[0177] For the above-mentioned UAV swarm under the local destruction situation, during the execution of the above-mentioned multi-target joint reconnaissance mission, some UAVs in one or more formations in the UAV swarm are destroyed. Taking the impact of this destruction as input, consider that each UAV has two states: normal state and fault state. When the UAV is destroyed, it enters the fault state. The scope of local destruction is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value of is randomly generated. Based on the generated local destruction range, the destruction situation of the drone cluster is analyzed.

[0178] In addition, in order to ensure the accuracy of subsequent modeling and strategy generation, it is necessary to make the arrival mode of drone cluster destruction as consistent as possible with the actual destruction behavior during the destruction modeling process. In the process of modeling the arrival mode of destruction, the main consideration is to analyze the number of destructions in the simulation process and the time point of each destruction. The non-homogeneous Poisson process method is used to model the number of local destructions, assuming that the intensity of extreme environmental events that cause large-scale destruction during the model operation is λ S , then the number of random destructions of environmental information is k s The probability is given by the following formula:

[0179] (1.20)

[0180] where λ S (t) is the mean of the number of times the damage occurs within the total duration of the mission execution (0, t]. After obtaining the number of times the damage occurs, the time of the damage is determined in the same way as the random damage. The coordinates of the damage center are determined by random sampling on the mission area plane, and the damage radius is determined according to the input parameters of the macro-environmental information. For the local damage situation, it is mostly caused by external attacks, and its generation method is determined by the preset parameters of the environmental information. Considering the randomness and uncertainty of the damage, the present invention abstracts the damage into a random area with a center point, uses a deterministic regional damage model to represent the impact of local damage on the drone cluster, and abstracts the regional damage caused by the local damage into a circular area randomly placed on the physical network plane, and all drones intersecting with this area are regarded as damaged.

[0181] 1.4.3 Analysis of Dynamic Reconfiguration of UAV Clusters

[0182] Under the above local destruction situation, the dynamic reconstruction of the UAV cluster is considered to be an autonomous behavior that can adapt to environmental changes to continue to perform the current task. When the UAV cluster is affected by dynamic changes during the task execution, it is necessary to generate a corresponding dynamic reconstruction strategy to achieve the recovery of the UAV cluster reconnaissance task performance and the redeployment of the topological configuration, so as to ensure the continuity of the current multi-target joint reconnaissance task. When local destruction occurs, the proposed DPPO dynamic reconstruction decision algorithm observes the destruction situation of the UAV cluster and generates a reconstruction strategy to each UAV. The UAV cluster then performs dynamic reconstruction between UAVs within the formation and between formations within the cluster. A multi-target joint reconnaissance task consisting of seven irregular polygonal reconnaissance task areas is randomly generated. The entire relative coordinate system map is represented as a 512×512 positive direction grid image. The reconnaissance range of each UAV in the UAV cluster is represented by a blue circular image. The reconnaissance area of ​​each formation needs to cover the corresponding white irregular polygonal task area. The destruction state is randomly generated, the destruction center is a random sampling point on the task area, and the destruction area is a randomly generated irregular polygon. The multi-target joint reconnaissance mission analyzed includes 7 mission areas. The drone cluster dispatched 7 formations, with a total of 124 drones. Table 2 shows the initial position information and UAV formation number information of the multi-formation drone cluster. The reconstruction action of each UAV in the reconstruction strategy is generated based on formula (1.8) and is determined by its initial position and terminal position, as shown in Table 2; Python is used to conduct a visual simulation study on the reconstruction of the drone cluster. The numerical simulation is described from the perspective of the optimal configuration of the drone cluster. The reconstruction results in different scenarios verify the effectiveness of the proposed DPPO dynamic reconstruction decision method. The above-mentioned drone cluster is for a quadcopter drone cluster. In addition, the proposed method is also applicable to other types of drone clusters. In addition, for the above-mentioned 7 formations consisting of a total of 124 drones, each position on the 512×512 relative coordinate system map can be used as a choice of the terminal position of each drone reconstruction action. Based on the properties of the decision variables of the dynamic reconstruction decision problem, it can be seen that the total number of strategies is 2 (512×512×124) =∞.

[0183] Table 2 UAV swarm formation configuration reconstruction information

[0184]

[0185] After the above dynamic process, the drone swarm completes the redeployment of the topological configuration. Assuming that all drones in the cluster move at a speed of 20 km / h, of course, the speed of the drone can also be set according to the actual situation. During the dynamic reconstruction process, the performance of the cluster shows a fluctuating upward trend.

[0186] 1.4.4 Discussion and Analysis

[0187] The purpose of solving the dynamic reconstruction decision problem of drone swarms is to generate a feasible optimization strategy. In the discussion and analysis part, an extended analysis is conducted on the performance of the proposed DPPO algorithm and the impact of various factors on the dynamic reconstruction problem.

[0188] (1) Different decision-making algorithms

[0189] This paper evaluates the performance of the proposed DPPO method. In addition, the case part also applies the DQN method and the cooperative game (CG) method to analyze the dynamic reconstruction decision problem of the studied drone cluster. The learning and training of the algorithm uses a server composed of an Intel i97980XE CPU and four RTX2080TI-A11G GPUs to train the DPPO network and the DQN network during the dynamic reconstruction process. During the training process, as described in the method section, each simulation will randomly generate a multi-target joint reconnaissance task and a local destruction situation, and then the algorithm will generate the corresponding dynamic reconstruction strategy.

[0190] The present invention uses the following method to evaluate the performance of each algorithm: pause after running every 100 episodes, use the current training state agent to generate reconstruction strategies for 10 test cases, and the test cases remain unchanged during the entire training and learning process. During the entire training and learning process, each method generates reconstruction strategies for 10 cases in the test case, calculates the cluster balance and then calculates the reward value, and calculates the average of the 10 reward values ​​as the average reward of the current test. Figure 6 Describe the average reward curves of the three methods throughout the training and learning process. Since the 10 cases in the test case set are fixed, the average reward of the CG method is a constant value, so the reward curve of the CG method is a straight line. The shadow around each reward curve represents the standard deviation of the 10 reward values. During the entire training process, both the DPPO method and the DQN method executed 100,000 episodes. The reward curves of both methods show a fluctuating upward trend. In the initial 17,000 episodes, the DQN method showed a faster growth trend than the DPPO method. However, DPPO grew to a higher reward value position than DQN after 20,000 episodes, and DPPO was significantly stronger than the other two methods in terms of dynamic reconstruction decision performance in the later stage of the training process.

[0191] (2) Alliance learning analysis

[0192] In addition, in the learning and training process of the proposed DPPO dynamic reconstruction decision algorithm, in order to realize the distributed learning and training of DPPO, a coalition learning model for DPPO is designed. Based on the designed coalition game and the virtual self-learning mechanism with priority considerations, the main agent, main explorer, and coalition explorer are used for learning and training. The DPPO data used for comparative analysis are all the main agent data. The main agent is the core agent of the three types of agents in the designed coalition learning. The main explorer and coalition explorer are both auxiliary agents, which support the main agent for better training in the entire coalition learning training, and compare and analyze the winning rates of the three types of agents in the coalition learning process.

[0193] (3) Case studies in different situations

[0194] For different multi-target joint reconnaissance tasks and UAV cluster sizes, the corresponding local destruction situations are randomly generated for the multi-formation UAV cluster related information of the two discussion and analysis cases in Tables 3 and 4, and the DPPO reconstruction strategy is executed. The geographical distribution of all mission areas in the two groups of discussion and analysis cases is different, and the size of each mission area is different. The overall size of the UAV formations in the two groups of cases is 126 and 129 UAVs respectively, and the formation configuration of the cluster is different. In addition, their respective destruction states are different. After the reconstruction process, the redeployment results of the formation configuration show that the proposed DPPO method shows stable performance for dynamic reconstruction decision problems under different tasks and different destruction situations. This is because the multi-target joint reconnaissance tasks and local destruction situations are randomly generated during the dynamic reconstruction agent training and learning process.

[0195] Table 3 Multi-formation UAV cluster information (discussion and analysis case 1)

[0196]

[0197] Table 4 Multi-formation UAV cluster information (discussion and analysis case 2)

[0198]

[0199] In addition, an extended analysis of the optimal dynamic reconstruction strategy is conducted to determine the impact of different algorithms on the optimal strategy. For the above two cases under different situations, the dynamic reconstruction strategy is generated based on the proposed DPPO deep reinforcement learning method. The DQN method and the CG method are also used to generate reconstruction strategies for the above two cases under different situations. Figure 7 As shown in Figure 2, the dynamic reconstruction strategy given by the DPPO method achieves better cluster balance ε than the other two methods. band less reconstruction time TTFR. The proposed DPPO method achieves a better dynamic reconstruction strategy solution, while the DQN method and the CG method may lead to local optimality, for example, during the dynamic reconstruction and redeployment process, multiple drones may spend more time executing the deployed formation configuration reconstruction strategy.

[0200] The present invention aims at typical problems such as operation management in cluster system operation and maintenance, and focuses on the representative frontier problem of this type of problem - the "dynamic reconstruction" decision-making problem of cluster system. It analyzes the task coordination and topological configuration characteristics of the three-level spatiotemporal dynamic cluster system during the execution of tasks, studies the dynamic reconstruction decision-making method based on DPPO deep reinforcement learning, constructs a dynamic reconstruction decision-making architecture under cross-level interaction of cluster system, designs a deep neural network to comprehensively analyze cluster system information and high-dimensional situation information of complex environment, and further studies the reinforcement learning algorithm model of DPPO on this basis, applies the policy gradient algorithm of advantage Actor-Critic, and performs asynchronous updates based on experience replay technology. By optimizing the strategy, the expectation of the reconstruction strategy reward is maximized, and a coalition learning model is designed to realize the distributed learning training of DPPO. Three types of intelligent agents are used for learning and training to realize dynamic reconstruction strategy optimization.

[0201] Based on the proposed DPPO dynamic reconstruction decision algorithm, the present invention verifies the dynamic reconstruction of multi-formation UAV clusters in the local destruction scenario in the case analysis part, analyzes cases under various situations such as different joint reconnaissance tasks, different cluster sizes, and different destruction situations, and generates dynamic reconstruction strategies based on the proposed DPPO deep reinforcement learning method. At the same time, multiple methods are applied to generate reconstruction strategies for dynamic reconstruction cases of different situations. Based on the simulation analysis of the UAV cluster reconnaissance task case, the dynamic reconstruction process under various strategies is executed. Compared with other strategies, the dynamic reconstruction strategy given by the proposed DPPO method can achieve higher cluster balance in a shorter reconstruction time during the reconstruction of the UAV cluster formation. In addition, while maintaining the best solution quality, the proposed DPPO method can also maintain a more efficient solution speed among multiple methods, and the solution speed shows a relatively stable state with the increase of task scale, cluster scale and destruction scale. In summary, the case analysis results show that the proposed DPPO dynamic reconstruction decision algorithm can provide an efficient dynamic reconstruction solution in the topological configuration redeployment work of multi-formation clusters for task collaboration, and realizes the redeployment of UAV clusters under local destruction to continue to perform the current cluster task.

Claims

1. A cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning, characterized in that: The method comprises the following steps: 1) Construct a dynamic reconstruction decision framework for cluster systems: Analyze the task coordination and topological configuration characteristics of the three-level spatiotemporal dynamic cluster system, construct a three-level dynamic reconstruction decision framework of cluster-cluster-node, and analyze the topological configuration characteristics between clusters within the cluster and between nodes within the cluster; 2) Multi-dimensional situation feature extraction: Design a deep neural network model, use the ResNet module to process the task area features, business coverage features, node location features, and damage area features, apply the LSTM module to extract the temporal features of high-dimensional information, and use the Attention mechanism to focus on the internal and cross-level interactions of the cluster to extract multi-dimensional situation features; 3) DPPO-based reinforcement learning algorithm model: Apply the policy gradient algorithm of the dominant actor-critic and combine it with the experience replay technology for asynchronous update. By maximizing the expected policy reward, the neural network is updated using TD (λ), V-trace and UPGO training to achieve dynamic reconstruction strategy optimization. 4) Design of DPPO alliance learning model: Design alliance games and virtual self-learning mechanisms, including three types of agents: master agent, master explorer and alliance explorer, conduct distributed learning training, and optimize dynamic reconstruction strategies; 5) Dynamic reconstruction strategy generation: In the dynamic reconstruction process, the dynamic reconstruction agent is used to generate a reconstruction action set based on the multi-dimensional situation information of the cluster system, and redeploy the cluster system topology configuration to restore the task performance; The cluster system dynamic reconstruction decision architecture in step 1) comprehensively analyzes cluster system information and complex environment high-dimensional situation information, supports cross-level dynamic reconstruction strategy optimization based on task collaboration requirements, and specifically includes the following steps: 1.1) Analyze the tasks, destruction and reconstruction of the three-level spatiotemporal dynamic cluster system of cluster-cluster-node, considering the multi-objective tasks including M There are different task areas, and the task area space is described as follows: （1.1） The mission area A m ,m∈{1,2,...,M} is an irregular polygonal region; 1.2) Send a team of M Clusters , then the cluster Clusters It can be expressed as: （1.2） in c m Indicates dispatch to mission area A m clusters; consider clusters c m Depend on N m nodes, then the cluster c m It is expressed as: （1.3） in n ∈{ 1,2,...,N m } represents a node node (m,n) In cluster c m The dynamic number in N m Cluster c m The total number of nodes, depending on the size of the task area A m Sure; 1.3) During the execution of the multi-objective task, the cluster system may be partially damaged, resulting in the destruction of some nodes in one or more clusters. Taking the impact of this damage as input, each node is considered to have two states: normal state and fault state. When a node is damaged, it enters the fault state. The scope of the local damage is represented by a point ( i d ,j d ) as the center, with r d is a circle with a radius of ( i d ,j d )and r d The value of is randomly generated; 1.4) When the cluster is affected by dynamic changes during task execution, it is necessary to generate corresponding dynamic reconstruction strategies to achieve cluster system task performance recovery and topology redeployment, so as to ensure the continuity of task execution; when local damage occurs, the dynamic reconstruction algorithm observes the damage situation of the cluster system and generates a reconstruction strategy to communicate to each node, and the cluster system then performs dynamic reconstruction between nodes within the cluster and between clusters within the cluster; Dynamic Reconfiguration of Cluster Systems Time, definition Cluster balance over time , which represents the mean square error of the redundancy values ​​of all nodes that can work normally in each cluster during the task execution phase, as shown below: （1.4） in M Cluster The total number of clusters in N m Represents clusters c m The total number of nodes, express Time Node node (m,n) The sum of the overlapping areas of the coverage area of ​​​​the node and the coverage areas of other nodes in the same cluster within the task area, express The average redundancy value of all nodes in each cluster at a certain time; 1.5) In view of the above-mentioned task coordination and topological configuration characteristics, the local damage scenario during the cluster system task is considered, and the cluster system dynamic reconstruction decision problem is constructed. The damage center and radius are described, the cluster failure scale is described, the threshold of reconfigurable nodes in each cluster is determined, the maximum moving distance of reconfigurable nodes is determined, the minimum coverage rate of the task area is specified, the cluster balance is described, and the reconstruction time threshold is determined. The optimization objectives, constraints and control variables based on the cluster system dynamic reconstruction decision problem are as follows: （1.5） in ε m Represents clusters c m The balance of ε m min Represents clusters c m The balance threshold of N m move Represents clusters c m The total number of reconfigurable nodes in N m normal Represents clusters c m The total number of nodes in normal state, d Represents the distance between two normal nodes, d min It represents the minimum allowed distance, that is, the safe distance between nodes. represents the total time of the reconstruction process, Indicates the reconstruction time threshold.

2. According to claim 1, the cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning is characterized in that: In step 1), the cross-level dynamic reconstruction decision architecture uses DNN to extract the multi-dimensional situational features of the cluster system by designing a dynamic reconstruction agent, focusing on the cross-level interactions within the cluster and between clusters, and then selecting reconstruction actions for each node in the cluster. The selection of reconstruction actions is expressed as: （1.8） in S t Represents the dynamic reconstruction process t The multi-dimensional situation tensor at the moment, Represents a cluster mobility feature set, which consists of the historical mobility features of the cluster { mov t-1 ,mov t-2 ,mov t-3 }; t The mobile characteristics of the nodes selected by the cluster at the moment are described as ,in loc t init and loc t final Represents the node position feature matrix, through I×J Map matrix to describe.

3. The cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning according to claim 2 is characterized in that: The multi-dimensional situation feature extraction in step 2) includes: 2.1) Task area feature extraction: The task area features are expressed as a matrix to describe the task area coverage in the relative coordinate system; 2.2) Service coverage feature extraction: Describe the service coverage of cluster nodes, expressed as a service coverage feature matrix; 2.3) Node position feature extraction: Design a node position feature matrix to represent the current position of each node in the cluster; 2.4) Damage area feature extraction: describes the scope and impact of local damage, including the location and radius of the damage center.

4. The cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning according to claim 3 is characterized in that: Transform multi-dimensional situation feature information into relevant I×J The map matrix describes it mathematically, and after determining the relative coordinate system according to the geographical location of the specific mission area, I×J The map matrix is ​​determined by the selected relative coordinate system; the task area feature matrix at time t in the dynamic reconstruction process S t mission It can be expressed as: （1.9） The matrix elements s ij m ( t )express t In the relative coordinate system ( i,j ) is covered by the task area, s ij m ( t )=1 means it is covered. s ij m ( t )=0 means not covered; dynamic reconstruction process t Business coverage feature matrix at the moment S t cover It is expressed as: （1.10） The matrix elements s ij c ( t ) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the business area of ​​the cluster node. s ij c ( t )=1 means it is covered. s ij c ( t ) = 0 represents not covered; the node position feature matrix at time t in the dynamic reconstruction process S t location It is expressed as: （1.11） The matrix elements s ij l ( t ) represents the relative coordinate system at time t ( i,j ) whether there is any node in the small square grid represented by node (m,n) , s ij l (t) = 1 means existence s ij l ( t ) = 0 means it does not exist; the characteristic matrix of the damaged area at time t in the dynamic reconstruction process S t damage It is expressed as: （1.12） The matrix elements s ij d ( t ) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area. s ij d ( t )=1 means it is covered. s ij d ( t )=0 means not covered; Based on the above description, the multidimensional situation tensor at time t in the dynamic reconstruction process in formula (1.8) S t It can be expressed as: （1.13）； in, S t mission is the task area feature matrix, S t cover is the business coverage feature matrix, S t location is the node position feature matrix, S t damage is the destruction area characteristic matrix; In the dynamic reconstruction process t At this moment, the Agent is dynamically reconstructed to move the feature set of the cluster system M t-1= { mov t-1 ,mov t-2 , mov t-3 } and multidimensional situation tensor S t As input, and output the reconstruction action at the current moment act t｜mn ; During the entire dynamic reconstruction process, a series of dynamic reconstruction actions are generated by the dynamic reconstruction agent. All nodes in the cluster system execute the dispatched reconstruction actions in sequence or simultaneously to complete the redeployment of the cluster topology configuration, so as to improve the cluster balance and restore it to a performance level that can continue to execute the current task; the dynamic reconstruction action set is expressed as: （1.14） in act t｜mn Represents a cluster node node (m,n) In the dynamic reconstruction process t The reconstruction action performed at the moment, and the reconstruction action can be defined as act t｜mn =[ s l i”j” ( t-1 ), s l ij ( t )], which defines the cluster nodes node (m,n) In the dynamic reconstruction process t The position of the moment in the relative coordinate system from the previous moment s l i”j” ( t-1 ) Move to the current position s l ij ( t ), the refactoring action defined act t｜mn According to the cluster nodes described in formula (1.8) node (m,n) Mobile features mov t |[ S t ,M t ] is transformed; further clustering c m The dynamic reconstruction action set is expressed as: （1.15）。 5. The cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning according to claim 1 is characterized in that: The deep neural network model in step 2) includes: 1) ResNet module: used to process mission area features, service coverage features, and damage area features, and extract multi-dimensional feature information; 2) LSTM module: used to extract the time series features of high-dimensional information and analyze the dynamic changes of cluster systems; 3) Attention mechanism: used to focus on the interactions within and between clusters, and support dynamic reconstruction across levels.

6. The cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning according to claim 1 is characterized in that: The reinforcement learning algorithm model in step 3) is optimized by a policy gradient reward function, and the reward function includes: 1) Cluster balance: represents the mean square error of the node redundancy values ​​in each cluster during the task execution phase; 2) Reconstruction time: the cumulative time of the dynamic reconstruction process, as one of the optimization objectives; 3) Strategy optimization: Through experience replay and policy gradient algorithm, TD (λ), V-trace and UPGO techniques are used to optimize the reconstruction strategy.

7. The cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning according to claim 6 is characterized in that: In step 3), for each dynamic reconstruction action act t｜mn Once the reconstruction action is completed, it is considered that the moving node starts to perform the current task at the new location, and then the service coverage feature matrix can be updated S t cover , the current cluster balance is calculated according to formula (1.4). The goal of the cluster system is to improve its cluster balance as efficiently as possible; the reward function should be defined for cluster balance and reconstruction time as follows: （1.16） in Represents the dynamic reconstruction process The reward value at that moment, Represents the dynamic reconstruction process ( The cumulative reconstruction time of the moment, Represents the dynamic reconstruction process The cumulative reconstruction time of the moment, Indicates the cluster system balance threshold. represents the discount factor of the reward function, Indicates the cumulative reconstruction time to complete the entire reconstruction process; Reward the dynamic reconstruction strategy and train the parameters of the DNN model through the sample data set θ i ; For the next policy iteration process i +1, generate new initial multi-dimensional situation feature information, and then use the dynamic reconstruction process θ i The DNN model as a parameter generates a new reconstruction strategy; for the first policy iteration process, the dynamic reconstruction process uses the parameters θ The DNN model is randomly initialized with 0. The model aims to approximate the value function of the reconstruction strategy and multiple policy functions describing the reconstruction actions at the same time. The DNN is trained in the policy iteration process of dynamic reconstruction. When executing the dynamic reconstruction process, the reconstruction agent selects a series of reconstruction actions to generate a complete strategy. The reconstruction agent needs to output three probability matrices for each selected reconstruction action, and at the same time use the reward R of the reconstruction action as the output. t As a value label; The reinforcement learning algorithm iteratively performs a dynamic reconstruction process during the policy iteration process: training the DNN parameters so that the reconstruction probability and value predicted by the neural network are closer to the reconstruction strategy and reward; by maximizing the similarity between the reconstruction action probability predicted by the DNN and the reconstruction strategy, the policy gradient is expressed as: （1.17） （1.18） in r ( s t ,a t )express t Reconstruct the reward value of the action at all times, s t express t The state of the moment, a t express t The reconstruction action of the moment, V ( s t ) represents the average value, A ( s t ,a t ) represents the action advantage value; asynchronous update based on experience replay technology, through optimization strategy Make the expectation of reward Maximization is expressed as: （1.19） Adjust the DNN to minimize the gap between the predicted value and the reward value of the reconstruction strategy. Apply TD(), V-trace and UPGO training to update the neural network and push the strategy to a trajectory that is better than the average return. When performing the dynamic reconstruction process in the next policy iteration, the dynamic reconstruction agent uses the new DNN parameters to better search for the optimal reconstruction action.

8. The cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning according to claim 1 is characterized in that: The DPPO alliance learning model includes: 1) Alliance game: Multi-target tasks and destruction scenarios are randomly generated, and the main agent plays against opponents in the alliance archive; 2) Virtual self-learning mechanism: select opponents with high winning rate in the alliance archive to play games and promote the training of the main agent; 3) Three types of agents: including the main agent, the main explorer and the alliance explorer, which work together to optimize the strategy.

9. The cluster system dynamic reconstruction decision method based on DPPO deep reinforcement learning according to claim 1 is characterized in that: The dynamic reconstruction strategy generation includes: 1) Dynamic reconstruction action generation: The dynamic reconstruction agent uses a deep neural network model to input multi-dimensional situation features and generate a series of reconstruction actions; 2) Topology redeployment: Execute the dispatched reconstruction action, redeploy the topology of the cluster system, and improve the cluster balance.

Images