A multi-dimensional situation awareness method and device for intelligent operation and maintenance of cluster systems

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a multi-dimensional situational awareness method and device for intelligent operation and maintenance of cluster systems. The present invention comprehensively improves the operation and maintenance capabilities of cluster systems through multi-level cluster system modeling, real-time multi-dimensional situation information collection, data fusion and processing, and situation prediction based on reinforcement learning. The present invention effectively improves the intelligent operation and maintenance capabilities of cluster systems in complex environments, can greatly improve the operating efficiency of the system, reduce operation and maintenance costs, and improve the stability of task execution and system responsiveness, and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-dimensional situation awareness method and device for intelligent operation and maintenance of a cluster system. Background Art

[0002] With the rapid development of information and intelligent technology, cluster systems have become an important foundation for the execution of complex tasks in many fields. Cluster systems are usually composed of multiple nodes (such as sensors, computing units, devices, etc.), which are connected and collaborated through the network to achieve dynamic allocation and coordination of tasks. Typical cluster system application scenarios include post-disaster repair, intelligent transportation, industrial automation, smart cities, national defense systems, etc. In these scenarios, cluster systems need to be able to cope with complex situations such as environmental changes, task adjustments, and equipment failures. Therefore, efficient operation and maintenance of cluster systems has become a key issue that needs to be solved urgently.

[0003] The traditional cluster system operation and maintenance methods mainly rely on regular maintenance and post-maintenance modes. Both methods have their limitations: 1. Limitations of regular maintenance: Although the regular maintenance mode can reduce the risk of system failure, it cannot reflect the operating status of the system in real time, which easily leads to waste of resources and increased operation and maintenance costs. In addition, in complex cluster systems, the differences between devices make it difficult for a unified maintenance cycle to effectively cover the actual needs of the entire system. 2. Limitations of post-maintenance: The post-maintenance mode relies on repairing the fault only after it occurs, which often causes a long system downtime, affecting the continuity and efficiency of the overall task execution. Especially in critical mission scenarios, the lag of post-maintenance will bring serious risks.

[0004] In order to improve the operating efficiency of cluster systems, reduce maintenance costs and reduce system downtime, intelligent operation and maintenance technology has emerged. Intelligent operation and maintenance aims to prevent possible failures in advance and optimize operation and maintenance strategies through real-time status monitoring, fault prediction and automated decision-making, thereby achieving more efficient and reliable system operation. However, the existing intelligent operation and maintenance systems still face the following challenges when dealing with the complex multi-dimensional situation of cluster systems:

[0005] 1. Comprehensive perception of multi-dimensional situation information: The multi-dimensional situation information in the cluster system includes node status, link status, task execution status, environmental changes, etc., which have spatiotemporal dynamic characteristics. How to comprehensively and real-time collect this situation information and integrate multi-source data for analysis is a key issue facing the intelligent operation and maintenance system;

[0006] 2. Accuracy of situation prediction: The operating environment of cluster systems is complex and changeable. Simply relying on historical data for situation prediction is difficult to cope with dynamic task adjustments and environmental disturbances. How to accurately predict the future operating situation of the system through efficient prediction algorithms to ensure the foresight and rationality of operation and maintenance decisions is still a technical difficulty.

[0007] In the early stage, the applicant aimed at the problem of "predictive maintenance" of three-level time dynamic cluster system under the condition of system performance degradation in the long-term operation stage. By comprehensively weighing the system degradation state and the imbalance characteristics of maintenance benefits, a predictive maintenance plan was generated through a method based on DQN deep reinforcement learning. For details, please refer to the Chinese invention patent application (application number: 202411357728X, application date: 20240927) disclosed a predictive maintenance decision method for cluster system based on DQN; for the problem of "post-disaster repair" of multiple teams in a two-level time dynamic cluster system facing large-scale local damage, a corrective maintenance plan was generated by coordinating the two-level decisions of maintenance timing and path planning through an Actor-Critic deep reinforcement learning method. For details, please refer to the Chinese invention patent application (application number: 2024112822477, application date: 20240913), which discloses a method for post-disaster restorative maintenance of cluster systems based on the AC-MCTS algorithm; for the "dynamic reconstruction" problem of a three-level spatiotemporal dynamic cluster system with multi-formation characteristics, a two-level dynamic reconstruction of functional reconstruction between nodes within the three-level cluster system and functional reconstruction between clusters is analyzed through a method based on DPPO deep reinforcement learning to generate a dynamic reconstruction plan. For details, please refer to the Chinese invention patent application (application number: 2024114796053, application date: 20241023), which discloses a dynamic reconstruction decision method for cluster systems based on DPPO deep reinforcement learning. Summary of the invention

[0008] In response to the above problems, the present invention proposes a multi-dimensional situation awareness method for intelligent operation and maintenance of cluster systems. Through multi-level cluster system modeling, real-time multi-dimensional situation information collection, data fusion and processing, and situation prediction based on reinforcement learning, the operation and maintenance capabilities of cluster systems are comprehensively improved. It effectively improves the intelligent operation and maintenance capabilities of cluster systems in complex environments, can greatly improve the operating efficiency of the system, reduce operation and maintenance costs, and improve the stability of task execution and system responsiveness, and has broad application prospects.

[0009] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0010] A multi-dimensional situation awareness method for intelligent operation and maintenance of a cluster system, wherein the cluster system is a power system cluster, a drone cluster system, or a large-scale battery cluster system in an industrial energy storage system, and the method comprises the following steps:

[0011] a) Cluster system modeling: A multi-dimensional situation model of the cluster system is constructed through a multi-level modeling method of cluster-cluster-node, and the model includes a time dynamic cluster system and a space-time dynamic cluster system;

[0012] b) Multi-dimensional situation information collection: The multi-dimensional situation information of the cluster system is collected in real time through sensors, including node health status, link status, task execution status and environmental change information;

[0013] c) Situation information processing: Classify, fuse and analyze the collected multi-dimensional situation information to generate an environmental information set H, which is used to describe the current state of the cluster system;

[0014] d) Situation Prediction: Situation prediction is performed based on reinforcement learning algorithms, including the Deep Q Network (DQN) and Distributed Proximal Policy Optimization (DPPO) algorithm, to predict future situation development trends.

[0015] Preferably, step 1) is implemented by constructing a node set N and a cluster set C of the cluster system, wherein the node set is defined as:

[0016] N={n1, n2,…, n n ,…}

[0017] Letter N n represents the nth node in the cluster, and the cluster set C is defined as:

[0018] C={c1, c2,…, c m …}

[0019] Among them, c m Represents the mth cluster in the cluster at the current moment.

[0020] Preferably, the execution of the situation prediction is based on the dynamic characteristics of the cluster system, specifically including the following formula:

[0021] a) Time dynamic features: Through the distance matrix L={l ij} and the adjacency matrix A={a ij}, calculate the shortest path d between node i and node j ij , and is predicted using the following formula:

[0022] d ij =min(l ij ,a ij )

[0023] b) Spatiotemporal dynamic characteristics: by constructing the cluster redundancy matrix M i∈ ={∈ ij}, calculate the task execution redundancy in each cluster, and adjust the link status between nodes based on the task distribution.

[0024] As a preferred method, situation information fusion adopts a multi-dimensional information fusion method based on environmental objects to fuse node and link information with external environmental change information to generate a situation graph G, and update the environmental information set H through the following formula:

[0025] G=f(N,C,L,A)

[0026] Among them, f represents the fusion function of the environment object.

[0027] As a preferred method, situation prediction is implemented through a reinforcement learning algorithm. Based on the deep Q network (DQN) algorithm, feature extraction and strategy optimization are performed on multi-dimensional situation information, which specifically includes the following update formula:

[0028] ;

[0029] R t represents the reward at time t, γ represents the discount factor, π represents the behavior strategy, s represents the state, a represents the action, S t represents the observation result of the state at time t, A t represents the action performed at time t;

[0030] During the policy iteration process, the updated loss function is expressed as:

[0031] ;

[0032] where γ represents the discount factor; θ i represents the network parameters of DQN in the i-th PI process; θ i - represents the network parameters of the target DQN in the i-th PI process; the network parameters of the target DQN are set to the network parameters of the i-1th PI process, θ i - =θ i-1 ; Q(s,a,θ i ) represents the approximate value function based on deep neural network, θ i represents the parameters of the neural network; (s,a,r,s ’ )~U(D) represents the experience data of the DQN agent, s represents the current state, a represents the current action, s ’ Indicates the state at the next moment, a ’ Indicates the next action.

[0033] Preferably, the final bias based on the Distributed Proximal Policy Optimization (DPPO) algorithm is generated by a two-layer hypernetwork with ReLU nonlinearity; the final loss function is:

[0034]

[0035] Where b represents the number of samples sampled from the experience pool, ,θ - is the parameter of the target network, so the TD error can be expressed as:

[0036] TD error =(r+γQ tot (target))-Q tot (evaluate)

[0037] Where Q tot (target) indicates the state s ' In the case of all behaviors, the maximum value Q is obtained tot ; According to the IGM condition, the input is the maximum action value of each agent in this state; Q tot (evaluate): In state s, the Q that can be obtained according to the current network strategy tot ; r represents the reward value under the current strategy.

[0038] Preferably, situational awareness includes the following scenarios:

[0039] a) Predictive maintenance: Preventive maintenance can be performed in advance by predicting node degradation status and task execution status;

[0040] b) Post-disaster repair: Plan the optimal repair path based on post-disaster node and link damage and situation information;

[0041] c) Dynamic Reconfiguration: Dynamically adjust the topology of the cluster system according to task requirements to maintain task execution capabilities.

[0042] Preferably, the node sets and cluster sets are dynamically managed based on hash tables to support real-time adjustment and expansion of the cluster system.

[0043] Preferably, the link set L is as follows:

[0044]

[0045] Where s(a ij ) represents the connection node n i and node n j The state of the edge, element l ij Represents node n i and nj The geographical distance between them.

[0046] Furthermore, the present invention also discloses a multi-dimensional situation awareness device for intelligent operation and maintenance of a cluster system, wherein the device executes the method, including:

[0047] a) a situation information collection module, which is used to collect multi-dimensional situation information in the cluster system in real time through sensors, the information including node status, link status, task execution status and environmental change information;

[0048] b) Data processing module, used to classify, integrate and analyze the collected multi-dimensional situation information, generate environmental information set, and describe the operation situation of the current cluster system;

[0049] c) A situation prediction module predicts future situation changes based on a reinforcement learning algorithm, which includes a deep Q network (DQN) and a distributed proximal policy optimization (DPPO) algorithm to generate prediction results of future situations.

[0050] The present invention adopts the above-mentioned technical solution, through multi-level cluster system modeling, real-time multi-dimensional situation information collection, data fusion and processing, and situation prediction based on reinforcement learning, to comprehensively improve the operation and maintenance capabilities of the cluster system. It effectively improves the intelligent operation and maintenance capabilities of the cluster system in complex environments, can greatly improve the operating efficiency of the system, reduce operation and maintenance costs, improve the stability of task execution and system responsiveness, and has broad application prospects. Specifically, it includes the following effects:

[0051] 1. Improve the situational awareness of the cluster system: The present invention divides the cluster system into three levels: cluster, cluster and node through a multi-level modeling method, which can accurately describe the status and interaction relationship of different levels in the cluster system. Through the situation information acquisition module, multi-dimensional situation information including node health status, link status, task execution status and environmental changes can be obtained in real time, thereby comprehensively improving the system's ability to perceive the current operating situation. Compared with traditional methods, the present invention can more comprehensively obtain and analyze multi-dimensional situation information.

[0052] 2. Enhance the accuracy and efficiency of situation information processing: By classifying, fusing and analyzing multi-dimensional situation information, the present invention can efficiently process data from different sensors and generate an accurate set of environmental information. The data fusion technology based on complex network theory can integrate information on nodes, links and external environments, providing a basis for subsequent situation prediction. This information processing mechanism effectively reduces information redundancy and errors, ensuring the efficiency and accuracy of situation awareness.

[0053] 3. Improve the accuracy and reliability of situation prediction: The present invention predicts the future situation of the cluster system based on reinforcement learning algorithms (such as deep Q network DQN and distributed proximal policy optimization DPPO). Reinforcement learning algorithms can improve the ability to predict complex situation changes through training and self-optimization of historical data. Compared with traditional rule-based or statistical prediction methods, the prediction accuracy of the present invention is higher, especially when facing complex and changeable cluster operating environments, it can provide more reliable situation prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the overall architecture diagram of the intelligent operation and maintenance of the present invention.

[0055] Figure 2 Modeling example diagrams for various types of cluster systems.

[0056] Figure 3 This is a cluster system operation and maintenance status diagram.

[0057] Figure 4 Degraded state analysis for cluster nodes. DETAILED DESCRIPTION

[0058] The present invention explains the concept connotation of cluster system, expounds the composition and characteristics of cluster system, and further classifies cluster system in terms of composition and characteristics. In terms of composition, the multi-level composition form of cluster-cluster-node is considered. In terms of characteristics, the temporal dynamic characteristics and spatiotemporal dynamic characteristics are targeted, and the conditions such as scale, synergy, and uncertainty are comprehensively considered. Four common technologies of modeling, perception, decision-making, and evaluation are integrated to build an intelligent operation and maintenance framework of cluster system, which includes: cluster system modeling module, multi-dimensional situation perception module, intelligent decision-making planning module, and cluster effectiveness evaluation module, and the complex interactive relationship between the above four modules is sorted out, such as Figure 1 As shown. The cluster system modeling module needs to conduct modeling analysis from multiple levels for different types of cluster systems; the multi-dimensional situation awareness module needs to deconstruct and classify the multi-dimensional situation information of the cluster system operation stage in a complex environment, and qualitatively and quantitatively describe the key information to support multi-dimensional situation awareness; the intelligent decision-making and planning module needs to sort out the decision-making and planning mathematical models of various operation and maintenance problems for the three main types of operation and maintenance problems, namely preventive maintenance, corrective maintenance, and operation management, and design a reinforcement learning architecture to support decision-making and planning; the cluster effectiveness evaluation module needs to construct its indicator system for the special and general characteristics of the cluster system, and propose an evaluation algorithm for key technical indicators to evaluate the performance of the cluster system in a complex environment and guide the decision-making and planning module to optimize the solution.

[0059] The construction of the intelligent operation and maintenance framework of cluster systems is a pioneering work, which aims to build an open, basic common technology framework to provide a high-precision, high-efficiency, and low-cost intelligent operation and maintenance solution system for various cluster systems in different operation and maintenance scenarios. The present invention builds its framework by integrating the common technologies of intelligent operation and maintenance, and conducts research on some algorithms on this basis. Future researchers can continue to expand the relevant research on intelligent operation and maintenance technologies based on the common technology framework. In addition, the proposed common technology framework can provide an environment for interaction and training for different types of reinforcement learning methods. By unifying the interface specifications in the intelligent operation and maintenance framework and refining the common processes of reinforcement learning algorithms, it can be compatible with many different types of deep reinforcement learning algorithms.

[0060] 1.1 Cluster system modeling module

[0061] The present invention discusses the cluster system modeling module in the intelligent operation and maintenance framework. It first explains the conceptual connotation of the cluster system, then analyzes the multi-level morphology of the cluster system from the perspective of composition, and analyzes the temporal and spatial dynamic characteristics from the perspective of characteristics. On this basis, modeling and analysis are carried out for different categories of cluster systems.

[0062] 1.1.1 Concept of Cluster System

[0063] At this stage, cluster systems have gradually become a very important representative product form and an important support for the national economy and national defense, such as various infrastructure systems, various equipment clusters, and various intelligent manufacturing systems. In a broad sense, a system refers to a whole formed by a certain internal connection within a certain space and time range. It is a system composed of different systems and has a certain order. From this perspective, the top level of a generalized system is a system, that is, a system of systems. For a system, what is usually considered is a whole composed of different heterogeneous systems. In contrast, clusters usually focus on a whole composed of homogeneous systems, which is the next level of the system. In addition, compared with network systems, clusters pay more attention to the synergy of large-scale homogeneous systems, but different systems in a cluster still have obvious network characteristics.

[0064] At present, a lot of exploratory research has been carried out on cluster systems at home and abroad. Cluster systems are usually composed of large-scale relatively simple systems, and global collaborative behavior is achieved through local collaboration between individual systems. In order to further study cluster systems, this paper will conduct a multi-level deconstruction analysis on the composition of cluster systems. In addition, cluster systems will be classified according to their different characteristics.

[0065] Regarding the composition of the cluster system, the present invention deconstructs it from the functional forms of three levels: Swarm, Cluster, and Node. The cluster system can construct a cluster-node (Swarm-Node, SN) form, in which the nodes inside the cluster are combined into the functional form of the cluster system according to a certain order and internal connections between nodes, and clusters can interact with each other through node cross-cluster connections; in addition, the cluster system can also construct a cluster-cluster-node (Swarm-Cluster-Node, SCN) form, in which the nodes inside the cluster are combined into clusters according to a certain order and internal connections, and clusters interact with different nodes and different clusters to form the functional form of the cluster system.

[0066] Regarding the characteristics of cluster systems, the present invention divides cluster systems into two categories: time dynamic clusters and space-time dynamic clusters, based on the two core characteristics of time dynamic characteristics and space-time dynamic characteristics. A time dynamic cluster refers to a cluster system whose spatial attributes are mainly static characteristics, while its temporal attributes are mainly dynamic characteristics. For example, various infrastructure systems such as power networks usually show a relatively static state in their spatial position characteristics, but their operation, failure, and degradation properties show dynamic change characteristics over time. A space-time dynamic cluster refers to a cluster system whose spatial and temporal attributes both show dynamic characteristics. For example, various equipment clusters such as cluster drone systems usually show a moving state in their spatial position characteristics during missions, and their operation, failure, and degradation properties still show dynamic change characteristics over time.

[0067] Based on the above explanation of the concept of cluster system, for the cluster system cluster-cluster-node multi-level form, considering the two core characteristics of cluster system time dynamics & spatiotemporal dynamics, further analyze the multi-dimensional situation that needs to be discussed in the operation of cluster system under scale, coordination and uncertainty conditions, focusing on the state information related to cluster system operation and maintenance. For example, for the cluster-node two-level form of time dynamic cluster system, assuming that the spatial position of the nodes in the cluster presents a large-scale distributed layout, in the long-term operation stage, each node and link in the cluster will experience operation and maintenance situations such as degradation and failure. At the same time, when facing large-scale natural disasters such as typhoons and earthquakes, there will be large-scale node destruction in the local area of ​​the cluster. In addition, for the cluster-cluster-node three-level form of spatiotemporal dynamic cluster system, assuming that the spatial position of each node in its operation stage is dynamically adjusted according to real-time changes, and clusters are constructed in real time according to a certain order and internal connections between nodes to complete the collaborative interaction between multiple clusters and multiple nodes. In such a scenario, the operation and maintenance situation inside the cluster system is further described: for the cluster level, it is necessary to consider failure, degradation, local attack and other situations; for the node level, it is necessary to consider the failure and degradation of nodes and links.

[0068] 1.1.2 Cluster System Modeling and Analysis

[0069] Based on the explanation of the concept of cluster system, this paper models and analyzes the cluster system from the perspective of composition, targeting the multi-level forms of clusters, clusters, and nodes, and from the perspective of characteristics, targeting the temporal dynamic characteristics and spatiotemporal dynamic characteristics of the cluster system. This topic mainly studies three typical cluster systems, including two-level temporal dynamic clusters, three-level temporal dynamic clusters, and three-level spatiotemporal dynamic clusters, such as Figure 2 As shown in the figure, a simple description is given from the perspectives of composition and dynamic characteristics. For two-level time dynamic clusters, clusters composed of large-scale distributed systems are representative examples. Considering that the distributed systems in the cluster are combined in a certain order and connected to the internal network of the cluster to form a two-level cluster-node form, each system in the cluster exhibits static characteristics in the spatial dimension, such as the spatial position and other attributes remain static, and exhibits dynamic characteristics in the time dimension, such as health, degradation, damage and other operation and maintenance status dynamically change over time. Specific case objects include infrastructure networks represented by large-scale distributed power systems. For three-level time dynamic clusters, clusters composed of large-scale series-parallel-bridge systems are representative examples. Considering that each system in the cluster is combined in a certain series-parallel-bridge order to form a cluster form, clusters are combined based on parallel or n / k redundant structures to form a cluster form. The static and dynamic characteristics of each system in the cluster in the spatial and temporal dimensions are similar to those of two-level time dynamic clusters. Specific case objects include large-scale battery clusters for industrial energy storage systems. For the three-level spatiotemporal dynamic cluster, the cluster composed of multiple formation systems is a representative example. Considering that the systems in the cluster are combined into a formation according to a certain topological configuration and collaborative relationship, the formations are combined into a cluster based on multi-task collaborative relationships. The systems in the cluster show dynamic characteristics in the spatial dimension. For example, their spatial position and other attributes need to be adjusted in real time according to changes in the situation in a complex environment. The dynamic characteristics shown in the time dimension are similar to those of the time dynamic cluster. Specific case objects include cluster systems composed of multiple UAV formations.

[0070] Based on the above analysis of the cluster system, the cluster system model is analyzed from the perspective of multi-level composition. The spatiotemporal dynamic and temporal dynamic characteristics are mainly reflected in the multi-dimensional attributes of the cluster system model, which will be described in detail in the next section on the introduction of the multi-dimensional situation awareness module. For the cluster system object S under study, a node set N is first established to describe all nodes n in the cluster S, as shown below:

[0071] N={n1, n2,…, n n,…}(1.1)

[0072] The lowercase cursive letter n n Represents the nth node in the cluster. The number counting variable n will also be used in the subsequent parts of the present invention. This parameter only represents the integer number. The node set N is the number of each node, which is the unique identifier of each node. In the present invention, the integer number is used as the unique identifier of the node. At the same time, it can also be used as the unique identifier based on the product code of each node in the actual scenario. In the intelligent operation and maintenance framework, for the cluster system object S, the data structure of its node set N is initialized in the form of a set, and the node set is constructed based on a hash table. The keyword of the hash table is the unique identifier of the node, which is the node number in the present invention. The stored data elements are the relevant attributes of the node, which involve spatiotemporal dynamics and time dynamics. It will be described in detail in the next section about the introduction of the multi-dimensional situation awareness module. In addition, before forming a cluster form, when some or all nodes in the node set need to be combined into a cluster form in a certain order, it is necessary to establish a cluster set C to describe all clusters c in the cluster at the current moment according to factors such as the specific task requirements in the current scenario, as shown below:

[0073] C={c1, c2,…, c m …} (1.2)

[0074] The lowercase cursive letter c m Represents the mth cluster in the cluster at the current moment. The number counting variable m will also be used in the subsequent parts of the present invention. This parameter only represents an integer number. Cluster set C The number of each cluster is a dynamic identifier of each cluster, which is related to factors such as tasks, and changes dynamically accordingly. It is generated when a cluster needs to be formed. The cluster set is constructed based on a hash table. The keyword of the hash table is the cluster number, and the stored data elements are the relevant attributes of the cluster, which involve spatiotemporal dynamics and time dynamics. It will be described in detail in the next section on the introduction of the multidimensional situational awareness module. When the cluster set needs to be dynamically adjusted due to changes in factors such as real-time tasks, the corresponding cluster data can be dynamically added, deleted or modified based on the data structure of the hash table.

[0075] On the basis of defining node sets and cluster sets, the multi-level form of clusters is further analyzed. For two-level clusters, all nodes in the node set can form a two-level cluster form by combining them in a certain order and the internal connections between nodes. By establishing its node set N, the one-to-many tree structure relationship between the cluster and all its nodes can be described. However, for a three-level cluster, after establishing its node set, before forming a three-level cluster form, it is necessary to define its three-level model. Specifically, the cluster system object S has a one-to-many tree structure relationship with all clusters in its cluster set, and each cluster has a one-to-many tree structure relationship with all nodes therein, thereby forming a cluster-cluster-node three-level model. The intelligent operation and maintenance framework of the present invention constructs the above three-level model by nesting hash tables in hash tables. First, a one-to-many data structure between clusters is constructed based on the hash table. The keyword of the hash table is still the dynamic number of the cluster, and the stored data element is the node set of the cluster. It is constructed using a nested hash table. The keyword of the nested hash table is the dynamic number of each node in the cluster, and the stored data element is the unique identifier of the node, that is, the number of the node in the node set N. Therefore, the three-level cluster model SM is constructed by nesting hash tables in hash tables. III , the mapping relationship between the unique identifier of a node and its dynamic number in the cluster can be described as follows:

[0076] (1.3)

[0077] where n (i,j) Represents cluster c i The number counting variables i and j will also be used in the subsequent parts of the present invention. This parameter only represents the integer number. n It represents the nth node in the node set N in formula (1.1). Its number n is the unique identifier of each node and is a unique fixed number. Cluster c i The dynamic number i of the cluster is consistent with the dynamic number of each cluster in formula (1.2). When the cluster C changes with factors such as real-time tasks and is dynamically adjusted, the three-level cluster model SM III Then, the corresponding cluster level mapping relationship is dynamically added, deleted or modified based on the data structure of the hash table. i It consists of multiple nodes, where node n (i,j)The subscript number is the dynamic identification of the node in the cluster, which is related to factors such as tasks and changes dynamically. The corresponding node level mapping relationship is dynamically added, deleted or modified based on the data structure of the nested hash table. The keyword of the nested hash table is the counting variable j in the dynamic identification of the node. The data element stored in the nested hash table is the unique identification of the node, that is, the number of the node in the node set N. In short, the number of each node in the cluster in the node set N is the unique identification of the node, which always remains unchanged. When the three-level cluster model SM III When a node in the node set N needs to be called to form a cluster according to factors such as task requirements, the node will be given a corresponding dynamic number. The dynamic numbers of the cluster and the node can be dynamically rebuilt according to factors such as changes in real-time task requirements.

[0078] 1.2 Multi-dimensional Situational Awareness Module

[0079] The present invention discusses the multi-dimensional situation awareness module in the intelligent operation and maintenance framework. Firstly, the conceptual connotation of cluster system operation and maintenance situation and perception is explained. On this basis, the multi-dimensional situation awareness of cluster system intelligent operation and maintenance is analyzed.

[0080] In the situational awareness of cluster system operation and maintenance, the "state" of operation and maintenance refers to the situation, state or environment of the cluster system in the current operation stage, while the "momentum" of operation and maintenance refers to the potential trend or influence of the cluster system in the future operation stage. There is interaction and influence between the "state" and "momentum" of operation and maintenance. When the "state" of operation and maintenance changes, it will also trigger changes in the "momentum" of operation and maintenance. For example, if a large-scale destruction occurs, including natural disasters such as hurricanes or earthquakes, this will cause the current state of the cluster system to change, and this change may trigger a series of subsequent influences and potential forces, such as the destruction of large-scale nodes of the cluster system, the rupture of links, the evacuation of personnel, etc. Therefore, the change of the "state" of operation and maintenance will trigger the change of the "momentum" of operation and maintenance. On the other hand, when the "momentum" of operation and maintenance changes, it will also trigger the change of the "state" of operation and maintenance. For example, fluctuations in the military situation or changes in the political situation will have an impact on the battlefield environment, the individual form and group form of weapons and equipment. This change in the "momentum" of operation and maintenance may cause changes in the behavior or form of the equipment system, thereby changing the state of the entire battlefield. Therefore, changes in the "momentum" of operation and maintenance will also trigger changes in the "state" of operation and maintenance. In the situational awareness of intelligent operation and maintenance of cluster systems, it is crucial to understand and analyze the interaction between the "momentum" and "state" of operation and maintenance. By observing and predicting this interaction, we can better respond to and adapt to different scenarios and environments to support better decisions and actions.

[0081] At the same time, in the situational awareness of intelligent operation and maintenance of cluster systems, changes in the "feeling" of operation and maintenance will trigger changes in the "knowing" of operation and maintenance. This is because the perception of operation and maintenance situation is the process of collecting and obtaining information through sensors or sensors. When sensors observe certain changes in complex environments, they will convert them into various forms of signals, which will then be processed, parsed and stored as meaningful operation and maintenance situation information. This is the generation of "knowing" of operation and maintenance. Conversely, changes in the "knowing" of operation and maintenance will also trigger changes in the "feeling" of operation and maintenance. In the process of perceiving the operation and maintenance situation of cluster systems, through the processing, parsing and storage of operation and maintenance situation information, we can obtain understanding and cognition of the complex environment, so that we can respond to changes in the complex environment. These responses may include adjusting the parameters of the cluster system perception devices, reorganizing the functional form of the cluster system, changing the goals or methods of operation and maintenance situation awareness, etc., which will affect the results of the cluster system operation and maintenance situation awareness. In summary, in the situational awareness of intelligent operation and maintenance of cluster systems, the "feeling" and "knowing" of operation and maintenance are interrelated and mutually influential. Changes in the "feeling" of operation and maintenance trigger changes in the "knowing" of operation and maintenance, and changes in the "knowing" of operation and maintenance will further affect changes in the "feeling" of operation and maintenance. This interaction enables the situational awareness of intelligent operation and maintenance of cluster systems to adapt to changes in complex environments and provide accurate and real-time operation and maintenance situation information to support decision-making planning and performance evaluation of intelligent operation and maintenance.

[0082] Based on the above discussion, to build a multi-dimensional situation awareness module for intelligent operation and maintenance of cluster systems, it is necessary to comprehensively analyze the interaction between the "state" and "potential" of cluster system operation and maintenance, as well as the interaction between the "sensing" and "knowledge" of operation and maintenance, and then construct its operation and maintenance situation map for the multi-dimensional situation information of the cluster system operation stage under complex environment, such as Figure 3 As shown. For the “state” of operation and maintenance, based on the definitions of storage attribute information for the node set N and the cluster set C respectively in formulas (1.1) and (1.2), and the definition of storage attribute information for the three-level cluster model SM in formula (1.3), III Based on the definition of , various information in the cluster system operation scenario is further deconstructed, and the environmental information, cluster-cluster-node operation information, degradation information, etc. are classified, and then the similar and highly correlated information is integrated, and then the cluster-cluster-node information is distinguished according to the temporal dynamic characteristics and spatiotemporal dynamic characteristics. On this basis, the environmental object εnv is defined and the environmental information set H is constructed, as shown in Figure 3It is shown that the set is composed of multi-dimensional situation information, including task information, damage information, map information, etc. It also includes the constructed cluster information. For preventive maintenance and corrective maintenance, it also needs to include the guarantee system information. For the "trend" of operation and maintenance, focus on the trend between the current "state" and the future "state", analyze the real-time observation data of the multi-dimensional situation information in the environmental information set H, study the relevant prediction methods, describe the potential trend of the multi-dimensional situation information in the future operation stage, and then predict and update the "state" of operation and maintenance. For the "sense" of operation and maintenance, it is necessary to collect and obtain the operation and maintenance situation information in a complex environment through sensors or sensors, and convert it into various forms of signals. For the "knowledge" of operation and maintenance, it is necessary to qualitatively and quantitatively describe the operation and maintenance situation information in various scenarios during the operation stage of the cluster system. The present invention focuses on the framework of common technologies for intelligent operation and maintenance. In the multidimensional situational awareness module, only the input interface design of multidimensional situation data is done for the "sense" of operation and maintenance, and no in-depth research is done on technologies such as detection and diagnosis. However, the intelligent operation and maintenance framework proposed by the present invention still supports future researchers to integrate related technologies such as detection and diagnosis in the multidimensional situational awareness module; in addition, the multidimensional situational awareness module is designed for the "knowledge" of operation and maintenance. By designing environmental objects and their environmental information set H, and associating the node set N and cluster set C defined by the cluster modeling module, it has been able to process, parse and store the operation and maintenance situation information of the cluster system to support the decision-making planning and performance evaluation of intelligent operation and maintenance. Therefore, in the subsequent chapters of the present invention, the "sense" and "knowledge" of operation and maintenance are no longer distinguished, but the word "perception" is used to uniformly describe the input of environmental observations, as well as the processing, parsing and storage of situation information. The subsequent content of the present invention will be simply discussed with some algorithms integrated in the multidimensional situational awareness module, including data structure examples of operation and maintenance "state" and prediction methods of operation and maintenance "potential".

[0083] (1) Data structure example of “state”

[0084] In the node set N defined in the cluster system modeling module, the node's state, location and other attributes can be stored. However, for some specific cluster systems, it is necessary to consider not only the state of all nodes in the node set, but also the state of all links between nodes.

[0085] Since the intelligent operation and maintenance framework of the present invention is developed in Python, starting from the requirement of analyzing the links between all nodes in the cluster system, it is easy to think of the NetworkX tool, which is an algorithm tool for graph theory and complex network modeling. This tool is also developed in Python and contains rich algorithm tools for graph theory and complex network modeling and analysis, which can facilitate researchers to carry out complex network simulation modeling and data analysis. Although the NetworkX tool is a quite mature tool in the field of graph theory and complex networks, its underlying code does not endow the data structures of nodes, edges, and adjacency matrices with the attributes of operation and maintenance situation. Therefore, when applying the NetworkX tool, the present invention comprehensively considers the operation and maintenance situation of the cluster system, and reconstructs the data structures of the nodes and edges of the cluster system based on the data structures of nodes and edges in the NetworkX underlying code, endowing them with relevant attributes of operation and maintenance situation.

[0086] Before introducing the data structures of the nodes and edges of the above cluster system, first, based on the complex network theory, the nodes and edges of the cluster system are simply defined and described. The cluster is represented as an undirected graph G with N nodes and K edges. Assume that the graph G is sparse, that is, K << N (N - 1) / 2. In addition, consider that the graph G is connected, that is, there is at least one path connecting any two nodes with a finite number of steps. Such a graph G needs two matrices to describe: the adjacency matrix A and the physical distance matrix L. A is an N × N adjacency matrix. If there is an edge connecting node n i and node n j , then the element a ij in the adjacency matrix is 1, otherwise it is 0. The element l ij of the matrix L represents the geographical distance between node n i and n j . Even if there is no edge element between node n i and node n j in the cluster, l ijIt is also known. According to the above definition, the G graph is a collection of nodes and known node pairs (which can be called edges, links, etc.). In the NetworkX tool, a node can be any hashable object, such as text, a string, another graph, a custom node object, etc. In addition, NetworkX can also add nodes containing node attributes at the same time. For nodes, after creating an empty G graph, regardless of whether the added node contains node attributes, the node set N of the cluster system object S of the present invention can generate the required data. The node set generates different node arrays for addition according to the normal, degraded, and damaged states of the nodes at the current moment. For edges, it is necessary to define the link set L of the cluster system, and then generate the edge data and add it to the created G graph. The present invention refers to the data structure of the adjacency matrix in the underlying encoding of NetworkX for the construction of the cluster system link set L, that is, hash nested hash, and adds attribute information such as operation and maintenance status on this basis. In addition, the required attribute information can still be added according to the actual scenario. The link set L is as follows:

[0087] (2.1)

[0088] Where s(a ij ) represents the connection node n i and node n j The state of the edge, the present invention mainly focuses on the operation and maintenance state, and different operation and maintenance problems may involve different states such as normal, degraded, and damaged. The state parameters can also be customized according to specific problems. In addition, the link set L only stores a ij =1 represents the link and its attributes. It should be noted that when analyzing the links between nodes in the cluster system, the present invention does not consider spin links for the time being. In addition, most of the problems in the present invention are analyzed for undirected graphs, that is, s(a ij ) and s(a ij ) has the same meaning, but the construction of this data structure still supports the description of the link status in the directed graph required in related problems.

[0089] Based on the above construction of the node set N and the link set L, we can generate node arrays and link arrays describing different states for the normal and damaged states of all nodes and their links at a specific moment in the present or future, and then use the NetworkX tool to simulate and analyze the complex network model of the cluster system.

[0090] (2) Examples of “potential” prediction methods

[0091] Unlike the "state" of operation and maintenance, which focuses on the current state of the cluster, the "potential" of operation and maintenance focuses on the future changing trend of the cluster. In the operation and maintenance of cluster systems, node degradation is a type of time dynamic feature that is of great concern. At this stage, the degradation of nodes can be analyzed using the State of Health (SOH), Remaining Useful Life (RUL), etc., and the corresponding prediction methods are also very rich. This invention introduces a SOH random degradation model as an example of the prediction method of the "potential" of operation and maintenance. The proposed intelligent operation and maintenance framework still supports the integration of richer prediction methods.

[0092] First, the SOH of the actual node object needs to be defined. For example, in the case study of Chapter 3 of this invention, for large-scale battery pack integration in industrial energy storage systems, based on the analysis of the capacity degradation model of each battery pack, its SOH is defined as the percentage of the current capacity to the initial capacity, so 0≤SOH≤100%. Then, in order to use the SOH of all nodes in the cluster to predict the trend of cluster status changes, it is assumed that the SOH of each node at any time follows the normal distribution N (µ,σ 2 ), this assumption means that the defined SOH only provides the average SOH of the nodes in the cluster, that is, µ=SOH, and there are still slight differences between each node due to materials, manufacturing processes and other environmental factors. Assume that the standard deviation σ is (1-µ) / 6, and its initial value is zero. Based on the assumption of discretization of the degradation process, there are multiple discrete degradation states in the node degradation process, and the cluster degradation state can also be constructed by the SOH of the discretized nodes. Based on the normal distribution N (µ, σ 2 ) As a result, by analyzing the probability density function of each node's SOH, we can obtain the probability density function of each node's SOH at each stage after a certain period of operation. For example, a node starts running from the state of SOH = 100%. According to a certain degradation law, when the node runs for 100 hours, 200 hours, 300 hours, 400 hours, 500 hours, 600 hours, and 700 hours, its SOH probability density function is as follows: Figure 4 shown.

[0093] The node SOH in the above figure is divided into six levels: ≥95%, 90~95%, 85~90%, 80~85% and ≤80%. The probability of each SOH level of the node in different operation stages is shown in Table 1.

[0094] Table 1 Probability of different SOH levels during cluster node operation

[0095] Running time (h) ≥95% 90~95% 85~90% 80~85% ≤80% 100 1 0 0 0 0 200 0.3730 0.6270 0 0 0 300 0.0133 0.9282 0.0584 0 0 400 0.0008 0.3722 0.6211 0.0060 0 500 0.0001 0.0722 0.7191 0.2076 0.0010 600 0 0.0133 0.3596 0.5686 0.0585 700 0 0.0029 0.1253 0.5586 0.3132

[0096] The present invention introduces the multidimensional situation awareness module in the intelligent operation and maintenance framework, explains the connotation of operation and maintenance "state" and "trend", and constructs a cluster system operation and maintenance situation diagram. On this basis, the data structure of the operation and maintenance "state" is designed, focusing on the introduction of environmental objects and their environmental information sets H, as well as cluster objects S and their related cluster models. Then, according to the temporal dynamic characteristics and spatiotemporal dynamic characteristics in the operation and maintenance situation diagram, the relevant prediction methods are studied to form a multidimensional situation awareness module to support the decision-making planning and performance evaluation of the cluster system.

[0097] 1.3 Based on the Deep Q Network (DQN) algorithm

[0098] For details about the deep Q network (DQN) algorithm, please refer to the Chinese invention patent application (application number: 202411357728X, application date: 20240927); among them: the objectives, constraints and variables of the predictive maintenance decision-making problem are as follows:

[0099] For the cluster system object under study, we focus on the three-level time dynamic clusters and establish a cluster set C To describe H clusters, consider each cluster c h (h=1,2,…,H) It consists of M×N nodes in a certain order and internal connections between nodes. Therefore, the predictive maintenance of cluster systems involves three decision-making levels, including node, cluster and cluster levels. The constraint is cost, the optimization goal is the reliability of the cluster, and the variable is the maintenance status of the node.

[0100] The goal of the predictive maintenance decision problem is to maximize reliability recovery under limited maintenance costs. For these three levels, the problem should be solved at the node level. The basic decision variables are clusters. c h Midpoint n (h,m,n) The state of predictive maintenance pms (h,m,n) (h=1,2,…,H;m=1,2,…,M;n=1,2,…,N) Therefore, when pms (h,m,n) =1, the node is subjected to predictive maintenance. pms (h,m,n) = 0, no predictive maintenance task is performed on the node. Consider a cluster of H×M×N, whose maintenance decision can be described by the maintenance strategy PMS as follows:

[0101] PMS=[PMS 1 ,PMS 2 ,…,PMS h ,…, PMS H ] (3.1)

[0102] In the formula PMS h It is a cluster c h The strategy matrix can be described as:

[0103] (3.2)

[0104] Each element of the maintenance strategy matrix pms (h,m,n) There are only two states (0-normal state, 1-repair and replacement state), then:

[0105] (3.3)

[0106] The initial state feature tensor of the cluster can be expressed as:

[0107] S=[S 1 rec ,S 2 rec ,…,S h rec ,…,S H rec ] (3.4)

[0108] Each element in the vector is a cluster c h The initial state characteristics can be expressed as:

[0109] (3.5)

[0110] In the formula, the matrix elements are nodes n (h,m,n) The initial state of the cluster. After the cluster runs for a period of time, the state of the node can be calculated according to formula (3.2). The state feature tensor of the cluster after node degradation can be expressed as:

[0111] S fad =[S 1 fad ,S 2 fad ,…,S h fad ,…,S H fad ] (3.6)

[0112] The vector elements are each cluster c h The degraded state characteristics can be expressed as:

[0113] (3.7)

[0114] The elements in the formula are nodes n (h,m,n) The degraded state.

[0115] In addition, the cost can be related to the variable pms (h,m,n) The cluster reliability after predictive maintenance is related to the cluster state feature tensor at this time. S mai Therefore, the predictive maintenance problem can be expressed as:

[0116] (3.8)

[0117] in C T is the expected total cost. R S is the reliability of the cluster, C thr is the cost threshold. The expected total cost of the cluster C T It can be calculated as follows:

[0118] (3.9)

[0119] in, C h Cluster c h Cost, C (h,m,n) For Node n (h,m,n) In addition, a cluster reliability evaluation model is established to calculate the reliability R S .

[0120] Constraints of the predictive maintenance decision problem C T With decision variables s (h,m,n) It is a linear relationship and easy to calculate. However, cluster reliability R S and the cluster state feature tensor S mai The relationship between the decision variables is complex and nonlinear. s (h,m,n) Depends on the node n (h,m,n) For the degenerate state feature tensor S fad , if in the decision-making process each group of maintenance strategies PMS The decision variables in s (h,m,n)Total maintenance cost C T and cluster reliability R S If the verification is not done, the computational efficiency will be very low and the computational load will be too large.

[0121] 1.3.1 DQN Algorithm Principle

[0122] In the field of artificial intelligence, with the introduction of DQN-based reinforcement learning methods, it has become possible to solve decision-making problems in high-dimensional state space and action space. The key to the DQN method is to use a deep neural network DQN to estimate the value function, and then perform policy iteration (PI) based on the idea of ​​Q-learning until a DQN agent with efficient and stable decision-making capabilities is trained. The DQN agent can generate highly accurate, efficient, and low-cost decision-making solutions for situation-based maintenance decision-making problems in high-dimensional state space and action space. The basic algorithm principle of DQN is to use a deep neural network as DQN based on the idea of ​​Q-learning to approximate the optimal value function Q. The optimal value function Q can be expressed as:

[0123] (3.10)

[0124] R t represents the reward at time t, γ represents the discount factor, π represents the behavior strategy, s represents the state, a represents the action, S t represents the observation result of the state at time t, A t represents the action performed at time t;

[0125] In addition, during the policy iteration process, the updated loss function can be expressed as:

[0126] (3.11)

[0127] where γ represents the discount factor; θ i represents the network parameters of DQN in the i-th PI process; θ i - represents the network parameters of the target DQN in the i-th PI process; the network parameters of the target DQN are set to the network parameters of the i-1th PI process, θ i - =θ i-1 ; Q(s,a,θ i ) represents the approximate value function based on deep neural network, θ i represents the parameters of the neural network; (s,a,r,s ’)~U(D) represents the experience data of the DQN agent, s represents the current state, a represents the current action, s ’ Indicates the state at the next moment, a ’ Represents the action at the next moment. For the predictive maintenance problem of cluster systems, RL uses information about the node state to select the optimal action in the state-action space to maximize the expected reliability under limited resources. The state-action space of the predictive maintenance problem represents the mapping relationship between the state characteristics of the cluster system and the maintenance action. Agent-environment interaction is a mathematically idealized form used to describe the reinforcement learning framework for learning from interaction to achieve the best maintenance strategy. The project management decision maker or learner is called the project management agent. During the reinforcement learning process, the predictive maintenance agent continuously interacts with the simulation environment, which includes all the contents simulated outside the predictive maintenance agent. The predictive maintenance agent selects maintenance behaviors, and then the simulation environment responds to these behaviors, presents new cluster system situations to the predictive maintenance agent, and evaluates the rewards that the predictive maintenance agent seeks to maximize through its behavior selection. After the learning process, the predictive maintenance agent working process takes the actual state of the cluster system as input and outputs the optimal predictive maintenance strategy. Taking the above factors into consideration, the present invention proposes a project management framework based on reinforcement learning, which includes a realistic working process and a simulated learning process. More specifically, the predictive maintenance agent interacts with the simulation environment at each step in a series of discrete time steps (t=1,2,…,T). At each time step t, the predictive maintenance agent receives the cluster system S from the simulation environment. t , then according to S t Choose an action. After executing one time step, the agent receives a numerical reward r for its behavior (t+1) , and in the new S t Keep searching.

[0128] 1.3.2 Predictive maintenance decision-making based on DQN

[0129] The predictive maintenance decision-making problem of cluster systems is of great significance for the operation and maintenance of cluster systems under the condition of system performance degradation during long-term operation. The difficulty of the problem lies in the imbalance of maintenance benefits of heterogeneous systems and components, the selectivity of the degree of system and component repair, the high impact of system and component repair timing on cluster performance recovery, and the overall NP-hard characteristics of the problem.

[0130] In view of the research content of predictive maintenance considering system degradation and unbalanced benefits, this paper regards it as the problem of determining the probability of maintenance states of multiple elements in the maintenance matrix and intends to solve it through a maintenance decision method based on DQN. By training a CNN neural network model fθ , approximate the optimal value function Q, provide an estimate of the value function Q, and evaluate the current cluster system maintenance state feature X; then based on DQN's estimate of the value function Q, a maintenance state probability π is obtained through the ε-greedy algorithm, and then the optimal maintenance action a of the current cluster maintenance state feature is generated * ; After executing a series of optimal maintenance actions, the cluster system performance recovers to a set threshold, and a predictive maintenance decision process (PMDP) is completed. A series of optimal maintenance actions constitute a complete maintenance strategy. After completing a PMDP, a set of maintenance decision experience data is obtained. Multiple sets of maintenance decision data constitute the current experience data set. CNN is trained based on this experience data set to provide a better value function estimate for the next PMDP.

[0131] In the above DQN-based predictive maintenance decision method, ε-greedy can be regarded as an efficient policy improvement operation; PMDP obtains the reward value r by executing the optimal maintenance action output by ε-greedy each time and evaluating the final maintenance status of the cluster system. This process can be regarded as an efficient policy evaluation operation. The reinforcement learning idea in the predictive maintenance decision method is mainly reflected in the policy iteration process of cyclically executing the above two operations: the CNN network parameters θ provide a more accurate estimate of the value function Q through learning and training; the new network parameters provide a better basis for searching the optimal maintenance action for the next policy iteration process.

[0132] The DQN-based predictive maintenance decision-making method of the present invention includes the following two core ideas: experience replay technology of the predictive maintenance decision-making process, and predictive maintenance strategy iteration based on DQN.

[0133] 1.3.2.1 Experience replay technology for predictive maintenance decision-making process

[0134] First, we need to design a DQN optimal value function Q for function approximation, randomly initialize the network parameters θ0, and define its input and output information.

[0135] The input of DQN is a cluster system state feature matrix s t =[S t ,S1]: At time t, the H×M×N-order state matrix composed of all clusters and their nodes in the cluster system is recorded as:

[0136] S t =[S 1 t ,S 2 t,…,S h t ,…,S H t ] (3.12)

[0137] Where S t h represents the cluster c at time t h The state characteristics can be expressed as:

[0138] (3.13)

[0139] The element in the formula is the node n at time t (h,m,n) status.

[0140] The output of DQN is the optimal estimate of the value function Q*(s,a). In the PMDP process, the repair state probability π and the final reward value r are obtained through the ε-greedy algorithm, and then the experience data set e is generated. t =(s t ,a t ,r t ,s t+1 ), based on the experience replay technology, the neural network parameters θ are trained using the gradient descent method, and the neural network f θ In a PMDP sequential decision-making process, a pipeline model (computer terminology: a linear communication model for exchanging data) is generated to generate experience data sets.

[0141] 1.3.2.2 Predictive maintenance strategy iteration based on DQN

[0142] Based on the idea of ​​Q-learning policy iteration, the above DQN is used to estimate the optimal value function Q*(s,a) to implement policy iteration, thereby designing a cluster system predictive maintenance Agent-Environment architecture based on the DQN method. The core idea of ​​this architecture is to complete policy iteration through the interaction between the predictive maintenance agent and the simulation environment. Among them, the simulation environment can describe the degradation state of the cluster system during operation and evaluate its reliability, while the predictive maintenance agent is a decision maker that generates predictive maintenance plans. It can generate a series of maintenance actions by executing a PMDP process. When executing the PMDP process, the predictive maintenance agent uses the ε-greedy strategy to select each maintenance action based on the optimal estimate Q*(s,a) of the value function by DQN.

[0143] The Agent-Environment architecture for predictive maintenance of cluster systems based on the DQN method, the main idea of ​​its reinforcement learning is: iteratively execute the PMDP process during the policy iteration process. Each policy iteration process is an episode, which includes four steps: First, the simulation environment generates the initial degradation state of the cluster system. Then, the predictive maintenance agent executes the PMDP process to generate a series of maintenance actions. Each maintenance action is based on the DQN estimation of the optimal value function Q*(s,a), and then selected using the ε-greedy strategy. Next, the simulation environment uses the reliability evaluation method to evaluate and reward the generated maintenance strategy, and stores the training samples e t =(s t ,a t ,r t ,s t+1 ) to the cache, and the maintenance strategy consists of a series of maintenance actions generated by the above PMDP process. Finally, DQN retrieves Mini-batch from the training sample data set and trains its network parameters based on the gradient descent method. The loss function is shown in (3.11). The new DQN will be used to estimate the optimal value function in the next policy iteration process.

[0144] In summary, the learning process is the interaction between the predictive maintenance agent and the simulation environment. Each iteration of the predictive maintenance agent learning process includes four modules: input feature generation, PMDP, reliability evaluation and reward, and DQN learning training. First, generate the initial state features and LRU set. Second, execute PMDP to generate maintenance strategies. Third, score the strategies and adjust the ResNet parameters θ through batch training. i For the next iteration i+1, a new initial state feature and LRU-set are generated, and then PMDP uses the parameter θ i For the first iteration, PMDP uses a randomly initialized ResNet with parameters θ0.

[0145] (1) Input feature generation

[0146] Before executing PMDP, the initial state feature tensor S1 is generated. The degraded health state feature tensor S is calculated according to (3.7) fad , then S1 can be expressed as follows:

[0147] S1=S fad (3.14)

[0148] According to (3.9) the total cost C T , the maintenance spare parts LRU set can be generated as follows:

[0149] SLRU =[S 1 LRU ,S 2 LRU ,...,S t LRU ,...,S T LRU ] (3.15)

[0150] Where S t LRU is the state characteristic of LRUt.

[0151] At time t during PMDP, LRUt is applied to repair a degraded node. Each state feature tensor S t Reduce the dimension from H*M*N to H*MN; then, the input feature X can be generated t , which contains the current state feature S t and the initial state feature S1. At each time t, it is necessary to retain the initial state feature. On the one hand, by comparing the current state feature with the initial state feature, the degree of recovery of the cluster performance can be estimated. On the other hand, the difference between the current state feature tensor and the initial state feature tensor also covers the maintenance state pms of each node in the cluster as of time t. (h,m,n) Therefore, the current input feature X t is a 2*H*MN tensor, as follows:

[0152] X t =[S t ,S1](3.16)

[0153] (2) Maintenance decision-making process

[0154] Initial input feature X1 and LRU set S LRU is the input of PMDP and the complete strategy is the output. The main steps of PMDP are as follows:

[0155] 1) Action value function based on DQN

[0156] At time t during the MDP, DQN takes X t As its input, and output maintenance action value v t . Value v t Is a normalization parameter, which estimates the current input feature X t The DQN architecture consists of convolutional layers with ReLU functions and fully connected layers. The DQN model can be expressed as:

[0157] (3.17)

[0158] Where DQN parameters θ(i-1) The adjustment was made during the previous iteration i-1. The initial network parameters θ0 are randomly initialized.

[0159] 2) Searching for maintenance actions based on the ε-greedy algorithm

[0160] Based on the ε-greedy algorithm, the repair action value estimated by DQN is used to search for the repair action a t (h,m，n) , indicating that the predictive maintenance task is selected at (h,m,n), that is, using LRU to replace node n (h,m,n) .

[0161] 3) Update input features

[0162] In performing higher quality actions a t * =a t (h*,m*,n*) After that, the old node is replaced by LRU. After that, the state features of time step t+1 can be generated as follows:

[0163] (3.18)

[0164] Input feature X t+1 It can be described as follows:

[0165] X t+1 =[S t+1 ,S1](3.19)

[0166] 4) Check loop conditions

[0167] If the LRU set is not empty, the process returns to the first step of PMDP. The MDP is completed at time step T.

[0168] 5) Calculate the recovery level of the cluster system

[0169] According to formula (3.11), the reliability R can be calculated S (G f ≥W,S T ) to evaluate the recovery ability after PMDP.

[0170] 6) Generate a complete maintenance strategy

[0171] By using a set of higher quality operations from the entire PMDP * ,a2 * ,...,a T *} to generate the complete policy.

[0172] (3) Rewards

[0173] After completion, PMDP is evaluated and the final reward is given. The reward for each time step can be described as:

[0174] r t =ΔR S (G f ≥W,t)

[0175] =R S (G f ≥W,S t )-R S (G f ≥W,S t-1 )(3.20)

[0176] Where R S (G f ≥W,S T ) is the cluster reliability at time t.

[0177] (4) Learning and training

[0178] Based on the PMDP process and reliability evaluation and reward, the training samples are stored. t =(s t ,a t ,r t ,s t+1 ) to the training dataset D. When DQN is learning and training, it first randomly and uniformly samples a Mini-batch from the training sample dataset D: {(s,a,r,s , )}, where s , Indicates the next state of s. The experience replay technique is applied to train the DQN network parameters based on the gradient descent method. In the i-th PI process, the training goal of DQN is to adjust the network parameters θ i To minimize the mean-squared error (MSE) in the Bellman equation, the optimal target value will be replaced by the estimate of the target value, which is the network parameter θ in the previous PI process. i - DQN generation. Therefore, the loss function of the i-th PI process can be expressed as:

[0179] (3.21)

[0180] where γ represents the discount factor; θ i represents the network parameters of DQN in the i-th PI process; θ i - represents the network parameters of the target DQN in the i-th PI process; the network parameters of the target DQN are set to the network parameters of the i-1th PI process, θi - =θ i-1 ; Q(s,a,θ i ) represents the approximate value function based on deep neural network, θ i represents the parameters of the neural network; (s,a,r,s ’ )~U(D) represents the experience data of the DQN agent, s represents the current state, a represents the current action, s ’ Indicates the state at the next moment, a ’ represents the action at the next moment. After the learning and training is completed, the new DQN network parameters will be used to estimate the optimal value function in the next PI process. Generally, when the average action value and the MSE of DQN tend to be stable, the predictive maintenance agent can effectively support the generation of predictive maintenance strategies for UAV clusters under various degradation states and improve the state of the UAV cluster. In addition, the DQN that has completed the learning and training does not need to be trained again when planning the predictive maintenance management work during the operation of the UAV cluster. The training of DQN in each iteration of the learning process is based on Mini-batch sampling and uses the empirical data of PMDP. After training, DQN can improve the strength of PMDP in the next iteration i+1. When the MSE tends to be stable, the DQN is well trained, and the predictive maintenance agent can effectively support the maintenance decision-making of large-scale predictive maintenance problems and improve the state of the cluster system. In addition, DQN does not need to be trained again during the predictive maintenance management work.

[0181] After the predictive maintenance agent completes learning and training, it takes the input feature X of the real-world cluster system and the SLRU of the LRU set as input. Then, the predictive maintenance agent uses the LRU set SLRU to perform a series of maintenance actions. For each time step, the trained DQN uses the input feature X to predict the prior information, and then ε-greedy selects a higher reward action to repair a degraded node based on the prior information. Finally, a series of high-reward maintenance actions will produce a complete predictive maintenance strategy.

[0182] 1.4 Based on Distributed Proximal Policy Optimization (DPPO) Algorithm

[0183] Based on the Distributed Proximal Policy Optimization (DPPO) algorithm, please refer to the Chinese invention patent application (application number: 2024114796053, application date: 20241023) for details.

[0184] First, we analyze the tasks, destruction and reconstruction of the three-level spatiotemporal dynamic cluster system of cluster-cluster-node, and consider the multi-objective tasks including M There are different task areas, and the task area space is described as follows:

[0185] Ω={A1, A2,…,A m ,…,A M} (4.1)

[0186] Mission area A m ,m∈{1,2,...,M} is an irregular polygonal area.

[0187] For this multi-target mission, a team consisting of M If a cluster S is composed of clusters, then the cluster set C of the cluster S can be expressed as:

[0188] C={ c1, c2,…,c m ,…,c M} (4.2)

[0189] where c m Indicates dispatch to mission area A m Consider cluster c m By N m nodes, then cluster c m It can be expressed as:

[0190] C m ={n (m,1) ,n (m,2) ,…, n (m,n) ,…, n (m,Nm)} (4.3)

[0191] where n∈{1,2,…,N m} represents node n (m,n) In cluster c m The dynamic number in N m Cluster c m The total number of nodes, depending on the size of the task area A m Sure.

[0192] During the execution of the multi-objective task, the cluster system may be partially damaged, resulting in the destruction of some nodes in one or more clusters. Taking the impact of this damage as input, consider that each node has two states: normal state and fault state. When a node is damaged, it enters the fault state. The scope of the local damage is represented by a point (i d ,j d ) as the center, with r d is a circle with a radius of d ,j d ) and r d The value is randomly generated.

[0193] Under the situation of local damage, the dynamic reconstruction of the cluster system is considered to be an autonomous behavior that can adapt to environmental changes to continue to perform the current task. When the cluster is affected by dynamic changes during task execution, it is necessary to generate a corresponding dynamic reconstruction strategy to achieve cluster system task performance recovery and topology redeployment, thereby ensuring the continuity of task execution. When local damage occurs, the dynamic reconstruction algorithm observes the damage situation of the cluster system and generates a reconstruction strategy to communicate to each node. The cluster system then performs dynamic reconstruction between nodes within the cluster and between clusters within the cluster.

[0194] For the time τ of the dynamic reconstruction process of the cluster system, define the cluster balance degree ε at time τ b (τ) represents the mean square error of the redundancy values ​​of all nodes that can work normally in each cluster during the task execution phase, as shown below:

[0195] (4.4)

[0196] Where M represents the total number of clusters in cluster S, N m Represents clusters c m The total number of nodes, ε mn (τ) represents the node n at time τ (m,n) The sum of the overlapping areas of the coverage area of ​​​​the node and the coverage areas of other nodes in the same cluster within the task area, Represents the average redundancy value of all nodes in each cluster at time τ.

[0197] In view of the above-mentioned task coordination and topological configuration characteristics, the local damage scenario during the cluster system task is considered, and the cluster system dynamic reconfiguration decision (DRD) problem is constructed to describe the damage center and radius, describe the cluster failure scale, determine the threshold of reconfigurable nodes in each cluster, determine the maximum moving distance of reconfigurable nodes, stipulate the minimum coverage of the task area, describe the cluster balance, and determine the reconstruction time threshold. Based on this, the optimization objectives, constraints and control variables of the DRD problem are defined as follows:

[0198] (4.5)

[0199] where ε m Represents cluster c m The degree of balance, ε m min Represents cluster c m The balance threshold, N m move Represents cluster c mThe total number of reconfigurable nodes in m normal Represents cluster c m The total number of nodes in the normal state, d represents the distance between two nodes in the normal state, d min It represents the minimum allowed distance, that is, the safe distance between nodes, τ tot represents the total time of the reconstruction process, τ thr Indicates the reconstruction time threshold.

[0200] 1.4.1 Cross-level dynamic reconstruction algorithm model based on DPPO

[0201] In order to solve the problem of dynamic reconstruction strategy generation of cluster system, during the execution of tasks by the cluster system, dynamic reconstruction within the cluster and between clusters is realized in the face of local destruction scenarios, and the cluster performance is restored to a certain extent to continue to complete the given tasks. The present invention proposes a dynamic reconstruction method based on DPPO. DPPO not only selects reconstruction actions for a single node and evaluates its action value, but also comprehensively considers the three levels of cluster-cluster-node that an effective reconstruction strategy needs to include, analyzes the overall task goals that each node has with other nodes in the cluster, and needs to build a two-level dynamic reconstruction architecture for the cluster system. The problems analyzed by traditional reinforcement learning methods are often limited to the scale of action space and sample space. Under normal circumstances, tasks that are closer to the actual situation are often more complex and need to consider large-scale and continuous state space and action space. When the input data is an image or sound, it often has a very high dimension, which is difficult to handle by traditional reinforcement learning. Deep reinforcement learning combines the high-dimensional input of deep learning with reinforcement learning. The proposed DPPO-based method includes three parts: a cluster system dynamic reconstruction decision architecture, a DPPO reinforcement learning algorithm model, and a DPPO alliance learning model. First, it is necessary to build a two-level dynamic reconstruction decision-making architecture for the cluster system, comprehensively analyze the cluster system information and the high-dimensional situation information of the complex environment, and apply the Reinforce mechanism to estimate the reconstruction action value under the two-level reconstruction to support cross-level dynamic reconstruction strategy optimization. Based on the above two-level dynamic reconstruction, the reinforcement learning algorithm model and alliance learning model of DPPO are further studied. The generation network of the deviation of the reinforcement learning agent training does not have an absolute value activation function, and the final deviation is generated by a two-layer super network with ReLU nonlinearity. The final loss function is:

[0202] L(θ)=∑ b i=1 [y i tot -Q tot (τ,u,s;θ) 2 ] (4.6)

[0203] Where b represents the number of samples sampled from the experience pool, ,θ - is the parameter of the target network, so the TD error can be expressed as:

[0204] TD error =(r+γQ tot (target))-Q tot (evaluate)(4.7)

[0205] Where Q tot (target) indicates the state s ' In the case of all behaviors, the maximum value Q is obtained tot According to the IGM condition, the input is the maximum action value of each agent in this state; Q tot (evaluate): In state s, the Q that can be obtained according to the current network strategy tot ; r represents the reward value under the current strategy, which refers to the degree of balance in the present invention.

[0206] 1.4.2 Dynamic Reconfiguration Decision Algorithm Considering Task Coordination and Topological Configuration

[0207] Considering the distributed functional coordination of multiple clusters in the cluster, which leads to cross-level interaction in the reconstruction space, a two-level dynamic reconstruction within the cluster and between clusters is comprehensively analyzed, and the dynamic reconstruction decision method based on DPPO is studied. First, a cross-level dynamic reconstruction decision architecture of the cluster system is constructed, and then reinforcement learning and federation learning models are designed based on DPPO to realize the reconstruction strategy optimization.

[0208] 1.4.3 Cluster system cross-level dynamic reconstruction decision architecture

[0209] Construct a cross-level dynamic reconstruction decision-making framework for cluster systems, analyze the multi-dimensional situation information of the three-level spatiotemporal dynamic cluster system, and then design a deep neural network model to extract multi-dimensional situation features and evaluate the value of reconstruction actions. On the basis of completing the cross-level dynamic reconstruction decision object, further study the reinforcement learning model based on DPPO and the alliance learning model to train the neural network. In the proposed cross-level dynamic reconstruction decision-making framework, by designing a dynamic reconstruction agent to use DNN to extract the multi-dimensional situation features of the cluster system, focus on the cross-level interactions within the cluster and between clusters, and then select reconstruction actions for each node in the cluster. The selection of reconstruction actions can be expressed as:

[0210] (4.8)

[0211] Where S t Represents the multidimensional situation tensor at time t in the dynamic reconstruction process, M t-1Represents a cluster mobility feature set, which consists of the historical mobility features of the cluster {mov t-1 ,mov t-2 ,mov t-3}.

[0212] The historical mobility characteristics of the cluster are necessary because the mobility trends of all nodes in the cluster cannot be fully observed only from the current mobility status. The mobility characteristics of the nodes selected by the cluster at time t can be described as mov t |[S t ,M t-1 ]=[loc t init ,loc t final ], where loc t init and v represent the node position feature matrix, which is described by the I×J map matrix. Each element in the node position feature matrix is ​​associated with a coordinate. If the element is 1, the associated coordinate is the selected target position. t init and loc t final There is only one element in it that is equal to 1, and the other elements are equal to 0. The node position feature matrix loc t init represents the initial position of the reconstruction action at time t, and the node position feature matrix loc t final Indicates the end position of the same reconstruction action. In addition, if t<1, then loc t init and loc t final are all zero matrices. From the nature of the defined reconstruction action, this problem is integer programming with binary variables. For each node, the node position feature matrix loc t final Any element in can be used as its current strategy, then the number of its strategies is 2 I×J ; For the entire cluster, the total number of strategies is , so the dynamic decision-making problem studied is an NP-hard problem. The entire dynamic reconstruction process includes task dispatch and local destruction generation, reconstruction action generation, and cluster balance recovery evaluation. Based on the above decision-making architecture, the DPPO reinforcement learning algorithm is further designed to support cross-level dynamic reconstruction strategy optimization.

[0213] (1) Multi-dimensional situation feature extraction

[0214] In the constructed cross-level dynamic reconstruction decision-making architecture of the cluster system, we first need to pay attention to the cluster system information and the multi-dimensional situation information of the complex environment, including environmental feature information such as destruction characteristics and task characteristics, as well as cluster feature information such as cluster node location characteristics and business coverage characteristics, and further convert the above cluster system feature information and complex environment feature information into mathematical language. The present invention converts these feature information into the related I×J map matrix to mathematically describe it. After determining the relative coordinate system according to the geographical location of the specific task area, the I×J map matrix is ​​determined by the selected relative coordinate system. For example, it can be defined as a 256×256 image matrix to describe the current map information. The matrix processes a 256×256 relative coordinate system into a 256×256 image grid, and each element in the matrix corresponds to a specific small square grid in the relative coordinate system. Therefore, the task area feature matrix S at time t of the dynamic reconstruction process t mission It can be expressed as:

[0215] (4.9)

[0216] The matrix element s m ij (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s m ij (t) = 1 means covered, s m ij (t) = 0 means no coverage. The service coverage characteristic matrix S at time t in the dynamic reconstruction process t cover It can be expressed as:

[0217] (4.10)

[0218] The matrix element s c ij (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the business area of ​​the cluster node, s c ij (t) = 1 means covered, s c ij (t) = 0 means not covered. The node position feature matrix S at time t in the dynamic reconstruction process t location It can be expressed as:

[0219] (4.11)

[0220] The matrix element s l ij(t) indicates whether there is any node n in the small square grid represented by (i, j) in the relative coordinate system at time t (m,n) ,s l ij (t) = 1 means existence, s l ij (t) = 0 means it does not exist; the characteristic matrix S of the damaged area at time t in the dynamic reconstruction process t damage It can be expressed as:

[0221] (4.12)

[0222] The matrix element s d ij (t) indicates whether the small square grid represented by (i, j) in the relative coordinate system at time t is covered by the task area, s d ij (t) = 1 means covered, s d ij (t) = 0 means not covered.

[0223] Based on the above description, the multidimensional situation tensor S at time t in the dynamic reconstruction process in formula (4.8) t It can be expressed as:

[0224] S t =[S t mission ,S t damage ,S t location ,S t cover ](4.13).

[0225] (2) Reconstruction strategy generation

[0226] At time t in the dynamic reconstruction process, the dynamic reconstruction agent moves the feature set M in the cluster system. t-1 ={mov t-1 ,mov t-2 ,mov t-3} and the multidimensional situation tensor S t As input, and output the reconstruction action act at the current moment t|mn During the entire dynamic reconstruction process, a series of dynamic reconstruction actions are generated by the dynamic reconstruction agent. All nodes in the cluster system execute the dispatched reconstruction actions in sequence or simultaneously to complete the redeployment of the cluster topology configuration, so as to improve the cluster balance and restore it to a performance level that can continue to execute the current task. The above dynamic reconstruction action set can be expressed as:

[0227] Φ = {act t|mn}|t=1toT,m∈{1,2,…,M},n∈{1,2,…,N m} (4.14)

[0228] Among them act t|mn Indicates cluster node n (m,n) The reconstruction action performed at time t in the dynamic reconstruction process, and the reconstruction action can be defined as act t|mn =[s l i"j" (t-1),s l ij (t)], which means cluster node n (m,n) In the dynamic reconstruction process, at time t, the position s in the relative coordinate system at the previous moment is l i"j" (t-1) Move to the current position s l ij (t), the defined reconstruction action act t|mn According to the cluster node n described in formula (4.3) (m,n) The moving feature mov t |[S t ,M t ] is transformed; further cluster c m The dynamic reconstruction action set is expressed as:

[0229] Φ = {act t|mn}|n=1toN m ,m∈{1,2,…,M},t∈{1,2,…,T} (4.15).

[0230] Based on the above description of the input and output of the dynamic reconstruction agent, a DNN model f is designed for the dynamic reconstruction agent. θ , to generate reconstruction actions, apply Res Net to extract features from cluster system information and environmental information, apply LSTM to process the temporal features of high-dimensional information, apply Attention mechanism to focus on task area and node position, apply Reinforce with Baseline principle to estimate the action value of the entire reconstruction process, and use value network as its baseline function only when training neural network. The designed DNN model for dynamic reconstruction of cluster system is a multi-input and multi-output neural network. As the reconstruction strategy generation model, the network parameters θ0 are randomly initialized. The DNN in the dynamic reconstruction agent uses a ResNet module composed of 5 convolutional layers with ReLU function to process the multi-dimensional situation input feature matrix, including the task area feature matrix S tmission , business coverage feature matrix S t cover , node position feature matrix S t location , damage area characteristic matrix S t damage , cluster system mobile feature set M t-1 ={mov t-1 ,mov t-2 ,mov t-3}. After the destruction information and task information are decoded by their respective ResNet modules, they are input into the scene kernel composed of residual MLP modules for processing. The business coverage information, historical reconstruction action information and cluster node location information are decoded by their respective ResNet modules and input into the cluster kernel composed of LSTM modules for processing. The output of the scene kernel and the cluster kernel is used as part of the value network input. The other part of the value network input is the Baseline feature of the entire dynamic reconstruction process. This feature tensor is composed of the reconstruction action set of the entire dynamic reconstruction process. The value network evaluates the cluster balance loss of the entire dynamic reconstruction process and is only used when training the parameters of the entire DNN neural network. In the dynamic reconstruction process, after the reconstruction Agent uses the DNN model to input the above-mentioned multi-dimensional situation features, it first outputs an I×J probability matrix through an Attention module to select S t location As the object for performing the reconstruction action at this moment, the Attention module inputs the processing information of the scene kernel and the cluster kernel, and inputs the reconstruction action features and cluster node position features through the skip connection of the residual mechanism. The ResNet module that previously processed the task information generates a task information input feature through the residual connection, and embeds the I×J probability matrix output by the above Attention module into it, and then inputs it to a new Attention module for processing, and then outputs an I×J probability matrix to select S t mission A task area in is taken as the target task area. The ResNet module that previously processed the cluster node position information generates a pole position information input feature through a residual connection, and embeds the feature information describing the target task area through an MLP module, inputs it to a Deconv residual module, and then outputs an I×J probability matrix to select the target position of the reconstruction action.

[0231] 1.4.4 Dynamic Reconstruction Strategy Optimization Based on DPPO

[0232] Based on the cross-level dynamic reconstruction decision-making architecture of the cluster system constructed in the previous section, a reinforcement learning algorithm based on DPPO and a federated learning algorithm model are designed to train the DNN model in the dynamic reconstruction Agent to achieve dynamic reconstruction strategy optimization.

[0233] (1) Reinforcement learning algorithm based on DPPO

[0234] Based on the cross-level dynamic reconstruction decision architecture in the previous section, after completing each dynamic reconstruction process through simulation, a set of multi-dimensional situation data and corresponding reconstruction strategy data can be generated. Then, it is necessary to evaluate the dynamic reconstruction process, calculate the cluster balance after the reconstruction, and the loss of cluster balance during the entire reconstruction process, so as to comprehensively evaluate the entire dynamic reconstruction process and design a reward function on this basis. For each dynamic reconstruction action act t|mn Once the reconstruction action is completed, it is considered that the moving node starts to perform the current task at the new location, and then the service coverage feature matrix S can be updated. t cover According to formula (4.4), the current cluster balance can be calculated. The goal of the cluster system is to improve its cluster balance as efficiently as possible. Therefore, the reward function should be defined for cluster balance and reconstruction time as follows:

[0235] (4.16)

[0236] Where R t represents the reward value at time t in the dynamic reconstruction process, τ t+ζ represents the cumulative reconstruction time at the time (t+ζ) of the dynamic reconstruction process, τ t+ζ-1 represents the cumulative reconstruction time of the dynamic reconstruction process (t+ζ-1), E_b^thr represents the cluster system balance threshold, δ represents the discount factor of the reward function, τ T Represents the cumulative reconstruction time (Time to finish reconfiguration, TTFR) to complete the entire reconstruction process. Based on the reward function and the data set of the input and output feature information of the dynamic reconstruction agent during the reconstruction process, a reinforcement learning algorithm is designed, involving technologies such as policy gradient, experience replay, TD(𝝀), V-trace and UPGO.

[0237] Reward the dynamic reconstruction strategy and train the parameters θ of the DNN model through the sample data set i For the next policy iteration process i+1, a new initial multi-dimensional situation feature information is generated, and then the dynamic reconstruction process uses θ iThe DNN model as a parameter generates a new reconstruction strategy. For the first policy iteration process, the dynamic reconstruction process uses a DNN model with randomly initialized parameters θ0. This model is designed to simultaneously approximate the value function of the reconstruction strategy and multiple policy functions that describe the reconstruction actions. The DNN is trained during the policy iteration process of dynamic reconstruction. When performing the dynamic reconstruction process, the reconstruction agent selects a series of reconstruction actions to generate a complete strategy. The reconstruction agent needs to output three probability matrices for each reconstruction action selected, and at the same time use the reward R of the reconstruction action as the reward. t As a value label. The main idea of ​​the reinforcement learning algorithm is to iteratively perform the dynamic reconstruction process during the policy iteration process: training the DNN parameters so that the reconstruction probability and value predicted by the neural network are closer to the reconstruction policy and reward. By maximizing the similarity between the probability of the reconstructed action predicted by the DNN and the reconstruction policy, the policy gradient can be expressed as:

[0238] (4.17)

[0239] (4.18)

[0240] Where r(s t ,a t ) represents the reward value of the reconstruction action at time t, s t represents the state at time t, a t represents the reconstruction action at time t, V(s t ) represents the average value, A(s t ,a t ) represents the action advantage value. Based on the experience replay technology, asynchronous update is performed by optimizing the strategy π θ The expected reward J(π θ ) is maximized, which can be expressed as:

[0241] (4.19)

[0242] In addition, the DNN is adjusted to minimize the gap between the predicted value and the reward value of the reconstruction strategy. TD(λ), V-trace and UPGO training are applied to update the neural network to push the strategy to a trajectory that is better than the average return. When the dynamic reconstruction process is performed in the next policy iteration, the dynamic reconstruction agent uses the new DNN parameters to better search for the optimal reconstruction action.

[0243] (2) Alliance learning model based on DPPO

[0244] In order to realize the distributed learning and training of DPPO, a coalition learning model is designed, coalition games and a virtual self-learning mechanism with priority are defined, and three types of agents are used for learning and training to achieve dynamic reconstruction strategy optimization.

[0245] Game definition: randomly generate n match multi-objective tasks and destruction. In a single game, two agents generate reconstruction strategies for a task. The one with the smaller loss of cluster balancing ability wins. Calculate n match Game winning rate.

[0246] Priority-based virtual self-learning (PFSP): Virtual self-learning (SP) archives the alliance at regular intervals and randomly and evenly selects opponents from the alliance to play against the agent being trained; priority refers to giving priority to opponents with a higher winning rate in the alliance archive.

[0247] The three types of agents include the main agent, the main explorer, and the alliance explorer. The main agent: selects opponents in the alliance archive based on PFSP to play against, and archives regularly; the main explorer: plays against the main agent of the current iteration, and reinitializes regularly; the alliance explorer: defeats all opponents in the alliance with a certain winning rate, and archives to the alliance, otherwise it reinitializes regularly.

Claims

1. A multi-dimensional situation awareness method for intelligent operation and maintenance of a cluster system, wherein the cluster system is a power system cluster, a drone cluster system, or a large-scale battery cluster system in an industrial energy storage system, characterized in that: The method comprises the following steps: a) Cluster system modeling: A multi-dimensional situation model of the cluster system is constructed through a multi-level modeling method of cluster-cluster-node, and the model includes a time dynamic cluster system and a space-time dynamic cluster system; b) Multi-dimensional situation information collection: The multi-dimensional situation information of the cluster system is collected in real time through sensors, including node health status, link status, task execution status and environmental change information; c) Situation information processing: classify, integrate and analyze the collected multi-dimensional situation information to generate a set of environmental information , used to describe the current status of the cluster system; d) Situation prediction: Situation prediction is performed based on reinforcement learning algorithms, including deep Q networks and distributed proximal strategy optimization algorithms, to predict future situation development trends; The multi-dimensional situation model is constructed by building a cluster system node set and clusters To achieve this, the node set is defined as: ； letter Indicates the first n nodes, clusters Defined as: ； in, c m Represents the mth cluster in the cluster at the current moment; The situation information fusion adopts a multi-dimensional information fusion method based on environmental objects to fuse node and link information with external environmental change information to generate situation information. G Figure, and update the environment information collection through the following formula : ； in, f represents the fusion function of the environment object, Indicates a set of links. A represents the adjacency matrix; Node-set based and clusters Definition of storage attribute information and three-level cluster model Based on the definition of cluster system, various information in the cluster system operation scenario is further deconstructed, and environmental information, cluster-cluster-node operation information, and degradation information are classified. Then, similar and highly correlated information is integrated, and cluster-cluster-node information is distinguished according to temporal dynamic characteristics and spatiotemporal dynamic characteristics. On this basis, the environmental object is defined. And build an environment information collection ; ； in Represents clusters The j nodes; Cluster system links The construction of the reference networkX underlying coding in the adjacency matrix data structure, that is, hash nested hash, and on this basis added operation and maintenance status attribute information, link set As shown below: （1.4）； in, s ( a ij ) represents a connection node n i and nodes n j The state of the edge, element l ij Representation Node n i and n j The geographical distance between them.

2. The multi-dimensional situation awareness method according to claim 1, characterized in that: Situation prediction is achieved through reinforcement learning algorithm. Based on the deep Q network (DQN) algorithm, feature extraction and strategy optimization are performed on multi-dimensional situation information, including the following update formula: ； r t represents the reward at time t, γ represents the discount factor, π represents the behavior strategy, s represents the state, a represents the action, S t represents the observation result of the state at time t, A t represents the action performed at time t; during the policy iteration process, the updated loss function is expressed as: ； where γ represents the discount factor; θ i Indicated in i The network parameters of the current DQN are the PI process; θ i - represents the network parameters of the target DQN in the i-th PI process; the network parameters of the target DQN are set to the network parameters of the i-1th PI process, θ i - =θ i-1 ; Q(s,a,θ i ) represents the approximate value function based on deep neural network; (s,a,r,s ’ ) ~ U(D) represents the experience data of the DQN agent, s represents the current state, a represents the current action, and s ’ Indicates the state at the next moment, a ’ Indicates the next action.

3. The multi-dimensional situation awareness method according to claim 2, characterized in that: The final bias based on the distributed proximal policy optimization algorithm is generated by a two-layer hypernetwork with ReLU nonlinearity; the final loss function is: ； Where b represents the number of samples sampled from the experience pool, ， θ - is the parameter of the target network, so the TD error can be expressed as: TD error =(r+γQ tot (target))-Q tot (evaluate); Where Q tot (target) indicates the state s ' In the case of all behaviors, the maximum value Q is obtained tot ; According to the IGM condition, the input is the maximum action value of each agent in this state; Q tot (evaluate): In state s, the Q that can be obtained according to the current network strategy tot ; r represents the reward value under the current strategy.

4. The multi-dimensional situation awareness method according to claim 1, characterized in that: Situational awareness includes the following scenarios: a) Predictive maintenance: Preventive maintenance can be performed in advance by predicting node degradation status and task execution status; b) Post-disaster repair: Based on the damaged nodes and links after the disaster, the optimal repair path is planned based on the situation information; c) Dynamic Reconfiguration: Dynamically adjust the topology of the cluster system according to task requirements to maintain task execution capabilities.

5. The multi-dimensional situation awareness method according to claim 1, characterized in that: Node sets and cluster sets are dynamically managed based on hash tables to support real-time adjustment and expansion of cluster systems.

6. A multi-dimensional situation awareness device for intelligent operation and maintenance of cluster systems, characterized in that: The device performs the method according to any one of claims 1 to 5, including: a) a situation information collection module, which is used to collect multi-dimensional situation information in the cluster system in real time through sensors, and the information includes node status, link status, task execution status and environmental change information; b) Data processing module, used to classify, integrate and analyze the collected multi-dimensional situation information, generate an environmental information set, and describe the current operation situation of the cluster system; c) Situation prediction module, which predicts future situation changes based on reinforcement learning algorithm, which includes deep Q network and distributed proximal strategy optimization algorithm to generate prediction results of future situation.

Images