Information system maintenance scheme generation method and system based on multi-agent cooperation

Abstract

The invention provides an information system maintenance scheme generation method and system based on multi-agent collaboration, and relates to the technical field of equipment state maintenance, and the method comprises the steps: collecting state data and historical maintenance records of each assembly in an information system; time sequence features are constructed according to the state data and historical maintenance records; an NARX-AD model is constructed; judging whether the information system has a fault or not through an NARX-AD model; predicting the maintenance cost of the information system, otherwise, continuing monitoring; predicting the maintenance cost of the information system through a pre-trained artificial neural network according to the fault type of the information system; taking the minimum maintenance cost as a target, referring to a multi-dimensional Markov decision process, and constructing a maintenance environment of the information system; based on the maintenance environment, a maintenance scheme of the information system is determined through a multi-agent deep reinforcement learning algorithm; and repairing the information system according to the maintenance scheme.

Description

Technical Field

[0001] This invention relates to the field of equipment condition-based maintenance technology, and in particular to a method and system for generating maintenance plans based on multi-agent collaboration information systems. Background Technology

[0002] Multi-agent systems refer to multiple autonomous decision-making "agents," which can be virtual software agents or physical devices. Each agent possesses perception, decision-making, and execution capabilities and can collaborate or compete within an environment. An information system is an integrated whole comprised of hardware (such as servers and storage), software (applications and databases), networks, and operational mechanisms, used to process, transmit, and store information. The information system maintenance plan generation method based on multi-agent collaboration leverages the cooperation between multiple agents, using intelligent algorithms and models to automatically generate a suitable maintenance plan when problems occur in the information system, thereby ensuring the system's normal operation more efficiently and at a lower cost.

[0003] In the modern information environment, information systems are becoming increasingly complex. By introducing a multi-agent collaborative mechanism, different parts of the information system can be regarded as being managed by different "agents". Each agent focuses on the collection and decision-making of local information, and then achieves global optimization through collaborative strategies. This provides an intelligent, automated and collaborative maintenance path for complex information systems, greatly improving the resilience and operation and maintenance efficiency of the system, and laying the foundation for future intelligent system management.

[0004] However, in current information system operation and maintenance practices, the generation of troubleshooting plans often relies on empirical rules and manual intervention, lacking dynamic adaptability. Once the system environment changes, the rules may become invalid. Secondly, many plans depend on human expert judgment, resulting in slow response times and susceptibility to subjective human factors. Furthermore, traditional methods typically focus on anomalies in individual components while neglecting the coupling relationships between components, making it difficult to effectively identify interconnected faults in complex systems. This leads to problems such as insufficient adaptability, lack of coordination, limited automation, and insufficient cost optimization, making it difficult to meet the high availability, high efficiency, and low cost operation and maintenance requirements of modern complex systems. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method for generating maintenance plans for information systems based on multi-agent collaboration. This method can solve the problems that traditional maintenance plan generation often relies on experience rules and manual intervention, lacks dynamic adaptability, and may become invalid once the system environment changes. This results in problems such as insufficient adaptability, lack of collaboration, limited automation level, and insufficient cost optimization, making it difficult to meet the technical problems of high availability, high efficiency, and low cost operation and maintenance requirements of modern complex systems.

[0006] A first aspect of this invention provides a method for generating maintenance plans for information systems based on multi-agent collaboration, comprising: S1: Collect status data and historical maintenance records of various components in the information system; S2: Construct time-series features based on status data and historical maintenance records; S3: Construct the NARX-AD model; S4: Based on the time-series characteristics, determine whether the information system has malfunctioned using the NARX-AD model; if yes, proceed to step S5; otherwise, continue monitoring. S5: Based on the fault type of the information system, determine the maintenance cost of the information system through a pre-trained artificial neural network; S6: With the goal of minimizing maintenance costs, construct the maintenance environment for the information system by referring to the multidimensional Markov decision process; S7: Based on the maintenance environment, determine the maintenance plan for the information system through a multi-agent deep reinforcement learning algorithm; S8: Repair the information system according to the maintenance plan.

[0007] A second aspect of this invention provides a system for generating maintenance solutions for information systems based on multi-agent collaboration, comprising: a processor and a memory; The memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, they implement the steps of the method for generating maintenance schemes for information systems based on multi-agent collaboration as described in the first aspect.

[0008] A third aspect of the present invention provides a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the method for generating an information system maintenance plan based on multi-agent collaboration as described in the first aspect are implemented.

[0009] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this embodiment of the invention, time-series features are constructed based on the collected status data and historical maintenance records of the information system. These features are then input into the NARX-AD model to determine whether the information system has experienced a fault. The NARX-AD model can not only predict system behavior but also adaptively adjust based on data characteristics, ensuring that the modeling results are highly consistent with the real system. Next, based on the fault type of the information system, the maintenance cost of the information system is predicted using a pre-trained artificial neural network. With the goal of minimizing maintenance costs, and referencing a multidimensional Markov decision process, the maintenance environment of the information system is constructed through the definitions of state space, action space, and transition function. Based on the maintenance environment, a maintenance plan for the information system is determined using a multi-agent deep reinforcement learning algorithm. By utilizing the parallel decision-making and collaborative capabilities of multiple agents, an optimal or near-optimal maintenance plan is generated. Compared to traditional single-agent methods, this approach is more adaptable to the distributed characteristics of complex systems. Finally, the information system is repaired according to the maintenance plan, improving the level of automation, reducing manual maintenance costs, increasing the accuracy and efficiency of maintenance, shortening repair time, further reducing downtime costs, and improving the availability and reliability of the system. Attached Figure Description

[0010] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0011] Figure 1 This is a flowchart illustrating a method for generating maintenance solutions for an information system based on multi-agent collaboration, as provided in an embodiment of the present invention.

[0012] Figure 2 This is a schematic diagram of the structure of an information system maintenance scheme generation system based on multi-agent collaboration provided in an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] The following description, in conjunction with the accompanying drawings, details the method for generating maintenance solutions for information systems based on multi-agent collaboration provided by the present invention through specific embodiments and application scenarios.

[0015] Reference manual attached Figure 1 The diagram shows a flowchart of a method for generating maintenance schemes for information systems based on multi-agent collaboration, provided by an embodiment of the present invention.

[0016] This invention provides a method for generating maintenance plans for information systems based on multi-agent collaboration, which may include the following steps: S1: Collect status data and historical maintenance records of various components in the information system.

[0017] Components refer to the hardware and software units that make up an information system, such as servers, databases, storage devices, network modules, and applications, and are the basic carriers for system operation. Status data describes dynamic indicators of component operation, including CPU utilization, memory usage, disk I / O speed, network traffic, and log files, reflecting the system's health and performance level. Historical maintenance records are complete records of past faults, maintenance measures, repair methods, and their effects, including information such as the cause of the fault, handling steps, time spent, and cost, providing experience for future maintenance.

[0018] It's important to note that the systematic collection of status data and historical maintenance records provides a solid data foundation for subsequent modeling, prediction, and optimization. First, status data reflects the system's operational status in real time, enabling timely detection of fault signs and improving detection sensitivity. Second, historical maintenance records provide the model with rich prior knowledge, allowing the system to not only identify known fault modes but also discover potential risks through mode comparison. Third, the combination of these two elements enables unified analysis of short-term anomalies and long-term trends, providing high-quality input for subsequent time-series feature extraction and predictive model construction. This process ensures data integrity and traceability, avoiding misjudgments caused by incomplete information.

[0019] In one possible implementation, the status data includes CPU utilization, memory usage, and log data.

[0020] S2: Construct time-series features based on status data and historical maintenance records.

[0021] Among them, time-series characteristics refer to the data characteristics that arise over time, such as trends, periodicity, and volatility, which are used to reveal the dynamic changes in the system state.

[0022] It's important to note that transforming raw status data and historical maintenance records into quantifiable and analyzable time-series features allows for a better capture of the dynamic patterns in the information system's operational status. This process reveals potential trend changes, periodic fluctuations, and short-term anomalies, making subsequent models more robust and interpretable when processing input data. Simultaneously, the extraction of time-series features eliminates redundant information, highlights key variables, and improves modeling efficiency and accuracy. Furthermore, features generated from historical maintenance records help the model establish a mapping between data and failure modes, enhancing its ability to predict future anomalies.

[0023] S3: Construct the NARX-AD model.

[0024] Among them, the adaptive nonlinear autoregressive model (NARX-AD) is an improved time series forecasting model. It is based on the nonlinear autoregressive structure (NARX) and incorporates an adaptive mechanism (AD). It can dynamically adjust parameters according to external inputs and the system's own historical state to improve forecast accuracy.

[0025] It should be noted that by constructing an adaptive nonlinear autoregressive model (NARX-AD), the accuracy and flexibility of information system state prediction and anomaly detection are effectively improved. Compared with traditional linear models, NARX-AD can better capture the nonlinear dynamic characteristics of complex systems, thereby improving the ability to identify sudden and variable faults. Simultaneously, the model's adaptive mechanism enables it to automatically adjust parameters based on real-time input and changes in the operating environment, ensuring that the prediction results maintain high accuracy and robustness. The introduction of external inputs further enriches the sources of modeling information, allowing the model to not only rely on historical data but also comprehensively consider the impact of external conditions on the system state. Overall, S3 provides scientifically reliable prediction results for subsequent fault detection, making the entire maintenance plan generation method forward-looking and dynamically adaptable.

[0026] S4: Based on the time-series characteristics, determine whether the information system has malfunctioned using the NARX-AD model; if yes, proceed to step S5; otherwise, continue monitoring.

[0027] In one possible implementation, S4 specifically includes: S401: Input the temporal features into the NARX-AD model:

[0028] in, express k The predicted value of the NARX-AD model at time 10:00. express k The model output at time step express kThe output of the NARX model at time step 1. Indicates the adaptive term. express k Enter the time number i The first variable j The adaptive parameters corresponding to the first-order delay, X ( k )express k Input data at any time, y ( k -1) represents the model k The actual output value at time -1, which is the model's previous prediction result. express k Random error at time, They follow a zero-mean normal distribution. express k - m Input data at any time, express k - n The actual output value at any given moment.

[0029] S402: Determine the fault detection signal of the information system by calculating the residual between the predicted and actual values ​​of the NARX-AD model.

[0030] in, r ( k )express k The residual at time, i.e., the fault detection signal. express k The actual measured value under fault conditions at any given time.

[0031] Among them, residual refers to the difference between the model prediction value and the actual observation value. It is used to measure the model prediction error and is also an important basis for fault detection.

[0032] S403: Eliminate the correlation of fault detection signals through whitening treatment:

[0033] in, d i Indicates the past number i The autoregressive coefficient at time t. i =1,2,…, N , N Represents the total number of moments. r ( k - i )express k - i Time residuals This represents the random error term in the AR(N) model that cannot be explained by historical residuals.

[0034] Among them, whitening refers to removing the autocorrelation of the residual sequence through an autoregressive model, making it closer to random noise, which facilitates subsequent analysis.

[0035] S404: Smooths the whitened fault detection signal using an exponentially weighted moving average filter.

[0036] in, z ( k )express k The fault detection signal after real-time smoothing. λ Represents the smoothing coefficient. z ( k -1) indicates k The fault detection signal after smoothing at time -1.

[0037] Among them, the Exponentially Weighted Moving Average (EWMA) filter is a smoothing method that highlights recent anomalies and suppresses random fluctuations by assigning exponentially decaying weights to historical residuals.

[0038] S405: Calculate the upper and lower control limits of the control chart based on the statistical characteristics of the fault detection signal.

[0039]

[0040] Wherein, UCL represents the maximum permissible value of the process output under normal conditions. This represents the average value of the fault detection signal after processing by the exponentially weighted moving average filter. c Indicates the control coefficient. The standard deviation of the fault detection signal after processing by the exponentially weighted moving average filter is represented by LCL, which represents the minimum allowable value of the process output under normal conditions.

[0041] Among them, the control limits (UCL / LCL) are thresholds in statistical process control used to determine whether the system output is within the normal fluctuation range; exceeding them indicates a possible fault.

[0042] S406: Determine whether the smoothed fault detection signal is greater than the upper control limit or less than the lower control limit. If yes, determine that the information system has failed and issue an alarm; otherwise, continue monitoring.

[0043] Among them, the fault detection signal is a discrimination index calculated based on the residual and control limit, which is used to determine whether a fault has occurred in the system.

[0044] It's important to note that, firstly, residual calculation directly reflects the deviation between model predictions and actual conditions, serving as a sensitive indicator for identifying potential faults. Secondly, whitening eliminates autocorrelation in the residual sequence, making the fault detection signal more consistent with the independent random hypothesis, thereby improving the reliability of the detection. Thirdly, EWMA smoothing effectively reduces noise interference, highlighting genuine abnormal fluctuations. Finally, by setting statistical control limits, complex time-series data is transformed into clear threshold judgment criteria, making fault detection operable and real-time. Overall, S4 ensures that the detection process can quickly identify anomalies while reducing false alarms and false negatives, providing a stable and reliable basis for subsequent maintenance decisions.

[0045] S5: Based on the fault type of the information system, determine the maintenance cost of the information system through a pre-trained artificial neural network.

[0046] Fault types refer to different categories of abnormal situations occurring in information systems, such as hardware failures, software errors, network outages, or resource bottlenecks. Different fault types determine different maintenance strategies. Pre-trained artificial neural networks (ANNs) are intelligent models trained on large-scale historical data. They can learn patterns from input features (such as fault types and system states) and output corresponding maintenance cost predictions. Maintenance costs refer to the resource consumption and economic costs required to perform a certain maintenance operation, including labor costs, spare parts costs, and system downtime losses.

[0047] It's important to note that compared to traditional methods relying on manual estimation or fixed rules, neural networks can automatically learn complex nonlinear relationships from a large amount of historical maintenance records, providing more accurate and personalized cost assessments. This approach allows for the quantification of the economic costs of different decisions during the maintenance plan generation process, avoiding resource waste or downtime losses due to blind repairs. Simultaneously, neural network models possess excellent generalization capabilities, enabling them to handle novel faults or situations under different system environments, improving the adaptability and robustness of cost prediction. Overall, this step adds an economic optimization dimension to the entire plan, making subsequent maintenance strategy selection more scientific and rational.

[0048] In one possible implementation, S5 specifically includes: S501: Determine the status of the information system based on the type of fault in the information system.

[0049] Among them, the information system status refers to the overall operating status of the system at a certain moment, including whether the components are normal, whether the performance indicators meet the standards, and whether the system is in an available or faulty state.

[0050] S502: Based on the fault type of the information system, different maintenance methods are used to maintain the information system and obtain the status of the information system after maintenance.

[0051] The post-maintenance information system status refers to the operational status of the system after specific maintenance operations are performed, and is used to measure the effectiveness of maintenance measures.

[0052] S503: Input the information system status and the post-maintenance information system status into a pre-trained artificial neural network, and output the maintenance cost of the information system.

[0053] S6: With the goal of minimizing maintenance costs, construct the maintenance environment for the information system by referring to the multidimensional Markov decision process.

[0054] Multidimensional Markov Decision Processes (MDPs) are mathematical modeling methods used to describe decision-making problems in uncertain environments. They include a state space, action space, state transition probabilities, and a reward function. The multidimensional representation allows for the simultaneous consideration of multiple system components and factors. A maintenance environment refers to a simulation or computational environment built upon MDPs to simulate system state changes, the effects of maintenance operations, and their associated costs and benefits, supporting subsequent intelligent decision-making.

[0055] It's important to note that, firstly, minimizing maintenance costs ensures that the maintenance plan considers not only repair effectiveness but also economic efficiency. Secondly, multi-dimensional modeling using MDP allows for the simultaneous depiction of the states and evolutionary patterns of multiple components, thus more realistically reflecting the overall system dynamics. Thirdly, the introduction of state transition and reward mechanisms quantifies maintenance effectiveness and costs, facilitating subsequent algorithm learning and optimization. Finally, the constructed maintenance environment provides a standardized training scenario for intelligent algorithms such as reinforcement learning, enabling iterative policy improvement. Overall, this approach ensures a scientific, rational, and scalable decision-making process, laying a solid foundation for generating intelligent, low-cost maintenance plans.

[0056] In one possible implementation, S6 specifically includes: S601: Define the state space of the information system based on the state data of each component in the information system.

[0057] The state space refers to the set of all possible operating states of an information system, such as normal, minor anomaly, severe anomaly, and shutdown, and is used to describe the overall health of the system.

[0058] S602: Define the action space of the information system based on the maintenance actions of each component in the information system.

[0059] The action space refers to the set of all maintenance operations that can be performed on the system, such as restarting services, replacing hardware, patching, and load migration.

[0060] S603: Define the state transition function of the information system based on the state transition matrices of each component in the information system:

[0061] in, Representation Component e From state u to state v The transition probability, Representation Component e From state u to state v Number of transfers, m e Representation Component e The maximum number of states, e =1,2,…, E , E This indicates the total number of components in an information system.

[0062] The state transition function describes the probabilistic model of the system transitioning from one state to another after performing a certain maintenance action, and is used to characterize the evolution of the system.

[0063] S604: Define the reward function for the information system based on its maintenance costs.

[0064] in, r j Indicates the first j Step rewards c j Indicates the first j Total cost of the step Indicates the first j The maintenance cost of the step Indicates the first j Status indicators of the step information system, when When the value equals 1, it indicates that the information system is in a fault state. A value of 0 indicates that the information system is in a normal state. This represents the downtime cost of the information system. This represents the maintenance cost function. s j Indicates the first j Step information system status, Indicates the first j The status of the information system after maintenance.

[0065] Among them, the reward function is used to quantify the evaluation index of the quality of maintenance decisions. It is usually related to maintenance costs and downtime costs. The larger the reward value, the better the solution.

[0066] S605: Define the discount factor.

[0067] The discount factor is a parameter used in reinforcement learning to balance short-term and long-term returns, ensuring that decisions take into account both immediate effects and long-term benefits.

[0068] S606: Determine the maintenance environment of the information system based on the state space, the action space, the state transition function, the reward function, and the discount factor.

[0069] It's important to note that, firstly, the definition of the state space and action space formalizes the system's operational status and possible maintenance operations, ensuring a clear problem structure. Secondly, the state transition function utilizes historical data to characterize the system's evolution under different operations, making the model more realistic. Thirdly, the reward function incorporates maintenance costs and downtime expenses into a unified consideration, ensuring that solution optimization not only focuses on fault repair but also on economic benefits. Finally, the introduction of a discount factor balances short-term costs and long-term benefits, making the decision more sustainable.

[0070] S7: Based on the maintenance environment, determine the maintenance plan for the information system through a multi-agent deep reinforcement learning algorithm.

[0071] Deep reinforcement learning algorithms are a method that combines deep neural networks with reinforcement learning, enabling the learning of optimal policies in complex, high-dimensional state and action spaces. A repair plan refers to a sequence of learned optimal actions, encompassing specific repair measures taken by different agents under different states.

[0072] It's important to note that, firstly, the multi-agent mechanism decomposes complex system problems, with each agent making independent decisions based on its own component state, while simultaneously achieving global optimization through collaboration, thus avoiding the bottleneck of centralized control. Secondly, deep reinforcement learning possesses powerful nonlinear modeling capabilities, enabling efficient searching in high-dimensional state and action spaces, making the solution more consistent with the dynamic characteristics of complex systems. Thirdly, the algorithm iteratively updates its strategy through continuous trial and error and experience replay, not only adapting to environmental changes but also continuously improving decision quality. Finally, the maintenance plan output by S7 balances economy and reliability, providing intelligent and data-driven guidance for actual operation and maintenance. Overall, this step significantly improves the scientific rigor, adaptability, and collaboration of the maintenance strategy, and is the core component of the entire method for achieving intelligent operation and maintenance.

[0073] In one possible implementation, S7 specifically includes: S701: Initialize training steps, experience replay buffer, policy network, target network, and exploration constant.

[0074] The training steps are the total number of model iterations during reinforcement learning, used to control the training scale. The experience replay buffer is a set storing samples of states, actions, rewards, and next states generated by the agent's interactions in the environment, used to break sample correlation and improve training efficiency. The policy network is a neural network used to approximate the value of the current decision and output the optimal action selection. The target network is a delayed copy of the policy network, used to calculate the target value, avoiding instability caused by overly rapid updates of the estimate during training. The exploration constant is a parameter that controls the balance between random exploration and utilizing existing knowledge, ensuring that the model can try new solutions while also utilizing known good policies.

[0075] In one possible implementation, both the policy network and the target network employ a branch-and-fight structure, with the target network having the same initial weights as the target network.

[0076] Among them, the branching dueling architecture is an improved neural network structure that estimates "state value" and "action advantage" separately and then combines them to better determine which states are important, even if the specific action choices are not significantly different in some states.

[0077] It's important to note that the branch-and-fight structure decouples state value and action advantage, allowing the network to better assess the importance of different states, thereby improving the accuracy and efficiency of action selection in a high-dimensional action space. This way, even if action differences are not significant in certain states, the network can still learn which states are more critical for long-term gains. Simultaneously, the policy network and target network have consistent parameters in the initial training phase, avoiding valuation bias issues in the early learning stages, reducing training oscillations, and accelerating convergence. Overall, this design not only improves the stability and robustness of reinforcement learning models but also accelerates policy convergence, ultimately generating better and more reliable troubleshooting strategies.

[0078] S702: Based on the information system status, uniformly sample scalar quantities.

[0079] Specifically, scalars are uniformly sampled from [0,1]. S703: Determine if the scalar is less than the exploration constant. If so, randomly select an action from the action space to perform maintenance on the information system. Otherwise, determine the optimal maintenance action for the information system through the policy network.

[0080] Where argmax represents maximization. Q tot() denotes the global value function after decomposition. Q tot ( s , a ) represents the global state s Next, take joint action a The long-term value (expected future cumulative rewards). A joint Represents the action space, Indicates the first e The optimal action of each agent in the global state. s Next, the e Each intelligent agent in its own individual action space Ae In the middle, find the local value function Maximize individual actions , e =1,2,…, E , E This represents the total number of components in an information system, i.e., the total number of intelligent agents.

[0081] In one possible implementation, S703 specifically involves determining the optimal maintenance action for the information system using a Value Decomposition Network (VDN) algorithm.

[0082] Value Decomposition Network (VDN) is a multi-agent reinforcement learning method that decomposes the global value function into the sum of individual value functions of multiple agents. This decomposition allows each agent to learn independently while also collectively optimizing the global objective. Optimal maintenance actions refer to maintenance decisions that minimize the long-term operating cost and maximize the benefits of the information system under the current conditions. These actions may include restarting services, replacing components, or adjusting configurations.

[0083] It's important to note that the advantage of using the VDN algorithm to determine the optimal maintenance actions lies in its transformation of the complex multi-agent collaborative problem into a parallel, solvable individual problem by decomposing the global value function into the sum of the individual value functions of each agent. This reduces computational complexity, making the training process more efficient. Furthermore, it ensures consistency between the global optimal goal and individual learning, avoiding conflicts between local decisions and overall optimization. In information system maintenance scenarios, the states and operations of each component are both independent and interconnected. VDN fully leverages this structural characteristic to achieve collaborative optimization among multiple components. The resulting maintenance actions not only quickly adapt to different fault scenarios but also reduce costs and improve reliability at the global level.

[0084] In one possible implementation, S703 specifically includes: S7031: The information system is broken down into multiple independent and maintainable components, wherein the maintenance actions of each component are determined by an intelligent agent.

[0085] In this context, the agent is the autonomous decision-making unit in reinforcement learning. Each agent corresponds to a component and is responsible for selecting the maintenance action for that component.

[0086] S7032: Determine the global value function using the Value Decomposition Network (VDN) algorithm:

[0087] in, Let D represent the joint action-value function, and D represent the expected operation. R j Indicates the first j The instant reward for each step, where γ represents the discount factor. S j Indicates the first j The state of the step, A j Indicates the first j The action of stepping.

[0088] Among them, the Value Decomposition Network (VDN) algorithm is a multi-agent reinforcement learning method that decomposes the global value function into the sum of individual value functions, thereby achieving consistency between distributed learning and global optimization.

[0089] S7033: Decompose the global value function into individual value functions for all agents:

[0090] in, Indicates the first e Action value function for each agent.

[0091] S7044: Determine the maintenance plan for the information system with the goal of maximizing the sum of the individual value functions of each agent.

[0092] It's important to note that, firstly, decomposing the information system into multiple independent components modularizes complex problems, facilitating parallel processing by multiple agents. Secondly, VDN decomposes the global value function into the sum of individual value functions, enabling each agent to learn independently while ensuring consistency between local decisions and global goals. Thirdly, this decomposition reduces the complexity of global computation, improving the efficiency of training and inference. Finally, by maximizing the sum of the value functions of each agent, an overall optimal maintenance plan can be generated, ensuring both global optimization and individual optimality, thus improving the reliability and economy of the plan. Overall, this method enhances the coordination of multi-agent collaboration and improves the scalability and intelligence level of system operation and maintenance.

[0093] S704: Execute the optimal maintenance action and record the information system status after executing the optimal maintenance action.

[0094] S705: Calculate the reward for the information system based on the information system status after performing the optimal maintenance action and the information system status.

[0095] S706: Based on the patterns of each component in the information system, the information system will naturally degenerate to the next state.

[0096] S707: The information system status, optimal maintenance action, reward, and next status are stored as state transition samples in the experience replay buffer for updating the experience replay buffer.

[0097] S708: Select a batch of samples uniformly from the updated experience replay buffer.

[0098] S709: Calculate the target value for each sample in the batch:

[0099] in, y j Indicates the first j The target value of the step, r j Indicates the first j Instant rewards for each step Represents the target global Q function. Indicates the state Below, the global Q function is made Maximize the combined action.

[0100] S710: Determine the objective function of the policy network based on each objective value and the mean square error of the estimated global value function.

[0101] S711: The goal is to minimize the value of the objective function and adjust the weights of the policy network.

[0102] S712: Update the target network based on the weights of the adjusted policy network.

[0103] S713: Repeat steps S702-S712 until the maximum number of iterations is reached, and output the trained policy network.

[0104] Specifically, the maximum number of iterations is the number of training steps.

[0105] S714: Determine the maintenance plan for the information system based on the trained policy network.

[0106] It's important to note that, firstly, the introduction of an experience replay buffer and a target network effectively improves training stability and convergence, avoiding the risk of overfitting due to sample correlation. Secondly, by designing an exploration constant, a balance is struck between random exploration and optimal action selection, enabling the model to continuously discover potentially better maintenance strategies. Thirdly, the decomposition mechanism of the global value function and individual value function ensures that multiple agents can make independent decisions while achieving global collaborative optimization. Finally, through iterative iteration and error-driven weight updates, the policy network gradually converges to the optimal solution, thus outputting a low-cost, high-reliability maintenance plan. Overall, this process ensures the scientific rigor, adaptability, and optimality of the maintenance plan, forming the core mechanism for achieving intelligent maintenance.

[0107] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this embodiment of the invention, time-series features are constructed based on the collected status data and historical maintenance records of the information system. These features are then input into the NARX-AD model to determine whether the information system has experienced a fault. The NARX-AD model can not only predict system behavior but also adaptively adjust based on data characteristics, ensuring that the modeling results are highly consistent with the real system. Next, based on the fault type of the information system, the maintenance cost of the information system is predicted using a pre-trained artificial neural network. With the goal of minimizing maintenance costs, and referencing a multidimensional Markov decision process, the maintenance environment of the information system is constructed through the definitions of state space, action space, and transition function. Based on the maintenance environment, a maintenance plan for the information system is determined using a multi-agent deep reinforcement learning algorithm. By utilizing the parallel decision-making and collaborative capabilities of multiple agents, an optimal or near-optimal maintenance plan is generated. Compared to traditional single-agent methods, this approach is more adaptable to the distributed characteristics of complex systems. Finally, the information system is repaired according to the maintenance plan, improving the level of automation, reducing manual maintenance costs, increasing the accuracy and efficiency of maintenance, shortening repair time, further reducing downtime costs, and improving the availability and reliability of the system.

[0108] Reference manual attached Figure 2 The diagram shows a structural schematic of an information system maintenance scheme generation system based on multi-agent collaboration provided by an embodiment of the present invention.

[0109] This invention provides an information system maintenance scheme generation system 20 based on multi-agent collaboration, comprising: a processor 201 and a memory 202; The memory 202 stores programs or instructions that can run on the processor 201. When the program or instructions are executed by the processor 201, they implement the steps of the above-mentioned method for generating maintenance schemes for information systems based on multi-agent collaboration and achieve the same technical effect. To avoid repetition, the present invention will not elaborate further.

[0110] It should be understood that the processor 201 in this embodiment of the invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0111] It should also be understood that the memory 202 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM).

[0112] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0113] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] This invention provides a readable storage medium that stores a program or instructions on the medium. When the program or instructions are executed by a processor, they implement the steps of the above-described method for generating a maintenance scheme for an information system based on multi-agent collaboration, and achieve the same technical effect. To avoid repetition, this invention will not elaborate further.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating maintenance plans for information systems based on multi-agent collaboration, characterized in that, include: S1: Collect status data and historical maintenance records of various components in the information system; S2: Construct time-series features based on the status data and the historical maintenance records; S3: Construct the NARX-AD model; S4: Based on the time-series characteristics, determine whether the information system has malfunctioned using the NARX-AD model; if yes, proceed to step S5; otherwise, continue monitoring. S5: Based on the fault type of the information system, determine the maintenance cost of the information system through a pre-trained artificial neural network; S6: With the goal of minimizing the maintenance cost, construct the maintenance environment of the information system by referring to the multidimensional Markov decision process; S7: Based on the maintenance environment, determine the maintenance plan for the information system using a multi-agent deep reinforcement learning algorithm; S8: Repair the information system according to the maintenance plan.

2. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, The status data includes CPU utilization, memory usage, and log data.

3. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, S4 specifically includes: S401: Input the time-series features into the NARX-AD model; S402: Determine the fault detection signal of the information system by calculating the residual between the predicted value and the actual value of the NARX-AD model; S403: Eliminate the correlation of the fault detection signal through whitening treatment; S404: Smooths the whitened fault detection signal using an exponentially weighted moving average filter; S405: Calculate the upper and lower control limits of the control chart based on the statistical characteristics of the fault detection signal; S406: Determine whether the smoothed fault detection signal is greater than the upper control limit or less than the lower control limit; if so, determine that the information system has malfunctioned and issue an alarm, proceed to step S5; otherwise, continue monitoring.

4. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, S5 specifically includes: S501: Determine the status of the information system based on the fault type of the information system; S502: Based on the fault type of the information system, the information system is repaired using different maintenance methods to obtain the status of the information system after maintenance; S503: Input the information system status and the information system status after maintenance into the pre-trained artificial neural network, and output the maintenance cost of the information system.

5. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, S6 specifically includes: S601: Define the state space of the information system based on the state data of each component in the information system; S602: Define the action space of the information system based on the maintenance actions of each component in the information system; S603: Define the state transition function of the information system based on the state transition matrix of each component in the information system; S604: Define the reward function of the information system based on the maintenance cost of the information system; S605: Define the discount factor; S606: Determine the maintenance environment of the information system based on the state space, the action space, the state transition function, the reward function, and the discount factor.

6. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 5, characterized in that, Specifically, S7 includes: S701: Initialize training steps, experience replay buffer, policy network, target network, and exploration constant; S702: Based on the state of the information system, perform uniform sampling of the scalar; S703: Determine whether the scalar is less than the exploration constant; if so, randomly select an action from the action space to repair the information system; otherwise, determine the optimal repair action for the information system through the policy network. S704: Execute the optimal maintenance action and record the information system status after executing the optimal maintenance action; S705: Calculate the reward for the information system based on the information system status after performing the optimal maintenance action and the information system status; S706: Based on the patterns of each component in the information system, the information system is naturally degraded to the next state; S707: Store the information system status, the optimal maintenance action, the reward, and the next status as state transition samples in the experience replay buffer to update the experience replay buffer; S708: Select a batch of samples evenly from the updated experience replay buffer; S709: Calculate the target value of each sample in the batch sample; S710: Determine the objective function of the policy network based on each of the stated objective values ​​and the mean square error of the estimated global value function; S711: To minimize the value of the objective function, the weights of the policy network are adjusted; S712: Update the target network according to the weights of the adjusted policy network; S713: Repeat steps S702-S712 until the maximum number of iterations is reached, and output the trained policy network. S714: Determine the maintenance plan for the information system based on the trained policy network.

7. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 6, characterized in that, Both the policy network and the target network adopt a branch-and-fight structure, and the initial weights of the target network are the same as those of the target network.

8. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 6, characterized in that, The process in S703 of determining the optimal maintenance action for the information system through the policy network specifically involves: The optimal maintenance action for the information system is determined using a value decomposition network algorithm.

9. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 8, characterized in that, The S703 specifically includes: S7031: The information system is disassembled into multiple independent components to be inspected, wherein the inspection actions of each component to be inspected are determined by an intelligent agent; S7032: Determine the global value function using the value decomposition network algorithm; S7033: Decompose the global value function into individual value functions for all agents; S7034: Determine the maintenance plan for the information system with the goal of maximizing the sum of the individual value functions of each intelligent agent.

10. A system for generating maintenance plans for information systems based on multi-agent collaboration, characterized in that, include: Processor and memory; The memory stores programs or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the method for generating maintenance schemes for information systems based on multi-agent collaboration as described in any one of claims 1 to 9.

Images