A method and system for generating an information system maintenance scheme based on multi-agent cooperation

Abstract

The application provides a kind of information system overhaul scheme generation method and system based on multi-agent cooperation, it is related to equipment state maintenance technical field, method includes: the state data and historical maintenance record of each component in information system are collected;According to state data and historical maintenance record, time sequence characteristics are constructed;NARX-AD model is constructed;Whether information system appears fault is judged by NARX-AD model, if yes;The maintenance cost of information system is predicted, otherwise, continue monitoring;According to the fault type of information system, the maintenance cost of information system is predicted by pre-training artificial neural network;With the goal of minimizing maintenance cost, reference multidimensional Markov decision process, the maintenance environment of information system is constructed;Based on maintenance environment, the maintenance scheme of information system is determined by multi-agent deep reinforcement learning algorithm;According to maintenance scheme, information system is repaired.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of equipment state maintenance, and in particular relates to a method and system for generating an information system maintenance scheme based on multi-agent collaboration. BACKGROUND

[0002] Multi-agent refers to multiple autonomous decision-making "agents", which can be virtual software agents or physical devices, each having perception, decision-making and execution capabilities, and being able to collaborate or game in an environment. An information system is a whole composed of hardware (such as servers, storage), software (applications, databases), networks and operation and maintenance mechanisms, and is used to process, transmit and store information. The method for generating an information system maintenance scheme based on multi-agent collaboration is to use the collaboration between multiple agents to automatically generate a suitable maintenance scheme for the information system when a problem occurs through intelligent algorithms and models, thereby more efficiently and at a lower cost ensuring the normal operation of the system.

[0003] In the modern information environment, the complexity of information systems is increasing. By introducing the mechanism of multi-agent collaboration, different parts of the information system can be regarded as being managed by different "agents", each of which focuses on the collection and decision-making of local information, and the global optimization is achieved through collaboration strategies, providing an intelligent, automated and collaborative maintenance path for complex information systems, greatly improving the resilience and operation efficiency of the system, and laying a foundation for future intelligent system management.

[0004] However, in the current information system operation and maintenance practice, the generation of maintenance schemes often relies on experience rules and manual intervention, lacks dynamic adaptability, and once the system environment changes, the rules may be invalid. Secondly, many schemes rely on manual expert judgment, resulting in slow response speed and being easily affected by human subjective factors. In addition, traditional methods usually focus on the abnormality of a single component, ignoring the coupling relationship between components, making it difficult to effectively identify the linkage failure in complex systems, resulting in problems such as insufficient adaptability, lack of collaboration, limited automation level and insufficient cost optimization, which is difficult to meet the high availability, high efficiency and low cost operation and maintenance needs of modern complex systems. SUMMARY

[0005] In view of the above deficiencies of the prior art, the purpose of the embodiments of the present application is to provide a method for generating an information system maintenance scheme based on multi-agent collaboration, which can solve the technical problem that the generation of traditional maintenance schemes often relies on experience rules and manual intervention, lacks dynamic adaptability, and once the system environment changes, the rules may be invalid, resulting in problems such as insufficient adaptability, lack of collaboration, limited automation level and insufficient cost optimization, which is difficult to meet the high availability, high efficiency and low cost operation and maintenance needs of modern complex systems.

[0006] In a first aspect, the present application provides a method for generating a maintenance scheme of an information system based on multi-agent cooperation, comprising:

[0007] S1: collecting state data and historical maintenance records of each component in the information system;

[0008] S2: constructing a time sequence feature according to the state data and the historical maintenance records;

[0009] S3: constructing an NARX-AD model;

[0010] S4: judging whether the information system has a fault according to the time sequence feature and the NARX-AD model; if yes, proceeding to step S5; otherwise, continuing to monitor;

[0011] S5: determining a maintenance cost of the information system according to a fault type of the information system and a pre-trained artificial neural network;

[0012] S6: constructing a maintenance environment of the information system by referring to a multi-dimensional Markov decision process, with the aim of minimizing the maintenance cost;

[0013] S7: determining a maintenance scheme of the information system based on the maintenance environment and a multi-agent deep reinforcement learning algorithm;

[0014] S8: repairing the information system according to the maintenance scheme.

[0015] In a second aspect, the present application provides a system for generating a maintenance scheme of an information system based on multi-agent cooperation, comprising: a processor and a memory.

[0016] The memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method for generating a maintenance scheme of an information system based on multi-agent cooperation according to the first aspect.

[0017] In a third aspect, the present application provides a readable storage medium, and the readable storage medium stores programs or instructions, which are executed by the processor to implement the steps of the method for generating a maintenance scheme of an information system based on multi-agent cooperation according to the first aspect.

[0018] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0019] In the embodiment of the present application, according to the collected state data and historical maintenance records of the information system, time series features are constructed, and the time series features are input into the NARX-AD model to determine whether the information system fails, the NARX-AD model can not only predict the system behavior, but also adaptively adjust according to the data characteristics to ensure that the modeling result is highly consistent with the real system, then, according to the fault type of the information system, the maintenance cost of the information system is predicted through the pre-trained artificial neural network, taking minimizing the maintenance cost as the goal, referring to the multi-dimensional Markov decision process, through the definition of state space, action space and transition function, the maintenance environment of the information system is constructed, based on the maintenance environment, through the multi-agent deep reinforcement learning algorithm, the maintenance scheme of the information system is determined, through the parallel decision and collaborative ability of the multi-agent, the optimal or approximate optimal maintenance scheme is generated, compared with the traditional single-agent method, it can better adapt to the distributed characteristics of the complex system, finally, according to the maintenance scheme, the information system is repaired, the automation level is improved, the manual maintenance cost is reduced, the accuracy and efficiency of the maintenance are improved, the repair time is shortened, the downtime cost is further reduced, and the availability and reliability of the system are improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0021] Figure 1 FIG. 1 is a flow diagram of a method for generating an information system maintenance scheme based on multi-agent collaboration according to an embodiment of the present application.

[0022] Figure 2 FIG. 2 is a structural diagram of a system for generating an information system maintenance scheme based on multi-agent collaboration according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order for those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions of the present application will be described clearly and completely below with reference to the drawings, obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. It should be understood that these descriptions are exemplary and not intended to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0024] The method for generating an information system maintenance scheme based on multi-agent cooperation provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings, specific examples and application scenarios.

[0025] Reference is made to the accompanying drawings Figure 1 , which shows a flowchart of a method for generating an information system maintenance scheme based on multi-agent cooperation provided by the embodiments of the present application.

[0026] The embodiments of the present application provide a method for generating an information system maintenance scheme based on multi-agent cooperation, which can include the following steps:

[0027] S1: Collecting state data and historical maintenance records of each component in the information system.

[0028] Among them, the component refers to the hardware and software units that constitute the information system, such as servers, databases, storage devices, network modules, application programs, etc., which are the basic carriers of system operation. The state data describes the dynamic indicators of the component running condition, including CPU usage, memory occupancy, disk I / O rate, network traffic and log files, etc., which can reflect the system health status and performance level. The historical maintenance record refers to the complete record of past faults, maintenance measures, repair methods and effects, containing fault causes, processing steps, time consumption and cost, etc., which provides experience basis for future maintenance.

[0029] It should be noted that the systematic collection of state data and historical maintenance records provides a solid data foundation for subsequent modeling, prediction and optimization. First, the state data can reflect the system running condition in real time, so that the fault signs can be captured in time, thereby improving the detection sensitivity. Second, the historical maintenance record provides rich prior knowledge for the model, so that the system can not only identify known fault patterns, but also find potential risks through pattern comparison. Third, the combination of the two can realize the unified analysis of short-term anomalies and long-term trends, providing high-quality input for subsequent time series feature extraction and prediction model construction. Through this link, the completeness and traceability of the data are guaranteed, avoiding misjudgment caused by one-sided information.

[0030] In one possible implementation, the state data includes CPU usage, memory usage and log data.

[0031] S2: Constructing time series features according to the state data and the historical maintenance records.

[0032] Among them, the time series feature refers to the data characteristics generated over time, such as trend, periodicity, volatility, etc., which is used to reveal the dynamic change law of system state.

[0033] It should be noted that the original state data and historical maintenance records are converted into quantifiable and analyzable time series features, thereby better capturing the dynamic rules of the information system operation state. Through this process, potential trend changes, periodic fluctuations, and short-term abnormalities can be revealed, making the subsequent model more robust and interpretable when processing input data. At the same time, the extraction of time series features can eliminate redundant information, highlight key variables, and improve modeling efficiency and accuracy. Features generated in combination with historical maintenance records can also help the model establish a mapping relationship between data and fault modes, enhancing the prediction ability of future abnormalities.

[0034] S3: Constructing a NARX-AD model.

[0035] Among them, the adaptive nonlinear autoregressive model (NARX-AD) is an improved time series prediction model based on nonlinear autoregressive structure (NARX) and adaptive mechanism (AD), which can dynamically adjust parameters according to external input and system historical state to improve prediction accuracy.

[0036] It should be noted that by constructing the adaptive nonlinear autoregressive model (NARX-AD), the accuracy and flexibility of information system state prediction and anomaly detection are effectively improved. Compared with traditional linear models, NARX-AD can better capture nonlinear dynamic characteristics in complex systems, thereby improving the ability to identify sudden and variable faults. At the same time, the adaptive mechanism of the model allows it to automatically adjust parameters according to real-time input and changes in the operating environment, ensuring that the prediction results remain highly accurate and robust. The introduction of external input further enriches the source of modeling information, making the model not only rely on historical data but also consider the impact of external conditions on system state. Overall, S3 provides scientific and reliable prediction results for the subsequent fault detection step, making the entire maintenance scheme generation method forward-looking and dynamically adaptive.

[0037] S4: According to the time series features, determine whether the information system has failed through the NARX-AD model; if so, go to step S5; otherwise, continue monitoring.

[0038] In one possible implementation, S4 specifically includes:

[0039] S401: Input the time series features into the NARX-AD model:

[0040]

[0041] Among them, represents k the predicted value of the NARX-AD model at time N, represents k the model output at time N, express k The output of the NARX model at time step 1. Indicates the adaptive term. express k Enter the time number i The first variable j The adaptive parameters corresponding to the first-order delay, X ( k )express k Input data at any time, y ( k -1) represents the model k The actual output value at time -1, which is the model's previous prediction result. express k Random error at time, They follow a zero-mean normal distribution. express k - m Input data at any time, express k - n The actual output value at any given moment.

[0042] S402: Determine the fault detection signal of the information system by calculating the residual between the predicted and actual values ​​of the NARX-AD model.

[0043]

[0044] in, r ( k )express k The residual at time, i.e., the fault detection signal. express k The actual measured value under fault conditions at any given time.

[0045] Among them, residual refers to the difference between the model prediction value and the actual observation value. It is used to measure the model prediction error and is also an important basis for fault detection.

[0046] S403: Eliminate the correlation of fault detection signals through whitening treatment:

[0047]

[0048] in, d i Indicates the past number i The autoregressive coefficient at time t. i =1,2,…, N , N Represents the total number of moments. r ( k - i )expressk - i the residual at time t, represents a random error term in the AR(N) model that cannot be explained by historical residuals.

[0049] wherein the whitening process refers to removing the autocorrelation of the residual sequence through an autoregressive model, so that it is closer to random noise, facilitating subsequent analysis.

[0050] S404: Smooth the fault detection signal after whitening by an exponential weighted moving average filter:

[0051]

[0052] wherein, z ( k ) represents k the smoothed fault detection signal at time t, Lambda represents a smoothing coefficient, z ( k -1) represents k the smoothed fault detection signal at time t-1.

[0053] wherein the exponential weighted moving average filter (EWMA) is a smoothing method that assigns exponential decay weights to historical residuals, highlighting recent anomalies and suppressing random fluctuations.

[0054] S405: Calculate the upper control limit and the lower control limit of the control chart according to the statistical characteristics of the fault detection signal:

[0055]

[0056]

[0057] wherein UCL represents the maximum allowable value of the process output under normal conditions, represents the average value of the fault detection signal after exponential weighted moving average filter processing, c represents a control coefficient, represents the standard deviation of the fault detection signal after exponential weighted moving average filter processing, and LCL represents the minimum allowable value of the process output under normal conditions.

[0058] wherein the control limit (UCL / LCL) is a threshold value in statistical process control, used to determine whether the system output is within the normal fluctuation range, and if it exceeds, it indicates that there may be a fault.

[0059] S406: Determine whether the smoothed fault detection signal is greater than the upper control limit or less than the lower control limit, if so, determine that the information system has failed, and issue an alarm, otherwise, continue monitoring.

[0060] wherein the fault detection signal is a discriminant index calculated based on the residual and the control limit, for judging whether the system has a fault.

[0061] It should be noted that first, residual calculation can intuitively reflect the deviation between model prediction and actual state, and is a sensitive index for discovering potential faults. Second, whitening eliminates the autocorrelation in the residual sequence, making the fault detection signal more consistent with the independent random hypothesis, thereby improving the reliability of the test. Third, EWMA smoothing effectively reduces noise interference and can highlight the true abnormal fluctuations. Finally, by setting a statistical control limit, complex time series data is converted into explicit threshold judgment criteria, making fault detection operable and real-time. Overall, S4 ensures that the detection link can quickly discover abnormalities while reducing false positives and false negatives, providing a stable and reliable basis for subsequent maintenance decisions.

[0062] S5: According to the fault type of the information system, determine the maintenance cost of the information system through a pre-trained artificial neural network.

[0063] wherein the fault type refers to different categories of abnormal situations occurring in the information system, such as hardware failure, software error, network interruption or resource bottleneck, and different fault types determine different maintenance strategies. The pre-trained artificial neural network (ANN) is an intelligent model trained based on large amounts of historical data, which can learn the rules from input features (such as fault type, system state) and output corresponding maintenance cost prediction results. Maintenance cost refers to the resource consumption and economic cost required to perform a certain maintenance operation, including labor cost, spare parts cost, system downtime loss, etc.

[0064] It should be noted that compared with traditional methods that rely on manual estimation or fixed rules, neural networks can automatically learn complex nonlinear relationships from a large number of historical maintenance records, providing more accurate and personalized cost estimates. This approach allows the economic cost of different decisions to be quantified during the maintenance scheme generation process, avoiding resource waste or downtime losses caused by blind repair. At the same time, the neural network model has good generalization ability and can handle new types of faults or different system environments, improving the adaptability and robustness of cost prediction. Overall, this step adds an economic optimization dimension to the entire scheme, making the subsequent maintenance strategy selection more scientific and reasonable.

[0065] In one possible implementation, S5 specifically includes:

[0066] S501: According to the fault type of the information system, determine the information system state.

[0067] The information system state refers to the overall operation of the system at a certain moment, including whether the components are normal, whether the performance indicators meet the standards, and whether the system is in a usable or faulty state.

[0068] S502: According to the fault type of the information system, the information system is repaired by different repair methods, and the repaired information system state is obtained.

[0069] The repaired information system state refers to the restored or improved operation state of the system after performing a specific repair operation, which is used to measure the effectiveness of the repair measures.

[0070] S503: The information system state and the repaired information system state are input into a pre-trained artificial neural network, and the repair cost of the information system is output.

[0071] S6: With the goal of minimizing repair cost, a repair environment for the information system is constructed by referring to a multi-dimensional Markov decision process.

[0072] The multi-dimensional Markov decision process (MDP) is a mathematical modeling method used to describe decision-making problems in uncertain environments, which includes state space, action space, state transition probability, and reward function, and multi-dimensional means that multiple system components and factors can be considered simultaneously. The repair environment refers to a simulation or calculation environment constructed based on MDP, which is used to simulate system state changes, repair operation effects, and the costs and benefits they bring, supporting subsequent intelligent decision-making.

[0073] It should be noted that first, the goal of minimizing repair cost makes the repair scheme not only focus on repair effect, but also consider economy. Second, through multi-dimensional modeling of MDP, the state and evolution law of multiple components can be described simultaneously, thus more truly reflecting the overall dynamics of the system. Third, the introduction of state transition and reward mechanism makes the repair effect and cost quantifiable, facilitating subsequent algorithm learning and optimization. Finally, the constructed repair environment provides a standardized training scenario for intelligent algorithms such as reinforcement learning, enabling iterative improvement of strategies. Overall, it ensures that the decision-making process is scientific, reasonable, and scalable, laying a solid foundation for subsequent generation of intelligent and low-cost repair schemes.

[0074] In one possible implementation, S6 specifically includes:

[0075] S601: According to the state data of each component in the information system, the state space of the information system is defined.

[0076] The state space refers to the set of all possible running states of the information system, such as normal, slight abnormal, serious abnormal, and fault shutdown, which is used to describe the overall health of the system.

[0077] S602: Define the action space of the information system according to the maintenance actions of each component in the information system.

[0078] wherein the action space refers to a set of all maintenance operations that can be taken on the system, such as restarting services, replacing hardware, patching, load migration, etc.

[0079] S603: Define the state transition function of the information system according to the state transition matrix of each component in the information system:

[0080]

[0081] wherein, represents the transition probability of component e from state u to state v , represents the number of transitions of component e from state u to state v , m e represents the maximum number of states of component e , e =1,2,…, E , E represents the total number of components in the information system.

[0082] wherein the state transition function describes the probability model of the system from one state to another after performing a certain maintenance action, which is used to describe the evolution law of the system.

[0083] S604: Define the reward function of the information system according to the maintenance cost of the information system:

[0084]

[0085] wherein, r j represents the reward of step j , c j represents the total cost of step j , represents the maintenance cost of step j , represents the state indicator of the information system at step j , when is equal to 1, it indicates that the information system is in a failure state, when is equal to 0, it indicates that the information system is in a normal state, represents the downtime cost of the information system, represents the maintenance cost function, sj represents the state of the information system at the first j step, represents the state of the information system after the first j step of maintenance.

[0086] wherein the reward function is used to quantify the evaluation index of the maintenance decision, and is usually related to the maintenance cost and downtime cost, and the greater the reward value indicates the better the scheme.

[0087] S605: defining a discount factor.

[0088] wherein the discount factor is a parameter used to balance short-term returns and long-term returns in reinforcement learning, ensuring that the decision takes into account both immediate effects and long-term benefits.

[0089] S606: determining the maintenance environment of the information system according to the state space, the action space, the state transition function, the reward function, and the discount factor.

[0090] It should be noted that first, the definition of the state space and the action space establishes a formal expression for the system operation and possible maintenance operations, ensuring that the problem structure is clear. Second, the state transition function uses historical data to describe the evolution law of the system under different operations, making the model more realistic. Third, the reward function unifies the maintenance cost and downtime cost into consideration, ensuring that the scheme optimization not only focuses on fault repair, but also takes into account economic benefits. Finally, the introduction of the discount factor can balance short-term costs and long-term benefits, making the decision more sustainable.

[0091] S7: based on the maintenance environment, determining the maintenance scheme of the information system through a multi-agent deep reinforcement learning algorithm.

[0092] wherein the deep reinforcement learning algorithm is a method that combines deep neural networks and reinforcement learning, which can learn the optimal strategy in a complex, high-dimensional state space and action space. The maintenance scheme refers to the optimal action sequence learned, which includes specific repair measures taken by different agents in different states.

[0093] It should be noted that first, the multi-agent mechanism can decompose complex system problems, each agent makes independent decisions according to its own component state, and realizes global optimization through cooperation, avoiding the bottleneck of centralized control. Second, deep reinforcement learning has strong nonlinear modeling capability, can efficiently search in high-dimensional state and action space, and makes the scheme more consistent with the dynamic characteristics of complex systems. Third, the algorithm realizes policy iteration update through continuous trial and error and experience replay, which not only adapts to environmental changes, but also continuously improves decision quality. Finally, the maintenance scheme output by S7 takes into account economy and reliability, providing intelligent and data-driven guidance for actual operation and maintenance. Overall, this step significantly improves the scientificity, adaptability and collaboration of the maintenance strategy, and is the core link of the whole method to realize intelligent operation and maintenance.

[0094] In a possible implementation, S7 specifically includes:

[0095] S701: Initialize the number of training steps, experience replay buffer, policy network, target network, and exploration constant.

[0096] The number of training steps is the total number of model iteration updates in the reinforcement learning process, used to control the training scale. The experience replay buffer is a collection of state, action, reward, and next state samples generated by the agent interacting with the environment, used to break sample correlation and improve training efficiency. The policy network is a neural network that approximates the value of the current decision and outputs the optimal action selection. The target network is a delayed copy of the policy network, used to calculate the target value to avoid instability caused by rapid value updates during training. The exploration constant is a parameter that controls the balance between random exploration and the use of existing knowledge, ensuring that the model can try new schemes and also use known good strategies.

[0097] In a possible implementation, the policy network and the target network both adopt a duel structure, and the initial weights of the target network are the same as the initial weights of the target network.

[0098] The duel structure is an improved neural network structure that estimates "state value" and "action advantage" separately and then combines them to better determine which states are important, even if the specific action selection differs little in some states.

[0099] It should be noted that the branch duel structure can decouple state value and action advantage, so that the network can better evaluate the importance of different states, thereby improving the accuracy and efficiency of action selection in a high-dimensional action space. In this way, even if the action difference is not obvious in some states, the network can still learn which states are more critical to long-term returns. At the same time, the parameters of the policy network and the target network are consistent at the initial stage of training, which can avoid the estimation bias problem in the early learning stage, reduce training shock, and speed up the convergence speed. Overall, this design not only improves the stability and robustness of the reinforcement learning model, but also speeds up policy convergence, and finally generates a more optimal and reliable maintenance scheme.

[0100] S702: uniformly sampling a scalar according to the information system state.

[0101] Specifically, uniformly sampling the scalar from [0, 1]

[0102] S703: determining whether the scalar is less than an exploration constant. If yes, randomly selecting an action from the action space to maintain the information system. Otherwise, determining the optimal maintenance action of the information system through the policy network:

[0103]

[0104] wherein argmax represents maximization, Q tot () represents the decomposed global value function, Q tot ( s , a ) represents the global state s , the long-term value (expectation of future cumulative rewards) of taking joint action a , A joint represents the action space, represents the individual optimal action of the i-th agent, i.e., in the global state e , the i-th agent finds the individual action s , e that maximizes the local value function Ae in its individual action space , , e =1,2,…, E , E represents the total number of components in the information system, i.e., the total number of agents.

[0105] In one possible implementation, S703 specifically comprises: determining the optimal maintenance action of the information system through a value decomposition network (VDN) algorithm.

[0106] Among them, the value decomposition network (VDN, Value Decomposition Network) algorithm is a multi-agent reinforcement learning method, which decomposes the global value function into the sum of individual value functions of multiple agents. Through this decomposition, each agent can learn independently and optimize the global goal together. The optimal maintenance action refers to the maintenance decision that can minimize the long-term operation cost and maximize the benefit of the information system under the current state, which may include operations such as restarting services, replacing components, or adjusting configurations.

[0107] It should be noted that the advantage of using the VDN algorithm to determine the optimal maintenance action is that it decomposes the global value function into the sum of individual value functions of each agent, which converts the complex multi-agent collaboration problem into individual problems that can be solved in parallel. This not only reduces the computational difficulty and makes the training process more efficient, but also ensures the consistency between the global optimal goal and individual learning, avoiding the conflict between local decision and overall optimization. In the information system maintenance scenario, the state and operation of each component are independent and interact with each other, and VDN can fully utilize this structural feature to achieve collaborative optimization among multiple components. The final maintenance action not only can quickly adapt to different fault scenarios, but also can reduce costs and improve reliability on a global level.

[0108] In one possible implementation, S703 specifically includes:

[0109] S7031: decompose the information system into multiple components that can be independently maintained, wherein the maintenance action of each component is determined by an agent.

[0110] Among them, the agent is an autonomous decision unit in reinforcement learning, and each agent corresponds to a component, responsible for selecting the maintenance action of the component.

[0111] S7032: determine the global value function by the value decomposition network (VDN) algorithm:

[0112]

[0113] Among them, represents the joint action value function, and D represents the expectation operation. R j represents the immediate reward at step j , and γ represents the discount factor. S j represents the state at step j . A j represents the action at step j .

[0114] Among them, the value decomposition network (VDN) algorithm is a multi-agent reinforcement learning method, which decomposes the global value function into the sum of individual value functions, realizes the consistency of distributed learning and global optimization.

[0115] S7033: decompose the global value function into individual value functions of all agents:

[0116]

[0117] Among them, Q (s, a) represents the action value function of the i-th agent. e

[0118] S7044: determine the maintenance scheme of the information system by maximizing the sum of individual value functions of each agent.

[0119] It should be noted that first, the information system is disassembled into multiple independent components, making the complex problem modularized, facilitating multi-agent parallel processing. Secondly, VDN decomposes the global value function into the sum of individual value functions, enabling each agent to learn independently while ensuring the consistency of local decision-making and global goals. Thirdly, this decomposition reduces the complexity of global computation and improves the efficiency of training and reasoning. Finally, by maximizing the sum of individual value functions, the overall optimal maintenance scheme can be generated, ensuring global optimization while considering individual optimization, improving the reliability and economy of the scheme. Overall, this method not only enhances the coordination of multi-agent collaboration, but also improves the scalability and intelligence level of system operation and maintenance.

[0120] S704: execute the optimal maintenance action and record the state of the information system after executing the optimal maintenance action.

[0121] S705: calculate the reward of the information system according to the state of the information system after executing the optimal maintenance action and the state of the information system.

[0122] S706: based on the rules of each component in the information system, naturally degrade the information system to the next state.

[0123] S707: store the information system state, optimal maintenance action, reward and next state to the experience replay buffer as a state transition sample to update the experience replay buffer.

[0124] S708: uniformly select a batch of samples from the updated experience replay buffer.

[0125] S709: calculate the target value of each sample in the batch of samples:

[0126]

[0127] ​wherein, y j denotes the target value of the i-th step, j r j denotes the immediate reward of the i-th step, j denotes the target global Q function, denotes the joint action that maximizes the global Q function in the state .

[0128] S710: determining a target function of the policy network according to each target value and an estimated mean square error of the global value function.

[0129] S711: adjusting the weight of the policy network to a target value of minimizing the value of the target function.

[0130] S712: updating the target network according to the adjusted weight of the policy network.

[0131] S713: repeating steps S702-S712 until a maximum iteration number is reached, and outputting the trained policy network.

[0132] Specifically, the maximum iteration number is the training step number.

[0133] S714: determining a maintenance scheme of the information system according to the trained policy network.

[0134] It should be noted that, first, the introduction of the experience replay buffer and the target network effectively improves the stability and convergence of the training, avoiding the overfitting risk caused by sample correlation. Second, by designing the exploration constant, a balance is achieved between random exploration and optimal action selection, enabling the model to continuously discover potential better maintenance strategies. Third, by using the decomposition mechanism of the global value function and the individual value function, the multi-agent can both make independent decisions and achieve global collaborative optimization. Finally, through the weight update driven by the error and the cyclic iteration, the policy network can gradually converge to the optimal solution, thereby outputting a low-cost and high-reliability maintenance scheme. Overall, this process guarantees the scientificity, adaptability and optimality of the maintenance scheme, and is the core mechanism for realizing intelligent maintenance.

[0135] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0136] ​​In the embodiment of the present application, according to the collected state data and historical maintenance records of the information system, time series features are constructed, and the time series features are input into the NARX-AD model to determine whether the information system fails, the NARX-AD model can not only predict the behavior of the system, but also adaptively adjust according to the data characteristics to ensure that the modeling result is highly consistent with the real system, then, according to the fault type of the information system, the maintenance cost of the information system is predicted through the pre-trained artificial neural network, taking minimizing the maintenance cost as the goal, referring to the multi-dimensional Markov decision process, the maintenance environment of the information system is constructed through the definition of state space, action space and transition function, based on the maintenance environment, the maintenance scheme of the information system is determined through the multi-agent deep reinforcement learning algorithm, the optimal or approximate optimal maintenance scheme is generated through the parallel decision and collaborative ability of the multi-agent, compared with the traditional single-agent method, it is more suitable for the distributed characteristics of complex systems, finally, the information system is repaired according to the maintenance scheme, which improves the automation level, reduces the artificial maintenance cost, improves the accuracy and efficiency of the maintenance, shortens the repair time, further reduces the downtime cost, improves the availability and reliability of the system.

[0137] Reference is made to the accompanying drawings that show a structural schematic diagram of an information system maintenance scheme generation system based on multi-agent collaboration provided by an embodiment of the present application. Figure 2 , shows a structural schematic diagram of an information system maintenance scheme generation system based on multi-agent collaboration provided by an embodiment of the present application.

[0138] An information system maintenance scheme generation system based on multi-agent collaboration provided by an embodiment of the present application comprises a processor 201 and a memory 202.

[0139] The memory 202 stores programs or instructions executable on the processor 201, and the programs or instructions are executed by the processor 201 to implement the steps of the information system maintenance scheme generation method based on multi-agent collaboration described above, and achieve the same technical effects. To avoid repetition, the present application will not be described again.

[0140] It should be understood that the processor 201 in the embodiment of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready-to-program gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0141] It is also to be understood that the memory 202 in embodiments of the application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. Nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically EPROM (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as external cache memory. By way of example, and not limitation, many forms of random access memory can be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double-data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DR RAM).

[0142] The above-described embodiments can be implemented in part or in whole through software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When loaded and executed by a computer, the computer instructions or computer programs can generate the flow or function according to the embodiments of the present application in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, such as from one website site, computer, server, or data center to another website site, computer, server, or data center through a wired (e.g., infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing a set of one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0143] It should be understood that the size of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0144] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses, and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0146] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0147] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0148] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0149] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts of the technical solutions that make contributions to the prior art, or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0150] The embodiments of the present application provide a readable storage medium, which includes: a program or instructions stored on the readable storage medium, the program or instructions are executed by a processor to implement the steps of the information system maintenance scheme generation method based on multi-agent cooperation described above, and the same technical effects can be achieved. To avoid repetition, the present application will not be described again.

[0151] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application.

Claims

1. A method for generating maintenance plans for information systems based on multi-agent collaboration, characterized in that, include: S1: Collect status data and historical maintenance records of various components in the information system; S2: Construct time-series features based on the status data and the historical maintenance records; S3: Construct the NARX-AD model; S4: Based on the time-series characteristics, determine whether the information system has malfunctioned using the NARX-AD model; if yes, proceed to step S5; otherwise, continue monitoring. S5: Based on the fault type of the information system, determine the maintenance cost of the information system through a pre-trained artificial neural network; S6: With the goal of minimizing the maintenance cost, construct the maintenance environment of the information system by referring to the multidimensional Markov decision process; S6 specifically includes: S601: Define the state space of the information system based on the state data of each component in the information system; S602: Define the action space of the information system based on the maintenance actions of each component in the information system; S603: Define the state transition function of the information system based on the state transition matrix of each component in the information system; S604: Define the reward function of the information system based on the maintenance cost of the information system; S605: Define the discount factor; S606: Determine the maintenance environment of the information system based on the state space, the action space, the state transition function, the reward function, and the discount factor; S7: Based on the maintenance environment, determine the maintenance plan for the information system using a multi-agent deep reinforcement learning algorithm; S8: Repair the information system according to the maintenance plan.

2. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, The status data includes CPU utilization, memory usage, and log data.

3. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, S4 specifically includes: S401: Input the time-series features into the NARX-AD model; S402: Determine the fault detection signal of the information system by calculating the residual between the predicted value and the actual value of the NARX-AD model; S403: Eliminate the correlation of the fault detection signal through whitening treatment; S404: Smooths the whitened fault detection signal using an exponentially weighted moving average filter; S405: Calculate the upper and lower control limits of the control chart based on the statistical characteristics of the fault detection signal; S406: Determine whether the smoothed fault detection signal is greater than the upper control limit or less than the lower control limit; if so, determine that the information system has malfunctioned and issue an alarm, proceed to step S5; otherwise, continue monitoring.

4. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, S5 specifically includes: S501: Determine the status of the information system based on the fault type of the information system; S502: Based on the fault type of the information system, the information system is repaired using different maintenance methods to obtain the status of the information system after maintenance; S503: Input the information system status and the information system status after maintenance into the pre-trained artificial neural network, and output the maintenance cost of the information system.

5. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 1, characterized in that, Specifically, S7 includes: S701: Initialize training steps, experience replay buffer, policy network, target network, and exploration constant; S702: Based on the state of the information system, perform uniform sampling of the scalar; S703: Determine whether the scalar is less than the exploration constant; if so, randomly select an action from the action space to repair the information system; otherwise, determine the optimal repair action for the information system through the policy network. S704: Execute the optimal maintenance action and record the information system status after executing the optimal maintenance action; S705: Calculate the reward for the information system based on the information system status after performing the optimal maintenance action and the information system status; S706: Based on the patterns of each component in the information system, the information system is naturally degraded to the next state; S707: Store the information system status, the optimal maintenance action, the reward, and the next status as state transition samples in the experience replay buffer to update the experience replay buffer; S708: Select a batch of samples evenly from the updated experience replay buffer; S709: Calculate the target value of each sample in the batch sample; S710: Determine the objective function of the policy network based on each of the stated objective values ​​and the mean square error of the estimated global value function; S711: To minimize the value of the objective function, the weights of the policy network are adjusted; S712: Update the target network according to the weights of the adjusted policy network; S713: Repeat steps S702-S712 until the maximum number of iterations is reached, and output the trained policy network. S714: Determine the maintenance plan for the information system based on the trained policy network.

6. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 5, characterized in that, Both the policy network and the target network adopt a branch-and-fight structure, and the initial weights of the target network are the same as those of the target network.

7. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 5, characterized in that, The specific steps in S703, where the optimal maintenance action for the information system is determined through the policy network, are as follows: The optimal maintenance action for the information system is determined using a value decomposition network algorithm.

8. The method for generating maintenance plans for information systems based on multi-agent collaboration according to claim 7, characterized in that, The S703 specifically includes: S7031: The information system is disassembled into multiple independent components to be inspected, wherein the inspection actions of each component to be inspected are determined by an intelligent agent; S7032: Determine the global value function using the value decomposition network algorithm; S7033: Decompose the global value function into individual value functions for all agents; S7034: Determine the maintenance plan for the information system with the goal of maximizing the sum of the individual value functions of each intelligent agent.

9. A system for generating maintenance plans for information systems based on multi-agent collaboration, characterized in that, include: Processor and memory; The memory stores programs or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the method for generating maintenance schemes for information systems based on multi-agent collaboration as described in any one of claims 1 to 8.

Images