Data center fault diagnosis and repair method based on multi-agent collaborative reasoning

Abstract

The invention relates to a data center fault diagnosis and repair method based on multi-agent collaborative reasoning, which effectively overcomes the problems of insufficient knowledge depth and illusion of a single model through a multi-agent collaborative reasoning mechanism, and realizes multi-view cross validation and accurate root cause positioning. Digital twinborn simulation and negative feedback iterative optimization are introduced, so that the secondary fault risk caused by direct execution of a repair script is solved, and the scheme is ensured to be safe and reliable; the understanding capability of a large model on heterogeneous data is improved through multi-modal data semantic mapping; a hierarchical execution and real-time fusing rollback system is constructed, and the hidden danger of service interruption in the automation process is eliminated, so that the transformation from manual assistance to safety autonomy is realized, and the accuracy, efficiency and reliability of operation and maintenance of the data center are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence, and in particular relates to a data center fault diagnosis and repair method based on multi-agent collaborative reasoning. Background Technology

[0002] With the rapid development of cloud computing, 5G, the Internet of Things, and artificial intelligence, data centers, as the core infrastructure supporting massive computing and storage, have seen a dramatic increase in scale and complexity. The number of hardware components such as servers, storage arrays, and network devices has multiplied, accompanied by massive amounts of multimodal monitoring data (time-series metrics, system logs, topology relationships) and highly coupled system dependencies. Traditional data center operations and maintenance primarily rely on the experience and manual operation of senior experts, resulting in slow response times, difficulties in knowledge transfer, and limitations in scalability. While rule-based automated systems have improved logical clarity, their rule bases suffer from high maintenance costs, delayed updates, and difficulty in covering complex and ever-changing fault scenarios, leading to insufficient system flexibility.

[0003] In recent years, breakthroughs in Large Language Modeling (LLM) technology have brought new opportunities to intelligent data center operations and maintenance. Existing solutions often utilize LLM combined with vector databases to build operations and maintenance support systems. By retrieving equipment parameters, historical cases, and operational knowledge, these systems provide fault query and repair suggestions to operations and maintenance personnel, significantly improving information processing efficiency. However, such solutions are essentially still "open-loop" support tools. The generated diagnostic conclusions and repair scripts require final human judgment, verification, and manual execution, failing to achieve true automated closed-loop.

[0004] Furthermore, single general-purpose models lack sufficient depth of knowledge in vertical domains, which can easily lead to "machine illusions" and limit diagnostic accuracy; they have weak semantic understanding capabilities for multimodal heterogeneous data, making it difficult to accurately locate deep-seated root causes; the direct application of repair scripts to the production environment lacks safety simulation verification, posing a risk of causing secondary failures; and no effective real-time feedback and risk prevention mechanisms have been established during execution, resulting in a high risk of business interruption. Summary of the Invention

[0005] The purpose of this invention is to provide a data center fault diagnosis and repair method based on multi-agent collaborative reasoning, in order to solve the problems in the prior art such as the separation of decision-making and execution, the susceptibility of a single large model to illusion, the lack of pre-execution security simulation verification, and the insufficient understanding of multimodal data.

[0006] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a data center fault diagnosis and repair method based on multi-agent cooperative reasoning, the method comprising:

[0007] Real-time collection of time-series monitoring metrics and system log data, and conversion into serialized environment state vectors;

[0008] Based on the environmental state vector, initial hypotheses are generated in parallel by multiple vertical domain agents, and the optimal root cause conclusion is output by the decision agent through multiple rounds of cross-verification.

[0009] Atomic operation sequences are generated based on the optimal root cause conclusion, and simulation verification and iterative optimization are performed in a digital twin environment to obtain the optimal repair scheme that has passed verification.

[0010] As a further improvement to one embodiment of the present invention, the method further includes, in which the real-time collection of time-series monitoring indicators and system log data through infrastructure probes and the conversion into a serialized environmental state vector, the following steps are taken:

[0011] Savitzky-Golay convolution is used to smooth the signal and solve the feature for the time-series monitoring index. First-order trend features and second-order fluctuation features are extracted and dynamic semantic labeling is performed through Z-Score dynamic threshold determination to form the semantic enhancement features of the time-series monitoring index.

[0012] Based on the aforementioned semantic enhancement features, the system logs are processed using online template extraction and variable regularization masking based on weighted edit distance to generate a standardized event set, thereby achieving semantic alignment between logs and metrics.

[0013] Based on the standardized event set, TF-IDF weights are calculated, and after pruning and sorting, a composite environment state vector is constructed.

[0014] As a further improvement to one embodiment of the present invention, the method further includes the step of generating initial hypotheses in parallel by multiple vertical domain agents, and outputting the optimal root cause conclusion by a decision agent through multiple rounds of cross-validation.

[0015] The multiple vertical domain agents generate initial hypotheses based on the composite environment state vector; the initial hypotheses include a unique hypothesis identifier, a source agent identifier, a root cause code, a chain of evidence, and an initial confidence level, to form a structured multi-dimensional set of fault hypotheses;

[0016] The decision-making agent iteratively updates the confidence level of each hypothesis through multiple rounds of cross-verification. The confidence level update formula is as follows:

[0017]

[0018] in, For the first Assumptions in round iteration confidence level For the learning rate step size, To determine the domain authority weight of agent j, Indicates feedback polarity, Represents the strength of evidence;

[0019] Based on the updated confidence level, convergence is determined and the optimal root cause conclusion is output when the confidence level meets the preset convergence condition.

[0020] As a further improvement to one embodiment of the present invention, the method further includes, in addition to, the step of the decision agent converging to output the optimal root cause conclusion through multiple rounds of cross-verification, further comprising,

[0021] The formula for calculating the completeness of evidence score is as follows:

[0022]

[0023] in, This represents the actual number of features matched. The standard indicator base number for this fault type defined in the knowledge base;

[0024] Based on the evidence completeness score, if the score is lower than a preset threshold, the risk of insufficient evidence is marked in the root cause analysis report, and the marked report is integrated into the optimal root cause conclusion.

[0025] As a further improvement to one embodiment of the present invention, the method further includes generating an atomic operation sequence based on the root cause conclusion and performing simulation verification and iterative optimization in a digital twin environment, which includes...

[0026] Based on the optimal root cause conclusion that integrates the risk markers for insufficient evidence, a remediation scheme consisting of a sequence of atomic operations is generated; each atomic operation in the sequence of atomic operations is bound to a corresponding rollback instruction and a verification metric.

[0027] After implementing the remediation plan on the digital twin instance, a feasibility score is calculated using the following formula:

[0028]

[0029] in, As a repair solution, The system state observed after performing a sequence of atomic operations in a digital twin environment. To restore the weight, Indicates the degree of service restoration. Risk weighting This indicates the risk value of system crash or secondary failure. As a cost weight, Indicates the execution time cost;

[0030] Based on the feasibility score, when the score is lower than a preset safety threshold, a difference diagnosis report is generated as a negative feedback signal to trigger iterative optimization of the repair scheme until an optimal repair scheme that satisfies all safety constraints and passes verification is obtained. .

[0031] As a further improvement to one embodiment of the present invention, the method further includes, performing the verified optimal repair solution in stages in a physical production environment, specifically including,

[0032] Based on topological betweenness centrality, low-importance nodes are selected as pilot batches to test and implement the optimal repair scheme. And within the high-frequency observation window, the relative deviation of key performance indicators is calculated in real time, using the following formula:

[0033]

[0034] in, Let j be the value of the j-th dimension of the real-time state vector. The value of the j-th dimension of the baseline state vector. To prevent smooth terms with a denominator of zero;

[0035] Based on the relative deviation, dynamic circuit breaker detection is performed to control execution risks, and the detection results are used as a quantitative basis for subsequent branch switching.

[0036] As a further improvement to one embodiment of the present invention, the method further includes dynamic circuit breaker detection through real-time indicator monitoring to achieve circuit breaker or rolling release, specifically including,

[0037] When any metric deviates beyond a preset hard constraint threshold or a high-risk log is generated, a hard interrupt is triggered and a pre-bound atomic rollback instruction is executed to restore the system state to the baseline level and prevent the risk from spreading.

[0038] Based on the execution result of the atomic rollback instruction, when all indicators deviate below the preset safety threshold, an exponential step strategy is adopted for rolling release until all affected nodes are covered, thereby achieving a smooth recovery of business services.

[0039] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a data center fault diagnosis and repair system based on multi-agent collaborative reasoning, the system including a semantic mapping module, a root cause analysis module, and a repair verification module;

[0040] The semantic mapping module is used to collect time-series monitoring indicators and system log data in real time and convert them into serialized environment state vectors.

[0041] The root cause analysis module is used to generate initial hypotheses in parallel by multiple vertical domain agents based on environmental state vectors, and the decision agent outputs the optimal root cause conclusion through multiple rounds of cross-verification.

[0042] The repair verification module is used to generate an atomic operation sequence based on the optimal root cause conclusion, and to perform simulation verification and iterative optimization in a digital twin environment to obtain the optimal repair scheme that has passed verification.

[0043] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the data center fault diagnosis and repair method based on multi-agent cooperative reasoning as described above.

[0044] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps in the data center fault diagnosis and repair method based on multi-agent cooperative reasoning as described above.

[0045] Compared with existing technologies, this invention provides a data center fault diagnosis and repair method based on multi-agent collaborative reasoning. This method effectively overcomes the limitations of single-model knowledge depth and the illusion problem through a multi-agent collaborative reasoning mechanism, achieving multi-perspective cross-validation and accurate root cause localization. It introduces digital twin simulation and negative feedback iterative optimization to solve the secondary fault risk caused by direct execution of repair scripts, ensuring the safety and reliability of the solution. Multimodal data semantic mapping enhances the large model's understanding of heterogeneous data. The construction of a hierarchical execution and real-time circuit breaker rollback system eliminates the risk of business interruption during automation, thereby achieving a shift from manual assistance to secure autonomy and significantly improving the accuracy, efficiency, and reliability of data center operation and maintenance. Attached Figure Description

[0046] Figure 1 This is an overall flowchart of the data center fault diagnosis and repair method based on multi-agent collaborative reasoning described in this invention.

[0047] Figure 2 This is a schematic diagram of the multi-agent collaboration and cross-verification process of the data center fault diagnosis and repair method based on multi-agent collaborative reasoning described in this invention.

[0048] Figure 3 This is a schematic diagram of the repair strategy generation and simulation verification process based on digital twin feedback closed loop in the data center fault diagnosis and repair method based on multi-agent collaborative reasoning described in this invention.

[0049] Figure 4This is a schematic diagram of the repair strategy generation and simulation verification process based on digital twin feedback closed loop in the data center fault diagnosis and repair method based on multi-agent collaborative reasoning described in this invention.

[0050] Figure 5 This is a schematic diagram of the architecture of the data center fault diagnosis and repair system based on multi-agent collaborative reasoning as described in this invention.

[0051] Figure 6 This is a detailed system architecture diagram of the data center fault diagnosis and repair system based on multi-agent collaborative reasoning described in this invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0053] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0054] In Embodiment 1 of the present invention, the present invention provides a data center fault diagnosis and repair method based on multi-agent cooperative reasoning, such as... Figure 1 As shown, the method includes,

[0055] S1: Real-time collection of time-series monitoring metrics and system log data, and conversion into serialized environment state vectors;

[0056] S2: Based on the environmental state vector, initial hypotheses are generated in parallel by multiple vertical domain agents, and the optimal root cause conclusion is output by the decision agent through multiple rounds of cross-verification.

[0057] S3: Generate an atomic operation sequence based on the optimal root cause conclusion, and perform simulation verification and iterative optimization in a digital twin environment to obtain the optimal repair solution that has passed verification.

[0058] In one specific embodiment of the present invention, time-series monitoring indicators and system log data are collected in real time through infrastructure probes and converted into serialized environmental state vectors. Specifically,

[0059] Savitzky-Golay convolution is used to smooth the signal and solve the feature for the time-series monitoring index. First-order trend features and second-order fluctuation features are extracted and dynamic semantic labeling is performed through Z-Score dynamic threshold determination to form the semantic enhancement features of the time-series monitoring index.

[0060] Based on the aforementioned semantic enhancement features, the system logs are processed using online template extraction and variable regularization masking based on weighted edit distance to generate a standardized event set, thereby achieving semantic alignment between logs and metrics.

[0061] Based on the standardized event set, TF-IDF weights are calculated, and after pruning and sorting, a composite environment state vector is constructed.

[0062] It should be noted that by collecting time-series monitoring indicators and system log data in real time through infrastructure probes and converting them into serialized environment state vectors, the aim is to address the technical challenges of high noise, strong heterogeneity, and difficulty in direct understanding by large language models in multimodal raw data from data centers. This step transforms the underlying, cold numerical signals and unstructured text into semantically rich, denoised, and key-highlighted standardized environment state vectors. This provides a unified and high-value contextual input for subsequent multi-agent collaborative reasoning, ensuring that all expert agents conduct independent deductions and cross-debates based on the same comprehensive set of facts, thereby significantly improving the accuracy and consistency of diagnosis.

[0063] Furthermore, Savitzky-Golay convolution is used to smooth the signal and calculate features for the time-series monitoring indicators. First-order trend features and second-order fluctuation features are extracted, and dynamic semantic labeling is performed using Z-Score dynamic thresholding to form semantically enhanced features for the time-series monitoring indicators. The purpose of this sub-step is to effectively suppress high-frequency noise while preserving the key form of the original signal, and to transform continuous values ​​into discrete semantic labels (such as "exponential surge", "high-frequency oscillation", "continuous saturation"), realizing the leap from pure numerical values ​​to interpretable knowledge and laying a semantic foundation for subsequent multimodal fusion.

[0064] In a specific embodiment of the present invention, the steps of performing signal smoothing and feature decomposition are as follows:

[0065] Constructing a sliding observation window: The system sets a sliding window of length 2m+1 (e.g., m=5, window length is 11), places the data point at the current time t in the center of the window, and extracts a local data subset S. local ={x t-m ,...,x t ,...,x t+m}

[0066] Perform local polynomial fitting: The system selects a polynomial p(n) = Σc of order k (usually k = 2 or 3). j ×n j The system fits the data points within the window, where n is the relative position of the data point with respect to the center of the window (-m ≤ n ≤ m). The system solves for the polynomial coefficient set {c0, c...} using the least squares method. 1, ...,c k This minimizes the sum of squared fitting errors.

[0067] Feature extraction through convolution calculation: The system uses a pre-calculated Savitzky-Golay convolution coefficient table to directly perform linear weighted summation on the data within the window, outputting feature values ​​in three dimensions at once.

[0068] Smoothing value calculation: The value of the fitted polynomial at the center point (n=0) is used as the smoothed value after denoising, i.e. This operation removes high-frequency random noise while preserving the peak height and width characteristics of the original signal to the greatest extent possible.

[0069] Solution of first-order trend characteristic (λ1): Directly calculate the first derivative of the center point using the coefficients of the linear term, i.e. This value accurately represents the instantaneous rate of change of the indicator at the current moment.

[0070] Solution of the second-order wave characteristic (λ2): The second derivative at the center point is calculated directly using the coefficients of the quadratic term, i.e. This value precisely characterizes the acceleration of the index change or the convexity / concavity of the curve (i.e., whether there are violent oscillations).

[0071] Dynamic semantic labeling: After completing the full sequence traversal calculation, the system introduces a Z-Score dynamic threshold determination mechanism. The system has a built-in multi-dimensional semantic mapping dictionary, and labeling is triggered only when the feature value exceeds μ±3σ.

[0072] If λ1>3σ and λ2>0, it is mapped to "Exponential Surge", which represents the risk of resources running out of control.

[0073] If λ1≈0 but the standard deviation σ increases significantly, it is mapped to "High-Freq Oscillation", which indicates that there may be resource contention or allocation jitter.

[0074] If the value remains above 99% for a long period of time, it is mapped to "saturated".

[0075] Furthermore, based on the aforementioned semantic enhancement features, online template extraction and variable regularization masking based on weighted edit distance are applied to the system logs to generate standardized event sets, thereby achieving semantic alignment between logs and metrics. This sub-step aims to eliminate interference from variable noise (such as IP addresses, timestamps, and session IDs) in the logs, compressing massive amounts of unstructured logs into a limited set of core event templates, along with frequency and severity information. This allows discrete log events and time-series metrics to be semantically aligned, forming mutually corroborating multi-dimensional fault indicators.

[0076] In a specific implementation scenario of this invention, online template extraction and variable regularization masking based on weighted edit distance are performed: For unstructured system logs filled with variable noise, the system performs structured feature extraction.

[0077] Variable regular expression mask: The raw log text stream Tlog is pre-scanned using a pre-built regular expression library to automatically identify non-fixed entities such as IP addresses, timestamps, and UUIDs, and then uniformly replaces them with generic placeholders (e.g., ...).<IP_ADDR> ,<SESSION_ID> ), to eliminate interference noise.

[0078] Streaming clustering extraction: An online clustering algorithm based on prefix tree or weighted edit distance is introduced. The system calculates the similarity of the segmented logs. If the similarity is higher than a preset threshold (e.g., 0.8), it is merged into an existing template and its trigger frequency f is updated. freq If the time distribution entropy is not equal to the time distribution entropy, then create a new event template Ttemplate.

[0079] Output standardization: Finally, massive discrete text is compressed into a set of standardized events containing "core semantic templates", "burst frequency" and "severity level" as fault indicator symbols.

[0080] Furthermore, after pruning and sorting the TF-IDF weights calculated based on the standardized event set, a composite environment state vector is constructed. This aims to filter out the few key pieces of information with the most specificity and diagnostic value in the current fault scenario from massive semantic features, eliminate normal background noise, and avoid the large language model context window being occupied by invalid data. Finally, a composite environment state vector containing local dependency subgraphs, key indicator semantic sets, high-risk log template sets, and metadata is constructed and serialized into JSON format prompt words. This composite vector serves as the sole common input for the subsequent multi-agent collaborative root cause analysis module, ensuring that all vertical domain expert agents and decision agents perform hypothesis generation, cross-questioning, and confidence iteration based on the same refined and consistent facts. This effectively avoids diagnostic discrepancies caused by input bias and provides a solid guarantee for the reliable convergence of root cause conclusions.

[0081] In a specific implementation scenario of this invention, multimodal feature vectorization concatenation and context construction are performed: In order to provide the highest value information within a limited large model context window, the system prioritizes the aforementioned semantic features:

[0082] Feature pruning and sorting: The system calculates the TF-IDF weight of each semantic label and log template, and prioritizes retaining features that are rare and highly specific at the current fault time (such as sudden "OOM Error"), while automatically removing normal background noise (such as timed "Health Check INFO").

[0083] State vector construction: Define the environment state vector X t For a composite data object: X t = <G sub Lmetrics topK ,Ttemplate topK ,Ω meta >. Among them G sub Lmetrics represents the fault local dependency subgraph. topK and Ttemplate topK These are the key indicator semantic set and the high-risk log template set, respectively, after weighted filtering.

[0084] JSON serialization: the vector X t Ultimately, it is serialized into a standard JSON-formatted prompt context, serving as the common input for all expert agents in subsequent steps. This ensures that all agents conduct subsequent root cause reasoning and debate based on the same semantically enhanced, denoised, and highlighted "panoramic medical record."

[0085] In one specific embodiment of the present invention, multiple vertical domain agents generate initial hypotheses in parallel, and a decision agent converges and outputs the optimal root cause conclusion through multiple rounds of cross-validation. Specifically,

[0086] The multiple vertical domain agents generate initial hypotheses based on the composite environment state vector; the initial hypotheses include a unique hypothesis identifier, a source agent identifier, a root cause code, a chain of evidence, and an initial confidence level, to form a structured multi-dimensional set of fault hypotheses;

[0087] The decision-making agent iteratively updates the confidence level of each hypothesis through multiple rounds of cross-verification. The confidence level update formula is as follows:

[0088]

[0089] in, For the first Assumptions in round iteration confidence level For the learning rate step size, To determine the domain authority weight of agent j, Indicates feedback polarity, Represents the strength of evidence;

[0090] Based on the updated confidence level, convergence is determined and the optimal root cause conclusion is output when the confidence level meets the preset convergence condition.

[0091] The formula for calculating the completeness of evidence score is as follows:

[0092]

[0093] in, This represents the actual number of features matched. The standard indicator base number for this fault type defined in the knowledge base;

[0094] Based on the evidence completeness score, if the score is lower than a preset threshold, the risk of insufficient evidence is marked in the root cause analysis report, and the marked report is integrated into the optimal root cause conclusion.

[0095] It should be noted that the initial hypotheses are generated in parallel by multiple vertical domain agents, and the optimal root cause conclusion is output by the decision agent through multiple rounds of cross-verification. This aims to overcome the shortcomings of a single general-purpose large language model, such as insufficient knowledge depth in vertical domains and susceptibility to "machine illusion." By simulating a multi-perspective collaborative debate mechanism of human expert groups, the objectivity, reliability, and sufficiency of evidence in the diagnostic conclusions are ensured. This step directly takes the composite environment state vector generated by the aforementioned multimodal data perception and semantic mapping steps as the sole input, and transmits the output optimal root cause conclusion and its structured report to the subsequent digital twin repair verification step. This provides a solid factual basis for the generation of a safe repair solution, forming a close connection between the diagnostic, verification, and execution stages.

[0096] Simultaneously, an evidence completeness score is calculated and risk labeling is performed based on the score. This aims to prevent the problem of missing evidence chains due to incomplete observational data or rare failure scenarios, ensuring that the output root cause conclusions not only have high confidence but also include a quantitative assessment of evidence sufficiency. This provides a more comprehensive and reliable decision-making basis for downstream remediation strategy generation. This supplementary step closely follows the optimal root cause hypothesis after the aforementioned multi-round confidence iteration convergence, directly conducting a secondary quantitative review of the supporting evidence for this hypothesis, and integrating the assessment results into the final root cause analysis report. This extends the diagnostic process from "consensus convergence" to "evidence completeness assurance," while introducing risk awareness into the subsequent digital twin simulation verification steps, enabling conservative strategies for scenarios with weak evidence to be prioritized when generating remediation plans.

[0097] Furthermore, such as Figure 2 As shown, multiple vertical domain agents generate initial hypotheses based on the composite environment state vector. This leverages the specialized knowledge bases of expert agents in each vertical domain (such as network experts, hardware experts, application experts, and security experts) to independently interpret the same environment state vector from different technical dimensions, rapidly generating diverse sets of fault hypotheses. This parallel and independent reasoning approach can cover multiple possible paths to cross-domain faults. Furthermore, the structured field design facilitates subsequent global tracking, evidence comparison, and confidence quantification, providing a standardized data foundation for multi-round debates.

[0098] In a specific embodiment of the present invention, the system loads the hyperparameter configuration O for multi-agent runtime. conf The inference temperature coefficient T for the large language model is set to 0.3 (to ensure deterministic inference), and the maximum generation length L is set to... max =2048, and the kernel sampling parameter P used to control diversity. top =0.9. Each expert agent A i A specific system prompt template is injected, which includes role constraints, domain knowledge retrieval paths, and output format specifications.

[0099] Upon receiving the environmental state vector X t Subsequently, each agent generates initial fault hypotheses in parallel. To facilitate internal system analysis and transfer, the generated hypotheses H... i It is encapsulated as a structured data object containing the following five key fields:

[0100] Assume a unique identifier (Assumption_ID): a UUID used for global tracking (e.g., UUID-v4-Net-01).

[0101] Source Agent: The expert role ID (e.g., Agent_Network_Expert) that generated the hypothesis.

[0102] Root Cause Code: The standard error code corresponding to the fault type (such as RC_DB_POOL_EXHAUSTION).

[0103] Evidence Chain: A list containing multiple sets of feature indices, each set recording the indicator ID, observation value, and decision threshold (e.g., ...).<Metric:M_TCP_WAIT,Value:850,Threshold:200> ).

[0104] Initial Confidence: A normalized score calculated based on the model's output probability.

[0105] Furthermore, the decision-making agent iteratively updates the confidence level of each hypothesis through multiple rounds of cross-verification. This is achieved by introducing an independent decision-making role to preside over structured debates and by using a broadcast verification request and feedback response mechanism to realize cross-questioning and mutual verification among expert agents. This simulates the questioning and corroboration process in human expert meetings, thereby effectively eliminating cognitive biases from a single perspective and improving the robustness of the overall diagnosis.

[0106] In one specific implementation scenario of this invention, after completing the initial configuration and independent thought chain reasoning, the system enters a multi-round iterative mutual verification phase. Decision agent A judge Maintain a fully connected verification matrix and set the convergence step size parameter. Maximum number of iterations In each round of interaction, the system generates a standardized verification request message and broadcasts it to all nodes.

[0107] When each expert agent acts as a validator, it must output a feedback response object containing polarity and intensity. This object includes the following key fields:

[0108] Validator ID: The ID of the agent that performs the validation.

[0109] Target Hypothesis Identifier (Target_ID): The UUID of the hypothesis being validated.

[0110] Verification Polarity: Indicates the direction of the verification result, with values ​​of {1, 0, -1}, representing support, irrelevant, and refutation, respectively.

[0111] Strength of evidence: A quantitative value based on the confidence level of the verification data, with a value range of [0,1].

[0112] Supporting Metric: Key-value pairs containing specific supporting data (e.g., ...)<Disk_Queue_Depth:12.5> ).

[0113] Furthermore, based on the iteratively updated confidence level, convergence is determined and the optimal root cause conclusion is output when the confidence level meets the preset convergence condition. This is to monitor the dynamic convergence of the debate process, ensure that the most consensus-based root cause explanation is selected after the hypothesis space has been fully explored, and transform it into a standardized root cause analysis report, which is directly used as input for the generation of downstream remediation strategies.

[0114] Furthermore, calculating the evidence completeness score involves objectively quantifying the evidence coverage of the current diagnostic conclusion by comparing the number of actually observed fault features with the predefined standard indicator base of the root cause type in the knowledge base. This provides a supplementary evaluation dimension independent of confidence level, thereby avoiding the situation where the model's subjective probability is relied upon alone and the lack of objective data support is ignored.

[0115] Furthermore, when the score falls below a preset threshold, the root cause analysis report is marked with a risk of insufficient evidence, and this marked report is integrated into the optimal root cause conclusion. This proactively identifies potential uncertainties when evidence coverage is insufficient, ensuring that the final root cause conclusion carries clear warning information. This marked, complete report serves as direct input to the subsequent remediation strategy generation module, guiding the agent to adopt more cautious scheme design when formulating atomic operation sequences (such as increasing diagnostic probing operations or reducing the magnitude of changes). This, in conjunction with digital twin simulation verification and hierarchical execution mechanisms, further strengthens the risk control capabilities of the entire closed-loop system, ensuring that the stability of the production environment is maintained to the maximum extent even in cases of insufficient evidence.

[0116] In one specific embodiment of the invention, the system monitors the confidence changes of all active hypotheses in real time. An absolute convergence threshold is set. With relative convergence threshold The system determines that the debate has converged when any of the following conditions are met:

[0117] 1. A certain hypothesis confidence level ;

[0118] 2. Maximum confidence increment over two consecutive iterations .

[0119] Finally, the system outputs the optimal solution. This is then transformed into the final root cause analysis report (RCAReport).

[0120] In addition, if Even with a high confidence level, the system will mark "risk of insufficient evidence" in the report, thus ensuring the rigor of the diagnostic results through quantitative parameter control.

[0121] In one specific embodiment of the present invention, the step of generating an atomic operation sequence based on root cause conclusions and performing simulation verification and iterative optimization in a digital twin environment includes,

[0122] Based on the optimal root cause conclusion that integrates the risk markers for insufficient evidence, a remediation scheme consisting of a sequence of atomic operations is generated; each atomic operation in the sequence of atomic operations is bound to a corresponding rollback instruction and a verification metric.

[0123] After implementing the remediation plan on the digital twin instance, a feasibility score is calculated using the following formula:

[0124]

[0125] in, As a repair solution, The system state observed after performing a sequence of atomic operations in a digital twin environment. To restore the weight, Indicates the degree of service restoration. Risk weighting This indicates the risk value of system crash or secondary failure. As a cost weight, Indicates the execution time cost;

[0126] Based on the feasibility score, when the score is lower than a preset safety threshold, a difference diagnosis report is generated as a negative feedback signal to trigger iterative optimization of the repair scheme until an optimal repair scheme that satisfies all safety constraints and passes verification is obtained. .

[0127] It should be noted that generating atomic operation sequences based on root cause conclusions and performing simulation verification and iterative optimization in a digital twin environment aims to address the technical bottleneck of existing technologies where directly applying repair scripts to the production environment can easily lead to secondary failures. By constructing a high-fidelity digital twin sandbox as a "safety filter," it ensures that only solutions verified in terms of both logical correctness and resource security can enter the physical execution stage. This step directly takes the optimal root cause conclusion (including a complete report with insufficient evidence risk markers) output from the aforementioned multi-agent collaborative root cause analysis step as input and passes the finally verified optimal repair solution P' to the subsequent hierarchical automated execution steps, thereby establishing a rigorous closed-loop verification link between diagnosis and execution, avoiding business risks caused by blind operation.

[0128] Furthermore, such as Figure 3 As shown, based on the optimal root cause conclusion that integrates the insufficient evidence risk marker, a remediation scheme consisting of a sequence of atomic operations is generated. This is achieved by utilizing a large language model combined with an operational knowledge base to transform the root cause conclusion into a structured, machine-executable, ordered sequence of atomic operations. Simultaneously, each operation is pre-bound with a reverse rollback instruction and a single-step verification metric. This design not only realizes the transformation from natural language suggestions to deterministic code but also provides an atomically controllable foundation for subsequent simulation evaluation and physical rollback. Especially when the insufficient evidence risk marker exists, it can guide the generation of a more conservative operation sequence.

[0129] In one specific implementation scenario of the present invention, the decision-making agent confirms the root cause of the fault. Then, the system activates the adaptive strategy generation engine. This engine combines standard operating procedures from the operations and maintenance knowledge base with the current system state, and uses the code generation capabilities of a large language model to construct a structured repair solution P. Solution P is defined as an ordered sequence of atomic operations. Each atomic operation It not only includes specific execution instructions (such as shell scripts or SQL statements), but also strictly binds the corresponding rollback instructions. Compared with single-step verification metrics Taking the "database connection pool exhaustion" fault diagnosed in step two as an example, the initial solution generated by the agent includes two steps: the first step is "dynamically adjusting database parameters," with the instruction SETGLOBALmax_connections=2000, and the corresponding rollback instruction is SETGLOBALmax_connections=500 (original value); the second step is "restarting the application service to reconnect," with the instruction systemctl restartapp_service. To standardize the interaction, this solution is encapsulated into a standard object conforming to a predefined JSONSchema, containing the target device IP, execution timeout threshold (e.g., 30s), and expected recovery effect (e.g., "TCP connection count <80%)," achieving a deterministic transformation from natural language suggestions to machine-executable code.

[0130] Furthermore, after executing the repair scheme on the digital twin instance, a feasibility score is calculated. This involves non-destructively replaying the repair process in a virtual sandbox that is highly consistent with the production environment. By comparing system snapshots before and after the operation through differential state analysis, the effectiveness of the recovery scheme, potential risks, and execution costs are quantitatively evaluated, providing an objective basis for feasibility judgment.

[0131] In one specific embodiment of the present invention, the repair scheme P is injected into a digital twin simulation environment for non-destructive verification. This environment utilizes containerization technology and a virtual network mapping algorithm to pull configuration snapshots and topology structures from the production environment in real time, constructing a mirror instance of the physical system. The system is in The simulation engine executes the operation sequence P and uses Differential State Analysis (DSA) to evaluate its performance. It compares system state snapshots before and after execution and calculates the incremental changes ΔM in key performance indicators. During this process, the simulation engine focuses on monitoring two key metrics: fault recovery effectiveness (verifying whether key metrics, such as response latency, return to normal ranges) and operational safety (detecting whether the operation causes side effects). Continuing with the above example, after executing `max_connections=2000` on the virtual database, the simulation engine detected that the virtual machine's memory usage spiked from 60% to 99% within 5 seconds, triggering the OutOfMemoryError (OOM) protection mechanism and causing the database process to crash. In this case, due to the excessively high Risk factor, the total score J(P) was far below the safety threshold J_min, leading the system to determine that the scheme was high-risk and block it.

[0132] Furthermore, when the score is lower than the preset safety threshold, a difference diagnosis report is generated as a negative feedback signal to trigger the iterative optimization of the repair scheme. When the simulation finds that the side effects or risks are too high, a difference report that clearly points out the semantic gap between the expected and actual states is automatically generated and sent back to the policy generation engine to drive the agent to perform multi-dimensional iterations such as parameter tuning, operation reconstruction or constraint strengthening until the scheme fully meets the safety requirements.

[0133] In a specific implementation scenario of this invention, when simulation verification fails, the simulation engine automatically generates a structured difference diagnostic report, clearly indicating the semantic gap between the "expected state" (Mem<85%) and the "actual state" (Mem=99%, OOMTriggered), and sends it back to the policy generation engine as a negative feedback signal. After receiving the signal, the agent triggers a multi-dimensional constraint satisfaction inference process:

[0134] The agent identifies "memory resources" as the hard constraint boundary of the current operation. It uses a reverse inference algorithm to calculate the average memory overhead of a single connection under the current physical memory limit, thus deriving that the safe maximum number of connections should be 1000 instead of 2000. Secondly, the agent analyzes the operation dependencies and realizes that simply adjusting parameters cannot solve the existing deadlocked connections. Therefore, it dynamically reconstructs the operation graph and inserts a high-priority atomic operation—"KillIdle Threads"—before adjusting the parameters to pre-release memory space.

[0135] After parameter tuning and structural reconstruction, the new scheme P' was once again sent to the simulation environment. The results showed that the peak memory usage stabilized below 85% and the connection blockage was successfully cleared. This process constitutes a rigorous "generation-trial and error-parameter optimization" convergence loop. Only when the system finds an optimal strategy P that can both eradicate the fault and strictly satisfy all resource safety constraints will its digital signature be marked as "Verified" and it be allowed to enter the physical execution phase.

[0136] In one specific embodiment of the present invention, the validated optimal repair solution is implemented in stages in the physical production environment, specifically,

[0137] Based on topological betweenness centrality, low-importance nodes are selected as pilot batches to test and implement the optimal repair scheme. And within the high-frequency observation window, the relative deviation of key performance indicators is calculated in real time, using the following formula:

[0138]

[0139] in, Let j be the value of the j-th dimension of the real-time state vector. The value of the j-th dimension of the baseline state vector. To prevent smooth terms with a denominator of zero;

[0140] Based on the relative deviation, dynamic circuit breaker detection is performed to control execution risks, and the detection results are used as a quantitative basis for subsequent branch switching.

[0141] It should be noted that the optimal repair solution, once verified, is executed in stages within the physical production environment. This aims to address business interruptions caused by uncertainties in the physical environment (such as network fluctuations, hardware jitter, or unforeseen interactions). By introducing pilot testing, real-time circuit breaking, and atomic rollback mechanisms, the repair process is ensured to proceed gradually under safe and controllable conditions. This step directly takes the optimal repair solution output from the aforementioned digital twin simulation verification step as input, safely projecting it into the actual production environment. Simultaneously, through real-time feedback signals and close collaboration with subsequent branch logic, a seamless closed loop from virtual verification to physical execution is formed, thus completing the final stage of the entire intelligent operation and maintenance system and ensuring business continuity and system stability.

[0142] Furthermore, such as Figure 4As shown, the real-time calculation of the relative deviation of key performance indicators within a high-frequency observation window abandons the high-risk mode of traditional full-scale concurrent execution. Instead, it prioritizes edge nodes (such as low-traffic backup machines) with the least impact on the overall system as the first gray-scale targets for small-scale exploratory changes. Simultaneously, by increasing the monitoring sampling frequency to the second or millisecond level, the deviation of key indicators from the pre-operation baseline is calculated in real time, providing a high-density, timely quantitative data foundation for subsequent risk detection. This design effectively reduces the potential impact range of initial execution and accumulates reliable observational evidence for dynamic decision-making.

[0143] In one specific implementation scenario of this invention, the execution component first initiates a pilot node selection algorithm, abandoning the traditional full-scale concurrent mode, and instead calculates the importance score of the affected devices based on a dynamic topology graph. System computing nodes betweenness centrality and prioritize The edge nodes with the lowest values ​​(such as non-core standby machines and low-traffic load nodes) are used as the first batch of objects for canary execution, and a trial execution batch is constructed. .

[0144] The system will distribute the simulated and verified atomic repair instruction set to... The device in the middle, and simultaneously opens an independent millisecond-level high-frequency observation time window. (Typical value is 30-60 seconds). During this window period, the sampling frequency of the perception layer automatically increases from minutes to seconds, focusing on the set of feature indicators strongly correlated with the repair operation. (Such as database connection count, disk I / O wait time), forming a high-density real-time monitoring data stream.

[0145] Furthermore, dynamic circuit breaker detection based on relative deviation is used to control execution risks. This involves utilizing a streaming computing engine to make real-time judgments on the deviation of multi-dimensional indicators and high-risk log events. Once abnormal fluctuations or undefined errors are detected, a hard interrupt mechanism is triggered. The detection result directly drives the branch selection of subsequent execution paths: if the risk is controllable, the scope continues to expand; if abnormal, it is immediately blocked, thereby achieving precise state machine-style control of the execution process and preventing the risk from spreading to the entire domain.

[0146] In one specific embodiment of the present invention, in the observation window Internally, the system utilizes a streaming computing engine to perform real-time deviation analysis. The system defines a baseline state vector. (Average value N minutes before operation) and real-time state vector The system calculates the relative deviation vector of multidimensional indicators in real time. And based on the built-in dynamic circuit breaker decision logic, the analysis is performed:

[0147] Hard constraint circuit breaker: If any key performance indicator (KPI) deviates by a certain percentage... Exceeding the preset high-risk threshold (For example, if disk I / O latency suddenly increases by 50%), the system will immediately trigger a hard interrupt.

[0148] Soft constraint exception: If an undefined high-risk error code (such as "SegmentationFault" or "Kernel Panic") appears in the log stream, the system immediately triggers a hard interrupt.

[0149] Once any of the above conditions are met, the system will generate a blocking event object containing the "circuit breaker reason code", "abnormal indicator snapshot" and "blocking timestamp" and broadcast it to the control center.

[0150] In one specific embodiment of the present invention, dynamic circuit breaker detection is performed through real-time indicator monitoring to achieve circuit breaker triggering or rolling release. Specifically,

[0151] When any metric deviates beyond a preset hard constraint threshold or a high-risk log is generated, a hard interrupt is triggered and a pre-bound atomic rollback instruction is executed to restore the system state to the baseline level and prevent the risk from spreading.

[0152] Based on the execution result of the atomic rollback instruction, when all indicators deviate below the preset safety threshold, an exponential step strategy is adopted for rolling release until all affected nodes are covered, thereby achieving a smooth recovery of business services.

[0153] It should be noted that dynamic circuit breaker detection through real-time indicator monitoring to achieve circuit breaker or rolling release is designed to address unpredictable dynamic uncertainties in the physical production environment, ensuring business continuity and zero interruption risk throughout the implementation of remediation solutions. This step follows the detection results after the pilot batch trial execution and real-time calculation of relative deviation, acting as a dynamic switcher for the execution path, directly determining whether to block risks or gradually expand the release. Thus, together with the preceding digital twin simulation verification step, it forms a complete protection loop from virtual security to physical reliability, providing the final execution guarantee for the entire intelligent operation and maintenance system.

[0154] Furthermore, when any metric deviates beyond a preset hard constraint threshold or a high-risk log appears, a hard interrupt is triggered, and pre-bound atomic rollback instructions are executed to restore the system state to the baseline level and prevent risk propagation. When real-time monitoring detects abnormal fluctuations or undefined errors, all subsequent batch tasks are immediately frozen, and the pilot node state is quickly restored to its pre-operation state using pre-bound reverse instructions for each atomic operation. This mechanism effectively prevents potential secondary failures from spreading throughout the entire system, ensuring that risks are strictly isolated to a minimum, while providing a safe exit point for the system without loss of performance.

[0155] Furthermore, based on the execution results of the atomic rollback command, an exponential step-by-step rolling release strategy is adopted when all metric deviations are below the preset safety threshold, until all affected nodes are covered. After confirming stability in the pilot phase, the coverage of the remediation plan is systematically advanced using a conservative to aggressive batch scaling approach (e.g., gradually increasing from 10% to full coverage). A brief health self-check is performed after each batch is completed to ensure a smooth and seamless change process, ultimately achieving complete eradication of the fault and full service restoration.

[0156] In one specific implementation scenario of the present invention, the system dynamically switches the execution path based on the circuit breaker detection result:

[0157] Branch A (Blocking and Rollback): If a hard interrupt signal is triggered, the execution engine immediately freezes all subsequent batches of pending tasks. Simultaneously, the system invokes the pre-defined atomic rollback instruction in the repair plan. (e.g., "Execute snapshot recovery script", "Roll back firewall rules"), perform reverse operations on the pilot nodes. The system continuously monitors the status after the rollback until the metrics return to normal. Within the allowable error range, ensure the environment is restored without damage.

[0158] Branch B (Rolling Release and Expansion): If the pilot nodes operate stably during the observation period and the indicators converge as expected. The system will unlock the full rolling update process. The execution engine will gradually expand the execution scope according to an "exponential step" strategy (e.g., 10% to 30% to the remaining full update). After each batch of changes is completed, the system will automatically perform a brief health self-check until the repair strategy covers all affected nodes, ultimately achieving a smooth recovery of business services.

[0159] In a second embodiment of the present invention, the present invention provides a data center fault diagnosis and repair system based on multi-agent collaborative reasoning, such as... Figure 5 As shown, the system includes a semantic mapping module 1, a root cause analysis module 2, and a repair verification module 3;

[0160] The semantic mapping module 1 is used to collect time-series monitoring indicators and system log data in real time and convert them into serialized environment state vectors;

[0161] The root cause analysis module 2 is used to generate initial hypotheses in parallel by multiple vertical domain agents based on the environmental state vector, and the decision agent outputs the optimal root cause conclusion through multiple rounds of cross-verification.

[0162] The repair verification module 3 is used to generate an atomic operation sequence based on the optimal root cause conclusion, and to perform simulation verification and iterative optimization in a digital twin environment to obtain the optimal repair scheme that has passed the verification.

[0163] Furthermore, such as Figure 6 As shown, the data center fault automatic diagnosis and repair system based on multi-agent collaboration provided by this invention is strictly divided into four layers from bottom to top in its logical architecture: the infrastructure global perception layer, the multi-agent cognitive decision-making layer, the simulation verification layer, and the automated execution layer. These four layers are tightly coupled through a standardized data bus and control protocol, forming a complete closed-loop system from data acquisition in the physical world to intelligent analysis in the digital world, and then back to the physical world for precise control.

[0164] The semantic mapping module 1 is located in the system's underlying infrastructure global perception layer. This layer mainly consists of physical hardware devices in the data center (including computing servers, storage arrays, network switches, firewalls, etc.) and software probe roles deployed on these devices. Its core function is to act as the system's perception antenna, responsible for maintaining the daily operation of business and generating massive amounts of multimodal data in real time, including time-series monitoring indicators, system operation logs, and alarm events. The data aggregation role in this layer is responsible for cleaning, aligning, and vectorizing the scattered heterogeneous raw data, transforming it into a standard environmental state vector that upper-layer intelligent agents can understand, thereby ensuring that subsequent analysis is based on comprehensive and accurate factual evidence.

[0165] The root cause analysis module 2 is located in the core multi-agent cognitive decision layer of the system. This layer is deployed on a cluster of inference servers with high computing power and runs a multi-agent inference engine driven by a large language model. This layer includes several specific roles: domain expert agents (playing the roles of network expert, hardware expert, application expert, and security expert, respectively) and decision agents. The role of the domain expert agents is to load their own exclusive operation and maintenance knowledge bases and independently deduce the same fault characteristics from different technical dimensions; while the role of the decision agents is to preside over multi-party debates, eliminate cognitive biases from a single perspective through cross-questioning mechanisms, output the final root cause analysis report by integrating the confidence levels of all parties, and generate a structured repair plan containing specific operation steps.

[0166] The repair verification module 3 is located in the simulation verification layer of the system's intermediate protection. This layer relies on the virtualization cluster and container orchestration platform to construct a digital twin sandbox role that is highly consistent with the production environment. The main function of this layer is to act as a "safety filter" before repair instructions are issued. After the cognition layer generates a repair plan, the simulation verification layer will replay the fault scenario in the virtual environment and test run the repair script, monitoring resource changes and service recovery in real time after the operation. Only plans that have been evaluated by this layer and determined to be safe and effective will be allowed to pass and be passed to the execution layer, thereby minimizing the potential risks of automated operations to the production environment.

[0167] The hierarchical execution module 4 is located in the top-level automated execution layer of the system. As the final executor, this layer plays a dual role as both the operator and the security guardian. Its main function is to receive simulated and verified instruction sets and perform substantive configuration changes, service restarts, or traffic scheduling on the physical infrastructure via SSH or API interfaces. Simultaneously, this layer has a built-in real-time feedback mechanism. Once abnormal fluctuations in performance metrics are detected during execution, an atomic rollback operation will be immediately triggered to restore the system state to its pre-operation state, ensuring uninterrupted business continuity.

[0168] In a third embodiment of the present invention, the present invention provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the data center fault diagnosis and repair method based on multi-agent cooperative reasoning as described above.

[0169] In Embodiment 4 of the present invention, the present invention provides a storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps in the data center fault diagnosis and repair method based on multi-agent cooperative reasoning as described above.

[0170] In summary, this invention provides a data center fault diagnosis and repair method based on multi-agent collaborative reasoning. This method effectively overcomes the limitations of single-model knowledge depth and the illusion problem through multi-agent collaborative reasoning, achieving multi-perspective cross-validation and accurate root cause localization. The introduction of digital twin simulation and negative feedback iterative optimization solves the risk of secondary failures caused by direct execution of repair scripts, ensuring the safety and reliability of the solution. Multimodal data semantic mapping enhances the large model's understanding of heterogeneous data. The construction of a hierarchical execution and real-time circuit breaker rollback system eliminates the risk of business interruption during automation, thereby achieving a shift from manual assistance to secure autonomy and significantly improving the accuracy, efficiency, and reliability of data center operation and maintenance.

[0171] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0172] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0173] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0174] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data center fault diagnosis and repair method based on multi-agent cooperative reasoning, characterized in that: include, Real-time collection of time-series monitoring metrics and system log data, and conversion into serialized environment state vectors; Based on the environmental state vector, initial hypotheses are generated in parallel by multiple vertical domain agents, and the optimal root cause conclusion is output by the decision agent through multiple rounds of cross-verification. Atomic operation sequences are generated based on the optimal root cause conclusion, and simulation verification and iterative optimization are performed in a digital twin environment to obtain the optimal repair scheme that has passed verification.

2. The data center fault diagnosis and repair method based on multi-agent cooperative reasoning according to claim 1, characterized in that: The process of collecting time-series monitoring indicators and system log data in real time through infrastructure probes and converting them into serialized environmental state vectors includes: Savitzky-Golay convolution is used to smooth the signal and solve the feature for the time-series monitoring index. First-order trend features and second-order fluctuation features are extracted and dynamic semantic labeling is performed through Z-Score dynamic threshold determination to form the semantic enhancement features of the time-series monitoring index. Based on the aforementioned semantic enhancement features, the system logs are processed using online template extraction and variable regularization masking based on weighted edit distance to generate a standardized event set, thereby achieving semantic alignment between logs and metrics. Based on the standardized event set, TF-IDF weights are calculated, and after pruning and sorting, a composite environment state vector is constructed.

3. The data center fault diagnosis and repair method based on multi-agent cooperative reasoning according to claim 2, characterized in that: The process involves multiple agents from different vertical domains generating initial hypotheses in parallel, and a decision agent converging the conclusion through multiple rounds of cross-validation to output the optimal root cause conclusion. The multiple vertical domain agents generate initial hypotheses based on the composite environment state vector; the initial hypotheses include a unique hypothesis identifier, a source agent identifier, a root cause code, a chain of evidence, and an initial confidence level, to form a structured multi-dimensional set of fault hypotheses; The decision-making agent iteratively updates the confidence level of each hypothesis through multiple rounds of cross-verification. The confidence level update formula is as follows: ； in, For the first Assumptions in round iteration Confidence level, For the learning rate step size, To determine the domain authority weight of agent j, Indicates feedback polarity, Represents the strength of evidence; Based on the updated confidence level, convergence is determined and the optimal root cause conclusion is output when the confidence level meets the preset convergence condition.

4. The data center fault diagnosis and repair method based on multi-agent cooperative reasoning according to claim 3, characterized in that: The optimal root cause conclusion, which is output by the decision-making agent through multiple rounds of cross-verification, also includes... The formula for calculating the completeness of evidence score is as follows: ； in, This represents the actual number of features matched. The standard indicator base number for this fault type defined in the knowledge base; Based on the evidence completeness score, if the score is lower than a preset threshold, the risk of insufficient evidence is marked in the root cause analysis report, and the marked report is integrated into the optimal root cause conclusion.

5. The data center fault diagnosis and repair method based on multi-agent cooperative reasoning according to claim 4, characterized in that: The process of generating atomic operation sequences based on root cause conclusions and performing simulation verification and iterative optimization in a digital twin environment includes: Based on the optimal root cause conclusion that integrates the risk markers of insufficient evidence, a repair scheme consisting of a sequence of atomic operations is generated; Each atomic operation in the atomic operation sequence is bound to a corresponding rollback instruction and verification metric. After implementing the remediation plan on the digital twin instance, a feasibility score is calculated using the following formula: ； in, As a repair solution, This refers to the system state observed after performing a sequence of atomic operations in a digital twin environment. To restore the weight, Indicates the degree of service restoration. Risk weighting This indicates the risk value of system crash or secondary failure. As a cost weight, Indicates the execution time cost; Based on the feasibility score, when the score is lower than a preset safety threshold, a difference diagnosis report is generated as a negative feedback signal to trigger iterative optimization of the repair scheme until an optimal repair scheme that satisfies all safety constraints and passes verification is obtained. .

6. The data center fault diagnosis and repair method based on multi-agent cooperative reasoning according to claim 1, characterized in that: It also includes, The validated optimal remediation solution will be implemented in stages in the physical production environment, specifically including: Based on topological betweenness centrality, low-importance nodes are selected as pilot batches to test and implement the optimal repair scheme. And within the high-frequency observation window, the relative deviation of key performance indicators is calculated in real time, using the following formula: ； in, Let j be the value of the j-th dimension of the real-time state vector. The value of the j-th dimension of the baseline state vector. To prevent smooth terms with a denominator of zero; Based on the relative deviation, dynamic circuit breaker detection is performed to control execution risks, and the detection results are used as a quantitative basis for subsequent branch switching.

7. The data center fault diagnosis and repair method based on multi-agent cooperative reasoning according to claim 6, characterized in that: Dynamic circuit breaker detection is achieved through real-time indicator monitoring to enable circuit breaker activation or rolling release, specifically including... When any metric deviates beyond a preset hard constraint threshold or a high-risk log is generated, a hard interrupt is triggered and a pre-bound atomic rollback instruction is executed to restore the system state to the baseline level and prevent the risk from spreading. Based on the execution result of the atomic rollback instruction, when all indicators deviate below the preset safety threshold, an exponential step strategy is adopted for rolling release until all affected nodes are covered, thereby achieving a smooth recovery of business services.

8. A data center fault diagnosis and repair system based on multi-agent cooperative reasoning, characterized in that: It includes a semantic mapping module, a root cause analysis module, and a repair verification module; The semantic mapping module is used to collect time-series monitoring indicators and system log data in real time and convert them into serialized environment state vectors. The root cause analysis module is used to generate initial hypotheses in parallel by multiple vertical domain agents based on environmental state vectors, and the decision agent outputs the optimal root cause conclusion through multiple rounds of cross-verification. The repair verification module is used to generate an atomic operation sequence based on the optimal root cause conclusion, and to perform simulation verification and iterative optimization in a digital twin environment to obtain the optimal repair solution that has passed verification.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the data center fault diagnosis and repair method based on multi-agent cooperative reasoning as described in any one of claims 1-7.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps in the data center fault diagnosis and repair method based on multi-agent cooperative reasoning as described in any one of claims 1-7.

Images