A large model cluster fault diagnosis method based on double-path reasoning

Abstract

The present application relates to a kind of big model cluster fault diagnosis method based on double path reasoning, belong to intelligent fault diagnosis technical field, method includes: obtaining the cluster fault request to be diagnosed, form the request input for diagnosis, request input is semantically vectorized and is characterized, and in experience memory network, retrieve the historical diagnosis memory and corresponding confidence degree matched with request input are obtained;According to confidence degree, reasoning path selection is selected when confidence degree meets preset condition, otherwise, analysis reasoning path is selected, and corresponding diagnosis result is generated;Receive disposal execution result or artificial confirmation feedback, request input, diagnosis result and feedback result are input into experience memory network to update memory weight, to realize the continuous learning of constantly improving with diagnosis process.The present application realizes the adaptive switching of intuition reasoning and analysis reasoning by experience memory retrieval and metacognition routing, and continuous learning is carried out in combination with disposal feedback.

Description

Technical Field

[0001] This invention relates to the field of intelligent fault diagnosis technology, and in particular to a fault diagnosis method for large model clusters based on dual-path reasoning. Background Technology

[0002] With the development of cloud-native technologies, cluster systems represented by Kubernetes have become the key infrastructure for supporting microservices and containerized applications. Due to the large number of internal components, complex dependencies, and dynamic changes in operating status, actual faults often manifest as a complex interweaving of multiple sources of information, including alarms, events, logs, configurations, and resource status. Operation and maintenance personnel need to perform correlation analysis, verify hypotheses, and provide solutions from a large amount of scattered evidence, resulting in a high diagnostic threshold and long response time.

[0003] In recent years, large-scale models, with their language understanding and reasoning capabilities, have been used for cluster fault diagnosis. By parsing alarm texts, log fragments, and kubectl outputs and generating repair suggestions, they can reduce operation and maintenance costs to some extent. However, existing cluster diagnosis solutions based on large-scale models still have the following shortcomings: 1. Lack of continuous learning from operation and maintenance experience: Many solutions rely on static knowledge bases or fixed retrieval content. Each diagnosis is relatively independent, making it impossible to distill successful experiences in handling historical faults into reusable patterns. This results in repeated reasoning for responses to high-frequency faults, and it is difficult to continuously improve with the accumulation of operation and maintenance practices. 2. Difficulty in balancing diagnostic speed and depth: For familiar and recurring faults, it is expected that actionable solutions can be provided quickly; for complex or novel faults, more in-depth causal analysis and verification are required. Existing solutions usually adopt a single reasoning approach, which is prone to either over-reasoning leading to high latency and high costs, or under-reasoning leading to inaccurate root causes and weak interpretability. 3. Insufficient consistency and operability of evidence: Cluster fault handling is heavily dependent on contextual conditions such as version constraints, permission policies, resource quotas, and network policies. Existing solutions may ignore cluster constraints when outputting disposal commands and steps, resulting in unexecutable or high-risk suggestions, affecting the reliability and implementability of diagnostic results. 4. Privacy and local deployment requirements are prominent: Logs, configurations, and alarm information in production environments are often sensitive. Actual deployments often require local diagnosis and learning to avoid uploading data to external services; however, under limited local resources, more efficient inference and experience reuse mechanisms are needed. Therefore, how to solve these problems is currently a matter that needs to be considered. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fault diagnosis method for large model clusters based on dual-path reasoning, thus solving the deficiencies of the prior art.

[0005] The objective of this invention is achieved through the following technical solution: a fault diagnosis method for large model clusters based on dual-path reasoning, the diagnosis method comprising:

[0006] Step 1: Obtain the cluster fault request to be diagnosed, form the request input for diagnosis, perform semantic vectorization representation on the request input, and retrieve the historical diagnostic memory and corresponding confidence level that match the request input from the experience memory network;

[0007] Step 2: Select the reasoning path based on the confidence level. When the confidence level meets the preset conditions, select the intuitive reasoning path; otherwise, select the analytical reasoning path and generate the corresponding diagnostic results.

[0008] Step 3: Receive the results of the treatment or manual confirmation feedback, and input the request input, diagnosis results and feedback results into the experience memory network to update the memory weights, thereby achieving continuous learning that improves with the diagnosis process.

[0009] Under the intuitive reasoning path, the large model combines historical diagnostic memory to perform fault pattern matching and answer generation, and outputs diagnostic results that include fault root cause judgment and handling steps;

[0010] Under the analytical reasoning path and guided by historical diagnostic memory, the causal knowledge graph is retrieved and causal chain reasoning is performed. The large model combines the graph reasoning results to generate diagnostic results containing causal explanations.

[0011] The step of obtaining the cluster fault request to be diagnosed, and forming the request input for diagnosis, includes:

[0012] Cluster fault request collection and structured encapsulation, including fault symptom information collection, cluster runtime context information collection, and evidence information alignment and encapsulation;

[0013] Diagnostic request input generation is used to convert diagnostic request payloads into input sequences or structured cue words that can be processed by large models, enabling them to be used for subsequent experience retrieval and inference path selection.

[0014] The step of semantically vectorizing the request input and retrieving historical diagnostic memories and corresponding confidence levels that match the request input from the experience memory network includes:

[0015] Query vector generation is used to encode diagnostic request inputs into query vectors using an embedding model, where the query vectors represent the semantics of fault symptoms, contextual constraints, and evidence features.

[0016] Experience memory retrieval and confidence calculation are used to retrieve the Top-K historical diagnostic memories from episodic and pattern memories in the experience memory network. The confidence of each historical diagnostic memory is calculated based on the similarity between the query vector and the historical diagnostic memory vector, the memory time decay factor, and the historical treatment success rate. The maximum confidence or aggregate confidence is output for path selection.

[0017] The contextual memory is used to record the symptom-context-treatment-outcome information of a single diagnostic round; the pattern memory is used to cluster and abstract multiple similar contextual memories to form reusable failure patterns. The pattern memory includes the semantic center representation of the failure pattern, the treatment strategy template, and the reliability parameter.

[0018] The selection of inference paths based on confidence level, where the intuitive inference path is selected when the confidence level meets a preset condition, and the analytical inference path is selected otherwise, includes:

[0019] Confidence assessment and routing decision are used to read the confidence level and compare it with a preset threshold to generate an inference path selection result. When the confidence level is higher than the preset threshold, the intuitive inference path is selected; otherwise, the analytical inference path is selected.

[0020] Inference parameter configuration is used to set corresponding inference control parameters for the selected inference path. The inference control parameters include at least one or more of the following: number of retrievals, generation length, maximum number of hops in the causal chain, and output structure constraints, thereby dynamically balancing diagnostic speed and diagnostic depth.

[0021] Under the intuitive reasoning path, the large model combines historical diagnostic memory to perform fault pattern matching and answer generation, outputting diagnostic results that include fault root cause judgment and handling steps, including:

[0022] Intuitive path context construction is used to convert the Top-K historical diagnostic memories into a large model context according to a preset template, where the template includes a summary of fault phenomena, common root cause candidates, verification command suggestions, and key points of handling steps.

[0023] Intuitive path diagnosis generation is used by large models to perform pattern matching and answer generation based on context, and outputs diagnostic results including root cause judgment, verification steps, handling steps and prevention suggestions.

[0024] The process of retrieving causal knowledge graphs and reasoning about causal chains under the guidance of historical diagnostic memory, and generating diagnostic results containing causal explanations by combining the graph reasoning results with a large model, includes:

[0025] Causal knowledge graph retrieval and causal chain generation are used to extract key entities and relational clues based on historical diagnostic memories and diagnostic request inputs, perform multi-hop retrieval in the causal knowledge graph and generate candidate causal chains, wherein the candidate causal chains at least represent the causal association between symptom nodes, intermediate state nodes and root cause nodes.

[0026] Causal chain verification and analysis path diagnosis generation are used to perform consistency and operability verification on candidate causal chains to obtain verified causal chains. The large model combines the verified causal chains and the diagnostic request input to generate diagnostic results that include causal explanations, root cause location basis, treatment steps, and rollback steps.

[0027] The consistency verification includes at least one or more of the following: matching with cluster version constraints, resource quota constraints, network policy constraints, or permission policy constraints; the operability verification includes at least one or more of the following: verification of command syntax, parameter completeness, or execution risk warnings given in the diagnostic results.

[0028] Step three includes:

[0029] Feedback results are collected to gather the results of the handling or the conclusions of manual confirmation. The feedback results shall include at least one or more of the following: whether the recovery was successful, the recovery time, the status of secondary alarms, or the manual confirmation label.

[0030] Experience memory write-back and weight update are used to input diagnostic request input, diagnostic results and feedback results into the experience memory network to form new contextual memories, and update the pattern memory reliability parameters corresponding to the contextual memories based on the feedback results.

[0031] This invention offers the following advantages: a fault diagnosis method for large model clusters based on dual-path reasoning, which adaptively switches between intuitive and analytical reasoning through experience memory retrieval and metacognitive routing, and continuously learns by incorporating handling feedback. Since the large model, experience memory network, and causal knowledge graph are all deployed locally, the diagnostic process does not require uploading alarms, logs, configurations, and context data to external servers, thereby reducing the risk of data leakage and meeting the requirements of localized processing of operational data in production environments. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the process of the present invention;

[0033] Figure 2 This is a diagram of the fault diagnosis and analysis architecture. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided below with reference to the accompanying drawings is not intended to limit the scope of protection of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The present invention will be further described below with reference to the accompanying drawings.

[0035] like Figure 1 and Figure 2 As shown, this invention relates to a large-scale cluster fault diagnosis method based on dual-path reasoning, aiming to solve the problems of existing cluster fault diagnosis schemes, such as difficulty in continuously learning from historical handling experience, difficulty in balancing diagnosis speed and depth, and insufficient interpretability and operability of diagnosis results, thereby improving the accuracy and engineering implementation capability of cluster fault diagnosis; specifically including the following:

[0036] S1. The diagnostic system obtains cluster fault requests and generates diagnostic request inputs;

[0037] The diagnostic system receives cluster fault requests for diagnosis and generates diagnostic request inputs for subsequent retrieval and inference. The diagnostic system can be a locally deployed operations and maintenance platform, command-line plugin, alarm platform interface component, or in-cluster diagnostic service.

[0038] The diagnostic request input must include at least fault symptom information and cluster runtime context information, including but not limited to:

[0039] 1) Fault symptom information: alarm title and alarm details, Kubernetes event information, error code, log fragments, exception stack information, output of kubectl get / describe / logs, and phenomenon descriptions provided by operations personnel in natural language;

[0040] 2) Cluster runtime context information: Cluster and component version information, namespaces and object identifiers (such as Pod, Deployment, Service, Node, etc.), resource usage and quota information (CPU, memory, disk, PVC, etc.), scheduling status and scheduling constraints (taints / tolerance, affinity / anti-affinity, priority, etc.), network-related information (Service, DNS, Ingress, NetworkPolicy, CNI, etc.), image and repository access status (such as ImagePullBackOff, authentication failure, rate limiting, etc.), configuration and permission information (ConfigMap / Secret, RBAC, admission control Webhook, etc.).

[0041] To facilitate processing by the large model (Kube large language model) and retrieval module, the diagnostic system standardizes and organizes the above information to generate structured or semi-structured diagnostic request payloads, and converts them into input sequences or prompt word templates that can be directly used by the large model. The prompt word templates can include fields such as "problem summary - observed evidence - environmental constraints - expected output structure" to reduce the impact of noise information on the diagnostic results.

[0042] S2. The diagnostic system vectorizes the diagnostic request input and retrieves historical diagnostic memories and confidence levels from the experience memory network.

[0043] The diagnostic system employs an embedding model to semantically vectorize the diagnostic request input, obtaining a query vector. This query vector represents the semantics of the fault symptoms, contextual constraints, and evidentiary features. The embedding model can be a text embedding model based on a deep neural network, preferably supporting long texts and operational technical corpora.

[0044] Experience memory network is used to store and manage historical diagnostic memories, which include episodic memories and pattern memories: (1) Episodic memories: record the "symptom-context-treatment action-result" information of a single diagnostic round, which is used to reflect the complete trajectory of a real fault treatment; (2) Pattern memories: cluster and abstract multiple similar episodic memories to form reusable fault patterns, including semantic center representation, treatment strategy template and reliability parameters, which are used to support fast matching.

[0045] During the retrieval phase, the experience memory network retrieves the top K historical diagnostic memories from episodic and pattern memories based on the query vector, and calculates the confidence score corresponding to each historical diagnostic memory. The confidence score calculation comprehensively considers: the similarity between the query vector and the historical memory vector, the time decay factor (to reflect the priority of recent experience), and the historical treatment success rate or human confirmation label (to reflect the reliability of the experience). The retrieval output can be the maximum confidence score, or an aggregated confidence score obtained by aggregating the confidence scores of the top K memories, which is used for subsequent inference path selection.

[0046] S3. The metacognitive controller selects inference paths and configures inference parameters based on confidence levels;

[0047] The metacognitive controller selects a reasoning path based on the confidence level to adaptively balance diagnostic speed and diagnostic depth. Specifically, the confidence level is compared with a preset threshold: when the confidence level meets a preset condition (e.g., it is higher than the preset threshold), the intuitive reasoning path is selected; otherwise, the analytical reasoning path is selected.

[0048] After path selection is completed, the metacognitive controller is also used to configure the corresponding inference control parameters. These parameters include, but are not limited to: the number of retrieval entries K, the maximum generated length, the number of evidence citations, the maximum number of hops in the causal chain, and output structure constraints (e.g., fixed output fields such as "phenomenon summary / root cause / verification / action / rollback / prevention"). Through the above configuration, the system can quickly output executable suggestions in high-similarity reproducibility fault scenarios, and trigger more rigorous causal inference and verification in low-similarity or complex fault scenarios to improve reliability.

[0049] S4. Based on historical diagnostic memory, diagnostic results are generated from a large model under the intuitive reasoning path;

[0050] When the metacognitive controller selects the intuitive reasoning path, the system injects the first K retrieved historical diagnostic memories into the large model context according to a preset template, forming an intuitive reasoning context. The template includes at least: a summary of the fault phenomenon, historical high-frequency root cause candidates, verification suggestions related to the root cause (such as check items or commands), key points of the handling steps, and precautions.

[0051] The large model performs experience-based matching and generation under an intuitive reasoning path, and outputs diagnostic results that include root cause identification and remediation steps. The diagnostic results preferably include at least: (1) root cause candidates and corresponding confidence statements; (2) verification steps (such as recommended kubectl commands or checks); (3) remediation steps (executable commands and configuration modification suggestions); and (4) rollback steps and prevention suggestions when necessary.

[0052] Intuitive reasoning paths are suitable for high-frequency, recurring fault scenarios (such as insufficient resources leading to OOMKilled, image pull failure, common configuration errors, etc.), achieving rapid response with low reasoning costs.

[0053] S5. Based on the causal knowledge graph, causal chain reasoning is performed under the analysis and reasoning path, and a diagnostic result containing causal explanation is generated by the large model.

[0054] When the metacognitive controller selects the analysis and reasoning path, the system, guided by the historical diagnostic memory, retrieves the causal knowledge graph and performs causal chain reasoning. The large model then combines the graph reasoning results to generate a diagnostic result containing causal explanations.

[0055] Furthermore, S5 specifically includes:

[0056] S501. Causal Knowledge Graph Retrieval and Candidate Causal Chain Generation: The system extracts key entities and relational clues based on the diagnostic request input and historical diagnostic memory, using them as the entry point for graph retrieval. It then performs multi-hop retrieval within the causal knowledge graph to generate candidate causal chains. These candidate causal chains characterize the causal association between "symptom nodes—intermediate state nodes—root cause nodes" and can output evidence citation information associated with the causal chain nodes (such as event types, key log lines, configuration fields, constraints, etc.). During graph retrieval, the diagnostic system can first locate nodes or document fragments in the graph knowledge base that are semantically similar to the diagnostic request input based on the semantic vector, using these as the starting nodes for multi-hop causal chain retrieval.

[0057] S502. Causal Chain Verification and Analysis Path Diagnosis Generation: The system performs consistency and operability checks on candidate causal chains to obtain verified causal chains. Consistency checks verify whether the causal chain matches cluster version constraints, resource quota constraints, network policy constraints, and permission policy constraints. Operability checks verify whether the handling steps are executable, whether the command syntax and parameters are complete, and whether there are any obvious risk points. Subsequently, the verified causal chain and the diagnostic request are input into the large model, which generates diagnostic results including causal explanations, root cause location basis, handling steps, and rollback steps.

[0058] The analytical reasoning path is suitable for complex fault scenarios with low similarity, novelty, or cross-component issues (such as coupling problems between network policies and service discovery, cascading unavailability caused by scheduling constraints, and control plane anomalies caused by certificate expiration), which can improve the interpretability of diagnostic results and reduce the probability of misjudgment.

[0059] S6. Receive execution feedback and write it back to the experience memory network to achieve continuous learning;

[0060] The diagnostic system receives the results of the treatment or manual confirmation feedback, and writes the diagnostic request input, diagnostic results, and feedback results into the experience memory network to update the memory network cache and achieve continuous learning. The feedback results include at least: whether recovery has been achieved, recovery time, whether a secondary alarm occurred, whether the process has been escalated to manual handling, and a manual confirmation tag.

[0061] The experience memory network updates its memories based on feedback: when feedback indicates a valid diagnosis, it increases the reliability parameter of the corresponding pattern memory and enhances its priority in subsequent retrievals; when feedback indicates an invalid diagnosis or a risk, it decreases the reliability parameter of the corresponding pattern memory or marks the corresponding context memory as a negative example to reduce the probability of future mismatches. Through this write-back mechanism, the system can continuously optimize retrieval confidence and path routing effectiveness as operational practices accumulate, gradually expanding the coverage of intuitive reasoning paths while maintaining the rigor of analytical reasoning paths when necessary.

[0062] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and improvements, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A fault diagnosis method for large model clusters based on dual-path reasoning, characterized in that: The diagnostic method includes: Step 1: Obtain the cluster fault request to be diagnosed, form the request input for diagnosis, perform semantic vectorization representation on the request input, and retrieve the historical diagnostic memory and corresponding confidence level that match the request input from the experience memory network; Step 2: Select the reasoning path based on the confidence level. When the confidence level meets the preset conditions, select the intuitive reasoning path; otherwise, select the analytical reasoning path and generate the corresponding diagnostic results. Step 3: Receive the results of the treatment or manual confirmation feedback, and input the request input, diagnosis results and feedback results into the experience memory network to update the memory weights, thereby achieving continuous learning that improves with the diagnosis process; The step of semantically vectorizing the request input and retrieving historical diagnostic memories and corresponding confidence levels that match the request input from the experience memory network includes: Query vector generation is used to encode diagnostic request inputs into query vectors using an embedding model, where the query vectors represent the semantics of fault symptoms, contextual constraints, and evidence features. Experience memory retrieval and confidence calculation are used to retrieve the Top-K historical diagnostic memories from episodic and pattern memories in the experience memory network, and calculate the confidence of each historical diagnostic memory based on the similarity between the query vector and the historical diagnostic memory vector, the memory time decay factor, and the historical treatment success rate, and output the maximum confidence or aggregate confidence for path selection. The selection of inference paths based on confidence level, where the intuitive inference path is selected when the confidence level meets a preset condition, and the analytical inference path is selected otherwise, includes: Confidence assessment and routing decision are used to read the confidence level and compare it with a preset threshold to generate an inference path selection result. When the confidence level is higher than the preset threshold, the intuitive inference path is selected; otherwise, the analytical inference path is selected. Inference parameter configuration is used to set corresponding inference control parameters for the selected inference path. The inference control parameters include at least one or more of the following: number of retrievals, generation length, maximum number of hops in the causal chain, and output structure constraints, thereby dynamically balancing diagnostic speed and diagnostic depth.

2. The fault diagnosis method for large model clusters based on dual-path reasoning according to claim 1, characterized in that: Under the intuitive reasoning path, the large model combines historical diagnostic memory to perform fault pattern matching and answer generation, and outputs diagnostic results that include fault root cause judgment and handling steps; Under the analytical reasoning path and guided by historical diagnostic memory, the causal knowledge graph is retrieved and causal chain reasoning is performed. The large model combines the graph reasoning results to generate diagnostic results containing causal explanations.

3. The fault diagnosis method for large model clusters based on dual-path reasoning according to claim 1, characterized in that: The step of obtaining the cluster fault request to be diagnosed, and forming the request input for diagnosis, includes: Cluster fault request collection and structured encapsulation, including fault symptom information collection, cluster runtime context information collection, and evidence information alignment and encapsulation; Diagnostic request input generation is used to convert diagnostic request payloads into input sequences or structured cue words that can be processed by large models, enabling them to be used for subsequent experience retrieval and inference path selection.

4. The fault diagnosis method for large model clusters based on dual-path reasoning according to claim 1, characterized in that: The contextual memory is used to record the symptom-context-treatment-outcome information of a single diagnostic round; the pattern memory is used to cluster and abstract multiple similar contextual memories to form reusable failure patterns. The pattern memory includes the semantic center representation of the failure pattern, the treatment strategy template, and the reliability parameter.

5. The method for fault diagnosis of large model clusters based on dual-path reasoning according to claim 2, characterized in that: Under the intuitive reasoning path, the large model combines historical diagnostic memory to perform fault pattern matching and answer generation, outputting diagnostic results that include fault root cause judgment and handling steps, including: Intuitive path context construction is used to convert the Top-K historical diagnostic memories into a large model context according to a preset template, where the template includes a summary of fault phenomena, common root cause candidates, verification command suggestions, and key points of handling steps. Intuitive path diagnosis generation is used by large models to perform pattern matching and answer generation based on context, and outputs diagnostic results including root cause judgment, verification steps, handling steps and prevention suggestions.

6. The fault diagnosis method for large model clusters based on dual-path reasoning according to claim 2, characterized in that: The process of retrieving causal knowledge graphs and reasoning about causal chains under the guidance of historical diagnostic memory, and generating diagnostic results containing causal explanations by combining the graph reasoning results with a large model, includes: Causal knowledge graph retrieval and causal chain generation are used to extract key entities and relational clues based on historical diagnostic memories and diagnostic request inputs, perform multi-hop retrieval in the causal knowledge graph and generate candidate causal chains, wherein the candidate causal chains at least represent the causal association between symptom nodes, intermediate state nodes and root cause nodes. Causal chain verification and analysis path diagnosis generation are used to perform consistency and operability verification on candidate causal chains to obtain verified causal chains. The large model combines the verified causal chains and the diagnostic request input to generate diagnostic results that include causal explanations, root cause location basis, treatment steps, and rollback steps.

7. The fault diagnosis method for large model clusters based on dual-path reasoning according to claim 6, characterized in that: The consistency verification includes at least one or more of the following: matching with cluster version constraints, resource quota constraints, network policy constraints, or permission policy constraints; the operability verification includes at least one or more of the following: verification of command syntax, parameter completeness, or execution risk warnings given in the diagnostic results.

8. The fault diagnosis method for large model clusters based on dual-path reasoning according to claim 1, characterized in that: Step three includes: Feedback results are collected to gather the results of the handling or the conclusions of manual confirmation. The feedback results shall include at least one or more of the following: whether the recovery was successful, the recovery time, the status of secondary alarms, or the manual confirmation label. Experience memory write-back and weight update are used to input diagnostic request input, diagnostic results and feedback results into the experience memory network to form new contextual memories, and update the pattern memory reliability parameters corresponding to the contextual memories based on the feedback results.

Images