Method and device for fault diagnosis

Abstract

The invention provides a fault diagnosis method and device, computing equipment, a computer readable storage medium, a computer program product and a chip. In some embodiments, diagnostic information corresponding to input information describing the fault may be derived by retrieving enhanced generation using a fault database storing a causal chain related to the fault information. The fault database can store richer and more comprehensive information related to the fault, so that in the process of generating and determining the diagnosis information through retrieval enhancement, the root cause of the fault can be quickly determined based on a causal chain, the fault positioning can be quickly realized, the fault processing efficiency is improved, the dependence on operation and maintenance personnel is reduced, and the fault processing efficiency is improved. And the cost is reduced.

Description

Technical Field

[0001] The embodiments disclosed herein primarily relate to the field of computers, and more specifically, to a method and apparatus for fault diagnosis. Background Technology

[0002] As cloud computing and other systems expand in scale, the occurrence of faults significantly increases the cost of manual maintenance. Intelligent operation and maintenance-based fault diagnosis technology can help operations personnel quickly identify faults, thereby improving the efficiency of fault resolution. However, the scale of cloud computing and other systems continues to expand, and the objects of management are extending from physical devices to virtual machines. The scale of network element management is also constantly growing. Therefore, there is an urgent need for more effective fault diagnosis solutions. Summary of the Invention

[0003] This disclosure provides a scheme for fault diagnosis that uses a fault database storing causal chains related to fault information to determine diagnostic information, enabling faster fault location and improving the efficiency of fault handling.

[0004] In a first aspect of this disclosure, a method for fault diagnosis is provided, comprising: acquiring input information describing a fault; and using retrieval-augmented generation (RAG) based on a fault database to determine diagnostic information corresponding to the input information describing the fault, wherein the fault database stores causal chains related to the fault information.

[0005] In this way, because the fault database stores causal chains related to fault information, that is, it stores richer and more comprehensive information related to faults, the root cause of the fault can be quickly determined based on the causal chain during the process of generating diagnostic information through enhanced retrieval. This allows for faster fault location, improves the efficiency of fault handling, reduces reliance on maintenance personnel, and thus reduces costs.

[0006] In some implementations, determining diagnostic information includes: generating prompts related to the input information describing the fault based on a fault database; and using a large RAG model to generate diagnostic information corresponding to the input information describing the fault based on the prompts.

[0007] In this way, embodiments of the present disclosure can generate prompts based on a fault database, thereby enabling large models to be adapted to fault diagnosis problems and thus obtaining more accurate diagnostic information.

[0008] In some implementations, generating prompts includes: using a word embedding model to convert the input information describing the fault into structured input information; identifying multiple information items associated with the structured input information from a fault database through vector retrieval; and generating prompts by reordering the multiple information items. Optionally, the reordering is performed using a reordering model or a large language model.

[0009] In this way, by reordering, information items at the top of the list can be filtered out, thereby enhancing the generation of prompts. Because causal chains are stored in the fault database, the efficiency and accuracy of generating prompts through reordering during the online diagnostic phase can be improved. Thus, using reordering in enhanced retrieval generation can improve the accuracy and quality of the responses.

[0010] In some implementations, causal chains related to fault information can represent the causal relationship between different fault times. Therefore, causal chains can be used to quickly determine the root cause of a fault, achieving efficient fault localization.

[0011] In some implementations, the fault database is obtained by: acquiring data related to system operation; and using causal analysis to generate the fault database based on that data.

[0012] In this way, causal analysis can enable the fault database to store richer information, making it easier to query the fault database later, thereby facilitating the control and analysis of faults in complex systems.

[0013] In some implementations, the data includes at least one of the following: log document data, trace call data, monitored key performance indicator (KPI) data, or historical fault diagnosis data. In other implementations, the data includes data from actual fault scenarios, as well as data from fault scenarios generated through binary instrumentation or fault injection.

[0014] In this way, richer data can be collected, thereby ensuring that the fault database built on the data can record fault-related information more comprehensively and completely.

[0015] In some implementations, obtaining data related to system operation includes retrieving data obtained using event extraction methods. In this way, event extraction can solve the problem of the difficulty in constructing a fault database.

[0016] In some implementations, the data can be preprocessed to obtain preprocessed data, which can then be used as input to an event extraction method. For example, fault information can be determined using an event extraction method.

[0017] This approach, by preprocessing the data, facilitates adaptation during event extraction.

[0018] In some implementations, preprocessing includes converting text data into vector data using a word embedding model. In other implementations, preprocessing includes state monitoring, time-series analysis, and data fusion. For example, data fusion includes using causal convolutional layers to perform multimodal fusion of different data for unified encoding.

[0019] In this way, the preprocessed data can be uniformly encoded, making it easier for subsequent event extraction operations.

[0020] In some implementations, event extraction methods include template-matching-based event extraction methods or deep learning-based event extraction methods. In other implementations, causal analysis includes Bayesian causal inference or rating-based causal discovery.

[0021] In some implementations, the fault database is updated based on new fault information. Thus, by updating the fault database, it is possible to adapt to the scaling of cloud or computing clusters, and the solutions of the embodiments of this disclosure are universal.

[0022] In some implementations, the fault database also stores fault information and diagnostic strategies for that fault information.

[0023] In a second aspect of this disclosure, an apparatus for fault diagnosis is provided, the apparatus comprising: an acquisition module configured to acquire input information describing a fault; and a diagnostic information determination module configured to determine diagnostic information corresponding to the input information describing the fault using retrieval enhancement generation based on a fault database, wherein the fault database stores causal chains related to the fault information.

[0024] The apparatus may also include one or more other modules for performing the methods according to the various implementations of the first aspect above.

[0025] In a third aspect of this disclosure, a computing device is provided, comprising: one or more processors configured to perform the method according to the first aspect above or any implementation thereof.

[0026] In some implementations, the computing device further includes a memory storing instructions executable by one or more processors, which, when executed by the processors, cause the computing device to implement the methods described in the first aspect above or any implementation thereof. The memory can also be used to store data such as data associated with system operation, fault databases, etc. The memory can also be used to store word embedding models, causal analysis algorithms, reordering models, large models, etc.

[0027] In a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the operation of the method according to the first aspect or any implementation thereof.

[0028] In a fifth aspect of this disclosure, a computing device cluster is provided, including at least one computing device. Each of the at least one computing device includes a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the operations of the methods in the first aspect or any implementation thereof described above.

[0029] In a sixth aspect of this disclosure, a chip or chip system is provided. The chip or chip system includes processing circuitry configured to perform operations according to the methods described in the first aspect or any implementation thereof.

[0030] In a seventh aspect of this disclosure, a computer program or computer program product is provided. The computer program or computer program product is tangibly stored on a computer-readable medium and includes computer-executable instructions that, when executed, implement the operations according to the method of the first aspect or any implementation thereof.

[0031] It should be understood that the technical effects in the first aspect and its various implementations also apply to each of the second to seventh aspects, therefore the technical effects of the second to seventh aspects will not be described again in this paper.

[0032] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0033] The above and other features, advantages and aspects of the embodiments of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description.

[0034] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0035] Figure 1 A schematic diagram of intelligent operation and maintenance is shown;

[0036] Figure 2 This disclosure illustrates example scenarios in which some embodiments can be applied;

[0037] Figure 3 A schematic flowchart of a method for determining a fault database according to some embodiments of the present disclosure is shown;

[0038] Figure 4 A schematic diagram of a causal chain according to some embodiments of the present disclosure is shown;

[0039] Figure 5 A schematic flowchart of a method for fault diagnosis according to some embodiments of the present disclosure is shown;

[0040] Figure 6 Schematic flowcharts are shown according to some embodiments of the present disclosure;

[0041] Figure 7 A schematic flowchart of an example apparatus for fault diagnosis according to embodiments of the present disclosure is shown;

[0042] Figure 8 A schematic flowchart of an example apparatus for determining a fault database according to embodiments of the present disclosure is shown;

[0043] Figure 9 A schematic block diagram of an example device that can be used to implement embodiments of the present disclosure is shown; and

[0044] Figure 10 A schematic block diagram of an example computing device cluster that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0045] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0046] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. The term "and / or" means at least one of the two items associated therewith. For example, "A and / or B" means A, B, or A and B. Other explicit and implicit definitions may also be included below. In the description of embodiments of this disclosure, unless expressly stated to the contrary, "a plurality" means at least two, i.e., two or more.

[0047] As cloud and computing clusters grow larger, frequent failures significantly increase the cost of manual maintenance. Intelligent operation and maintenance (O&M) fault diagnosis technology, combined with big data, modeling and simulation analysis, and machine learning, enables intelligent fault screening and diagnosis. This helps operations personnel quickly identify, handle, and resolve performance degradation and operational interruptions, shortening recovery time and reducing operating costs. For example, fault recovery time can be characterized by mean time to recovery (MTTR), which represents the average time it takes for an object to transition from a faulty state to a working state.

[0048] Figure 1 This diagram illustrates Intelligent Operations 100. Intelligent operations, such as those defined as Algorithm IT Operations (AIOps), can automate operations by leveraging operational algorithms. For example, it can use artificial intelligence (AI) capabilities to automatically process and streamline the workflow of IT service management and operations.

[0049] Intelligent operations and maintenance can collect and aggregate massive amounts of data continuously generated by multiple IT infrastructure components, application requirements and performance monitoring tools, and service ticket systems. It can intelligently identify "signals" from the "noise," recognize important events and patterns related to system performance and availability issues, diagnose root causes, and report them to IT and development operations teams so that they can respond quickly and take remedial measures, or in some cases, resolve issues automatically without human intervention.

[0050] However, in cloud or computing scenarios, the management objects range from thousands to tens of thousands or even hundreds of thousands of network interface cards (NICs), increasing the scale of NIC management by tens of times. Furthermore, the number of paths for traffic forwarding between nodes increases exponentially with the number of network nodes, making fault localization more difficult. For example, real-time analysis is required in actual business operations, increasing the granularity of device metric collection from minutes to milliseconds. Fault analysis also needs to be combined with analysis of actual forwarding traffic flows, resulting in a nearly thousand-fold increase in data volume. Additionally, high network reliability and high bandwidth require load balancing for forwarding traffic, with inter-node traffic selecting paths via hash algorithms. The probability of forwarding paths increases exponentially with the number of network nodes, making traditional fault localization methods time-consuming and heavily reliant on the personal experience of operations and maintenance personnel.

[0051] In view of this, embodiments of the present disclosure provide a fault diagnosis scheme. In this scheme, a fault database storing causal chains related to fault information is used to obtain diagnostic information corresponding to the input information describing the fault through retrieval-enhanced generation. The fault database can store richer and more comprehensive fault-related information, including the causal chains of the fault. Therefore, during the process of determining diagnostic information through retrieval-enhanced generation, the root cause of the fault can be quickly determined based on the causal chains, enabling faster fault localization, improving the efficiency of fault handling, reducing reliance on maintenance personnel, and thus lowering costs.

[0052] Figure 2 Example scenarios 200 in which some embodiments of this disclosure can be applied are illustrated. Scenario 200 may include a cluster, such as computer devices 201-1 to 201-M and / or clouds 202-1 to 202-N. It is understood that the magnitudes of M and N can be arbitrary, such as thousands, tens of thousands, hundreds of thousands, or even larger. Any cloud 202 may be a virtual machine, a database, storage space, etc. The cluster in the embodiments of this disclosure may also be referred to as a system; it is understood that the terms cluster and system are used interchangeably in the following text.

[0053] Different computer devices 201 can be located in different physical locations. For example, computer device 201-1 and computer device 201-M are located in different locations and form part of a cluster through network 250. Different computer devices 201 can also be located in the same physical location. For example, computer device 201-1 and computer device 201-2 are connected to each other and are located in the same location, forming part of a cluster.

[0054] Different clouds 202 can reside on the same or different physical entities. For example, a portion of the software or program of a cloud 202 can be installed on one physical device, while another portion of the software or program can be installed on another physical device. Alternatively, a portion of the software or program of a cloud 202 can also be installed on a computer device 201.

[0055] It should be understood that, despite Figure 2 The image simultaneously shows computer device 201 and cloud 202. In other scenarios, computer device 201 can be omitted to form a cloud cluster, or cloud 202 can be omitted to form a computer cluster. Although Figure 2 The present disclosure illustrates network 250, but the embodiments of the present disclosure do not limit the form and implementation of network 250. For example, the connection method and network type between different computer devices 201 / cloud 202 may be different.

[0056] Understandably, Figure 2 The scenario 200 shown is merely one example of an embodiment of this disclosure that can be applied to. Embodiments of this disclosure can also be applied to other scenarios, such as various clusters that include multiple objects.

[0057] The methods of embodiments of this disclosure can be performed by one or more devices for managing or maintaining a cloud or computer cluster. Exemplarily, the device used to perform the methods of embodiments of this disclosure can be a single device, such as a device with management and maintenance functions within a cluster, or a separate device independent of the cluster. Exemplarily, multiple devices used to perform the methods of embodiments of this disclosure can be physically located in the same or different locations; for example, multiple devices may all belong to the cluster, or none may belong to the cluster, or some may belong to the cluster while others do not.

[0058] Figure 3 A schematic flowchart of a method 300 for determining a fault database according to some embodiments of the present disclosure is shown. Figure 3 As shown, in box 310, data related to system operation is acquired. In box 220, causal analysis is used to generate a fault database based on the data related to system operation, where the fault database stores causal chains related to fault information.

[0059] For example, the system can be as follows Figure 2 The cluster shown, correspondingly, the system operation can refer to... Figure 2The cloud or computing cluster shown represents the operation of all or some of the objects included. For example, data items during system operation can be recorded by an intelligent operations and maintenance system or devices deployed with such systems. For instance, existing or future methods can be used to collect data related to system operation.

[0060] For example, data associated with system operation may include some or all of the following: log document data, tracing call data, monitored KPI data, or historical fault diagnosis data. In some examples, data associated with system operation may include monitoring data, such as log document data, tracing call data, and monitored KPI data. In some examples, data associated with system operation may include historical diagnostic data, such as historical fault diagnosis data obtained manually or automatically or semi-automatically.

[0061] In some embodiments of this disclosure, an event extraction method can be used to obtain fault data based on data associated with system operation. Alternatively, the data associated with system operation can be obtained after using an event extraction method.

[0062] The embodiments of this disclosure relate to event extraction methods, a crucial task in natural language processing, aiming to extract meaningful event information from text. Event extraction methods can be implemented using various algorithms, such as rule-based, template-matching, statistical, and deep learning (or machine learning) based methods. It should be noted that the embodiments of this disclosure do not limit the specific implementation of the event extraction method; for example, existing algorithms or future-developed algorithms can be used. Furthermore, the embodiments of this disclosure do not limit the name of the event extraction method; it can be referred to as adaptive event extraction, multivariate adaptive event extraction, fault information extraction, adaptive pre-extraction, key information extraction, or other names.

[0063] The input (or parameter input) of the event extraction method may include data associated with system operation, or data obtained by preprocessing data associated with the system, and the output may include fault information. For example, the event information with specific meaning extracted by the event extraction method is fault information.

[0064] Data associated with system operation can include log document data, which may also be called business logs or simply logs. For example, log document data may record some or all of the following: operation data, fault data, alarm data, etc. Data associated with system operation can also include trace call data, such as the number of requests, error rate, average latency, latency distribution, etc. Data associated with system operation can include key performance indicator (KPI) data. Data associated with system operation can also include data from historical operation and maintenance cases, i.e., existing fault operation and maintenance data or historical diagnostic data. For example, historical operation and maintenance cases may record historical operation and maintenance management logs, such as anomaly detection data, root cause diagnosis data, etc.

[0065] In some embodiments of this disclosure, the data associated with system operation may include data from actual fault scenarios in the live network, and optionally, data after manual intervention. For example, data from actual fault scenarios may be real data of faults that actually occur during system operation. For example, data after manual intervention may be data from fault scenarios generated through manual intervention. Exemplarily, fault data in typical scenarios can be collected through methods such as binary instrumentation or fault injection. In this way, richer fault data can be collected.

[0066] Binary instrumentation refers to inserting new code at any point in an existing binary program and observing or modifying the behavior of the binary program in some way, including static binary instrumentation or dynamic binary instrumentation.

[0067] Fault injection is a software and hardware testing technique that assesses a system's reliability and robustness by simulating or forcing unexpected error conditions during operation. The main purpose of fault injection is to detect the system's behavior in abnormal states, identify potential design flaws, recovery mechanism failures, and security vulnerabilities, thereby improving system design and enhancing its resistance to real-world failures. In embodiments of this disclosure, hardware fault injection (e.g., contact fault injection) can be employed, using direct contact between circuit pins and the target system hardware. Exemplarily, direct contact between circuit pins and the target system hardware can also be referred to as "pin-level injection," which can use active probes or pluggable techniques to alter the current and voltage at the pins, thereby reducing the impact on the target system.

[0068] For example, data associated with system operation can be preprocessed to obtain preprocessed data. Preprocessing may include using word embedding models to convert unstructured information into structured information; for instance, word embedding models can be used to convert text data in monitoring data into vector data. For example, word embedding models can be used to preprocess log document data, trace call data, monitoring KPI data, and historical maintenance case data, respectively. Preprocessing may also include time-series analysis of status monitoring and data fusion; for example, vector data can be fused according to time sequence. For instance, a multimodal fusion model can be used for unified encoding to obtain the input. This allows for adaptation to subsequent processing operations.

[0069] The embodiments of this disclosure relate to word embedding models, which are widely used representation methods in natural language processing. Word embedding transforms discrete words into vectors in a continuous vector space, and the resulting vectors reflect the semantic and syntactic relationships between words. Optionally, the word embedding model is based on word embedding algorithms such as Word2Vec and GloVe. Optionally, word embedding can also be referred to as corpus embedding, vectorization, or encoding, etc.

[0070] The embodiments of this disclosure relate to multimodal fusion models, which refer to technical methods for combining data from different modalities to improve information processing and understanding capabilities. Multimodal data typically possesses different physical properties and informational characteristics; by fusing these multimodal data, a more comprehensive and accurate understanding can be obtained. Optionally, in the embodiments of this disclosure, log document data, trace call data, monitoring KPI data, historical operation and maintenance case data, etc. (or their vector data transformed by a word embedding model) can be considered as data with different modalities (i.e., multivariate data).

[0071] For example, log document data, trace call data, monitoring KPI data, historical operation and maintenance case data, etc. (or vector data after they are transformed by word embedding model) can be used to obtain time series data through time series analysis. Then, multimodal fusion can be performed using causal convolutional layers for unified encoding, thereby obtaining the input of the event extraction method.

[0072] For example, the output, i.e., fault information, can be obtained using template matching-based event extraction methods, deep learning-based event extraction methods, etc. For instance, fault information can be referred to as the status information of critical faults, and a fault information item can record one or more of the following: fault event, time of fault occurrence, location of fault occurrence, and fault description information, etc.

[0073] In some embodiments of this disclosure, causal chains related to fault information can be determined using causal analysis based on fault information. Exemplarily, causal chains may also be referred to as fault propagation models, fault propagation chains, propagation chains, or other names, and this disclosure is not limited thereto. Exemplarily, causal chains related to fault information can be used to represent causal relationships between faults. As an example of a causal chain, event A (cause) leads to event B1 (effect), and events B1 (cause) and B2 (cause) lead to event C (effect). For example, a fault in high bandwidth memory (HBM) causes a fault in the network processing unit (NPU), which in turn causes a software fault (such as a training framework error), ultimately leading to the interruption of the training process.

[0074] For example, the causal chain associated with fault information can also represent the degree of influence of an event as a cause (such as event B1) on another event as a result (such as event C). For example, a number between 0 and 1 can be used as an influence coefficient to represent the degree of influence.

[0075] As an example, a causal chain can be represented as a causal graph, where nodes represent events that are either causes or effects. Causal relationships between nodes are indicated by one-way arrows, such as arrows pointing from cause to effect, and each arrow has a corresponding influence coefficient (e.g., greater than 0 and less than or equal to 1). Figure 4 A schematic diagram of a causal chain 400 according to some embodiments of the present disclosure is shown. Figure 4 The diagram illustrates nodes A1, A2, and B1, as well as the directed edges from A1 to B1 and from A2 to B1. Specifically, the directed edge from A1 to B1 indicates that A1 is a cause of B1, and the influence coefficient af1 associated with this directed edge represents the degree of influence of cause A1 on result B1. Similarly, the directed edge from A2 to B1 indicates that A2 is a cause of B1, and the influence coefficient af2 associated with this directed edge represents the degree of influence of cause A2 on result B1. It should be noted that... Figure 4 The causal chain 400 shown is merely illustrative. In reality, the causes leading to B1 may include multiple factors. A1 or A2 may be the result of other events, and B1 may be the cause of other events, etc. This disclosure does not limit this. It is understood that causal chains can be represented in forms other than causal diagrams, and this disclosure does not limit this.

[0076] Embodiments of this disclosure relate to causal analysis, which can be used to analyze the causal relationships between multiple different information items. In some examples, causal analysis may include causal association analysis or causal inference analysis. Exemplarily, causal analysis can be performed using techniques such as Bayesian causal inference and scored causal discovery. For example, Bayesian causal inference can perform time-series causal relationship learning to determine whether a causal relationship exists between faults at different times. For example, scored causal discovery can determine the highest-scoring causal graph based on scores, where the causal graph can represent the causal relationship and degree of influence between faults. Exemplarily, during causal analysis, methods such as cluster analysis and association mining can be used to identify multiple key components within the system and their inter-component dependencies. For example, multiple key components may include fault information. Exemplarily, diffusion models can be used to analyze the impact of fault diffusion, thereby constructing causal chains related to fault information. Taking diffusion models as an example, a trained diffusion model suitable for embodiments of this disclosure can be generated through a forward training process and a backward evaluation process.

[0077] In some embodiments of this disclosure, causal chains related to fault information can be stored in a fault database. That is, the fault database can store causal chains related to fault information. The fault database can also be called a fault vector database, a fault vector knowledge base, or other names, and this disclosure is not limited thereto. Exemplarily, the fault database also stores fault information and / or diagnostic strategies. For example, diagnostic strategies can be determined using currently known or future-developed methods, and the fault information and diagnostic strategies can be stored. In some examples, the fault database includes corpus information about faults, which includes: fault information, fault diagnostic strategies, causal chains related to the fault information, etc.

[0078] Understandably, due to the difficulty in extracting fault data, its redundancy, and the high number of errors, coupled with the complexity of fault causal relationships, traditional mathematical and statistical methods are not readily available to obtain accurate and in-depth understanding of fault causal relationships, thus hindering the control and analysis of complex system faults. In the embodiments of this disclosure, by combining... Figure 3 The described method can solve the problem of difficult construction of fault database through event extraction, and can make the fault database store richer information through causal analysis, which facilitates subsequent querying of the fault database, thereby facilitating the management and analysis of faults in complex systems.

[0079] In the embodiments of this disclosure, as shown Figure 3 The process of generating the fault database shown can be performed during the offline preparation phase of the business, and can be referred to as the offline database creation process.

[0080] In some embodiments of this disclosure, the fault database can be further supplemented and / or updated. For example, as the number of network nodes in the cluster increases, new fault information may be monitored, and the fault database can then be updated in real time or periodically based on the new fault information. Exemplarily, the process of updating the fault database based on new fault information is similar to the aforementioned process of using causal analysis to obtain the causal propagation chain, and will not be described in detail here.

[0081] In this way, the embodiments of this disclosure can adapt to the growth in the scale of cloud or computer clusters, and can obtain a fault database accordingly even as the number of network nodes continues to increase.

[0082] Figure 5 A schematic flowchart of a method 500 for fault diagnosis according to some embodiments of the present disclosure is shown. In block 510, input information describing a fault is obtained. In block 520, diagnostic information corresponding to the input information describing the fault is determined using enhanced generation based on a fault database retrieval.

[0083] In some embodiments of this disclosure, a fault database may also be acquired before block 510, for example, the acquired fault database may be obtained through... Figure 3 The fault database generated / established by method 300 shown, or it can be generated through... Figure 3 The fault database generated / established by method 300 is further updated based on the fault database.

[0084] In some examples, input information describing the fault can be obtained, such as user input or information generated by the application during runtime. For instance, the input information describing the fault may include a question describing the fault condition or a problem to be diagnosed. For example, the input information describing the fault may be a description of a fault situation during the actual operation of the cluster. For example, the input information describing the fault may be a question posed by a user that includes a description of the fault condition.

[0085] Embodiments of this disclosure relate to retrieval-enhanced generation, a generative method oriented towards large models (LMs). Retrieval-enhanced generation combines a retrieval model and a generative model to assist in tasks such as text generation. Exemplarily, retrieval-enhanced generation combines the retrieval and generation processes by retrieving a large amount of relevant information before generating text output, and then inputting this retrieved information as context into the generative model, thus mitigating the illusion problem.

[0086] Retrieval-enhanced generation based on a fault database can be interpreted as referencing / accessing / using a large RAG model (hereinafter referred to as the large model) of the fault database before generation. Specifically, based on the fault database, prompts related to the input information describing the fault can be generated; then, the large RAG model is used to generate diagnostic information corresponding to the input information describing the fault based on the prompts. In this way, prompts can be generated based on the fault database, allowing the large model to adapt to fault diagnosis problems and thus obtain more accurate diagnostic information. Since the fault database stores causal chains related to fault information, the corresponding prompts can guide the large model to more quickly and accurately determine the root cause of the fault.

[0087] For input information describing a fault, a word embedding model can be used to transform it into structured input information, such as vector data. For example, based on the structured input information, vector retrieval can be used to determine multiple information items associated with the input information from a fault database. Optionally, the vector retrieval method can be nearest neighbor retrieval, approximate nearest neighbor retrieval, or other methods, which are not limited in this disclosure. In this way, by using a word embedding model to transform the input information into structured input information, the efficiency of retrieval can be ensured; for example, for vector data, vector retrieval can more quickly determine multiple information items associated with the input information.

[0088] Optionally, the multiple information items identified by retrieval and associated with the input information may also be referred to as multiple associated corpora or other names, which are not limited in this disclosure. For example, the number of multiple information items can be an integer, and the value of the integer can be predefined. For example, the correlation between each of the multiple information items and the structured input information exceeds an association threshold, and the association threshold can be predefined. For example, each of the multiple information items identified from the fault database may include corpus information of a fault. Taking fault F as an example, the corpus information of fault F may include: fault information of fault F, causal chains related to fault F, etc.

[0089] Prompt information can be generated based on the re-ranking of multiple information items. Re-ranking assesses the relevance between multiple contexts and prioritizes the contexts most likely to provide accurate and relevant answers. In embodiments of this disclosure, the multiple contexts considered in the re-ranking can be considered as multiple information items. In this way, re-ranking allows for the filtering of top-ranked fault information items, thereby enhancing the generation of prompt information. Thus, using re-ranking in RAG can improve the accuracy and quality of responses.

[0090] For example, fault information can be reordered for multiple information items associated with the input information, such as using a reordering model or a large language model (LLM). Optionally, the reordering model takes the query and context as input, for example, considering the interaction features between the context and the query, and uses cross-entropy loss to optimize and directly output a similarity score. Optionally, LLM can capture semantic information more comprehensively by thoroughly understanding all context and queries. For example, reordering based on LLM can include: fine-tuning the LLM using a reordering task; prompting the LLM to reorder; and using LLM for data augmentation during training.

[0091] For example, using a large model, diagnostic information corresponding to the input information describing the fault can be generated based on the prompts. Optionally, the diagnostic information can be the output of the model, or it can be understood as the answer to the input information, etc. For example, the input information describes the fault condition, and the diagnostic information can provide root cause localization and diagnostic assistance, etc.

[0092] The large model in this embodiment, which can be referred to as an end-to-end fault diagnosis information output model, can be fine-tuned to improve the accuracy of fault diagnosis. As described above, since the prompt information is generated based on the reordering of multiple information items, and these multiple information items are corpus information of faults determined from the fault database, fine-tuning the large model based on this prompt information enables the fine-tuned large model to be suitable for fault diagnosis tasks. Exemplarily, the prompt information can guide the fine-tuned large model to output diagnostic information. Optionally, the diagnostic information may include one or more of the following: fault location results, fault causes, fault resolution strategies, etc.

[0093] In the embodiments of this disclosure, as shown Figure 5 The fault diagnosis process shown can be performed during the actual operation of the business, and may be referred to as an online diagnosis process.

[0094] In this way, embodiments of this disclosure can generate prompting information based on a fault database, thereby enabling large models to adapt to fault diagnosis problems targeting clusters. Specifically, the fault database is generated based on cluster operation data and stores causal chains related to fault information, allowing the generated prompting information to better reflect the analysis of the input problem, and thus enabling large models to output diagnostic information corresponding to the fault problems of the cluster.

[0095] The embodiments of this disclosure can generate rich prompts based on queries to a fault database, thereby guiding a large model to generate more accurate and comprehensive output. In this way, during real-time operation, for diagnostic questions input by the user that include a description of the fault condition, the embodiments of this disclosure can filter key fault information from a fault database containing causal relationships by reordering the data, thereby determining prompts and outputting diagnostic information through a large model.

[0096] Figure 6 A schematic flowchart of an overall process 600 according to some embodiments of the present disclosure is shown. Process 600 includes offline library construction 610 and online diagnostics 620. Exemplarily, offline library construction 610 can be referred to as in conjunction with... Figure 3 The described method 300 can be implemented, and online diagnosis 620 can be referred to as follows. Figure 5 The method described is implemented using 500.

[0097] In the offline database construction stage 610, parameter input 601 can be obtained and converted into vector data through a word embedding model 602. Subsequently, key fault information can be determined through event extraction 603. Furthermore, the fault propagation chain 605 can be obtained through causal analysis 604. Thus, the fault database 655 is generated or updated.

[0098] In the online diagnosis stage 620, the input information 621 describing the fault can be converted into vector data through a word embedding model 622. Subsequently, multiple related corpora 624 can be selected from the fault database 655 through vector retrieval 623. And prompt information is generated by reordering 625. Thus, the output, namely the diagnostic information 628, is obtained using a large model 626.

[0099] It should be noted that word embedding model 602 and word embedding model 622 may have completely identical model parameters and model structures, or they may have partially different model parameters or structures, but both achieve essentially the same function. In the embodiments of this disclosure, the word embedding models 602 / 622 used may be deployed on the same or different devices / systems as the devices / systems performing the methods of the embodiments of this disclosure. For example, a word embedding model located on another different device, cloud storage, or system may be invoked via an interface. Similarly, event extraction model 603 may be implemented by deploying an event extraction model or invoking an event extraction model from another device / system, and causal analysis model 604 may be implemented by deploying a causal analysis model or invoking a causal analysis model from another device / system; this disclosure does not limit this.

[0100] In this way, the embodiments of this disclosure can utilize a fault database that stores causal chains related to fault information to determine diagnostic information corresponding to the input information describing the fault, thereby achieving faster fault location and improving the efficiency of fault resolution.

[0101] Storing causal chains (such as fault causal information) in the fault database can improve the efficiency and accuracy of generating alerts through reordering during the online diagnostic phase. For example, the time required to generate alerts can be reduced from minutes to seconds. With the improved efficiency and accuracy of alert generation, large models also achieve performance improvements in fault root cause localization.

[0102] In this way, the embodiments of this disclosure provide a scheme for constructing a fault database based on causal analysis and enhancing a large-scale model for fault diagnosis. For scenarios such as fault location, the large-scale model tool is used to quickly and efficiently assist in the diagnosis of cluster faults, thus shortening the time required for manual inspection and location after massive faults occur in large-scale clusters, and further optimizing intelligent operation and maintenance.

[0103] It should be understood that although the above embodiments are illustrated using cloud or computing clusters as examples, the embodiments of this disclosure are not limited thereto, and can be used, for example, for database clusters, hybrid clusters of computing devices and the cloud, etc. It should be understood that the embodiments of this disclosure can be applied to the operating system of computing devices (such as terminal devices), for example, for high-performance computing (HPC), AI computing, general-purpose computing, and other fields.

[0104] It should be understood that the methods, situations, categories, and classifications of embodiments in this disclosure are for the convenience of description only and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other where logically consistent.

[0105] It should also be understood that the foregoing is merely to help those skilled in the art better understand the embodiments of this disclosure, and is not intended to limit the scope of the embodiments of this disclosure. Those skilled in the art can make various modifications, variations, or combinations based on the foregoing. Such modifications, variations, or combinations are also within the scope of the embodiments of this disclosure.

[0106] It should also be understood that the above description focuses on highlighting the differences between the various embodiments. Similarities or commonalities can be referenced or learned from each other, and for the sake of brevity, they will not be elaborated here.

[0107] Figure 7A schematic block diagram of an example apparatus 700 for fault diagnosis according to some embodiments of the present disclosure is shown. As illustrated, the apparatus 700 may include an acquisition module 710 and a diagnostic information determination module 720. The acquisition module 710 may be configured to acquire input information describing a fault. The diagnostic information determination module 720 may be configured to determine diagnostic information corresponding to the input information describing the fault using enhanced generation based on a fault database, wherein the fault database stores causal chains related to the fault information.

[0108] In some implementations, the diagnostic information determination module 720 can be configured to: convert the input information describing the fault into structured input information using a word embedding model; determine multiple information items associated with the structured input information from a fault database through vector retrieval; generate prompt information by reordering the multiple information items; and generate diagnostic information corresponding to the input information describing the fault based on the prompt information using a large model. For example, the reordering is performed using a reordering model or a large language model.

[0109] In some implementations, the fault database can be obtained by: acquiring data associated with system operation; and generating the fault database based on the data associated with system operation using causal analysis, wherein the fault database stores causal chains related to fault information. Exemplarily, device 700 may also include a fault database generation module; alternatively, the fault database may be generated by another device (device 800 described below). It is understood that the methods by which device 700 or device 800 generates the fault database are similar, and to avoid repetition, they will not be repeated here.

[0110] For example, the fault database is updated based on new fault information. For example, the fault database also stores fault information and diagnostic strategies based on the fault information.

[0111] Figure 8 A schematic flowchart of an example apparatus 800 for determining a fault database according to an embodiment of the present disclosure is shown. As illustrated, the apparatus 800 may include an acquisition module 810 and a causal analysis module 820. The acquisition module 810 may be configured to acquire data associated with system operation. The causal analysis module 820 may be configured to generate a fault database based on the data associated with system operation using causal analysis, wherein the fault database stores causal chains related to fault information.

[0112] Optionally, the data associated with system operation includes at least one of the following: log document data, trace call data, monitored KPI data, or historical fault diagnosis data. Optionally, the data associated with system operation includes data under actual fault scenarios, as well as data under fault scenarios generated through binary instrumentation or fault injection.

[0113] In some implementations, the acquisition module 810 can be configured to acquire data related to system operation obtained using an event extraction method. Optionally, the event extraction method includes a template matching-based event extraction method or a deep learning-based event extraction method.

[0114] In some implementations, causal analysis includes Bayesian causal inference or rating causal discovery.

[0115] For example, the fault database is updated based on new fault information. For example, the fault database also stores fault information and diagnostic strategies based on the fault information.

[0116] The division of modules or units in the embodiments of this disclosure is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the disclosed embodiments may be integrated into one unit, exist as separate physical entities, or two or more units may be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0117] For example, Figure 7 and Figure 8 The devices 700 and 800 can be implemented as computing devices or device clusters, or as chips or chip systems within devices. Figure 7 and Figure 8 Devices 700 and 800 in the above-mentioned configuration can be used to achieve the above-mentioned combination. Figures 3 to 6 For the sake of brevity, the corresponding operations in each process described herein will not be repeated here.

[0118] Figure 9 A schematic block diagram of an example device 900 that can be used to implement embodiments of the present disclosure is shown. Exemplarily, Figure 9 The device 900 in this disclosure can be implemented as a computing device, or as a chip or chip system in a computing device, and the embodiments herein do not limit this.

[0119] As shown in the figure, device 900 includes a processor 901 and a memory 902. The memory 902 can store computer program instructions, as well as various programs and data required for the operation of device 900. The processor 901 and the memory 902 are connected to each other via a bus 903.

[0120] The processor 901 may include a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in the memory 902. The memory 902 may include, for example, read-only memory (ROM), random access memory (RAM), and storage units such as magnetic disks and optical disks.

[0121] Device 900 may further include an input unit 904 and an output unit 905, which may be connected to bus 903, for example, via an input / output (I / O) interface. Input unit 904 may include, for example, a keyboard, mouse, etc. Output unit 905 may include, for example, various types of displays, speakers, etc. Device 900 may also include a communication unit (not shown), which allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. Optionally, input unit 904 and output unit 905 may be implemented as input / output units, such as a touch screen.

[0122] A CPU can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Examples of such implementations include, but are not limited to, a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc., which may accordingly be referred to as a computing unit. The CPU performs the various methods and processes described above. For example, in some embodiments, the various processes described above may be implemented as computer software programs tangibly contained in a computer-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM and / or RAM and / or communication units. When the computer program is loaded into ROM and / or RAM and executed by the CPU, one or more steps of the processes described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the various processes described above by any other suitable means (e.g., by means of firmware).

[0123] Figure 10 A schematic block diagram of an example computing device cluster 1000 that can be used to implement embodiments of the present disclosure is shown. Figure 10 As shown, the computing device cluster 1000 includes at least one computing device, such as... Figure 10 The computing device 1050-1 and computing device 1050-2 shown in the figure (collectively referred to as computing device 1050)

[0124] The computing device 1050 can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device 1050 can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone. As shown in the figure, the computing device 1050 includes a processor 1004, a memory 1006, and a communication interface 1008 connected via a bus 1002. In some examples, the computing device 1050 can be implemented as follows: Figure 9 The device shown is 900.

[0125] One or more computing devices 1050 in the computing device cluster 1000 can be connected via a network, which can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 As shown, computing device 1050-1 and computing device 1050-2 are connected via network 1060. Specifically, different computing devices 1050 are connected to network 1060 through their respective communication interfaces 1008.

[0126] In some possible implementations, the memory 1006 of one or more computing devices 1050 in the computing device cluster 1000 may store the same instructions for executing the aforementioned combination. Figures 3 to 6 The method described. For example, a single computing device 1050 can implement the aforementioned combination. Figures 3 to 6 The method described.

[0127] In some possible implementations, the memory 1006 of one or more computing devices 1050 in the computing device cluster 1000 may respectively store information for performing the aforementioned combination. Figures 3 to 6 Part of the instructions for the described method. For example, the memory of a computing device may store instructions for performing the aforementioned combination. Figure 3 The instructions for the described method can be stored in the memory of another computing device for performing the aforementioned combination. Figure 3 The instructions for the described method. In other words, a combination of multiple computing devices can jointly execute the aforementioned combination. Figures 3 to 6 The method described.

[0128] In other possible implementations, the combination can also be achieved by combining multiple computing devices. Figures 3 to 6 The method described.

[0129] It should be noted that the memories 1006 in the different computing devices 1050 within the computing device cluster 1000 can store different instructions, each used to execute a portion of the functions of the device 1000. That is, the instructions stored in the memories 1006 of the different computing devices 1050 can respectively implement… Figure 10 The functions of one or more modules in the system.

[0130] It should be noted that, although in Figure 10 Two computing devices are shown, namely computing device 1050-1 and computing device 1050-2. However, it is understood that in specific implementations, the methods according to the various embodiments of this disclosure may be implemented by only one computing device, or the methods according to the various embodiments of this disclosure may be implemented by two or more computing devices together. This disclosure is not limited in this respect.

[0131] Embodiments of this disclosure also provide a chip, which may include an input interface, an output interface, and processing circuitry. In embodiments of this disclosure, the input and output interfaces can be used to complete the interaction of signaling or data, while the processing circuitry can be used to generate and process the signaling or data information.

[0132] Embodiments of this disclosure also provide a chip system including a processor for supporting a computing device to implement the functions involved in any of the foregoing embodiments. In one possible design, the chip system may further include a memory for storing necessary program instructions and data, which, when executed by the processor, cause the device on which the chip system is mounted to implement the methods involved in any of the foregoing embodiments. Exemplarily, the chip system may consist of one or more chips, or may include chips and other discrete devices.

[0133] Embodiments of this disclosure also provide a processor for coupling with a memory storing instructions that, when executed by the processor, cause the processor to perform the methods and functions involved in any of the above embodiments.

[0134] Embodiments of this disclosure also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions involved in any of the embodiments described above.

[0135] Embodiments of this disclosure also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods and functions involved in any of the above embodiments.

[0136] Embodiments of this disclosure also provide a fault database that stores causal chains related to fault information. Exemplarily, this fault database is generated and / or updated based on causal analysis.

[0137] Embodiments of this disclosure also provide a large model, which can be a RAG-based model. Exemplarily, this large model can be constructed based on the aforementioned fault database.

[0138] Generally, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0139] This disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods as described above with reference to the accompanying drawings. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0140] Computer program code used to implement the methods of this disclosure may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the computer or other programmable data processing apparatus, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be performed. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0141] In the context of this disclosure, computer program code or related data may be carried on any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and so on. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0142] A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More detailed examples of computer-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0144] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to well explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for fault diagnosis, comprising: Obtain input information describing the fault; as well as Using a fault database-based retrieval enhancement to generate a RAG, diagnostic information corresponding to the input information describing the fault is determined, wherein the fault database stores causal chains related to the fault information.

2. The method according to claim 1, wherein the fault database is obtained by: Acquire data related to system operation; and The fault database is generated based on the data using causal analysis.

3. The method according to claim 2, wherein the data includes at least one of the following: log document data, trace call data, monitored key performance indicator (KPI) data, or historical fault diagnosis data.

4. The method according to claim 2 or 3, wherein acquiring data associated with system operation includes: Retrieve data related to system operation obtained using event extraction methods.

5. The method according to any one of claims 2 to 4, wherein the causal analysis includes Bayesian causal inference or rating causal discovery.

6. The method according to any one of claims 1 to 5, wherein determining the diagnostic information comprises: Based on the fault database, generate prompt information related to the input information describing the fault; as well as Using the RAG large model, diagnostic information corresponding to the input information describing the fault is generated based on the prompt information.

7. The method according to claim 6, wherein generating the prompt information comprises: The input information describing the fault is converted into structured input information using a word embedding model. Multiple information items associated with the structured input information are determined from the fault database through vector retrieval; as well as The prompt message is generated by reordering the multiple information items.

8. The method according to any one of claims 1 to 7, wherein the fault database is updated based on new fault information.

9. The method according to any one of claims 1 to 8, wherein the causal chain associated with the fault information is used to represent the causal relationship between different faults.

10. An apparatus for fault diagnosis, comprising a module for implementing the method according to any one of claims 1 to 9.

11. A computing device, comprising: One or more processors; as well as A memory storing instructions that, when executed by the one or more processors, cause the computing device to perform the method according to any one of claims 1 to 9.

12. A computer program product storing instructions that, when executed, cause a device to perform the method according to any one of claims 1 to 9.

13. A chip or chip system configured to perform the method according to any one of claims 1 to 9.

Images