A system fault detection method, apparatus, equipment, medium, and program product

Abstract

This application provides a system fault detection method, apparatus, device, medium, and program product, relating to the field of artificial intelligence technology. The method includes: upon receiving a first input from a user, acquiring an instruction corresponding to the first input; in response to the instruction, scheduling a first intelligent agent among multiple intelligent agents to detect whether there are anomalies in the services of a target system, the multiple intelligent agents including agents located at different levels within the target system; if an anomaly is detected in the services of the target system, scheduling a second intelligent agent among the multiple intelligent agents to perform fault detection on the target system, obtaining detection results, and determining the faulty node in the target system based on the detection results. By scheduling multiple intelligent agents at different levels within the target system for fault detection, the different levels of intelligent agents can perform fault analysis for specific fault scenarios, thereby improving the accuracy of system fault diagnosis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a system fault detection method, apparatus, equipment, medium, and program product. Background Technology

[0002] The complexity of information technology (IT) systems in large enterprises is increasing daily, and system failures can lead to substantial economic losses. Therefore, leveraging artificial intelligence (AI) big data modeling techniques to quickly locate faults has become particularly important.

[0003] Currently, mainstream fault diagnosis methods mainly rely on technical frameworks based on artificial intelligence and machine learning. This involves training models on operational data to further accumulate fault characteristics and uncover fault correlations, enabling root cause localization when a fault occurs. However, as the structure of communication services becomes increasingly complex, large models lack sufficient accumulation of service characteristics, resulting in lower diagnostic accuracy when facing complex faults. Summary of the Invention

[0004] This application provides a system fault detection method, apparatus, device, medium, and program product to address the problem of low diagnostic accuracy of large models for complex faults.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a system fault detection method, the method comprising:

[0007] Upon receiving the user's first input, obtain the instruction corresponding to the first input;

[0008] In response to the instruction, the first intelligent agent is scheduled among multiple intelligent agents to detect whether there are any anomalies in the services of the target system through the target intelligent agent, wherein the multiple intelligent agents include intelligent agents located at different levels of the target system, and the target intelligent agent is used to manage the multiple intelligent agents;

[0009] If an anomaly is detected in the service of the target system, the target agent schedules a second agent among the multiple agents to perform fault detection on the target system to obtain the detection result, and determines the faulty node in the target system based on the detection result.

[0010] Secondly, embodiments of this application provide a system fault detection device, the device comprising:

[0011] The first acquisition module is used to acquire the instruction corresponding to the first input when the user's first input is received;

[0012] The first scheduling module is used to respond to the instruction and schedule the first intelligent agent among multiple intelligent agents to detect whether there is an anomaly in the service of the target system. The multiple intelligent agents include intelligent agents located at different levels of the target system, and the target intelligent agent is used to manage the multiple intelligent agents.

[0013] The detection module is used to, when an anomaly is detected in the service of the target system, schedule a second agent among the multiple agents through the target agent to perform fault detection on the target system, obtain the detection result, and determine the faulty node in the target system based on the detection result.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the system fault detection method described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the system fault detection method described in the first aspect.

[0016] Fifthly, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the system fault detection method as described in the first aspect.

[0017] In this embodiment, multiple agents at different levels of the target system are scheduled by the target agent to perform fault detection. Since agents at different levels can perform fault analysis for specific fault scenarios, the accuracy of system fault diagnosis can be improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts of a system fault detection method provided in the embodiments of this application;

[0020] Figure 2This is a second flowchart of a system fault detection method provided in the embodiments of this application;

[0021] Figure 3 This is a schematic diagram of a fault detection system for an intelligent body provided in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of a platform architecture for an intelligent agent provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram illustrating a multi-agent collaborative decision-making process provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of a knowledge base retrieval provided in an embodiment of this application;

[0025] Figure 7 This is a schematic diagram illustrating the development of an operation and maintenance plugin provided in an embodiment of this application;

[0026] Figure 8 This is a schematic diagram illustrating the development of an operation and maintenance workflow provided in an embodiment of this application;

[0027] Figure 9 This is a framework diagram of a system fault detection system provided in an embodiment of this application;

[0028] Figure 10 This is one of the ReAct technology implementation logic diagrams provided in the embodiments of this application;

[0029] Figure 11 This is the second implementation logic diagram of ReAct technology provided in the embodiments of this application;

[0030] Figure 12 This is a schematic diagram of the intelligent agent execution fault detection method provided in the embodiments of this application;

[0031] Figure 13 This is a schematic diagram of collaborative decision-making by database intelligent agents provided in an embodiment of this application;

[0032] Figure 14 This is a schematic diagram illustrating the identification of user input information provided in an embodiment of this application;

[0033] Figure 15 This is a schematic diagram of the structure of a system fault detection device provided in an embodiment of this application;

[0034] Figure 16 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Currently, the diagnosis of IT system failures includes root cause localization based on alarm data and topology data, root cause localization methods based on microservices (Tracelog), and root cause localization combined with large model fine-tuning techniques.

[0037] For example, root cause localization based on time-series anomaly alarm information involves receiving anomaly alarm information and determining the anomaly initiation node and anomaly detection time window based on this information. The anomaly detection time window includes the anomaly start time and the anomaly end time. Based on the anomaly initiation node and a preset application deployment topology (a knowledge graph-based graph topology), associated nodes are determined. Anomaly detection is then performed on each associated node from top to bottom based on the anomaly detection time window and the fault status information corresponding to each associated node, resulting in anomaly-related nodes. Finally, based on the depth of each anomaly-related node within the application deployment topology, the root cause recommendation node corresponding to the anomaly alarm information is determined. This approach avoids redundant anomaly detection and improves the efficiency of root cause localization.

[0038] The method for locating the root cause of performance anomalies in container cloud microservices based on a large language model involves: collecting and monitoring metrics for each container microservice in the container cloud and storing them in a time-series database; using the large language model to fine-tune and train a baseline model for performance anomalies based on the time-series data and constructing a global topology graph for container microservices; collecting real-time metrics for both hosts and container microservices, and using a discovery module to infer pairs of abnormal container microservices based on the baseline model; constructing subgraphs of the microservice anomaly topology based on the global topology graph; and utilizing the mind map prompts of the large language model to sort and output a list of root causes of performance anomalies in container microservices. This approach enhances the robustness, accessibility, and reliability of container cloud microservices.

[0039] A cloud data center fault root cause localization method extracts time-series data of key performance indicators from each virtual machine (VM) in the cloud data center and monitors the time-series data for anomalies in each VM. When a cloud data center fault is detected, a weighted fault impact graph is generated among the VMs. The correlation between VMs with causal relationships in the weighted fault impact graph is calculated, as well as the correlation between the initially faulty VM and other VMs. Based on the correlation and the weights in the weighted fault impact graph, the state transition probability is obtained, and the VM is walked on the weighted fault impact graph, recording the number of visits to each VM. The root cause score is calculated by combining the number of visits to each VM with the anomaly score, and the VMs are sorted according to the root cause score to finally determine the VM causing the fault. By considering the propagation delay of VMs, the accuracy of root cause localization can be improved.

[0040] However, the above solution has the following drawbacks:

[0041] When data quality is poor and the learning of relevant time-series features of operation and maintenance data is incomplete, the timeliness of the data is insufficient, i.e., there is a certain lag, which will affect the accuracy of fault diagnosis. Due to the complex structure of communication services, when the model accumulation is insufficient, the fault diagnosis process needs to be carried out across multiple levels of the system (Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS)) step by step, resulting in long processing times and affecting the efficiency and accuracy of fault diagnosis.

[0042] Because small machine learning models rely on large amounts of operational data for training and real-time inference, the training and inference costs of machine learning algorithm models are relatively high.

[0043] Furthermore, during the machine learning process, a large amount of core operational data needs to be accessed, which poses security risks such as data leakage and privacy breaches during data transmission.

[0044] This application provides a system fault detection method, apparatus, device, medium, and program product to address the problem that large models have insufficient accumulation of business features, resulting in low diagnostic accuracy when facing complex faults.

[0045] See Figure 1 , Figure 1 This is a flowchart of a system fault detection method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0046] Step 101: Upon receiving the user's first input, obtain the instruction corresponding to the first input;

[0047] Step 102: In response to the instruction, the first intelligent agent is scheduled among multiple intelligent agents to detect whether there is an anomaly in the service of the target system through the target intelligent agent, wherein the multiple intelligent agents include intelligent agents located at different levels of the target system, and the target intelligent agent is used to manage the multiple intelligent agents;

[0048] Step 103: If an anomaly is detected in the service of the target system, the target agent schedules a second agent among the multiple agents to perform fault detection on the target system to obtain the detection result, and determines the fault node in the target system based on the detection result.

[0049] The first input can be text or voice information entered by the user. By recognizing the first input, the corresponding instruction can be obtained. For example, receiving the user's instruction to "detect whether the business system is abnormal". In some implementations, a large model combined with contextual information can be used to recognize the instruction corresponding to the user's input.

[0050] In some implementations, after recognizing the instruction of the first input, the target agent can schedule the first agent to detect the services of the target system according to the instruction; in other implementations, the intent of the first input is understood through a large model, a fault diagnosis task is generated, and the target agent schedules the first agent to detect the services of the target system based on the diagnosis task.

[0051] For example, after a failure occurs in a business system, users can initiate troubleshooting by interacting with the large model and inputting their intent. After recognizing the user intent, the large model or intelligent agent will further coordinate multiple intelligent agents to conduct troubleshooting through the supervisor intelligent agent, i.e., the target intelligent agent.

[0052] The target system can be an IT system that needs to be monitored, such as an enterprise's business systems or cloud platforms. The target system can include multiple layers, such as SaaS, PaaS, and IaaS layers, each with different functions. The target intelligent agent manages multiple agents at different layers; it can also be called the supervisory intelligent agent. It schedules multiple agents to collaborate according to instructions and detects whether there are any anomalies in the target system's services.

[0053] The target agent first schedules a first agent to detect whether the target system's services are abnormal. If the services are abnormal, a second agent is then scheduled to troubleshoot the problem. The second agent may include one or more agents.

[0054] In some implementations, the target agent schedules multiple second agents at different levels to troubleshoot the problem;

[0055] In some implementations, the target agent schedules a fault-bounding agent (a second agent), which in turn schedules other second agents to further investigate the fault.

[0056] For example, the health check agent at the SaaS layer initially determines whether the service is abnormal. If an anomaly is detected, the target agent further schedules the database agent at the PaaS layer and the host agent at the IaaS layer for in-depth investigation. Through multi-agent collaborative analysis (such as database deadlock or network latency), the specific faulty node (such as a database instance or network node) is located.

[0057] The intelligent agent detection process is as follows: Figure 2 As shown.

[0058] As a specific example, such as Figure 3 As shown. For example, user intent is identified through a large language model (LLM) of the target agent, and then the target agent is used to schedule the system health check agent of the SaaS layer to determine service anomalies.

[0059] The system health check agent includes the skill to inspect the health indicators of SaaS layer services. It performs a check on the service health indicators one by one based on the business health check prompts. The health check indicators are generally the core indicators of the service, including business success rate, average business response latency, business call volume, etc. The agent checks these indicators to find out if there are any anomalies. If there are any anomalies, the agent will summarize the health check results for the supervisor agent to make scheduling decisions.

[0060] If an anomaly is detected, the supervisor agent will automatically route the fault delimitation agent for further assessment. The fault delimitation agent primarily uses link tracing technology to identify the fault location. It also utilizes a multi-agent supervisor agent to automatically invoke agents at different layers for fault investigation. For example, the SaaS layer provides a Configuration Management Database (CMDB) agent to analyze the relationships between faulty resources and assist in the fault delimitation process. The PaaS layer provides database agents, middleware agents, and foundation agents for the fault delimitation agent to diagnose faults based on these relationships. The IaaS layer provides host agents and network agents for agents to perform fault delimitation analysis for host and network anomalies.

[0061] To facilitate further understanding of this solution, for example, if the first agent detects an anomaly in the business success rate indicator, the supervisor agent uses the fault delimitation agent as the command and dispatch center for business faults. First, it identifies possible fault points through fault delimitation. For instance, if an anomaly is found in the middleware service, but the CMDB agent discovers that the middleware is also related to the database, the agent corresponding to the data block is further dispatched, and multiple agents collaborate to make decisions, ultimately discovering that a deadlock fault in the database is the cause of this fault.

[0062] In this embodiment, by scheduling multi-level intelligent agents to cooperate, the fault node can be quickly located, reducing manual intervention and improving the accuracy of system fault location.

[0063] Optionally, the step of scheduling a second agent among the plurality of agents by the target agent to perform fault detection on the target system and obtain the detection result includes:

[0064] Obtain parameters for each of the plurality of agents, the parameters including at least one of contribution and support, the contribution being used to characterize the degree of participation of each agent in fault detection, and the support being used to characterize the degree of acceptance of each agent's fault detection proposal;

[0065] The target agent, based on the parameters, calls the second agent among the multiple agents to perform fault detection and obtain the detection result.

[0066] In the process of managing multiple agents to collaborate, each agent can propose diagnostic solutions or repair suggestions during the fault diagnosis process.

[0067] Contribution level can characterize the active participation of an agent in collaboration with the supervisor agent's Large Language Model (LLM). For example, if agent A submits multiple valid proposals, its contribution level is high.

[0068] The degree of contribution can be expressed using the following formula:

[0069]

[0070] in, This represents the contribution index (or contribution level) of a certain Agent, and its value is initialized to 1.0. This represents a random decay rate, ranging from 0 to 0.03. This value is adjusted after each iteration to incentivize agents to increase their contribution (participate in interactions and submit planning proposals) and to prevent an agent's weight from becoming too concentrated. It is the unit step of contribution. If an agent actively participates in (excluding abstention) the LLM interaction and submits a planning proposal, the contribution index of the evaluation module increases by 0.1. This means that, to ensure fairness, the upper limit of the contribution index is set at 1.5.

[0071] Support is used to characterize the degree to which an agent's proposal is accepted. For example, high support indicates that the proposal is adopted by the target agent. To improve the reliability of fault diagnosis in large models and to incentivize multi-agent collaboration and fairness to generate optimal planning strategies, expert contribution is introduced, where expert contribution is expressed as:

[0072]

[0073] in, It represents the expert's contribution and does not automatically decay over time, reflecting the professional skills that the expert agent accumulates over time. It is the unit step of the expert contribution. If the expert agent moves towards the final goal, it increases by 0.01 in each iteration; otherwise, it decreases by 0.01. This mechanism rewards the agent for making accurate predictions and punishes inaccurate predictions. This means that the maximum value for expert agents is set to 1.5 to maintain fairness and prevent experienced agents from having an undue influence.

[0074] To evaluate the merits of various solutions generated by multi-agent systems, evaluation parameters are defined as follows:

[0075] , ;

[0076] Where K is the total number of agents involved in the decision-making process. It is the contribution weight of the j-th agent. If ≥ α (agent support is the support level of the participating decision-making agent), and If α ≥ β (the agent's participation rate is the percentage of participation in the decision-making process), the agent's planning proposal is considered approved, where α and β are predefined threshold percentages. The indicator function I(·) outputs when the i-th agent outputs approval. If the vote is abstained, output 0. And if the vote is opposed, the indicator function I′(·) outputs the result for the i-th Agent in favor or against. If you abstain, output 0.

[0077] In this way, when the target agent obtains proposals for fault detection from multiple agents, it can obtain proposals from agents with high support and participation, and make decisions on the adoption and scheduling of proposals to improve the reliability of proposals.

[0078] The above algorithm is encapsulated into a multi-agent development platform, resulting in the following: Figure 4 The intelligent agent platform shown.

[0079] Intelligent agents can include the following components and modules:

[0080] Core control module: Decision engine, responsible for making decisions based on the current environmental state and historical data;

[0081] Perception and Interaction Module: Collects data required for system operation and provides a user interface;

[0082] Knowledge base: Provides case knowledge, rule base, etc. required for operation and maintenance;

[0083] Memory module: Provides long and short-term memory capabilities to assist large models in decision-making;

[0084] Skill plugins: Provides operation and maintenance plugin tools and operation and maintenance workflows;

[0085] Single agent: By organically combining the above capabilities, a single agent is formed to solve relevant business scenarios, such as fault delimitation agent, database agent, etc.

[0086] Multi-agent: By orchestrating multiple single agents and driving business needs, multiple agents are collaboratively configured to solve business scenario problems.

[0087] The target agent, based on the parameters (contribution or support) of each agent, decides to invoke a second agent (e.g., an agent with a contribution or support higher than a preset value) to perform fault detection. Figure 5 As shown. The second agent may include one or more agents.

[0088] By using the above methods, parameters such as contribution and support are used to further determine how to select the appropriate agent among multiple agents to perform overall fault diagnosis. For example, selecting agents with high contribution and support for fault detection can improve detection efficiency, reduce misjudgments by a single agent, and improve the accuracy of fault detection.

[0089] Optionally, the step of scheduling a second agent among the plurality of agents by the target agent to perform fault detection on the target system and obtain the detection result includes:

[0090] The target intelligent agent schedules the second intelligent agent among the multiple intelligent agents, and the second intelligent agent schedules at least one of the knowledge base, plugin, and workflow corresponding to the second intelligent agent to perform fault diagnosis and obtain the diagnosis result.

[0091] The detection result is determined based on the investigation results.

[0092] When performing fault detection, the target agent can call one or more of its internal knowledge base, plugins, and workflows for analysis to decide which second agent needs to be scheduled, thereby improving the efficiency of fault detection and the accuracy of scheduling agents.

[0093] After the target agent schedules the second agent, the second agent can further invoke plugins, knowledge bases, or workflows corresponding to itself (i.e., within the second agent) to perform fault detection. For example, if an abnormal number of database connections is detected, the second agent invokes the distributed database system anomaly detection plugin to analyze the trend of connection fluctuations.

[0094] The second agent may include at least one agent, which may be located at the same or different levels.

[0095] When the second intelligent agent comprises multiple intelligent agents, the target intelligent agent can make decisions based on the troubleshooting results of these multiple agents. During the decision-making process, it can also invoke knowledge bases, plugins, workflows, etc., within the target intelligent agent.

[0096] For example, the target agent calls the network agent and detects abnormal network latency. The target agent retrieves cases of "insufficient network bandwidth" from the knowledge base and suggests increasing capacity. Based on the obtained investigation results, the target agent confirms the insufficient network bandwidth and generates an expansion plan.

[0097] The target agent and each of the multiple agents may include at least one of the following: a knowledge base, a plugin, and a workflow. Wherein:

[0098] Knowledge Base: Quickly accumulates troubleshooting manuals, solutions, operation manuals, etc. from local daily operations and maintenance for use by intelligent agents.

[0099] Plugins: These are operations and maintenance plugins, which are sets of operations and maintenance tools. An operations and maintenance plugin can contain one or more tools (APIs). To complete this proposal, various built-in toolsets were used based on business needs, and custom toolsets are also supported.

[0100] Workflow: Provides orchestration of operation and maintenance processes for complex business fault diagnosis scenarios. It can improve the efficiency of intelligent agents in handling complex tasks by dragging and dropping different nodes to design complex multi-step tasks.

[0101] Intelligent Agents: Provides single-agent and multi-agent orchestration modes, allowing intelligent agents to be built according to the actual needs of business scenarios.

[0102] Before implementing this solution, the knowledge base, plugins, and workflows can be pre-built or developed. The knowledge base building process includes:

[0103] (1) First, knowledge is created, and then knowledge units are created. Knowledge units can be created for operation and maintenance document units such as TXT, Word, and Excel, and document upload function is provided.

[0104] (2) The uploaded operation and maintenance knowledge units can be segmented and configured according to the data characteristics of operation and maintenance knowledge through the segmented configuration capability;

[0105] (3) Knowledge Base Document Unit Details View: This view allows users to view the details of the knowledge base. Switching between content view and segmented view facilitates the agent's configuration of the knowledge base retrieval strategy, maximum recall quantity, and recall method. The target agent or the second agent can invoke instructions from the knowledge base. The invocation process is as follows: Figure 6 As shown.

[0106] (4) After the knowledge base is added to the agent, the vectorized retrieval of knowledge can be carried out through the above process. At the same time, the knowledge is further combined by the big model of the agent and replied to the agent by the big model in combination with the preset prompt words.

[0107] Plugins can include time series anomaly detection, time series prediction, etc. These small models can be packaged into skill plugins for intelligent agents to call.

[0108] Operations and maintenance (O&M) plugin tools primarily provide atomic operations for specific O&M tasks. Each tool is an atomic and independent task, such as querying container unit (pod) information or restarting a pod. These atomic O&M capabilities across different platforms can be encapsulated into standardized, platform-recognizable, and registrable tool application programming interfaces (APIs). The developed tool APIs are then registered with the system, including basic information, input parameters, output parameters, preview, and debugging steps.

[0109] A plugin can contain multiple tools, i.e., a toolset. The relationship between plugins and tools is analogous to the relationship between an API service and an API endpoint. For example... Figure 7 As shown, the main steps in creating a plugin are as follows:

[0110] (1) Configure tool parameters:

[0111] First, you need to register this API service as a plugin. The plugin definition includes basic information, a list of headers, and the authorization method. Next, add the API endpoint of this service to the plugin for use as a tool. The tool parameter definition includes basic information, input parameters, and output parameters.

[0112] Basic information includes defining the tool's name, description, whether to send SMS notifications, whether to confirm parameters, and whether to execute asynchronously.

[0113] Tool input parameters refer to the definition of information such as the name, description, type, and whether the input parameters are required.

[0114] Tool output parameters refer to the definition of the output parameters of the tool execution, and the parsing of the output parameter results based on the definition of the output parameters.

[0115] (2) Tool debugging

[0116] Tool debugging refers to testing and verifying the usability of a tool before its release. Only after successful debugging can the tool be released.

[0117] (3) Release of plug-in tools.

[0118] Tool debugging refers to testing and verifying the usability of a tool before its release. Only after successful debugging can the tool be released. Once successfully released, the tool can be invoked by intelligent agents.

[0119] For complex fault diagnosis processes, workflows can be orchestrated, such as in-depth inspection and fault delimitation workflows.

[0120] like Figure 8 As shown, the main steps in developing an operations and maintenance workflow are as follows:

[0121] (1) Configure basic information. After configuring the basic information of the workflow, the system will provide start and end nodes by default. Other nodes mainly include code, large model, plugin, knowledge base and selector.

[0122] (2) The workflow nodes can be arranged according to the relevant nodes required in the fault diagnosis process. The relevant nodes are described as follows:

[0123] Large model nodes: Supports large model configuration selection, adjustable temperature coefficient, and supports the definition and configuration of node input and output parameters.

[0124] Code node: Supports writing code (such as Python) and allows configuration of input and output parameters.

[0125] Plugin Nodes: Supports referencing existing plugin tools in workflows, and also supports creating new plugin tools and then referencing them. Node configuration is completed by defining the input and output parameters.

[0126] Knowledge base node: Supports configuration of knowledge base references, as well as adjustment and configuration of information such as knowledge base retrieval strategies. The node configuration can be completed by defining the input and output parameters.

[0127] Selector node: This is the conditional branch selection of the workflow. It supports the configuration of different conditions and can connect to the next node according to different conditions.

[0128] (3) Run and debug the workflow. After the workflow is orchestrated, if you want to know whether the workflow can output as expected, you can debug it. During the debugging process, you can see the specific status, success / failure. If the node is successfully debugged, you can see the input and output parameters of each node to facilitate further understanding of the specific information. If the debugging fails, you can see the specific error log to facilitate further location of the failure cause and further debugging.

[0129] After debugging, the workflow can be published and then invoked by the operations and maintenance agent.

[0130] The development based on the multi-agent development platform includes four main modules: knowledge base, plugins, workflow, and agents. Figure 9 As shown, the construction of a single agent can be completed by dragging and dropping the operation and maintenance knowledge base, operation and maintenance plugins, and operation and maintenance workflow as needed. The construction of a multi-agent system can be completed by arranging multiple single agents used in the business fault diagnosis process according to business characteristics to complete the fault diagnosis process of complex businesses.

[0131] The above modules combine to form a method for diagnosing complex business faults. The overall diagnosis process is mainly divided into three stages: perception, planning, and action.

[0132] Among them, perception is used to collect operation and maintenance requirements and further extract operation and maintenance data and knowledge so that the intelligent agent can build a knowledge base and query operation and maintenance data (empowering operation and maintenance plugins).

[0133] Planning refers to the planning of operation and maintenance intelligent agent scenarios, which involves unified planning and scheduling of the execution of operation and maintenance intelligent agents and skills based on the operation and maintenance scenarios.

[0134] Action refers to the execution of operation and maintenance operations, mainly referring to the execution of plug-in skills of the operation and maintenance intelligent agent, knowledge base query, workflow execution, and further completion of operations such as fault analysis, fault diagnosis, and fault self-healing.

[0135] The detailed descriptions of each functional module are as follows:

[0136] Perception primarily involves collecting and acquiring operational data based on business scenarios, and extracting relevant knowledge accordingly. The specific implementation process is as follows:

[0137] Step 1: Define the components of the business scenario

[0138] Organize the resource data related to this business (the resource data mainly involves network, IaaS / SaaS / PaaS, business, etc.), and clarify the relationship between the system resource relationship tree and the business resource relationship tree.

[0139] Step 2: Operation and Maintenance Data Collection

[0140] Operational data collection is a crucial step in ensuring stable system operation, performance optimization, and rapid fault response. Effective data collection helps the operations team promptly identify problems, analyze causes, and take appropriate measures. Common data collection methods include using ELK Stack for log data and Prometheus and Zabbix for metrics and alarm data.

[0141] Resource data collection: Based on the resource data sorted out in step 1, unified collection and management are carried out, and resource mapping management is carried out according to the system resource tree and business resource relationship tree, that is, resource topology data collection and management is carried out.

[0142] Performance metrics collection: After identifying the resources, relevant operation and maintenance metrics are collected based on the resource types, such as hosts and middleware, including platform metrics, gold metrics, and North Star metrics.

[0143] Platform metrics: These mainly include basic platform metrics such as CPU utilization and memory utilization.

[0144] Golden metrics: These mainly include golden metrics defined by business characteristics, such as average processing time and average response time.

[0145] North Star Metrics: Key metrics defined primarily based on business characteristics, such as mobile service resumption timeliness and consistency of service suspension / resumption status.

[0146] Alarm data collection: After the above performance indicators are collected, they are trained by an AI model and then both dynamic and static thresholds are defined to generate abnormal alarms.

[0147] Log data collection: Collect log data based on the composition of the business system.

[0148] Work order data collection: Collect data based on relevant historical work orders from the business system.

[0149] Step 3: Gathering Operation and Maintenance Knowledge

[0150] After completing the above two steps, it is necessary to collect operation and maintenance knowledge (mainly by vectorizing the operation and maintenance knowledge). The main knowledge to be collected includes troubleshooting experience manuals, operation and maintenance manuals, fault reports, work order processing suggestions, etc.

[0151] After completing the perception part, the next step is to plan the fault diagnosis scenario for the operation and maintenance intelligent agent. The main focus of this scenario planning process is the development of the intelligent agent. It utilizes ReAct technology to generate verbal reasoning trajectories and actions for tasks from a large model, and continuously interacts with the environment to ensure the reliability and accuracy of the AI ​​application. This is to complete the generation of the "fault diagnosis" execution strategy. The ReAct technology implementation logic is as follows: Figure 10 As shown. Its main interactive process includes "thinking," "execution," "observation," and iterations between them, until the optimal planning solution is generated, such as... Figure 11 As shown.

[0152] The target intelligent agent performs fault detection by calling a second intelligent agent. The second intelligent agent can further call corresponding plugins, workflows, and knowledge bases, which can adapt to different fault scenarios, improve the flexibility and efficiency of fault detection.

[0153] Optionally, the plugin includes at least one of the following: an indicator anomaly detection plugin, a log anomaly detection plugin, a distributed database system anomaly detection plugin, a business indicator analysis plugin, and a service plugin, wherein each plugin is obtained by encapsulating the corresponding skill.

[0154] When performing fault detection, the second intelligent agent can select and invoke necessary plugins. For example, based on knowledge from the operations and maintenance knowledge base, it can identify potential faults and invoke the corresponding plugins for detection. Each plugin encapsulates a specific function, which is executed when the plugin is invoked.

[0155] In some implementations, each second agent can invoke one or more plugins to perform corresponding fault detection, and the aforementioned plugins can be invoked by one or more second agents.

[0156] Among them, the indicator anomaly detection plugin can be used to monitor real-time indicators of system resources (such as memory and network bandwidth) and identify abnormal fluctuations.

[0157] Log anomaly detection plugins can be used to analyze log data and detect error codes, abnormal behavior, or potential failure modes.

[0158] The distributed database system anomaly detection plugin can be used for specific analysis and detection of database deadlocks, slow queries, abnormal connection counts, etc.

[0159] The business metrics analysis plugin can be used to assess the scope of a failure's impact by combining key business metrics such as transaction success rate and response time.

[0160] Service plugins can be used to call specific operation and maintenance tools (such as restarting services or adjusting configurations) to perform repair operations.

[0161] The target intelligent agent generates fault diagnosis results by acquiring the detection results of the plugin.

[0162] The second intelligent agent can flexibly call upon plugins for fault detection, which can improve the fault diagnosis effect.

[0163] Optionally, before scheduling the first agent to detect whether the service of the system is abnormal among multiple agents based on the instruction, the method further includes:

[0164] Receive the user's second input;

[0165] In response to the second input, business information corresponding to the business fault detection requirement is obtained, and the business information includes at least one of a knowledge base, a plugin, and a workflow.

[0166] Based on the business information, construct any one of the multiple intelligent agents.

[0167] The second input can be information entered by the user through natural language or a visual interface. Based on the second input, business information is obtained, which may include one or more of the following: a knowledge base, plugins, and workflows corresponding to the fault detection requirements.

[0168] For example, if a user inputs "analyze network latency issues," the system retrieves cases of insufficient network bandwidth from the knowledge base, obtains network monitoring plugins and log anomaly detection plugins, and generates a process to detect latency, analyze logs, and increase bandwidth. An intelligent agent is then constructed based on the acquired information.

[0169] In the fault diagnosis process, each of the multiple agents can consist of a single agent or multiple agents. For example... Figure 12 As shown.

[0170] For the development of single agents, agents can be dragged and dropped to select the skills (operation and maintenance knowledge base, operation and maintenance workflow, operation and maintenance plugin) required by business fault diagnosis needs. If it is found that the developed skills do not meet the conditions, the target system can also recreate or edit the knowledge base and workflow to complete the orchestration of single agents.

[0171] The agent can be debugged based on the specific skills selected to determine if it meets expectations. The debugging page provides a reference to the specific execution process and records. If it does not meet expectations, the relevant skills can be reconfigured and the debugging process can be repeated until it is successful.

[0172] Once successfully debugged, the agent can be deployed and then applied.

[0173] The development of the aforementioned intelligent agents, such as the health check intelligent agent, the Panji intelligent agent, and the Redis intelligent agent, can all be carried out by following the steps described above.

[0174] For multi-agent systems, multiple agents can be dragged and dropped to select the agents required for business fault diagnosis, and further different condition processes can be configured.

[0175] Debugging is performed based on the configuration of the multi-agent system to see if it can locate the root cause of the fault as expected and meet the fault diagnosis process in different situations.

[0176] The debugged multi-agent system can be deployed, and the deployed multi-agent system can be used for specific fault root cause localization.

[0177] After the single intelligences mentioned above are orchestrated as a whole by the multi-agent orchestration center, the final operation and maintenance fault diagnosis is performed by an intelligence in a multi-agent mode. Among them, the single intelligences also utilize small model skills, such as time series anomaly detection, time series prediction, log pattern clustering, etc., which are encapsulated as plug-ins.

[0178] By using the above methods, intelligent agents can be configured on demand, which can improve the accuracy of intelligent agents in fault detection, thereby improving the accuracy and efficiency of fault detection.

[0179] Optionally, after obtaining the detection result by scheduling a second agent among the plurality of agents through the target agent to perform fault detection, and determining the faulty node based on the detection result, the method further includes at least one of the following:

[0180] The target intelligent agent schedules the intelligent agent corresponding to the faulty node to perform fault repair.

[0181] After the faulty node has been repaired, the service status of the faulty node is verified by the target intelligent agent.

[0182] Once the fault node is identified, a specific intelligent agent can be invoked to repair the fault based on the fault node.

[0183] For example, if a deadlock fault is confirmed in the database, the system further invokes the database agent's overall fault diagnosis skills to perform fault self-healing (i.e., fault repair) operations. The database agent is composed of the following components: Figure 13 As shown, the database agent performs corresponding troubleshooting by scheduling multiple agents at different levels of the system.

[0184] After the fault self-healing is completed, the supervisor agent of the multi-agent system can further verify the fault to determine whether there are still anomalies. If there are anomalies, the above steps can be repeated until the fault is resolved.

[0185] The above methods enable rapid fault location and repair, improving fault repair efficiency.

[0186] Optionally, the instruction to obtain the first input upon receiving the user's first input includes:

[0187] Upon receiving the user's first input, retrieve the user's historical input records;

[0188] Based on the historical input records, the information entered by the first input is identified to obtain the instruction of the first input.

[0189] If the user's initial input is obtained, the user's input history can be further obtained, and the information entered in the initial input can be identified based on the history, thereby improving the recognition effect.

[0190] Historical input records can include input content, input time, and context information (such as current task status, fault type, etc.). Historical input records can be stored in categories such as time, fault type, or task type.

[0191] Based on historical input records and the first input content of the current input, the system identifies the instruction of the first input. For example, the user's historical input might be "Check database connection count" or "Analyze error codes in the logs"; currently, only "Check for anomalies" is input. The system analyzes the historical records and infers that the user's intention is to check for anomalies in the database or logs. The system then obtains the user's instruction as either "Detect database connection count anomalies" or "Analyze error codes in the logs".

[0192] like Figure 14 As shown, the agent's memory module is used to enhance its decision-making ability. This agent can be a target agent or an agent used to recognize user intent. For the agent's intra-session memory (multi-turn dialogue) and inter-session memory (persistent memory), historical conversation records are extracted, continuously optimized, and provided to a large model for skill learning. This improves the agent's accuracy in diagnosing faults, thereby increasing diagnostic accuracy and efficiency.

[0193] This application presents a fault diagnosis method based on multi-agent technology. The perception component combines multi-agent technology, data acquisition, and knowledge graph construction to achieve rapid and accurate fault diagnosis and optimized operation and maintenance. The planning component utilizes the planning capabilities of a large model, a multi-agent collaboration mechanism, a modular process for knowledge base and plugin development, and workflow orchestration to construct an intelligent operation and maintenance system capable of efficiently completing fault diagnosis tasks. The action component, through the collaborative work of multiple agents and the assistance of LLM and memory modules, constructs an efficient intelligent operation and maintenance system that automatically completes the entire process from fault detection to repair and continuously optimizes diagnostic capabilities.

[0194] Meanwhile, through large-scale model-driven conversational multi-agent collaborative operations, combined with the encapsulation of skills in large-scale operation and maintenance models and plug-in small models, evaluation parameters such as contribution and support are introduced to further quantify the optimal decision-making scheme of the agents. Furthermore, a modular agent design pattern is adopted, allowing for flexible and rapid on-demand selection of skills to build agents based on operation and maintenance needs. It also supports rapid expansion and free combination to adapt to various operation and maintenance scenarios, thereby improving the efficiency and accuracy of IT operation and maintenance fault diagnosis. This reduces the strong dependence of centralized systems on the computing power of operation and maintenance data from various provincial departments, enabling large-scale deployment.

[0195] See Figure 15 , Figure 15 This is a schematic diagram of the structure of a system fault detection device provided in an embodiment of this application, as shown below. Figure 15 As shown, the system fault detection device 1500 includes:

[0196] The first acquisition module 1501 is used to acquire the instruction corresponding to the first input when the user's first input is received;

[0197] The first scheduling module 1502 is used to respond to the instruction and schedule the first intelligent agent among multiple intelligent agents to detect whether there is an anomaly in the service of the target system. The multiple intelligent agents include intelligent agents located at different levels of the target system, and the target intelligent agent is used to manage the multiple intelligent agents.

[0198] The detection module 1503 is used to, when an anomaly is detected in the service of the target system, schedule a second agent among the multiple agents through the target agent to perform fault detection on the target system to obtain the detection result, and determine the fault node in the target system based on the detection result.

[0199] Optionally, the first scheduling module includes:

[0200] The first acquisition submodule is used to acquire parameters of each of the plurality of intelligent agents. The parameters include at least one of contribution and support. The contribution is used to characterize the degree of participation of each intelligent agent in fault detection, and the support is used to characterize the degree of acceptance of each intelligent agent's proposal for fault detection.

[0201] The calling submodule is used to call the second intelligent agent among the multiple intelligent agents according to the parameters through the target intelligent agent to perform fault detection and obtain the detection result.

[0202] Optionally, the first scheduling module includes:

[0203] The scheduling submodule is used to schedule a second agent among the multiple agents through the target agent, and to schedule at least one of the knowledge base, plugins, and workflows corresponding to the second agent to perform fault diagnosis and obtain the diagnosis results.

[0204] The determination module is used to determine the detection result based on the investigation results.

[0205] Optionally, the plugin includes at least one of the following: an indicator anomaly detection plugin, a log anomaly detection plugin, a distributed database system anomaly detection plugin, a business indicator analysis plugin, and a service plugin, wherein each plugin is obtained by encapsulating the corresponding skill.

[0206] Optionally, the device further includes:

[0207] The receiving module is used to receive the user's second input;

[0208] The second acquisition module is used to acquire business information corresponding to the business fault detection requirement in response to the second input. The business information includes at least one of a knowledge base, a plugin, and a workflow.

[0209] A construction module is used to construct any one of the multiple intelligent agents based on the business information.

[0210] Optionally, the device further includes at least one of the following:

[0211] The second scheduling module is used to schedule the agent corresponding to the faulty node to perform fault repair through the target agent;

[0212] The verification module is used to verify the service status of the faulty node through the target intelligent agent after the faulty node has been repaired.

[0213] Optionally, the first acquisition module includes:

[0214] The second acquisition submodule is used to acquire the user's historical input records upon receiving the user's first input.

[0215] The recognition submodule is used to recognize the information input by the first input based on the historical input record, and obtain the instruction of the first input.

[0216] The system fault detection device can achieve Figure 1 The various processes implemented in the method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0217] It should be noted that the electronic device provided in this application embodiment is capable of executing the above-described system fault detection device. Therefore, all implementation methods in the above-described system fault detection method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0218] This application also provides a terminal. Since the principle by which the terminal solves the problem is similar to the system fault detection method in this application, the implementation of this terminal can be found in the implementation of the method, and repeated details will not be described again. Figure 16 As shown, the terminal in this embodiment includes a processor 1600, configured to read a program from a memory 1620 and execute the following processes:

[0219] Upon receiving the user's first input, obtain the instruction corresponding to the first input;

[0220] In response to the instruction, the first intelligent agent is scheduled among multiple intelligent agents to detect whether there are any anomalies in the services of the target system through the target intelligent agent, wherein the multiple intelligent agents include intelligent agents located at different levels of the target system, and the target intelligent agent is used to manage the multiple intelligent agents;

[0221] If an anomaly is detected in the service of the target system, the target agent schedules a second agent among the multiple agents to perform fault detection on the target system to obtain the detection result, and determines the faulty node in the target system based on the detection result.

[0222] Transceiver 1610 is used to receive and send data under the control of processor 1600.

[0223] Among them, Figure 16In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 1600) and memory (memory 1620). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1610 may be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. The processor 1600 is responsible for managing the bus architecture and general processing, and the memory 1620 may store data used by the processor 1600 during operation.

[0224] Optionally, the processor 1600 reads the program from the memory 1620 and executes the step of scheduling a second agent among multiple agents to perform fault detection on the target system and obtain detection results, including:

[0225] Obtain parameters for each of the plurality of agents, the parameters including at least one of contribution and support, the contribution being used to characterize the degree of participation of each agent in fault detection, and the support being used to characterize the degree of acceptance of each agent's fault detection proposal;

[0226] The target agent, based on the parameters, calls the second agent among the multiple agents to perform fault detection and obtain the detection result.

[0227] Optionally, the processor 1600 reads the program from the memory 1620, wherein the step of scheduling a second agent among the plurality of agents to perform fault detection on the target system and obtain detection results by the target agent includes:

[0228] The target intelligent agent schedules the second intelligent agent among the multiple intelligent agents, and the second intelligent agent schedules at least one of the knowledge base, plugin, and workflow corresponding to the second intelligent agent to perform fault diagnosis and obtain the diagnosis result.

[0229] The detection result is determined based on the investigation results.

[0230] Optionally, the plugin includes at least one of the following: an indicator anomaly detection plugin, a log anomaly detection plugin, a distributed database system anomaly detection plugin, a business indicator analysis plugin, and a service plugin, wherein each plugin is obtained by encapsulating the corresponding skill.

[0231] Optionally, the processor 1600 reads the program from the memory 1620 and is also used to perform the following steps:

[0232] Receive the user's second input;

[0233] In response to the second input, business information corresponding to the business fault detection requirement is obtained, and the business information includes at least one of a knowledge base, a plugin, and a workflow.

[0234] Based on the business information, construct any one of the multiple intelligent agents.

[0235] Optionally, the processor 1600 reads the program from the memory 1620 and is also used to execute at least one of the following:

[0236] The target intelligent agent schedules the intelligent agent corresponding to the faulty node to perform fault repair.

[0237] After the faulty node has been repaired, the service status of the faulty node is verified by the target intelligent agent.

[0238] Optionally, the processor 1600 reads the program from the memory 1620 and executes the instruction to obtain the first input upon receiving the first input from the user, including:

[0239] Upon receiving the user's first input, retrieve the user's historical input records;

[0240] Based on the historical input records, the information entered by the first input is identified to obtain the instruction of the first input.

[0241] The terminal provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0242] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described system fault detection method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0243] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0244] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0245] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0246] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A system fault detection method, characterized in that, include: Upon receiving the user's first input, obtain the instruction corresponding to the first input; In response to the instruction, the first intelligent agent is scheduled among multiple intelligent agents to detect whether there are any anomalies in the services of the target system through the target intelligent agent, wherein the multiple intelligent agents include intelligent agents located at different levels of the target system, and the target intelligent agent is used to manage the multiple intelligent agents; If an anomaly is detected in the service of the target system, the target agent schedules a second agent among the multiple agents to perform fault detection on the target system to obtain the detection result, and determines the faulty node in the target system based on the detection result; The step of scheduling a second agent among multiple agents by the target agent to perform fault detection on the target system and obtain detection results includes: Obtain parameters for each of the plurality of agents, the parameters including at least one of contribution and support, the contribution being used to characterize the degree of participation of each agent in fault detection, and the support being used to characterize the degree of acceptance of each agent's fault detection proposal; Through the target intelligent agent, the second intelligent agent is invoked among the multiple intelligent agents according to the parameters to perform fault detection and obtain the detection result; The step of scheduling a second agent among multiple agents by the target agent to perform fault detection on the target system and obtain detection results includes: The target intelligent agent schedules the second intelligent agent among the multiple intelligent agents, and the second intelligent agent schedules at least one of the knowledge base, plugin, and workflow corresponding to the second intelligent agent to perform fault diagnosis and obtain the diagnosis result. The detection result is determined based on the investigation results.

2. The method according to claim 1, characterized in that, The plugins include at least one of the following: an anomaly detection plugin for metrics, an anomaly detection plugin for logs, an anomaly detection plugin for distributed database systems, a business metrics analysis plugin, and a service plugin. Each plugin is obtained by encapsulating the corresponding skill.

3. The method according to any one of claims 1 to 2, characterized in that, Before scheduling the first agent detection system's service to detect anomalies among multiple agents based on the instruction, the method further includes: Receive the user's second input; In response to the second input, business information corresponding to the business fault detection requirement is obtained, and the business information includes at least one of a knowledge base, a plugin, and a workflow. Based on the business information, construct any one of the multiple intelligent agents.

4. The method according to any one of claims 1 to 2, characterized in that, After obtaining the detection result by scheduling a second agent among the multiple agents through the target agent to perform fault detection, and determining the faulty node based on the detection result, the method further includes at least one of the following: The target intelligent agent schedules the intelligent agent corresponding to the faulty node to perform fault repair. After the faulty node has been repaired, the service status of the faulty node is verified by the target intelligent agent.

5. The method according to any one of claims 1 to 2, characterized in that, The step of obtaining the instruction for the first input upon receiving the user's first input includes: Upon receiving the user's first input, retrieve the user's historical input records; Based on the historical input records, the information input by the first input is identified to obtain the instruction of the first input.

6. A system fault detection device, characterized in that, include: The first acquisition module is used to acquire the instruction corresponding to the first input when the user's first input is received; The first scheduling module is used to respond to the instruction and schedule the first intelligent agent among multiple intelligent agents to detect whether there is an anomaly in the service of the target system. The multiple intelligent agents include intelligent agents located at different levels of the target system, and the target intelligent agent is used to manage the multiple intelligent agents. The detection module is used to, when an anomaly is detected in the service of the target system, schedule a second agent among the multiple agents through the target agent to perform fault detection on the target system, obtain the detection result, and determine the fault node in the target system based on the detection result; The first scheduling module includes: The first acquisition submodule is used to acquire parameters of each of the plurality of intelligent agents. The parameters include at least one of contribution and support. The contribution is used to characterize the degree of participation of each intelligent agent in fault detection, and the support is used to characterize the degree of acceptance of each intelligent agent's proposal for fault detection. The calling submodule is used to call the second intelligent agent among the multiple intelligent agents according to the parameters through the target intelligent agent to perform fault detection and obtain the detection result; The first scheduling module includes: The scheduling submodule is used to schedule a second agent among the multiple agents through the target agent, and to schedule at least one of the knowledge base, plugins, and workflows corresponding to the second agent to perform fault diagnosis and obtain the diagnosis results. The determination module is used to determine the detection result based on the investigation results.

7. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the system fault detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the system fault detection method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the system fault detection method as described in any one of claims 1 to 5.

Images