Multi-modal collaborative root cause analysis method fused with large language model

Abstract

The invention provides a multi-modal collaborative root cause analysis method fused with a large language model, and relates to the technical field of IT service fault analysis. According to the method, a tool feature vector space is constructed, a mixed retrieval algorithm (semantic similarity and historical calling frequency) is adopted to realize real-time dynamic sorting of a tool set, and a Top-K high-matching-degree tool is pushed based on a sliding window strategy; an Attention-Path mechanism based on the fault propagation weight is established, and a topology reasoning path related to the context is dynamically generated; multi-source heterogeneous data such as equipment logs, performance indexes and topological relations are integrated, and feature level fusion is achieved through a cross-modal alignment module. The problems of cognitive redundancy caused by full-amount context information overload, decision complexity of a multi-tool calling path, inherent illusion deviation diffusion of a generative model and high consumption of computing resources caused by long sequence processing in existing IT system fault diagnosis can be effectively solved.

Description

Technical Field

[0001] This invention relates to the field of IT business failure analysis technology, and in particular to a multimodal collaborative root cause analysis method that integrates a large language model. Background Technology

[0002] As businesses grow, IT enterprise architectures become increasingly complex, with a massive amount of structured data metrics, topology data, unstructured data logs, and alerts. Traditional root cause analysis has the following pain points: Rule engines rely on expert experience: they require manual sorting of fault trees or causal rules, making it difficult to cover the dynamic interaction relationships of complex systems (such as microservice architectures and multi-device linkage scenarios); Statistical learning methods are highly dependent on data: they require a large amount of labeled historical fault data to train the model, and their generalization ability is insufficient in small samples or new fault scenarios; among them, labeling historical fault data is a large workload and difficult to implement.

[0003] Direct application of large models: Due to the complexity of the scenarios, different alarm analyses rely on different data, making it difficult to provide data for different scenarios. Providing full data can easily cause Prompt bloat, leading to an unbearable burden on the context, a surge in token consumption, and a severe illusion of a large model.

[0004] Decision overload: When directly using large models to analyze mixed log and metric data, problems such as "causal jumps" (e.g., misjudging chronological relationships as causal relationships) or "knowledge obsolescence" (changes in processes within the machine and real-time changes in data in the database) often occur. If MCP is used to acquire relevant knowledge in real time, LLM needs to make more complex decisions when faced with a long list that may contain similar tools. The more decision-making steps there are, the higher the probability of errors, and even the "illusion" of calling non-existent APIs may occur. Summary of the Invention

[0005] Purpose of the invention: To propose a multimodal collaborative root cause analysis method that integrates large language models, aiming to solve the four major technical bottlenecks existing in the fault diagnosis of IT systems: 1) Cognitive redundancy caused by overload of full contextual information; 2) The decision-making complexity of multiple tool invocation paths; 3) The inherent diffusion of illusion bias in generative models; 4) The problem of high computational resource consumption caused by long sequence processing.

[0006] To achieve the above objectives, this invention proposes a multimodal collaborative root cause analysis method that integrates a large language model, comprising the following steps: S1: Construct node feature values, which include interface call frequency, data traffic matrix, fault propagation path, and version compatibility mapping; S2: Construct a relevant information tag library This is used for subsequent tool matching; S3: Construct a dependency graph based on the node feature values. The dependency graph is a weighted directed graph. ,in For a set of nodes, For directed edges, The weight of the edge; S4: Obtain candidate nodes based on the faulty node and the dependency graph; S5: A tool for verifying candidate fault nodes and filtering matching nodes based on the aforementioned tag library; S6: Use an Agent combined with a large language model and an MCP model context protocol to analyze whether the candidate nodes have faults and whether they are the root cause; S7: If the root cause is not located, iterate back, expand the range of candidate nodes and repeat S4 to S6.

[0007] As a preferred embodiment, the version compatibility mapping matrix The calculation formula is:

[0008] in, , , The matching degree is used as the weight for the hierarchy; , , The decay rate; for Version number difference; for Version number difference; for The difference in version numbers.

[0009] As a preferred embodiment, in step S2, the tag library The tags included are at least one of the following: server, CPU, memory, disk, disk I / O, process, message queue, Kafka, MQ, cache, Redis, Memcache, network device, SNMP, and IPMI; And define alarms The tag set is ,node The tag set is ,tool The tag set is ,in .

[0010] As a preferred embodiment, in step S3, the edge weights of the weighted directed graph... The calculation formula is:

[0011] in, This is the fault propagation probability weight, with a value of 0.5; if both nodes are hardware nodes, then... The value is 0.5. , The value is 0; if both nodes are service nodes, then The value is 0.1. The value is 0.3. The value is 0.1.

[0012] As a preferred approach, in step S5, when verifying candidate fault nodes, the alarm-tool local matching degree is first calculated. Node-tool local matching degree Then calculate the overall matching score of the tool; The formula for calculating the local matching degree of the alarm-tool is as follows: ; The formula for calculating the node-tool local matching degree is as follows: ; The formula for calculating the overall matching score of the tool is as follows: ,in For alarm node weight factors.

[0013] As a preferred embodiment, the alarm-node weight factor The value is 0.6, and the Top N tools are selected from high to low based on the overall matching score of the tools for candidate node fault verification.

[0014] As a preferred solution, in step S4, when obtaining candidate nodes, the calculation is performed from the candidate nodes. To the fault node The weighted path prioritizes nodes with higher weights. The rule for the number of candidate nodes K is: if 1 / 3 of the number of nodes in the dependency graph is less than 10, then all nodes are selected; otherwise, 1 / 3 of the number of nodes in the dependency graph is selected.

[0015] Beneficial Effects: This invention proposes a multimodal collaborative root cause analysis method that integrates a large language model. By constructing a tool feature vector space, it uses a hybrid retrieval algorithm (semantic similarity + historical call frequency) to achieve real-time dynamic ranking of the toolset, and pushes Top-K high-matching tools based on a sliding window strategy. It establishes an Attention-Path mechanism based on fault propagation weights to dynamically generate context-related topological reasoning paths. It integrates multi-source heterogeneous data such as device logs, performance indicators, and topological relationships, and achieves feature-level fusion through a cross-modal alignment module. This method can effectively solve the problems of cognitive redundancy caused by the overload of full context information, decision complexity of multi-tool call paths, the diffusion of illusion bias inherent in generative models, and high computational resource consumption caused by long sequence processing in the fault diagnosis of existing IT systems. Attached Figure Description

[0016] Figure 1 This is a flowchart of the alarm analysis process in the embodiment.

[0017] Figure 2 This is a flowchart of the alarm-tool call process in the embodiment. Detailed Implementation

[0018] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0019] Before describing the embodiments, some specific terms that may appear in the following text will be explained.

[0020] MCP: Model Context Protocol (MCP) is an open-source protocol launched by Anthropic. It aims to integrate large language models with external data sources and tools, and to establish a secure, bidirectional connection between large models and data sources.

[0021] Node: refers to an independent server, virtual machine, or service instance that can run independently, undertaking specific tasks (such as computing, storage, database, microservice instance, etc.).

[0022] Topology: refers to the physical / logical layout of device connections (such as bus, star, mesh).

[0023] In a microservice architecture, there are numerous nodes and complex relationships between them. Each node generates a large amount of log information during its operation and calls. Nodes include physical machines, virtual machines, microservice clusters, caches, databases, message queues, and various middleware. In this scenario, real-time analysis of the root cause of failures requires the integration of various heterogeneous data, including: alarm information, basic node information, node performance data, logs, propagation information, service versions, data traffic, and API call frequency.

[0024] It is known that tools exist that provide fusion query capabilities for multimodal data. Tool T1: CMDB Asset Basic Information Query Module Used to obtain static information about assets such as hosts, services, and network devices, including metadata such as IP address, system type, and associated business.

[0025] Tool T2: Alarm Information Query Module Used to query alarm records generated by a specific asset within a certain time period from the alarm system, including alarm type, level, time, status, etc.

[0026] Tool T3: Microservice Call Chain Tracing Module It is used to obtain information such as call relationships, link time, and abnormal nodes between services in a distributed system, and to construct a complete call chain path.

[0027] Tool T4: Node Performance Data Acquisition Module Used to collect the running status information of each node, including performance indicators such as CPU, memory, disk, and network, and supports time series queries.

[0028] Tool T5: Database Query Tool Module It supports SQL query interfaces, allowing for sampling and viewing of key business data in the business database. It also supports viewing database management commands to assist in analyzing whether the problem is caused by data layer anomalies.

[0029] Tool T6: Linux command execution tool module It supports remote execution of common diagnostic commands (such as top, df, netstat, etc.) on the host or container environment and returns the results for further analysis.

[0030] Tool T7: Network Topology Query Module It can query the logical connection relationships between services and between hosts in the system, and is used for visual analysis of dependency paths and breakpoints.

[0031] Tool T8: Log Query Module Searches can be performed based on log collection platforms or log libraries, including keyword matching, regular expression matching, and time range queries. When business anomalies are detected, error logs can be viewed by log level or keywords. When timeouts or access times are too slow, exception logs can be viewed by time range. Request access volume logs can also be viewed.

[0032] Tool T9: Message Queue Query Module Used to view message backlog, consumption status, and exception information in message middleware such as Kafka, RabbitMQ, and RocketMQ.

[0033] Tool T10: Cache Query Tool Module It supports querying and statistical analysis of data structures, cache hit rates, and expiration status in caching services such as Redis and Memcached.

[0034] Tool T11: Container Operation and Maintenance Tool Module Used for collecting and controlling information about Docker or other container runtime environments in the system.

[0035] Tool T12: Kubernetes Operation and Maintenance Tools Module It is mainly used to query various resources in Kubernetes clusters and provide a unified resource information collection capability.

[0036] Tool T13: Configuration Center Query Tool Module It is used to query application configuration items, change history, and effective status, and supports multiple configuration centers (such as Nacos, Apollo, and Spring Cloud Config) to facilitate troubleshooting problems caused by configuration changes.

[0037] Tool T14: Middleware runtime status query module It supports querying the running status and key indicators of common middleware components (such as MySQL master-slave status, Kafka Broker status, ES node status, Zookeeper status, etc.).

[0038] Tool T15: Change Operation Record Query Module It is used to query various change operation records within the system, such as code deployment records, deployment records, container change records, configuration modifications, etc., to facilitate change tracing and problem rollback analysis.

[0039] Tool T16: Network Connection Status Query Tool It supports querying the network connection status between hosts or containers, such as connectivity test records like ping, traceroute, and telnet. Combined with network policy configuration, it can be used for network troubleshooting.

[0040] Tool T17: Security Event Query Module It is used to query security event records such as host intrusion detection (IDS), abnormal logins, privilege escalation, and container escape, providing security posture visualization and tracing capabilities.

[0041] Tool T18: Task Scheduling and Execution Record Query Module Query task execution records and logs such as scheduled tasks (cronjobs), automated maintenance jobs, and CI / CD pipeline operation history to assist in fault analysis.

[0042] The multimodal collaborative root cause analysis method integrating a large language model proposed in this invention has the following specific steps: Step 1: Construct node feature values API call frequency (API requests / second): Use tools T3 and T4 to collect the frequency of node and node interface calls in the IT system architecture, and use a matrix... It means that, among them Represents a node Call Node The quantity.

[0043] Data traffic matrix (GB / hour): Data flow between nodes is collected using tools T3 and T4, and a matrix is ​​used. It means that, among them Representation module To module Data traffic.

[0044] Fault propagation path (probability of anomaly transmission): By combining historical alarms and topology relationships with T2 and T7, a probability matrix for node-to-node fault propagation is constructed. It means that, among them Representation module The fault propagates to the module The probability of.

[0045] Version compatibility mapping (semantic version correlation): Using matrices It means that, among them Represents a node and nodes The semantic version matching degree is in the range of [0,1]. The version change record is obtained using T13, and a matrix is ​​constructed based on the semantic version number and the decay coefficient.

[0046]

[0047] in: Matching degree hierarchy weights =0.6, =0.3, =0.1 : decay rate, =0.5, =0.1, =0.01 The difference between version numbers The difference between version numbers The difference between version numbers Step 2: Building an information tag library Establish a unified tag library For subsequent tool matching, L includes: server | CPU | memory | disk | disk I / O | process | message queue | Kafka | MQ | cache | Redis | memcache | network device | SNMP | IPMI, etc. Let the tag set of alarm 'a' be:

[0048] Let the label set of node n be:

[0049] Let the label set of tool t be:

[0050] in .

[0051] Step 3: Construct the dependency graph A directed weighted topology is constructed based on interface call frequency, data traffic matrix, version compatibility mapping, fault propagation path, and current alarms, and subsequent fault node filtering is performed.

[0052] View modules and PaaS components as nodes in a graph, with dependencies represented by directed edges, and edge weights determined by eigenvalues. Use a weighted directed graph. ,in: Node set : Indicates a directed edge.

[0053] The edge weights are calculated based on eigenvalues, using the following method:

[0054] : Represents the probability of anomaly propagation from module i to module j : Represents the current flow from module i to module j : Indicates the frequency of calls from module i to module j Note: When a server includes both hardware and software services, it is treated as two nodes during topology construction. : represents the weight values ​​of each parameter, where The weight value representing the probability of fault propagation is typically set to 0.5. This represents the node and its traffic weight. If both nodes belong to hardware such as network devices and hosts, this value is set to 0.5. , Setting it to 0 indicates that if the two nodes are service-to-service, such as a database or authentication service, then... Set to 0.1, Set to 0.3, Set it to 0.1.

[0055] : The weights from module i to module j Step 4: Obtain candidate nodes based on the faulty node and the dependency graph. Calculate the weighted path from candidate node i to faulty node j, prioritizing nodes with high weights, select the Top K relevant nodes, and obtain the current alarm information of each node from the alarm center.

[0056] K: The value is preferably 1 / 3 of the number of nodes in the dependency graph. If 1 / 3 of the number of nodes in the dependency graph is less than 10, then all nodes are taken.

[0057] Step 5: Verify candidate fault nodes Based on the faulty nodes, obtain the TopN of the MCP tool and call the tool to verify whether the candidate nodes are faulty.

[0058] Alarm-tool local matching aims to filter tools needed to acquire multimodal data based on alarm information. The matching algorithm is as follows:

[0059] in: : A collection of tags representing alarm 'a' : A collection of tags representing tool t : Indicates the number of common tags for alarm 'a' and tool 't'. Node-tool local matching aims to filter the tools needed to acquire multimodal data based on basic node information. The matching algorithm is as follows:

[0060] in: : Represents the set of labels for node n : A collection of tags representing tool t : Indicates the number of common labels between node n and tool t Overall match score:

[0061] in , The final matching score for tool j. It can be set to 0.6.

[0062] TopN filtering: The matching score for each tool is calculated using the algorithm described above, and the Top N tools are selected from the highest to the lowest scores.

[0063] Step 6: Use Agent + Large Model + MCP to analyze whether the node has a fault and whether it is the root cause. Step 7: Iterative Tracing If the candidate node is not the root cause, expand the scope of TopK in Step 4 based on the dependency graph constructed in Step 3, and repeat Steps 4-6 to analyze the root cause.

[0064] This embodiment uses a large-scale IT system as an example. The system includes multiple PaaS components and infrastructure modules (such as databases, caches, message queues, virtual machines, physical hosts, network devices, etc.) and has a complex dependency topology. To achieve root cause fault location, the following steps are adopted: Step 1: Construct node feature values Using the CMDB information acquisition tool (T3) and performance monitoring tool (T4), the following four types of inter-node features are extracted from the running system to construct a feature matrix: 1. Interface call frequency matrix If the database service is called by the authentication service Auth 120 times per second, then...

[0065] 2. Data Flow Matrix (Unit: GB / hour) If there is 3G of data traffic per hour between the database service and the authentication service Auth,

[0066] 3. Fault propagation path matrix Database service failures cause Auth service failures in 80% of cases.

[0067] 4. Version compatibility matrix The version has good compatibility.

[0068] Similarly, build the tag library and dependency library according to Step 2 and Step 3 respectively.

[0069] Follow the flowchart: "Alarm Analysis Flowchart" to process and analyze alarms.

[0070] S1 generates an alarm: "Alarm A_203: Redis node xxx.xxx.xxx.xxx response time exceeds 100ms and has not recovered for 5 consecutive minutes." S2 retrieves alert nodes: Node1, Redis service node S3 Obtain relevant nodes: Based on step 4 above, obtain the relevant dependent nodes: "Host: Node1, Node2; Middleware / Database: mysql, MQ, redis; Services: Auth, Gateway". S4 Obtain tools based on nodes and alarms: Obtain tools "T1, T2, T3, T4, T5, T6, T9, T10" according to Step 5 above.

[0071] See the tool call flowchart. Figure 1 The system uses tools, alarms, and nodes to invoke the intelligent agent, which then selects the appropriate tool combination for diagnosis based on Table 1 below. Table 1. Reference Table for Tool Combination Diagnosis

[0072] Output the root cause: The large model agent performs inference and judgment after collecting multimodal data: The current direct cause of Redis response timeouts is blocking operations and CPU / IO saturation; Redis was overwhelmed by a large number of write commands, and the write pressure originated from the pressure on the upstream database. Slow database queries and connection pool bottlenecks cause a large backlog of requests to the authentication service, which in turn squeezes the Redis request queue. The root cause is inferred to be: slow queries on the MySQL node caused timeouts in the response of the downstream Redis node.

[0073] The logical ideas behind the methods disclosed in the above embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs.

[0074] When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium may be a solid-state drive (SSD).

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A multimodal collaborative root cause analysis method integrating a large language model, characterized in that, Includes the following steps: S1: Construct node feature values, which include interface call frequency, data traffic matrix, fault propagation path, and version compatibility mapping; S2: Construct a relevant information tag library This is used for subsequent tool matching; S3: Construct a dependency graph based on the node feature values. The dependency graph is a weighted directed graph. ,in For a set of nodes, For directed edges, The weight of the edge; S4: Obtain candidate nodes based on the faulty node and the dependency graph; S5: A tool for verifying candidate fault nodes and filtering matching nodes based on the aforementioned tag library; S6: Use an Agent combined with a large language model and an MCP model context protocol to analyze whether the candidate nodes have faults and whether they are the root cause; S7: If the root cause is not located, iterate back, expand the range of candidate nodes and repeat S4 to S6.

2. The multimodal collaborative root cause analysis method integrating large language models according to claim 1, characterized in that, The version compatibility mapping matrix The calculation formula is: in, , , The matching degree is used as the weight for the hierarchy; , , The decay rate; for Version number difference; for Version number difference; for The difference in version numbers.

3. The multimodal collaborative root cause analysis method integrating large language models according to claim 1, characterized in that, In step S2, the tag library The tags included are at least one of the following: server, CPU, memory, disk, disk I / O, process, message queue, Kafka, MQ, cache, Redis, Memcache, network device, SNMP, and IPMI; And define alarms The tag set is ,node The tag set is ,tool The tag set is ,in .

4. The multimodal collaborative root cause analysis method integrating large language models according to claim 1, characterized in that, In step S3, the edge weights of the weighted directed graph The calculation formula is: in, This is the fault propagation probability weight, with a value of 0.5; if both nodes are hardware nodes, then... The value is 0.

5. , The value is 0; if both nodes are service nodes, then The value is 0.

1. The value is 0.

3. The value is 0.

1.

5. The multimodal collaborative root cause analysis method integrating large language models according to claim 3, characterized in that, In step S5, when verifying candidate fault nodes, the alarm-tool local matching degree is first calculated. Node-tool local matching degree Then calculate the overall matching score of the tool; The formula for calculating the local matching degree of the alarm-tool is as follows: ; The formula for calculating the node-tool local matching degree is as follows: ; The formula for calculating the overall matching score of the tool is as follows: ,in For alarm node weight factors.

6. The multimodal collaborative root cause analysis method integrating large language models according to claim 5, characterized in that, The alarm-node weight factor The value is 0.6, and the Top N tools are selected from high to low based on the overall matching score of the tools for candidate node fault verification.

7. The multimodal collaborative root cause analysis method integrating large language models according to claim 1, characterized in that, In step S4, when obtaining candidate nodes, the calculation is performed from the candidate nodes. To the fault node The weighted path prioritizes nodes with higher weights. The rule for the number of candidate nodes K is: if 1 / 3 of the number of nodes in the dependency graph is less than 10, then all nodes are selected; otherwise, 1 / 3 of the number of nodes in the dependency graph is selected.

8. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the multimodal collaborative root cause analysis method for fusing large language models as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on an electronic device, causes the electronic device to perform the multimodal collaborative root cause analysis method for a fusion large language model as described in any one of claims 1 to 7.

Images