Problem positioning method and device for distributed system, equipment and storage medium
By utilizing a large model to construct a call chain topology and identify fault modes in a distributed system, the problem of low efficiency and poor accuracy of manual fault location in existing technologies is solved, achieving efficient and accurate automatic fault root cause location.
Patent Information
- Application Number
- CN202511801233.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-01-16
AI Technical Summary
In existing technologies, the localization of problems in distributed systems is time-consuming, inefficient, and relies on human experience, which can easily lead to the omission of key nodes and inaccurate localization results.
By extracting TraceID from error logs, upstream and downstream logs are obtained and input into a large model to construct a topology graph of the service call chain. Combining log data and node environment data, the large model is used to identify the target error pattern and root cause of the fault.
It significantly improves the efficiency and accuracy of problem localization, shortens investigation time, reduces manual intervention, and achieves intelligent root cause generation.
Smart Images
Figure CN121356993A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a problem localization method, apparatus, device and storage medium for a distributed system. Background Technology
[0002] With the widespread adoption of distributed systems in critical business areas such as finance and payment, the rapid location of system failures has become a core challenge in ensuring business continuity and stability.
[0003] In existing technologies, when locating problems in distributed systems, developers typically manually switch between multiple systems (such as log platforms, monitoring systems, code repositories, etc.) to analyze the error messages and determine the problem location results.
[0004] However, manual positioning is time-consuming and inefficient, and it relies heavily on personal experience, making it easy to miss key details and potentially leading to inaccurate positioning results. Summary of the Invention
[0005] This application provides a method, apparatus, device, and storage medium for locating problems in a distributed system, thereby improving the accuracy and efficiency of problem location in a distributed system.
[0006] In a first aspect, embodiments of this application provide a problem localization method for a distributed system, the method comprising: When an error occurs on the target service node, extract the error log corresponding to the target service node. The error log contains: TraceID; Obtain upstream and downstream logs associated with TraceID from the log collection system. Each log in the upstream and downstream logs includes: TraceID, log data, and node environment data. Error logs, upstream and downstream logs, and preset prompts are input into the large model. Guided by the prompts, the large model constructs a service call chain topology based on log data, identifies the target error pattern of the failure based on the error logs, and determines the target root cause of the failure based on the target error pattern, the topology, and node environment data.
[0007] In one alternative embodiment, before constructing the call chain based on log data, the following steps are also included: Filter logs from upstream and downstream sources that are irrelevant to problem identification.
[0008] In one optional embodiment, a topology graph of the service call chain is constructed based on log data, including: Based on the span identifier SpanID and parent span identifier Parent SpanID contained in the log data, construct a topology graph of the service call chain.
[0009] In one optional embodiment, based on the error log, the target error pattern of this failure is identified, including: Based on the abnormal feature data contained in the error log, the knowledge base is queried to identify multiple candidate error patterns that match the error log. Semantic analysis was performed on the error log and multiple candidate error patterns to obtain the target error pattern of this fault.
[0010] In one optional embodiment, the target root cause of the failure is determined based on the target error pattern, topology graph, and node environment data, including: Based on the root cause reasoning template set, the target error mode, topology graph and node environment data are analyzed to determine the target root cause of the failure. Each root cause reasoning template in the root cause reasoning template set includes: root cause determination rules and root cause conclusions.
[0011] In an optional embodiment, after determining the target root cause of the current failure based on the target error pattern, topology graph, and node environment data, the method further includes: Based on the root cause, the knowledge base is queried to determine the target solution for this fault; Based on the code indication field contained in the error log, obtain the code snippet corresponding to this fault; The problem localization result report should include the target root cause, topology diagram, target solution, and code snippets, and then be sent to the target object.
[0012] In an optional embodiment, after sending the problem location result report to the target object, the method further includes: When a confirmation instruction is received from the target object regarding the problem location results report, the relevant data of this fault is stored in the knowledge base. The relevant data includes: error logs, target root causes, and solutions.
[0013] Secondly, embodiments of this application also provide a problem location device for a distributed system, the device comprising: The acquisition module is used to extract the error logs corresponding to the target service node when an error occurs. The error logs include: TraceID; The aggregation module is used to obtain upstream and downstream logs associated with TraceID from the log collection system. Each log in the upstream and downstream logs includes: TraceID, log data and node environment data. The large model analysis engine module is used to input error logs, upstream and downstream logs, and preset prompt words into the large model. Guided by the prompt words, the large model constructs a topology diagram of the service call chain based on log data, identifies the target error pattern of the failure based on the error logs, and determines the target root cause of the failure based on the target error pattern, topology diagram, and node environment data.
[0014] In one alternative embodiment, before constructing the call chain based on log data, the large model analysis engine module is also used for: Filter logs from upstream and downstream sources that are irrelevant to problem identification.
[0015] In an optional embodiment, when constructing a topology graph of the service call chain based on log data, the large model analysis engine module is also used for: Based on the span identifier SpanID and parent span identifier Parent SpanID contained in the log data, construct a topology graph of the service call chain.
[0016] In an optional embodiment, when identifying the target error pattern of this fault based on the error log, the large model analysis engine module is also used to: Based on the abnormal feature data contained in the error log, the knowledge base is queried to identify multiple candidate error patterns that match the error log. Semantic analysis was performed on the error log and multiple candidate error patterns to obtain the target error pattern of this fault.
[0017] In an optional embodiment, when determining the target root cause of the current failure based on the target error pattern, topology graph, and node environment data, the large model analysis engine module is also used for: Based on the root cause reasoning template set, the target error mode, topology graph and node environment data are analyzed to determine the target root cause of the failure. Each root cause reasoning template in the root cause reasoning template set includes: root cause determination rules and root cause conclusions.
[0018] In one optional embodiment, after determining the target root cause of the failure based on the target error pattern, topology graph, and node environment data, the large model analysis engine module is further used for: Based on the root cause, the knowledge base is queried to determine the target solution for this fault; Based on the code indication field contained in the error log, obtain the code snippet corresponding to this fault; The problem localization result report should include the target root cause, topology diagram, target solution, and code snippets, and then be sent to the target object.
[0019] In an optional embodiment, after sending the problem location result report to the target object, the device further includes an update module, which is used to: When a confirmation instruction is received from the target object regarding the problem location results report, the relevant data of this fault is stored in the knowledge base. The relevant data includes: error logs, target root causes, and solutions.
[0020] Thirdly, embodiments of this application also provide an electronic device, including: Processor; and Stored program memory, The program includes instructions that, when executed by a processor, cause the processor to perform the problem localization method for a distributed system as described in the first aspect.
[0021] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the problem localization method of the distributed system as described in the first aspect.
[0022] Fifthly, this application provides a computer program product that, when invoked by a computer, causes the computer to execute the problem localization method steps of the distributed system as described in the first aspect.
[0023] The beneficial effects of this application are as follows: In the distributed system problem localization method provided in this application embodiment, when a target service node reports an error, the error log corresponding to the target service node is extracted. The error log contains a TraceID. Then, upstream and downstream logs associated with the TraceID are obtained from the log collection system. Each log in the upstream and downstream logs includes a TraceID, log data, and node environment data. Finally, the error log, upstream and downstream logs, and preset prompt words are input into a large model. Guided by the prompt words, the large model constructs a topology graph of the service call chain based on the log data. Based on the error log, the target error pattern of this failure is identified. Based on the target error pattern, the topology graph, and the node environment data, the target root cause of this failure is determined. In this way, upstream and downstream logs are directly obtained through TraceID to construct a topology graph of the service call chain, eliminating the need for manual information piecing together, significantly improving efficiency. Furthermore, the large model proxy enables intelligent generation of root causes, automatically locating the root cause of the problem, greatly shortening the problem investigation time, and significantly improving the efficiency and accuracy of problem localization.
[0024] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described herein are used to provide a further understanding of this application, constitute a part of this application, and do not constitute an improper limitation of this application. In the accompanying drawings: Figure 1 This is a schematic diagram of an optional system architecture applicable to the embodiments of this application; Figure 2 A schematic diagram illustrating the implementation process of a problem localization method for a distributed system provided in this application embodiment; Figure 3 A logical schematic diagram of a problem localization method for a distributed system provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a problem location device for a distributed system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0026] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.
[0027] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.
[0028] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.
[0029] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0030] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.
[0031] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.
[0032] (1) TraceID: A trace ID used to uniquely identify a request in a distributed system. This ID can be used to link the logs of various services.
[0033] (2) Large Language Model (LLM): Refers to large models, such as GPT and Tongyi 1000 Questions. Large models represent a significant technological breakthrough in the field of artificial intelligence in recent years. Based on deep learning and neural network architecture, they generate complex language models through training on massive amounts of data. Their core advantage lies in their powerful language understanding and generation capabilities, enabling them to handle various tasks in Natural Language Processing (NLP), such as text generation, translation, and question answering. By learning patterns and relationships in a large amount of text, large models can generate coherent and accurate text content, demonstrating broad application potential in multiple fields.
[0034] (3) Log aggregation: Collect and store logs from different systems and services together for centralized querying and analysis.
[0035] (4) Call Chain: All service nodes that a request passes through in a distributed system and their mutual calling relationships.
[0036] Based on the above explanations of terms and related terminology, the design concept of the embodiments of this application will be briefly introduced below: With the widespread adoption of distributed systems in critical business areas such as finance and payment, the rapid location of system failures has become a core challenge in ensuring business continuity and stability.
[0037] In existing technologies, when locating problems in distributed systems, developers typically manually switch between multiple systems (e.g., log platforms, monitoring systems, code repositories, etc.) to analyze error messages and determine the problem location. However, manual location is time-consuming, inefficient, and highly dependent on personal experience, making it easy to miss key nodes and potentially leading to inaccurate results. Alternatively, a basic alarm system can be used to trigger alarms based on preset rules (e.g., error codes), but the alarms only indicate that an error has occurred, still requiring manual problem location.
[0038] In view of this, this application provides a method for locating problems in a distributed system. Specifically, it may include: when a target service node reports an error, extracting the error log corresponding to the target service node, the error log containing a TraceID; then obtaining upstream and downstream logs associated with the TraceID from a log collection system, each log in the upstream and downstream logs including a TraceID, log data, and node environment data; finally, inputting the error log, upstream and downstream logs, and preset prompt words into a large model, so that the large model, guided by the prompt words, constructs a topology graph of the service call chain based on the log data, identifies the target error pattern of the current fault based on the error log, and determines the target root cause of the current fault based on the target error pattern, the topology graph, and the node environment data. In this way, by directly obtaining upstream and downstream logs through the TraceID and constructing a topology graph of the service call chain, there is no need for manual information piecing together, significantly improving efficiency. Furthermore, by utilizing the large model proxy to achieve intelligent generation of the root cause, automatically locating the root cause of the problem, it greatly shortens the problem investigation time and significantly improves the efficiency and accuracy of problem location.
[0039] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.
[0040] See Figure 1The diagram illustrates an optional system architecture applicable to an embodiment of this application. This system architecture may include: terminal devices (101a, 101b) and server 102. The terminal devices (101a, 101b) and server 102 can interact via a communication network. The communication network may employ wireless communication or wired communication methods. For example, the terminal devices (101a, 101b) can access the network and communicate with server 102 via cellular mobile communication technology. This cellular mobile communication technology may include, for example, 5G (5th generation mobile networks) or next-generation mobile communication technology. Optionally, the terminal devices (101a, 101b) can access the network and communicate with server 102 via short-range wireless communication. This short-range wireless communication method may include, for example, wireless fidelity (Wi-Fi) technology.
[0041] This application embodiment does not impose any limitation on the number of communication devices involved in the above system architecture. For example, the above system architecture may include more terminal devices, or it may include fewer terminal devices, or it may also include other network devices. Figure 1 As shown, only terminal devices (101a, 101b) and server 102 are described as examples. The following is a brief introduction to the above communication devices and their respective functions.
[0042] A terminal device (101a, 101b) is a device that can provide voice and / or data connectivity to a user, and may be a device that supports wired and / or wireless connections.
[0043] For example, terminal devices (101a, 101b) may include, but are not limited to: mobile phones, tablets, laptops, handheld computers, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in autonomous driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.
[0044] In addition, the terminal devices (101a, 101b) may have related clients installed. The client may be software, such as an application (APP), browser, short video software, or webpage, mini-program, etc. It should be noted that the terminal devices (101a, 101b) in this application embodiment may be clients related to problem localization.
[0045] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0046] Optionally, a pre-trained large model can be deployed on server 102. After obtaining the error log, upstream and downstream logs, and preset prompts, server 102 inputs these logs and prompts into the large model to obtain the root cause of the current failure. The large model can be any large language model, and this application does not limit its application to this.
[0047] The following describes the problem localization method for a distributed system provided by an exemplary embodiment of this application, in conjunction with the above-described system architecture and with reference to the accompanying drawings. It should be noted that the above-described system architecture is only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.
[0048] See Figure 2 The diagram shown illustrates the implementation flow of a problem localization method for a distributed system provided in this application embodiment. Taking a server as an example, the specific implementation flow of this method is as follows: S20: When an error occurs at the target service node, extract the error log corresponding to the target service node.
[0049] The error log includes: TraceID.
[0050] In this embodiment of the application, the target service node is any service node in the distributed system.
[0051] For example, if service node A reports an error (e.g., an HTTP 500 error), the error log corresponding to service node A is extracted, and the TraceID in the error log is obtained.
[0052] S21: Obtain upstream and downstream logs associated with TraceID from the log collection system.
[0053] Each log in the upstream and downstream logs includes: TraceID, log data, and node environment data. The node environment data is the environment data of the service node corresponding to the log. The log data includes: call chain identification data (such as span identifier span_id, parent span identifier parent_span_id, service name service_name, etc.) and error diagnosis data (such as level, error code error_code, error message error_message).
[0054] In this embodiment of the application, all logs containing TraceID are retrieved through a log collection system (e.g., Fluentd), wherein the log collection system includes the logs of each service node.
[0055] For example, assuming TraceID is 12345, all logs with TraceID 12345 in the log collection system will be used as context logs.
[0056] S22: Input the error logs, upstream and downstream logs, and preset prompt words into the large model. Guided by the prompt words, the large model constructs a topology diagram of the service call chain based on the log data. Based on the error logs, it identifies the target error pattern of this failure. Based on the target error pattern, the topology diagram, and node environment data, it determines the target root cause of this failure.
[0057] Optionally, in this embodiment, to reduce the amount of data that the large model needs to process and reduce the consumption of computing resources, logs unrelated to problem localization can be filtered from upstream and downstream logs before constructing the topology graph of the service call chain based on log data. This filtering can be done by the large model or by the processing module; this embodiment does not impose any limitations on this.
[0058] Logs unrelated to problem identification are those with a level lower than the preset level (e.g., WARN level) and where the error is not true.
[0059] For example, if a log in the upstream or downstream logs has a DEBUG level, then that log will be deleted.
[0060] In this way, by eliminating interfering information, the large model can focus more on logs related to problem localization, while reducing the amount of data that the large model needs to process, reducing the consumption of computing resources, and maximizing system throughput while ensuring the analysis effect.
[0061] Optionally, in this embodiment, when constructing the topology graph of the service call chain based on log data, specifically, the topology graph of the service call chain is constructed based on the span identifier (SpanID) and parent span identifier (Parent SpanID) contained in the log data, and the target service node and the node closest to the target service node are highlighted in the topology graph. The topology graph is a directed acyclic graph, and each node in the graph contains the logs of the service node corresponding to that node. The form of the topology graph includes, but is not limited to, DOT language (call_graph_dot) and text format (call_graph_text).
[0062] Optionally, in this application embodiment, a possible implementation is provided for identifying the target error pattern of this fault based on the error log, specifically by performing the following operations: S220: Based on the abnormal feature data contained in the error log, query the knowledge base to determine multiple candidate error patterns that match the error log.
[0063] The abnormal feature data includes, but is not limited to, error codes, error messages, and exception types. The knowledge base includes, for example, reference root cause cases. Each reference root cause case includes, for example, reference error logs, reference upstream and downstream logs, reference root cause, reference solution, and other data related to the reference case.
[0064] For example, a reference root cause case error pattern could be "Pattern number: P042, Error pattern: Database connection pool exhausted, Description: The service has no available connections in the connection pool under high concurrency, Solution: Increase the connection pool limit, limit upstream flow, monitor connection pool utilization".
[0065] In this embodiment, the abnormal feature data contained in the error log are concatenated to form query features. The query features are then matched with the reference root cause cases in the knowledge base to obtain the N reference root cause cases with the highest similarity. The error patterns of the N reference root cause cases are then used as N candidate error patterns. Here, N is an integer greater than 1.
[0066] S221: Perform semantic analysis on the error log and multiple candidate error patterns to obtain the target error pattern of this fault.
[0067] In this embodiment of the application, a large model is used to perform semantic analysis on the error log and multiple candidate error patterns. One candidate error pattern is selected from the multiple candidate error patterns as the target error pattern for this fault, or multiple candidate error patterns are combined to generate the target error pattern for this fault.
[0068] In this way, the dual verification mechanism of multiple candidate error patterns and semantic analysis improves the confirmation accuracy of the target error model, thereby enhancing the accuracy of problem localization in distributed systems.
[0069] Optionally, in this embodiment of the application, when determining the target root cause of the current failure based on the target error pattern, topology graph, and node environment data, the determination is made in conjunction with a root cause reasoning template set. Specifically, the large model analyzes the target error pattern, topology graph, and node environment data based on the root cause reasoning template set to determine the target root cause of the current failure.
[0070] Each root cause reasoning template in the root cause reasoning template set includes: root cause determination rules and root cause conclusions.
[0071] In this embodiment, the target error pattern, topology graph, and node environment data are used as multi-dimensional evidence for root cause inference to determine the target root cause of the current failure. It is worth noting that there can be multiple or a single target root cause; this embodiment does not impose any limitation on this.
[0072] For example, Root Cause Inference Template 1 is "If the error mode is timeout, and the query per second (QPS) of upstream service Y increases by more than 150% in the last 5 minutes, then service X times out when processing requests from service Y because a surge in traffic to service Y leads to resource contention"; Root Cause Inference Template 2 is "If the error mode is connection pool exhaustion, and the connection pool limit of X is n (default 50), and the QPS of upstream service Y suddenly increases, then the maximum configuration of service X's database connection pool (n connections) is insufficient, and the connection pool is exhausted due to a sudden surge in traffic to service Y"; Root Cause Inference Template 3 is "If monitoring shows that CPU utilization is >90% and memory usage is >85%, and the error log contains "Out Of Memory", then service X triggers Out Of Memory (OOM) under high CPU and memory usage, possibly due to unrestricted queries leading to resource leaks."
[0073] In this way, multi-dimensional verification by integrating topology diagrams, error modes, and node environment data improves the accuracy of root cause identification, allowing the large model to learn the analytical thinking of operations and maintenance personnel through the root cause reasoning template set, and intelligently and quickly identify the target root cause.
[0074] Furthermore, in this embodiment of the application, after obtaining the target root cause of the current failure, relevant data is integrated to generate a problem location result report and send it to the target object so that the target object can review and confirm it.
[0075] S23: Based on the root cause, query the knowledge base to determine the target solution for this fault.
[0076] In this embodiment, the target root cause is matched with each reference root cause case in the knowledge base to obtain at least one reference root cause case with the highest similarity, and the solution of the at least one reference root cause case with the highest similarity is used as the target solution for this fault. Alternatively, if there are no reference root cause cases with a similarity greater than a similarity threshold among the reference root cause cases, the target solution for this fault is generated using a large model.
[0077] S24: Based on the code indication field contained in the error log, obtain the code snippet corresponding to this fault.
[0078] The code indicator field is used to locate the source code file and line number.
[0079] Additionally, it's worth noting that even if the error log lacks a code indicator field, the code snippet corresponding to the current fault can be obtained from the code file corresponding to the target error model. For example, if the target error model is "database connection pool exhausted," the code snippet corresponding to the current fault can be obtained by retrieving the connection pool configuration file for the target service node.
[0080] S25: Include the target root cause, topology diagram, target solution, and code snippets in the problem localization result report, and send the problem localization result report to the target object.
[0081] In this embodiment, the large model uses a preset report template to generate a problem location result report, which includes the target root cause, topology diagram, target solution, and code snippets. The problem location result report is then sent to the target object via WeChat, email, or an internal work order system.
[0082] For example, the problem localization result report is as follows: "{ "trace_id":"…", "call_graph_dot":"…", / / can be omitted "call_graph_text":"…", / / can be omitted "error_nodes":["serviceA:span12","…"], "error_origin":"serviceC:span45", "semantic_summary":[ {"pattern_id":"P042","description":"Database connection pool exhausted","count":1,"confidence":0.96} ], "root_cause":"Service C's database connection pool maximum configuration (50 connections) is insufficient, and the connection pool is exhausted due to a sudden traffic surge from Service B.", "remediation":[ "Increase the database connection pool limit of Service C to 200;", "Add traffic surge detection and enable rate limiting on the Service B side;", "Monitor the QPS and connection pool utilization of Service C, and set the alarm threshold to 80%." " ], "related_code_snippets":[ {"service":"serviceC","file":"db_pool.go","line_range":"45-58","snippet":"if poolSize>= maxSize { returnerrors.New("pool exhausted")}"} ]}.
[0083] In this way, the problem location result report not only locates the problem, but also provides specific and feasible solutions. The code snippets in the report are directly related to accelerate the understanding of the problem and the implementation of the repair. The problem location result report improves the convenience and efficiency of the repair process.
[0084] Furthermore, in this embodiment of the application, in order to provide feedback and update the knowledge base for continuous optimization, the knowledge base can be updated after the problem location result report is sent to the target object. Specifically, when a confirmation instruction from the target object regarding the problem location result report is received, the relevant data of this fault is stored in the knowledge base. When a modification instruction from the target object regarding the problem location result report is received, the relevant data of this fault is modified according to the modification instruction, and the modified relevant data of this fault is stored in the knowledge base.
[0085] The relevant data includes, but is not limited to: error logs, target root causes, and solutions.
[0086] In this embodiment, error logs, upstream and downstream logs, and preset prompts are input into the large model to obtain a problem location result report automatically output by the large model.
[0087] For example, the prompt message is "You are the root cause analysis engine. Based on the input TraceID, error logs, and upstream and downstream logs, complete the following steps and return a fixed JSON. All Chinese fields should use Simplified Chinese, full-width punctuation, and the text should not exceed 120 characters." 1. Input Validation: If TraceID or logs is empty, return {"error":"trace_id or empty logs"}. 2. Noise Reduction: Only retain logs with a level ≥WARN or marked error:true. 3. Call Chain Reconstruction: Construct a directed acyclic graph using span_id and parent_span_id, generating call_graph_dot (Graphviz). DOT) and call_graph_text (indentation level). Mark all nodes containing error logs (logs including TraceID) as error_nodes, and the error node of the nearest target service node as error_origin. 4. Semantic Analysis: Extract error_code, error_message, and exception_type from the error logs. Match using a knowledge base KB, calling search_kb(query) (query is the concatenation of the above text), taking the first 3 results and returning pattern_id, description, count, and confidence, forming a semantic_summary. KB example: - Pattern number P042 - Error Mode: Database Connection Pool Exhaustion - Description: The service has no available connections in the connection pool under high concurrency. - Solution: Increase the connection pool limit, implement upstream rate limiting, and monitor connection pool utilization. 5. Root Cause Inference: Based on three types of evidence (call chain topology, target error mode, and node environment data), use the following root cause inference template set to generate root causes (if there are multiple possibilities, list a maximum of 3, ordered in descending order of confidence): Template 1 (Timeout): If the target error mode is TIMEOUT, and the QPS of upstream service Y has increased by >150% in the last 5 minutes, then "Service X timed out while processing a request from service Y because a surge in traffic from service Y led to resource contention"; Template 2 (Connection Pool): If the target error mode is POOL_EXHAUSTED, and the connection pool limit of X is N (default 50), and the QPS of upstream service Y suddenly increases, then "Service X's database connection pool maximum configuration (N connections) is insufficient, and the connection pool is exhausted due to a sudden surge in traffic from service Y"; Template 3 (CPU / Memory OOM): If monitoring shows CPU >90%... If memory usage is greater than 85% and the logs contain "out of memory", then "Service X triggers OOM under high CPU and memory usage, which may be due to unrestricted queries leading to resource leaks".6. Solution generation: For the root cause, retrieve the corresponding solution from the KB; if the KB does not have it, synthesize it manually (such as "increase the connection pool limit to 200", "enable rate limiting upstream", "enable circuit breaker", etc.), each ending with ";", forming a remediation list. 7. Return JSON (must be complete): { "trace_id":"…", "call_graph_dot":"…", / / can be omitted "call_graph_text":"…", / / can be omitted "error_nodes":["serviceA:span12","…"], "error_origin":"serviceC:span45", "semantic_summary":[ {"pattern_id":"P042","description":"Database connection pool exhausted","count":1,"confidence":0.96} ], "root_cause":"Service C's database connection pool maximum configuration (50 connections) is insufficient, the connection pool is exhausted due to sudden traffic from Service B.", "remediation":[ "Increase the database connection pool limit of Service C to 200;", "Add traffic surge detection and enable rate limiting on the Service B side;", "Monitor the QPS and connection pool utilization of Service C, and set the alarm threshold to 80%." ], "related_code_snippets":[ {"service":"serviceC","file":"db_pool.go","line_range":"45-58","snippet":"if poolSize>=maxSize { return errors.New("pool exhausted")}"} ]}。 8. Error Handling: If any step fails, only {"error":"detailed description"} is returned, without any other fields.
[0088] Based on the above embodiments, see Figure 3 As shown, this is a logical diagram of obtaining the problem location of the distributed system in an embodiment of this application. When any service node reports an error, the log aggregation platform obtains the upstream and downstream logs associated with TraceID from the log collection system, and then performs root cause location and solution generation through the large model analysis engine. Finally, a problem location result report is generated and pushed to the target object.
[0089] Furthermore, based on the same technical concept, embodiments of this application provide a problem localization device for a distributed system, which is used to implement the above-described method flow of embodiments of this application. For example, see [link to relevant documentation]. Figure 4 As shown, the problem location device 400 of the distributed system may include: an acquisition module 401, an aggregation module 402, a large model analysis engine module 403, and an update module 404.
[0090] The acquisition module is used to extract the error logs corresponding to the target service node when an error occurs. The error logs include: TraceID; The aggregation module is used to obtain upstream and downstream logs associated with TraceID from the log collection system. Each log in the upstream and downstream logs includes: TraceID, log data and node environment data. The large model analysis engine module is used to input error logs, upstream and downstream logs, and preset prompt words into the large model. Guided by the prompt words, the large model constructs a topology diagram of the service call chain based on log data, identifies the target error pattern of the failure based on the error logs, and determines the target root cause of the failure based on the target error pattern, topology diagram, and node environment data.
[0091] In an alternative embodiment, before constructing the call chain based on log data, the large model analysis engine module 403 is also used for: Filter logs from upstream and downstream sources that are irrelevant to problem identification.
[0092] In an optional embodiment, when constructing a topology graph of the service call chain based on log data, the large model analysis engine module 403 is also used for: Based on the span identifier SpanID and parent span identifier Parent SpanID contained in the log data, construct a topology graph of the service call chain.
[0093] In an optional embodiment, when identifying the target error pattern of this fault based on the error log, the large model analysis engine module 403 is further configured to: Based on the abnormal feature data contained in the error log, the knowledge base is queried to identify multiple candidate error patterns that match the error log. Semantic analysis was performed on the error log and multiple candidate error patterns to obtain the target error pattern of this fault.
[0094] In an optional embodiment, when determining the target root cause of the current failure based on the target error pattern, topology graph, and node environment data, the large model analysis engine module 403 is further configured to: Based on the root cause reasoning template set, the target error mode, topology graph and node environment data are analyzed to determine the target root cause of the failure. Each root cause reasoning template in the root cause reasoning template set includes: root cause determination rules and root cause conclusions.
[0095] In an optional embodiment, after determining the target root cause of the failure based on the target error pattern, topology graph, and node environment data, the large model analysis engine module 403 is further configured to: Based on the root cause, the knowledge base is queried to determine the target solution for this fault; Based on the code indication field contained in the error log, obtain the code snippet corresponding to this fault; The problem localization result report should include the target root cause, topology diagram, target solution, and code snippets, and then be sent to the target object.
[0096] In an optional embodiment, after the problem location result report is sent to the target object, the update module 404 is used to: When a confirmation instruction is received from the target object regarding the problem location results report, the relevant data of this fault is stored in the knowledge base. The relevant data includes: error logs, target root causes, and solutions.
[0097] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of the present invention.
[0098] This application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.
[0099] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.
[0100] See Figure 5The diagram shown below illustrates the structure of an electronic device 500 that can serve as a server or client in this application, and is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.
[0101] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0102] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, WiFi devices, worldwide interoperability for microwave access (WiMax) devices, cellular communication devices, and / or the like.
[0103] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the problem localization method of the above-described distributed system can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the problem localization method of the above-described distributed system by any other suitable means (e.g., by means of firmware).
[0104] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0105] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0106] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device, PLD) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0109] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.
[0110] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of this invention are still within the scope of this application.
Claims
1. A method for problem localization in a distributed system, the method comprising: The method comprises the following steps: When an error occurs in a target service node, an error log corresponding to the target service node is extracted, and the error log comprises a TraceID; Obtain the upstream and downstream logs associated with the TraceID from the log collection system, wherein each log in the upstream and downstream logs comprises the TraceID, log data, and node environment data; Input the error log, the upstream and downstream logs, and a preset prompt word into a large model, so as to construct a topology graph of a service call chain based on the log data under the guidance of the prompt word by the large model, identify a target error mode of the current fault based on the error log, and determine a target root cause of the current fault based on the target error mode, the topology graph, and the node environment data.
2. The method of claim 1, wherein, Before constructing the call chain based on the log data, the method further comprises the following step: Filter the logs in the upstream and downstream logs that are irrelevant to problem positioning.
3. The method of claim 1, wherein, The method of constructing the topology graph of the service call chain based on the log data comprises the following step: Construct the topology graph of the service call chain based on the SpanID and the Parent SpanID contained in the log data.
4. The method of claim 1, wherein, The method of identifying the target error mode of the current fault based on the error log comprises the following steps: Query a knowledge base based on the abnormal feature data contained in the error log to determine a plurality of candidate error modes matching the error log; Perform semantic analysis on the error log and the plurality of candidate error modes to obtain the target error mode of the current fault.
5. The method of claim 1, wherein, The method of determining the target root cause of the current fault based on the target error mode, the topology graph, and the node environment data comprises the following steps: Analyze the target error mode, the topology graph, and the node environment data based on a root cause reasoning template set to determine the target root cause of the current fault, wherein each root cause reasoning template in the root cause reasoning template set comprises a root cause determination rule and a root cause conclusion.
6. The method of claim 1, wherein, After determining the target root cause of the current fault based on the target error mode, the topology graph, and the node environment data, the method further comprises the following steps: Query the knowledge base based on the target root cause to determine a target solution to the current fault; Obtain a code segment corresponding to the current fault based on a code indication field contained in the error log; Report the target root cause, the topology graph, the target solution, and the code segment as contents of a problem positioning result report, and send the problem positioning result report to a target object.
7. The method of claim 6, wherein, After sending the problem positioning result report to the target object, the method further comprises the following step: When receiving an acknowledgment indication of the problem positioning result report from the target object, store related data of the current fault in the knowledge base, wherein the related data comprises the error log, the target root cause, and the solution.
8. A problem location device for a distributed system, characterized in that, The method comprises the following steps: An acquisition module is configured to extract an error log corresponding to a target service node when an error occurs in the target service node, and the error log comprises a TraceID; The aggregation module is configured to acquire upstream and downstream logs associated with the TraceID from a log collection system, each of the upstream and downstream logs including the TraceID, log data, and node environment data. The large model analysis engine module is configured to input the error log, the upstream and downstream logs, and a preset prompt word into a large model, to construct a topology graph of a service call chain based on the log data under guidance of the prompt word by the large model, to identify a target error mode of the current fault based on the error log, and to determine a target root cause of the current fault based on the target error mode, the topology graph, and the node environment data.
9. An electronic device, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-7.
10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are configured to cause a computer to perform the method of any one of claims 1-7.