Log analysis method, device, server and computer readable storage medium

Abstract

Embodiments of the present application provide a log analysis method and device, a server and a computer readable storage medium, and relate to the technical field of data processing. Log data of each service node is obtained; an invocation dependency topology graph between the service nodes is constructed according to the invocation chain ID and the invocation chain data corresponding to each log data, each log data is subjected to semantic density clustering and screening, and at least one target log clustering cluster is obtained; each target log clustering cluster includes a plurality of target log data, and the plurality of target log data correspond to the same abnormal event; each target log data is mapped to a service node in the invocation dependency topology graph according to the timestamp information of the target log data, and a target invocation dependency topology graph is obtained; a cause-effect analysis is performed based on the target invocation dependency topology graph, and a source event and a source node of the abnormal event are determined according to the cause-effect analysis result, so as to improve the efficiency and accuracy of abnormal event and abnormal reason analysis, and reduce the human pressure.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a log analysis method, apparatus, server, and computer-readable storage medium. Background Technology

[0002] In today's context of widespread adoption of large-scale distributed systems and microservice architectures, a typical application system often consists of dozens or even hundreds of independently deployed service units. These service units communicate via a network and collaborate to complete business logic. While this architecture enhances system scalability and fault tolerance, it also significantly exacerbates the concealment and complexity of runtime problems—failures can originate from any service node and propagate rapidly along the call chain, leading to cascading failures. Therefore, log analysis has become one of the core means to ensure system stability and observability.

[0003] Current mainstream log analysis technologies generally rely on centralized log platforms for implementation, such as identifying abnormal events through keyword matching or static threshold alerts. However, because logs from various service nodes are often highly fragmented and isolated, and the data volume is large, these technologies often suffer from low efficiency and accuracy in identifying abnormal data. Furthermore, after identifying abnormal events, these technologies often require engineers to manually investigate the causes, which is not only labor-intensive but also suffers from low efficiency and poor accuracy in locating the root cause. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a log analysis method, apparatus, server and computer-readable storage medium to improve the efficiency and accuracy of abnormal event analysis and abnormal cause analysis, and reduce manpower pressure.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: Firstly, this application provides a log analysis method, the method comprising: Obtain log data from each service node; Based on the call chain ID and call chain data corresponding to each log data, a call dependency topology graph between each service node is constructed. Semantic density clustering and filtering are performed on each log data to obtain at least one target log cluster. Each target log cluster includes multiple target log data, and the multiple target log data correspond to the same abnormal event. Based on the timestamp information of the target log data, each target log data is mapped to a service node in the call dependency topology graph to obtain the target call dependency topology graph; Causal analysis is performed based on the target call dependency topology graph, and the source event and source node of the abnormal event are determined according to the causal analysis results.

[0006] In an optional implementation, the step of performing semantic density clustering and filtering on each of the log data to obtain at least one target log cluster includes: The log data is filtered according to a preset whitelist rule base to obtain candidate log data; Semantic features are extracted from the candidate log data, and density clustering is performed on the candidate log data based on the semantic features to obtain multiple log clusters. By filtering each of the log clusters using a preset filtering algorithm, at least one target log cluster is obtained.

[0007] In an optional implementation, the filtering algorithm includes at least one of noise filtering, semantic importance filtering, and preset rule filtering.

[0008] In an optional implementation, the method further includes: Obtain resource monitoring data corresponding to each target log data in the target log cluster; the resource monitoring data characterizes the system operating performance when the target log data is generated. Map the resource monitoring data corresponding to the target log data to the call dependency topology graph.

[0009] In an optional implementation, the target call dependency topology graph includes multiple target service nodes, and the target service nodes are mapped with the target log data and the resource monitoring data corresponding to the target log data; The step of performing causal analysis based on the target call dependency topology graph, and determining the source event and source node of the abnormal event based on the causal analysis results, includes: For each target service node, calculate the intra-node causality score between the resource monitoring data and the target log data corresponding to the target service node, and calculate the inter-node causality score between the upstream service node and the target service node. The validity of the intra-node causal scores and inter-node causal scores is verified to obtain valid causal scores. Based on the valid causal scores, the target call dependency topology is traced in reverse to determine the source event and source node of the abnormal event.

[0010] In an optional implementation, the method further includes: The log data is input into a pre-trained prediction model for processing to obtain anomaly prediction results. If the anomaly prediction result indicates that there is an abnormal risk within a preset time period, an early warning message will be issued.

[0011] In an optional implementation, the method further includes: Obtain user questions regarding the log data; The user's question information is processed using a large language model, and the response information is output.

[0012] Secondly, this application provides a log analysis apparatus, the apparatus comprising: The acquisition module is used to acquire log data from each service node; The analysis module is used to construct a call dependency topology graph between the service nodes based on the call chain ID and call chain data corresponding to each log data, and to perform semantic density clustering and filtering on each log data to obtain at least one target log cluster; wherein, each target log cluster includes multiple target log data, and the multiple target log data correspond to the same abnormal event; The analysis module is also used to map each target log data to the service node in the call dependency topology graph according to the timestamp information of the target log data, so as to obtain the target call dependency topology graph; The analysis module is also used to perform causal analysis based on the target call dependency topology graph, and determine the source event and source node of the abnormal event based on the causal analysis results.

[0013] Thirdly, this application provides a server including a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the method described in any of the foregoing embodiments.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the foregoing embodiments.

[0015] The log analysis method, apparatus, server, and computer-readable storage medium provided in this application embodiment allow the server to construct a call dependency topology graph between service nodes based on the call chain ID and call chain data corresponding to each log data. Semantic density clustering and filtering are then performed on each log data to obtain at least one target log cluster corresponding to an abnormal event. This target log cluster includes multiple target log data. The server can then map each target log data to a service node in the call dependency topology graph according to the timestamp information of the target log data, obtaining a target call dependency topology graph. Causal analysis is then performed based on the target call dependency topology graph, and the source event and source node of the abnormal event are determined based on the causal analysis results. In this way, by performing semantic density clustering and filtering on log data, isolated log data can be integrated through semantic information, and effective log data can be filtered from massive amounts of log data for analysis, thereby accurately identifying abnormal events and improving the efficiency of abnormal event identification. Furthermore, causal analysis can be performed by mapping the target log data in the target log cluster representing the abnormal event to the call dependency topology graph. This allows for the analysis of the root cause of the abnormal event through call dependencies and log data, thereby improving the efficiency and accuracy of abnormal cause analysis and reducing manpower pressure.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A block diagram of a server provided in an embodiment of this application is shown; Figure 2 This paper illustrates a flowchart of a log analysis method provided in an embodiment of this application. Figure 3 This illustration shows another flowchart of the log analysis method provided in an embodiment of this application; Figure 4 A functional block diagram of a log analysis device provided in an embodiment of this application is shown.

[0019] Icons: 100 - Server; 110 - Memory; 120 - Processor; 130 - Communication Module. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] Current mainstream log analysis technologies generally rely on centralized log platforms, such as the ELK stack (an open-source technology stack consisting of Elasticsearch, Logstash, and Kibana), Splunk, or similar commercial log analysis software. These platforms share a highly similar technical approach: first, they use lightweight agents like Filebeat or Fluentd to collect raw, unstructured text logs in real time from various server nodes; then, the collected log data is buffered in a Kafka message queue to handle sudden traffic surges; finally, the buffered logs are written in batches to a backend storage engine, such as Elasticsearch. At the storage level, these platforms primarily use inverted index technology to index the full text of the logs and divide them into multiple shards based on time to support efficient retrieval and horizontal scaling.

[0024] When performing anomaly analysis, the relevant technologies mainly rely on two passive response mechanisms: one is keyword matching, which involves operations and maintenance personnel manually writing query statements based on experience, using Boolean logical operators (AND / OR) and regular expressions to filter log content in an attempt to find log entries containing specific error codes, exception class names, or keywords; the other is static threshold alarms, which involves setting a fixed rule in advance, such as counting the number of logs with the level field value of ERROR within one minute, and triggering an alarm immediately once the number exceeds one hundred.

[0025] While these two methods seem simple and direct, they still present the following problems in real production environments: First, log information itself is highly fragmented and lacks contextual coherence. Individual log entries exist in isolation, making it impossible to automatically identify the manifestation of the same anomaly on different service nodes, let alone determine their causal relationship on the timeline. Second, crucial clues are often buried under massive amounts of repetitive, low-value logs, such as periodic heartbeat logs, known-to-be-ignorable WARN-level logs, or repeatedly printed identical stack traces, making it difficult to discover truly valuable fault signals. Therefore, related technologies often suffer from low efficiency and low accuracy in anomaly event identification.

[0026] Furthermore, in related technologies, after an abnormal event is identified, engineers often need to manually investigate the cause of the abnormality. Since the three types of data—logs, monitoring metrics (CPU / memory), and call chains (Trace)—are stored in different systems, engineers must manually switch between multiple interfaces, compare the timestamps of the three types of data at different time scales, and then try to align and infer based on their personal experience. The whole process is not only inefficient but also prone to errors, which prolongs the duration of the fault and increases labor costs and technical barriers.

[0027] Based on this, embodiments of this application provide a log analysis method, apparatus, server, and computer-readable storage medium to solve the above-mentioned problems.

[0028] Specifically, Figure 1 Please refer to the block diagram of the server 100 provided in the embodiments of this application. Figure 1 The server 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0029] The memory 110 is used to store computer programs or data. The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0030] The processor 120 is used to read / write data or computer programs stored in the memory, and execute the computer program to implement the log analysis method provided in the embodiments of this application.

[0031] The communication module 130 is used to establish a communication connection between the server and other communication terminals via the network, and to send and receive data via the network.

[0032] It should be understood that, Figure 1 The structure shown is only a schematic diagram of the server structure; the server may also include components such as... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0033] Optionally, the server may be equipped with a centralized logging platform, which can communicate and connect with multiple service nodes to obtain log data from each service node.

[0034] The following is based on the above. Figure 1 The server in this application is the execution entity. The log analysis method provided in this embodiment is illustrated with a flowchart. Specifically, Figure 2 For a flowchart illustrating the log analysis method provided in this application embodiment, please refer to [link / reference]. Figure 2 The method includes: Step S20: Obtain log data from each service node.

[0035] Step S21: Based on the call chain ID and call chain data corresponding to each log data, construct a call dependency topology graph between each service node, perform semantic density clustering and filtering on each log data, and obtain at least one target log cluster.

[0036] Each target log cluster contains multiple target log data, and these multiple target log data correspond to the same abnormal event.

[0037] In this embodiment, log data is a record text automatically generated by the distributed system during operation, including unstructured text logs output by each service node, such as error stacks, interface call status, configuration loading results, heartbeat reports, etc.

[0038] In practical applications, the server can acquire log data from multiple service nodes such as order service, user service, and payment gateway, and synchronously collect associated call chain IDs (Trace ID, a distributed tracing identifier used to uniquely identify a complete request call chain) and call chain data (Span Data, i.e., call span data generated by distributed tracing systems such as Jaeger or SkyWalking, including service node name, parent span ID, time consumption, status code, etc.). Based on this, the server can use the call chain ID as the core association key to bind logs generated by different service nodes with multiple spans under the same call chain, thereby identifying the call relationships between services. This constructs a service node topology graph that reflects the true dependency relationships, where each node represents a service node and each edge represents a cross-service call. This graph is the call dependency topology graph.

[0039] Semantic density clustering and filtering of log data refers to mapping the original log text into semantic vectors and then automatically merging logs with different but similar meanings in a high-dimensional space based on their distribution density. For example, "cache isnull", "redis cache miss", and "CacheNotFoundException" fall into similar regions after vectorization and can be clustered into the same cluster. The filtering process identifies those clusters with anomaly indications in the clustering results. The filtering criteria are not keyword matching, but rather the sparsity of the cluster in the semantic space, its distance from the mainstream log pattern, and / or whether its frequency of occurrence within the time window significantly deviates from the normal. The clusters that are ultimately retained are the target log clusters. Multiple target log data in each cluster point to the same type of abnormal event, such as "database connection timeout" or "thread pool rejects task". Although they differ in dynamic fields such as parameter values, timestamps, and thread IDs, they are consistent at the level of the fault essence.

[0040] In this way, the server can reliably extract clusters of abnormal events with diagnostic value from massive mixed logs based solely on the semantic distribution characteristics and spatiotemporal correlation of the logs themselves, without pre-setting fault templates.

[0041] Step S22: Based on the timestamp information of the target log data, map each target log data to the service node in the call dependency topology graph to obtain the target call dependency topology graph.

[0042] Step S23: Perform causal analysis based on the target call dependency topology graph, and determine the source event and source node of the abnormal event based on the causal analysis results.

[0043] It's important to note that log data is inherently discrete and unordered text records, while faults in microservice systems rarely occur in isolation. Instead, they propagate along the service call chain and evolve over time. Simply categorizing logs by service name fails to reflect the true propagation path and timing of anomalies within the system, making it difficult to distinguish between apparent symptoms and initial causes.

[0044] Based on this, in practical applications, the server can first utilize the key metadata—the call chain ID extracted from the logs—combined with the original call chain data to construct a call dependency topology graph between service nodes. Then, based on the timestamp information carried by each target log data entry, it accurately maps it to the corresponding service node in the topology graph and marks its occurrence time. In this way, the originally scattered log events are given spatial locations and service affiliations, while being embedded into a unified timeline, thus forming a dynamic graph that includes both inter-service connections and the spatiotemporal coordinates of exception occurrences—the target call dependency topology graph.

[0045] Afterwards, the server can perform root cause analysis on the abnormal events corresponding to the target log clusters based on the target call dependency topology graph. In this process, causal analysis does not refer to statistical correlation in general, but rather to identifying which node state changes are more likely to cause abnormal behavior in subsequent nodes under the constraints of the target call dependency topology graph. Thus, the source event and source node of the abnormal event can be determined based on the causal analysis results.

[0046] Among them, the source event refers to the earliest abnormal behavior that appears in the entire abnormal propagation chain and has been verified to have sufficient causal effect, such as a configuration change log or a GC pause log; the source node is the service node that carries the source event, that is, the location where the fault truly originates, rather than just the downstream node that shows an error.

[0047] The log analysis method, apparatus, server, and computer-readable storage medium provided in this application embodiment allow the server to construct a call dependency topology graph between service nodes based on the call chain ID and call chain data corresponding to each log data. Semantic density clustering and filtering are then performed on each log data to obtain at least one target log cluster corresponding to an abnormal event. This target log cluster includes multiple target log data. The server can then map each target log data to a service node in the call dependency topology graph according to the timestamp information of the target log data, obtaining a target call dependency topology graph. Causal analysis is then performed based on the target call dependency topology graph, and the source event and source node of the abnormal event are determined based on the causal analysis results. In this way, by performing semantic density clustering and filtering on log data, isolated log data can be integrated through semantic information, and effective log data can be filtered from massive amounts of log data for analysis, thereby accurately identifying abnormal events and improving the efficiency of abnormal event identification. Furthermore, causal analysis can be performed by mapping the target log data in the target log cluster representing the abnormal event to the call dependency topology graph. This allows for the analysis of the root cause of the abnormal event through call dependencies and log data, thereby improving the efficiency and accuracy of abnormal cause analysis and reducing manpower pressure.

[0048] The following provides a possible implementation method for performing semantic density clustering and filtering on various log data to obtain at least one target log cluster.

[0049] Specifically, in Figure 2 On this basis, Figure 3 For another flowchart illustrating the log analysis method provided in this application embodiment, please refer to [link / reference]. Figure 3 The method includes: Step S21-1: Filter the log data according to the preset whitelist rule base to obtain candidate log data.

[0050] In this embodiment, the server can first perform structured parsing and metadata binding on the log data generated by each service node.

[0051] Specifically, the server can automatically parse the received raw unstructured text logs, extract key fields such as TraceID, SpanID, and Host / IP, and encapsulate these fields as metadata along with the log content. At the same time, the server can also synchronously access monitoring metrics (i.e., time-series monitoring data of the system and application layers, such as CPU utilization and memory usage) and call chain data (Traces, i.e. distributed tracing data Span), and use high-precision timestamp technology to strictly align these three types of data at the millisecond time granularity, and finally generate a multi-dimensional unified spatiotemporal identifier for each log event, which includes "time, node, link ID, and metric characteristics".

[0052] Based on this, the server can perform preliminary filtering of log data according to the preset whitelist rule base to obtain candidate log data.

[0053] The whitelist rule base here refers to a set of matching rules that can be hot-updated. In one possible implementation, the whitelist rule base can include three types of rules: health heartbeat templates, known whitelist warning libraries, and duplicate stack fingerprint libraries.

[0054] Among them, the health heartbeat template is used to identify periodic, routine operation logs with no diagnostic value, such as fixed expressions like "service is running normally" and "heartbeat detection passed"; the known whitelist warning library is used to collect prompt logs that have been confirmed to be within the scope of business permission, such as "cache miss WARN" and "degradation switch enabled" in specific scenarios; and the duplicate stack fingerprint library generates a unique identifier based on the structural characteristics of the exception stack, which is used to identify redundant logs formed by the repeated printing of the same exception in a short period of time.

[0055] In this embodiment, the rules in the whitelist rule base can be dynamically updated. When the server discovers a log pattern that occurs frequently in subsequent analysis and is verified as a benign fluctuation by manual confirmation or resource health correlation analysis, the server can automatically extract its structured template features, generate new whitelist rules, and add them to the rule base to achieve continuous updates of the rule set.

[0056] In this embodiment, the server can perform millisecond-level rule matching on the acquired raw log stream. For logs that match any rule in the whitelist rule base, the server determines that they belong to deterministic noise and discards them directly or archives them to cold storage. For logs that do not match any rule, they are marked as unknown or suspected anomalies and output as candidate log data to subsequent processing stages. In this way, both the determinism and real-time nature of the filtering process are ensured, while also preserving a sufficiently rich collection of potential anomaly samples for in-depth semantic analysis.

[0057] Step S21-2: Extract semantic features from the candidate log data, and perform density clustering on the candidate log data based on the semantic features to obtain multiple log clusters.

[0058] Step S21-3: Filter each log cluster using a preset filtering algorithm to obtain at least one target log cluster.

[0059] It should be understood that if log analysis relies solely on character matching or fixed rules, it is difficult to identify exception information that has the same semantics but different expressions. For example, "NPE", "NullPointerException" and "java.lang.NullPointerException" actually point to the same type of program error, but are easily regarded as unrelated events by the server because of their different wording.

[0060] Based on this, in practical applications, the server can first extract the semantic features of the message body of the candidate log data that has been initially screened by the whitelist rules, and then perform clustering operations based on these features.

[0061] Semantic feature extraction refers to using lightweight pre-trained language models, such as DistilBERT or LogBERT fine-tuned from log corpora, to transform the message body of each log into a high-dimensional semantic vector. This vector can represent the actual meaning expressed by the log, rather than just reflecting the combination of characters.

[0062] Density clustering refers to algorithms like DBSCAN or HDBSCAN, which do not require a pre-defined number of clusters. Within the space defined by the semantic vectors, it automatically divides the logs into several clustered regions based on the local density relationships between points. Each region is a log cluster, representing an inherently consistent system behavior pattern, such as "user login successful" or "database connection timeout." During this process, the algorithm itself does not determine which behavior is abnormal; it only groups semantically similar logs into the same cluster. The specific business meaning needs to be confirmed later based on the original log text and vector distribution characteristics.

[0063] Furthermore, the server can perform filtering operations on the multiple log clusters already formed. This involves selecting the target log cluster corresponding to the anomalous event from all clusters based on the distribution characteristics and semantic attributes of the clustering results. This operation does not simply remove individual log entries, but rather selects clusters at a time. The aim is to retain patterns that may indicate genuine anomalies while excluding clusters that significantly deviate from normal behavior or lack diagnostic significance. In this way, the server can output at least one target log cluster, serving as the basic input for subsequent root cause localization and causal analysis.

[0064] In one possible implementation, the filtering algorithm may include at least one of noise filtering, semantic importance filtering, and preset rule filtering.

[0065] In practical applications, noise filtering is a natural screening process that is performed simultaneously during the density clustering process of DBSCAN (Density-Based Spatial Clustering of Applications with Noise). When the server semantically vectorizes the candidate log data and inputs it into the DBSCAN algorithm, the algorithm can identify dense and sparse regions in the high-dimensional semantic space based on the preset neighborhood radius parameter eps. Log vectors that cannot belong to any core cluster and do not meet the edge point conditions are automatically marked as noise points. If it is further determined that its semantic vector is too far from the known error pattern and has no actual semantic meaning (e.g., it is represented by garbled text, truncated logs, or severely distorted character sequences), then the noise point is directly discarded and will not enter the subsequent aggregation analysis process.

[0066] Semantic importance filtering applies to multiple log clusters that have already been clustered, essentially assigning each cluster a quantifiable value weight. This weight is obtained by a weighted fusion of semantic rarity score and distribution outlier score.

[0067] The semantic rarity score is obtained by extracting keyword features of logs within a cluster based on the TF-IDF (Term Frequency–Inverse Document Frequency) algorithm, which measures the rarity of the semantic pattern in the entire log corpus. The distribution outlier score refers to a comprehensive consideration of the number of samples within the cluster (the fewer the samples, the higher the weight) and the semantic distance between the centroid vector and the centroid of the mainstream normal cluster (the farther the distance, the higher the weight).

[0068] The server can linearly weight the aforementioned two items using preset weighting coefficients to generate a final importance score. When this score is lower than a preset threshold, the entire cluster is considered semantically redundant and filtered out. For example, logs such as "System is running... progress 1%" and "progress 2%" may have slightly different semantics, but they are highly clustered and evenly distributed in the vector space. Their information entropy is close to zero and their TF-IDF weight is extremely low. Based on this, the server can identify their high-frequency, low-information-content nature and filter out the entire cluster.

[0069] Preset rule filtering refers to whitelist / blacklist filtering, where the blacklist or whitelist can be a preset rule or a dynamically updated rule.

[0070] Taking whitelists as an example, servers can use resource health correlation analysis to compare whether CPU utilization, memory usage, and interface latency during the outbreak period are all within the healthy baseline range to determine whether a certain cluster is a normal business fluctuation; they can also call the historical pattern matching function to search whether the cluster template fingerprint has appeared regularly in the same period in the past seven days and has not been associated with a fault to determine whether a certain cluster is a normal business fluctuation; they can also receive manual annotation instructions, or adopt the high-confidence benign judgment given by the built-in LLM (Large Language Model) based on context to determine whether a certain cluster is a normal business fluctuation.

[0071] Once the cluster is determined to be a normal business fluctuation, the server can extract the log template structure of the cluster after removing dynamic variables, generate temporary semantic filtering rules, and use them as a new whitelist to participate in the filtering of log clusters within a set validity period (such as two hours or permanently).

[0072] Understandably, one of the three types of filtering algorithms mentioned above can be used, or they can be used in combination. For example, noise filtering can be performed first, followed by semantic importance filtering, and then filtering based on preset rules, thereby achieving fine-grained filtering and obtaining target log clusters with more analytical value.

[0073] Optionally, after obtaining the target log cluster, the server can also uniformly sort and organize the log content, merging multiple original logs corresponding to the same type of abnormal event into a single aggregated event record with complete context.

[0074] Specifically, the server can identify that logs belonging to the same target log cluster differ only in dynamic fields such as timestamps, thread numbers, process numbers, user identifiers, and order numbers, while maintaining the same core error semantics. Based on this, the server can extract the host IP and trace ID carried in each log as key anchors for subsequent cross-service tracing. During compression, the server retains the original text of the earliest, latest, and highest-scoring logs within the cluster, and each sample log fully retains its trace ID, ensuring accurate reconstruction of the scene during root cause backtracking.

[0075] The final output structured target log cluster can contain several fixed fields: a standard text template describing the anomaly, the total number of times the template appears within a specified time period, the time of the first occurrence, the time of the last occurrence, a list of the three sample logs mentioned above, and a list of all service node addresses involved.

[0076] In this way, the massive amount of logs that were originally scattered across different times and nodes are integrated into a very small number of structured event units with extremely high information density. This not only makes it easier for people to quickly grasp the whole picture of the problem, but also provides a clear and stable input foundation for mapping the propagation path in the service topology graph.

[0077] The server can then perform root cause analysis based on the selected target log clusters.

[0078] In this embodiment, the server can first map each target log data to a service node in the call dependency topology graph according to the timestamp information of the target log data. Simultaneously, it can also obtain resource monitoring data corresponding to each target log data in the target log cluster; and map the resource monitoring data corresponding to the target log data to the call dependency topology graph. The resource monitoring data characterizes the system performance at the time the target log data was generated.

[0079] In this embodiment, resource monitoring data refers to time-series performance indicators that reflect the running status of the service node, collected at the same time as the target log data is generated.

[0080] In one possible implementation, resource monitoring data can include key parameters characterizing system performance, such as CPU (Central Processing Unit) utilization, memory usage, I / O (Input / Output) throughput, database connection count, and average interface response latency. This data is not collected independently but is acquired synchronously with log data during the multimodal data access phase. Using millisecond-level timestamp alignment technology, it is bound to the spatiotemporal identifier corresponding to each log record during the initial parsing stage. That is, when the server parses a log record for the first time, it not only extracts its message body, TraceID, and HostIP metadata but also pulls the latest set of monitoring snapshots of the node to which it belongs at that millisecond from the monitoring acquisition agent, incorporating them into a unified data context.

[0081] In practical applications, the process of mapping resource monitoring data to the calling dependency topology is not a simple one-to-one mounting, but rather uses the timestamp of the target log data as the anchor point and combines it with a dynamic time window for association mapping.

[0082] Specifically, the server first maps each target log data point to a service node in the call dependency topology graph based on the timestamp information of the target log data. This locates the service node and corresponding time point to which the log belongs. Then, a preset time window (e.g., five minutes, the specific value of which can be adaptively adjusted based on the average service link latency) is automatically slid across the window. Within this window, the monitoring data sampling point closest to the target log time and meeting the accuracy requirements is retrieved. For example, if the log timestamp is 2026-07-28T14:23:15.892, and the monitoring data is collected at the second granularity, the system selects the complete set of metrics at 14:23:15 or 14:23:16 as the associated resource context for the log event. In this way, each target log data point mapped to a service node in the topology graph carries a set of resource monitoring data tightly coupled before and after its occurrence, thus forming a node attribute of log + metrics in the graph structure.

[0083] This can be understood as a spatiotemporal collaborative process: in the time dimension, high-precision timestamps are used to achieve millisecond alignment; in the spatial dimension, Trace IDs are used to bind logs to call chain nodes; and then monitoring data is mapped to the same service node according to time nodes. Ultimately, the call dependency topology graph no longer merely reflects static call relationships between services, but evolves into a dynamic fault propagation graph with real-time performance indicators. Thus, when subsequent causal analysis is performed, the server can simultaneously compare the semantics of abnormal logs and resource fluctuation trends on the same node to determine whether the anomaly stems from code logic defects or is induced by underlying resource bottlenecks.

[0084] The following section provides a possible implementation method for performing causal analysis based on the target call dependency topology graph and determining the source event and source node of the abnormal event based on the causal analysis results.

[0085] In this embodiment, the target call dependency topology graph includes multiple target service nodes, each mapped with target log data and corresponding resource monitoring data. The server can calculate the intra-node causal score between the resource monitoring data and the target log data for each target service node, and the inter-node causal score between the upstream service node and the target service node. The server then verifies the validity of the intra-node and inter-node causal scores to obtain valid causal scores. Based on these valid causal scores, the server performs reverse tracing of the target call dependency topology graph to determine the source event and source node of the abnormal event.

[0086] In this embodiment, when determining the source node of an abnormal event, the server does not rely solely on single-point log errors or failure markers at the end of the call chain, but instead uses a combination of dual causal score calculation and multi-layer validity verification to achieve a reliable reconstruction of the fault propagation path.

[0087] In practical applications, the server can first distinguish two types of causal relationships: one is the correlation between the resource monitoring data inside the target service node and the target log data generated by that node, i.e., the intra-node causal score; the other is the degree of influence of the state changes of the upstream service node on the abnormal state of the target service node, i.e., the inter-node causal score. These two types of scores together constitute a complete characterization of the anomaly propagation mechanism.

[0088] Specifically, the server can use a causal discovery algorithm (such as the PC algorithm) to calculate the causal score within a node based on resource monitoring data (such as CPU utilization, memory usage, and number of database connections) and the structured output (including template content, occurrence frequency, and time distribution) corresponding to the target log cluster to which the target log data belongs.

[0089] Furthermore, the server can also extract the state time-series data of the currently abnormal target service node and its upstream associated nodes based on the target call dependency topology graph, and calculate the inter-node causality score characterizing the probability that the upstream node caused the abnormality of the target service node. This state time-series data can be overall data on the service node, such as log data and resource monitoring data.

[0090] In addition, the server can use the initial time of the target log cluster as a reference to retrieve log data within a preset time window and use it as a preliminary cause to participate in the calculation of causal scores between nodes.

[0091] In this embodiment, the window is not of fixed length but adaptively adjusted based on the average service link latency, initially set to 5 to 15 minutes. Within this window, the server not only scans abnormal logs already marked as target log clusters but also performs a full search of all log types, including normal business log clusters, to uncover potential precursors, such as configuration change logs, connection pool warnings, or GC pause logs. Although these precursors do not trigger regular alarms, they occur before the anomaly outbreak and possess semantic recognizability and temporal coupling, thus allowing them to be incorporated into the data source for upstream node state modeling, thereby supporting the accurate calculation of causal scores between nodes.

[0092] In this process, the server does not directly accept the originally calculated intra-node causal scores and inter-node causal scores. Instead, it performs three parallel validity checks. The first is a statistical significance threshold determination, used for initial screening at the data level. Results with scores below a preset threshold (e.g., 0.85) are removed to ensure that the retained causal relationships have a basic level of confidence. The second is a topological prior constraint, based on the physical architecture, allowing causal scores to enter subsequent analysis only when two service nodes have a real call edge in the call dependency topology graph, preventing the algorithm from fabricating false causal paths without call relationships. The third is a temporal causality check, which further restricts the temporal logic, requiring that the occurrence time of the causal event must be earlier than the result event, and the time interval between the two must fall within the reasonable propagation range of the service call chain. This eliminates erroneous inferences that reverse cause and effect and filters out abnormal associations with significantly distorted time spans.

[0093] In summary, the causal score retained after the above triple verification is the valid causal score. The server can use this score to trace back the target call dependency topology, comprehensively evaluating the positional weight and score of each node along the confirmed valid causal links. Ultimately, the node and event at the very top with the highest overall score are identified as the source node and source event. Understandably, this source event may be the aforementioned preceding triggering event, or it may be a performance event reflected in resource monitoring data.

[0094] Furthermore, in related technologies, static thresholds are commonly used for anomaly detection in log analysis and alarm processes. Because static thresholds cannot detect trends and rates of change, "slow faults" such as slow resource leaks and gradual traffic surges will not trigger any alarms or be identified as risk signals before evolving into irreversible collapse. This results in the system remaining in a passive response state until the fault fully erupts, at which point emergency response is not initiated. To address these issues, the server can input log data into a pre-trained prediction model for processing to obtain anomaly prediction results. If the anomaly prediction results indicate an anomaly risk within a preset timeframe, a warning message will be issued.

[0095] In this embodiment, the server can input the acquired log data into the prediction model for anomaly prediction. In one possible implementation, to reduce computational load, the semantic features, occurrence frequency, and other information corresponding to the log clusters obtained after clustering, along with corresponding resource monitoring data (such as CPU utilization, memory usage, database connection count, and interface response latency), can be used to construct a multivariate time series as input features for the prediction model.

[0096] In this embodiment, the prediction model can be a time series prediction model based on the Transformer architecture, an Informer model, or a Long Short-Term Memory (LSTM) network. Its training process uses the actual change trajectory of the above-mentioned multidimensional indicators in the historical period as a supervision signal to learn the dynamic coupling relationship and lag dependency law between the variables.

[0097] Optionally, the server can perform rolling forecasts in real time, extrapolating the trend curves of key indicators over a future period, with the forecast period set to one hour. The model output not only includes point estimates for each future time point but also covers the forecast confidence interval, thereby supporting dynamic threshold determination.

[0098] Specifically, the server can determine whether the prediction curve will exceed a preset risk threshold at some point in the future. This threshold is not a fixed value, but a boundary condition that is adaptively adjusted based on the business SLA, the service historical baseline, and the current load status.

[0099] For example, if the model predicts that the error log density will exceed the critical value of 300 logs per minute after 42 minutes, and this upward trend continues for more than three consecutive prediction steps, while the memory usage prediction curve enters a high plateau period, the server can be confirmed to have a risk of slow failure caused by memory leaks.

[0100] In this way, the server can proactively trigger the early warning mechanism before the anomaly actually occurs, generating early warning information in natural language format. The content includes the name of the predicted indicator, the type of change trend, the expected time point of exceeding the limit, the corresponding risk threshold, and the specific type of fault that may be induced, so as to achieve early warning and facilitate proactive response by the server or engineers to carry out emergency early warning processing in advance.

[0101] In addition, to lower the barrier to entry for users and make it easier for them to view relevant analysis results at any time, the server can also be equipped with a large language model. The server can then obtain user questions about the log data, process the user questions through the large language model, and output response information.

[0102] Understandably, this large language model has learned from a vast amount of specialized corpus, including system logs, fault cases, indicator meanings, call chain structures, and handling strategies. When a user enters a question in natural language, such as "What anomalies have occurred in the past half hour?", the large language model can first perform intent parsing on the statement, identifying the time range, query object, and question type. Then, it rewrites the intent into internal instructions executable by the device, such as extracting target log clusters of all anomalies within the last 30 minutes, loading status snapshots of corresponding service nodes, and backtracking original log samples. The entire process requires no knowledge of SQL syntax or regular expressions from the user and does not rely on preset dashboard paths.

[0103] In this process, the large language model first transforms the user's question into intent, mapping vague spoken language into precise operational instructions. Then, it performs a comprehensive analysis, proactively integrating all existing analytical results, such as root cause localization conclusions, predicted trend curves, key log fragments, related monitoring metric values, and call chain topology paths, to generate a clearly structured natural language diagnostic report. Finally, for the confirmed fault type, it can match or instantly derive feasible handling strategies from the built-in knowledge base, such as "recommend restarting service X" or "recommend downgrading service Y," and can further generate corresponding execution scripts or code patch (Pull Request, PR) content.

[0104] For example, if the model determines that an interface timeout is caused by the database connection pool being exhausted and there are no obvious resource anomalies upstream, it will not recommend expanding the cluster, but will instead prioritize generating lightweight intervention instructions such as releasing connections and adjusting the maximum number of active connections.

[0105] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of a log analysis device is given below. Optionally, the log analysis device may employ the methods described above. Figure 1 The server's device structure is shown. For further details, please refer to... Figure 4 , Figure 4 This is a functional block diagram of a log analysis device provided in an embodiment of this application. It should be noted that the log analysis device provided in this embodiment has the same basic principle and technical effects as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The log analysis device includes: an acquisition module and an analysis module.

[0106] This acquisition module is used to acquire log data from each service node.

[0107] Understandably, this acquisition module can be used to perform the above step S20.

[0108] This analysis module is used to construct a call dependency topology graph between service nodes based on the call chain ID and call chain data corresponding to each log data, perform semantic density clustering and filtering on each log data, and obtain at least one target log cluster; wherein, each target log cluster includes multiple target log data, and multiple target log data correspond to the same abnormal event.

[0109] This analysis module is also used to map each target log data to a service node in the call dependency topology graph according to the timestamp information of the target log data, thereby obtaining the target call dependency topology graph.

[0110] This analysis module is also used to perform causal analysis based on the target call dependency topology graph, and to determine the source event and source node of the abnormal event based on the causal analysis results.

[0111] Understandably, this analysis module can be used to perform steps S21 to S23 as described above.

[0112] Optionally, the analysis module is also used to filter log data according to a preset whitelist rule base to obtain candidate log data; extract semantic features from the candidate log data, and perform density clustering on the candidate log data according to the semantic features to obtain multiple log clusters; and filter each log cluster through a preset filtering algorithm to obtain at least one target log cluster.

[0113] Optionally, this analysis module is also used to obtain resource monitoring data corresponding to each target log data in the target log cluster; the resource monitoring data characterizes the system running performance when the target log data is generated; and the resource monitoring data corresponding to the target log data is mapped to the call dependency topology graph.

[0114] Optionally, the analysis module is also used to calculate the intra-node causal score between the resource monitoring data and the target log data corresponding to each target service node, and to calculate the inter-node causal score between the upstream service node and the target service node; to verify the validity of the intra-node causal score and the inter-node causal score, to obtain a valid causal score, and to perform reverse tracing of the target call dependency topology based on the valid causal score to determine the source event and source node of the abnormal event.

[0115] Optionally, the analysis module is also used to input log data into a pre-trained prediction model for processing to obtain anomaly prediction results; if the anomaly prediction results indicate that there is an abnormal risk within a preset time period, an early warning message is issued.

[0116] Optionally, this analysis module is also used to obtain user questions about the log data; process the user questions using a large language model; and output response information.

[0117] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory shown is either stored in or embedded in the server's operating system (OS), and can be used by... Figure 1 The processor executes the commands. Meanwhile, the data and program code required to execute these modules can be stored in memory.

[0118] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the log analysis method provided in this application.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0121] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0122] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A log analysis method, characterized in that, The method includes: Obtain log data from each service node; Based on the call chain ID and call chain data corresponding to each log data, a call dependency topology graph between each service node is constructed. Semantic density clustering and filtering are performed on each log data to obtain at least one target log cluster. Each target log cluster includes multiple target log data, and the multiple target log data correspond to the same abnormal event. Based on the timestamp information of the target log data, each target log data is mapped to a service node in the call dependency topology graph to obtain the target call dependency topology graph; Causal analysis is performed based on the target call dependency topology graph, and the source event and source node of the abnormal event are determined according to the causal analysis results.

2. The method according to claim 1, characterized in that, The step of performing semantic density clustering and filtering on each of the log data to obtain at least one target log cluster includes: The log data is filtered according to a preset whitelist rule base to obtain candidate log data; Semantic features are extracted from the candidate log data, and density clustering is performed on the candidate log data based on the semantic features to obtain multiple log clusters. By filtering each of the log clusters using a preset filtering algorithm, at least one target log cluster is obtained.

3. The method according to claim 2, characterized in that, The filtering algorithm includes at least one of noise filtering, semantic importance filtering, and preset rule filtering.

4. The method according to claim 1, characterized in that, The method further includes: Obtain resource monitoring data corresponding to each target log data in the target log cluster; the resource monitoring data characterizes the system operating performance when the target log data is generated. Map the resource monitoring data corresponding to the target log data to the call dependency topology graph.

5. The method according to claim 4, characterized in that, The target call dependency topology diagram includes multiple target service nodes, and the target service nodes are mapped with the target log data and the corresponding resource monitoring data. The step of performing causal analysis based on the target call dependency topology graph, and determining the source event and source node of the abnormal event based on the causal analysis results, includes: For each target service node, calculate the intra-node causality score between the resource monitoring data and the target log data corresponding to the target service node, and calculate the inter-node causality score between the upstream service node and the target service node. The validity of the intra-node causal scores and inter-node causal scores is verified to obtain valid causal scores. Based on the valid causal scores, the target call dependency topology is traced in reverse to determine the source event and source node of the abnormal event.

6. The method according to claim 1, characterized in that, The method further includes: The log data is input into a pre-trained prediction model for processing to obtain anomaly prediction results. If the anomaly prediction result indicates that there is an abnormal risk within a preset time period, an early warning message will be issued.

7. The method according to claim 1, characterized in that, The method further includes: Obtain user questions regarding the log data; The user's question information is processed using a large language model, and the response information is output.

8. A log analysis device, characterized in that, The device includes: The acquisition module is used to acquire log data from each service node; The analysis module is used to construct a call dependency topology graph between the service nodes based on the call chain ID and call chain data corresponding to each log data, and to perform semantic density clustering and filtering on each log data to obtain at least one target log cluster; wherein, each target log cluster includes multiple target log data, and the multiple target log data correspond to the same abnormal event; The analysis module is also used to map each target log data to a service node in the call dependency topology graph according to the timestamp information of the target log data, so as to obtain the target call dependency topology graph; The analysis module is also used to perform causal analysis based on the target call dependency topology graph, and to determine the source event and source node of the abnormal event based on the causal analysis results.

9. A server, characterized in that, It includes a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the method of any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images