Log intelligent analysis fault diagnosis system for distributed software

Abstract

This invention discloses a log intelligent analysis and fault diagnosis system for distributed software, relating to the field of software fault diagnosis technology. It includes a data acquisition module for acquiring business log streams and orchestration event streams. Each log entry in the business log stream includes a service name and a first timestamp. The orchestration event stream includes discrete events for each container under the corresponding service name and the running nodes deployed by each container. This invention transforms fault analysis from traditional instantaneous, discrete, static deployment fault analysis to dynamic, continuous analysis through the generated enhanced trajectory. It can accurately reconstruct the true trajectory of business requests during execution, thereby detecting a series of novel, hidden faults directly caused by container dynamism, effectively improving the accuracy of fault detection.

Description

Technical Field

[0001] This invention relates to the field of software fault diagnosis technology, specifically to a log intelligent analysis fault diagnosis system for distributed software. Background Technology

[0002] With the development of cloud computing and microservice architecture, distributed software systems are widely used on container orchestration platforms. Log analysis is one of the core methods for fault diagnosis in distributed software systems.

[0003] Traditional call chain tracing assigns a globally unique tracing identifier to each external request and, during inter-service calls, can afterwards link records belonging to the same business request that are scattered across different service logs to reconstruct the complete request path. This method can effectively locate performance bottlenecks or error nodes at the service level.

[0004] However, in a container-driven, dynamically scheduled environment, the containers and runtime nodes of services change frequently. Traditional call chain tracing directly analyzes static deployments provided by orchestration platforms, which orchestrate the instantaneous states of discrete events within containers. This only records "Service A called Service B," completely losing crucial physical context information such as "which specific container and runtime node of Service A initiated the call, and which container and runtime node of Service B responded." When containers start and stop frequently, log associations based on static deployments become completely ineffective, making it impossible to accurately reconstruct the true trajectory of business requests during execution. Consequently, it fails to detect a series of new, hidden faults directly caused by the dynamism of containers, thus affecting the accuracy of diagnostic results. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a log intelligent analysis and fault diagnosis system for distributed software.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A log intelligent analysis and fault diagnosis system for distributed software includes:

[0008] Data acquisition module: used to acquire business log streams and orchestration event streams. Each log in the business log stream includes a service name and a first timestamp. The orchestration event stream includes discrete events of each container under the corresponding service name and the running nodes deployed by each container.

[0009] Data processing module: used to associate the creation and termination events in the discrete events of each container, calculate the survival time interval of the container under the corresponding service name, and associate the corresponding service name, container and running node with the survival time interval to form a spatiotemporal mapping table;

[0010] Take any log in the business log stream as the target log, match the service name and first timestamp of the target log with the spatiotemporal mapping table, and extract the container and running node that successfully match.

[0011] The extracted containers and running nodes are used as physical contexts and merged with the target logs to form context-enhanced logs;

[0012] The business log stream is traversed for matching, and all generated context enhancement logs are sorted and aggregated according to the first timestamp to form an enhanced business trajectory;

[0013] Data Judgment Module: Used to extract all context enhancement logs that are within a preset time window and run on the same node from the enhanced business trajectory;

[0014] The extracted context enhancement logs are grouped by service name to obtain a set of service names running simultaneously on the running node, and a sequence of performance indicators representing resource utilization or response latency is extracted from the context enhancement logs of the corresponding service name.

[0015] Calculate the Pearson correlation coefficient of the performance index sequence corresponding to different service names within a preset time window;

[0016] If there is at least one pair of Pearson correlation coefficients greater than a preset threshold, and the corresponding performance index sequence shows an upward trend within a preset time window, then a resource contention failure is determined to have occurred on the running node.

[0017] Data output module: Used to generate diagnostic reports based on the judgment results.

[0018] A method for intelligent log analysis and fault diagnosis in distributed software, applied to the system for intelligent log analysis and fault diagnosis in distributed software, includes the following steps:

[0019] Acquire business log streams and orchestration event streams. Each log entry in the business log stream includes a service name and a first timestamp. The orchestration event stream includes multiple containers under the corresponding service name, discrete events of each container, and running nodes deployed by each container.

[0020] The creation and termination events of each container are associated with discrete events. The survival time interval of the container under the corresponding service name is calculated, and the corresponding service name, container and running node are associated with the survival time interval to form a spatiotemporal mapping table.

[0021] Take any log in the business log stream as the target log, match the service name and first timestamp of the target log with the spatiotemporal mapping table, and extract the container and running node that successfully match.

[0022] The extracted containers and running nodes are used as physical contexts and merged with the target logs to form context-enhanced logs;

[0023] The business log stream is traversed for matching, and all generated context enhancement logs are sorted and aggregated according to the first timestamp to form an enhanced business trajectory;

[0024] In the enhanced business trajectory, the following judgments are made:

[0025] Extract all context-enhanced logs from nodes that are running within a preset time window;

[0026] The extracted context enhancement logs are grouped by service name to obtain a set of service names running simultaneously on the running node, and a sequence of performance indicators representing resource utilization or response latency is extracted from the context enhancement logs of the corresponding service name.

[0027] Calculate the Pearson correlation coefficient of the performance index sequence corresponding to different service names within a preset time window;

[0028] If there is at least one pair of Pearson correlation coefficients greater than a preset threshold, and the corresponding performance index sequence shows an upward trend within a preset time window, then a resource contention failure is determined to have occurred on the running node.

[0029] A diagnostic report is generated based on the assessment results.

[0030] Preferably, the determination of the enhanced service trajectory further includes:

[0031] In the enhanced business trajectory, select two context enhancement logs that belong to the same service name and have consecutive first timestamps;

[0032] Obtain the containers and corresponding first timestamps of the two context-enhanced logs respectively;

[0033] Calculate the time difference between two first timestamps;

[0034] If two context-enhanced logs are in different containers and the time difference is less than the preset minimum stable operation threshold, then an instance skipping failure is determined to have occurred.

[0035] Preferably, the determination of the enhanced service trajectory further includes:

[0036] Traverse the enhanced business trajectory, and for each pair of context enhancement logs with a direct business call sequence, extract the running nodes of the two context enhancement logs respectively;

[0037] A network policy matrix is ​​pre-defined, where the element (x, y) indicates whether running node x is allowed to send a service request to running node y;

[0038] If both running nodes are prohibited when querying the network policy matrix, a network topology violation fault event is generated for the service call represented by the context enhancement log.

[0039] Preferably, the service name and first timestamp of the target log are matched with the spatiotemporal mapping table, specifically:

[0040] Using the service name and first timestamp of the target log as the query key, search the spatiotemporal mapping table for records with the same service name and whose first timestamp falls within the survival time interval;

[0041] If a unique record is found, the match is successful; if multiple records are found, the record with the most recent time is selected.

[0042] If no record is found, the match fails, and a context missing exception event is generated and logged.

[0043] Based on the context-deficient exception event, it can be directly determined that there is a log loss failure caused by the instantaneous destruction of the container or the loss of events.

[0044] Preferably, the generation of diagnostic reports includes credibility quantification, specifically including:

[0045] Assign an initial credibility score to the enhanced business trajectory;

[0046] Based on the number of detected resource contention failures, instance skipping failures, network topology violation failures, and log disconnection failures, the initial credibility score is deducted according to the preset first deduction weight, second deduction weight, third deduction weight, and fourth deduction weight, respectively.

[0047] The final credibility score obtained after deduction is used as the reliability measure of the business trajectory and written into the diagnostic report.

[0048] Preferably, generating a diagnostic report also includes root cause correlation reasoning, specifically including:

[0049] Construct a fault knowledge graph, whose nodes include fault type, discrete events of each container, and running nodes deployed by each container;

[0050] The specific fault events detected and their physical context information are mapped to phenomenon nodes in the fault knowledge graph;

[0051] In the fault knowledge graph, reverse reasoning is performed along the causal relationship edges to locate the upstream root cause node that has the highest probability of causing the phenomenon node to occur.

[0052] One or more root cause node information obtained through reasoning is converted into natural language descriptions and attached to the diagnostic report.

[0053] Preferably, the orchestration event stream is obtained in real time through the event interface of the container orchestration platform. The discrete events include at least container creation events, scheduling events, running events, and termination events, and each event record contains a corresponding event timestamp.

[0054] Preferably, the spatiotemporal mapping table is continuously and dynamically updated, specifically as follows:

[0055] When a container creation event is detected, extract the corresponding service name, container, running node, and event timestamp, insert a new record in the time-space mapping table, and set the corresponding event timestamp to the container startup time.

[0056] When a termination event of the same container is detected, the corresponding event timestamp is extracted, and the container termination time of the corresponding record is updated.

[0057] Preferably, it also includes the maintenance steps of the spacetime mapping table:

[0058] Set an expiration time threshold for historical data;

[0059] Periodically scan each record in the spatiotemporal mapping table and calculate the time difference between the current moment and the event timestamp of the container's termination event;

[0060] If the time difference is greater than the expiration time threshold, the record is deleted from the spatiotemporal mapping table.

[0061] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0062] This invention achieves end-to-end business process reconstruction through the generated enhanced trajectories, breaking through the limitations of traditional business trajectories that only contain business information. By using both business and physical dimensions, it completely reconstructs the entire execution process of business requests, revealing which service is in which container and node, and what operation it performed at what time. This enables a visual correlation between business flow and physical deployment, providing a precise analysis unit for subsequent fault diagnosis. The corresponding enhanced business trajectory can be quickly located through the Trace ID, and the business status and physical environment of each link can be traced along the trajectory time sequence. This eliminates the need to sift through massive discrete logs to find related information, effectively reducing fault location time, improving operational efficiency, and transforming fault analysis from traditional instantaneous discrete static deployment fault analysis to dynamic continuous analysis. It can accurately reconstruct the true trajectory of business requests during execution, thereby detecting a series of new hidden faults directly caused by container dynamism, effectively improving the accuracy of fault detection. Attached Figure Description

[0063] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0064] Figure 1 This is a flowchart illustrating the enhanced business trajectory generation process of the present invention.

[0065] Figure 2 This is a flowchart of the resource contention fault judgment process of the present invention;

[0066] Figure 3 This is a flowchart illustrating the jump fault judgment process of this invention.

[0067] Figure 4 This is a flowchart for determining network topology violations according to the present invention. Detailed Implementation

[0068] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0069] like Figure 1-4 As shown, a log intelligent analysis and fault diagnosis system for distributed software includes:

[0070] Data acquisition module: used to acquire business log streams and orchestration event streams. Each log in the business log stream includes the service name and the first timestamp. The orchestration event stream includes discrete events of each container under the corresponding service name and the running nodes deployed by each container.

[0071] Data processing module: used to associate the creation and termination events in the discrete events of each container, calculate the survival time interval of the container under the corresponding service name, and associate the corresponding service name, container and running node with the survival time interval to form a spatiotemporal mapping table;

[0072] Take any log in the business log stream as the target log, match the service name and first timestamp of the target log with the spatiotemporal mapping table, and extract the container and running node that successfully match.

[0073] The extracted containers and running nodes are used as physical contexts and merged with the target logs to form context-enhanced logs;

[0074] The business log stream is traversed for matching, and all generated context enhancement logs are sorted and aggregated according to the first timestamp to form an enhanced business trajectory;

[0075] Data Judgment Module: Used to perform the following judgments in enhanced business trajectories:

[0076] Extract all context-enhanced logs from nodes that are running within a preset time window;

[0077] The extracted context enhancement logs are grouped by service name to obtain a set of service names running simultaneously on the running node, and a sequence of performance indicators representing resource utilization or response latency is extracted from the context enhancement logs of the corresponding service name.

[0078] Calculate the Pearson correlation coefficient of the performance index sequence corresponding to different service names within a preset time window;

[0079] If there is at least one pair of Pearson correlation coefficients greater than a preset threshold, and the corresponding performance index sequence shows an upward trend within a preset time window, then a resource contention failure is determined to have occurred on the running node.

[0080] Data output module: Used to generate diagnostic reports based on the judgment results.

[0081] A method for intelligent log analysis and fault diagnosis in distributed software, applied in an intelligent log analysis and fault diagnosis system for distributed software, includes the following steps:

[0082] Acquire business log streams and orchestration event streams. Each log entry in the business log stream includes the service name and the first timestamp. The orchestration event stream includes multiple containers under the corresponding service name, discrete events of each container, and the running nodes deployed by each container.

[0083] Specifically, the acquired data is explained as follows:

[0084] Business log streams are structured or semi-structured log sequences generated by microservices or business modules in distributed software when executing specific business logic. Essentially, they are textual records of the business execution process, reflecting the status, results, and time points of business operations. Examples include order creation and payment callback logs in an e-commerce system's order service, and login verification and information query logs in a user service.

[0085] By deploying a log collection agent inside each container, new log content is captured in real time. The collected logs are then transmitted to a centralized log storage system through a publish or subscribe model, forming a continuously input business log stream.

[0086] Service name: Used to identify which microservice or business module the current log belongs to.

[0087] First timestamp: Used to record the precise time when the log was generated.

[0088] Orchestration event streams are sequences of events generated by a container orchestration platform during the entire lifecycle management of containers. Essentially, they are records of state changes in the container's runtime environment. Orchestration event streams reflect which node a service's corresponding container is running on, as well as state changes such as container creation, scheduling, operation, and termination. Examples include events such as an order service container being scheduled to node A, or a payment service container abnormally terminating.

[0089] Containers are lightweight, portable virtualization technologies that encapsulate applications, their libraries, configuration files, and runtime environments together. This enables environment isolation and on-demand resource allocation, ensuring consistent application performance across different deployment environments. A single business service can typically correspond to multiple containers; that is, a service name can represent multiple containers.

[0090] It obtains data in real time through the event interface of the container orchestration platform.

[0091] Discrete events for each container: Discrete events are used to detect changes in the container's running state and include at least key node events such as container creation events, scheduling events, running events, and termination events.

[0092] The running nodes of each container deployment: The running node is the location of the physical machine or virtual machine where the container is actually deployed. Common forms include standardized identifiers such as node IP address and node name.

[0093] The creation and termination events of each container are associated with discrete events. The survival time interval of the container under the corresponding service name is calculated, and the corresponding service name, container and running node are associated with the survival time interval to form a spatiotemporal mapping table.

[0094] Specifically, the construction steps of the spatiotemporal mapping table are as follows:

[0095] Step 1: Filter the core discrete events for each container from the orchestrated event stream, namely the container creation and termination events, and extract the key fields for each type of event:

[0096] The key field of a container: It gives the container a unique identifier to distinguish different containers;

[0097] Key fields of the service name: Associated with the container identifier, used to distinguish the business services carried by different containers;

[0098] Key fields for running nodes: the IP address or name of the node where the container is actually deployed;

[0099] The key field of the event timestamp is used to identify the specific time when the corresponding event occurred.

[0100] Key fields for creating and terminating events: assigning an event type identifier to the corresponding event.

[0101] Step 2: Group the filtered creation and termination events according to the container's unique identifier to ensure that all events of the same container are grouped together. Within the event group of the same container, a creation event is matched with a corresponding termination event. Under normal circumstances, a container will generate one creation event and one termination event from startup to termination, forming a complete container lifecycle loop.

[0102] If the same container generates multiple creation or termination events due to reasons such as restart, they are paired in chronological order, with the previous termination event corresponding to the next creation event, forming multiple consecutive survival intervals. If the container is still running, i.e. no termination event has been generated, it is marked as not terminated, and the termination event is temporarily empty, to be added dynamically later.

[0103] Step 3: Based on the event timestamps of the creation and termination events of the same grouped container, calculate the lifespan interval of the corresponding container:

[0104] Survival time interval = [creation event timestamp, termination event timestamp], during which the event is in a valid running state;

[0105] Example: Container A for a certain order service:

[0106] The key field for creating the event timestamp is: 2025-12-25T10:00:00.000Z;

[0107] The key field of the event stamp for the termination event is: 2025-12-25T10:30:00.000Z;

[0108] The key fields for the survival event interval of container A are: [2025-12-25T10:00:00.000Z, 2025-12-25T10:30:00.000Z].

[0109] Step 4: Link and store the service name, container, running node, and survival range to form a structured spatiotemporal mapping table. The key fields of the spatiotemporal mapping table are:

[0110] Service Name: The name of the associated business service, such as "order-service" and "pay-service".

[0111] Container: A unique identifier for a container (ContainerID), such as "docker: / / a1b2c3d4e5f6";

[0112] Running node: The physical node identifier (NodeID / NodeIP) of the container deployment, such as "node-01" or "192.168.1.100";

[0113] Lifetime interval: e.g., [2025-12-25T10:00:00.000Z, 2025-12-25T10:30:00.000Z];

[0114] Status flag: Marks the current state of the container, such as running or terminated;

[0115] The spatiotemporal mapping table uses dynamically updated structured storage, supporting high-concurrency queries and real-time updates.

[0116] Example of some tables:

[0117] ServiceName ContainerID NodeID LifetimeInterval Status order-service docker: / / a1b2c3d4e5f6 node-01 [2025-12-25T10:00:00.000Z, 2025-12-25T10:30:00.000Z] Terminated pay-service docker: / / f6e5d4c3b2a1 node-02 [2025-12-25T10:05:00.000Z, 2025-12-25T11:00:00.000Z] Running

[0118] Furthermore, by creating a spatiotemporal mapping table, the corresponding container and running node can be queried from the spatiotemporal mapping table using the service name and the first timestamp. This solves the problem that business logs only contain service names and cannot locate the physical running environment. The container creation and termination events and running node information in the orchestration event stream are associated with the service name to form a complete mapping relationship. Then, by matching the service name and the first timestamp in two dimensions, the original logs are supplemented with accurate physical context, realizing one-click traceability of business logs and physical deployments, which greatly reduces the difficulty and time cost of fault location.

[0119] Take any log in the business log stream as the target log, match the service name and first timestamp of the target log with the spatiotemporal mapping table, and extract the container and running node that successfully match.

[0120] Specifically, the matching process is as follows:

[0121] Step 1: Use the service name of the target log as the first query condition, and filter out all records whose service names match exactly in the spatiotemporal mapping table. This narrows the query range and avoids full table scans. Especially in large-scale distributed clusters, the spatiotemporal mapping table may contain a massive number of container records. Filtering by service name can greatly improve matching efficiency.

[0122] Step 2: Further validate the filtered records with the same service name, verifying whether the first timestamp of the target log falls within the record's lifespan interval. The specific validation is as follows:

[0123] If the first timestamp of the target log is later than the container's creation event timestamp but earlier than the container's termination time timestamp, then the record passes the time verification.

[0124] Otherwise, exclude the record.

[0125] It should be noted that for containers that are running in the time-space mapping table, there is no termination event. As long as the first timestamp of the target log is not later than the current time, the time verification can be passed. For terminated containers, it is necessary to strictly ensure that the first timestamp of the target log is within the survival time interval.

[0126] Step 3: After filtering by service name and verifying by timestamp, confirm the matching results based on the number of remaining records:

[0127] If there is one record remaining, then that record will be directly identified as the final matching record, and the match will be successful.

[0128] If there are multiple remaining records, meaning there are multiple containers running under the same service name and within the same time interval, then according to the principle of closest time, the difference between the first timestamp of the target log and the creation event timestamp of each record is calculated, and the record with the smallest difference is selected as the final matching record, and the match is successful.

[0129] If no records remain, the match fails.

[0130] The extracted containers and running nodes are used as physical contexts and merged with the target logs to form context-enhanced logs;

[0131] Specifically, after a successful match, the key fields of the container and the key fields of the running node are extracted from the record obtained after the successful match. The two core physical context information are then merged with the original target log to generate a context-enhanced log.

[0132] For example, the original target log is:

[0133] Service name: order-service;

[0134] First timestamp: 2025-12-25T10:05:30Z;

[0135] Log content: Order created successfully;

[0136] The matching record is:

[0137] Service name: order-service;

[0138] Container: docker: / / a1b2c3d4e5f6;

[0139] Running node: node-01;

[0140] Survival range: [2025-12-25T10:00:00Z, 2025-12-25T10:30:00Z];

[0141] The merged context-enhanced log is as follows:

[0142] Service name: order-service;

[0143] First timestamp: 2025-12-25T10:05:30Z;

[0144] Container: docker: / / a1b2c3d4e5f6;

[0145] Running node: node-01;

[0146] Log content: Order created successfully.

[0147] The business log stream is traversed for matching, and all generated context enhancement logs are sorted and aggregated according to the first timestamp to form an enhanced business trajectory;

[0148] Specifically, the steps for generating enhanced business trajectories are as follows:

[0149] Step 1: Following the process of generating context-enhanced logs by matching single target logs, perform batch operations on each log in the business log stream. Use a streaming processing mode to traverse the business log stream, extracting key fields from each original log in turn. For each original log encountered, use the service name and first timestamp as the time verification basis to query and match the spatiotemporal mapping table, extract the corresponding key fields and key fields of the running node, and generate context-enhanced logs containing business information and physical context. Temporarily store the generated context-enhanced logs in a memory buffer queue to avoid processing congestion due to excessive instantaneous log volume, while providing a stable data stream input for subsequent sorting steps.

[0150] Step Two: In a distributed environment, different services and logs may experience time-series disorder due to network transmission delays, node clock skew, etc. For example, the log of service A may actually be generated earlier than that of service B, but arrive at the processing node later. Therefore, time-series alignment needs to be achieved through unified sorting. The specific process is as follows:

[0151] For all context-enhanced logs in the buffer queue, verify the format and precision of the first timestamp, ensuring that the precision is uniform to the millisecond level to avoid sorting deviations due to inconsistent formats;

[0152] All context enhancement logs are sorted in ascending order using the first timestamp as the sorting key. This ensures that logs with the same timestamp, i.e. logs generated by different services or containers at the same time, retain their original relative order and avoid relationship confusion caused by sorting.

[0153] If there are sorting anomalies caused by node time drift, such as log A having a first timestamp earlier than log B, but A should actually be later than B in the business logic, a secondary correction should be performed based on the business causal markers in the logs. The log order should be adjusted according to the causal relationship of the business call chain to ensure that the timing is consistent with the business execution logic. For example:

[0154] First log entry: Service name: pay-service, running node: node-02; First timestamp: 2025-12-25T10:05:32.123Z;

[0155] Second log entry: Service name: order-service; Running node: node-01; First timestamp: 2025-12-25T10:05:30.456Z;

[0156] Third log entry: Service name: user-service, running node: node-03; First timestamp: 2025-12-25T10:05:31.789Z;

[0157] The first, second, and third log entries correspond to the payment service, order service, and user service, respectively. Based on the causal relationship of the business call chain, the order service is generated first, followed by the user service, and finally the payment service. The order has been rearranged as follows:

[0158] Second log entry: Service name: order-service, running node: node-01; First timestamp: 2025-12-25T10:05:30.456Z;

[0159] Third log entry: Service name: user-service, running node: node-03; First timestamp: 2025-12-25T10:05:31.789Z;

[0160] First log entry: Service name: pay-service, running node: node-02; First timestamp: 2025-12-25T10:05:32.123Z.

[0161] Step 3: The sorted enhanced logs are still discrete entries. They need to be aggregated across multiple dimensions to create a traceable business chain, while retaining the correlation information from the physical environment dimension. Specifically:

[0162] A business chain is a business request, such as creating an order or paying for an order. These complete business operations typically correspond to multiple log entries. A business request is a complete process across services. For example, a request to create an order will trigger multiple services such as the order service, user service, and payment service to work together. Each triggered service will generate corresponding log records. These logs from different services are associated with the same business chain identifier (Trace ID) and aggregated to form the complete trajectory of the business request. This results in a single-request full-chain enhanced trajectory, which fully restores the execution process of the corresponding request from the entry service to all downstream dependencies, and includes the container and runtime node information of each execution stage.

[0163] The following auxiliary trajectory is generated by combining multiple dimensions:

[0164] Aggregate all context-enhanced logs for the same service name to form a single-service business trajectory;

[0165] Aggregate all context-enhanced logs on the same running node to form a single-node multi-service trajectory;

[0166] Aggregate all the upper and lower enhancement logs of the same container to form a single container business trajectory;

[0167] The aggregated trajectory is lightweighted, and only key fields are retained for storage.

[0168] Furthermore, by generating enhanced trajectories, the entire business process can be restored, breaking through the limitations of traditional business trajectories that only contain business information. Through dual-dimensional information of business and physical data, the entire execution process of business requests can be fully restored, showing which service is in which container and node, and what operation it performed at what time. This enables a visual correlation between business flow and physical deployment, providing a precise analysis unit for subsequent fault diagnosis. The corresponding enhanced business trajectory can be quickly located through the Trace ID, and the business status and physical environment of each link can be traced along the trajectory time sequence. There is no need to sift through massive discrete logs to find related information, which will effectively reduce fault location time, improve operation and maintenance efficiency, and transform fault analysis from traditional instantaneous discrete static deployment fault analysis to dynamic continuous analysis. It can accurately restore the real trajectory of business requests during execution, thereby detecting a series of new hidden faults directly caused by container dynamism, effectively improving the accuracy of fault detection.

[0169] In the enhanced business trajectory, the following judgments are made:

[0170] Extract all context-enhanced logs from nodes that are running within a preset time window;

[0171] Specifically, using both the running node and the time window as dual filtering conditions, the running node to be verified is selected from the enhanced business trajectory, all context enhancement logs corresponding to the running node are extracted, a fixed time interval is preset, and the context enhancement logs whose first timestamp falls within the interval are filtered out.

[0172] The extracted context enhancement logs are grouped by service name to obtain a set of service names running simultaneously on the running node, and a sequence of performance indicators representing resource utilization or response latency is extracted from the context enhancement logs of the corresponding service name.

[0173] Specifically, the filtered and extracted context-enhanced logs are grouped and aggregated according to service name. Each group contains only the context-enhanced logs of the corresponding service within the target running node and target time window. Key performance indicators that characterize resource competition are selected. These key performance indicators need to include resource utilization and business performance. When collecting raw logs, the performance indicators of containers or running nodes are collected synchronously and recorded in each log. When extracting keywords, the keywords of key performance indicators are extracted synchronously, and context-enhanced logs are generated synchronously.

[0174] Resource utilization includes CPU utilization (cpu usage, in %), memory utilization (memory usage, in %), network I / O utilization (network I / O usage, in %), and disk I / O utilization (disk I / O usage, in %).

[0175] Business performance includes interface response delay (in milliseconds) and database query delay (in milliseconds).

[0176] Furthermore, for each service group, sort them in ascending order by the first timestamp, and arrange the values ​​of the same performance metric in chronological order to form a performance metric sequence for that service.

[0177] Calculate the Pearson correlation coefficient of the performance index sequence corresponding to different service names within a preset time window;

[0178] Specifically, for different service sequences with the same running node and the same performance index, pairwise calculations are performed. If there are N services, N×(N-1) / 2 pairs of correlation coefficients need to be calculated to ensure coverage of all service combinations. The Pearson correlation coefficient r ranges from [-1, 1], where r greater than 0 indicates a positive correlation, meaning that when one index rises, the other also rises; r less than 0 indicates a negative correlation; and r=0 indicates no positive correlation. The closer the absolute value of r is to 1, the stronger the correlation.

[0179] The core calculation formula is: ;

[0180] in Let X be the covariance of sequences X and Y. , denoted as the standard deviations of sequences X and Y, respectively.

[0181] If there is at least one pair of Pearson correlation coefficients greater than a preset threshold, and the corresponding performance index sequence shows an upward trend within a preset time window, then a resource contention failure is determined to have occurred on the running node.

[0182] Specifically, it is determined whether the calculated Pearson correlation coefficient r is greater than a preset threshold. The preset threshold is set according to the business scenario and is usually set to 0.8. If there is a service pair where the Pearson correlation coefficient r is greater than the preset threshold and the performance index value of the subsequent time is greater than the performance index value of the previous time, then it is determined that a resource contention failure has occurred on the running node and the key information is recorded; if not, then it is determined that there is no resource contention failure.

[0183] Furthermore, based on the multi-dimensional correlation information of enhanced business trajectories, it breaks through the limitations of traditional methods that only look at the performance of a single service. Through a data-driven approach, it achieves automated and accurate identification of resource contention faults, eliminating the need for manual intervention in filtering logs and significantly improving fault diagnosis efficiency.

[0184] A diagnostic report is generated based on the assessment results.

[0185] The determination of enhanced business trajectory also includes:

[0186] In the enhanced business trajectory, select two context enhancement logs that belong to the same service name and have consecutive first timestamps;

[0187] Obtain the containers and corresponding first timestamps of the two context-enhanced logs respectively;

[0188] Calculate the time difference between two first timestamps;

[0189] If two context-enhanced logs are in different containers and the time difference is less than the preset minimum stable operation threshold, then an instance skipping failure is determined to have occurred.

[0190] Specifically, the target service is selected from the single-service business trajectory of the enhanced business trajectory, and all context enhancement logs corresponding to that service are extracted. The context enhancement logs under the same service are sorted in ascending order according to the key field of the first timestamp. The two adjacent context enhancement logs after sorting are selected as the judgment unit. Here, the continuous core is defined as temporally adjacent, and there are no other context enhancement logs of the same service between the two context enhancement logs.

[0191] Furthermore, from the two consecutive context-enhanced logs selected, the core decision-making key fields are precisely extracted to ensure the completeness and validity of the key fields. The specific extraction rules are as follows:

[0192] Extract the key fields of the containers of the two context-enhanced logs. These key fields come from the physical context information of the context-enhanced logs. If the key field of the container of a context-enhanced log is empty or has an invalid format, such as not conforming to the container runtime identification specification, it is marked as "container identification abnormal", the current judgment is terminated, the log is stored in the abnormal log library, and the key fields of the first timestamp of the two context-enhanced logs are extracted as the basis for calculating the time difference.

[0193] Furthermore, the time interval between two consecutive context enhancement logs is quantized by time difference to determine the timeliness of container switching and whether it is a short-term switch, as detailed below:

[0194] The time interval is calculated by subtracting the first timestamp of the previous context-enhanced log from the first timestamp of the subsequent context-enhanced log. If the result is negative, it indicates that there is a logical anomaly in the timestamps, which is marked as "time sequence anomaly" and the judgment is terminated. If the result is 0, it indicates that the two context-enhanced logs were generated at the same time, and the judgment needs to be combined with the business scenario.

[0195] Furthermore, the following judgments are made:

[0196] Determine if the key fields of the containers are the same. If they are the same, it means that the continuous context enhancement logs of the same service are still being processed by the same container; if they are different, it means that the container has been switched.

[0197] Determine if the time difference is less than the preset minimum stable operation threshold. This threshold refers to the shortest time for a business service container to run normally and stably. It is used to distinguish between abnormal short-term switching and normal scheduling switching, such as container switching caused by expansion, migration, or fault recovery. There is usually a reasonable time interval. The threshold needs to be preset in combination with the business scenario. It is usually set to 30 seconds. For core businesses, such as payment services, it can be reduced to 10 seconds, and for non-core businesses, it can be expanded to 60 seconds.

[0198] If two context-enhanced logs simultaneously satisfy the conditions of different containers and a time difference less than the preset minimum stable operation threshold, then an instance skipping failure is determined to have occurred; if only one condition is met, such as different containers but a time difference greater than the threshold, or a time difference less than the threshold but the containers are the same, then no instance skipping failure is determined, and key information is recorded.

[0199] The determination of enhanced business trajectory also includes:

[0200] Traverse the enhanced business trajectory, and for each pair of context enhancement logs with a direct business call sequence, extract the running nodes of the two context enhancement logs respectively;

[0201] A network policy matrix is ​​pre-defined, where the element (x, y) indicates whether running node x is allowed to send a service request to running node y;

[0202] If both running nodes are prohibited when querying the network policy matrix, a network topology violation fault event is generated for the service call represented by the context enhancement log.

[0203] Specifically, the enhanced business tracing traces are traversed through the single-request full-link enhanced tracing traces, and the tracing traces are aggregated by Trace ID. Since the same Trace ID corresponds to the complete execution link of the same business request, the direct call relationship is determined based on the core identifier of the full-link tracing. In the distributed tracing system, each business operation corresponds to a Span (operation unit). The Span ID is the unique identifier of the current operation, and the Parent Span ID is the identifier of the upstream operation that initiated the current operation. If the Span ID of log A is equal to the Parent Span ID of log B, and the Trace IDs of the two logs are the same, it is determined that there is a direct business call sequence between the upstream operation of log A and the downstream operation of log B, forming a log pair to be verified (upstream log A, downstream log B).

[0204] Furthermore, from each selected log pair, the running nodes corresponding to upstream and downstream operations are precisely extracted to clarify the actual node call path. The specific extraction rules are as follows:

[0205] Extract key fields of the running nodes, such as node-01 and node-03, from the upstream and downstream logs respectively, and form a call relationship from the upstream node to the downstream node, denoted as (x, y), where x is the running node of the upstream log and y is the running node of the downstream log.

[0206] For example: Extracting node call relationships from the two log pairs above:

[0207] Log pair 1: Upstream node x is node-01, downstream node y is node-03, and the call relationship is (node-01, node-03).

[0208] Log pair 1: Upstream node x is node-01, downstream node y is node-02, and the call relationship is (node-01, node-02).

[0209] Furthermore, the pre-defined network policy matrix is ​​an N×N two-dimensional matrix (N is the total number of running nodes in the distributed system). The matrix element (x, y) is used to define "whether running node x is allowed to initiate a service request to running node y". The core rule is:

[0210] An element value of "Allow" (or 1) indicates that node x can initiate cross-node business calls to node y.

[0211] An element with a value of "prohibited" (or 0) indicates that node x cannot initiate cross-node business calls to node y;

[0212] An element with a value of "default" (or null) indicates that it is not explicitly configured and is handled according to the system's global policy. It is usually "disabled" by default to ensure security.

[0213] Furthermore, for the obtained node call relationship (x, y), query the corresponding element (x, y) in the network policy matrix:

[0214] If the element value is "prohibited", it means that the actual node call of the current upstream and downstream log pair violated the preset network policy, which is judged as a network topology violation fault, and key information is recorded.

[0215] If the element value is "Allowed": the call is valid and there are no errors;

[0216] If the element value is "default": it will be handled according to the global disallow policy and judged as a fault.

[0217] Match the service name and first timestamp of the target log with the spatiotemporal mapping table, specifically:

[0218] Using the service name and first timestamp of the target log as the query key, search the spatiotemporal mapping table for records with the same service name and whose first timestamp falls within the survival time interval;

[0219] If a unique record is found, the match is successful; if multiple records are found, the record with the most recent time is selected.

[0220] If no record is found, the match fails, and a context missing exception event is generated and logged.

[0221] Based on the context-deficient exception event, it can be directly determined that there is a log loss failure caused by the instantaneous destruction of the container or the loss of events.

[0222] Specifically, in a normally operating distributed system, when a legitimate business log is generated, the container to which it belongs must be in a live state, and the spatiotemporal mapping table should have a corresponding live record. If there is no matching record, it means that the physical context association link is broken, which falls under the category of log disconnection. Therefore, if the context missing exception event corresponding to the target log is a valid event, that is, the service name is valid and the timestamp format is correct, then it is directly determined that a log disconnection failure has occurred, and the key information is recorded.

[0223] Generating diagnostic reports includes credibility metrics, specifically:

[0224] Assign an initial credibility score to the enhanced business trajectory;

[0225] Based on the number of detected resource contention failures, instance skipping failures, network topology violation failures, and log disconnection failures, the initial credibility score is deducted according to the preset first deduction weight, second deduction weight, third deduction weight, and fourth deduction weight, respectively.

[0226] The final credibility score obtained after deduction is used as the reliability measure of the business trajectory and written into the diagnostic report.

[0227] Specifically, the steps for credibility metric measurement are as follows:

[0228] Step 1: Record the core identifiers of the enhanced business trajectory, which are used to locate the business scenario corresponding to the trajectory, including Trace ID (unique identifier across the entire link), trajectory start time and end time, list of services involved, list of running nodes involved, list of containers involved, and total number of original logs or total number of enhanced logs;

[0229] Summarize the various fault information associated with this enhanced service trajectory. Required fields include the number of resource contention faults, the number of instance skipping faults, the number of network topology violation faults, the number of log disconnection faults, and specific details of each type of fault.

[0230] Record the complete calculation chain of the credibility score. Required fields include the initial credibility score, the deduction weight for each type of failure, the deduction score for each type of failure, and the final credibility score.

[0231] The track reliability is determined by the final credibility score, and the classification rules can be customized.

[0232] Based on reliability levels and fault statistics, targeted optimization suggestions are given, such as prioritizing the repair of log disconnection issues when trajectory reliability is low, and optimizing node resource configuration when there are many resource contention faults.

[0233] Step 2: Set the initial reliability score using a 100-point scale, with a default initial score of 100. The basis for this setting is: during the construction of the enhanced business trajectory, if no faults are detected, it indicates that the trajectory data is complete, the business execution link is normal, the physical context is accurately associated, and it possesses the highest reliability. A score of 100 can intuitively reflect the optimal reliability under fault-free conditions.

[0234] Differentiated weights are assigned based on the severity of the fault, and deductions are made cumulatively based on the number of faults.

[0235] The deduction for a certain type of fault = the number of faults of that type × the corresponding deduction weight;

[0236] The deduction weights need to be set in conjunction with the degree of impact of the fault on the reliability of the trajectory. The greater the impact, the higher the weight. The weight configuration and basis are as follows:

[0237] First deduction weight: Resource contention failure weight 10 points / failure. Since resource contention failure affects business performance but does not destroy the integrity and correlation of trajectory data, it has little interference with the diagnostic results, so it has the lowest weight.

[0238] The second deduction weight: Instance jump failure weight is 15 points / instance. Since instance jump failure will cause the interruption of business execution continuity, it may cause some context information in the trajectory to be broken, which will interfere with the accuracy of fault diagnosis. The weight is higher than that of resource contention failure.

[0239] The third deduction weight: Network topology violation faults have a weight of 20 points per fault. Since network topology violation faults involve security risks and may lead to the leakage of business link data, as well as disrupt the consistency of business calls and network policies, they have a significant impact on the confidence of diagnostic results and have a higher weight than instance skip faults.

[0240] Fourth deduction weight: Log disconnection failure weight 25 points / failure. Since log disconnection failure directly leads to incomplete trajectory data and broken physical context association, it will cause blind spots in fault diagnosis. It is the failure with the greatest impact on trajectory reliability, so it has the highest weight.

[0241] Since the final score must be non-negative, a score of 0 indicates that the trajectory data is seriously unreliable and the fault diagnosis results based on it are of no reference value. Therefore, the cumulative deduction of points shall not exceed the initial credibility score, that is, the final credibility score shall be at least 0 points. If the sum of the deductions for various faults is greater than or equal to the initial score, the final score shall be counted as 0 points.

[0242] The final credibility score is directly used as the reliability metric for enhanced business trajectories. It needs to be combined with score ranges to classify reliability levels, facilitating quick and intuitive judgment by operations and maintenance personnel. The level classification is as follows:

[0243] Grade A (High Reliability): Final score ≥ 85 points. Explanation: The trajectory data is complete, with only a few minor faults. Fault diagnosis results based on this trajectory have extremely high confidence and can be directly used as a basis for operation and maintenance decisions.

[0244] Level B (Relatively Reliable): 70 points ≤ Final Score < 85 points. Explanation: The trajectory data is basically complete, but there are some moderately severe faults, such as one resource contention fault and one instance jump fault. The diagnostic results are generally reliable, but require some manual verification.

[0245] Level C (Low Reliability): 50 points ≤ Final Score < 70 points. Explanation: The trajectory data is significantly missing or contains numerous faults, such as two network topology violations. The diagnostic results have low confidence and require priority to repair the faults and supplement the trajectory data before re-diagnosing.

[0246] Level D (Unreliable): Final score < 50 points. Explanation: The trajectory data is severely incomplete, such as multiple log loss failures or a large number of serious failures, such as two log loss failures and one network topology violation failure. The diagnostic results are of no reference value, and the trajectory construction process and business execution environment need to be thoroughly investigated.

[0247] Step 3: After completing the reliability quantification calculation, the basic information, fault statistics, quantification process, reliability level and diagnostic suggestions should be structured and written into the report to ensure that the output format is standardized and the information is traceable.

[0248] Furthermore, by objectively measuring trajectory reliability with scores, operations and maintenance personnel can clearly understand the confidence level of fault diagnosis results. At the same time, the quantification process fully records the link from the initial score to the fault deduction to the final score, which facilitates tracing the root cause of low reliability and provides a precise basis for prioritizing the repair of critical faults and improving trajectory reliability, effectively improving the efficiency and accuracy of fault diagnosis in distributed systems.

[0249] Generating diagnostic reports also includes root cause correlation reasoning, specifically including:

[0250] Construct a fault knowledge graph, whose nodes include fault type, discrete events of each container, and running nodes deployed by each container;

[0251] The specific fault events detected and their physical context information are mapped to phenomenon nodes in the fault knowledge graph;

[0252] In the fault knowledge graph, reverse reasoning is performed along the causal relationship edges to locate the upstream root cause node that has the highest probability of causing the phenomenon node to occur.

[0253] One or more root cause node information obtained through reasoning is converted into natural language descriptions and attached to the diagnostic report.

[0254] Specifically, the steps of root cause reasoning are as follows:

[0255] Step 1: The fault knowledge graph is the foundation of root cause reasoning. It needs to be built in advance and dynamically maintained. The core is to clarify the node type, define the node attributes, and establish causal relationship edges to ensure that it can cover the causal relationships of various faults in the distributed container environment.

[0256] Node Type 1: Fault Type Node, used to characterize the diagnosed fault results. Attributes include fault type, fault name, fault severity, scope of impact (service / node / container), and typical causes.

[0257] Node Type 2: Container Discrete Event Node, used to characterize various events (including normal / abnormal events) throughout the container's entire lifecycle. It is the direct cause of a failure or an intermediate transmission link. Attributes include event, event name, event type (normal or abnormal), associated container, occurrence time, and event details.

[0258] Node type 3: The node that runs the node, which represents the physical or virtual node of the container deployment. It is the carrier of faults. Attributes include node, node name, node IP, resource configuration, list of deployed containers, and node status.

[0259] Edges in a knowledge graph are used to represent causal relationships between nodes. Each edge carries a causal direction and a confidence level attribute. The confidence level ranges from 0 to 1, representing the strength of the causal relationship. Core relationship types and examples are as follows:

[0260] Container Discrete Events - Fault Types: Container abnormal events directly trigger a specific type of fault. Example:

[0261] Insufficient container memory caused eviction, resulting in an instance skipping failure (confidence level = 0.92).

[0262] Container CPU intensive usage → resource contention failure (confidence level = 0.88).

[0263] Furthermore, by accessing real-time fault diagnosis data, when new fault causes are associated, they are added to the graph after confirmation by operations and maintenance personnel. The confidence of the relationship edges is calibrated regularly based on new fault cases to improve the accuracy of inference.

[0264] Step 2: Retrieve specific fault events from the fault statistics module of the diagnostic report. These events must include complete physical context information, such as fault type, involved services, containers and nodes, occurrence time, and fault details.

[0265] The mapping rules are as follows:

[0266] Combined mapping: If the actual failure event is a superposition of multiple types of failures, such as resource contention and instance jump, then it is mapped to multiple phenomenon nodes, and the root cause is then jointly inferred.

[0267] Step 3: Based on the causal relationships in the knowledge graph, trace back from the phenomenon node to identify the upstream root cause node with the highest probability of causing the phenomenon node to appear, ensuring the accuracy of root cause localization. Specifically:

[0268] Directional rule: Tracing only along the reverse of the causal relationship edge, that is, from the effect to the cause, such as from the phenomenon node to the upstream container discrete event node, and then to the upstream running node node. Forward traversal is not allowed.

[0269] Termination rule: Termination occurs when tracing back to a node with no upstream causal node, i.e., the root cause node, such as when the running node has insufficient resource configuration or hardware failure.

[0270] Starting with the phenomenon node, perform the reasoning as follows:

[0271] Find all causal edges in the knowledge graph that have the phenomenon node as the target node, i.e., the upstream nodes that directly cause the phenomenon;

[0272] For each upstream node, continue tracing the causal relationship edges that serve as the target node until the termination rule is met;

[0273] Calculate the probability of each tracing path and summarize the final impact probability of each root cause node;

[0274] Nodes are sorted in descending order of their final impact probability, and those with a probability greater than or equal to a preset threshold are selected as core root cause nodes. The preset threshold is usually 0.7.

[0275] Step 4: Transform the root cause information obtained from the reasoning into natural language that is easy for operations and maintenance personnel to understand, and attach it to the diagnostic report in a structured manner to ensure the practicality of the report.

[0276] Furthermore, root cause correlation reasoning, through causal modeling and probabilistic tracing of knowledge graphs, breaks through the limitations of traditional fault diagnosis that focuses on phenomena while neglecting root causes. It upgrades the diagnostic report from a list of faults to a root cause-oriented approach, allowing maintenance personnel to quickly address issues based on the root cause descriptions and verification suggestions in the report, avoiding blind troubleshooting and significantly shortening fault repair time. At the same time, the reasoning data feeds back into the knowledge graph iteration, forming a closed loop of diagnosis, reasoning, and optimization, effectively improving the intelligence level of distributed system fault diagnosis.

[0277] The orchestration event stream is obtained in real time by listening to the event interface of the container orchestration platform. Discrete events include at least container creation events, scheduling events, running events, and termination events, and each event record contains a corresponding event timestamp.

[0278] The spacetime mapping table is continuously and dynamically updated, specifically as follows:

[0279] When a container creation event is detected, extract the corresponding service name, container, running node, and event timestamp, insert a new record in the time-space mapping table, and set the corresponding event timestamp to the container startup time.

[0280] When a termination event of the same container is detected, the corresponding event timestamp is extracted, and the container termination time of the corresponding record is updated.

[0281] It also includes the maintenance steps for the spacetime mapping table:

[0282] Set an expiration time threshold for historical data;

[0283] Periodically scan each record in the spatiotemporal mapping table and calculate the time difference between the current moment and the event timestamp of the container's termination event;

[0284] If the time difference is greater than the expiration time threshold, the record is deleted from the spatiotemporal mapping table.

[0285] Specifically, if the spatiotemporal mapping table retains all container records for an extended period, it will lead to a surge in table data volume, thereby reducing query efficiency and increasing storage resource consumption. Therefore, a periodic maintenance mechanism needs to be established, with the core objective of cleaning up expired historical data to ensure lightweight table operation. Details are as follows:

[0286] The expiration time threshold is the core criterion for determining whether historical records need to be deleted. It needs to be set scientifically in combination with factors such as business needs, storage resources, and log backtracking cycles. Specific requirements are as follows:

[0287] After a container terminates, the maximum retention period for its corresponding mapping table record is determined to be expired when the timestamp from the current moment to the container termination event is greater than the expiration threshold.

[0288] In general scenarios, the log backtracking requirements of most distributed systems typically do not exceed 30 days, and the amount of historical data for 30 days can meet the backtracking requirements for fault diagnosis without excessively consuming storage resources.

[0289] In core business scenarios, the period can be extended to 90 days. Core businesses, such as financial transactions and payment services, require a longer log traceability period to meet compliance audit requirements and need to reserve sufficient historical data support.

[0290] In non-core business scenarios, the time can be shortened to 7-15 days. Non-core businesses, such as back-end management and data statistics, have low requirements for log backtracking, and shortening the threshold can significantly reduce storage pressure.

[0291] Furthermore, a periodic scan is performed through a background scheduled task to traverse the records in the table, filter expired data, prioritize the records of terminated containers by filtering conditions, skip live container records, avoid invalid scans, and quickly locate all terminated container records by using the key field of the termination event timestamp in the spatiotemporal mapping table, reducing the performance consumption of full table scans. For each filtered terminated container record, a time difference calculation is performed: the time difference equals the current time minus the container termination event timestamp.

[0292] Furthermore, if the time difference of a record is greater than the preset expiration threshold, it is determined to be an expired record; if the time difference is less than or equal to the preset expiration threshold, the record is retained.

[0293] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A log intelligent analysis and fault diagnosis system for distributed software, characterized in that, include: Data acquisition module: used to acquire business log streams and orchestration event streams. Each log in the business log stream includes a service name and a first timestamp. The orchestration event stream includes discrete events of each container under the corresponding service name and the running nodes deployed by each container. Data processing module: used to associate the creation and termination events in the discrete events of each container, calculate the survival time interval of the container under the corresponding service name, and associate the corresponding service name, container and running node with the survival time interval to form a spatiotemporal mapping table; Take any log in the business log stream as the target log, match the service name and first timestamp of the target log with the spatiotemporal mapping table, and extract the container and running node that successfully match. The extracted containers and running nodes are used as physical contexts and merged with the target logs to form context-enhanced logs; The business log stream is traversed for matching, and all generated context enhancement logs are sorted and aggregated according to the first timestamp to form an enhanced business trajectory; Data Judgment Module: Used to perform the following judgments in enhanced business trajectories: Extract all context-enhanced logs from nodes that are running within a preset time window; The extracted context enhancement logs are grouped by service name to obtain a set of service names running simultaneously on the running node, and a sequence of performance indicators representing resource utilization or response latency is extracted from the context enhancement logs of the corresponding service name. Calculate the Pearson correlation coefficient of the performance index sequence corresponding to different service names within a preset time window; If there is at least one pair of Pearson correlation coefficients greater than a preset threshold, and the corresponding performance index sequence shows an upward trend within a preset time window, then a resource contention failure is determined to have occurred on the running node. Data output module: Used to generate diagnostic reports based on the judgment results.

2. A method for intelligent log analysis and fault diagnosis in distributed software, characterized in that, The application in the log intelligent analysis and fault diagnosis system for distributed software as described in claim 1 includes the following steps: Acquire business log streams and orchestration event streams. Each log entry in the business log stream includes a service name and a first timestamp. The orchestration event stream includes multiple containers under the corresponding service name, discrete events of each container, and running nodes deployed by each container. The creation and termination events of each container are associated with discrete events. The survival time interval of the container under the corresponding service name is calculated, and the corresponding service name, container and running node are associated with the survival time interval to form a spatiotemporal mapping table. Take any log in the business log stream as the target log, match the service name and first timestamp of the target log with the spatiotemporal mapping table, and extract the container and running node that successfully match. The extracted containers and running nodes are used as physical contexts and merged with the target logs to form context-enhanced logs; The business log stream is traversed for matching, and all generated context enhancement logs are sorted and aggregated according to the first timestamp to form an enhanced business trajectory; In the enhanced business trajectory, the following judgments are made: Extract all context-enhanced logs from nodes that are running within a preset time window; The extracted context enhancement logs are grouped by service name to obtain a set of service names running simultaneously on the running node, and a sequence of performance indicators representing resource utilization or response latency is extracted from the context enhancement logs of the corresponding service name. Calculate the Pearson correlation coefficient of the performance index sequence corresponding to different service names within a preset time window; If there is at least one pair of Pearson correlation coefficients greater than a preset threshold, and the corresponding performance index sequence shows an upward trend within a preset time window, then a resource contention failure is determined to have occurred on the running node. A diagnostic report is generated based on the assessment results.

3. The intelligent log analysis and fault diagnosis method for distributed software according to claim 2, characterized in that: The determination of the enhanced business trajectory also includes: In the enhanced business trajectory, select two context enhancement logs that belong to the same service name and have consecutive first timestamps; Obtain the containers and corresponding first timestamps of the two context-enhanced logs respectively; Calculate the time difference between two first timestamps; If two context-enhanced logs are in different containers and the time difference is less than the preset minimum stable operation threshold, then an instance skipping failure is determined to have occurred.

4. The intelligent log analysis and fault diagnosis method for distributed software according to claim 3, characterized in that: The determination of the enhanced business trajectory also includes: Traverse the enhanced business trajectory, and for each pair of context enhancement logs with a direct business call sequence, extract the running nodes of the two context enhancement logs respectively; A network policy matrix is ​​pre-defined, where the element (x, y) indicates whether running node x is allowed to send a service request to running node y; If both running nodes are prohibited when querying the network policy matrix, a network topology violation fault event is generated for the service call represented by the context enhancement log.

5. The intelligent log analysis and fault diagnosis method for distributed software according to claim 2, characterized in that: Match the service name and first timestamp of the target log with the spatiotemporal mapping table, specifically: Using the service name and first timestamp of the target log as the query key, search the spatiotemporal mapping table for records with the same service name and whose first timestamp falls within the survival time interval; If a unique record is found, the match is successful; if multiple records are found, the record with the most recent time is selected. If no record is found, the match fails, and a context missing exception event is generated and logged. Based on the context-deficient exception event, it can be directly determined that there is a log loss failure caused by the instantaneous destruction of the container or the loss of events.

6. The intelligent log analysis and fault diagnosis method for distributed software according to claim 5, characterized in that: Generating diagnostic reports includes credibility metrics, specifically: Assign an initial credibility score to the enhanced business trajectory; Based on the number of detected resource contention failures, instance skipping failures, network topology violation failures, and log disconnection failures, the initial credibility score is deducted according to the preset first deduction weight, second deduction weight, third deduction weight, and fourth deduction weight, respectively. The final credibility score obtained after deduction is used as the reliability measure of the business trajectory and written into the diagnostic report.

7. The intelligent log analysis and fault diagnosis method for distributed software according to claim 6, characterized in that: Generating diagnostic reports also includes root cause correlation reasoning, specifically including: Construct a fault knowledge graph, whose nodes include fault type, discrete events of each container, and running nodes deployed by each container; The specific fault events detected and their physical context information are mapped to phenomenon nodes in the fault knowledge graph; In the fault knowledge graph, reverse reasoning is performed along the causal relationship edges to locate the upstream root cause node that has the highest probability of causing the phenomenon node to occur. One or more root cause node information obtained through reasoning is converted into natural language descriptions and attached to the diagnostic report.

8. The intelligent log analysis and fault diagnosis method for distributed software according to claim 2, characterized in that: The orchestration event stream is obtained in real time by listening to the event interface of the container orchestration platform. The discrete events include at least container creation events, scheduling events, running events, and termination events, and each event record contains a corresponding event timestamp.

9. The intelligent log analysis and fault diagnosis method for distributed software according to claim 8, characterized in that: The spatiotemporal mapping table is continuously and dynamically updated, specifically as follows: When a container creation event is detected, extract the corresponding service name, container, running node, and event timestamp, insert a new record in the time-space mapping table, and set the corresponding event timestamp to the container startup time. When a termination event of the same container is detected, the corresponding event timestamp is extracted, and the container termination time of the corresponding record is updated.

10. The intelligent log analysis and fault diagnosis method for distributed software according to claim 9, characterized in that: It also includes the maintenance steps for the spacetime mapping table: Set an expiration time threshold for historical data; Periodically scan each record in the spatiotemporal mapping table and calculate the time difference between the current moment and the event timestamp of the container's termination event; If the time difference is greater than the expiration time threshold, the record is deleted from the spatiotemporal mapping table.

Images