AIOps anomaly detection and root cause localization method

Abstract

The application discloses an AIOps anomaly detection and root cause positioning method and relates to the technical field of anomaly detection. The method comprises the following steps: step 1: in a unified timeline, preset time windows are processed according to services and instances, monitoring class anomaly segments are generated for monitoring indexes, and client and server spans in distributed tracking are extracted for segment pairing; step 2: stitching is performed according to the distributed tracking to obtain a candidate evidence chain set, and the last segment of each evidence chain unit is used as a candidate root cause pointing; step 3: a root cause list is output according to a certainty rule for the candidate evidence chain set, and relevant template text and the time range of adjacent monitoring class anomaly segments are given. The application can accurately identify an abnormal propagation path in multi-source heterogeneous data, verify the correlation between upstream manifestations and downstream resource failures, and provide an explicit root cause target and evidence interpretation.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology, and more particularly to an AIOps anomaly detection and root cause localization method. Background Technology

[0002] With the widespread adoption of cloud computing, large-scale distributed systems, and microservice architectures, application systems and infrastructure generate a large amount of monitoring metrics, operational logs, and distributed tracing data during operation. This data contains critical information such as system status, request paths, performance metrics, and anomalies, and is widely used for operation and maintenance monitoring and troubleshooting. Currently, the industry typically relies on various automated monitoring and analysis methods to process this data, attempting to automate anomaly detection and root cause analysis. However, existing methods still have significant shortcomings and struggle to meet the demands for rapid and accurate fault diagnosis in complex business scenarios.

[0003] In existing technologies, anomaly detection of monitoring metrics often employs threshold alarms or statistical methods. For example, many systems configure monitoring metrics using static thresholds, triggering alarms when values ​​exceed preset ranges. While simple to implement, this approach is prone to false alarms and missed alarms in dynamic and complex environments, particularly under conditions of drastic load fluctuations or multi-tenant scenarios. To improve this, some technologies introduce time series prediction models to predict future trends in metrics and identify anomalies based on residuals. However, these methods still focus on anomaly detection for single metric sequences, lacking holistic correlation analysis across metrics and instances, making it difficult to reveal system-level fault propagation chains. Log analysis techniques in existing publicly available methods mainly focus on pattern matching and keyword recognition. On the one hand, existing log parsing methods often rely on regular expressions or fixed templates for classification, but system upgrades or application iterations frequently change log formats, making static rule-based methods costly to maintain and difficult to maintain stable accuracy when faced with diverse logs. On the other hand, although existing studies have proposed using word vectors or deep learning models for log semantic representation to improve automated recognition capabilities, these methods often fail to effectively merge in environments where multi-source log data coexist, making it difficult to form unified abnormal fragments and increasing the complexity of localization. Summary of the Invention

[0004] In view of this, the present invention provides an AIOps anomaly detection and root cause localization method. By collaboratively processing monitoring metrics, operational logs, and distributed tracing data under a unified timeline, it first generates and aligns monitoring-related and log-related anomaly fragments. Then, it generates call fragments guided by client-side and server-side spans, and forms evidence chain units through anchor point type labeling and time continuity constraints. Finally, it performs conflict pruning in the candidate evidence chain set and outputs a root cause list. This method can accurately identify anomaly propagation paths in multi-source heterogeneous data, correlate upstream manifestations with downstream resource failures for verification, provide clear root cause targets and evidentiary explanations, effectively reduce false positives and false negatives, improve the accuracy and certainty of anomaly localization, significantly shorten diagnosis time, and enhance the automation and intelligence level of complex distributed system operation and maintenance.

[0005] The technical solution adopted in this invention is as follows:

[0006] AIOps anomaly detection and root cause localization method, which includes:

[0007] Step 1: Under a unified timeline, process preset time windows by service and instance, generate monitoring-type anomaly fragments for monitoring metrics, generate log-type anomaly fragments by merging runtime logs with templates and identifying trigger words, and extract the client and server spans from distributed tracing for fragment pairing.

[0008] Step 2: Perform stitching based on distributed tracing. Pair client spans and server spans with the same tracing identifier to generate call fragments, and record the source service, source instance, destination service, destination instance, and occurrence time. Search for monitoring-type exception fragments that overlap with or are adjacent to each other for no more than one second on the source and destination sides, as well as log-type exception fragments containing phrases such as "reject new connection," "insufficient threads," "queue full," and "disk unwritable." Mark the anchor type for categorizable templates. When a call fragment contains a log-type exception fragment containing downstream failure or timeout trigger words on the source side and has a capacity anchor or storage anchor on the destination side, it is recorded as a stitchable call fragment. Starting from any stitchable call fragment, generate evidence chain units by continuing according to the destination instance and time continuity. Prune evidence chain units that simultaneously cover the same source and destination services and have opposite anchor semantics according to the conflict pruning rules to obtain a candidate evidence chain set, and use the last fragment of each evidence chain unit as the candidate root cause pointer.

[0009] Step 3: Output a root cause list for the candidate evidence chain set according to deterministic rules. If there is a capacity anchor or storage anchor at the end of the evidence chain, then take its purpose service and purpose instance as the root cause target, and give the relevant template text and the time range of adjacent monitoring anomaly fragments.

[0010] Furthermore, in step 1, within the instance identifier set under the same second and the same service identifier, the monitoring values ​​of all instances are sorted by size, and the median is taken as the reference for the same group. When the value of an instance is continuously greater than twice the reference for the same group or continuously less than half the reference for the same group within three consecutive seconds, these three seconds are merged into a monitoring-type abnormal segment. If the condition is still met later, the end second of the segment is extended.

[0011] Furthermore, in step 1, the runtime log data is template-merged, replacing numeric strings, hexadecimal strings, network addresses, file paths, and unique identifiers with placeholders. A fixed-order text is used as the template. When the same instance identifier appears at least ten times within any consecutive ten-second interval, or when the template text contains any of the trigger words for failure, timeout, rejection, unreachable, exhausted, full, insufficient permissions, rollback, or crash, a log-type exception fragment is generated using the first and last occurrence seconds. Span records within the current time window are retained from the distributed tracing data, saving the tracing identifier, span identifier, parent span identifier, service identifier, instance identifier, span type, start second, end second, status field, and destination network address field. When monitoring-type exception fragments and log-type exception fragments have the same service identifier and instance identifier and overlap in time or are adjacent for no more than two seconds, a fragment pairing relationship is established and recorded in the exception fragment set.

[0012] Furthermore, in step 2, the process of generating call fragments includes: filtering records with span types of client and server from the distributed tracing data; generating a call fragment for pairs with the same tracing identifier and where the parent span identifier of the server record is equal to the span identifier of the client record; the call fragment includes the source service identifier, source instance identifier, destination service identifier, destination instance identifier, start second, and end second; generating a call fragment when the client record and the server record do not have the above pairing but their tracing identifiers are the same, the difference in start second is no more than one second, and the destination network address field is consistent; deduplicating multiple call fragments with the same source service identifier, source instance identifier, destination service identifier, and destination instance identifier within the same second and retaining the one with the earliest start second.

[0013] Furthermore, in step 2, anchor point fragments are generated based on template text in log-type exception fragments. Among them, "rejecting new connections," "invalid certificate," "insufficient permissions," and "expired key" are classified as access control anchor points; "insufficient threads," "queue full," "connection pool exhausted," and "file handle full" are classified as capacity anchor points; "disk not writable," "directory not existing," "device read-only," and "verification failed" are classified as storage anchor points; and "configuration has taken effect," "restart completed," and "rollback successful" are classified as change anchor points. For each call fragment, monitoring-type exception fragments and anchor point fragments that overlap with or are adjacent to the start and end seconds of the call fragment for no more than one second are searched on the source instance identifier and the destination instance identifier to form an alignment result. When there is a log-type exception fragment containing downstream call failure, timeout, or retry trigger words on the source side and there is a capacity anchor point or storage anchor point on the destination side, or there is a monitoring-type exception fragment on the destination side, the call fragment is marked as a call fragment that can be stitched together.

[0014] Furthermore, in step 2, message system-related keys are searched from the span attributes of the distributed tracking data. When the target name of the sender's span and the receiver's span are found to be the same and the difference in start seconds is no more than one second, an intermediate call fragment with the target name as the intermediate identifier is generated and the fragment is aligned with the call according to the aforementioned rules. At the same time, log-type abnormal fragments containing keywords such as "published to queue", "consumed from queue", "topic", "partition", and "offset" are searched in the running log data. The queue or topic name in the fragment is extracted as the intermediate identifier. When the intermediate identifier is the same as the intermediate call fragment and the time overlaps or is adjacent for no more than one second, the intermediate call fragment is marked as a call fragment that can be stitched together.

[0015] Furthermore, in step 2, the process of generating an evidence chain unit includes: starting from any stitchable call segment, expanding from early to late according to the occurrence time; searching for the next stitchable call segment with the same source instance identifier as the target instance identifier and whose start second is the same as or adjacent to the end second of the current call segment by no more than one second at the target instance identifier of the call segment, and continuing it; if the continued call segment has neither an anchor segment nor a monitored anomaly segment on the target side, or if the difference between its start second and the end second of the previous segment exceeds one second, then the expansion stops. The resulting continuous call segment sequence and the monitored anomaly segment and anchor segment aligned to it together constitute an evidence chain unit.

[0016] Furthermore, in step 2, when two evidence chain units cover the same source service identifier and destination service identifier in the same second but the corresponding anchor type semantics are opposite, the evidence chain unit containing the access control anchor, capacity anchor, or storage anchor is retained and the other one is deleted; when any call fragment in the evidence chain unit fails to align to the source side or destination side monitoring-type abnormal fragment or anchor fragment, the call fragment and its subsequent fragments are deleted, and the remaining part is solidified into a candidate evidence chain set; the destination instance identifier of the last fragment of each evidence chain unit and its anchor type are output as root cause candidates. If the last fragment does not have an anchor fragment but has a monitoring-type abnormal fragment, the destination instance identifier and the monitoring-type abnormal fragment are output as root cause candidates.

[0017] Furthermore, step 3 specifically includes: when the last end of the evidence chain unit has an anchor point of capacity, storage, or access control type, the destination service identifier and destination instance identifier of the last segment are used as the root cause target; when the last end of the evidence chain unit only has a change anchor point and its upstream adjacent segment contains a downstream failure or timeout trigger word on the source side, but there are no anchor segments or monitoring-type abnormal segments on the destination side, the destination service identifier and destination instance identifier of the last segment are used as the root cause target; when the evidence chain unit has no anchor segments but there are three consecutive call segments whose destination sides are all aligned to monitoring-type abnormal segments, the destination service identifier and destination instance identifier of the last call segment are used as the root cause target; for each root cause target, an evidentiary explanation is output, the explanation content includes at least the original template text of the trigger, the call segment sequence involved in the stitching listed in order of source service identifier, destination service identifier and occurrence time, and the time range of the monitoring-type abnormal segment directly adjacent to the root cause target.

[0018] By adopting the above technical solutions, this invention achieves the following beneficial effects: Firstly, by generating monitoring-type and log-type anomaly fragments and combining client-side and server-side span pairing, this invention can establish strict temporal correlations between multi-source heterogeneous data, ensuring that anomaly signals no longer exist in isolation but are effectively correlated along the complete call chain. This not only avoids the fragmentation problem of separating indicator and log analysis in existing technologies but also significantly reduces the latency caused by manual comparison. Secondly, this invention proposes a structured annotation method using anchor point type classification to identify typical phrases in logs such as "rejecting new connections," "insufficient threads," "queue full," and "disk unwritable," aligning them on the source and destination sides of the call fragments. This allows for bidirectional verification between anomaly manifestations and underlying resource or storage failures. This design effectively reduces false positives and false negatives, ensuring the accuracy of root cause identification. Thirdly, this invention introduces an evidence chain unit generation mechanism. Through constraints of target instances and temporal continuity, stitchable call fragments are sequentially connected to obtain an evidence chain reflecting the fault propagation path. Subsequently, in the candidate evidence chain set, semantically contradictory redundant chains are eliminated through conflict pruning rules, thus retaining the most explanatory propagation sequence. This approach ensures the simplicity and certainty of the final root cause analysis results, avoiding ambiguity caused by concurrent events or recovery operations. Finally, when outputting the root cause list, this invention not only provides the target service and target instance, but also simultaneously outputs the original template text and the time range of adjacent monitored anomaly segments, enabling operations personnel to intuitively understand the context and duration of the anomaly. This evidence-based interpretation enhances the verifiability and credibility of the results, thereby significantly shortening anomaly diagnosis time and improving the automation and intelligence level of system operations. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the AIOps anomaly detection and root cause localization method in an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the median anomaly detection of AIOps monitoring metrics in an embodiment of the present invention. Detailed Implementation

[0021] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0022] Any feature disclosed in this specification, unless otherwise stated, may be replaced by other equivalent or similar features. That is, unless otherwise stated, each feature is merely one example of a series of equivalent or similar features.

[0023] refer to Figure 1AIOps anomaly detection and root cause localization method, which includes:

[0024] Step 1: Under a unified timeline, process preset time windows by service and instance, generate monitoring-type anomaly fragments for monitoring metrics, generate log-type anomaly fragments by merging runtime logs using templates and identifying trigger words, and extract the client and server spans from distributed tracing for fragment pairing.

[0025] Specifically, in the implementation process, Coordinated Universal Time (UTC) is used as the unified timeline. The preset time window is set as a continuous time interval with the current time as its right boundary, and its length is determined by system configuration, for example, 60 seconds. The purpose of choosing a unified timeline is to eliminate local clock differences between different data sources, ensuring that cross-source data can be compared and combined on the same time coordinate. For each record from the monitoring system, log collection system, and distributed tracing system, the timestamp is rounded to the nearest whole second. When the reference time difference between a single record and its collection channel exceeds 3 seconds, it is marked as misaligned and excluded from this processing. Using whole-second alignment can obtain a stable time bucket without introducing formulas, facilitating subsequent sorting, counting, and interval merging in seconds. Service identifiers and instance identifiers are added to all records. When a record only contains a hostname or container identifier, the corresponding service identifier and instance identifier are obtained through a pre-established mapping table, ensuring consistent referencing of the same running entity from different sources, thus preventing subsequent fragment generation and fragment pairing from being affected by naming differences.

[0026] For each second on a unified timeline, monitoring values ​​corresponding to all instance identifiers within that second are collected under the same service identifier, sorted by size, and the median is used as the reference for the same group. Using the median instead of the average value can maintain the stability of the reference value when there are a few extreme values, thus avoiding misjudgments caused by the pull of a single abnormal instance on the reference value. If the monitoring value of an instance identifier is continuously greater than twice the reference value of the same group or continuously less than half the reference value of the same group within a 3-second interval, these 3-second time points are marked as abnormal points for that instance identifier. Setting the requirement of 3 consecutive seconds can filter out instantaneous fluctuations and ensure that only instances that continuously deviate from the behavior of the same group are marked. Adjacent abnormal points of the same instance identifier are merged into intervals to generate monitoring-type abnormal segments with clear start and end seconds; when subsequent seconds still meet the aforementioned conditions, the end second of the monitoring-type abnormal segment is extended; when the interval between two adjacent abnormal intervals does not exceed 2 seconds, they are merged into a longer monitoring-type abnormal segment. Through merging, the quantitative length of the abnormality persistence is preserved, while avoiding the fragmentation of the same round of abnormalities into multiple short segments. If all instance identifiers increase or decrease simultaneously within the same second, they are not immediately identified as multiple monitoring-related anomalies. Instead, only the change in the direction of the median is retained for cross-validation with log-related anomalies and span information in step 2. This approach avoids misinterpreting global fluctuations as anomalies from multiple independent instances.

[0027] For each log record, while maintaining the word order, replace the numeric strings, hexadecimal strings, network addresses, file paths, and unique identifiers with uniform placeholders to obtain stable text. Maintaining the word order ensures that the generated templates are stable across versions under the same semantics, and replacing variable content can eliminate noise related to specific environments. The stable text is used as the template key for merging to obtain the correspondence between templates and their occurrence times. For the same instance identifier, if the same template appears 10 times within any consecutive 10-second interval, a log-type exception segment is generated, starting with the second of the first occurrence and ending with the second of the last occurrence. High-frequency repetition of the same template usually corresponds to loops of retries, blocking, or fast failures; aggregating this behavior into log-type exception segments provides a time range comparable to monitoring-type exception segments. For log records containing any of the trigger words such as failure, timeout, rejection, unreachable, exhaustion, full, insufficient permissions, rollback, or crash, they are aggregated within the current preset time window based on the instance identifier. If the template corresponding to the same trigger word appears 5 times between its first and last occurrences within 10 seconds, a log-type exception fragment is directly generated, defined by the first and last occurrence seconds. Introducing a trigger word threshold ensures that only when a certain type of exception semantics occurs consistently is it promoted to a fragment, avoiding meaningless fragments triggered by single, occasional log occurrences. When the same instance identifier generates log-type exception fragments both due to high-frequency template triggering and due to trigger word thresholding within the same second range, the intervals are merged, retaining the fragment with the earlier start second and the later end second, ensuring that only one log-type exception fragment appears at the same time period, facilitating subsequent one-to-one or one-to-many association with monitoring exception fragments.

[0028] Within a unified timeline and a preset time window, distributed tracing data is read, retaining the tracing identifier, span identifier, parent span identifier, service identifier, instance identifier, span type, start second, end second, status field, and destination network address field. These fields are retained because subsequent fragment pairing and fragment generation rely entirely on this verifiable factual information, without requiring prior system topology. Based on span type, the initiator is marked as a client span, and the callee as a server span; when some tracing systems do not explicitly specify the type, it is determined based on whether outbound and inbound network attributes are included. Retaining this record only when the type determination is clear avoids introducing uncertainty during subsequent pairing. Time indexes are created for client and server spans based on the start second, and a fast lookup index is created based on the tracing identifier, providing constant-level access capability for pairing in step 2. Limiting index building to the current preset time window limits memory consumption and keeps processing latency at the second level.

[0029] For the same service identifier and the same instance identifier, under a unified timeline, if the time intervals of monitoring-type anomaly segments and log-type anomaly segments overlap or their boundaries are adjacent for no more than 2 seconds, a segment pairing relationship is recorded. This pairing relationship is only registered in step 1 and is not propagated or used for causal judgment, thus avoiding overstepping the responsibilities of step 2. If a monitoring-type anomaly segment and multiple log-type anomaly segments meet the alignment conditions, all pairing relationships are retained; similarly, if multiple monitoring-type anomaly segments and one log-type anomaly segment meet the alignment conditions, all pairing relationships are also retained. Completely retaining these pairing relationships allows for more precise evidence stitching in step 2 without losing potential root cause clues.

[0030] If, within a given second, only a single instance of the same service identifier has a monitored value, and no reference group can be constructed, no monitoring-related anomaly fragment is generated for that second. Evidence is supplemented in step 2 using log-related anomaly fragments and span information. This avoids misleading references when samples are insufficient. For duplicate or out-of-order logs, deduplication is performed based on timestamps, retaining only the earliest received record within the same second. For out-of-order tracking records, if their timestamps fall outside the current preset time window, processing is delayed until a later time window to avoid disrupting the time consistency of the current time window. Log text and tracking tags are uniformly encoded to ensure consistent template merging and trigger word recognition performance across language scenarios.

[0031] The granularity of time alignment can be set to 500 milliseconds. In this optional implementation, all conditions involving seconds are executed at a granularity of 500 milliseconds; for example, six consecutive 500-millisecond time points replace three consecutive seconds. This improves sensitivity to brief fluctuations and is suitable for low-latency service scenarios. A truncated average can be used for the same group reference. In this optional implementation, the top 10 percentile and bottom 10 percentile data after sorting are removed, and the remaining values ​​are averaged as the same group reference. This method can further reduce the impact of extreme values ​​when the number of instances is large, making it suitable for large-scale clusters. The set of trigger words can be expanded according to the scenario. In storage-intensive scenarios, words such as insufficient space and quota exceeding limits can be added; in network-sensitive scenarios, words such as handshake failure and certificate expiration can be added. When expanding, the occurrence threshold should still be retained to ensure that the text promoted to log-type exception fragments has stable repetition. The generation of log-type exception fragments can introduce an adjacent interval merging threshold. In this optional implementation, if the interval between two log-type exception fragments does not exceed 2 seconds and their templates are identical, they are merged into a longer log-type exception fragment to establish a more stable one-to-one correspondence with the monitoring-type exception fragments. The determination of client-side span and server-side span can be achieved through the network direction field. When the span type is defaulted but outbound or inbound direction is recorded, the outbound direction is used to determine the client-side span, and the inbound direction is used to determine the server-side span, reducing reliance on specific field naming in the tracking system.

[0032] Step 2: Perform stitching based on distributed tracing. Pair client-side and server-side spans with the same tracing identifier to generate call fragments, and record the source service, source instance, destination service, destination instance, and occurrence time. Search for monitoring-type exception fragments that overlap with or are adjacent to each other for no more than one second on both the source and destination sides, as well as log-type exception fragments containing phrases such as "reject new connection," "insufficient threads," "queue full," and "disk unwritable." Mark the anchor type for categorizable templates. When a call fragment contains a log-type exception fragment containing downstream failure or timeout trigger words on the source side and a capacity anchor or storage anchor on the destination side, it is recorded as a stitchable call fragment. Starting from any stitchable call fragment, generate evidence chain units by continuing according to the destination instance and time continuity. Prune evidence chain units that simultaneously cover the same source and destination services and have opposite anchor semantics according to the conflict pruning rules to obtain a candidate evidence chain set, and use the last fragment of each evidence chain unit as the candidate root cause pointer.

[0033] In the specific implementation, distributed tracing data is read within a preset time window, retaining only the client-side and server-side spans, as well as the fields of tracing identifier, span identifier, parent span identifier, service identifier, instance identifier, start second, end second, and destination network address. Client-side and server-side span indexes are created separately based on the tracing identifier, while a time index is created based on the start second. The tracing identifier index ensures pairing within the same call context, while the time index is used to quickly locate matching records within the same second or adjacent seconds. The reason for using two types of indexes is that distributed systems may exhibit both strict parent-child relationships within the same tracing identifier and slight time drift caused by network jitter.

[0034] For each client span, the server span is searched under the same tracing identifier. If the parent span identifier of the server span is equal to the span identifier of the client span, they are considered to belong to the same call, and a call fragment is generated. The call fragment includes the source service identifier, source instance identifier, destination service identifier, destination instance identifier, start second, end second, and occurrence time. The occurrence time uses the start second of the client span. The start second of the client span is chosen as the occurrence time because the call is initiated actively by the source side, and this time point best reflects the causal starting point and is not affected by queuing or processing delays on the destination side. When the main rule is not hit, the server span and client span are searched under the same tracing identifier. If the destination network address field is consistent and the difference between their start seconds does not exceed 1 second, they are considered to belong to the same call, and a call fragment is generated. The reason for adding a fallback pair is that some tracing implementations do not record parent and child identifiers or the parent and child fields are missing when cross-process or cross-language transmission occurs. A call can still be reliably reconstructed through network address and adjacent time constraints. For multiple call fragments within the same second where the source service identifier, source instance identifier, destination service identifier, and destination instance identifier are completely identical, only the one with the earliest start second is retained to avoid misinterpreting duplicate reports of the same request as multiple calls. Exception records with a start second later than the end second are discarded to ensure semantic consistency within the interval.

[0035] For each call segment, under the source instance identifier, search for monitoring-related and logging-related anomaly segments whose time intervals overlap or are adjacent for no more than 1 second with the call segment. The reason for using overlap or adjacent intervals of no more than 1 second is that there are sampling boundaries and output buffer delays in monitoring sampling and log writing; allowing a minimal time slack can improve the recall rate of true correlations while controlling the risk of mismatches. In the same way, under the destination instance identifier, search for monitoring-related and logging-related anomaly segments whose time intervals overlap or are adjacent for no more than 1 second with the call segment. Simultaneous alignment on the source and destination sides can distinguish between initiator anomalies and callee anomalies, avoiding unilateral judgments based on evidence from only one side. For each call segment, record the time ranges of monitoring-related and logging-related anomaly segments matched on the source side, and the time ranges of monitoring-related and logging-related anomaly segments matched on the destination side, ensuring that subsequent judgments are traceable.

[0036] Phrase matching is performed on the template text within log-type exception fragments. If any of the phrases "reject new connection," "insufficient threads," "queue full," or "disk not writable" are included, the template is marked as a categorizable template. "Reject new connection" is mapped to an access control anchor, "insufficient threads" and "queue full" to capacity anchors, and "disk not writable" to a storage anchor. This mapping is used because these three anchor types point to three stable and verifiable failure mechanisms: restricted access or connection, exhaustion of computational or concurrent resources, and persistent storage write failure, respectively, providing a clear explanation in cross-source evidence. Anchor type annotation applies to specific templates, not the entire log-type exception fragment. When the same log-type exception fragment contains multiple categorizable templates, each is annotated individually, and its occurrence time range is preserved to facilitate precise location of the trigger point when aligned with the time of the calling fragment.

[0037] If the source-side log-type exception fragment contains a template with a downstream call failure or timeout trigger word, it indicates that the source side has detected a downstream call exception. If the destination-side log-type exception fragment contains a capacity anchor or storage anchor, or if the destination side has a monitoring-type exception fragment that overlaps with or is adjacent to the call fragment by no more than 1 second, it indicates that the downstream instance has experienced objective exceptions such as insufficient resources or write failure in the same time neighborhood. When both the source-side trigger condition and the destination-side evidence condition are met, the call fragment is marked as a stitchable call fragment. The reason for using bilateral constraints is that the source-side trigger word can indicate the appearance of call failure, and the destination-side evidence can provide the objective state of the callee; combining the two can significantly reduce false alarms. For each stitchable call fragment, the template text corresponding to the source-side trigger word, the destination-side anchor point type, or the time range of the monitoring-type exception fragment is recorded and saved together with the occurrence time of the call fragment, providing complete evidence for the generation of subsequent evidence chain units and conflict trimming.

[0038] If only a source-side trigger word exists but the destination-side has no evidence, it will not be marked as a stitchable call fragment and will be retained as a normal call fragment awaiting supplementary evidence in a subsequent time window. This avoids misjudging an upstream failure as a downstream root cause. If both capacity and storage anchors appear on the destination side, both will be recorded, and the anchor type whose time is closer to the call fragment's occurrence time will be prioritized when generating subsequent evidence chain units to improve time consistency. If the same call fragment matches multiple templates containing downstream failure or timeout trigger words on the source side, the time corresponding to the first appearing template will be selected as the source-side trigger time. This is because the first trigger is closer to the anomaly's origin, reducing the duplication effect caused by retries.

[0039] The occurrence time can be determined by using the earlier of the client-side span start second and the server-side span start second. In scenarios with highly unstable network latency, this method can further approximate the actual trigger time, reducing the impact of time offset caused by destination-side queuing. The adjacent time threshold can be set to 2 seconds. In systems with significant buffering delays in log writes to disk, this threshold can improve the recall rate of true association alignment. To control the risk of mismatches, the 2-second threshold can be enabled only when a client-side span and server-side span pairing has been confirmed. The anchor phrase set can be expanded according to business type. For example, in scenarios with strict access control, invalid certificates and expired keys can be added; in storage-intensive scenarios, directories not found and devices read-only can be added. When expanding, the anchor type should still be mapped to three categories: access control anchors, capacity anchors, and storage anchors to ensure terminology consistency. The deduplication rule can add end-second discrimination. If identical call segments within the same second have an end-second difference of less than 1 second, only the one with the longer end-second is retained to cover the entire fault duration range.

[0040] Within the preset time window, a set of call fragments has been obtained. Each call fragment includes the source service identifier, source instance identifier, destination service identifier, destination instance identifier, start second, end second, and occurrence time. Monitoring-type and logging-type exception fragments aligned to instance identifier and time already exist. Anchor point type annotations have been completed for templates containing phrases such as "reject new connection," "insufficient threads," "queue full," and "disk unwritable." "Reject new connection" belongs to the access control anchor, "insufficient threads" and "queue full" belong to the capacity anchor, and "disk unwritable" belongs to the storage anchor. The time alignment granularity is seconds, allowing a relaxation of no more than 1 second between adjacent calls to buffer differences between log writes to disk and sampling boundaries.

[0041] Source-side trigger confirmation includes: at the source instance identifier, searching for log-type abnormal segments whose time interval overlaps with or is adjacent to the call segment by no more than 1 second. If the template text contains downstream call failure or timeout trigger words, it confirms that the source side has shown signs of call abnormality. This approach uses proactive error reporting from the source side as a starting point, avoiding misinterpreting the destination side's own alarms as upstream problems. Destination-side evidence confirmation includes: at the destination instance identifier, searching for log-type and monitoring-type abnormal segments whose time interval overlaps with or is adjacent to the call segment by no more than 1 second. If a capacity anchor or storage anchor exists in the log-type abnormal segment, or if any monitoring-type abnormal segment is time-aligned with the call segment, it confirms that an observable resource or persistence abnormality has occurred in the same time neighborhood on the destination side. Introducing objective evidence from the destination side can create a closed loop correspondence between the source-side manifestations and the callee's state, reducing misjudgments. Bilateral concatenation judgment includes: when both source-side trigger confirmation and destination-side evidence confirmation are satisfied, the call segment is marked as a call segment that can be stitched together, and the source-side trigger template text, destination-side anchor type, or destination-side monitoring-type abnormal segment time range used for judgment is recorded. Recording these details ensures the traceability of subsequent results and provides a basis for conflict trimming. When a single side is not valid, the following processing is performed: if there is only source-side trigger confirmation but no evidence on the destination side, it is not marked and is retained as a normal call fragment; if there is only destination-side evidence confirmation but no trigger word on the source side, it is not marked as a call fragment that can be stitched together. Here, we insist on both sides being valid to distinguish between upstream self-faults and downstream true faults.

[0042] Starting from any stitchable call segment, evidence chain units are generated in ascending order of occurrence time. Stitchable call segments are chosen as the starting point because they simultaneously present upstream manifestations and downstream evidence, making them closer to the first hop in fault propagation. At the destination instance identifier of the current call segment, the next call segment with that destination instance identifier as its source instance identifier is searched. This requires that its start second is the same as or no more than one second adjacent to the end second of the current call segment, and that the next call segment itself is a stitchable call segment. If these conditions are met, it is continued into the current sequence. Using the destination instance identifier as the continuation key allows for downstream progression along the actual service call direction, while the time continuity constraint ensures that the propagation path closely follows the actual order of events. If no call segment satisfying the continuation rules is found, or if the next call segment found has neither an anchor type nor a monitored anomaly segment on the destination side, or if the start second of the next call segment differs from the end second of the current call segment by more than one second, then propagation stops. The stopping condition limits the propagation time and evidence continuity, preventing unrelated events from being chained together. If a source service identifier and destination service identifier already appear in the current sequence during the continuation process, the expansion of that branch is terminated, and the last call segment is used as the end of the sequence. This constraint avoids loops in retries or circular calls. The consecutive call segment sequences that satisfy the above rules, along with monitoring and log exception segments aligned with each segment, are collectively solidified into an evidence chain unit. The start second, end second, and the sequence index of the segments within the sequence are saved for easy comparison and pruning later.

[0043] A conflict is identified when two evidence chain units cover the same source service identifier and destination service identifier in the same second, and the anchor semantics corresponding to the two chains in that second are opposite. Opposite anchor semantics include one chain showing access restriction, resource exhaustion, or write failure, while the other chain shows recovery semantics such as change completion or successful rollback. In the event of a conflict, evidence chain units containing access control anchors, capacity anchors, or storage anchors are retained, while evidence chain units containing recovery semantics are deleted. This is based on the fact that rejection of new connections, insufficient threads, full queues, and disk unwritable directly represent failure states, while restart completion and successful rollback are often post-event actions, and their occurrence does not mean that the failure has been eliminated at that moment. When two evidence chain units both contain failure-type anchors but with different phrases, the evidence chain unit whose anchor occurrence time is closer to the time of the call segment is retained. The failure phrase closer to the occurrence time is usually more closely related to the causal relationship of the call. When the time proximity is the same, selection is based on the phrase order, with priority as follows: disk unwritable, full queue, insufficient threads, and rejection of new connections. This order reflects the ranking of failure strength from irreversible to remediable, which helps to make deterministic choices in the face of concurrent conflicts. After conflict pruning, if the deleted chain still contains segments that do not conflict with any chain at other times, these non-conflicting segments are retained as independent evidence chain units to avoid mistakenly deleting valid evidence related to other paths.

[0044] All evidence chain units, after being pruned and pruned to separate points and conflicts, are arranged from earliest to latest according to their start time, forming a candidate evidence chain set. When different evidence chain units have most segments identical but differ in their endings, they are retained, as they represent two possible paths of propagation to different downstream branches, requiring separate evaluation of subsequent candidate root causes. For each evidence chain unit within the candidate evidence chain set, a segment list, details of the aligned monitoring-type and log-type anomaly segments, and the specific rule number used in the pruning process are saved to ensure the results can be reviewed and replayed.

[0045] For each unit of the evidence chain within the candidate evidence chain set, read the last segment of the sequence. The destination instance identifier of the last segment, along with the anchor type or monitoring-type anomaly segment directly aligned with it, are used as candidate root cause pointers. The last segment is selected as the pointer because the propagation no longer possesses temporal or evidentiary continuity at that point, indicating that the failure has reached its boundary and is most likely the direct source of upstream failures. For each candidate root cause pointer, output the destination service identifier, destination instance identifier, the occurrence time of the last segment, the time range of the anchor phrase or monitoring-type anomaly segment aligned with it, and a list of upstream call segments along the chain, facilitating the generation of the root cause list according to deterministic rules in step 3.

[0046] If the final segment aligns to both capacity and storage anchors, the anchor type that appears earlier is prioritized as the semantic description of the candidate root cause, as earlier failures are more likely to be the trigger for subsequent phenomena. If the evidence chain unit contains only one stitchable callable segment, a candidate root cause is still generated to avoid missing isolated but definitive failure events. If the chain terminates due to a time interval exceeding 1 second and the final segment has no anchor type but contains a monitoring-type anomaly segment, then the monitoring-type anomaly segment and the target instance identifier are used as the candidate root cause, covering scenarios where logs are missing but metrics are exceptionally clear.

[0047] In systems with significant log write latency, the adjacency relaxation can be set to 2 seconds. To reduce mismatch risk, this setting only applies to call segments that already have client-side and server-side span pairings. To avoid excessively long chains during anomaly storms, the maximum continuation length can be set to 32. Once the length is reached, an evidence chain unit is immediately generated and expansion stops to ensure controllable computation and clear interpretation. The anchor phrase set can be expanded according to industry, but the three types of mapping relationships with access control anchors, capacity anchors, and storage anchors must remain unchanged. For example, insufficient quota can be added in the financial industry, and quota overrun can be added in a multi-tenant platform. When distributed tracing data is missing downstream, it is permissible to use the destination instance identifier to identify inbound network error logs and monitoring anomaly segments appearing within the same second to jointly replace the next call segment for a one-time extension, but not to continue downwards, avoiding the introduction of uncertain paths.

[0048] Step 3: Output a root cause list for the candidate evidence chain set according to deterministic rules. If there is a capacity anchor or storage anchor at the end of the evidence chain, then take its purpose service and purpose instance as the root cause target, and give the relevant template text and the time range of adjacent monitoring anomaly fragments.

[0049] In the specific implementation process, the first input is a set of candidate evidence chains. Each evidence chain unit consists of several call segments sequentially arranged by occurrence time, and retains the time range of monitoring-type and log-type exception segments aligned with each call segment, as well as the original template text. The overall goal is to output a root cause list based on deterministic rules within a unified timeline and preset time window. Each root cause list entry includes at least the destination service identifier, destination instance identifier, occurrence time, anchor type or the time range of the monitoring-type exception segment, and a list of call segment sequences along the chain. For each evidence chain unit, the last segment is read, and the destination service identifier, destination instance identifier, and occurrence time of that segment are determined. The last segment is chosen as the judgment criterion because the evidence cannot continue to satisfy time continuity or evidence continuity at that point, exhibiting a continuity interruption. An interruption usually means that the callee at that point cannot provide a stable response, thus becoming the direct point of failure propagation.

[0050] For cases where the final segment contains a capacity anchor or a storage anchor: the destination service identifier and destination instance identifier of the final segment are identified as the root cause targets. This method is chosen because capacity anchors and storage anchors correspond to concurrent resource exhaustion and persistent write failure, respectively. Both immediately create a state where the callee cannot continue to provide services locally, directly explaining the upstream call failure and retries. The original template text is output, along with the time range of the monitoring-type exception segments adjacent to the final segment at the destination instance identifier. Providing the original template text accurately presents the trigger semantics, and providing the time range of adjacent monitoring-type exception segments quantifies the duration of the callee's local state, facilitating verification. If both capacity and storage anchors appear in the same final segment, the anchor type that occurred earlier is selected as the anchor type for that entry, and the existence of the other anchor type is added to the description. The earlier anchor type is closer to the initial failure point, avoiding treating secondary phenomena as the primary cause.

[0051] For cases where the final segment only has access control anchors or only change anchors: When the final segment has an access control anchor and its upstream neighboring segment matches a template containing downstream call failure or timeout trigger words on the source side, the destination service identifier and destination instance identifier of the final segment are identified as the root cause target, and the original template text related to access restrictions and the time range of adjacent monitoring-type abnormal segments are output. Access restrictions will be immediately detected on the inbound path, and the simultaneous occurrence of upstream errors and inbound rejections can form a self-consistent explanation. When the final segment only has change anchors and its upstream neighboring segment contains downstream call failure or timeout trigger words on the source side, and the final segment does not have other anchor types or monitoring-type abnormal segments on the destination side, the destination service identifier and destination instance identifier of the final segment are identified as the root cause target, and the original template text related to the change is output. Change completion usually means the configuration switch or restart is over. If an upstream failure occurs immediately afterward and there is no other abnormal evidence locally, the incompatibility or cold start state introduced by the change is most likely to explain the failure.

[0052] When the final segment does not contain any anchor type on the destination side, but there are monitoring-related anomaly segments that overlap in time with or are adjacent to the final segment by no more than 1 second, the destination service identifier and destination instance identifier of the final segment are identified as the root cause targets, and the time range of the monitoring-related anomaly segment is output. This determination method is chosen because missing logs are common in distributed systems, and monitoring-related anomaly segments can reflect local instability from a numerical behavior perspective. Aligning the two can avoid omitting numerical evidence that has a more direct impact on the actual situation.

[0053] When multiple evidence chain units point to the same destination service identifier and destination instance identifier within two consecutive seconds, they are merged into a single root cause target entry. The earliest occurrence time is used, and the original template text is listed in ascending order of occurrence time. Merging avoids repeatedly outputting multiple neighboring entries for the same entity, making the list more compact and readable. When different evidence chain units cover the same source service identifier and destination service identifier within the same second and have opposite anchor semantics, entries containing capacity or storage anchors are prioritized. Insufficient capacity and write failures directly interrupt external services, and their evidence is closer to the root cause of the fault; recovery-related semantics more often represent post-event actions and cannot independently explain upstream failures at the time. When two entries are both failure-related anchors and occur at the same time, their output priority is selected according to a fixed order: storage anchor, capacity anchor, and access control anchor. This priority reflects the order from unsustainable to manageable through adjustments, which is beneficial for prioritizing actual handling.

[0054] The organization and output of the root cause list includes: generating a structured description of each root cause target, containing the following information: destination service identifier and destination instance identifier; occurrence time; anchor type or time range of the monitored exception segment; original template text, if multiple original template texts exist, arranged in ascending order of occurrence time; a list of call segment order, listed in the following order: source service identifier, source instance identifier, destination service identifier, destination instance identifier, and occurrence time. The root cause list is output in ascending order of occurrence time. When the list length exceeds the operational acceptance limit, entries with earlier occurrence times and more concentrated evidence on the same destination instance identifier are output first, ensuring that issues with the greatest potential impact are addressed first.

[0055] For each root cause target, the source identifier from the candidate evidence chain set is saved, including the index of the evidence chain unit, the position of the last segment in the chain, the time range of the monitoring-type and log-type abnormal segments involved in the judgment, and the original template text. Operations personnel can verify each segment sequentially from upstream to downstream, first confirming the target instance identifier of the last segment on-site, and then checking the directly adjacent upstream segments to eliminate false associations caused by concurrent events.

[0056] For cases where the final segment has multiple anchor point types and occurs at different times: select the anchor point type with the earlier occurrence time as the judgment criterion, and list the original template text and occurrence time of other anchor point types in the root cause list to avoid omitting statements that have diagnostic value for the fault chain. For cases where the evidence chain unit contains only one stitchable callable segment: still generate a root cause target. Single-hop chains often appear at edge dependencies or middleware entry points; failing to output them will miss direct and actionable remediation suggestions. For cases where the evidence chain unit terminates due to a time interval exceeding 1 second and the final segment has no anchor point type: if there is a monitoring-type abnormal segment at the target instance identifier, output it as if there is no anchor point but an abnormal indicator; if there is no monitoring-type abnormal segment, do not output the entry and note the reason in the review record to avoid elevating a chain lacking evidence to a root cause target.

[0057] When log write-to-disk latency is significant, the adjacent 2-second intervals in conflict and duplicate handling can be adjusted to adjacent 3-second intervals to accommodate time jitter caused by larger output buffers. To control the risk of erroneous merging, this threshold should only be enabled for entries that already contain capacity or storage anchors. In scenarios requiring rapid handling, only entries with capacity or storage anchors can be output, while the output of entries related to access control anchors and change anchors can be delayed to prioritize resolving faults with the greatest impact on availability. When the original template text length exceeds the single-entry output limit, priority should be given to retaining one line of context before and after the key phrase to improve readability without affecting the judgment criteria. Consistency annotations can be added to root cause list entries, including whether the source-side trigger exists, whether the destination-side monitoring anomaly fragment exists, and whether the anchor type of the final fragment exists. This annotation helps to quickly identify whether the evidence is complete without changing the judgment rules.

[0058] refer to Figure 2Five instance nodes (node-01 to node-05) with the same service identifier were used as monitoring targets. Their CPU utilization was continuously collected within a preset time window, spanning 10 seconds, with a sampling frequency of once per second. Within each time second, the system automatically collected monitoring values ​​from all instances under the same service identifier and sorted them by size. Taking T3 as an example, the collected CPU utilization values ​​of the five instances were [2%, 5%, 10%, 10%, 10%], and the median of 10% was taken as the reference benchmark for the group. Then, the abnormal threshold boundaries were calculated: the upper threshold was twice the median (i.e., 20%), and the lower threshold was 0.5 times the median (i.e., 5%). When the value of an instance in three consecutive seconds was greater than twice the reference benchmark, the system merged these three seconds into a single monitoring-type abnormal segment. As shown in the figure, the CPU utilization of instance node-02 consistently reached 40% during the T4-T6 time period, far exceeding the threshold of 30% (twice the median of 15%), triggering the generation of abnormal segment 1. The segment starts at T4 and ends at T6. If this condition is met again, the end time of the segment is extended. Similarly, when an instance's value is continuously less than half of the reference value in the same group, a corresponding abnormal segment is generated. Instance node-03 experienced a value below half the median during the T8-T9 time period, but because it only lasted 2 seconds, it did not meet the trigger condition of 3 consecutive seconds, and therefore no valid abnormal segment was generated. This invention effectively avoids the limitations of traditional fixed threshold methods by calculating the median of the same group in real time. Experimental data shows that during the T1-T6 time period, the median benchmark for each instance dynamically adjusts with load changes (gradually increasing from 10% to 25%), ensuring the accuracy and adaptability of anomaly detection. This method is particularly suitable for scenarios with multiple instances running concurrently in a microservice architecture, accurately identifying abnormal instances that deviate from the group's behavior pattern, providing a reliable data foundation for subsequent root cause localization.

[0059] Define the following service scenario: The preset time window is from second 0 to second 59. There are three service identifiers: Order Service, Inventory Service, and Database Service; the corresponding instance identifiers are Order Service Instance A, Order Service Instance B, Inventory Service Instance A, Inventory Service Instance B, and Database Service Instance A, respectively. The unified timeline granularity is seconds. Monitoring selects the error rate percentage per second as the numerical monitoring. Logs are recorded using template text. Distributed tracing records the client-side and server-side spans. For each second and each service identifier, collect the monitoring values ​​of all instance identifiers for that service and calculate the reference for the same group.

[0060] Let x be the monitoring value of the i-th instance identifier under the same service identifier at second t. i,t (x i,t (This represents the instance monitoring value), where the reference value for this service in the same group at second t is m. t (m t(This represents the median within the same second). The calculation method is to sort all instance values ​​within that second and then take the median. Taking the inventory service as an example, at seconds 20, 21, and 22, the four sequences are as follows: Second 20: Inventory service instance A is 12, inventory service instance B is 3; other instances in the same service are assumed to be 4 and 5, then the median m of {3, 4, 5, 12} is... 20 =4.5. At the 21st second: Inventory service instance A is 14, Inventory service instance B is 4, and the others are 5 and 5, then m 21 =4.5. At the 22nd second: Inventory service instance A is 15, Inventory service instance B is 4, and the others are 5 and 6, then m 22 =4.5.

[0061] When x is satisfied for 3 consecutive seconds i,t ≥2·m t or x i,t ≤0.5m t When the value is 12, 14, and 15 at seconds 20 to 22, respectively, all satisfying the condition of being no less than twice the reference value in the same group (e.g., 12 at second 20 ≥ 2 × 4.5), a monitoring-type anomaly segment is generated for inventory service instance A, starting at second 20 and ending at second 22. If the value is 10 at second 23 and the reference value in the same group is 4, the condition is still met, and the ending second is extended to second 23. Adjacent anomaly intervals with an interval of no more than 2 seconds are merged into a longer interval.

[0062] Log processing: For each log record, template merging is performed, replacing numeric strings, hexadecimal strings, network addresses, file paths, and unique identifiers with placeholders while maintaining the word order, to obtain the template key. If the same instance identifier appears 10 times within any consecutive 10 seconds using the same template, or if the template text contains words from the trigger word set (failure, timeout, rejection, unreachable, exhausted, full, insufficient permissions, rollback, crash), then a log-type exception fragment is generated using the seconds of the first and last occurrences.

[0063] Example 1: Inventory service instance A experiences the "Queue Full" error 12 times between seconds 21 and 25, forming a log exception fragment. The start time is 21 seconds, the end time is 25 seconds, and the anchor point type is labeled as a capacity anchor point. Example 2: Database service instance A experiences the "Disk Not Writable" error 8 times between seconds 23 and 27, forming a log exception fragment. The start time is 23 seconds, the end time is 27 seconds, and the anchor point type is labeled as a storage anchor point. Example 3: Order service instance A experiences the "Downstream Call Failed" error 7 times between seconds 22 and 26, forming a log exception fragment. The start time is 22 seconds, the end time is 26 seconds.

[0064] Distributed tracing extraction: Retaining a set of fields within a unified timeline. Let the tracing identifier be r (r represents the identifier of a complete call chain), the span identifier be s (s represents the identifier of a single span), the parent span identifier be p (p represents the parent span identifier), the service identifier be S (S represents the service name), the instance identifier be I (I represents the instance name), and the start second be t. start The end time is t. end Example tracing 1: r = 1001 contains two spans, the client span is in order service instance A, (S = order service, I = instance A, t start =22,t end =22); The server span is in inventory service instance A, (S = inventory service, I = instance A, t start =22,t end =22), and the parent-child relationship satisfies p 服务端 =s 客户端 Example Tracking 2: r=1002 includes client span in inventory service instance A and server span in database service instance A, with start seconds of 23 and 23 respectively, and end seconds of 23 and 23 respectively. The parent-child relationship is the same as above.

[0065] Call fragment generation: when there exists the same r and satisfies p 服务端 =s 客户端 At that time, a call fragment is generated. Let the call fragment be... src ,I src ,S dst ,I dst ,t h >, where S src with I src S represents the source service and the source instance. dst with I dst Indicates the purpose service and the purpose instance, t h Indicates the occurrence time, taking the first second of the client span. The call fragment obtained from Example Trace 1 is <Order Service, Instance A, Inventory Service, Instance A, 22>. The call fragment obtained from Example Trace 2 is <Inventory Service, Instance A, Database Service, Instance A, 23>.

[0066] Stitchable call segment determination: Define a time overlap function overlap([a,b],[c,d]) (overlap indicates whether two closed intervals overlap or are adjacent for no more than 1 second). When the source side is in [t h ,t h ​If within a 1-second range of a given point, there exists a log-type exception segment containing the trigger words "downstream call failed" or "timeout", and the destination side has a capacity anchor or storage anchor, or a monitoring-type exception segment within the same range, then the call segment is a stitchable call segment. For <Order Service, Instance A, Inventory Service, Instance A, 22>: The log-type exception segment range of the source side Order Service Instance A is [22, 26], containing "downstream call failed"; the log-type exception segment range of the destination side Inventory Service Instance A is [21, 25], marked as a capacity anchor, and the monitoring-type exception segment range is [20, 23]. These conditions satisfy overlap([22, 22], [21, 25]) and overlap([22, 22], [20, 23]), therefore, this call segment is a stitchable call segment. For <Inventory Service, Instance A, Database Service, Instance A, 23>: The source-side inventory service instance A has a "failed to call downstream" log (a record of the inventory service's failed database request) in the interval [21, 25]. The destination-side database service instance A has a "disk not writable" log-type exception fragment in [23, 27] and is marked as a storage anchor. At the same time, the database side may also have monitoring-type exception fragments (for example, the error rate continues to rise in [23, 26]). Therefore, this call fragment is also a call fragment that can be stitched together.

[0067] Evidence chain unit generation: Starting with the first stitchable call fragment <Order Service, Instance A, Inventory Service, Instance A, 22>, the next stitchable call fragment with Inventory Service instance A as the source instance and whose occurrence time is no more than 1 second apart is obtained along the destination instance, resulting in the second fragment <Inventory Service, Instance A, Database Service, Instance A, 23>. The two fragments satisfy temporal continuity, forming the evidence chain unit [<Order Service, Instance A, Inventory Service, Instance A, 22>, <Inventory Service, Instance A, Database Service, Instance A, 23>]. In this instance, there are no concurrent chains with opposite anchor semantics, so no conflict pruning is required. The candidate evidence chain set therefore contains 1 evidence chain unit.

[0068] Read the last segment of the evidence chain unit <Inventory Service, Instance A, Database Service, Instance A, 23>. The target database service instance A has a log-type exception segment "Disk not writable" at [23, 27], with an anchor type of storage anchor, satisfying the deterministic rule that "a storage anchor exists at the end of the evidence chain." Therefore, the root cause target is the database service and database service instance A. Output items include: the target service identifier is database service, the target instance identifier is database service instance A, the occurrence time is 23, the original template text is "Disk not writable," and the time range of adjacent monitoring exception segments is [23, 26] (e.g., the error rate continuously deviates from the same group reference within this range). A list of the call segments along the chain is also provided, in the order of order service instance A to inventory service instance A (occurrence time 22), and inventory service instance A to database service instance A (occurrence time 23), for verification.

[0069] If another evidence chain unit points to the same second of database service instance A within the same preset time window, it is merged into a single entry according to the adjacent 2-second merging rule, and the earliest occurrence time is selected as the output time. If the last segment does not have an anchor type but database service instance A has a monitoring-type abnormal segment in [23,26], the root cause target can also be output as database service instance A according to the rule of "no anchor but abnormal indicator", and the interval of the monitoring-type abnormal segment is provided as an explanation. The adjacent relaxation of the unified timeline can be adjusted from 1 second to 2 seconds for systems with large log disk latency; the merging interval of monitoring-type abnormal segments can be adjusted from 2 seconds to 1 second to improve segmentation sensitivity; phrases such as "directory does not exist" and "device read-only" can be expanded in the anchor type set, and they are still merged into storage anchors. The above substitutions do not change the judgment process, only the threshold size and the content of the phrase set.

[0070] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. An AIOps anomaly detection and root cause localization method, characterized in that, The method includes: Step 1: Under a unified timeline, process preset time windows by service and instance, generate monitoring-type anomaly fragments for monitoring metrics, generate log-type anomaly fragments by merging runtime logs with templates and identifying trigger words, and extract the client and server spans from distributed tracing for fragment pairing. Step 2: Perform stitching based on distributed tracing. Pair client spans and server spans with the same tracing identifier to generate call fragments, and record the source service, source instance, destination service, destination instance, and occurrence time. Search for monitoring-type exception fragments that overlap with or are adjacent to each other for no more than one second on the source and destination sides, as well as log-type exception fragments containing phrases such as "reject new connection," "insufficient threads," "queue full," and "disk unwritable." Mark the anchor type for categorizable templates. When a call fragment contains a log-type exception fragment containing downstream failure or timeout trigger words on the source side and has a capacity anchor or storage anchor on the destination side, it is recorded as a stitchable call fragment. Starting from any stitchable call fragment, generate evidence chain units by continuing according to the destination instance and time continuity. Prune evidence chain units that simultaneously cover the same source and destination services and have opposite anchor semantics according to the conflict pruning rules to obtain a candidate evidence chain set, and use the last fragment of each evidence chain unit as the candidate root cause pointer. Step 3: Output a root cause list for the candidate evidence chain set according to deterministic rules. If the last end of the evidence chain contains a capacity anchor or storage anchor, then its target service and target instance are the root cause targets, and provide relevant template text and the time range of adjacent monitoring anomaly segments. Specifically, when the last end of an evidence chain unit contains an anchor of type capacity anchor, storage anchor, or access control anchor, the target service identifier and target instance identifier of that last segment are the root cause targets; when the last end of an evidence chain unit only contains a change anchor and its upstream adjacent segment contains downstream call failure or timeout trigger words on the source side, but the target side does not have any anchors... When dealing with point segments and monitoring-type abnormal segments, the destination service identifier and destination instance identifier of the last segment are used as the root cause target. When the evidence chain unit has no anchor segment but has three consecutive call segments whose destinations are all aligned to the monitoring-type abnormal segment, the destination service identifier and destination instance identifier of the last call segment are used as the root cause target. For each root cause target, an evidentiary explanation is output. The explanation includes at least the original template text that triggered the call segment, the call segment sequence involved in the stitching, listed in order of source service identifier, destination service identifier and occurrence time, and the time range of the monitoring-type abnormal segment directly adjacent to the root cause target.

2. The AIOps anomaly detection and root cause localization method as described in claim 1, characterized in that, In step 1, within the instance identifier set under the same second and the same service identifier, the monitoring values ​​of all instances are sorted by size, and the median is taken as the reference for the same group. When the value of an instance in three consecutive seconds meets the condition of being continuously greater than twice the reference for the same group or continuously less than half the reference for the same group, these three seconds are merged into a monitoring-type abnormal segment. If the condition is still met in the future, the end second of the segment is extended.

3. The AIOps anomaly detection and root cause localization method as described in claim 2, characterized in that, In step 1, the runtime log data is template merged, and numeric strings, hexadecimal strings, network addresses, file paths and unique identifiers are replaced with placeholders. Fixed word order text is used as template. When the same instance identifier appears at least ten times in any consecutive ten-second interval, or when the template text contains any of the trigger words such as failure, timeout, rejection, unreachable, exhausted, full, insufficient permissions, rollback and crash, log-type exception fragments are generated based on the first and last occurrence seconds. Retain span records within the current time window from the distributed tracing data, saving the tracing identifier, span identifier, parent span identifier, service identifier, instance identifier, span type, start second, end second, status field, and destination network address field; when monitoring-type abnormal segments and log-type abnormal segments have the same service identifier and instance identifier and overlap or are adjacent for no more than two seconds, establish a segment pairing relationship and record it in the abnormal segment set.

4. The AIOps anomaly detection and root cause localization method as described in claim 3, characterized in that, In step 2, the process of generating call fragments includes: filtering records with span types of client and server from the distributed tracing data; generating a call fragment for pairs with the same tracing identifier and whose parent span identifier of the server record is equal to the span identifier of the client record; the call fragment includes the source service identifier, source instance identifier, destination service identifier, destination instance identifier, start second, and end second; generating a call fragment when the client record and the server record do not have the above pairing but their tracing identifiers are the same, the difference in start second is no more than one second, and the destination network address field is the same; deduplicating multiple call fragments with the same source service identifier, source instance identifier, destination service identifier, and destination instance identifier within the same second and retaining the one with the earliest start second.

5. The AIOps anomaly detection and root cause localization method as described in claim 4, characterized in that, In step 2, anchor point types are identified and generated in log exception fragments based on template text. Among them, new connection rejection, invalid certificate, insufficient permissions, and expired key are classified as access control anchor points; insufficient threads, full queue, exhausted connection pool, and full file handle are classified as capacity anchor points; disk cannot be written, directory does not exist, device is read-only, and verification failure are classified as storage anchor points; configuration has taken effect, restart completed, and rollback successful are classified as change anchor points. For each call segment, search for monitoring-type abnormal segments and anchor segments that overlap with or are adjacent to the start and end seconds of the call segment by no more than one second on the source instance identifier and destination instance identifier to form an alignment result. When there is a log-type abnormal segment on the source side containing downstream call failure, timeout, or retry trigger words and there is a capacity anchor or storage anchor on the destination side, or there is a monitoring-type abnormal segment on the destination side, mark the call segment as a call segment that can be stitched.

6. The AIOps anomaly detection and root cause localization method as described in claim 5, characterized in that, In step 2, message system-related keys are searched from the span attributes of the distributed tracing data. When the target name of the sender's span and the receiver's span are found to be the same and the difference in start seconds is no more than one second, an intermediate call fragment with the target name as the intermediate identifier is generated and the fragment is aligned with the call according to the corresponding rules. At the same time, log-type abnormal fragments containing keywords such as "published to queue", "consumed from queue", "topic", "partition", and "offset" are searched in the running log data. The queue or topic name in the fragment is extracted as the intermediate identifier. When the intermediate identifier is the same as the intermediate call fragment and the time overlaps or is adjacent for no more than one second, the intermediate call fragment is marked as a call fragment that can be stitched together.

7. The AIOps anomaly detection and root cause localization method as described in claim 6, characterized in that, In step 2, the process of generating an evidence chain unit includes: starting from any stitchable call segment, expanding from early to late according to the occurrence time; finding the next stitchable call segment with the same source instance identifier as the target instance identifier and whose start second is the same as or adjacent to the end second of the current call segment by no more than one second at the target instance identifier of the call segment and continuing it; if the continued call segment has neither an anchor segment nor a monitored anomaly segment on the target side, or if the difference between its start second and the end second of the previous segment exceeds one second, then the expansion stops. The resulting continuous call segment sequence and the monitored anomaly segment and anchor segment aligned to it together constitute an evidence chain unit.

8. The AIOps anomaly detection and root cause localization method as described in claim 7, characterized in that, In step 2, when two evidence chain units cover the same source service identifier and destination service identifier in the same second but the corresponding anchor type semantics are opposite, the evidence chain unit containing the access control anchor, capacity anchor, or storage anchor is retained and the other one is deleted; when any call fragment in the evidence chain unit fails to align to the source side or the destination side monitoring-type abnormal fragment or anchor fragment, the call fragment and its subsequent fragments are deleted, and the remaining part is solidified into a candidate evidence chain set; the destination instance identifier of the last fragment of each evidence chain unit and its anchor type are output as root cause candidates. If the last fragment does not have an anchor fragment but has a monitoring-type abnormal fragment, the destination instance identifier and the monitoring-type abnormal fragment are output as root cause candidates.

Images