Troubleshooting methods, devices, equipment, media, and program products

Abstract

This application discloses a fault handling method, apparatus, device, medium, and program product. The method includes: analyzing fault information of a service topology network to obtain faulty services and fault times; performing fault context awareness on the faulty service based on the call dependencies between services in the service topology network and the fault time to obtain associated services of the faulty service in the service topology network and the running status data of the associated services during the fault time; performing statistical causality checks on abnormal events existing in the running status data based on the call dependencies to obtain target abnormal events, wherein the target abnormal event is an event among multiple abnormal events that does not have temporal statistical causality; and determining a fault solution matching the target abnormal event.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of data processing technology, and in particular to a fault handling method, a fault handling apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the widespread adoption of cloud-native architecture and microservices technology, modern business systems are generally deployed on large-scale computer clusters. These clusters are characterized by a large number of nodes, dense service instances, dynamic resource scheduling, and complex network topology, which leads to an explosive growth in observable data and makes faults more frequent, more complex in their manifestation, and longer in their propagation path.

[0003] When such clusters fail, it is necessary to quickly locate and handle the fault. In related technologies, fault alarm information is usually processed manually, logs, metrics and tracking data are manually correlated and analyzed, and the root cause is inferred and repairs are attempted based on personal experience. However, in practical applications, this approach relies heavily on human experience, making it difficult to accurately locate the cause of the fault, resulting in excessively long fault repair times and low repair efficiency. Summary of the Invention

[0004] This application provides at least one fault handling method, fault handling device, computer equipment, computer-readable storage medium, and computer program product.

[0005] The technical solution of this application embodiment is implemented as follows: On one hand, embodiments of this application provide a fault handling method, the method comprising: analyzing fault information of a service topology network to obtain faulty services and fault times; performing fault context awareness on the faulty service based on the call dependencies between services in the service topology network and the fault times to obtain associated services of the faulty service in the service topology network, and the running status data of the associated services during the fault times; performing statistical causality checks on abnormal events existing in the running status data based on the call dependencies to obtain target abnormal events, wherein the target abnormal event is an event among multiple abnormal events that does not have temporal statistical causality; and determining a fault solution matching the target abnormal event.

[0006] Secondly, embodiments of this application provide a fault handling apparatus, the apparatus comprising: an acquisition module, configured to analyze fault information of a service topology network to obtain fault services and fault times; a perception module, configured to perform fault context perception on the fault service based on the call dependencies between services in the service topology network and the fault time, to obtain associated services of the fault service in the service topology network and the running status data of the associated services during the fault time; a verification module, configured to perform statistical causal verification on abnormal events existing in the running status data based on the call dependencies to obtain a target abnormal event, wherein the target abnormal event is an event among multiple abnormal events that does not have temporal statistical causality; and a determination module, configured to determine a fault solution matching the target abnormal event.

[0007] Thirdly, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the method provided in embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in the method provided in embodiments of this application.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements the steps in the method provided in embodiments of this application.

[0010] This application provides a fault handling method. By analyzing fault information in a service topology network, a fault service and fault time are obtained. Based on the call dependencies between services in the service topology network and the fault time, fault context awareness is performed on the fault service to obtain associated services in the service topology network and the operational status data of the associated services during the fault time. Statistical causality checks are performed on abnormal events in the operational status data based on the call dependencies to obtain a target abnormal event, wherein the target abnormal event is an event among multiple abnormal events that does not have temporal statistical causality. A fault solution matching the target abnormal event is determined. In this way, by using the call dependencies in the service topology network, statistical causality checks can be performed on the operational status data of associated services that trigger the fault service to obtain the target abnormal event that triggers the fault service. Based on the target abnormal event, a matching fault solution is obtained. This method can accurately determine the root cause of the fault based on the call dependencies, avoiding the problem in related technologies where reliance on human experience makes it difficult to accurately locate the root cause of the fault. This enables precise fault cause localization and improves fault repair efficiency.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0012] Figure 1 This is a flowchart of a fault handling method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a fault handling device provided in an embodiment of this application; Figure 3 This is a flowchart of another fault handling method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application.

[0013] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0017] Currently, with the widespread application of cloud-native architecture and microservice technology, modern business systems are generally deployed on large-scale computer clusters. These clusters are characterized by a large number of nodes, dense service instances, dynamic resource scheduling, and complex network topology, which leads to an explosive growth in the observable data of the system. At the same time, it also makes the occurrence of faults more frequent, the manifestations more complex, and the propagation links longer.

[0018] Specifically, within a cluster, multiple services form a service topology network due to complex call dependencies. A user request flows through multiple services within this network. Therefore, faults rarely occur in isolation; instead, they propagate rapidly along the dependency chains within the service topology network, forming fault domains that make it difficult to accurately pinpoint the cause. Furthermore, manual analysis struggles to quickly identify the root cause of anomalies from the massive amounts of alerts within these fault domains. When multiple independent faults occur concurrently, limited human resources are insufficient for effective parallel diagnosis and handling, easily leading to the cumulative and amplified impact of faults.

[0019] Moreover, the explosive growth of observable data in the cluster means that any failure will cause anomalies in massive amounts of data across different dimensions due to the existence of network dependencies. Manual analysis requires cross-data source and cross-timeline correlation searches and pattern recognition in massive logs, metrics, and tracking data, which consumes a lot of time to locate the cause of the failure, resulting in excessively long failure repair times and low repair efficiency.

[0020] To address the aforementioned problems, this application proposes a fault handling method, a fault handling apparatus, a computer device, a computer-readable storage medium, and a computer program product. This fault handling method, through the call dependencies in the service topology network, can perform statistical causal checks on the runtime status data of associated services that trigger the faulty service to obtain the target abnormal event that triggered the faulty service. Based on the target abnormal event, a matching fault solution is obtained. This method can accurately determine the root cause of the fault based on call dependencies, avoiding the problem in related technologies where reliance on human experience makes it difficult to accurately locate the root fault source. It enables precise fault cause localization and improves fault repair efficiency.

[0021] Figure 1 This is a flowchart of a fault handling method provided in an embodiment of this application, such as... Figure 1 As shown, this method can be executed by a cluster agent, which can be a lightweight agent running on the cluster management node or each service node, or a centralized automated operation and maintenance system deployed on a standalone server or cloud platform. The method may include the following steps.

[0022] Step S101: Analyze the fault information of the service topology network to obtain the fault service and fault time.

[0023] The service topology network may include, but is not limited to, the connection relationships between multiple nodes and any nodes. Each node may refer to different services or application components in the cluster. The connection relationships between any nodes may refer to the call relationships and dependency directions between any services, the call relationships and dependency directions between any application components, or the call relationships and dependency directions between any service and any application component. The service topology network may be a logical network used to characterize the real-time call and dependency relationships between services or application components in the cluster.

[0024] In this embodiment, the fault information of the service topology network can be derived based on information input by the user. Specifically, the information input by the user can be information actively entered into the intelligent agent by the user through a natural language interaction interface.

[0025] In actual use, when users use the intelligent assistant integrated into the operation and maintenance management platform or instant messaging tool, if users find that there is an anomaly in the cluster, they can directly describe the fault phenomenon in natural language. When the intelligent agent receives the information input by the user, it can start the problem understanding and information extraction module inside the intelligent agent, so that the problem understanding and information extraction module can analyze the information input by the user to obtain the fault information of the service topology network.

[0026] Upon receiving user input, the problem understanding and information extraction module uses a pre-trained domain semantic model to parse the input, identifying the fault entity, task identifier, trigger time, and keywords describing the fault's state. After obtaining multiple keywords, these keywords are populated into a predefined fault information template to obtain the fault information for the service topology network. The predefined fault information template can include multiple fields, such as fault object, fault type, fault time, and impact scope. If missing fields are detected after populating the predefined fault information template with extracted keywords, single-point follow-up questions can be generated according to a preset priority to proactively guide the user to supplement the information.

[0027] For example, when a user uses the operation and maintenance management platform, if they find that the order service response is slow, they can directly output "The order service response is slow" to the intelligent agent in natural language. When the problem understanding and information extraction module receives the user's input "The order service response is slow", it can start its internal pre-trained domain semantic model to parse the statement and extract key entities. Among them, "order service" is identified as the fault object entity, and "slow response" is identified and classified as a fault type keyword that represents performance abnormality.

[0028] Subsequently, the extracted key information can be populated into the corresponding fields of the predefined fault information template. Specifically, "Order Service" is populated into the fault object field, and "Slow Response" is mapped to "High Latency" and populated into the fault type field. At this point, the problem understanding and information extraction module will detect that the fault time and impact scope fields in the fault information template are still empty. In this case, the problem understanding and information extraction module can generate a single follow-up question for the high-priority fault time field based on preset priority logic, such as "When did this problem begin approximately?", and present it to the user through the interactive interface to guide them to provide additional information. After the user provides time information based on the follow-up question (e.g., "Around 2 PM today"), it is parsed again, the time expression is extracted, normalized, and then populated into the fault time field. Meanwhile, for the still empty impact scope field, inferences can be made based on the current dialogue context. Since the user description only involves a single service, it is assigned the default value of "Single Service". Once all fields of the fault information template are populated, a structured fault information object can be finally output. This object represents the fault information of the clear service topology network derived from the user's original natural language input.

[0029] It should be noted that throughout the multi-round interaction process, the dialogue state can be persistently managed by maintaining a session-level fault context object, and the consistency of information can be ensured by adopting a three-state update mechanism of overriding, supplementing, and correcting.

[0030] Considering that the fault object in the user input information is only a superficial, vague functional description or a high-level business module name, it does not directly correspond to a specific service instance that can be precisely addressed in the service topology network. For example, the fault object in the user input "order payment is slow" is order payment, but in the service topology network, it may contain multiple independently deployed microservices such as "order service," "payment service," "inventory service," and "risk control service." Among them, the business function of "order payment" involves the complete link of "order service" calling "payment service."

[0031] Therefore, in this embodiment, after obtaining the fault information of the serving topology network, the fault information can be analyzed to obtain the fault service and fault time. Specifically, multiple field rows corresponding to the fault information in the serving topology network can be analyzed to obtain the fault service and fault time.

[0032] In one possible implementation, based on the fault objects in the service topology network's fault information, a query and matching process can be performed within the service topology network to select one or more candidate services associated with the functional description. Then, performance metrics and error logs of all candidate services within a time window before and after the user-reported fault time can be retrieved and aggregated to identify the specific service exhibiting the most significant anomaly, thereby determining that service as the faulty service in this fault. Furthermore, time series analysis can be performed on the abnormal metrics of each candidate service, and the time point at which the first sustained deviation of the metric from the normal fluctuation range occurs can be detected, thereby determining that time point as the fault time of this fault.

[0033] Step S102: Based on the call dependencies and failure times among services in the service topology network, perform failure context awareness on the failure service to obtain the associated services of the failure service in the service topology network, as well as the running status data of the associated services during the failure time.

[0034] Among them, fault context awareness can refer to obtaining the running status data of related services that have a direct or indirect relationship with the faulty service within a preset time period corresponding to the fault time, based on the real-time call dependency relationship between services in the service topology network and with the faulty service as the center.

[0035] In this embodiment, related services that have a direct or indirect relationship with the faulty service may include, but are not limited to, upstream caller services, downstream dependent services, and peer services that share critical resources. Upstream caller services may be clients or front-end services that directly or indirectly call the faulty service; downstream dependent services may be back-end services actively called by the faulty service when processing requests; peer services may be services that can share resources such as databases, message queues, and caches with the faulty service. It should be noted that when determining related services that have a direct or indirect relationship with the faulty service, the scope of the relationship can be set to a single layer with direct relationships, or it can be set to extend to multiple layers with indirect relationships. The specific scope setting can be determined based on the current resource status of the agent, and this application is not limited to a specific scope here.

[0036] After obtaining the associated services of the failed service in the service topology network, a preset time window can be extended forward and backward based on the failure time. Monitoring tools can then be used to collect multi-dimensional operational status data of all associated services within this time window. This operational status data may include, but is not limited to, performance metrics, business and log data, and distributed link data.

[0037] Specifically, performance metrics can refer to quantified numerical data that changes over time and reflects the status and behavioral characteristics of service or system resources. Performance metrics can include, but are not limited to, resource metrics, service metrics, and dependency metrics. For example, resource metrics can include CPU utilization, memory usage, disk I / O, network bandwidth, etc.; service metrics can include request rate, error rate, response time, request throughput, etc.; dependency metrics can include database connection pool utilization, message queue backlog length, cache hit rate, etc.

[0038] Business and log data can refer to discrete event texts generated by services, applications, or system components during operation and recorded in chronological order. These records include errors and warnings related to program execution. Examples include database connection failures, timeouts when calling the target service, and memory overflow exceptions.

[0039] Distributed link data can refer to the complete path and lifecycle of an external request flowing within a distributed system. Specifically, it can include all services the external request passes through, the dependencies between services, the processing time in each service, and key events. For example, it can include the call path of the external request from the gateway, service A, service B, to the database; it can also include the time spent in each service it passes through.

[0040] Step S103: Perform statistical causal verification on the abnormal events existing in the runtime status data based on the call dependency relationship to obtain the target abnormal event.

[0041] Among these, the target anomaly event is an event among multiple anomalies that does not have a temporal statistical causal relationship. An anomaly event can refer to an event identified from the operational status data of a related service that corresponds to a sudden change in an indicator that significantly deviates from its historical baseline or expected pattern; for example, a service's response time spikes in a short period of time, its error rate increases sharply, or its throughput drops drastically.

[0042] In this step, we first identify anomalous events that significantly deviate from their historical baseline or expected patterns from the operational status data of related services. Then, we construct and filter causal networks for multiple anomalous events based on the call dependencies provided by the service topology network. Specifically, we construct causal hypothesis paths between anomalous events based on the call direction of the service call chain in the call dependencies. Then, for any two adjacent anomalous events on the causal hypothesis path, we extract detailed time-series data of these two anomalous events within the failure time window and perform statistical causal tests on the time-series data.

[0043] If an anomalous event is verified to be predictable by another anomalous event that occurred earlier in time, then the anomalous event can be labeled as having statistical causality; if an anomalous event is verified to be unpredictable by another anomalous event that occurred earlier in time, then the anomalous event can be labeled as not having statistical causality.

[0044] Step S104: Determine the fault solution that matches the target abnormal event.

[0045] In this embodiment, a fault solution corresponding to the target abnormal event can be selected from the fault knowledge base, and the fault solution can be executed to process the fault information of the service topology network.

[0046] The fault knowledge base may include a pre-built set of solution records, and each record in the solution record set may be associated with a specific fault mode.

[0047] In actual execution, the fault solutions corresponding to the target abnormal event can be queried from the fault knowledge base.

[0048] In one possible implementation, if no fault solution directly matching the target anomaly is found in the fault knowledge base, it indicates that the current fault is an unrecorded fault mode. In this case, a network query can be performed based on the attribute information of the target anomaly, such as its anomaly type, associated service components, or resource types, to obtain general repair operation knowledge and provide it to the user for decision-making. Alternatively, a prompt message can be output to inform the user that the target anomaly is an unrecorded fault mode, prompting the user to take independent action. With user authorization and after resolving the fault, the complete context, solution, and final result of this fault are recorded as a new case in the fault knowledge base, completing the accumulation of knowledge.

[0049] By adopting the above technical solution, statistical causal verification can be performed on the running status data of related services that trigger the faulty service through the call dependency relationship in the service topology network. This allows us to obtain the target abnormal event that triggers the faulty service and obtain a matching fault solution based on the target abnormal event. It can accurately determine the root cause of the fault based on the call dependency relationship, avoiding the problem of difficulty in accurately locating the root fault source due to reliance on human experience in related technologies. This enables accurate location of the fault cause and improves fault repair efficiency.

[0050] In some embodiments, step S102 described above can be implemented through the following steps.

[0051] Step S1021: Based on the preset field mapping relationship, determine the query field that has a mapping relationship with the service identifier of the fault service.

[0052] The field mapping relationship includes the correspondence between the service identifier and the corresponding query field.

[0053] In this step, the service identifier can be a specific key or label used by the faulty service when storing data in various observable data sources. For example, the service identifier can be represented as "order-service". Query fields can represent different types of data; query fields include trace identifier fields and timestamp fields. The trace identifier field can be log data generated by all services on the same request chain across services, and the timestamp field can be the time period used to determine all services associated across services. The field mapping relationship between the service identifier and the corresponding query field can define the scope of fault perception when performing fault context awareness on the faulty service. That is, the faulty service can be used as the fault perception center. Through the trace identifier field, the agent can associate all upstream and downstream services that actually interacted with the faulty service at the time of the fault along the call dependency relationship, thus determining the scope of fault perception in the service topology space. And through the timestamp field, logs, metrics, and other data of all associated services can be uniformly aligned to a preset time period centered on the fault time for collection, thus determining the scope of fault perception in the time dimension.

[0054] In actual execution, when a service identifier for a faulty service is input, the agent can automatically retrieve the accurate tracking identifier field and timestamp field bound to that service identifier based on the field mapping relationship. Subsequently, the agent can use these fields as unified coordinates to automatically perform cross-data source queries and association operations, thereby constructing a complete fault context.

[0055] In one possible implementation, since faulty services have different service identifiers in different data sources—for example, different tag keys such as app, job, and service.name can be used in logs, metrics, and tracing systems—it's not feasible to directly use a single logical service name for cross-data source queries. Therefore, the agent can internally set up a dynamic Service Signal Map (SSM) to map a single service to different data sources.

[0056] When context awareness of a faulty service is required, the system only needs to query the SSM (Service Management System) for the logical service name to automatically obtain the corresponding logical name needed to accurately locate the data in all relevant data sources. This avoids query failures caused by differences in data source labeling systems or manual configuration errors.

[0057] Step S1022: Based on the tracing identifier field and the call dependency, determine the associated services of the faulty service in the service topology network.

[0058] In some embodiments, the service identifiers on the request chain to which the faulty service belongs can first be determined based on the tracking identifier field; then, based on the call dependencies in the service topology network, the services that have call dependencies on the faulty service among all service identifiers can be identified as associated services.

[0059] Specifically, after obtaining the query field of the fault service in the tracing system by querying the Dynamic Service Signal Mapping Table (SSM) in step S1021, the agent can retrieve all tracing services involved in the fault and extract a set of key tracing identifiers, namely the tracing identifier list, in which the specific request links that flowed through the service during the fault can be identified.

[0060] Once the specific request path is obtained, the agent can execute cross-data source relational queries. Specifically, since all services in the cluster conform to the OTel (OpenTelemetry) specification, the tracing identifier of the current request path can be automatically injected into each application log entry through the mapping diagnostic context of the log framework. Therefore, the agent's data search module can execute standard query statements for each tracing identifier in the log aggregation system. This query will automatically return all log entries distributed across different nodes in the cluster, belonging to different services, but all belonging to the same complete request path.

[0061] Then, the data search module of the intelligent agent can parse the source tags in each log record to accurately extract the service name that generated the log. By performing source analysis on all logs belonging to the same tracking identifier list, the set of all services involved in the specific business request can be summarized. This set of services is generated based on the actual call facts and includes the direct upstream caller and direct downstream dependency of the faulty service.

[0062] Step S1023: Based on the timestamp field, the running status data of the associated service within the preset time period corresponding to the fault time is used as the running status data of the associated service during the fault time.

[0063] In this step, the agent's data search module can extract data fragments related to the current failure event in the time dimension for each associated service from the identified set of associated services, and align all multi-source heterogeneous data in a unified time coordinate system, thereby obtaining the operational status data of the associated services during the failure period.

[0064] In actual execution, the agent's data search module can utilize the identification information of each related service in logs, metrics, and tracking systems, combined with its corresponding timestamp field, to initiate queries to various observable data sources. Query conditions can include ensuring that the timestamp of the data record falls within the aforementioned preset fault time window. For example, it can query the log system for all entries of a specific service within the window, and the metrics database for all time-series data samples of the corresponding service within that time period.

[0065] By employing the above technical solution and utilizing tracking identifiers that adhere to open standards, the agent can reconstruct the actual call neighborhood at the moment of failure, ensuring the accuracy of the associated range. Moreover, the static topology completion mechanism can prevent the omission of key dependencies due to the failure of calls caused by the failure. This avoids the problems of inaccurate range definition and low efficiency in traditional manual investigation or static topology analysis. Furthermore, the windowed data pruning and alignment based on a unified timestamp field can efficiently extract data strongly related to the failure from massive amounts of data, improving the data quality and processing efficiency of subsequent analysis.

[0066] Considering that the logs, metrics, and other data generated within a time window during a fault occurrence may amount to hundreds of thousands or even millions of records, feeding all of this data into the root cause analysis module of the intelligent agent would result in extremely high computational load and significant processing latency, making it impossible for fault diagnosis to meet real-time requirements.

[0067] Therefore, in some embodiments, the running status data of the associated service during the fault time can be used as candidate running data, and the candidate running data can be denoised and ranked by scoring priority. Only the data with a score not less than a preset scoring threshold can be used as the running status data to be passed into the root cause analysis module. This can greatly improve the analysis efficiency while ensuring coverage of key information.

[0068] In one possible implementation, the runtime status data of the associated service within a preset time period corresponding to the fault time can be selected as candidate runtime status data for the faulty service based on the timestamp field. Then, each candidate data in the candidate runtime status data is scored according to different preset dimensions to obtain the relevance score of each candidate data in different preset dimensions. Based on the relevance score of each candidate data in different preset dimensions, the target score of each candidate data is obtained. Finally, the candidate runtime status data with a target score not less than a preset score threshold can be selected as runtime status data.

[0069] Specifically, when obtaining the target score for each candidate data, the relevance scores of each candidate data in different preset dimensions can be weighted and summed to obtain the target score for each candidate data.

[0070] Among them, the preset dimensions may include, but are not limited to, the time correlation dimension used to characterize the proximity of the timestamp of the data to the time of the failure, the entity correlation dimension used to characterize the degree of association between the service to which the data belongs and the service of the failure in the service topology network, and the signal strength correlation dimension used to characterize the degree of data anomaly.

[0071] Regarding the time-related dimension, since the occurrence, propagation, and impact of faults are strongly time-dependent, data closer to the fault's onset time is more likely to contain clues to the root cause. Therefore, the score for the time-related dimension decreases as time distance increases. Regarding the entity-related dimension, data generated by services that have direct call dependencies on the faulty service in the service topology (upstream callers or downstream dependencies) receive higher scores because they are the first link in the fault propagation. Data from services that are far away in the topology receive very low or zero scores. This limits the scope of analysis and prevents fault analysis from spreading to the entire cluster. Regarding the signal strength-related dimension, for log data, error-level logs can be assigned the highest score, warning-level logs the next lowest score, and information-level and lower-level logs the lowest score. For metric data, the greater the percentage deviation of the current value from the historical baseline (or static threshold), the higher the score. For example, a CPU utilization increase from 30% to 95% scores higher than an increase from 30% to 40%.

[0072] In this implementation, the three key judgment criteria of time proximity, topological proximity and signal severity can be integrated into a quantifiable comprehensive scoring index. The state data most relevant to the fault can be selected from the candidate operating state data as the operating state data of the associated service during the fault time, which can greatly improve the analysis efficiency while ensuring coverage of key information.

[0073] In some embodiments, before performing statistical causal checks on abnormal events in the runtime status data based on call dependencies to obtain the target abnormal event, the abnormal event triggered by the fault service can also be determined based on the statistical characteristics of the runtime status data.

[0074] Specifically, at least one of the following can be identified as an abnormal event: log abnormal event, resource abnormal event, and performance abnormal event.

[0075] Method 1: If abnormal data matching the preset error pattern is identified in the running status data, then a log abnormal event is triggered.

[0076] Among them, the preset error pattern can refer to the rules or features that the agent predefines to represent faults. For example, the preset error pattern can be represented as a regular expression used to match specific error information such as connection timeout and memory overflow.

[0077] Specifically, by scanning log data, when the content of a log entry matches any preset error pattern, it is considered that a log exception event has been captured.

[0078] Method 2: Compare each data point in the running status data with the dynamic baseline threshold. If there is a data point that is not less than the dynamic baseline threshold and the duration is not less than the preset duration, then a resource anomaly event is triggered.

[0079] Here, "operational status data" can specifically refer to resource metrics such as CPU utilization, memory usage, disk I / O, and network bandwidth. "Dynamic baseline threshold" refers to the fluctuation range calculated based on historical data, which can be automatically adjusted according to time, date patterns, and business cycles. For example, it could be the CPU utilization fluctuation range calculated based on CPU utilization during the same period over the past 7 days.

[0080] In actual execution, the agent can employ a dual-condition approach, judging operational status data based on a dynamic baseline threshold. This dual-condition approach can include, but is not limited to, threshold triggering and duration triggering. Specifically, threshold triggering can refer to the current resource metric exceeding (or not falling below) the calculated dynamic baseline upper limit, and duration triggering can mean that the exceeding state must persist for a preset duration. This avoids false alarms caused by instantaneous traffic spikes. Only when at least both of these conditions are met simultaneously is it determined to be a resource anomaly event.

[0081] Method 3: Compare each data point in the running status data with the historical performance baseline. If any data point is not less than the historical performance baseline, then an abnormal performance event is identified.

[0082] Here, the operational status data can specifically refer to the performance metrics data within the operational status data; for example, it may include, but is not limited to, data such as service response time and call latency; historical performance baseline can refer to a dynamic standard established based on historical statistical quantiles.

[0083] In actual execution, the agent can statistically analyze the distribution of the service's response time during normal periods in real time and calculate the statistical quantiles, such as P95 (95th percentile) or P99 (99th percentile), as a baseline. For example, if 95% of requests have historically had a response time of less than 100 milliseconds, then 100 milliseconds can be set as the baseline.

[0084] By employing the above technical solutions and performing log analysis based on preset error patterns, known fault signatures can be quickly identified, ensuring both the capture rate and real-time performance of explicit errors. Furthermore, the dual determination of dynamic baseline thresholds and duration, automatically calculated based on historical data, can adapt to the cyclical fluctuations and growth trends of business operations, filtering out noise caused by short-term interference such as instantaneous traffic spikes, thereby accurately identifying real and persistent resource bottlenecks and reducing false alarm rates. Moreover, by comparing current metrics with a performance baseline constructed based on historical statistical quantiles, the degradation trend of service response time can be identified, helping to discover potential performance degradation issues before a significant deterioration in user experience.

[0085] In some embodiments, step S103 can be implemented by the following steps.

[0086] Step S1031: Determine the abnormal time sequence corresponding to the abnormal event based on the timestamp of the abnormal event.

[0087] The abnormal time series includes multiple entries arranged in chronological order, with each entry corresponding to an abnormal event at a different time.

[0088] In this embodiment, the agent has identified multiple abnormal events triggered by the fault service based on the statistical characteristics of the operational status data. However, these abnormal events are independent. Only by sorting the events by time can the algorithm analyze the chronological relationship, interval patterns, and potential statistical causal relationships between the events, thereby inferring the propagation path of the fault. Therefore, by extracting the timestamp as a common attribute of each event, they can be arranged in chronological order to form a sequence with timeline significance.

[0089] Step S1032: Perform statistical causal tests on the abnormal time series based on the call dependency relationship to obtain the target abnormal event.

[0090] Among them, statistical causality testing is used to verify the propagation direction of the faulty service on the service dependency chain, thereby locating the initial root cause event that triggered the entire fault chain from multiple abnormal events that occurred sequentially in time.

[0091] In this embodiment, if two abnormal events (abnormal event A and abnormal event B) occur sequentially in time, it does not necessarily mean that the earlier abnormal event A will cause the later abnormal event B. Therefore, it is necessary to use mathematical methods to analyze the time series data of the abnormal indicators of abnormal events A and B to determine whether the change of the earlier abnormal event A will lead to the occurrence of the later abnormal event B in a mathematically statistical manner. Only when the test results meet the preset conditions, such as the probability value not being less than the preset threshold, can it be determined that the earlier abnormal event A will lead to the later abnormal event B.

[0092] Each abnormal event in the abnormal time series can be arranged according to the call dependency relationship to obtain a fault situation map that can determine when and where the abnormality occurs. Then, based on the call relationship of each abnormal event in the fault situation map, the fault propagation path can be determined.

[0093] For example, in a microservice architecture, there are three services: an order service, a payment service, and a database. Their call dependencies can be represented as follows: the order service calls the payment service for authentication when creating an order; both the payment service and the order service need to query the database. When a user reports a slow order submission, the agent can construct the following fault situation diagram: a resource anomaly event ("abnormal CPU utilization, timestamp T1") is marked on the database node; a log anomaly event ("call timeout, timestamp T2") is marked on the payment service node; and a performance anomaly event ("API response timeout, timestamp T3") is marked on the order service node. At this point, based on the call dependencies in the service topology diagram, all directly adjacent pairs of anomalous nodes can be identified, thereby determining multiple candidate paths for fault propagation that need to be verified.

[0094] Specifically, two candidate paths can be generated. The first path is: database, payment service; this is because the payment service depends on the database, and both of them are experiencing anomalies. The second path is: payment service, order service; this is because the order service calls the payment service, and both of them are experiencing anomalies.

[0095] After obtaining the fault propagation path, the operational status data corresponding to each pair of adjacent abnormal service nodes on the fault propagation path can be extracted, and statistical causal tests can be performed on the extracted operational status data to determine whether there is a causal relationship between adjacent abnormal service nodes.

[0096] In all the causal relationships confirmed by statistical testing, the agent can trace back in the reverse direction through the confirmed causal relationships until it finds one or more abnormal nodes without any upstream cause. In this case, the abnormal event corresponding to the abnormal node without any upstream cause is the target abnormal event.

[0097] In one possible implementation, candidate causal paths in the service topology network can be determined based on call dependencies. These candidate causal paths include adjacent exception events with call dependencies.

[0098] In this implementation, the agent can determine candidate causal paths based on the call dependencies in the service topology network and the currently identified set of abnormal events.

[0099] Specifically, the agent can scan the service topology network to identify all node pairs. These pairs can be service nodes with a direct call dependency and both nodes have been flagged with at least one anomalous event within the current failure time window. Each pair of adjacent anomalous nodes can constitute a candidate failure propagation path to be verified. For example, if service A calls service B, and both experience high CPU usage and a response timeout within a similar timeframe, then the edge "A to B" would be listed as a candidate path.

[0100] Then, the abnormal time series can be verified based on the candidate causal paths to determine the parent node events adjacent to each abnormal event in the abnormal time series. The parent node event is the event that triggered the adjacent abnormal event.

[0101] In this implementation, the candidate time series corresponding to the candidate causal path can be determined based on each pair of adjacent abnormal events with call dependencies in the candidate causal path; then, statistical causal tests can be performed on the abnormal time series based on the time statistical causal data corresponding to the candidate time series to determine the parent node event corresponding to each abnormal event in the abnormal time series.

[0102] Specifically, the agent can extract the temporal statistical causal data of the abnormal events corresponding to the upstream and downstream nodes in each candidate causal path and perform statistical causality tests to determine whether the abnormal changes in the upstream nodes statistically lead to the abnormal changes in the downstream nodes. If the test passes, the causal relationship can be confirmed, and the upstream abnormal event can be marked as the parent node event of the downstream abnormal event. This parent node event is the event that can directly trigger adjacent downstream anomalies in the verified propagation chain.

[0103] Finally, abnormal events that do not have parent node events in the abnormal time series can be used as target abnormal events.

[0104] Specifically, after the agent completes the verification of all candidate causal paths, the anomaly event that does not have any parent node event in the anomaly time series can be taken as the target anomaly event. In other words, this anomaly event can be the starting point of all verified fault propagation chains, and there is no statistical evidence that this anomaly event was caused by anomalies in other services in the topology. Therefore, this anomaly event can be determined as the root cause of the entire fault chain.

[0105] For example, in a call chain of order service, inventory service, and database, if it is verified that the database exception is the parent node of the inventory service exception, and the inventory service exception is the parent node of the order service exception, then the database exception is an event without a parent node, and thus the database exception can be located as the target exception event (root cause).

[0106] In one possible implementation, the Granger causality test can be used to perform statistical causality testing. This method can mathematically verify whether an anomaly in one service statistically caused an anomaly in another dependent service. The specific testing steps can employ existing techniques, which will not be elaborated upon here.

[0107] By adopting the above technical solution, service node pairs that have direct call dependencies and are simultaneously experiencing anomalies can be selected from the service topology network as fault propagation paths to be verified. This avoids computational noise and resource consumption caused by combinations of irrelevant events, thereby improving the processing efficiency of the root cause analysis process. Furthermore, by mathematically verifying the causal relationship of each candidate path, it is possible to distinguish between temporal correlations and true causal relationships, thus improving the accuracy and reliability of root cause localization.

[0108] In some embodiments, the method may further include the following steps.

[0109] Step S105: Select the candidate repair solution corresponding to the target abnormal event from the fault knowledge base.

[0110] Step S106: Based on the preset operation step template and the resource status and topology information of multiple services in the service topology network, adjust the operation steps of the candidate repair scheme to obtain the candidate fault solution.

[0111] Step S107: Perform a repair simulation of the candidate fault solutions in the repair sandbox, and if the simulation is successful, use the candidate fault solutions as fault solutions and execute the fault solutions to process the fault information of the service topology network.

[0112] Specifically, a simulation sandbox environment isolated from the production environment can be constructed based on traffic mirroring and state snapshots of multiple services in the service topology network. In the simulation sandbox environment, repair operations are performed sequentially according to the operation steps of the candidate fault solutions. After each operation step is executed, the operating indicators in the simulation sandbox environment are collected. The collected operating indicators can then be compared and verified with the preset expected indicator range. If the operating indicators are restored to the expected indicator range after all operation steps are completed, the repair simulation is considered successful.

[0113] By adopting the above technical solution, matching candidate repair solutions are automatically retrieved from the historical fault knowledge base based on the identified target anomaly event. Then, an executable solution is generated and the repair is completed. This avoids the problem of the separation between diagnosis and repair in related technologies. The analysis conclusions can be directly converted into operable instructions, which can shorten the time from problem discovery to repair implementation. Furthermore, by conducting repair simulations in the repair sandbox, the risk of secondary damage to production operations caused by defects in the repair solution itself or unpredictable side effects in complex environments can be reduced, ensuring the robust operation of the automated operation and maintenance process.

[0114] Figure 2 This is a schematic diagram of the structure of a fault handling device provided in an embodiment of this application, as shown below. Figure 2 As shown, the fault handling device 200 includes an acquisition module 201, a sensing module 202, an inspection module 203, and a determination module 204.

[0115] In this embodiment, The acquisition module 201 is used to analyze the fault information of the service topology network to obtain the fault service and fault time. The perception module 202 is used to perform fault context perception on the faulty service based on the call dependency relationship and fault time between services in the service topology network, and to obtain the associated services of the faulty service in the service topology network, as well as the running status data of the associated services during the fault time. The inspection module 203 is used to perform statistical causal inspection on abnormal events in the running status data according to the call dependency relationship to obtain the target abnormal event, wherein the target abnormal event is an event that does not have time statistical causality among multiple abnormal events; Module 204 is used to determine the fault solution matching the target abnormal event.

[0116] The description of the above device-side embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0117] The following describes the application of the fault handling method provided in the embodiments of this application in a real-world scenario, mainly involving an intelligent agent-based cluster fault intelligent diagnosis and automatic repair system.

[0118] Currently, handling cluster failures mainly relies on manual intervention and experience-based judgment by operations and maintenance personnel. Typically, it is triggered by user feedback or alarms from the monitoring system. Subsequently, based on a limited problem description, operations and maintenance personnel can manually cross-reference and search for data fragments related to the failure in multiple independent observability systems (such as logs, metrics, and tracing systems). By screening log error information, analyzing the time-series fluctuations of monitoring metrics, and tracing the distributed call chain, operations and maintenance personnel can locate the abnormal service components based on their experience.

[0119] Based on this, determining the root cause of the fault relies on individual experience and intuition to deduce repair actions, such as restarting service instances, adjusting operating parameters, or expanding computing resources. Afterwards, operations personnel observe the system status to verify the effectiveness of the repairs. If the problem persists, the above operations are repeated. This process is not only entirely manual, time-consuming and labor-intensive, but also extremely difficult and time-consuming when dealing with faults in distributed systems involving multiple services and complex data relationships. Moreover, this approach still falls under the manual mode, and the problems of difficult fault location and slow response remain.

[0120] Moreover, this approach has several drawbacks. First, at the data correlation level, when a failure occurs, operations personnel need to manually retrieve and correlate information across data sources. In complex distributed scenarios, this process is extremely time-consuming and prone to missing crucial data clues with causal relationships across services. Second, the entire analysis and diagnosis process heavily relies on the personal experience and intuitive judgment of operations personnel, making it difficult to pinpoint the root cause. Furthermore, after proposing hypothetical root causes and remedial solutions, the lack of secure verification mechanisms forces trial-and-error operations in the production environment, posing a risk of amplifying the failure due to misjudgment. Ultimately, this results in a lengthy end-to-end response time from failure occurrence to eventual recovery, impacting business continuity.

[0121] Therefore, this application proposes an intelligent cluster fault diagnosis and automatic repair system based on intelligent agents, which can realize the automatic association and analysis of fault-related data, automatically infer the root cause of the fault based on historical experience and current data, and automatically verify and repair the fault under safe and controllable conditions, thereby significantly shortening the fault handling time and reducing the dependence on human experience.

[0122] Figure 3 This is a flowchart of another fault handling method provided in the embodiments of this application, such as... Figure 3 As shown, this method can be executed by a cluster agent, which can be a lightweight agent running on the cluster management node or each service node, or a centralized automated operation and maintenance system deployed on a standalone server or cloud platform. The method may include the following steps.

[0123] Step S301: Automatically search and associate monitoring data corresponding to the fault described by the user based on the user description.

[0124] In this step, the intelligent agent's problem understanding and information extraction module receives fault descriptions from users in natural language, such as slow order service response or GPU tasks stuck in a pending state. The module then extracts key elements from these descriptions, including but not limited to the faulty object (service name, task identifier, node, etc.), fault type (slow response, failure, stuck, etc.), time range, and scope of impact. The scope of impact can be an enumerated value type, including but not limited to four levels: single instance, all instances of a single service, cross-service, and the entire cluster. During extraction, the number of affected services and users can be identified first from the user description. If these cannot be identified, the default level is single service, which will be confirmed in subsequent follow-up questions.

[0125] In one possible implementation, when the problem understanding and information extraction module extracts key elements from the description, if the extracted information is incomplete, it can proactively ask the user for more details. These key elements may include, but are not limited to, the fault object (i.e., the fault service in the above embodiment), fault type, time range, scope of impact, and current status description. The trigger condition for follow-up questions can be that a missing item exists among the key elements. The triggering priority can be in the order of fault object, time range, and fault type. In each round of questioning, at most one element can be added to avoid asking multiple questions at once.

[0126] During the questioning process, the dialogue history can be persistently recorded through a session-level fault context object, which can be updated using a three-state update mechanism. Specifically, it can first overwrite existing fields when new information contradicts their semantics, directly replacing the corresponding fields; secondly, it can supplement missing fields in the fault context object with new information; and thirdly, it can correct certain states when the user explicitly states that it is not X but Y, marking X as excluded before filling in Y. The complete state of the fault context object is persisted at the end of each dialogue round, ensuring that the context can be restored if the user interrupts subsequent conversations.

[0127] Step S302: Based on the temporal causal propagation analysis of the dynamic service topology graph, machine inference of the distributed fault propagation path is realized to locate the fault origin node.

[0128] In some embodiments, the intelligent agent can use the internally configured automatic data search and association module to search for key elements collected by the problem understanding and information extraction module. The faulty object and time range in the key elements can be used as the main search anchor points, while the service topology map is used to expand the search scope to upstream and downstream dependent services to ensure that no causal relationship data is missed.

[0129] Specifically, the automatic data search and association module can maintain a dynamic Service Signal Map (SSM). This map stores the mapping relationship between each service name and its corresponding field in each observation system. For example, for the service name "order-service," its corresponding label in the log system is "app=order-service," in the metrics system it is "job=order-service," and in the tracking system it is "service.name=order-service." The existence of the service signal map allows the agent to automatically and accurately route the service name described by the user to the correct query field in each observation system without manually maintaining these mapping configurations.

[0130] Then, the intelligent agent can automatically search for relevant logs in the log system based on the fault object and time range. Among them, logs can be assigned weights according to their level (for example, error log weight = 3, warning log weight = 1, information log weight = 0). High-weight logs that occur within ±5 minutes of the fault time window are given priority for subsequent root cause analysis.

[0131] After obtaining the relevant logs, the agent can construct a dynamic service topology graph reflecting the current true state of the system. This dynamic service topology graph can be represented by services as nodes and directed edges representing the call dependencies between services. Subsequently, the agent maps various abnormal events (e.g., high-weighted error logs, resource metrics exceeding the dynamic baseline, and significantly slowed call chain spans) collected and filtered by the previous modules to the corresponding service nodes in the topology graph, and marks each abnormal event with its timestamp. Thus, a dynamic fault situation map can be obtained to characterize when, where, and what kind of anomaly occurred.

[0132] After obtaining the dynamic fault situation map, the agent can traverse the topology graph and automatically identify all pairs of adjacent service nodes with direct call dependencies that are marked as abnormal within the current time window, forming multiple candidate fault propagation paths to be verified. For each candidate path, the agent extracts the time series data of the abnormal indicators corresponding to upstream service A and downstream service B, and performs statistical analysis on these two sets of time series using methods such as Galanz causality tests. This statistical method can mathematically determine whether the historical data of A contains statistical information that can be used to significantly predict future changes in B. If the test result meets the preset conditions (e.g., the statistical probability value is not greater than 0.05), the agent determines that the abnormality of A is the statistical cause of the abnormality of B, that is, confirms the causal relationship on the edge, and marks the abnormal event of A as the parent node event of the abnormal event of B.

[0133] After statistically verifying all candidate paths, the agent obtains a network of anomalous events connected by statistically validated causal relationships. It can then trace back along the verified causal chain to ultimately locate the anomalous node that lacks any statistically validated upstream parent node events. The anomalous event corresponding to this node is the initial source triggering the entire fault propagation chain, i.e., the fault origin node (root cause).

[0134] For example, statistical verification revealed that database connection pool exhaustion caused inventory service query timeouts, which in turn led to slow order service response. This accurately pinpointed the database as the root cause, rather than focusing on the order service as the final performance issue.

[0135] Step S303: Query the historical fault knowledge base and obtain a repair plan based on historical repair plans of historical cases that match the target abnormal event.

[0136] In some embodiments, after completing the root cause localization of the fault and obtaining the target abnormal event, the agent can intelligently match similar historical fault cases from the knowledge base and refer to their successful handling experience to generate a repair plan for the current scenario.

[0137] Specifically, the agent can extract features of the current fault (such as root cause type, affected services, and abnormal indicators) and encode them into structured feature vectors. Then, it can retrieve the most relevant historical cases from the knowledge base by calculating vector similarity. These historical cases can record validated and effective repair operations.

[0138] The intelligent agent can use historical repair operations from past cases as a basic blueprint. Specifically, it can extract standard operation step templates from matched historical cases, and then automatically populate the variable parameters in the templates by combining the specific context of the current fault (such as the specific fault service name, the current resource status and topology of the cluster), thereby obtaining an executable repair plan that matches the current situation.

[0139] Step S304: Perform a pre-repair plan in the simulation environment using the repair sandbox.

[0140] In some embodiments, a full operational rehearsal can be performed in a simulation sandbox isolated from the production environment before any repair operations are submitted to the real production cluster.

[0141] Specifically, the intelligent agent can import real-time copies of user requests into the sandbox through the service mesh's traffic mirroring function to simulate real load; at the same time, through the snapshot capability of the underlying infrastructure, it records and reproduces the precise state of the production cluster before execution, which serves as the initial starting point for the sandbox.

[0142] In this simulated sandbox environment, the agent can execute all operational instructions step by step according to the predetermined repair plan. During execution, the agent can continuously monitor and record changes in core metrics directly related to business recovery (such as service response time and error rate). After the simulation, the agent can compare the results of these metrics in the sandbox with the preset normal baseline range. If all key metrics have recovered to a healthy state, the simulation is considered successful.

[0143] Step S305: If the pre-run is successful, dynamically and adaptively adjust the repair operation based on intermediate state indicators during the actual execution process.

[0144] In one possible implementation, the intelligent agent can sample key business metrics directly related to fault recovery in real time while executing the repair plan in a real production environment. These key business metrics can be compared and verified in real time with the expected intermediate states predefined for each operational step in the repair plan.

[0145] If the metrics after the current step meet expectations, the step is considered successful, and subsequent steps in the plan will automatically continue. Conversely, if the metrics deteriorate unexpectedly or fail to achieve the expected improvement, the entire repair plan can be immediately and automatically paused. Subsequently, a rapid reassessment can be performed based on the latest cluster state, and the optimal solution in the current state can be selected from a pre-prepared list of alternative solutions. To enable intervention and rollback at each step, the agent can use incremental snapshot technology before executing each step, backing up only the specific resources to be modified by that step. When a rollback is needed, the state before each step can be precisely restored in reverse order of execution, minimizing the impact of the rollback operation. This avoids the overhead and risks of a full recovery of the entire cluster or unrelated services.

[0146] Step S306: Establish a fault knowledge base.

[0147] In some embodiments, the following specific examples can be used to illustrate the use cases of this application.

[0148] Example 1: Automatically diagnose GPU task scheduling problems.

[0149] When a user reports "My submitted training task has been stuck in the waiting state for 30 minutes", the AI ​​will first extract key information (task status, waiting time) and actively ask for the task ID to obtain the specific context.

[0150] After the user provides a task identifier (for example, the task identifier can be represented as train-job-12345), the agent will automatically perform multi-source data retrieval: query the scheduling log and find the record of "insufficient GPU resources"; check the cluster monitoring indicators to confirm that the total GPU utilization has reached 98%; and list all currently running tasks that are occupying GPUs.

[0151] Based on this correlation data, the agent can perform root cause analysis and provide clear feedback to the user: "The direct reason why the task cannot start is that the cluster GPU resources are saturated. The current utilization rate is 98%, your task requires 4 GPUs, but there are no available resources at the moment." The agent can then provide structured solution options, such as suggesting automatic retrying after a specified task has finished, pausing low-priority tasks in emergencies, or adjusting the number of GPUs required for a task. If the user chooses to wait, the agent can be set to automatically monitor and alert the user when resources are released, thus achieving closed-loop processing.

[0152] Example 2: Automatically repairing service memory overflow issues.

[0153] The intelligent agent can proactively monitor the cluster status and trigger an automated diagnostic process when it detects that a service is frequently restarting abnormally.

[0154] Specifically, the intelligent agent can automatically collect and correlate the service's error logs, memory usage metrics, and heap memory dump analysis results. Based on this data, the intelligent agent can determine the root cause: the service is experiencing a memory leak due to the cache not having an expiration setting.

[0155] To address this root cause, the agent can generate a tiered remediation plan: short-term measures include temporarily increasing service memory limits by modifying configurations and dynamically injecting cache capacity and expiration policy configurations. With the appropriate user authorization, the agent can automatically execute these configuration changes and service restart operations.

[0156] Example 3: Assisting in diagnosing network problems between services.

[0157] When a user provides a vague description of "frequent timeouts when calling service B from service A," the agent can first analyze the call chain between the two services in the distributed tracing system, statistically analyze the latency distribution, and find that the P99 latency is as high as 5 seconds, far exceeding the normal baseline (100ms). Next, the agent can correlate and query network layer metrics to pinpoint an abnormal network packet loss rate (e.g., 10%) between the physical nodes where service A and service B reside, and check the network configuration rules that may be affected. Through correlation analysis of multi-source data (tracing, metrics, configuration), the agent can determine the root cause: "The call timeout stems from degraded network quality between nodes, with a severe packet loss rate, possibly caused by underlying network device failure or firewall policies." Based on this analysis, the agent can provide actionable solutions: the short-term recommendation is to migrate the containers for service B to a node with normal network connectivity via the scheduler to quickly restore service; the long-term recommendation is to generate a support ticket with detailed evidence to drive the network team to thoroughly overhaul the physical infrastructure. With authorization, the agent can directly execute container migration operations to achieve rapid fault repair.

[0158] The technical solution provided in this application can automate the search, association, and analysis of monitoring data, reducing the time required for manual troubleshooting and location in related technologies, thereby shortening the average time to resolve faults and effectively ensuring business continuity. Secondly, through a continuously accumulating fault knowledge base, it can reduce over-reliance on the personal experience of specific operation and maintenance experts, enabling standardized reuse of experience.

[0159] Furthermore, by integrating multi-source data such as logs, metrics, and traces, and performing causal inference based on statistics and topology, the root causes of complex fault chains can be better identified compared to manual single-point analysis, thus improving the accuracy and reliability of diagnosis. Based on this, with user authorization, standardized repair operations can be automatically executed or precise decision-making suggestions can be provided, enabling rapid self-healing and closed-loop management of common faults.

[0160] Meanwhile, through fine-grained access control and interventionist process nodes, users can flexibly balance between automatic processing and manual review, achieving an effective balance between intelligent operation and maintenance and risk control.

[0161] In some embodiments, for fault diagnosis, a rule-based expert system can be used to predefine various fault modes and corresponding diagnostic rules; for data association, full-text search and time-series data association methods can be used; and for automatic repair, it can be based on predefined repair scripts.

[0162] This application provides a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps in the method provided in this application.

[0163] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in this application.

[0164] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements the steps in the method provided in this application.

[0165] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0166] Figure 4 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application, such as... Figure 4As shown, the hardware entity of the computer device 400 includes a processor 401 and a memory 402, wherein the memory 402 stores a computer program that can run on the processor 401, and the processor 401 executes the program to implement the steps in the method of any of the above embodiments.

[0167] The memory 402 stores computer programs that can run on the processor. The memory 402 is configured to store instructions and applications that can be executed by the processor 401. It can also cache data to be processed or already processed by the processor 401 and various modules in the computer device 400 (e.g., image data, audio data, voice communication data and video communication data). It can be implemented by flash memory or random access memory (RAM).

[0168] The processor 401 executes the steps of the above-described task execution method when executing the program. The processor 401 typically controls the overall operation of the computer device 400.

[0169] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0170] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0171] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0172] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0174] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, in the various embodiments of this application, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0176] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0177] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A failure handling method characterized by, The method includes: Analyze the fault information of the service topology network to obtain the fault service and fault time; Based on the call dependencies between services in the service topology network and the fault time, the fault service is subjected to fault context awareness to obtain the associated services of the fault service in the service topology network, as well as the running status data of the associated services during the fault time. Based on the call dependency relationship, a statistical causal test is performed on the abnormal events existing in the running status data to obtain the target abnormal event, wherein the target abnormal event is an event among the multiple abnormal events that does not have a temporal statistical causal relationship; Determine the fault solution that matches the target abnormal event.

2. The method of claim 1, wherein, The step of performing fault context awareness on the faulty service based on the call dependencies between services in the service topology network and the fault time, to obtain the associated services of the faulty service in the service topology network, and the running status data of the associated services during the fault time, includes: Based on a preset field mapping relationship, a query field that has a mapping relationship with the service identifier of the fault service is determined; wherein, the field mapping relationship includes the correspondence between the service identifier and the corresponding query field; the query field includes a tracking identifier field and a timestamp field; Based on the tracking identifier field and the call dependency relationship, determine the associated services of the faulty service in the service topology network; Based on the timestamp field, the running status data of the associated service that falls within a preset time period corresponding to the fault time is taken as the running status data of the associated service during the fault time.

3. The method of claim 2, wherein, The step of using the timestamp field to select the operational status data of the associated service within a preset time period corresponding to the fault time as the operational status data of the associated service during the fault time includes: Based on the timestamp field, the running status data of the associated service that falls within the preset time period corresponding to the fault time is taken as the candidate running status data corresponding to the fault service. For different preset dimensions, each candidate data in the candidate running status data is scored to obtain the correlation score of each candidate data in different preset dimensions; Based on the relevance scores of each candidate data point across different preset dimensions, a target score is obtained for each candidate data point. The candidate running status data in which the target score is not less than a preset score threshold is taken as the running status data.

4. The method of claim 3, wherein, The preset dimensions include a time correlation dimension, which characterizes the proximity of the timestamp of the data to the time of the fault; an entity correlation dimension, which characterizes the degree of association between the service to which the data belongs and the faulty service in the service topology network; and a signal strength correlation dimension, which characterizes the degree of data anomaly.

5. The method according to claim 2, characterized in that, The step of determining the associated services of the faulty service in the service topology network based on the tracing identifier field and the call dependency relationship includes: Based on the tracking identifier field, determine all service identifiers on the request link to which the faulty service belongs; Based on the call dependencies in the service topology network, the services among all service identifiers that have call dependencies on the faulty service are identified as the associated services.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The abnormal event triggered by the fault service is determined based on the statistical characteristics of the operational status data; The step of performing statistical causal checks on abnormal events existing in the runtime status data based on the call dependencies to obtain target abnormal events includes: The abnormal time sequence corresponding to the abnormal event is determined based on the timestamp of the abnormal event; wherein, the abnormal time sequence includes multiple entries arranged in chronological order, and each entry corresponds to the abnormal event at a different time; Based on the aforementioned call dependencies, a statistical causal test is performed on the abnormal time series to obtain the target abnormal event.

7. The method according to claim 6, characterized in that, The step of performing statistical causal testing on the abnormal time series based on the call dependency relationship to obtain the target abnormal event includes: Based on the invocation dependency, candidate causal paths in the service topology network are determined; wherein, the candidate causal paths include adjacent abnormal events with the invocation dependency. The abnormal time series is verified based on the candidate causal path to determine the parent node event adjacent to each abnormal event in the abnormal time series; wherein, the parent node event is the event that triggers the adjacent abnormal event; The abnormal event in the abnormal time series that does not contain the parent node event is taken as the target abnormal event.

8. The method according to claim 7, characterized in that, The step of verifying the abnormal time series based on the candidate causal path and determining the parent node event adjacent to each abnormal event in the abnormal time series includes: Based on each pair of adjacent abnormal events with the call dependency relationship in the candidate causal path, determine the candidate time series corresponding to the candidate causal path; Based on the time statistical causal data corresponding to the candidate time series, a statistical causal test is performed on the abnormal time series to determine the parent node event corresponding to each abnormal event in the abnormal time series.

9. The method according to claim 6, characterized in that, The step of determining the abnormal event triggered by the fault service based on the statistical characteristics of the operating status data includes: If abnormal data matching a preset error pattern is identified in the running status data, then a log abnormal event is determined to be triggered. Each data point in the running status data is compared with the dynamic baseline threshold. If there is a data point that is not less than the dynamic baseline threshold and the duration is not less than the preset duration, then a resource abnormal event is determined to be triggered. Each data point in the operational status data is compared with the historical performance baseline. If any data point is not less than the historical performance baseline, then a performance anomaly event is identified. At least one of the log anomaly event, the resource anomaly event, and the performance anomaly event is identified as the anomaly event.

10. A fault handling device, characterized in that, The device includes: The acquisition module is used to analyze fault information of the service topology network to obtain fault service and fault time; The perception module is used to perform fault context perception on the faulty service based on the call dependency relationship between services in the service topology network and the fault time, and to obtain the associated services of the faulty service in the service topology network, as well as the running status data of the associated services during the fault time. The verification module is used to perform statistical causal verification on the abnormal events existing in the running status data according to the call dependency relationship, and obtain the target abnormal event, wherein the target abnormal event is an event that does not have temporal statistical causality among the multiple abnormal events; The determination module is used to determine the fault solution that matches the target abnormal event.

11. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, characterized in that, When the computer program is read and executed by a computer, it implements the method described in any one of claims 1 to 9.

Images