An AI-driven DevOps adaptive fault repair method and system

Abstract

This invention discloses an AI-driven DevOps adaptive fault repair method and system. The method specifically includes: extracting semantic features from code and logs generated during system runtime to generate semantic code fingerprints; using the semantic code fingerprints as initial node features to construct an error propagation graph based on the software system's call relationships; utilizing graph convolutional networks to perform high-order topological feature learning on the error propagation graph to accurately deduplicate error information and locate the root cause of the fault; generating elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters based on the deduplicated error information and the system's real-time load pattern; and generating corresponding repair plans and matching the optimal responsible party based on the located root cause of the fault and the real-time status characteristics of team members. This invention automates the entire DevOps fault repair process, significantly reducing the mean time to recovery and improving system stability and operational efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AI-driven DevOps adaptive fault repair method and system. Background Technology

[0002] DevOps is a combination of Development and Operations. It's a set of concepts, cultures, and practices designed to break down the barriers between development and operations, enabling efficient collaboration and automation across the entire software development, testing, deployment, and operations process. In DevOps practices, system stability and operational efficiency are paramount, and Mean Time To Repair (MTTR) is a key metric for measuring both.

[0003] First, in the traditional model, relying solely on manual log analysis is not only difficult to manage due to the sheer volume and complexity of logs in complex systems, but the logs themselves may also be incomplete or inaccurate. For example, some critical error messages may be missed due to imperfect logging mechanisms, or the logs may contain a large amount of redundant information that interferes with the judgment of developers and operations personnel, further increasing the difficulty and time-consuming nature of fault localization. While existing automated tools can perform preliminary filtering and analysis of logs using preset rules, these rules often have limitations. Fault manifestations vary significantly across different systems and scenarios, and preset rules cannot comprehensively cover all fault modes. For some novel or complex faults, they still cannot effectively assist in localization, resulting in limited improvements in the accuracy and efficiency of fault localization.

[0004] Secondly, in the traditional model, developers manually write fixes, which is not only inefficient and prone to introducing new errors, but also requires a high level of experience and skill. Different developers have varying levels of understanding of system architecture and business logic, resulting in inconsistent quality of the fixes they write. Furthermore, for some complex faults, even experienced developers may need to spend a significant amount of time debugging and optimizing. Existing automation tools cannot provide effective automation support in the fix phase, leaving the fix process still reliant on manual intervention and hindering significant improvements in overall operational efficiency. For example, some tools may only provide simple error messages but cannot generate specific fixes or solutions, requiring developers to write code from scratch.

[0005] Furthermore, in the traditional model, after developers write the fix code, they need to conduct rigorous testing and verification to ensure that the fix is ​​effective and will not negatively impact other parts of the system. However, the testing and verification process is often complex, requiring the design of comprehensive test cases to cover various possible scenarios and boundary conditions. For large and complex systems, the number of test cases can be enormous, making the testing process time-consuming and labor-intensive. Existing automated tools also have limited support for testing and verification, and may only be able to perform some basic tests, unable to fully assess the impact of the fix. For example, some tools may only check the syntax correctness of the code, but cannot detect whether the fix will introduce new performance issues or security vulnerabilities.

[0006] Furthermore, communication and collaboration between developers and operations personnel are costly during troubleshooting. Due to their different responsibilities, their understanding and handling of system faults differ, requiring frequent communication to coordinate their work. This not only consumes a significant amount of time but can also lead to misunderstandings due to poor communication, further impacting troubleshooting efficiency. In the traditional model, even if developers and operations personnel can communicate and collaborate effectively, the lack of a unified troubleshooting process and tools makes it difficult to significantly improve team collaboration efficiency.

[0007] While some existing automation tools have assisted in fault location to some extent—for example, some tools can perform preliminary filtering and analysis of logs through preset rules to help narrow down the scope of the fault—these tools still cannot provide effective automation support in the remediation phase. The remediation process still heavily relies on manual writing and testing of remediation code, resulting in limited improvement in overall operational efficiency. Some tools may only provide fault information display functions but cannot achieve the allocation and tracking of fault handling tasks. Summary of the Invention

[0008] The purpose of this invention is to provide an AI-driven DevOps adaptive fault repair method and system, which automates the entire DevOps fault repair process, significantly reduces the mean time to recovery (MTTR), and improves system stability and operational efficiency, thereby solving at least one of the aforementioned problems in the prior art.

[0009] In a first aspect, the present invention provides an AI-driven DevOps adaptive fault repair method, the method specifically comprising: Semantic features are extracted from the code and logs generated during system runtime using a self-supervised contrastive learning model to generate semantic code fingerprints that uniquely identify error types. Using semantic code fingerprints as initial node features, an error propagation graph is constructed based on the call relationships of the software system; Graph convolutional networks are used to learn high-order topological features of the error propagation graph in order to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the failure. Based on the deduplicated error messages and the real-time system load pattern, a reinforcement learning-based resource prediction model is invoked to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters. Based on the location of the root cause of the failure and the real-time status characteristics of team members, a heterogeneous multi-agent engine containing a repair agent and a scheduling agent is invoked to generate a corresponding repair plan and match the optimal responsible person.

[0010] Secondly, the present invention provides an AI-driven DevOps adaptive fault repair system, the system specifically comprising: The code fingerprint module is used to extract semantic features from the code and logs generated during system runtime using a self-supervised contrastive learning model, and generate a semantic code fingerprint that uniquely identifies the error type. The error identification module is used to construct an error propagation graph based on the call relationships of the software system, using semantic code fingerprints as the initial features of nodes. The deduplication and localization module is used to learn high-order topological features of the error propagation graph using graph convolutional networks, so as to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the fault. The resource optimization module is used to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters by calling a resource prediction model based on reinforcement learning, based on the deduplicated error information and the real-time system load mode. The multi-agent module is used to generate corresponding repair solutions and match the optimal responsible party based on the location of the root cause of the failure and the real-time status characteristics of team members by calling a heterogeneous multi-agent engine containing repair agents and scheduling agents.

[0011] Thirdly, the present invention provides a computer device, including: a memory and a processor, and a computer program stored in the memory, wherein when the computer program is executed on the processor, it implements the AI-driven DevOps adaptive fault repair method as described in any of the above methods.

[0012] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the AI-driven DevOps adaptive fault repair method as described in any of the above methods.

[0013] Compared with the prior art, the present invention has at least one of the following technical effects: 1. This invention automates the entire DevOps fault repair process, significantly reduces the mean time to recovery (MTTR), and improves system stability and operational efficiency.

[0014] 2. This invention can quickly and accurately identify error types, providing a strong basis for subsequent fault location, effectively reducing the time spent on manual log analysis, and improving fault location efficiency.

[0015] 3. This invention can accurately remove duplicate and similar error information and accurately locate the root cause of the fault, avoiding misjudgment and omission that may occur when manually locating faults.

[0016] 4. This invention can rationally allocate resources according to the actual needs of the system, improve the system resource utilization rate, and ensure the stable operation of the system.

[0017] 5. This invention enables the automated generation of repair solutions and the intelligent matching of responsible persons, reducing the workload of manually writing repair code, lowering communication costs, and improving the efficiency and quality of the entire fault repair process.

[0018] 6. This invention utilizes a self-supervised contrastive learning model to accurately extract semantic features from code and logs, quickly generating a semantic code fingerprint that uniquely identifies the error type, effectively assisting in subsequent fault localization.

[0019] 7. This invention constructs an error propagation graph based on software call relationships and semantic code fingerprints, clearly presenting the fault propagation path and providing an intuitive basis for accurately locating the root cause of the fault.

[0020] 8. This invention improves the accuracy of fault location by querying and filtering call chain instances and combining them with heterogeneous information network graphs to accurately extract error propagation graphs containing faulty nodes and their upstream and downstream nodes.

[0021] 9. This invention constructs a detailed and accurate error propagation graph by identifying abnormal spans, deduplicating and merging nodes, performing breadth-first traversal, and adding dynamic attributes, thus comprehensively reflecting the fault propagation situation.

[0022] 10. This invention learns high-order topological features of the error propagation graph through graph convolutional networks, thereby achieving accurate deduplication of duplicate and similar errors and accurate identification of the root cause of the fault.

[0023] 11. This invention generates a feature importance ranking list by calculating the Shapley value of each word in the fault root cause feature vector, providing interpretable basis for understanding the decision-making of the repair agent and enhancing the credibility of the solution. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating an AI-driven DevOps adaptive fault repair method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an AI-driven DevOps adaptive fault repair system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] In this application embodiment, the entity executing the process includes a terminal device. This terminal device includes, but is not limited to, devices capable of executing the methods disclosed in this application, such as servers, computers, smartphones, and tablets. Figure 1 A flowchart illustrating an AI-driven DevOps adaptive fault repair method according to an embodiment of the present invention is shown below: S101 uses a self-supervised contrastive learning model to extract semantic features from the code and logs generated during system runtime, generating a semantic code fingerprint to uniquely identify error types. S102, using semantic code fingerprints as initial features of nodes, constructs an error propagation graph based on the call relationships of the software system; S103 utilizes graph convolutional networks to learn high-order topological features of the error propagation graph in order to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the fault. S104: Based on the deduplicated error information and the real-time system load pattern, call the reinforcement learning-based resource prediction model to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters. S105, based on the location of the root cause of the failure and the real-time status characteristics of team members, calls the heterogeneous multi-agent engine containing the repair agent and the scheduling agent to generate the corresponding repair plan and match the optimal person in charge.

[0033] In this embodiment, a large amount of code and log information is continuously generated during system operation. A self-supervised contrastive learning model is used to extract semantic features from this code and logs. By learning from a large amount of unlabeled data, the self-supervised contrastive learning model can automatically capture the inherent semantic relationships within the data. Specifically, the code and log data generated during system operation are input into the self-supervised contrastive learning model, which performs in-depth analysis on this data and extracts representative semantic features. These semantic features accurately reflect the meaning expressed by the code and logs and the relationships between them. Based on the extracted semantic features, a semantic code fingerprint is generated to uniquely identify the error type. Each semantic code fingerprint corresponds to a specific error type, accurately distinguishing different error situations. For example, when a memory leak error occurs in the system, the semantic features of the relevant code and logs are extracted using the self-supervised contrastive learning model to generate a specific semantic code fingerprint that uniquely identifies the memory leak error type.

[0034] After generating semantic code fingerprints, these fingerprints are used as the initial features of the nodes. Simultaneously, considering the call relationships within the software system, which exhibit complex relationships between components reflecting the logical flow of the system's runtime, an error propagation graph is constructed. In this graph, nodes represent different error types (identified by semantic code fingerprints), and edges represent the propagation paths of errors within the system. For example, if component A calls component B, and component B encounters an error, that error may propagate to component A. In this way, the error propagation graph would contain an edge pointing from the node representing the error in component B to the node representing the error in component A. This method constructs an error propagation graph that visually illustrates the spread of errors within the system.

[0035] Graph convolutional networks (GCNNs) are used to learn high-order topological features from a constructed error propagation graph. GCNNs can process graph-structured data, learning the features of nodes and edges to uncover hidden high-order topological information. Error propagation graphs may contain numerous duplicate and similar errors, which can interfere with root cause localization. By learning from the error propagation graph, GCNNs can identify these duplicates and perform precise deduplication. Simultaneously, based on the graph's topology and node features, the propagation path and impact range of the error are analyzed, thus pinpointing the root cause. For example, analyzing the error propagation graph using a GCNN reveals that an error originates from a low-level component, propagates through multiple intermediate components, and ultimately affects the upper-level application; therefore, the low-level component can be identified as the root cause of the failure.

[0036] After identifying the root cause of the fault, the deduplicated error information and the system's real-time load pattern are obtained. The real-time load pattern reflects the system's current operating status, including metrics such as CPU utilization, memory usage, and network traffic. This information is then input into a reinforcement learning-based resource prediction model. Through continuous learning and optimization via interaction with the environment, the reinforcement learning model can predict the system's resource requirements over a future period based on the system's real-time status and historical data. Based on the prediction results of the reinforcement learning model, elastic scaling strategies and resource configuration optimization schemes for the cloud-native cluster are dynamically generated. For example, if a significant increase in system load is predicted in the future, the model will generate a scaling strategy that increases the number of nodes in the cloud-native cluster and optimize the resource configuration of each node to ensure stable system operation.

[0037] Based on the identified root cause of the fault and the real-time status characteristics of team members, a heterogeneous multi-agent engine containing a repair agent and a scheduling agent is invoked. The repair agent possesses rich knowledge and experience in fault repair and can generate multiple possible repair solutions based on the root cause. The scheduling agent is responsible for considering the real-time status characteristics of team members, such as their current tasks, skill levels, and workload. By analyzing this information, the scheduling agent selects the most suitable solution from the multiple repair options generated by the repair agent and assigns it to the optimal responsible person. For example, if a fault requires a member with specific programming language skills to repair, and team member A happens to possess that skill and has a relatively light workload, the scheduling agent will assign the repair solution to team member A to improve the efficiency and quality of fault repair.

[0038] In some embodiments, step S101 above, which involves extracting semantic features from the code and logs generated during system runtime using a self-supervised contrastive learning model to generate a semantic code fingerprint for uniquely identifying error types, specifically includes: Positive sample pairs are generated by performing semantically preserved data augmentation operations on the original code snippets, and negative sample pairs are generated by sampling code snippets with different error types, thus constructing a training sample set for self-supervised contrastive learning; The training sample set is input into the encoder network, and the encoder network is trained according to the contrastive learning objective function. The contrastive learning objective function is used to bring the positive sample pairs closer and push the negative sample pairs further apart in the vector space, so that the encoder network can distinguish the semantic similarity of the code. The trained encoder network is deployed to an online pipeline. When a new erroneous code fragment is captured, it is fed into the encoder network for forward computation, and a high-dimensional continuous value vector is output as a semantic code fingerprint.

[0039] In this embodiment, when constructing the training sample set for self-supervised contrastive learning, semantically preserving data augmentation operations are first performed on the original code snippets. Semantic-preserving data augmentation aims to transform the code in various ways without changing its original semantics, thereby generating code snippets with the same semantics as the original code but different presentations. These corresponding code snippets constitute positive sample pairs. For example, for a piece of code that implements a specific function, data augmentation can be performed by renaming variables or adjusting the order of code statements (without changing the logic). Through such operations, the number of positive samples can be increased, enabling the model to better learn the semantic features of the code.

[0040] Simultaneously, to help the model distinguish between different error types of code, it is necessary to sample code snippets of different error types to generate negative sample pairs. From historical system failure records or a pre-collected error code library, code snippets with error types different from the current original code snippet are selected and paired to form negative sample pairs. By constructing a training sample set containing both positive and negative sample pairs, a rich and targeted data foundation is provided for the subsequent training of the encoder network, enabling the model to learn the ability to distinguish between different code semantics and error types.

[0041] The constructed training sample set is input into the encoder network. The encoder network is the core of the entire self-supervised contrastive learning model; its role is to convert the input code snippets into high-dimensional continuous-valued vectors, thereby extracting semantic features of the code. During training, the encoder network is optimized according to the contrastive learning objective function. The design philosophy of the contrastive learning objective function is to bring positive sample pairs closer together in the vector space while simultaneously increasing the distance between negative sample pairs. Specifically, when a positive sample pair is input into the encoder network, the resulting two high-dimensional vectors should be as close as possible in the vector space, indicating that they have similar semantics; while the vectors obtained after inputting a negative sample pair should be as far apart as possible to reflect their semantic differences. By continuously adjusting the parameters of the encoder network, the value of the contrastive learning objective function gradually decreases until a preset convergence condition is reached. At this point, the encoder network gains the ability to distinguish the semantic similarity of code. This ability enables the encoder network to accurately extract key semantic features from the code, laying the foundation for subsequent generation of semantic code fingerprints.

[0042] After training the encoder network, it is deployed to an online pipeline. The online pipeline is a crucial step in the system's real-time processing of erroneous code segments. When the system captures a new erroneous code segment, it is immediately fed into the deployed encoder network. Upon receiving the input, the encoder network performs forward computation. Forward computation is the process by which the encoder network uses its learned parameters to extract and transform features from the input erroneous code segment.

[0043] After forward computation, the encoder network outputs a high-dimensional continuous value vector, which is the semantic code fingerprint generated for the erroneous code segment. Because the encoder network has undergone rigorous training, it possesses the ability to distinguish semantic similarities in code, thus the generated semantic code fingerprint uniquely identifies the error type to which the erroneous code segment belongs. For example, different memory leak error code segments, after processing by the encoder network, will generate similar semantic code fingerprints that can be distinguished from other types of errors (such as array out-of-bounds errors). In this way, semantic code fingerprints can be generated quickly and accurately for erroneous code segments that occur during system runtime, providing strong support for subsequent fault localization and repair.

[0044] In some embodiments, step S102 above, which involves using semantic code fingerprints as initial node features and constructing an error propagation graph based on the call relationships of the software system, specifically includes: Extract the call relationships between services, classes, and methods from the runtime data and static code of the software system, and construct a heterogeneous information network graph to describe the dependencies between system components; Semantic fingerprint vectors are used as initial features of nodes in a heterogeneous information network graph; When a system failure event is detected, an error propagation graph containing the failure node and its upstream and downstream nodes is extracted based on the link tracing data within the failure time window and the heterogeneous information network graph.

[0045] In this embodiment, to accurately describe the dependencies between various components in the software system, it is necessary to extract the call relationships between services, classes, and methods from the runtime data and static code of the software system. Runtime data contains interaction information between various components during actual system operation, such as service requests and responses, and the order in which class methods are called. Static code provides the structure and definition information of system components; by analyzing function calls, class inheritance, and other relationships in the code, the dependencies between components can be further clarified.

[0046] After collecting this call relationship information, a heterogeneous information network graph is constructed to describe the dependencies between system components. A heterogeneous information network graph is a network structure capable of representing various types of nodes and edges. In this graph, different types of nodes can represent system components such as services, classes, and methods, while edges represent the call relationships between them. For example, if service A calls service B, then in the heterogeneous information network graph there will be an edge from the node representing service A to the node representing service B. In this way, a comprehensive and accurate heterogeneous information network graph is constructed, laying the foundation for the subsequent construction of an error propagation graph.

[0047] After constructing the heterogeneous information network graph, the semantic fingerprint vectors generated earlier through the self-supervised contrastive learning model are used as the initial features of the nodes in this graph. The semantic fingerprint vector uniquely identifies the error type and contains semantic feature information from the code and logs. Assigning semantic fingerprint vectors to nodes in the heterogeneous information network graph ensures that each node not only represents a system component but also carries error-related semantic information. For example, a node might represent a specific method; when this method encounters an error, its corresponding semantic fingerprint vector can reflect the type and characteristics of that error. Thus, in subsequent error propagation analysis, the nature and propagation of the error can be determined based on the node's semantic fingerprint vector, providing strong support for accurately locating the root cause of the failure.

[0048] When the system detects a fault event, it needs to determine the time range in which the fault occurred, i.e., the fault time window. The fault time window can be reasonably set according to the time point of the fault occurrence and the operating characteristics of the system. For example, a period of time before and after the fault occurrence can be selected as the fault time window.

[0049] After determining the failure time window, link tracing data is used to obtain the component call information of the system during that time period. Link tracing data records the request and response paths between various components in the system, clearly showing the data flow and component interaction process. Combined with the previously constructed heterogeneous information network diagram, information including the failed node and its upstream and downstream nodes is extracted from the link tracing data.

[0050] A faulty node is a system component node that malfunctions within the fault time window. Upstream and downstream nodes are nodes that have a calling relationship with the faulty node; an upstream node is a component that calls the faulty node, and a downstream node is a component that is called by the faulty node. Extracting these nodes and their calling relationships constitutes an error propagation graph. An error propagation graph can visually display the propagation path and impact range of a fault in the system, helping developers and operations personnel quickly locate the root cause of the fault and thus take effective remedial measures.

[0051] Furthermore, the step of extracting an error propagation graph containing the faulty node and its upstream and downstream nodes based on the link tracing data within the fault time window and the heterogeneous information network graph specifically includes: Based on the timestamp and service identifier of the system failure event, query all call chain instances within a preset time window from the distributed tracing system. Each call chain instance includes several spans, and each span is used to record detailed information of remote procedure calls or local method calls. Filter the call chain instances obtained from the query and retain call chain instances whose call status is failed, contain exception logs, or whose response time exceeds the historical threshold. Based on the heterogeneous information network graph, an error propagation graph containing the faulty node and its upstream and downstream nodes is extracted from the preserved call chain instances.

[0052] In this embodiment, when the system detects a fault event, it records the timestamp of the fault event and the identifier of the service involved. The distributed tracing system is a system used to record the call relationships between various components in the system; it can record detailed information about each remote procedure call or local method call. Based on the recorded fault event timestamp and service identifier, all call chain instances within a preset time window are queried from the distributed tracing system.

[0053] The preset time window here is a time period set according to the characteristics of the system failure and actual needs; for example, it can be set to one hour before and after the failure occurs. Each call chain instance is like a "path" that records the order of system component calls, containing several spans. Each span records detailed information about remote procedure calls or local method calls, such as the start time, end time, method name, passed parameters, and returned result. By querying these call chain instances, the complete call situation of the system within the failure time window can be obtained, providing basic data for subsequent analysis of the failure propagation path.

[0054] Since the number of call chain instances retrieved may be large, and not all of them are related to faults, these call chain instances need to be filtered to improve the efficiency and accuracy of subsequent analysis. Call chain instances with a failed call status are retained, as call failure often indicates a system problem and is likely a direct manifestation of a fault. Call chain instances containing exception logs are also retained; exception logs record crucial information when errors occur during system operation and can provide key clues for fault analysis. Additionally, call chain instances with response times exceeding historical thresholds are also retained, as excessively long response times may indicate performance issues during that call or could be an indirect result of a fault.

[0055] By filtering call chain instances, we can identify call chains highly relevant to the fault, reducing the amount of data for subsequent analysis and making the analysis more focused and effective. For example, in an e-commerce system, if, within the fault time window, a call chain instance for a payment service experiences call failures, error logs recording payment interface errors, or response times significantly exceeding normal levels, then these call chain instances will be retained as crucial evidence for fault analysis.

[0056] After filtering the call chain instances, based on the previously constructed heterogeneous information network graph, an error propagation graph containing the faulty node and its upstream and downstream nodes is extracted from the retained call chain instances. The heterogeneous information network graph describes the dependencies between system components, where nodes represent system components such as services, classes, and methods, and edges represent the call relationships between them.

[0057] From the preserved call chain instances, the specific node where the failure occurred, i.e., the faulty node, can be identified. Then, based on the call relationships in the heterogeneous information network graph, the upstream and downstream nodes that have call relationships with the faulty node are found. The upstream node is the component that calls the faulty node, and the downstream node is the component that is called by the faulty node. Extracting these faulty nodes and their upstream and downstream nodes, and preserving the call relationships between them, constitutes the error propagation graph.

[0058] Error propagation diagrams can visually illustrate the propagation path and scope of a fault within a system. For example, in a distributed system, if a node of a database service fails, the error propagation diagram can show the upstream application service nodes that call the database service, as well as the downstream cache service nodes that are called by the database service. This provides a clear understanding of how the fault propagates within the system, offering strong support for subsequently locating the root cause of the fault and developing a remediation plan.

[0059] Furthermore, the step of extracting an error propagation graph containing the faulty node and its upstream and downstream nodes based on the retained call chain instances, according to the heterogeneous information network graph, specifically includes: Identify abnormal spans as candidate fault nodes from the remaining call chain instances; Based on the preserved call chain instances, the call relationships between services, classes and methods are extracted from the heterogeneous information network graph, and nodes with the same service, class and method are deduplicated and merged to construct a call relationship graph; Using candidate fault nodes as seed nodes, a breadth-first traversal algorithm is used to trace upstream to the request origin and downstream to the fault-affected end in the call relationship graph, and a connected subgraph containing the seed node and its upstream and downstream related nodes is extracted as the error propagation graph. Attach dynamic attributes to each node and edge in the error propagation graph, including the number of calls, the number of failures, and the rate of change of response time during the fault.

[0060] In this embodiment, after filtering the call chain instances, a series of call chain instances highly correlated with the fault are obtained. From these retained call chain instances, abnormal spans are identified as candidate fault nodes. Abnormal spans refer to span records where abnormal situations occur during the call process. For example, the call status may show as failed, indicating that the remote procedure call or local method call represented by that span did not complete normally; or the span may contain exception logs, which are usually detailed information recorded when the system encounters errors during operation and can directly reflect the characteristics of the fault; or the span's response time may exceed a historical threshold, and an excessively long response time may mean that the system has encountered performance problems during the call, which is likely caused by a fault. By identifying these abnormal spans, nodes that may cause faults can be initially identified as candidate fault nodes, providing a basis for further analysis of the fault propagation path. For example, in the order processing flow of an e-commerce system, if a call failure occurs in the span corresponding to the payment service in a certain call chain instance, then the node corresponding to this payment service can be identified as a candidate fault node.

[0061] Based on the retained call chain instances, the call relationships between services, classes, and methods are extracted from the heterogeneous information network graph. The heterogeneous information network graph already describes the dependencies between system components; the call chain instances further clarify the actual call situation within the failure time window. During the extraction of call relationships, nodes with the same service, class, and method are deduplicated and merged. This is because in a real system, there may be multiple instances of the same component, but when analyzing fault propagation, the focus is on the call logic relationships between components, not the specific number of instances. Deduplication and merging simplify the structure of the call relationship graph, making it clearer and easier to analyze. For example, if there are multiple instances of the same service in the system, they are merged into a single node when constructing the call relationship graph, retaining only their call relationships with other components. The final constructed call relationship graph can intuitively display the call paths and dependencies of system components within the failure time window.

[0062] Using identified candidate fault nodes as seed nodes, a breadth-first search algorithm is employed to traverse the call relationship graph. Breadth-first search is a search algorithm that starts from the initial node and expands outwards layer by layer, systematically traversing the entire call relationship graph. During the traversal, it traces upstream to the origin of the request, finding the component that called the seed node, and continuing to the beginning of the entire call chain; simultaneously, it extends downstream to the end of the fault's impact, finding the component called by the seed node, until the fault's impact no longer spreads. In this way, a connected subgraph containing the seed node and its upstream and downstream related nodes is extracted; this connected subgraph is the error propagation graph. The error propagation graph clearly shows the propagation path and scope of the fault in the system, helping developers and operations personnel quickly locate the root cause of the fault and the affected components. For example, in the e-commerce system example above, if the payment service is a candidate fault node, the breadth-first search algorithm can find the order service node (upstream) that calls the payment service, and the financial system node (downstream) that is called by the payment service, thus constructing an error propagation graph containing these nodes.

[0063] To gain a more comprehensive understanding of the state changes of each node and edge during fault propagation, dynamic attributes are attached to each node and edge in the error propagation graph. These dynamic attributes include the number of calls, the number of failures, and the rate of change of response time during the fault. The number of calls reflects how frequently the node or edge is called within the fault time window; a higher number of calls indicates a greater importance of the component in the system, and may also mean a greater impact of the fault on it. The number of failures directly reflects the frequency of problems encountered by the node or edge during the fault; a higher number of failures indicates a stronger correlation between the component and the fault. The rate of change of response time refers to the magnitude of the change in the response time of the node or edge during the fault compared to the response time under normal conditions; a larger rate of change of response time indicates a more significant performance degradation of the component during the fault. By attaching these dynamic attributes, more detailed and accurate information can be provided for fault analysis, helping developers and operations personnel better understand the nature and impact of the fault, thereby developing more effective remediation plans. For example, in the error propagation graph, if a node has a high number of calls, a high number of failures, and a large rate of change of response time during the fault, then this node can be given special attention to analyze the cause of its fault and implement remediation.

[0064] In some embodiments, step S103 above, which involves using a graph convolutional network to perform high-order topological feature learning on the error propagation graph to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the fault, specifically includes: The error propagation graph and its initial node features are input into a graph convolutional network. Through multi-layer neighbor node feature aggregation, the feature representation of each node is updated to obtain a node embedding vector that integrates semantic and topological information. Perform graph pooling on the updated node embedding vectors to obtain a graph-level vector representation of the error propagation graph; The similarity between the graph-level vector representation and the historical graph-level vector representation is compared. If the similarity exceeds a preset similarity threshold, it is determined to be a duplicate fault. By analyzing the contribution of each node in the graph convolutional network to the graph-level vector representation, the node with the highest importance in error propagation is identified, and the code location corresponding to that node is output as the root cause of the fault.

[0065] In this embodiment, the constructed error propagation graph and the initial features of each node are input into a graph convolutional network (GCNN). GCNNs possess powerful feature extraction capabilities, considering both the topological relationships between nodes and the node's own feature information. Within the GCNN, node features are updated through multi-layer neighbor node feature aggregation. Specifically, for each node in the error propagation graph, the GCNN collects feature information from its neighboring nodes and fuses and aggregates these neighboring node features with the node's own initial features. This process iterates through multiple layers, with each layer further expanding the range of neighboring nodes, thus incorporating broader topological information into the node features. After multi-layer neighbor node feature aggregation, the feature representation of each node is updated, resulting in a node embedding vector that integrates semantic and topological information. These node embedding vectors not only contain the node's own semantic features but also reflect the node's topological position and relationships within the entire error propagation graph, providing more comprehensive and accurate information for subsequent analysis and processing. For example, in a complex software system, the initial features of a node may only be semantic code fingerprints generated based on its code and logs. After aggregating the features of its neighboring nodes through a multi-layer graph convolutional network, the embedding vector of that node also contains information about its upstream and downstream nodes, which can more accurately reflect its role in fault propagation.

[0066] After obtaining the updated node embedding vectors, graph pooling is performed. Graph pooling aggregates and compresses the node information of the entire error propagation graph, generating a graph-level vector representation that represents the entire graph. Graph pooling can employ various methods, such as average pooling or max pooling of the embedding vectors of all nodes. Through graph pooling, the local information of each node in the error propagation graph is integrated into a global graph-level vector representation. This representation reflects the overall characteristics and state of the error propagation graph. For example, if multiple nodes in the error propagation graph are associated with a specific fault mode, the graph-level vector representation after pooling will reflect the overall characteristics of this fault mode, providing a foundation for subsequent recurrence fault identification and root cause localization.

[0067] The similarity of the generated current error propagation graph's graph-level vector representation with its historical graph-level vector representations is compared. Historical graph-level vector representations are extracted and stored from previously processed fault cases, representing different types of fault modes. Common similarity metrics such as cosine similarity can be used for the comparison. By calculating the similarity between the current and historical graph-level vector representations, it is determined whether the current fault is similar to a historical fault. If the similarity exceeds a preset similarity threshold, the current fault is considered a duplicate fault. The preset similarity threshold can be analyzed and determined based on the characteristics of the actual system and historical fault data. Generally, a higher threshold leads to higher accuracy in identifying duplicate faults, but may miss some similar faults; a lower threshold broadens the range of duplicate faults, but may increase the probability of false positives. For example, in a large e-commerce system, by analyzing historical fault data, a similarity threshold of 0.8 is set. When the similarity between the current error propagation graph's graph-level vector representation and a certain historical graph-level vector representation exceeds 0.8, the current fault is considered a duplicate of that historical fault.

[0068] By analyzing the contribution of each node in a graph convolutional network to the graph-level vector representation, the nodes with the highest importance in error propagation can be identified. In a graph convolutional network, the feature update and aggregation processes of each node affect the final graph-level vector representation, with different nodes having varying degrees of influence. Several methods can be used to calculate the contribution of each node to the graph-level vector representation, such as gradient-based methods or attention mechanisms. By calculating the contribution, the nodes that play a key role in error propagation can be identified. The code location corresponding to the node with the highest importance is then output as the root cause of the fault. This code location can be a specific function, class, or module, allowing developers to directly pinpoint the specific location of the fault and perform targeted repairs. For example, in the e-commerce system mentioned above, analysis revealed that a certain payment service node has the highest contribution to the graph-level vector representation. Therefore, the code location corresponding to this payment service node can be identified as the root cause of the fault, and developers can then inspect and repair the relevant code of that payment service.

[0069] In some embodiments, step S104 above, which involves calling a reinforcement learning-based resource prediction model to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters based on the deduplicated error information and the real-time system load pattern, specifically includes: A state space is constructed, which includes cluster resource usage characteristics, system load mode characteristics, and fault context characteristics. The fault context characteristics are obtained by embedding and encoding the deduplicated error information. Construct an action space, which includes elastic scaling operations for adjusting the number of workload replicas and resource configuration optimization operations for adjusting container resource requests and limits; A reward function is set up, which is used to calculate the immediate reward based on the service level agreement violation, resource cost changes and stability fluctuations of the system after the action is executed, so as to guide the model to learn to minimize resource costs while ensuring service quality. Based on the state space, action space, and reward function, a reinforcement learning-based resource prediction model is used to simulate the behavior of cloud-native clusters, and output elastic scaling strategies and resource allocation optimization schemes for cloud-native clusters.

[0070] In this embodiment, a state space is constructed to describe the current state of the cloud-native cluster. The state space encompasses feature information from multiple key dimensions to comprehensively reflect the cluster's operational status. The state space includes cluster resource usage characteristics, system load pattern characteristics, and fault context characteristics.

[0071] (1) Cluster resource usage characteristics: Collect the usage of various resources in the cloud-native cluster, including but not limited to CPU utilization, memory usage, disk I / O rate and network bandwidth usage. This resource usage data can intuitively show the current resource consumption level of the cluster. For example, by continuously monitoring CPU utilization, we can understand the busy level of computing resources in the cluster, providing a basis for subsequent resource adjustments.

[0072] (2) System load pattern characteristics: Conduct in-depth analysis of the system load pattern and extract relevant characteristics. This includes indicators such as the system's request arrival rate, request processing time, and number of concurrent users. System load pattern characteristics can reflect the changes in business pressure of the system in different time periods. For example, during the promotional activities of the e-commerce system, the request arrival rate will increase significantly. By capturing this change in load pattern, we can prepare for resource adjustments in advance.

[0073] (3) Fault Context Features: The deduplicated error information is embedded and encoded to generate fault context features. Embedding encoding converts discrete error information into continuous vector representations, enabling them to be effectively processed by reinforcement learning models. For example, for different types of error logs, embedding encoding techniques are used to transform them into vectors with specific semantic meanings. These vectors can carry contextual information about the fault occurrence, such as the module where the fault occurred, the state of related components, etc., helping the model to better understand the relationship between the fault and the system state. The above cluster resource usage features, system load pattern features, and fault context features are integrated to form a complete state space, providing comprehensive system state information for reinforcement learning models.

[0074] Construct the action space of the reinforcement learning model. The action space defines a series of operations that the model can take to adjust the resource configuration of the cloud-native cluster, including elastic scaling and resource configuration optimization operations.

[0075] (1) Elastic scaling up and down operation: Adjusting the number of replicas of workloads in a cloud-native cluster. Based on the current system load and resource usage, the model can decide to increase or decrease the number of replicas for the workload. For example, when the system load is too high and the existing replicas cannot meet business needs, the model can generate an action to increase the number of replicas to improve the system's processing capacity; conversely, when the system load is low and there is resource waste, the model can generate an action to decrease the number of replicas to reduce resource costs.

[0076] (2) Resource Configuration Optimization Operations: Adjusting container resource requests and limits. Containers are the basic operating units in cloud-native clusters, and properly setting container resource requests and limits is crucial for ensuring system stability and resource utilization. The model can dynamically adjust parameters such as container CPU requests and limits, and memory requests and limits based on system status and fault information. For example, when a container frequently fails due to insufficient memory, the model can generate an action to increase the container's memory limit to prevent the failure from recurring; at the same time, for some containers with low resource utilization efficiency, the model can appropriately reduce their resource requests to improve overall resource utilization. Integrating elastic scaling operations and resource configuration optimization operations together constitutes an action space, providing the model with diverse resource adjustment methods.

[0077] Setting a reasonable reward function is a crucial step in guiding a reinforcement learning model to learn the optimal strategy. The reward function calculates immediate rewards based on several key system metrics after an action is executed, incentivizing the model to minimize resource costs while ensuring service quality. Key metrics include service level agreement violations, changes in resource costs, and stability fluctuations.

[0078] (1) Service Level Agreement (SLA) Violation: The Service Level Agreement (SLA) defines the service quality standards that the system must meet, such as response time and availability. When an action performed by the model causes the system to violate the SLA, a corresponding negative reward is given. The more severe the violation, the greater the negative reward. For example, if an action causes the system's response time to exceed the threshold specified in the SLA, the model will receive a negative reward value to encourage the model to avoid taking similar actions that would lead to a decrease in service quality.

[0079] (2) Resource cost changes: Focus on the changes in system resource costs after the action is executed. If the action can reduce resource costs, such as reducing unnecessary replicas or optimizing container resource configuration, a positive reward is given; conversely, if the action leads to an increase in resource costs, a negative reward is given. For example, when the model reduces the system's resource usage by reasonably adjusting resource configuration, thereby reducing the cloud service provider's charges, the model will receive a positive reward value to encourage the model to continue to take similar resource optimization actions.

[0080] (3) Stability Fluctuation: System stability is an important indicator for measuring the effectiveness of operation and maintenance. When the system stability fluctuates after an action is performed, such as an increase in the failure rate or instability in system performance, a negative reward is given; if the action helps improve system stability, a positive reward is given. For example, if an action causes a new failure in the system or prolongs the repair time of an existing failure, the model will receive a negative reward value to guide the model to prioritize ensuring system stability. Taking into account service level agreement violations, changes in resource costs, and stability fluctuations, a reasonable reward function is set to guide the reinforcement learning model to learn the optimal strategy of minimizing resource costs while ensuring service quality.

[0081] Based on a pre-constructed state space, action space, and reward function, a reinforcement learning-based resource prediction model simulates the behavior of a cloud-native cluster. During the simulation, the model selects appropriate actions from the action space based on the current state space information and receives immediate rewards according to the reward function. By continuously interacting with the environment—adjusting its action selection strategy based on actual system feedback—the model gradually learns the optimal actions to take in different states. After extensive simulation training, the model can output elastic scaling strategies and resource configuration optimization schemes for the cloud-native cluster based on the current system state. For example, the model might suggest increasing the number of replicas for a workload or adjusting the memory limits of a container. Operations personnel can then make real-time adjustments to the cloud-native cluster based on these suggestions to achieve stable system operation and efficient resource utilization.

[0082] This embodiment can effectively improve system stability and operational efficiency, and reduce resource costs.

[0083] In some embodiments, in step S105 above, the step of generating a corresponding repair plan and matching the optimal responsible person based on the location of the root cause of the fault and the real-time status characteristics of team members, by invoking a heterogeneous multi-agent engine containing a repair agent and a scheduling agent, specifically includes: Collect real-time status data of team members, normalize and vectorize the real-time status data, and construct the real-time status features of each member. The fault root cause information output from the graph convolutional network is structured and encoded, and a fault root cause feature vector is constructed through embedding mapping and semantic extraction. The fault root cause information includes root cause node type, code location, error type and code fragment. A repair agent based on a large language model is constructed. The root cause feature vector of the fault is converted into contextual prompts and input into the repair agent to generate one or more candidate repair schemes. Static syntactic verification and dynamic test verification are performed on each candidate repair scheme. A scheduling agent based on an attention mechanism is constructed. The root cause feature vector of the fault is used as the query and the real-time state feature is used as the key. The matching degree score between each member and the current fault is calculated through attention, and the member with the highest score is selected as the optimal person in charge.

[0084] In this embodiment, real-time status data of team members is collected. This data comes from a wide range of sources and covers multiple aspects, such as the type of task a member is currently working on, the task progress, the development or maintenance tools used, and the network status of their work environment. By setting up data collection points on multiple channels, including team members' work terminals, project management tools, and network monitoring systems, this information can be obtained comprehensively and in real time.

[0085] After collecting real-time status data, it's necessary to normalize it because different types of data have different dimensions and value ranges. Direct use of such data may affect the accuracy of subsequent analysis. Normalization maps the data to a specific interval, such as [0, 1], making different data points comparable.

[0086] After normalization, the data needs to be vectorized to facilitate computer processing and analysis. Vectorization converts discrete data into continuous vector representations, enabling it to be effectively recognized and utilized by subsequent intelligent agent models. Through specific vector encoding methods, the real-time state data of each member is converted into fixed-dimensional vectors. These vectors constitute the real-time state features of each member, providing the foundational data for matching the optimal responsible party.

[0087] The root cause information output from the graph convolutional network is structured and encoded. This information contains several key elements, such as the root cause node type (e.g., whether the problem occurred in a function, module, or interface), code location (specific line or section within the code file), error type (e.g., syntax error, logical error, runtime error), and related code snippets. Structured encoding of this information organizes it into a standardized and ordered format, facilitating subsequent processing and analysis. Next, a root cause feature vector is constructed using embedding mapping and semantic extraction techniques. Embedding mapping transforms discrete structured information into points in a continuous vector space, ensuring that information with similar semantics is grouped closer together. Semantic extraction further mines the deeper semantic features within the root cause information, such as using natural language processing techniques to understand the function and intent of code snippets. Through these steps, the root cause information is transformed into a feature vector that accurately represents its semantics and characteristics, providing crucial input for the repair agent to generate a repair plan.

[0088] A repair agent based on a large language model is constructed. Large language models possess powerful language understanding and generation capabilities, enabling them to generate grammatically and logically consistent text based on input information. In this embodiment, the root cause feature vector of the fault is converted into contextual cues. Contextual cues are information that guides the large language model to generate specific content; they contain key information about the fault, helping the model understand the background and needs of the current fault.

[0089] After contextual hints are input into the repair agent, the agent generates one or more candidate repair solutions. These candidate solutions are generated based on the large language model's learning from a large amount of code and fault handling experience, and thus possess a certain degree of rationality and feasibility.

[0090] However, the generated candidate fixes may contain syntax errors or fail to function correctly in a real system. Therefore, each candidate fix needs to undergo static syntax verification and dynamic testing verification. Static syntax verification uses syntax analysis tools to check whether the code of the fix conforms to the syntax rules of the programming language, ensuring that the code is error-free at the syntax level. Dynamic testing verification applies the fix to the actual system environment, running test cases to verify whether the fix can resolve the fault and will not negatively impact other parts of the system. After these two rounds of verification, candidate fixes that meet the requirements are selected, providing a basis for subsequently selecting the optimal fix.

[0091] A scheduling agent based on an attention mechanism is constructed. The attention mechanism is a technique that simulates human attention allocation, dynamically adjusting the weights of different parts based on the importance of the input information. In this embodiment, the root cause feature vector of the fault is used as the query, and the real-time status features of team members are used as the key.

[0092] Through attention computation, the scheduling agent can analyze the correlation between the real-time state features of each member and the current fault root cause feature vector, obtaining a matching score between each member and the current fault. The higher the matching score, the better the match between the member's real-time state and the current fault, and the more capable it is of handling the fault.

[0093] The member with the highest matching score is selected as the optimal responsible person. This ensures that faults are assigned to the most suitable member, improving the efficiency and quality of fault repair. Simultaneously, the scheduling agent can dynamically adjust the matching scores of members based on actual circumstances. For example, when a member's task changes or a new fault occurs, the matching score is recalculated promptly, ensuring the accuracy and real-time nature of responsible person matching.

[0094] Furthermore, the method also includes: Each word in the current root cause feature vector is used as a feature unit to be interpreted. The probability of the repair agent generating a complete repair code snippet or the conditional probability of generating a specific modified part is used as the target output value to be explained. A background dataset is constructed based on historical fault root cause feature vectors randomly sampled from the repair case library, and a set of randomly sampled feature subset masks are generated for the current fault root cause feature vector. For each feature subset mask, the masked feature values ​​are replaced by the background dataset to construct a masked input sequence, which is then input into the repair agent to calculate the corresponding target output value. Based on all feature subset masks and their target output values, the Shapley value of each word to the target output value is estimated by linear regression fitting. The tokens are sorted according to the absolute value of their Shapley values ​​to generate a feature importance ranking list. The Shapley value of each token is then mapped back to its position in the original input, and associated with the corresponding line of code and token.

[0095] In this embodiment, the current root cause feature vector is meticulously broken down, and each term is explicitly defined as a feature unit to be interpreted. These terms may be words describing the type of root cause node, identifiers indicating code locations, terms representing error types, or specific characters or words in code snippets.

[0096] Simultaneously, the target output value to be explained is determined. Here, the target output value is selected as either the probability that the repair agent generates a complete repair code snippet, or the conditional probability of generating a specific modified part. These two probability values ​​directly reflect the likelihood of the repair agent generating an effective repair solution when processing the current root cause feature vector of the fault, and are important indicators for measuring the role of lexical units.

[0097] A large number of historical failure root cause feature vectors were randomly sampled from a pre-established repair case library. These historical feature vectors cover various types of failure scenarios and were integrated to construct a background dataset. The background dataset provides rich reference information for subsequent analysis and can simulate various possible failure scenarios.

[0098] For the current root cause feature vector of the fault, a set of randomly sampled feature subset masks is generated. A feature subset mask is a tool used to identify which words are selected and which are ignored. By randomly generating different masks, different combinations of words in the feature vector can be covered, thereby comprehensively analyzing the role of each word.

[0099] For each generated feature subset mask, the feature values ​​corresponding to the masked terms in the current root cause feature vector are replaced with data from the background dataset. Specifically, according to the mask, the values ​​of certain terms in the current feature vector are replaced with the values ​​at the corresponding positions in the background dataset, thereby constructing a masked input sequence.

[0100] The constructed masked input sequence is fed into the repair agent, which then performs calculations based on this modified input to obtain the corresponding target output value: the probability of generating a complete repair code snippet or the conditional probability of generating a specific modified part. In this way, the changes in the repair agent's output can be observed when different word combinations are replaced.

[0101] After obtaining all feature subset masks and their corresponding target output values, a linear regression fitting method is used to estimate the Shapley value of each word in relation to the target output value. The Shapley value is an indicator used to measure the contribution of each participant to the overall payoff in cooperative games, and it is applied here to the analysis of word importance.

[0102] The process of linear regression fitting involves establishing a mathematical model that correlates the possible word combinations in the feature subset mask with the corresponding target output value, thereby calculating the contribution of each word to the target output value, i.e., the Shapley value. This process comprehensively considers all possible word combinations, ensuring a more accurate and comprehensive assessment of the importance of each word.

[0103] Based on the absolute value of the Shapley value for each term, all terms are sorted to generate a feature importance ranking list. The larger the absolute value, the greater the influence of the term on the target output value of the repair agent in generating the repair plan, that is, the higher its importance in the root cause feature vector.

[0104] Finally, the Shapley value of each term is mapped back to its position in the original root cause feature vector, and further associated with the corresponding line of code and term. This allows developers and operations personnel to clearly understand which terms in the root cause feature vector played a key role in generating the remediation plan, thereby better understanding the nature of the fault and providing strong support for subsequent fault repair and system optimization.

[0105] Reference Figure 2 An embodiment of the present invention provides an AI-driven DevOps adaptive fault repair system 2, wherein the system 2 specifically includes: The code fingerprint module 201 is used to extract semantic features from the code and logs generated during system runtime through a self-supervised contrastive learning model, and generate a semantic code fingerprint for uniquely identifying error types. Error identification module 202 is used to construct an error propagation graph based on the call relationship of the software system, using semantic code fingerprints as initial features of nodes. The deduplication and localization module 203 is used to learn high-order topological features of the error propagation graph using graph convolutional networks in order to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the fault. The resource optimization module 204 is used to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters by calling a resource prediction model based on reinforcement learning, based on the deduplicated error information and the real-time load mode of the system. The multi-agent module 205 is used to generate corresponding repair solutions and match the optimal responsible person based on the location of the root cause of the failure and the real-time status characteristics of team members by calling the heterogeneous multi-agent engine containing repair agents and scheduling agents.

[0106] It is understandable that, such as Figure 1 The content of the AI-driven DevOps adaptive fault repair method embodiment shown is applicable to the AI-driven DevOps adaptive fault repair system embodiment. The specific functions implemented in the AI-driven DevOps adaptive fault repair system embodiment are the same as those shown in the example. Figure 1 The illustrated AI-driven DevOps adaptive fault repair method is the same as the one shown, and achieves the same beneficial effects. Figure 1 The beneficial effects achieved by the AI-driven DevOps adaptive fault repair method embodiment shown are also the same.

[0107] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0109] Reference Figure 3 The present invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, it implements the AI-driven DevOps adaptive fault repair method as described in any of the above methods.

[0110] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0111] The processor 301 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0112] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0113] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the AI-driven DevOps adaptive fault repair method as described in any of the above methods.

[0114] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0115] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0116] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. An AI-driven DevOps adaptive fault repair method, characterized in that, The method specifically includes: Semantic features are extracted from the code and logs generated during system runtime using a self-supervised contrastive learning model to generate semantic code fingerprints that uniquely identify error types. Using semantic code fingerprints as initial node features, an error propagation graph is constructed based on the call relationships of the software system; Graph convolutional networks are used to learn high-order topological features of the error propagation graph in order to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the failure. Based on the deduplicated error messages and the real-time system load pattern, a reinforcement learning-based resource prediction model is invoked to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters. Based on the location of the root cause of the failure and the real-time status characteristics of team members, a heterogeneous multi-agent engine containing a repair agent and a scheduling agent is invoked to generate a corresponding repair plan and match the optimal responsible person.

2. The method according to claim 1, characterized in that, The step of extracting semantic features from the code and logs generated during system runtime using a self-supervised contrastive learning model to generate a semantic code fingerprint for uniquely identifying error types specifically includes: Positive sample pairs are generated by performing semantically preserved data augmentation operations on the original code snippets, and negative sample pairs are generated by sampling code snippets with different error types, thus constructing a training sample set for self-supervised contrastive learning; The training sample set is input into the encoder network, and the encoder network is trained according to the contrastive learning objective function. The contrastive learning objective function is used to bring the positive sample pairs closer and push the negative sample pairs further apart in the vector space, so that the encoder network can distinguish the semantic similarity of the code. The trained encoder network is deployed to an online pipeline. When a new erroneous code fragment is captured, it is fed into the encoder network for forward computation, and a high-dimensional continuous value vector is output as a semantic code fingerprint.

3. The method according to claim 1, characterized in that, The step of using semantic code fingerprints as initial node features and constructing an error propagation graph based on the call relationships of the software system specifically includes: Extract the call relationships between services, classes, and methods from the runtime data and static code of the software system, and construct a heterogeneous information network graph to describe the dependencies between system components; Semantic fingerprint vectors are used as initial features of nodes in a heterogeneous information network graph; When a system failure event is detected, an error propagation graph containing the failure node and its upstream and downstream nodes is extracted based on the link tracing data within the failure time window and the heterogeneous information network graph.

4. The method according to claim 3, characterized in that, The step of extracting an error propagation graph containing the faulty node and its upstream and downstream nodes based on link tracing data within the fault time window and combining it with a heterogeneous information network graph specifically includes: Based on the timestamp and service identifier of the system failure event, query all call chain instances within a preset time window from the distributed tracing system. Each call chain instance includes several spans, and each span is used to record detailed information of remote procedure calls or local method calls. Filter the call chain instances obtained from the query and retain call chain instances whose call status is failed, contain exception logs, or whose response time exceeds the historical threshold. Based on the heterogeneous information network graph, an error propagation graph containing the faulty node and its upstream and downstream nodes is extracted from the preserved call chain instances.

5. The method according to claim 4, characterized in that, The heterogeneous information network graph-based method extracts an error propagation graph containing the faulty node and its upstream and downstream nodes based on the retained call chain instances, specifically including: Identify abnormal spans as candidate fault nodes from the remaining call chain instances; Based on the preserved call chain instances, the call relationships between services, classes and methods are extracted from the heterogeneous information network graph, and nodes with the same service, class and method are deduplicated and merged to construct a call relationship graph; Using candidate fault nodes as seed nodes, a breadth-first traversal algorithm is used to trace upstream to the request origin and downstream to the fault-affected end in the call relationship graph, and a connected subgraph containing the seed node and its upstream and downstream related nodes is extracted as the error propagation graph. Attach dynamic attributes to each node and edge in the error propagation graph, including the number of calls, the number of failures, and the rate of change of response time during the fault.

6. The method according to claim 3, characterized in that, The method of using graph convolutional networks to learn high-order topological features of the error propagation graph to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the failure specifically includes: The error propagation graph and its initial node features are input into a graph convolutional network. Through multi-layer neighbor node feature aggregation, the feature representation of each node is updated to obtain a node embedding vector that integrates semantic and topological information. Perform graph pooling on the updated node embedding vectors to obtain a graph-level vector representation of the error propagation graph; The similarity between the graph-level vector representation and the historical graph-level vector representation is compared. If the similarity exceeds a preset similarity threshold, it is determined to be a duplicate fault. By analyzing the contribution of each node in the graph convolutional network to the graph-level vector representation, the node with the highest importance in error propagation is identified, and the code location corresponding to that node is output as the root cause of the fault.

7. The method according to claim 1, characterized in that, Based on the deduplicated error information and the real-time system load pattern, the process involves calling a reinforcement learning-based resource prediction model to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters. Specifically, this includes: A state space is constructed, which includes cluster resource usage characteristics, system load mode characteristics, and fault context characteristics. The fault context characteristics are obtained by embedding and encoding the deduplicated error information. Construct an action space, which includes elastic scaling operations for adjusting the number of workload replicas and resource configuration optimization operations for adjusting container resource requests and limits; A reward function is set up, which is used to calculate the immediate reward based on the service level agreement violation, resource cost changes and stability fluctuations of the system after the action is executed, so as to guide the model to learn to minimize resource costs while ensuring service quality. Based on the state space, action space, and reward function, a reinforcement learning-based resource prediction model is used to simulate the behavior of cloud-native clusters, and output elastic scaling strategies and resource allocation optimization schemes for cloud-native clusters.

8. The method according to claim 1, characterized in that, Based on the location-based root cause of the failure and the real-time status characteristics of team members, a heterogeneous multi-agent engine containing a repair agent and a scheduling agent is invoked to generate a corresponding repair plan and match the optimal responsible person. Specifically, this includes: Collect real-time status data of team members, normalize and vectorize the real-time status data, and construct the real-time status features of each member. The fault root cause information output from the graph convolutional network is structured and encoded, and a fault root cause feature vector is constructed through embedding mapping and semantic extraction. The fault root cause information includes root cause node type, code location, error type and code fragment. A repair agent based on a large language model is constructed. The root cause feature vector of the fault is converted into contextual prompts and input into the repair agent to generate one or more candidate repair schemes. Static syntactic verification and dynamic test verification are performed on each candidate repair scheme. A scheduling agent based on an attention mechanism is constructed. The root cause feature vector of the fault is used as the query and the real-time state feature is used as the key. The matching degree score between each member and the current fault is calculated through attention, and the member with the highest score is selected as the optimal person in charge.

9. The method according to claim 8, characterized in that, The method further includes: Each word in the current root cause feature vector is used as a feature unit to be interpreted. The probability of the repair agent generating a complete repair code snippet or the conditional probability of generating a specific modified part is used as the target output value to be explained. A background dataset is constructed based on historical fault root cause feature vectors randomly sampled from the repair case library, and a set of randomly sampled feature subset masks are generated for the current fault root cause feature vector. For each feature subset mask, the masked feature values ​​are replaced by the background dataset to construct a masked input sequence, which is then input into the repair agent to calculate the corresponding target output value. Based on all feature subset masks and their target output values, the Shapley value of each word to the target output value is estimated by linear regression fitting. The tokens are sorted according to the absolute value of their Shapley values ​​to generate a feature importance ranking list. The Shapley value of each token is then mapped back to its position in the original input, and associated with the corresponding line of code and token.

10. An AI-driven DevOps adaptive fault repair system, characterized in that, The system specifically includes: The code fingerprint module is used to extract semantic features from the code and logs generated during system runtime using a self-supervised contrastive learning model, and generate a semantic code fingerprint that uniquely identifies the error type. The error identification module is used to construct an error propagation graph based on the call relationships of the software system, using semantic code fingerprints as the initial features of nodes. The deduplication and localization module is used to learn high-order topological features of the error propagation graph using graph convolutional networks, so as to accurately deduplicate error information containing duplicate and similar errors and locate the root cause of the fault. The resource optimization module is used to dynamically generate elastic scaling strategies and resource configuration optimization schemes for cloud-native clusters by calling a resource prediction model based on reinforcement learning, based on the deduplicated error information and the real-time system load mode. The multi-agent module is used to generate corresponding repair solutions and match the optimal responsible party based on the location of the root cause of the failure and the real-time status characteristics of team members by calling a heterogeneous multi-agent engine containing repair agents and scheduling agents.

Images