Operation and maintenance method of distributed system

Abstract

The invention discloses an operation and maintenance method of a distributed system. Relates to the distributed field and the financial science and technology field, and the method comprises the steps that under the condition that an alarm target of a distributed system generates alarm information, a label combination of the alarm target is extracted, the alarm target is an object on the distributed system, and the object is a resource in the distributed system or an application running on the distributed system; a target root cause node of the alarm information is positioned from a first map based on the label combination of the alarm target, each node in the first map represents an object on the distributed system, the attribute of each node is determined by the label of the object, and each edge represents the incidence relation between the two objects; operation and maintenance information is inquired according to the target root cause node, operation and maintenance operation is executed based on the operation and maintenance information, and the operation and maintenance operation is used for eliminating faults indicated by the alarm information. Through the method and the device, the problem of low fault processing accuracy and efficiency of an operation and maintenance scheme of a distributed system in related technologies is solved.

Description

Technical Field

[0001] This application relates to the fields of distributed systems and financial technology, and more specifically, to a method for operating and maintaining a distributed system. Background Technology

[0002] Current distributed system operation and maintenance practices primarily rely on monitoring tools, rule engines, and call chain analysis for fault detection and response. Basic monitoring tools are used to set static thresholds to trigger alarms, rule engines execute predefined fault handling scripts, and distributed tracing systems are used for service dependency visualization and root cause analysis. However, these solutions reveal a series of limitations and challenges when dealing with increasingly complex cloud-native environments.

[0003] On the one hand, the monitoring, logging, and configuration management systems of distributed systems each employ different tagging systems, leading to inconsistent and untimely tag updates after resource changes. This inconsistency increases the burden on operations and maintenance personnel when handling faults, requiring them to manually compare and unify environmental information across multiple systems. Furthermore, for cross-system alarm events, the lack of consistent tag associations makes it difficult for operations and maintenance personnel to quickly pinpoint the root cause of the problem, which is particularly pronounced in scenarios requiring emergency response.

[0004] On the other hand, there is a disconnect between infrastructure alerts and application-layer alerts. When encountering common failures such as database connection exhaustion, operations personnel often need to spend a lot of time recalling the specific topology of the service deployment and searching logs across the entire cluster in order to find the service instances or components that may be causing the problem. The time-consuming process of finding and correlating alert information across systems impacts system availability and user experience.

[0005] In summary, existing distributed system operation and maintenance (O&M) solutions have significant shortcomings in terms of environmental identification consistency, fault location efficiency, and the security and intelligence of automated handling. These issues not only limit the fault handling speed of O&M personnel but also increase the O&M risks and costs. Therefore, there is a need to achieve accurate and rapid self-healing of faults, significantly improving O&M efficiency and system stability.

[0006] There is currently no effective solution to the problem of low accuracy and efficiency in fault handling of distributed system operation and maintenance schemes in related technologies. Summary of the Invention

[0007] The main objective of this application is to provide a method for the operation and maintenance of distributed systems, so as to solve the problems of low accuracy and efficiency in fault handling of distributed system operation and maintenance schemes in related technologies.

[0008] To achieve the above objectives, according to one aspect of this application, a method for operation and maintenance of a distributed system is provided. The method includes: when an alarm target in the distributed system generates alarm information, extracting a combination of tags for the alarm target, wherein the alarm target is an object on the distributed system, and the object is a resource in the distributed system or an application running on the distributed system; locating the target root cause node of the alarm information from a first graph based on the tag combination of the alarm target, wherein each node in the first graph represents an object on the distributed system, the attributes of each node are determined by the object's tag, and each edge represents the association relationship between two objects; querying operation and maintenance information based on the target root cause node, and performing operation and maintenance operations based on the operation and maintenance information, wherein the operation and maintenance operations are used to eliminate the fault indicated by the alarm information.

[0009] Optionally, the first graph is constructed as follows: Tags of objects on the distributed system are obtained from multiple sources to obtain a tag set; when the same object in the tag set has tags with the same name from different sources, the tags with the same name from different sources are overridden according to a tag overriding rule to obtain a first tag set, wherein the tag overriding rule indicates that for tags with the same name from different sources, the tag from the higher-level source is used to overwrite the tag from the lower-level source; when a resource change event occurs, the first tag set is updated based on the resource change event and the tag overriding rule to obtain a second tag set, wherein the resource change event includes at least one of the following: resource creation event, resource update event, application exception event; nodes are determined based on objects on the distributed system, the attributes of nodes are determined based on the tags carried by the objects, and edges are determined based on the relationships between objects to obtain the first graph.

[0010] Optionally, obtaining tags for objects on the distributed system from multiple sources to obtain a tag set includes: obtaining metadata of the cloud infrastructure of the distributed system and using the cloud infrastructure metadata as a first type of tag, wherein the first type of tag is a resource tag; obtaining pipeline information from the application development deployment pipeline and using the pipeline information as a second type of tag, wherein the second type of tag is a resource tag and an application tag; obtaining business semantic tags defined through a resource manifest file and using the business semantic tags as a third type of tag, wherein the third type of tag is a resource tag and an application tag; determining an initial tag set based on at least one of the first type of tag, the second type of tag, and the third type of tag; validating the tags in the initial tag set, and performing enhancement operations on tags that fail to be validated to obtain a tag set, wherein the enhancement operation indication includes one of the following: tag completion operation and tag correction operation.

[0011] Optionally, the overriding process for the same-named tags from different sources of the same object according to the tag overriding rules includes: if the same-named tags from different sources include user-defined tags, then the user-defined tags are used to overridden tags other than user-defined tags in the same-named tags; if the same-named tags from different sources do not include user-defined tags but include the object's own tags, then the object's own tags are used to overridden tags other than the object's own tags in the same-named tags; if the same-named tags from different sources do not include user-defined tags, the object's own tags, but include at least two tags inherited by the object from the hierarchical structure, then the tags other than ...

[0012] Optionally, in the event of a resource change event, updating the first tag set based on the resource change event and tag overriding rules includes: in the event of a resource creation event, obtaining the tags of the created resource; if there are tags with the same name from different sources for the same resource after creation, overriding the tags with the same name from different sources for the same resource according to the tag overriding rules, and adding the overridden tags to the first tag set; in the event of a resource update event, obtaining the tags of the updated resource; if there are tags with the same name from different sources for the same resource after update, overriding the tags with the same name from different sources for the same resource according to the tag overriding rules, and adding the overridden tags to the first tag set; in the event of an application exception event, determining the application and resources associated with the application exception event, adding fault tags to the associated application and resources, and adding the fault tags to the first tag set.

[0013] Optionally, locating the target root cause node of the alarm information from the first graph based on the tag combination of the alarm target includes: determining the abnormal nodes covered by the tag combination of the alarm target from the first graph; determining the upstream nodes of the abnormal nodes from the first graph to obtain a set of candidate root cause nodes; calculating the suspicious score of each candidate root cause node in the set of candidate root cause nodes; performing multi-dimensional verification on each candidate root cause node in the set of candidate root cause nodes to obtain the verification result; and filtering the target root cause node from the candidate root cause nodes according to the suspicious score and the verification result.

[0014] Optionally, calculating the suspiciousness score of each candidate root cause node in the candidate root cause node set includes: for a candidate root cause node, calculating the product of the basic weight of the lost resource dimension and the indicator value of the resource dimension to obtain a first suspiciousness score, wherein the basic weight of the resource dimension represents the degree of resource contention; calculating the product of the basic weight of the application dimension and the indicator value of the application dimension to obtain a second suspiciousness score, wherein the basic weight of the application dimension represents the degree of application abnormality; and calculating the suspiciousness score of the candidate root cause node based on the first suspiciousness score and the second suspiciousness score.

[0015] Optionally, multi-dimensional verification of each candidate root cause node in the candidate root cause node set includes at least one of the following: for a candidate root cause node, determining whether the total number of abnormal services and resources under the label combination of the candidate root cause node reaches a quantity threshold; determining whether the proportion of faults associated with historical label combinations similar to the label combination of the candidate root cause node reaches a fault proportion threshold; determining whether the candidate root cause node propagates along the first graph and affects services or resources at two or more levels; determining whether the difference between the candidate root cause node and healthy nodes under the same label reaches a difference threshold.

[0016] Optionally, querying operation and maintenance information based on the target root cause node and performing operation and maintenance operations based on the operation and maintenance information includes: querying the operation and maintenance knowledge base based on the tag combination of the target root cause node to obtain operation and maintenance information, wherein the operation and maintenance knowledge base includes different preset tag combinations and preset operation and maintenance information associated with the preset tag combinations; calculating the impact degree value of the operation and maintenance operation in the operation and maintenance information, and performing the operation and maintenance operation if the impact degree value is less than the impact degree value threshold; generating an approval request if the impact degree value is greater than or equal to the impact degree value threshold, sending the approval application to the approval end, and performing the operation and maintenance operation if the approval approval message is received from the approval end.

[0017] Optionally, after extracting the tag combination of the alarm target, the method further includes: extracting key information from the handling process of historical faults, wherein the key information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path, and timeline; converting the key information into elements and relationships, defining the elements as nodes, and defining the relationships as edges to obtain a second graph, wherein the nodes in the second graph include at least one of the following: fault mode nodes, root cause nodes, and operation and maintenance nodes; matching similar historical fault modes in the second graph according to the tag combination of the alarm target, obtaining fault information of the historical fault modes, and pushing fault information to the operation and maintenance end, wherein the fault information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path, and timeline.

[0018] In this embodiment, when an alarm target in a distributed system generates alarm information, a combination of tags for the alarm target is extracted. The alarm target is an object on the distributed system, and the object can be a resource or an application running on the distributed system. Based on the tag combination of the alarm target, the root cause node of the alarm information is located from a first graph. Each node in the first graph represents an object on the distributed system, the attributes of each node are determined by the object's tag, and each edge represents the association between two objects. Operation and maintenance information is queried based on the root cause node, and operation and maintenance operations are performed based on this information. These operations are used to eliminate the fault indicated by the alarm information, solving the problem of low accuracy and efficiency in fault handling of distributed system operation and maintenance schemes in related technologies. By extracting the tag combination of the alarm target, locating the root cause node of the alarm information from the first graph, matching the operation and maintenance information, and performing operation and maintenance operations, the effect of quickly and accurately locating faults in the distributed system and efficiently and accurately handling faults is achieved. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 A hardware structure block diagram of a computer terminal for implementing an operation and maintenance method for a distributed system is shown.

[0021] Figure 2 This is a flowchart of an operation and maintenance method for a distributed system according to an embodiment of this application;

[0022] Figure 3 This is a partial schematic diagram of the first map in the operation and maintenance method of a distributed system according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram illustrating tag acquisition in the operation and maintenance method of a distributed system according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of label overlay in the operation and maintenance method of a distributed system according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the operation and maintenance system of the distributed system according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of an operation and maintenance device for a distributed system according to an embodiment of this application;

[0027] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding operation entry points for them to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage. If the user chooses to agree, the user can view the purpose of data use in real time through the authorization interface and has the right to withdraw authorization or delete data at any time. After authorization is withdrawn, the system will terminate the relevant data processing within 24 hours.

[0031] Example 1

[0032] According to an embodiment of this application, an embodiment of a method for operating and maintaining a distributed system is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an operation and maintenance method for a distributed system is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (illustrated as 102a, 102b, ..., 102n) may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA), etc., a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the operation and maintenance method of the distributed system in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned operation and maintenance method of the distributed system. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0037] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0038] Under the aforementioned operating environment, this application provides the following: Figure 2 The operation and maintenance methods for the distributed system are shown. Figure 2 This is a flowchart of an operation and maintenance method for a distributed system according to an embodiment of this application.

[0039] Step S201: When an alarm target in the distributed system generates alarm information, extract the tag combination of the alarm target, wherein the alarm target is an object on the distributed system, and the object is a resource in the distributed system or an application running on the distributed system.

[0040] An alarm target refers to an entity or service in a distributed system. When an entity's resource or application service in a distributed system malfunctions (e.g., a monitoring metric exceeds a preset threshold), an alarm signal is generated. For example, an excessively long delay in a payment service will generate an alarm signal; the payment service is the alarm target.

[0041] Each object (resource or application) in a distributed system has one or more tags to describe its characteristics, location, function, or status. When an alarm is generated, the engine extracts a combination of tags closely related to the alarm target to facilitate subsequent root cause analysis. For example, when a payment service generates an alarm, the extracted tag combination might include service=payment (the service is a payment service), zone=ap-east-1c (the availability zone is Asia Pacific East 1c), and error_type=timeout (the error type is timeout), etc.

[0042] This step, by extracting the tag combinations of alarm targets, can quickly and accurately identify the related information behind the alarm information, narrowing the scope of the fault search and providing important clues and context for subsequent fault localization.

[0043] Step S202: Locate the target root cause node of the alarm information from the first graph based on the combination of alarm target labels. Each node in the first graph represents an object on the distributed system, the attributes of each node are determined by the object's label, and each edge represents the association between two objects.

[0044] The first graph refers to the relational graph of entity resources or application services in a distributed system. The nodes in the graph represent objects (entity resources (physical and virtual infrastructure) and service applications (such as elastic computing service instances, databases, payment services, etc.)) on the distributed system, and the edges represent the relationships between these objects (such as dependencies, running on, calling, etc.).

[0045] The root cause refers to the fundamental reason that triggers the alarm message. The engine can locate the most likely cause node, i.e., the target root cause node, from the first graph based on the tag combination of the alarm target. For example, the engine obtains the nodes covered by the tag combination of the alarm target and searches for the target root cause node from these nodes. For instance, excessive latency in the payment service may be caused by a full database connection pool that it depends on.

[0046] This step traces the fault propagation path backward from the first graph by combining the labels of the alarm targets, and locates the specific and most likely source of the fault (target root cause node). This overcomes the drawbacks of difficult and time-consuming fault root cause location in related technologies, and achieves accurate and efficient fault root cause location.

[0047] Step S203: Query the operation and maintenance information based on the target root cause node, and perform operation and maintenance operations based on the operation and maintenance information. The operation and maintenance operations are used to eliminate the fault indicated by the alarm information.

[0048] After identifying the target root cause node, you can query the predefined operations and maintenance knowledge base (i.e., the remediation operation knowledge base) to obtain the operations and maintenance information for that node. The operations and maintenance information may include specific operations and maintenance operations, the risk level of the operations and maintenance operations, the possible scope of impact, and rollback strategies, etc.

[0049] Operational and maintenance (O&M) operations are the safe remediation actions performed on faults indicated by appropriate alarm information in the O&M information. For example, when the database connection pool is full, O&M operations could include: urgently increasing the number of connections, restarting the connection pool service, or vertically scaling up the database instance size.

[0050] This step queries the operation and maintenance information based on the target root cause node and performs operation and maintenance operations based on the operation and maintenance information to eliminate the fault indicated by the alarm signal and ensure efficient and reliable fault recovery.

[0051] The distributed system operation and maintenance method provided in this application extracts a combination of tags from the alarm target when an alarm message is generated in the distributed system. The alarm target is an object on the distributed system, which can be a resource or an application running on the distributed system. Based on the tag combination of the alarm target, the root cause node of the alarm message is located from a first graph. Each node in the first graph represents an object on the distributed system, the attributes of each node are determined by the object's tag, and each edge represents the association between two objects. Operation and maintenance information is queried based on the root cause node, and operation and maintenance operations are performed based on the information. These operations are used to eliminate the fault indicated by the alarm message, thus solving the problems of low accuracy and efficiency in fault handling of distributed system operation and maintenance schemes in related technologies. By extracting the tag combination of the alarm target, locating the root cause node of the alarm message from the first graph, matching the operation and maintenance information, and performing operation and maintenance operations, the method achieves the effect of quickly and accurately locating faults in the distributed system and efficiently and accurately handling them.

[0052] Optionally, in the operation and maintenance method of the distributed system provided in this application embodiment, the first graph is constructed in the following manner: obtaining labels of objects on the distributed system from multiple sources to obtain a label set; when the same object in the label set has the same name labels from different sources, the same name labels from different sources of the same object are overridden according to the label overriding rule to obtain the first label set, wherein the label overriding rule indicates that for the same name labels from different sources, the label of the higher-level source is used to overridden the label of the lower-level source; when a resource change event occurs, the first label set is updated based on the resource change event and the label overriding rule to obtain the second label set, wherein the resource change event includes at least one of the following: resource creation event, resource update event, application exception event; determining nodes according to objects on the distributed system, determining the attributes of nodes according to the labels carried by the objects, and determining edges according to the association relationship between objects to obtain the first graph.

[0053] For example, the dynamic tag propagation engine can be controlled to continuously collect or define tags from multiple levels and sources to form an initial tag set. These sources can include cloud infrastructure layers, deployment pipelines, policy controllers, and business semantic tags explicitly defined by developers. Through multi-source tag injection, multi-dimensional information about objects in the distributed system can be captured, enhancing the descriptive capabilities of the constructed first map and the accuracy of fault location.

[0054] When faced with identically named tags received from different sources for the same object (e.g., region tags within the same container group), conflict resolution is performed according to tag override rules. Tag override rules state that for identically named tags from different sources, the tag from the higher-level source should override the tag from the lower-level source. For example, user-defined tags have the highest priority and override any automatically injected or inherited tags with the same name. Since tags are used for node attributes, conflict resolution ensures the consistency and accuracy of node attributes in the first graph.

[0055] In distributed systems, resource creation, updates, and abnormal events occur frequently. When such resource change events are triggered, the dynamic tag propagation engine updates the tag information in real time based on the existing first tag set and tag coverage rules, and generates a second tag set, so that the first graph always remains up-to-date.

[0056] Finally, the dynamic label propagation engine can be controlled to determine nodes based on objects (resource entities or application services) in the distributed system, determine edges based on the relationships between objects, and determine the attributes of nodes based on the labels carried by the objects, thus obtaining the first graph. The first graph can represent the complex dependencies and influence relationships between resources and applications based on a directed graph. Node types can include resource nodes and application nodes. Resource nodes refer to physical or virtual entities, such as elastic computing instances, compute nodes, container groups, containers, database instances, etc. Application nodes (also called service nodes) refer to business logic units, such as payment services. Node types can also include logical fault nodes, which are used to aggregate and represent specific fault scenarios or abstract fault points, generated or labeled by the analysis engine after location. For example, it can include network-partition-zone-a (network partition a) and config-error-db-connection (database connection configuration error). The relationships between edges can be inheritance, runtime, dependency, invocation, propagation, etc. The attributes of nodes are also multi-source labels (infrastructure, deployment, policy, business, runtime context).

[0057] Figure 3 This is a partial schematic diagram of the first graph in the operation and maintenance method of a distributed system according to an embodiment of this application, including nodes of the following objects, where the corresponding root cause node can be found by following the graph when a physical layer obstacle occurs.

[0058] The availability zone ap-east-1c, tagged with zone=ap-east-1c, status=down, indicates that the cloud service in the Asia Pacific East 1c zone is "down," meaning that the service in this availability zone is currently unavailable.

[0059] The availability zone ap-east-1c is connected to an ECS (Elastic Container Service) instance i-0a1b2c. The ECS instance i-0a1b2c has the tag: instance_type=c6g.4xlarge, role=database host. This means that the instance type of ECS instance i-0a1b2c is c6g.4xlarge, and its role is database host.

[0060] ECS instance i-0a1b2c is connected to the database master node. The database master node has the tags: db_role=primary, conn_count=98%. This indicates that the database's role is the master node, and its connection count has reached 98%, suggesting that the database is nearing its connection limit and may be a performance bottleneck.

[0061] The database master node is connected to a payment service database connection pool. The payment service database connection pool has the tags: service=payment, conn_wait=1500ms. This indicates that the service is a payment service, and the connection wait time is 1500 milliseconds, suggesting a performance issue with the database connection pool.

[0062] Availability zone ap-east-1c is connected to an NLB (Network Load Balancer). An NLB load balancer, tagged with lb_type=internet-facing, is an internet-facing load balancer used to distribute incoming network traffic to multiple destinations.

[0063] The NLB load balancer is connected to a payment service Pod (container). The payment service Pod has the tags: version=v2.3, biz_unit=payment. This indicates that the version is v2.3 and the business unit is payment.

[0064] The payment service Pod connects to both an order service and a risk control service. The order service, tagged with `biz_unit=order` and `error_rate=45%`, indicates that the error rate for the order service has reached 45%. The risk control service, tagged with `biz_unit=risk` and `timeout_count=120 / min`, indicates that the timeout count for the risk control service is 120 times per minute, suggesting potential performance issues or a large volume of requests being processed.

[0065] The first graph constructed in this embodiment not only structurally covers the complex dependencies of objects in the distributed system, but also carries real-time updated multi-dimensional tags on the attributes of each node. The real-time performance and accuracy of the first graph exceed those of static configuration management databases in related technologies, enabling the rapid location of the problem source and precise execution of repair operations based on the first graph when a fault occurs, thereby improving operational efficiency and system stability.

[0066] Optionally, in the distributed system operation and maintenance method provided in this application embodiment, obtaining tags of objects on the distributed system from multiple sources to obtain a tag set includes: obtaining metadata of the cloud infrastructure of the distributed system and using the metadata of the cloud infrastructure as a first type of tag, wherein the first type of tag is a resource tag; obtaining pipeline information in the application development deployment pipeline and using the pipeline information as a second type of tag, wherein the second type of tag is a resource tag and an application tag; obtaining business semantic tags defined by the resource manifest file and using the business semantic tags as a third type of tag, wherein the third type of tag is a resource tag and an application tag; determining an initial tag set based on at least one of the first type of tag, the second type of tag, and the third type of tag; verifying the tags in the initial tag set, and performing enhancement operations on tags that fail verification to obtain a tag set, wherein the enhancement operation indication includes one of the following: tag completion operation and tag correction operation.

[0067] Collect or define tags from different levels and sources. Figure 4 This is a schematic diagram illustrating the tag acquisition process in the operation and maintenance method of a distributed system according to an embodiment of this application.

[0068] like Figure 4 As shown, tags can be injected into a tag repository via cloud infrastructure layer injection. A lightweight agent deployed on compute nodes automatically captures metadata information from the cloud infrastructure, such as region, availability zone, and instance type. This information is converted into first-class tags, serving as resource-level identifiers and attributes. For example, the tags `region: ap-southeast-1`, `zone: ap-southeast-1a`, and `instance_type: c6g.4xlarge` indicate that the region is Southeast Asia, the availability zone is Southeast Asia 1a, and the instance type is `c6g.4xlarge`.

[0069] like Figure 4As shown, tags can be injected into the tag repository through deployment pipeline injection: In the CI / CD (Continuous Integration and Continuous Delivery / Deployment) process, build and deployment information is automatically embedded into the resource definition file of the application during deployment as a second type of tag, and is assigned to the application and service, forming a resource-application layer association tag. For example, the tag could be: deploy_id: ${CI_PIPELINE_ID}, git_commit: 8a3fd2, build_time: 2023-06-15T14:30, where deploy_id is the deployment identifier, CI_PIPELINE_ID is the identifier of the continuous integration pipeline, git_commit: 8a3fd2e is the hash value of the code commit, and build_time: 2023-06-15T14:30 is the build time.

[0070] like Figure 4 As shown, tags can be injected into the tag repository through explicit developer definition: application developers define business semantic tags through the `metadata.labels` or `annotations` fields in the Kubernetes (container orchestration system) resource manifest file, resulting in the third type of tags. These tags are persistently stored through Kubernetes' interface service admission control. The third type of tags includes information such as business unit and owner, enabling a more precise expression of the application's business ownership and operational responsibility. For example, the tag could be `business_unit:payment`, `owner:team-ops`, where `business_unit:payment` indicates that the business unit is a payment business, and `owner:team-ops` specifies that the resource owner is a team operation.

[0071] Information is obtained from three categories of tags: Category 1, Category 2, and Category 3. An initial tag set is constructed based on at least one of these categories to ensure comprehensive and diverse tag coverage. Tags in the initial tag set are then validated and enhanced. A policy controller can be used to validate the tags according to predefined rules, checking their completeness, compliance, and consistency with the actual state of the distributed system. For tags that fail validation, corresponding enhancement operations are performed. These can include tag completion (automatically adding missing necessary tags) and tag correction (correcting erroneous or outdated tag information) to ensure the tag set is accurate, complete, and conforms to the latest distributed system state and business rules.

[0072] This embodiment first collects tags from multiple sources and performs verification and tag enhancement operations. The injection of multi-source tags and tag verification enhance the identifiability of resources and applications, while the tag enhancement operation improves tag quality, laying the foundation for subsequent fault location and maintenance operations.

[0073] To ensure label consistency and accuracy, optionally, in the operation and maintenance method of the distributed system provided in this application embodiment, the overriding processing of the same-named labels from different sources for the same object according to the label overriding rules includes: when the same-named labels from different sources include user-defined labels, the user-defined labels are used to override the labels other than the user-defined labels in the same-named labels; when the same-named labels from different sources do not include user-defined labels but include the object's own labels, the object's own labels are used to override the labels other than the object's own labels in the same-named labels; when the same-named labels from different sources do not include user-defined labels, the object's own labels, but include at least two labels inherited by the object from the hierarchical architecture, the labels other than ...

[0074] For example, if an object has a tag with the same name from multiple different sources, and this includes a user-defined tag, the tag override rule is strictly followed. The user-defined tag is automatically used, overriding all other tags with the same name from all other sources. For instance, suppose a payment service has a directly defined tag of {business_unit: payment}, and the cloud infrastructure metadata hosting it also has a tag named business_unit, but with a value of non-critical. Since the user-defined tag has the highest priority, {business_unit: payment} is used to override the non-critical tag, ensuring that the business unit tag always reflects the user's intent.

[0075] For example, if an object has a tag with the same name from multiple different sources, in cases where there is no user-defined tag with the same name but a tag of its own exists, the tag of its own name is used, overriding all tags of the same source that are at a lower level than the object. For instance, if a container explicitly sets `zone: ap-southeast-1a` (the tag for Southeast Asia Pacific 1a) in its resource definition, and there is a tag with the same name `zone: ap-northeast-1b` (the tag for Northeast Asia Pacific 1b) in its namespace, the value of `ap-southeast-1a` is retained because the tag of the resource itself has higher priority, thus ensuring that the node attribute accurately reflects the latest state of the resource itself.

[0076] For example, if an object's same-name tag comes from multiple different sources, and there is neither a user-defined tag nor a tag from the resource itself, but the object inherits multiple tags with the same name from its hierarchical structure, for example, the object is a resource, the resource's parent level is a controller template, the controller template's parent level is a namespace, the namespace's parent level is a node, and the node's parent level is a cluster, then the tag closest to the object's hierarchy will be used first, covering all tags with the same name from more distant levels.

[0077] Table 1 illustrates the hierarchical architecture of resources. As shown in Table 1, resource labels originate from containers, have a scope limited to the current resource, and can be `Pod.metadata.labels` (container metadata labels), with a priority of 100. Controller template labels originate from controllers, have a scope limited to resources managed by the controller, and can be `Deployment.spec.template` (deployment space template), affecting all containers managed by it, with a priority of 80. Namespace labels originate from namespaces, have a scope limited to all resources within the namespace, and can be `Namespace.metadata` (namespace metadata), inherited by all resources within it, with a priority of 60. Node labels originate from nodes, have a scope limited to all containers on the node, and can be `Node.metadata` (node ​​metadata), inherited by containers scheduled to that node, with a priority of 40. Cluster-level labels originate from the cluster, have a scope limited to the entire cluster, and can be `ClusterInfo CRD` (ClusterInfo Custom Resource Definition), with a priority of 20.

[0078] Table 1

[0079]

[0080] Figure 5 This is a schematic diagram of label overlay in the operation and maintenance method of a distributed system according to an embodiment of this application, such as... Figure 5As shown, the object type of a resource object is determined. If it's a cluster, the cluster tag library is read; if it's a namespace, cluster tags are inherited and namespace tags are appended; if it's a container, namespace tags and container-specific tags are merged. Then, tag conflict detection is performed. Tag conflict refers to an object having tags with the same name from multiple different sources. If there's no conflict, the tag is written to the tag database; if a conflict exists, weight priority resolution is performed. For example, consider a container that inherits the tag `security_level: medium` (medium security level) from the controller and the tag `security_level: low` (low security level) from the namespace. Since controller template tags have higher priority than namespace tags, `security_level: medium` is retained as a Pod attribute to ensure that the description of security levels in the graph most closely reflects the actual resource management strategy.

[0081] In this embodiment, when tags with the same name from different sources conflict, the priority rule is strictly followed for overriding. User-defined tags have the highest priority and override any automatically injected or inherited tags with the same name. Automatically injected tags override inherited tags with the same name, and the lowest-level tag is retained among the inherited tags with the same name. This effectively eliminates conflicts caused by multi-source tag injection, ensuring that the attributes of each node in the constructed first graph are accurate, consistent, and reflect the true state of the distributed system in a timely manner.

[0082] Optionally, in the operation and maintenance method of the distributed system provided in the embodiments of this application, when a resource change event occurs, updating the first tag set based on the resource change event and the tag overriding rule includes: when a resource creation event occurs, obtaining the tag of the created resource; when there are tags with the same name from different sources for the same resource after creation, overriding the tags with the same name from different sources for the same resource according to the tag overriding rule, and adding the overriding tags to the first tag set; when a resource update event occurs, obtaining the tag of the updated resource; when there are tags with the same name from different sources for the same resource after update, overriding the tags with the same name from different sources for the same resource according to the tag overriding rule, and adding the overriding tags to the first tag set; when an application exception event occurs, determining the application and the associated resources associated with the application exception event, adding fault tags to the associated application and the associated resources, and adding the fault tags to the first tag set.

[0083] For example, resource change events of Kubernetes interface services can be continuously detected. For each resource creation event and resource update event, the engine calculates its final tag set according to the configured inheritance rules.

[0084] When a resource creation event occurs, all tag information for the newly created resource is collected, including but not limited to tags injected from cloud infrastructure metadata, deployment pipelines, policy controllers, and explicitly defined tags by developers. For conflicts between tags with the same name, the engine automatically applies tag override rules, prioritizing tags with higher priority, such as user-defined tags. The processed tags are then added to the first tag set.

[0085] When a resource undergoes an update event, the engine captures this change and collects the updated resource tag information. Similar to the resource creation event handling, for potential tag conflicts with the same name in update events, the engine follows the same tag overriding rules, prioritizing the tag from the higher-level source, and the updated tag is then integrated into the first tag set.

[0086] When an application anomaly is detected, such as a payment service call failure, in addition to tracing the call chain to identify the possible source of the failure, the engine dynamically creates and adds fault tags to the application and resources associated with the anomaly. When a failure is detected in a call from service A to service B, the engine automatically attaches fault tags identifying the source of the failure (e.g., caused_by: service-a (caused cause: service A), fault_initiator: true (fault initiator: true)) to the temporary context of service B and its related underlying resources (such as Pods, containers, and even dependent database instances). These fault tags then become part of the first tag set to construct the potential impact path of the failure.

[0087] This embodiment captures changes in resource status in real time during resource creation and updating, and in handling application anomalies, reasonably handles tag conflicts, and accurately records fault information. This lays the foundation for building an association view that reflects the real-time dependencies of a distributed system and improving the efficiency and accuracy of fault location and repair.

[0088] Optionally, in the distributed system operation and maintenance method provided in this application embodiment, locating the target root cause node of alarm information from the first graph based on the tag combination of alarm target includes: determining the abnormal nodes covered by the tag combination of alarm target from the first graph; determining the upstream nodes of the abnormal nodes from the first graph to obtain a set of candidate root cause nodes; calculating the suspicious score of each candidate root cause node in the set of candidate root cause nodes; performing multi-dimensional verification on each candidate root cause node in the set of candidate root cause nodes to obtain verification results; and filtering out the target root cause node from the candidate root cause nodes based on the suspicious score and verification results.

[0089] For example, when an alarm message is received, such as a payment service delay exceeding a threshold (e.g., 5 seconds), the engine first extracts the key tags of the alarm target (payment service), such as biz_unit=payment, zone=ap-east-1c, service_type=backend, etc.

[0090] Next, using the maintained first graph, all nodes covered by these tag combinations are identified as anomalous nodes. Coverage means that a node in the first graph has the exact same tag combination as the alarm target, or partially overlapping tags. Anomalous nodes are directly or indirectly associated with the alarm target. For example, an anomalous node can be a service node or resource node whose monitoring metrics related to the alarm target in terms of space (same zone, node, cluster), time (recent, such as within 5 minutes), or logic (same owner, deploy_id, upstream and downstream of the call chain) exceed the threshold.

[0091] To further narrow down the source of the fault, starting from the abnormal node, the engine traces upstream along the dependency edges (depends on) and fault propagation edges (caused by) in the graph. By analyzing the call chain relationships between nodes, all potential upstream influencing nodes are identified, forming a set of candidate root cause nodes. For example, when a payment service experiences a delay, the engine not only checks the payment service itself but also its dependencies (graph edges), searching upwards along the call chain for dependent database services. It checks the status / metrics of these database services, finding an abnormal connection pool wait time (1500ms > 200ms). It then locates the database container providing the service and further penetrates to examine the metrics of its running ECS-001 node, finding that CPUsteal (Central Processing Unit time stolen by hypervisor or lower priority tasks) = 40% (exceeding the threshold), thus including all nodes involved in the process in the candidate root cause node set.

[0092] Then, for each node in the candidate root cause node set, its suspicion score is calculated based on multi-dimensional information, including but not limited to the deviation of the node's monitoring indicators, the historical reproducibility rate of the failure mode, and the time correlation with the alarm target. For example, if an ECS instance shows an abnormally high CPU steal indicator, and other services in the same region and time period also show similar performance degradation, then the instance has a high suspicion score because it is likely to be the main cause of the alarm.

[0093] To further improve the accuracy of root cause localization, multi-dimensional verification is performed on each node in the candidate root cause node set, including label clustering verification, historical reproducibility verification, propagation depth verification, propagation breadth verification, and health comparison verification. For example, if multiple services are found to have encountered performance bottlenecks on ECS instances in the same region, and the historical knowledge base indicates that ECS instances in that region have caused similar failures multiple times, then that ECS instance is highly likely to be the source of the failure.

[0094] Finally, based on the calculated suspiciousness score and the results of multi-dimensional verification, the target root cause node with the highest confidence level was selected, thereby ensuring the accuracy and rationality of fault location and avoiding misdiagnosis that might be caused by a single indicator decision. For example, after comprehensively considering the characteristics of suspiciousness score, high clustering, high recurrence rate and wide propagation, resource contention (CPU Steal) of ECS instance (ID: i-0a1b2c3d) was the most likely cause of payment service delay, with a confidence level of 0.92.

[0095] This embodiment extracts abnormal nodes from the first graph, determines the set of candidate root cause nodes by tracing the fault propagation path in reverse based on the abnormal nodes, quantifies the importance of the candidate root cause nodes by suspicious scores and verifies them from multiple angles, which can quickly pinpoint the real source of the fault, improve the accuracy and response efficiency of fault location, and reduce the time cost of fault investigation and recovery.

[0096] Optionally, in the distributed system operation and maintenance method provided in this application embodiment, calculating the suspiciousness score of each candidate root cause node in the candidate root cause node set includes: for a candidate root cause node, calculating the product of the basic weight of the lost resource dimension and the indicator value of the resource dimension to obtain a first suspiciousness score, wherein the basic weight of the resource dimension represents the degree of resource contention; calculating the product of the basic weight of the application dimension and the indicator value of the application dimension to obtain a second suspiciousness score, wherein the basic weight of the application dimension represents the degree of application abnormality; and calculating the suspiciousness score of the candidate root cause node based on the first suspiciousness score and the second suspiciousness score.

[0097] For example, when calculating the first suspicion score for the resource dimension, a base weight for the resource dimension is first determined for each candidate root cause node to reflect the estimated strength of the impact of resource contention on the system. For instance, the base weight for the resource dimension of an ECS instance (preset or learned based on the fault type) is 0.7, indicating that physical resource contention is a relatively common cause of failure.

[0098] Then, obtain the resource dimension indicator values ​​of the candidate nodes. For example, for ECS instance i-0a1b2c3d, read the indicators related to resource contention in its real-time monitoring data, such as propagation strength (the propagation path length from the candidate point to the alarm point) and time correlation (the proximity of the time of the anomaly at the candidate point to the alarm time).

[0099] The base weight of the resource dimension is then multiplied by the indicator value of the resource dimension to obtain the first suspicion score. This score quantifies the likelihood that the candidate node is the source of the failure in terms of resource contention. For example, for ECS instance i-0a1b2c3d, its CPU steal indicator value is 40%, then the first suspicion score ECS-001 score = base weight (physical resource contention). Transmission intensity Time correlation ≈ 0.7 0.93 0.98 ≈ 0.64.

[0100] For example, when calculating the second suspicion score for the application dimension, a basic weight for the application dimension is first determined for each candidate root cause node, reflecting the potential impact of the application's own abnormality on the system. Suppose that the Payment-Service itself causes latency abnormalities due to code logic problems, its basic weight for the application dimension can be set to 0.5, indicating that application bugs (defects) are also common sources of failure.

[0101] Then, the analysis engine acquires application-level metrics, starting from the alert target. These metrics are related to the application anomaly, such as metric deviation (the severity of the candidate point's anomaly, e.g., the degree to which application latency exceeds the normal range), and the strength of the underlying propagation chain (the length of the propagation path from the candidate point to the alert point, and the confidence level of each edge on the path; confidence is determined based on call chain success rate and historical data; the shorter the path and the higher the confidence, the greater the strength). The application-level base weight is then multiplied by the application-level metric value to obtain the second suspicion score. For example, the Payment-Service's own score = base weight (application error). The strength of the propagation chain without a bottom layer is approximately 0.5. 1.0 = 0.5.

[0102] Finally, based on the first and second suspicion scores, a comprehensive suspicion score is calculated for each candidate root cause node. Specifically, this can be achieved by weighted summation, ensuring the score comprehensively reflects the overall likelihood of the candidate node being the source of the fault. For example:

[0103] This embodiment calculates a suspicious score for each candidate root cause node in the candidate root cause node set, combining two key dimensions: resource contention and application anomaly. It locates the root cause from the application layer to the underlying physical machine, and calculates the probability of each candidate node being the source of the fault, laying the foundation for improving the accuracy and efficiency of fault location.

[0104] To enhance the credibility of root cause nodes, optionally, in the operation and maintenance method of the distributed system provided in this application embodiment, multi-dimensional verification of each candidate root cause node in the candidate root cause node set includes at least one of the following: for a candidate root cause node, determining whether the total number of abnormal services and resources under the label combination of the candidate root cause node reaches a quantity threshold; determining whether the proportion of faults associated with historical label combinations similar to the label combination of the candidate root cause node reaches a fault proportion threshold; determining whether the candidate root cause node propagates along the first graph and affects services or resources at two or more levels; determining whether the difference between the candidate root cause node and healthy nodes under the same label reaches a difference threshold.

[0105] Multi-dimensional verification can include label clustering verification, used to assess whether a candidate root cause node belongs to an environment with high clustering anomalies. This can be determined by whether the total number of services or resources exhibiting anomalies under the same set of key labels reaches a pre-set threshold. For example, traversing the first graph, the number of all anomalous services and resources under similar label combinations to the candidate root cause node (e.g., zone=ap-east-1c) is counted. Within a specific availability zone, it is checked whether other services or resources have also reported performance issues or alarms. If this total exceeds the threshold (e.g., services from three different businesses within the zone are all anomalous), it is considered to have high clustering, indicating that the candidate node is likely a problem with underlying shared resources (e.g., the physical machine or network of the zone).

[0106] Multi-dimensional verification can include historical recurrence rate verification, which uses historical knowledge base queries to examine whether the proportion of failures caused by historical label combinations similar to the candidate root cause node in past failure events reaches a failure proportion threshold, thus using historical data to support the accuracy of current failure localization. For example, the key labels of the current candidate node can be input into the knowledge platform to search for the proportion of similar label combinations in past failures. For instance, if, under the condition of zone=ap-east-1c and abnormal CPU steal metric, historical failure records show that 90% of cases are related to underlying hardware resource contention, then the correctness of the current judgment is strongly supported.

[0107] Multi-dimensional verification can include propagation depth / breadth verification to determine whether the current fault has propagated along dependency and propagation edges in the first graph to services or resources at least two levels higher, thus measuring the impact range of the fault source. For example, starting from the candidate root cause node, tracing the associated paths in the graph, checking whether the fault has propagated to directly downstream services, and then further extending to the next level of associated services or resources. If the impact range of the fault does indeed exceed a single level (payment → order → risk control), it indicates a wide propagation range, and the candidate node may be a problem in the underlying or core services, thus having a higher confidence level.

[0108] Multi-dimensional verification can include health comparison verification, which compares whether the differences between candidate root cause nodes and healthy nodes reach a difference threshold under the same key dimensions, to help determine the true fault state of candidate nodes. For example, select healthy nodes with the same label as candidate root cause nodes (e.g., zone=ap-east-1c) but that have not reported anomalies, and compare their core indicators and label attributes. If the CPU steal indicator of the candidate node is found to be significantly higher than that of the healthy node, and this difference exceeds a preset difference threshold, this comparison result will serve as additional evidence to support the judgment that the candidate node is the current source of the fault.

[0109] This embodiment performs multi-dimensional verification on each node in the candidate root cause node set to ensure the accuracy and confidence of fault location. Specifically, label clustering verification helps identify common environmental problems, historical recurrence rate verification uses historical data to strengthen the basis for current judgment, propagation depth / breadth verification reveals the scope of influence of the fault source, and health comparison verification accurately points out the differences between candidate nodes and other nodes. By comprehensively using multiple dimensions to verify candidate nodes, support for fault analysis from different perspectives is provided, ensuring the reliability of the final location result.

[0110] To quickly repair faults while ensuring operational safety, optionally, in the distributed system operation and maintenance method provided in this application embodiment, querying operation and maintenance information based on the target root cause node and performing operation and maintenance operations based on the operation and maintenance information includes: querying the operation and maintenance knowledge base based on the tag combination of the target root cause node to obtain operation and maintenance information, wherein the operation and maintenance knowledge base includes different preset tag combinations and preset operation and maintenance information associated with the preset tag combinations; calculating the impact degree value of the operation and maintenance operation in the operation and maintenance information, and performing the operation and maintenance operation if the impact degree value is less than the impact degree value threshold; generating an approval request and sending the approval application to the approval end if the approval is received from the approval end, and performing the operation and maintenance operation.

[0111] For example, first query the predefined operation and maintenance knowledge base based on the key tags carried by the target root cause location result, such as the fault resource type resource_type=ecs_instance (resource type: Elastic Cloud Server Instance), the specific fault mode fault_type=cpu_steal (fault type: CPU stealing time) or db_type=dbA (database type: database A), and fault_detail=conn_pool_full (fault details: connection pool is full).

[0112] For example, if the target root cause node is ECS instance i-0a1b2c3d, and the fault mode is db_type=dbA and fault_detail=conn_pool_full, it will access the operations and maintenance knowledge base to find the preset operations and maintenance information corresponding to the tag combination {db_type=dbAAND fault_detail=conn_pool_full}. The operations and maintenance knowledge base stores various preset tag combinations and their associated preset operations and maintenance information, including a list of suggested actions for specific fault scenarios. An example of the knowledge base entry structure is: [Matching condition: Tag combination] => [Suggested action list].

[0113] After retrieving operational information from the knowledge base, the impact value of the operational operation is calculated. This impact value comprehensively considers factors such as the risk level of the operation, the expected scope of its impact on the business (e.g., based on the first graph, affecting the payment service and all its direct callers), and recovery time. For example, for a CPU Stealth failure of an ECS instance, an emergency expansion of the connection count (executing the SQL (Structured Query Language) command SET GLOBAL max_connections=200, i.e., setting the global maximum number of connections to 200) carries low risk, is a temporary operation with minimal impact, and can be executed directly. Another example is restarting the connection pool service, which carries medium risk and requires a restart; its impact value exceeds the preset impact threshold. Yet another example is vertically scaling up the database instance specifications; due to cost changes and potential service interruptions, this carries high risk, and its impact value exceeds the preset impact threshold.

[0114] If the impact level of a certain operation and maintenance (O&M) operation is determined to be greater than or equal to the impact level threshold, the O&M system will not execute the operation immediately. Instead, it will generate an approval request and send it to the designated approval authority. The approval request includes an analysis summary of the target root cause node, details of the proposed O&M operation, the expected scope of impact, and a rollback plan for the operation, ensuring that the approval authority fully understands the necessity and potential consequences of the operation.

[0115] Upon receiving approval, the operations and maintenance system executes operations atomically, ensuring idempotency and rollback capability at each step. For example, when the database connection pool is full, the system calls the cloud platform interface or Kubernetes interface service to temporarily remove the target database instance from the load balancer pool to prevent new requests from entering. Then, a repair atomic operation is performed, safely executing the scaling command `SET GLOBAL max_connections=200` on the database instance. Next, a verification atomic operation is performed, executing a pre-defined health check script (checking for a decrease in connection count and normal service response). After the health check passes, a recovery atomic operation is performed, calling the interface to re-add the database instance to the load balancer pool. The system can also monitor to confirm that service metrics have returned to normal and perform status tracking and notifications, providing real-time feedback on the execution status (in progress, successful, failed) to the collaboration platform, allowing relevant personnel to stay informed of the progress.

[0116] The closed-loop execution strategy in this embodiment matches appropriate operation and maintenance operations based on fault label combinations, which greatly improves the targeting and efficiency of operations. The introduction of an approval process effectively controls the execution of high-risk operation and maintenance operations, ensuring the safety and controllability of operation and maintenance operations, and reducing the impact of human error and uncertainties on the stable operation of the distributed system.

[0117] Optionally, in the distributed system operation and maintenance method provided in this application embodiment, after extracting the tag combination of alarm targets, the method further includes: extracting key information in the handling process of historical faults, wherein the key information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path, and timeline; converting the key information into elements and relationships, determining the elements as nodes, and determining the relationships as edges to obtain a second graph, wherein the nodes in the second graph include at least one of the following: fault mode nodes, root cause nodes, and operation and maintenance nodes; matching similar historical fault modes in the second graph according to the tag combination of alarm targets, obtaining fault information of historical fault modes, and pushing fault information to the operation and maintenance end, wherein the fault information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path, and timeline.

[0118] For example, during the fault handling process (from alarm generation to recovery), key information about historical faults can be extracted from data sources at each stage. This information includes at least fault characteristics, fault analysis process information, fault handling process information, fault propagation path, and timeline.

[0119] For example, for past payment delay alerts, the following key information can be extracted: Fault characteristics: [service=payment, error_type=timeout, root_cause=dbA_conn_pool, zone=ap-east-1c]. Analysis process: root cause localization path, weight calculation parameters, and multi-dimensional verification results. Handling actions: actual executed operation commands, parameters, and execution results (success / failure / rollback). Propagation path: payment service → order service → risk control service determined based on the fault propagation graph snapshot (nodes, edges, key status labels), down to the underlying ECS ​​instance. Timeline: key event timestamps on the complete timeline from alert generation, analysis, approval, execution, and recovery. Collaboration information: approval records, manual intervention records, and communication summaries.

[0120] Next, this key information is transformed into elements and relationships in the second graph. Entities in the information are converted into nodes in the graph; for example, insufficient database connection pool becomes a failure mode node, ECS CPU Steal becomes a root cause node, and SQLSET GLOBAL max_connections becomes an operation node. Relationships in the information are converted into edges, building logical bridges between nodes.

[0121] Then, a second graph is constructed, transforming historical fault information into an easily understandable and applicable graph format to facilitate rapid identification of similar fault modes and their effective handling solutions. Nodes in the second graph encompass fault modes, specific root causes, and operational repair operations. For example, adding or updating fault mode nodes (e.g., insufficient database connection pool) and associating them with feature tags; adding or updating root cause nodes (e.g., CPU Steal) and associating them with feature tags; adding or updating repair operation nodes (e.g., SQL SET GLOBAL max_connections) and recording their applicable conditions, risks, and effects. Edges establish the relationships between these nodes; for example, fault mode - [hasRootCause] -> root cause, fault mode - [resolvedBy] -> repair operation, root cause - [locatedOn] -> resource type. While setting nodes and edges, the weights or confidence levels of nodes / edges can also be updated, for example, incrementing the success count of a repair operation by 1. Furthermore, fault propagation path patterns can be recorded as subgraph templates.

[0122] After identifying the alarm information of the target, and matching a highly similar historical fault pattern in the second graph based on the combination of the alarm target's tags, fault information is pushed to the operations and maintenance team. This information includes not only a basic description of the fault characteristics, i.e., a summary of the matched historical fault cases (e.g., time, root cause, propagation impact), but also fault analysis process information (including relevant root cause analysis clues, such as "Historically, 90% of this phenomenon is caused by underlying CPU contention, and it is recommended to prioritize checking the CPU Steal metric of the host ECS"), fault handling process information (suggested successful repair solutions and operational details), and relevant historical propagation paths and timelines, providing operations and maintenance personnel with a clear framework for fault handling.

[0123] In addition, it can provide an interface for operations and maintenance personnel to query historical faults and solutions by tags, services, root cause types, etc.

[0124] By constructing and utilizing a second graph of historical failure modes, we can not only respond quickly to current alarms, but also provide handling suggestions based on historical knowledge. This accelerates the fault analysis and recovery process, improves the efficiency and accuracy of fault handling, and enhances the operation and maintenance management level and system stability of distributed systems.

[0125] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0126] According to an embodiment of this application, an operation and maintenance system for a distributed system is also provided. Figure 6 This is a schematic diagram of the operation and maintenance system of the distributed system according to the embodiments of this application, such as... Figure 6 As shown, the operation and maintenance system includes:

[0127] Dynamic Tag Propagation Engine: Constructs the first graph of objects (resources and applications) across application and resource levels. Specifically, it first collects or defines tags from different levels and sources, and then maintains the tag status and object associations in real time to generate a dynamic object association view, which is the first graph.

[0128] Intelligent Root Cause Analysis Engine: Based on the first graph, it performs multi-dimensional root cause localization. Specifically, it first extracts alarm features and associates them with the context to obtain abnormal nodes. Then, it performs root cause localization analysis based on the abnormal nodes and the first graph to obtain candidate root causes. It performs multi-dimensional cross-validation on the candidate root causes and determines the target root cause based on the validation results.

[0129] Closed-loop execution engine: Implements root cause-based safe remediation operations. Specifically, it first performs intelligent matching of remediation operations, and then executes the remediation operations while ensuring safety.

[0130] Collaboration and Knowledge Platform: Supports collaborative handling and enables the accumulation and reuse of fault knowledge. Specifically, fault knowledge is first automatically extracted, then a fault knowledge graph is created based on the fault knowledge to obtain a second graph, and then fault information with fault patterns similar to the current fault is determined based on the second graph, and the fault information is intelligently pushed and reused.

[0131] This embodiment constructs a first graph of objects across application and resource levels, performs multi-dimensional root cause localization based on the first graph, implements root cause-based security repair operations, and realizes fault knowledge accumulation and reuse, thereby achieving the effect of quickly and accurately locating faults in the distributed system and efficiently and accurately handling faults.

[0132] Example 2

[0133] This application also provides an operation and maintenance device for a distributed system. It should be noted that this device can be used to execute the operation and maintenance method for a distributed system provided in this application. The following describes the operation and maintenance device for a distributed system provided in this application.

[0134] According to an embodiment of this application, an apparatus for implementing the above-described operation and maintenance method for a distributed system is also provided. Figure 7 This is a schematic diagram of an operation and maintenance device for a distributed system according to an embodiment of this application, such as... Figure 7 As shown, the device includes:

[0135] Extraction unit 701 is used to extract the tag combination of the alarm target when the alarm target of the distributed system generates alarm information, wherein the alarm target is an object on the distributed system, and the object is a resource in the distributed system or an application running on the distributed system.

[0136] The positioning unit 702 is used to locate the target root cause node of alarm information from the first graph based on the combination of alarm target labels. Each node in the first graph represents an object on the distributed system, the attributes of each node are determined by the label of the object, and each edge represents the association relationship between two objects.

[0137] The query unit 703 is used to query operation and maintenance information based on the target root cause node and perform operation and maintenance operations based on the operation and maintenance information, wherein the operation and maintenance operations are used to eliminate the fault indicated by the alarm information.

[0138] The distributed system operation and maintenance device provided in this application embodiment, through an extraction unit 701, extracts the tag combination of the alarm target when an alarm target in the distributed system generates alarm information. The alarm target is an object on the distributed system, and the object is a resource or an application running on the distributed system. A location unit 702 locates the target root cause node of the alarm information from a first graph based on the tag combination of the alarm target. Each node in the first graph represents an object on the distributed system, the attributes of each node are determined by the object's tag, and each edge represents the association between two objects. A query unit 703 queries operation and maintenance information based on the target root cause node and performs operation and maintenance operations based on the operation and maintenance information. The operation and maintenance operations are used to eliminate the fault indicated by the alarm information, solving the problem of low accuracy and efficiency in fault handling of distributed system operation and maintenance schemes in related technologies. By extracting the tag combination of the alarm target, locating the target root cause node of the alarm information from the first graph based on this, matching the operation and maintenance information, and performing operation and maintenance operations, the device achieves the effect of quickly and accurately locating faults in the distributed system and efficiently and accurately handling faults.

[0139] Optionally, the operation and maintenance device for the distributed system provided in this application embodiment further includes: a first graph construction unit, comprising: an acquisition module, configured to acquire tags of objects on the distributed system from multiple sources to obtain a tag set; an overlay processing module, configured to, when the same object in the tag set has tags with the same name from different sources, perform overlay processing on the tags with the same name from different sources for the same object according to the tag overlay rule to obtain a first tag set, wherein the tag overlay rule indicates that for tags with the same name from different sources, the tag from the higher-level source is used to overlay the tag from the lower-level source; an update module, configured to, when a resource change event occurs, update the first tag set based on the resource change event and the tag overlay rule to obtain a second tag set, wherein the resource change event includes at least one of the following: a resource creation event, a resource update event, and an application exception event; and a construction module, configured to determine nodes based on objects on the distributed system, determine the attributes of nodes based on the tags carried by the objects, and determine edges based on the association relationships between objects to obtain a first graph.

[0140] Optionally, in the distributed system operation and maintenance device provided in this application embodiment, the acquisition module includes: a first acquisition submodule, used to acquire metadata of the cloud infrastructure of the distributed system and use the metadata of the cloud infrastructure as a first type of tag, wherein the first type of tag is a resource tag; a second acquisition submodule, used to acquire pipeline information in the deployment pipeline of application development and use the pipeline information as a second type of tag, wherein the second type of tag is a resource tag and an application tag; a third acquisition submodule, used to acquire business semantic tags defined by the resource manifest file and use the business semantic tags as a third type of tag, wherein the third type of tag is a resource tag and an application tag; a determination submodule, used to determine an initial tag set based on at least one of the first type of tag, the second type of tag, and the third type of tag; and an enhancement submodule, used to verify the tags in the initial tag set, perform enhancement operations on the tags that fail verification, and obtain a tag set, wherein the enhancement operation indication includes one of the following: tag completion operation and tag correction operation.

[0141] Optionally, in the operation and maintenance device of the distributed system provided in the embodiments of this application, the overlay processing module includes: a first processing submodule, used to overlay tags other than user-defined tags in the same-name tag by using user-defined tags when the same-name tags from different sources include user-defined tags; a second processing submodule, used to overlay tags other than object-defined tags in the same-name tag by using object-defined tags when the same-name tags from different sources do not include user-defined tags but include the object's own tags; and a third processing submodule, used to overlay other tags in the same-name tag according to the tag of the level closest to the object's level when the same-name tags from different sources do not include user-defined tags or the object's own tags but include at least two tags inherited by the object from the hierarchical architecture, where other tags are tags other than the tag of the level closest to the object's level.

[0142] Optionally, in the operation and maintenance device of the distributed system provided in the embodiments of this application, the update module includes: a first update submodule, used to obtain the tag of the created resource when a resource creation event occurs, and when there are tags with the same name from different sources for the same resource after creation, to perform overwriting processing on the tags with the same name from different sources for the same resource according to the tag overwriting rule, and add the overwritten tags to a first tag set; a second update submodule, used to obtain the tag of the updated resource when a resource update event occurs, and when there are tags with the same name from different sources for the same resource after update, to perform overwriting processing on the tags with the same name from different sources for the same resource according to the tag overwriting rule, and add the overwritten tags to a first tag set; a third update submodule, used to determine the application and associated resources associated with the application exception event when an application exception event occurs, and to add fault tags to the associated application and associated resources, and add the fault tags to a first tag set.

[0143] Optionally, in the distributed system operation and maintenance device provided in this application embodiment, the positioning unit 702 includes: a first determining module, used to determine the abnormal nodes covered by the tag combination of the alarm target from the first graph; a second determining module, used to determine the upstream nodes of the abnormal nodes from the first graph to obtain a candidate root cause node set; a calculation module, used to calculate the suspicious score of each candidate root cause node in the candidate root cause node set; a verification module, used to perform multi-dimensional verification on each candidate root cause node in the candidate root cause node set to obtain a verification result; and a filtering module, used to filter out the target root cause node from the candidate root cause nodes according to the suspicious score and the verification result.

[0144] Optionally, in the operation and maintenance device of the distributed system provided in this application embodiment, the calculation module includes: a first calculation submodule, used to calculate, for a candidate root cause node, the product of the basic weight of the lost resource dimension and the indicator value of the resource dimension to obtain a first suspicion score, wherein the basic weight of the resource dimension represents the degree of resource contention; a second calculation submodule, used to calculate, for a candidate root cause node, the product of the basic weight of the application dimension and the indicator value of the application dimension to obtain a second suspicion score, wherein the basic weight of the application dimension represents the degree of application abnormality; and a third calculation submodule, used to calculate the suspicion score of the candidate root cause node based on the first suspicion score and the second suspicion score.

[0145] Optionally, in the distributed system operation and maintenance device provided in the embodiments of this application, the verification module includes at least one of the following: a first judgment submodule, used to determine whether the total number of abnormal services and resources under the label combination of a candidate root cause node reaches a quantity threshold; a second judgment submodule, used to determine whether the proportion of faults associated with historical label combinations similar to the label combination of the candidate root cause node reaches a fault proportion threshold; a third judgment submodule, used to determine whether the candidate root cause node has spread along the first graph to affect services or resources at two or more levels; and a fourth judgment submodule, used to determine whether the difference between the candidate root cause node and healthy nodes under the same label reaches a difference threshold.

[0146] Optionally, in the operation and maintenance device of the distributed system provided in this application embodiment, the query unit 703 includes: a query module, used to query the operation and maintenance knowledge base according to the tag combination of the target root cause node to obtain operation and maintenance information, wherein the operation and maintenance knowledge base includes different preset tag combinations and preset operation and maintenance information associated with the preset tag combinations; a first execution module, used to calculate the impact degree value of the operation and maintenance operation in the operation and maintenance information, and execute the operation and maintenance operation when the impact degree value is less than the impact degree value threshold; and a second execution module, used to generate an approval request when the impact degree value is greater than or equal to the impact degree value threshold, and send the approval application to the approval end, and execute the operation and maintenance operation when the approval approval message is received from the approval end.

[0147] Optionally, in the distributed system operation and maintenance device provided in this application embodiment, the device further includes: an extraction unit, used to extract key information in the handling process of historical faults, wherein the key information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path, and timeline; a determination unit, used to convert the key information into elements and relationships, determine the elements as nodes, and determine the relationships as edges to obtain a second graph, wherein the nodes in the second graph include at least one of the following: fault mode nodes, root cause nodes, and operation and maintenance nodes; and a matching unit, used to match similar historical fault modes in the second graph according to the tag combination of the alarm target, obtain fault information of the historical fault modes, and push fault information to the operation and maintenance end, wherein the fault information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path, and timeline.

[0148] It should be noted that the above-mentioned units and modules correspond to the steps in Embodiment 1, and the instances and application scenarios implemented by the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above-mentioned modules may also be part of the device and can run in the computer terminal 10 provided in Embodiment 1.

[0149] Example 3

[0150] Embodiments of this application may provide an electronic device. Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 8 As shown, the electronic device may include: one or more ( Figure 8 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0151] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] The processor can access information and applications stored in memory via a transmission device to execute steps of the operation and maintenance methods for a distributed system.

[0153] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs (tablet computers), and other terminal devices. Figure 8 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 8 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 8 The different configurations shown.

[0154] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0155] Example 4

[0156] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the operation and maintenance method of the distributed system provided in Embodiment 1.

[0157] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0158] This application also provides a computer program product, which, when executed on a data processing device, is suitable for performing steps of a distributed system operation and maintenance method.

[0159] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0160] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0165] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for operating and maintaining a distributed system, characterized in that, include: When an alarm target in a distributed system generates alarm information, the tag combination of the alarm target is extracted, wherein the alarm target is an object on the distributed system, and the object is a resource in the distributed system or an application running on the distributed system; Based on the tag combination of the alarm target, the target root cause node of the alarm information is located in the first graph, wherein each node in the first graph represents an object on the distributed system, the attributes of each node are determined by the tag of the object, and each edge represents the association relationship between two objects. The system queries the operation and maintenance information based on the target root cause node and performs operation and maintenance operations based on the operation and maintenance information, wherein the operation and maintenance operations are used to eliminate the fault indicated by the alarm information.

2. The method according to claim 1, characterized in that, The first map is constructed in the following way: Obtain tags for objects on the distributed system from multiple sources to obtain a tag set; When there are identical tags with the same name from different sources for the same object in the tag set, the identical tags with the same name from different sources for the same object are overridden according to the tag overriding rule to obtain the first tag set. The tag overriding rule indicates that for identical tags with the same name from different sources, the tag from the higher-level source is used to overridden the tag from the lower-level source. In the event of a resource change event, the first tag set is updated based on the resource change event and the tag coverage rule to obtain a second tag set, wherein the resource change event includes at least one of the following: resource creation event, resource update event, and application exception event; The first graph is obtained by determining nodes based on objects in the distributed system, determining node attributes based on labels carried by objects, and determining edges based on the relationships between objects.

3. The method according to claim 2, characterized in that, Tags of objects on the distributed system are obtained from multiple sources, resulting in a tag set including: Obtain the metadata of the cloud infrastructure of the distributed system, and use the metadata of the cloud infrastructure as a first type of tag, wherein the first type of tag is a tag of the resource; Obtain pipeline information from the application development deployment pipeline, and use the pipeline information as a second type of tag, wherein the second type of tag is a resource tag and an application tag; Obtain the business semantic tags defined by the resource manifest file, and use the business semantic tags as a third type of tag, wherein the third type of tag is a tag for resources and a tag for applications; An initial tag set is determined based on at least one of the first type of tag, the second type of tag, and the third type of tag; The tags in the initial tag set are verified, and the tags that fail verification are enhanced to obtain the tag set. The enhancement operation instruction includes one of the following: tag completion operation and tag correction operation.

4. The method according to claim 2, characterized in that, Overriding tags with the same name from different sources for the same object according to tag overriding rules includes: In cases where the same-name tags from different sources include user-defined tags, the user-defined tags shall cover the tags other than the user-defined tags among the same-name tags; If the same-name tags from different sources do not include the user-defined tags but include the object's own tags, then the tags other than the object's own tags in the same-name tags are overridden by the object's own tags. If the same-name tags from different sources do not include the user-defined tags, the object's own tags, or at least two tags inherited by the object from the hierarchical structure, then other tags among the same-name tags are overridden by the tag of the hierarchy closest to the object's hierarchy, where the other tags are tags other than the tag of the hierarchy closest to the object's hierarchy.

5. The method according to claim 2, characterized in that, In the event of a resource change event, updating the first tag set based on the resource change event and the tag coverage rule includes: When the resource creation event occurs, the tags of the created resource are obtained. If there are tags with the same name from different sources for the same resource after creation, the tags with the same name from different sources for the same resource are overwritten according to the tag overwriting rules, and the overwritten tags are added to the first tag set. When the resource update event occurs, the updated resource's tag is obtained. If there are tags with the same name from different sources for the same updated resource, the tags with the same name from different sources for the same resource are overwritten according to the tag overwriting rule, and the overwritten tags are added to the first tag set. In the event of the application exception, the application and associated resources associated with the application exception are determined, and fault tags are added to the associated application and associated resources, and the fault tags are added to the first tag set.

6. The method according to claim 1, characterized in that, Locating the target root cause node of the alarm information from the first map based on the tag combination of the alarm target includes: Determine the abnormal nodes covered by the tag combination of the alarm target from the first map; The upstream nodes of the abnormal nodes are determined from the first graph to obtain a set of candidate root cause nodes; Calculate the suspicious score for each candidate root cause node in the candidate root cause node set; Each candidate root cause node in the candidate root cause node set is verified in multiple dimensions to obtain the verification results; The target root cause node is selected from the candidate root cause nodes based on the suspicious scores and verification results.

7. The method according to claim 6, characterized in that, Calculating the suspicious score for each candidate root cause node in the candidate root cause node set includes: For a candidate root cause node, the product of the basic weight of the lost resource dimension and the index value of the resource dimension is calculated to obtain the first suspicion score, wherein the basic weight of the resource dimension represents the degree of resource contention. The product of the basic weight of the application dimension and the index value of the application dimension of the candidate root cause node is calculated to obtain the second suspicion score, wherein the basic weight of the application dimension represents the degree of abnormality of the application. The suspicion score of the candidate root cause node is calculated based on the first suspicion score and the second suspicion score.

8. The method according to claim 6, characterized in that, Multi-dimensional verification of each candidate root cause node in the candidate root cause node set includes at least one of the following: For a candidate root cause node, determine whether the total number of services and resources that are abnormal under the tag combination of the candidate root cause node reaches the quantity threshold. Determine whether the proportion of faults associated with historical label combinations similar to the label combinations of the candidate root cause nodes reaches the fault proportion threshold. Determine whether the candidate root cause node propagates along the first graph and affects services or resources at two or more levels; Determine whether the difference between the candidate root cause node and the healthy node under the same label reaches the difference threshold.

9. The method according to claim 1, characterized in that, Querying maintenance information based on the target root cause node and performing maintenance operations based on the maintenance information includes: The operation and maintenance information is obtained by querying the operation and maintenance knowledge base based on the tag combination of the target root cause node. The operation and maintenance knowledge base includes different preset tag combinations and preset operation and maintenance information associated with the preset tag combinations. Calculate the impact degree value of the operation and maintenance operation in the operation and maintenance information, and execute the operation and maintenance operation if the impact degree value is less than the impact degree value threshold; If the impact level value is greater than or equal to the impact level value threshold, an approval request is generated and the approval application is sent to the approval end. If the approval is approved by the approval end, the operation and maintenance operation is executed.

10. The method according to claim 1, characterized in that, After extracting the tag combination of the alarm target, the method further includes: Extract key information from the handling process of historical faults, wherein the key information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path and timeline; The key information is transformed into elements and relationships. Elements are defined as nodes and relationships are defined as edges to obtain a second graph. The nodes in the second graph include at least one of the following: fault mode nodes, root cause nodes, and operation and maintenance nodes. Based on the tag combination of the alarm target, match similar historical fault patterns in the second graph, obtain the fault information of the historical fault patterns, and push the fault information to the operation and maintenance terminal. The fault information includes at least one of the following: fault characteristics, fault analysis process information, fault handling process information, propagation path, and timeline.

Images