A fault positioning method, device, equipment and storage medium

Abstract

The application discloses a fault positioning method and device, equipment and storage medium, and relates to the technical field of operation and maintenance, which comprises the following steps: when a service alarm event is received, a target service corresponding to the service alarm event is determined; a target resource ownership chain corresponding to the target service is queried from a current resource ownership graph; the resource ownership graph is used for recording hierarchical ownership relationships among service, physical resources and virtual resources; a corresponding checking task is generated for each resource object in the target resource ownership chain based on a preset checking strategy, and each checking task is executed to obtain a corresponding checking result; the checking result is analyzed based on a preset root cause analysis rule to determine a fault root cause, and a root cause link generated based on the fault root cause is sent to a preset front end. The application realizes the automation of fault positioning, can generate checking tasks and analyze checking results according to a unified process, forms a standardized and reproducible fault positioning process, and reduces the operation and maintenance threshold.

Description

Technical Field

[0001] This invention relates to the field of operation and maintenance technology, and in particular to a fault location method, device, equipment and storage medium. Background Technology

[0002] In the complex IT environments of cloud computing and virtualization, the deployment and operation of business systems exhibit high complexity and dynamism. Business services typically rely on multi-layered resource stacks, including virtual resources and underlying physical resources. When business systems fail, such as slow service response or inaccessibility, quickly and accurately locating the root cause of the failure becomes crucial to ensuring business continuity.

[0003] Traditional fault location solutions rely on monitoring systems that typically monitor individual resource objects in isolation. When an alarm occurs, maintenance personnel manually review documentation or rely on their personal experience to identify resource dependencies, then manually log into multiple devices to check processes, logs, resource utilization, and other information to locate the fault. This approach suffers from drawbacks such as missing resource relationships, cumbersome and unautomated troubleshooting processes, and over-reliance on expert experience.

[0004] Therefore, the aforementioned technical problems urgently need to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a fault location method, apparatus, device, and storage medium that automates fault location, generates inspection tasks and analyzes inspection results according to a unified process, forming a standardized and reproducible fault location process, thus lowering the maintenance threshold. The specific solution is as follows: The first aspect of this application provides a fault location method, comprising: Upon receiving a service alarm event, the target service corresponding to the service alarm event is determined; Query the target resource ownership chain corresponding to the target business service from the current resource ownership graph; the resource ownership graph is used to record the hierarchical ownership relationship between business services, physical resources and virtual resources. Based on a preset inspection strategy, a corresponding inspection task is generated for each resource object in the target resource ownership chain, and each inspection task is executed to obtain the corresponding inspection result. The inspection results are analyzed based on preset root cause analysis rules to determine the root cause of the fault, and the root cause link generated based on the root cause of the fault is sent to a preset front end.

[0006] Optionally, after receiving a service alarm event, the process further includes: Determine whether the type of the service alarm event belongs to a preset alarm type; wherein, the preset alarm types include dial-up service alarms, cloud host threshold alarms, and database alarms; If so, the step of determining the target service corresponding to the service alarm event is allowed to be executed.

[0007] Optionally, the process of establishing the resource ownership map includes: Query each database table in the current database; each database table records the configuration information of any resource object when it is created; the configuration information includes the attribute information of the resource object and its affiliation with other resource objects; The configuration information in each of the database tables is correlated to obtain the resource ownership graph.

[0008] Optionally, generating a corresponding inspection task for each resource object in the target resource ownership chain based on a preset inspection strategy includes: For each resource object in the target resource ownership chain, the indicator items to be checked are determined according to the resource type of the resource object; wherein, the resource type includes services, virtual machines, physical resources and virtual resources; Based on the indicators to be inspected, a corresponding inspection task is generated.

[0009] Optionally, performing each of the inspection tasks to obtain the corresponding inspection results includes: Each of the aforementioned inspection tasks is executed in parallel to obtain the corresponding inspection results; Alternatively, a directed acyclic graph (DAG) pre-constructed based on preset dependencies between resource objects can be obtained, and each of the aforementioned inspection tasks can be executed sequentially based on the DAG to obtain the corresponding inspection results.

[0010] Optionally, the preset root cause analysis rule is a pre-established root cause localization rule tree, where each node in the root cause localization rule tree corresponds to an anomaly detection rule, and the edges between nodes represent the causal relationship between the anomaly detection rules. Accordingly, the analysis of the inspection results based on preset root cause analysis rules to determine the root cause of the fault includes: Traverse the root cause localization rule tree to match each node in the root cause localization rule tree with the anomaly detection results in the inspection results; The matched nodes are sorted according to a preset sorting rule to obtain a sorting result, and the node that ranks first in the sorting result is taken as the root cause of the fault.

[0011] Optionally, sending the root cause link generated based on the root cause of the fault to a preset front end includes: Root cause links are constructed for the matched nodes based on the causal relationships between nodes; The root cause link is sent to a preset front end, so that the anomaly detection results corresponding to different nodes in the root cause link are classified and displayed in different boundary views of the preset front end; wherein, the boundary classification method is determined based on the boundary attributes of each node in the root cause localization rule tree.

[0012] Optionally, the preset root cause analysis rule is a pre-trained target classification model; wherein, the target classification model is obtained by training an initial classification model based on historical data samples, and each historical data sample includes the corresponding historical inspection results and fault root cause labels; Accordingly, the analysis of the inspection results based on preset root cause analysis rules to determine the root cause of the fault includes: The abnormal inspection results in the inspection results are input into the target classification model to output the probability distribution of each abnormal inspection result as the root cause of the fault, and the abnormal inspection result corresponding to the maximum probability is taken as the root cause of the fault.

[0013] A second aspect of this application provides a fault location device, comprising: The alarm receiving module is used to determine the target service corresponding to the service alarm event when a service alarm event is received. The query module is used to query the target resource ownership chain corresponding to the target business service from the current resource ownership relationship graph; the resource ownership relationship graph is used to record the hierarchical ownership relationship between business services, physical resources and virtual resources; The inspection task generation module is used to generate corresponding inspection tasks for each resource object in the target resource ownership chain based on a preset inspection strategy, and execute each inspection task to obtain the corresponding inspection results. The fault root cause determination module is used to analyze the inspection results based on preset root cause analysis rules to determine the fault root cause, and send the root cause link generated based on the fault root cause to a preset front end.

[0014] A third aspect of this application provides an electronic device including a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned fault location method.

[0015] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the aforementioned fault location method.

[0016] In this application, upon receiving a service alarm event, the target service corresponding to the service alarm event is first determined; the target resource ownership chain corresponding to the target service is queried from the current resource ownership graph; the resource ownership graph is used to record the hierarchical ownership relationship between service, physical resources, and virtual resources; a corresponding inspection task is generated for each resource object in the target resource ownership chain based on a preset inspection strategy, and each inspection task is executed to obtain the corresponding inspection result; the inspection result is analyzed based on a preset root cause analysis rule to determine the root cause of the fault, and the root cause link generated based on the root cause of the fault is sent to a preset front end.

[0017] Beneficial Effects: Upon receiving a business alarm event, this application first identifies the target business service corresponding to the alarm event. Then, it directly queries the target resource ownership chain corresponding to the target business service from a pre-defined resource ownership graph. Since the resource ownership graph completely records the hierarchical ownership relationships between business services, physical resources, and virtual resources, there is no need for manual sorting of resource dependencies. Furthermore, this application automatically generates and executes inspection tasks based on a pre-defined inspection strategy, replacing the tedious process of manually logging into each device for troubleshooting, significantly shortening fault location time. Moreover, the queried target resource ownership chain clarifies the precise scope of fault investigation, and the root cause analysis process only focuses on resource objects within the ownership chain, thereby avoiding interference from irrelevant resources and improving location efficiency. Then, based on the pre-defined root cause analysis rules, the inspection results are analyzed to accurately locate the specific root cause of the fault. Finally, the root cause link generated based on the fault root cause is sent to a pre-defined front end, which can intuitively display the propagation path and root cause of the fault. In other words, this solution automates fault location, generates inspection tasks and analyzes inspection results according to a unified process, and forms a standardized and reproducible fault location process, thus lowering the operation and maintenance threshold. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart of a fault location method disclosed in this application; Figure 2 This is a flowchart of a specific fault location method disclosed in this application; Figure 3 This is a schematic diagram of the overall process of a boundary demarcation system disclosed in this application; Figure 4This is a partial schematic diagram of a root cause localization rule tree disclosed in this application; Figure 5 This is a schematic diagram illustrating the delimitation display effect in a front-end interface disclosed in this application; Figure 6 This is a schematic diagram of the structure of a fault location device disclosed in this application; Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Traditional fault location solutions and monitoring systems typically monitor individual resource objects in isolation. When an alarm occurs, maintenance personnel rely on manual documentation or personal experience to identify resource dependencies, then manually log into multiple devices to check processes, logs, resource utilization, and other information to locate the fault. This approach suffers from drawbacks such as missing resource relationships, cumbersome and unautomated troubleshooting processes, and over-reliance on expert experience. To address these issues, this application provides a fault location method, device, equipment, and storage medium that automates fault location. It generates inspection tasks and analyzes inspection results according to a unified process, forming a standardized and reproducible fault location workflow, thus lowering the maintenance threshold.

[0022] Figure 1 A flowchart illustrating a fault location method provided in an embodiment of this application. See also... Figure 1 As shown, the fault location method includes: Step S11: Upon receiving a service alarm event, determine the target service corresponding to the service alarm event.

[0023] In this embodiment, it should first be noted that after receiving a business alarm event, the process further includes: determining whether the type of the business alarm event belongs to a preset alarm type; wherein, the preset alarm types include dial-up service alarms, cloud host threshold alarms, and database alarms; if so, the step of determining the target business service corresponding to the business alarm event is allowed to be executed. That is, the alarms supported for location in this embodiment mainly include all dial-up alarms. Dial-up is a proactive network and service monitoring technology that periodically sends probe requests to the target address by simulating user access behavior to monitor network connectivity and service availability in real time. Therefore, dial-up alarms can serve as the entry point that best reflects business problems. Since dial-up can directly simulate user access, its alarms mean that the real user experience or business function has been damaged, and alarm location analysis needs to be performed quickly to provide analysis results. In addition, this application also supports cloud host threshold alarms, such as CPU IO (Input / Output) waiting utilization rate, average disk IO latency, etc., and supports database alarms, such as database monitoring data reporting anomaly alarms. The filtering logic for these alerts is similar, all based on whether they may directly or seriously affect the business.

[0024] When the system receives such alarms, it first parses the key identification information carried in the alarm event. For example, the business URL (Uniform Resource Locator) corresponding to the probing alarm, the listening port, and the probing task ID; or the vm_id (Virtual Machine ID) corresponding to the cloud host alarm, and the db_id (Database ID) corresponding to the database alarm. Further, using this key identification information as keys, the system queries the probing task configuration table or service registry to determine the specific business service entity bound to these identifiers, such as OA office service or e-commerce payment service. The locked business service is the target business service. For example, if the alarm is "https: / / api.example.com / order response timeout," and the query determines that the service name corresponding to this URL is "Order Service," then this "Order Service" is the target business service.

[0025] Step S12: Query the target resource ownership chain corresponding to the target business service from the current resource ownership relationship graph; the resource ownership relationship graph is used to record the hierarchical ownership relationship between business services, physical resources and virtual resources.

[0026] In this embodiment, after determining the corresponding target business service, the target resource ownership chain corresponding to the target business service is directly queried from the preset resource ownership relationship graph. Since the resource ownership relationship graph completely records the hierarchical ownership relationship between business services, physical resources, and virtual resources, there is no need to manually sort out the resource dependency relationship.

[0027] The process of establishing the resource ownership graph specifically includes: querying each database table in the current database; wherein each database table records the configuration information of any resource object at the time of its creation; the configuration information includes the attribute information of the resource object and its ownership relationship with other resource objects; and associating the configuration information in each database table to obtain the resource ownership graph. That is, the resource ownership graph in this application is dynamically constructed by real-time association queries of each database table in the current database.

[0028] Understandably, when all resource objects are created, their configuration information is automatically recorded in the corresponding database tables. Specifically, starting from the lowest-level physical resources, such as CPUs, memory modules, hard drives, and network cards, data is collected and recorded in the database. Then, virtualization is performed based on these physical resources, creating corresponding virtual resources, such as virtual machines. The configuration information filled in during creation is saved in the table. Then, the corresponding physical resource to which the virtual resource, such as virtual memory or virtual hard drive, corresponds is also recorded during creation. Other similar relationships, such as probing tasks, are constructed using the same method, establishing relationships with other resources when the resource is created. Furthermore, each resource object corresponds to a dedicated database table, such as a virtual machine table for recording virtual machine configurations, a physical server table for recording server hardware information, a probing task table for recording which virtual machines or services are associated with the probing task, and a database instance table. The configuration information recorded in the tables includes two types of core data: firstly, the resource's own attribute information, such as the virtual machine's vm_id, server name, and database's db_id; and secondly, cross-resource attribution relationships, such as the volume_id (associated with physical hard disk) and server_name (associated with physical server) recorded in the virtual machine table, and the associated virtual machine ID (associated with cloud host) recorded in the probing task table. It should be noted that these database tables can be stored in the Configuration Management Database (CMDB). In addition to synchronizing from the CMDB to build the resource relationship graph, the system can also actively discover, such as automatically discovering the association between virtual machines and hosts through the cloud platform API (Application Programming Interface), service mesh topology data, or application performance management (APM) call chain data to supplement and build the resource relationship graph.

[0029] Therefore, this application embodiment can link the data of multiple tables together according to the ownership relationship between resources to obtain a resource ownership relationship graph. Since each table has reserved an association interface through key identifiers (such as vm_id, volume_id, db_id), there is no need to manually maintain the relationship. As shown in the resource relationship of a certain virtual machine below, for example, volume_id indicates which physical hard disk block it belongs to, server_name indicates which physical server this virtual machine was created on, etc. "_province_name": null, "_raid": "", "ips": [], "_rated_power": "", "_cpu_module": "", "_cpu_module": "", "_single_power": false, "_last_monitor_time": null, "_boot_time": null, "_sla": null, "server_name": "test", "_sn": "", "volume_id": "81136bb1_vs_vol_rep2", "_os_name": "", "_az_name": "".

[0030] Therefore, the resource ownership graph query in this application is essentially a table lookup process. For example, the test task table shows which virtual machines are associated with the test task. By querying the corresponding virtual machine table, we can find out which physical servers and hard drives are associated with this virtual machine. For instance, suppose we want to build the resource ownership relationship of OA business services. We would first query the test task table to find the vm_id associated with the test task corresponding to OA; then query the virtual machine table using the vm_id to obtain the server_name (physical server) and volume_id (physical hard drive) associated with that virtual machine; next, query the physical server table using the server_name to find the HCI (Hyper Converged Infrastructure) cluster to which it belongs; finally, we would connect the relationship of "OA business service → virtual machine → physical server → physical hard drive → HCI cluster". Through the above multi-table join queries, the resource ownership graph finally formed is a many-to-many relationship network covering business services, virtual resources, and physical resources.

[0031] It's also important to note that when resources are created, updated, or deleted (e.g., virtual machine migration, server expansion), the corresponding data tables are updated synchronously. Therefore, subsequent queries to the resource ownership graph will be re-associated based on the latest table data, ensuring that the graph always reflects the true ownership relationships of the current resources. Furthermore, the resource ownership graph can be considered a large graph constructed from all related resource objects; it's a "graph" in data structures, representing a many-to-many relationship. The resource ownership chain is a path within the graph, similar to a line connecting things in a certain direction. In this case, this direction refers to the path from the business logic to the underlying physical devices during troubleshooting.

[0032] Step S13: Generate corresponding inspection tasks for each resource object in the target resource ownership chain based on the preset inspection strategy, and execute each inspection task to obtain the corresponding inspection results.

[0033] In this embodiment, an inspection task is automatically generated and executed for each resource object in the target resource ownership chain based on a preset inspection strategy. This replaces the tedious process of manually logging into each device one by one for troubleshooting, significantly shortening the fault location time. Furthermore, the queried target resource ownership chain clarifies the precise scope of fault troubleshooting, and the root cause analysis process focuses only on resource objects within the ownership chain, thereby avoiding interference from irrelevant resources and improving location efficiency.

[0034] Step S14: Analyze the inspection results based on preset root cause analysis rules to determine the root cause of the fault, and send the root cause link generated based on the root cause of the fault to a preset front end.

[0035] In this embodiment, the inspection results are analyzed based on preset root cause analysis rules, which can accurately locate the specific root cause of the fault. Finally, the root cause chain generated based on the fault root cause is sent to a preset front end, which can intuitively display the propagation path and root cause of the fault. That is, this solution realizes the automation of fault location, can generate inspection tasks and analyze inspection results according to a unified process, form a standardized and reproducible fault location process, and reduce the operation and maintenance threshold.

[0036] As can be seen, upon receiving a business alarm event, this application first identifies the target business service corresponding to the alarm event. Then, it directly queries the target resource ownership chain corresponding to the target business service from a pre-defined resource ownership graph. Since the resource ownership graph completely records the hierarchical ownership relationships between business services, physical resources, and virtual resources, there is no need for manual sorting of resource dependencies. Furthermore, this application automatically generates and executes inspection tasks based on a pre-defined inspection strategy, replacing the tedious process of manually logging into each device for troubleshooting, significantly shortening the fault location time. Moreover, the queried target resource ownership chain clarifies the precise scope of fault investigation, and the root cause analysis process only revolves around resource objects within the ownership chain, thereby avoiding interference from irrelevant resources and improving location efficiency. Then, based on the pre-defined root cause analysis rules, the inspection results are analyzed to accurately locate the specific root cause of the fault. Finally, the root cause chain generated based on the fault root cause is sent to a pre-defined front end, which can intuitively display the propagation path and root cause of the fault. In other words, this solution automates fault location, generates inspection tasks and analyzes inspection results according to a unified process, and forms a standardized and reproducible fault location process, thus lowering the operation and maintenance threshold.

[0037] Figure 2 A flowchart illustrating a specific fault location method provided in this application embodiment. See also... Figure 2 As shown, the fault location method includes: Step S21: Upon receiving a service alarm event, determine the target service corresponding to the service alarm event.

[0038] In this embodiment, as Figure 3 As shown, taking dial-up alarm as an example, a dial-up testing system and a boundary system are specifically set up. The core function of the dial-up testing system is to simulate user access behavior (such as accessing a service URL, probing port connectivity), proactively send probe requests to the target service, and monitor the service's availability, connectivity, and response performance in real time. When an anomaly is detected, such as URL access timeout, return of an error status code, or port unresponsiveness, a dial-up service alarm event is generated, and the alarm information (including the dial-up source, target cloud host, port, error details, etc.) is pushed to the boundary system. The core function of the boundary system is to receive the alarm events pushed by the dial-up testing system, query the resource ownership chain corresponding to the target service through the resource ownership relationship graph, automatically generate and execute the inspection tasks of resources at each level, analyze the inspection results according to preset rules and algorithms, and finally define the scope of impact, propagation path, and root cause of the fault, and output the root cause chain and visualization results. The boundary system also determines whether the type of the service alarm event belongs to a preset alarm type; if so, further fault location analysis is performed; otherwise, the system listens for the next alarm.

[0039] Step S22: Query the target resource ownership chain corresponding to the target business service from the current resource ownership relationship graph; the resource ownership relationship graph is used to record the hierarchical ownership relationship between business services, physical resources and virtual resources.

[0040] Step S23: For each resource object in the target resource ownership chain, determine the indicator items to be checked according to the resource type of the resource object; wherein, the resource type includes services, virtual machines, physical resources and virtual resources.

[0041] In this embodiment, since the target resource ownership chain covers all levels of resources from "service → virtual machine / virtual resource → physical resource", the delimitation system will first identify the specific resource type of each resource object, and then match the corresponding type of kernel check indicators to determine the indicator items to be checked.

[0042] For service / process types, the purpose of the check is to confirm whether the application software itself is running normally. The corresponding indicators to be checked include: checking the service listening port status (whether it is open), checking the process liveness status (whether it is running or dead), checking process log error keywords (such as error reports, timeout logs), checking process resource usage (CPU / memory / file descriptors / TCP connection count), and checking service response status codes / latency.

[0043] For each virtual machine type, the purpose of the check is to confirm whether the virtual machine's operating system is healthy and whether its resources are sufficient. The corresponding indicators to be checked include: checking the overall resource utilization of the virtual machine (vCPU, virtual memory, virtual disk I / O, virtual network adapter bandwidth), checking the virtual machine system logs, checking the running status of the virtual machine agent, and checking the association status of the storage / network resources bound to the virtual machine.

[0044] For physical resource types, the purpose of the check is to confirm whether the underlying hardware infrastructure is functioning properly. The corresponding indicators to be checked include: checking the physical server hardware health status (such as fans and power supplies), checking server physical resource utilization (CPU / memory / disk capacity / network bandwidth), checking hardware alarm information managed by the server, checking the overall health status and load of the cluster, and checking physical disk I / O latency / error counts, etc.

[0045] For virtual resource types, the purpose of the check is to confirm whether the abstract resource services provided by the virtualization layer are functioning correctly. The corresponding indicators to be checked include: checking the overall health status, performance load, and event logs of the HCI cluster to which the virtual machine belongs; checking the IO latency, capacity, and error count of the specific storage resource object bound to the virtual machine; and checking the connectivity and policy configuration of the virtual network resource object.

[0046] Step S24: Generate corresponding inspection tasks based on the indicators to be inspected, and execute each inspection task to obtain the corresponding inspection results.

[0047] In this embodiment, for each resource object's metric to be checked, the system automatically generates a structured inspection task, that is, transforms the metric to be checked into specific, executable instructions. For example, to check whether a port is listening, the command `netstat -tlnp | grep :<port number>` is generated (executed via Agent); to check disk I / O latency, a query is generated: a PromQL query for the `disk_io_delay` metric of a specific `vm_id` is sent to the monitoring database (such as VictoriaMetrics); to check hardware alarms, a call is generated: the API of the hardware management platform is called to query the alarm list for that server ID.

[0048] Understandably, the inspection task should include the following: Task objective: Identify the resource objects to be inspected, such as virtual machine VM-001; Inspection methods: query the indicator database to obtain time-series data, call the cloud platform API to obtain resource status, and parse system logs to extract key fields; Result format: Uniform output structured data: {Resource object ID, check item, result status}, such as {Resource object ID: VM-001, check item: vCPU utilization, result status: abnormal (85%), threshold: ≤80%}).

[0049] It should be noted that this application pre-installs the agent on various virtual machines and physical machines to collect information such as metrics, logs, processes, network, and storage. The collected information is then reported to the data warehouse via Skyops. For example, time-series data such as metrics are stored in VictoriaMetrics (an open-source time-series database), relational databases are stored in MongoDB (a document-oriented NoSQL database), and then synchronized to ES (Elasticsearch, a search and analysis engine) for delimited system queries.

[0050] Specifically, the execution of each of the inspection tasks to obtain the corresponding inspection results includes: executing each of the inspection tasks in parallel to obtain the corresponding inspection results; or, obtaining a directed acyclic graph pre-constructed based on a preset dependency relationship between resource objects, and executing each of the inspection tasks sequentially based on the directed acyclic graph to obtain the corresponding inspection results.

[0051] In one specific implementation, this application can adopt a parallel execution mechanism to simultaneously initiate checks on all resource objects on the ownership chain, avoiding inefficiency caused by serial waiting.

[0052] In another specific implementation, a Directed Acyclic Graph (DAG) can be pre-constructed based on the dependencies of resource objects. For example, resource dependencies could be "cluster → physical server → virtual machine → service" or "storage pool → virtual disk → virtual machine". The DAG will determine the order of task execution according to the logic of upper-layer resources depending on lower-layer resources, such as checking the cluster first, then the physical server, and finally the virtual machine and service. In this way, the delimitation system will trigger the check tasks sequentially according to the order defined by the DAG. Only when the preceding dependent tasks are completed and the results meet the preset conditions will the subsequent dependent tasks be executed.

[0053] Step S25: Analyze the inspection results based on preset root cause analysis rules to determine the root cause of the fault, and send the root cause link generated based on the root cause of the fault to a preset front end.

[0054] In one specific implementation, the preset root cause analysis rule is a pre-established root cause localization rule tree, where each node in the root cause localization rule tree corresponds to an anomaly detection rule, and the edges between nodes represent the causal relationship between the anomaly detection rules. Accordingly, the step of analyzing the inspection results based on the preset root cause analysis rule to determine the root cause of the fault includes: traversing the root cause localization rule tree to match each node in the root cause localization rule tree with the anomaly inspection results in the inspection results; sorting the matched nodes based on a preset sorting rule to obtain a sorting result, and taking the node at the top of the sorting result as the root cause of the fault.

[0055] Understandably, the boundary delineation system can have a built-in visual rule tree configuration engine, supporting flexible rule construction and orchestration through a graphical interface. This engine provides a browser-based editing environment, allowing users to connect multiple detection processes into a complete root cause localization rule tree by dragging and dropping pre-defined rule node templates, and supports one-click deployment to the runtime environment. Combined with the system's excellent scalability, users can continuously integrate new abnormal pattern detection rules, thereby driving the continuous evolution of boundary delineation capabilities. In other words, this application can analyze the inspection results based on the root cause localization rule tree to determine the root cause of the fault.

[0056] Each node in the tree corresponds to an anomaly detection rule. This rule includes the anomaly phenomenon and its judgment criteria, such as storage pool capacity utilization ≥99%, OA-service process not running, and virtual machine vCPU utilization ≥85%. These are nodes in the rule tree, corresponding to possible anomaly results in the inspection task. Furthermore, each node can be pre-configured with attributes such as domain and root cause level. Edges between nodes represent the causal relationships between anomaly detection rules; that is, the connections (edges) between nodes represent the "cause-effect" logic between anomalies. For example, a storage pool capacity utilization ≥99% will cause virtual machine disk read / write failures; and virtual machine disk read / write failures will cause the OA-service process to fail to load page files, ultimately leading to a test alarm. Therefore, this edge defines a causal chain of fault propagation, indicating that if an anomaly is found in a child node (cause), it may lead to an anomaly in the parent node (effect). It should also be noted that the rule tree is a bottom-up causal chain, with bottom nodes being root cause candidates (such as a full storage pool) and top nodes being business alarm manifestations (such as a 404 error during a test), which corresponds exactly to the propagation path of the fault from the root cause to the manifestation. Figure 4 This is a partial schematic diagram of a root cause localization rule tree disclosed in this application.

[0057] Specifically, after obtaining the inspection results of all inspection tasks, the system first filters out the abnormal inspection results. Then, starting from the root node or the node that best matches the current alarm phenomenon, the system traverses the root cause localization rule tree, precisely matching the filtered abnormal inspection results with each node (anomaly detection rule) in the rule tree. For example, a storage pool S1 utilization rate of 99% will perfectly match the node "storage pool capacity utilization rate ≥ 99%" in the rule tree, and a virtual machine disk I / O latency of 200ms will match the node "virtual machine disk I / O average latency ≥ 100ms" in the rule tree. If multiple abnormal results exist, multiple corresponding nodes in the rule tree will be matched simultaneously. Furthermore, it should be noted that the matching in this application is not isolated; the system also performs causal inference along the edges between nodes. For example, if the node "high storage I / O latency" is matched, the system will then check the possible causes of its child nodes based on the causal relationship between nodes, such as the inspection results required by the physical hard disk failure rule, such as whether there are hardware alarms. In simple terms, if a test task D receives an alarm, and the rule tree is used to probe downwards, it is found that the cause of the test alarm may be a service issue. At this point, the system will check whether the service C associated with D has a problem. If there is a problem, the system will continue to look for abnormal rules that may be causing the service issue, such as virtual machine anomalies. Then, the system will check whether the virtual machine B associated with service C has a problem. In addition, if a problem is found with virtual machine E that is not associated with C, but there is no causal relationship between them, E will not be investigated.

[0058] Finally, this embodiment of the application needs to determine the most critical root cause from multiple matched nodes. Specifically, the matched nodes can be sorted based on a preset sorting rule to obtain a sorting result, and then the node at the top of the sorting result is taken as the root cause of the fault. The preset sorting rule is specifically embodied as follows: First priority: Whether it occurs on the same resource object. This is the most critical rule. If multiple anomalies occur on the same resource (e.g., high CPU and memory usage on the same physical server, and hard drive alerts on the same server), then these anomalies are highly correlated, and the probability that the resource is the root cause increases dramatically.

[0059] Second priority: Whether it is the root cause node / whether it is the last item. The node marked with the smallest root_cause_level value in the rule tree is designed to be the lowest-level root cause, and if it is matched, it should be prioritized as the root cause.

[0060] Third priority: the severity and number of alerts; the more urgent alerts a resource has, the more likely it is to be the root cause.

[0061] Fourth priority: Root cause level of the rule. The lower the level value, the more likely the node is to be the root cause.

[0062] Fifth priority: The height of the rule in the graph. In the rule tree editing interface, the higher the position of a node at the same level, the higher its priority.

[0063] After sorting all the matched abnormal nodes using the above sorting rules, the abnormality represented by the node ranked first is determined to be the root cause of this failure. At the same time, the path traversed from this node back to the initial alarm node naturally forms the root cause chain.

[0064] Furthermore, sending the root cause link generated based on the root cause of the fault to a preset front end includes: constructing a root cause link for the matched nodes based on the causal relationship between nodes; sending the root cause link to the preset front end to classify and display the anomaly check results corresponding to different nodes in the root cause link in different boundary views of the preset front end; wherein, the boundary classification method is determined based on the boundary attributes of each node in the root cause localization rule tree. That is, after rule tree matching and sorting, the root cause node of the fault has been determined, and it is known which intermediate nodes are activated during the traversal process. Therefore, this application will follow the causal relationship edge between the activated nodes in the rule tree, starting from the root cause node at the bottom layer, and backtracking upstream (towards the business side), connecting these nodes in causal order to form a linear, interpretable text or data structure chain, that is, obtaining the root cause link. For example, "hard disk failure → high storage IO latency → poor virtual machine performance → service response timeout → dial-up alarm".

[0065] Furthermore, this application categorizes all anomaly information into five domains: external internet, internal LAN, application, cloud host operating system, and platform. Each node in the root cause localization rule tree is pre-configured with a domain attribute. Therefore, this embodiment reads the domain attribute of each node in the root cause chain and categorizes the corresponding anomaly check results into the corresponding domain for categorized display in different domain views on the preset front end. In a specific implementation, the front end presents the anomaly check results under each domain in the form of domain cards or dedicated modules. Simultaneously, the front end visually presents the complete propagation path of the root cause chain (e.g., connecting anomaly nodes in different domains with arrows), allowing operations personnel to see anomalies at each level and clearly trace which anomaly in which domain is the root cause of the fault. Figure 5 As shown in the image.

[0066] In another specific implementation, the preset root cause analysis rule is a pre-trained target classification model; wherein, the target classification model is obtained by training an initial classification model based on historical data samples, and each historical data sample includes a corresponding historical inspection result and a fault root cause label; accordingly, the step of analyzing the inspection results based on the preset root cause analysis rule to determine the fault root cause includes: inputting the abnormal inspection results in the inspection results into the target classification model to output the probability distribution of each abnormal inspection result as the fault root cause, and taking the abnormal inspection result corresponding to the maximum probability as the fault root cause.

[0067] In other words, in addition to using a rule tree-based engine, the root cause analysis rules can also incorporate machine learning models. By learning from historical failure cases and corresponding monitoring indicator sequences, the model can directly perform pattern recognition on the inspection results when a new failure occurs, providing a root cause probability prediction. Each historical data sample contains two sets of key data: first, historical inspection results, i.e., abnormal inspection data of each resource on the resource ownership chain when past failures occurred, such as structured abnormal information like storage pool utilization of 99%, virtual machine disk I / O latency of 200ms, and service processes not running; second, the root cause label, i.e., the actual root cause of the failure as confirmed manually or determined by rules, such as insufficient storage pool capacity. Simply put, the sample is a data set of correspondences between "past abnormal phenomena → actual root causes." In this application, an initial classification model is first selected, such as a simple logistic regression model or a complex deep learning model. Then, a large number of historical data samples are input into the model: the model learns the patterns of which abnormal combinations correspond to which root causes, continuously adjusts the model parameters, and finally trains a target classification model that can output root cause probabilities from the input abnormal inspection results. Then, by extracting the anomaly detection result with the highest probability value from the probability distribution, it is taken as the ultimate root cause of this failure.

[0068] In this embodiment, the specific process of step S22 can be referred to the corresponding content disclosed in the previous embodiments, and will not be repeated here.

[0069] As can be seen, by constructing and utilizing a refined resource ownership graph, business services are dynamically linked to the various levels of hardware and software resources supporting their operation, such as processes, virtual machines, physical servers, storage network resources, and clusters. This enables automatic and systematic troubleshooting down the resource ownership chain after a business alarm is triggered, quickly locating the root cause of the fault, thereby improving operational efficiency and ensuring business continuity. This solves the problem of difficult and time-consuming root cause location when business systems experience failures or performance degradation in complex cloud and virtualization environments. The proposed solution offers the following advantages: Fast location speed: Automation replaces numerous manual troubleshooting steps, shortening fault location time; High location accuracy: Systematic investigation based on a complete resource ownership chain avoids omissions and accurately locates specific resource objects, rather than vague server or network issues; Standardized operation and maintenance: Expert troubleshooting experience is solidified into inspection strategies and rules, forming a unified and repeatable delineation process, reducing reliance on individual experience; Comprehensive coverage: It bridges the monitoring gap between business applications and infrastructure, effectively monitoring and associating abstract virtual resource objects in virtualized and cloud environments; Proactive prevention: Through routine checks of objects on the resource chain, potential risks to underlying resources (such as disk exhaustion or slow memory leakage) can be detected before business alarms occur, enabling proactive operation and maintenance.

[0070] Figure 6 A flowchart illustrating a fault location device provided in an embodiment of this application. See also... Figure 6 As shown, the fault device method includes: The alarm receiving module 11 is used to determine the target service corresponding to the service alarm event when a service alarm event is received; The query module 12 is used to query the target resource ownership chain corresponding to the target business service from the current resource ownership relationship graph; the resource ownership relationship graph is used to record the hierarchical ownership relationship between business services, physical resources and virtual resources; The inspection task generation module 13 is used to generate corresponding inspection tasks for each resource object in the target resource ownership chain based on a preset inspection strategy, and execute each inspection task to obtain the corresponding inspection results. The fault root cause determination module 14 is used to analyze the inspection results based on preset root cause analysis rules to determine the fault root cause, and send the root cause link generated based on the fault root cause to a preset front end.

[0071] As can be seen, upon receiving a business alarm event, this application first identifies the target business service corresponding to the alarm event. Then, it directly queries the target resource ownership chain corresponding to the target business service from a pre-defined resource ownership graph. Since the resource ownership graph completely records the hierarchical ownership relationships between business services, physical resources, and virtual resources, there is no need for manual sorting of resource dependencies. Furthermore, this application automatically generates and executes inspection tasks based on a pre-defined inspection strategy, replacing the tedious process of manually logging into each device for troubleshooting, significantly shortening the fault location time. Moreover, the queried target resource ownership chain clarifies the precise scope of fault investigation, and the root cause analysis process only revolves around resource objects within the ownership chain, thereby avoiding interference from irrelevant resources and improving location efficiency. Then, based on the pre-defined root cause analysis rules, the inspection results are analyzed to accurately locate the specific root cause of the fault. Finally, the root cause chain generated based on the fault root cause is sent to a pre-defined front end, which can intuitively display the propagation path and root cause of the fault. In other words, this solution automates fault location, generates inspection tasks and analyzes inspection results according to a unified process, and forms a standardized and reproducible fault location process, thus lowering the operation and maintenance threshold.

[0072] Furthermore, embodiments of this application also provide an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0073] Figure 7This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the fault location method disclosed in any of the foregoing embodiments.

[0074] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0075] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0076] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the fault location method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks. The data 223 may include image quality control strategies collected by the electronic device 20, etc.

[0077] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the fault location method steps disclosed in any of the foregoing embodiments.

[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0079] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] The above provides a detailed description of the fault location method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A fault location method, characterized in that, include: Upon receiving a service alarm event, the target service corresponding to the service alarm event is determined; Query the target resource ownership chain corresponding to the target business service from the current resource ownership graph; The resource ownership map is used to record the hierarchical ownership relationships between business services, physical resources, and virtual resources. Based on a preset inspection strategy, a corresponding inspection task is generated for each resource object in the target resource ownership chain, and each inspection task is executed to obtain the corresponding inspection result. The inspection results are analyzed based on preset root cause analysis rules to determine the root cause of the fault, and the root cause link generated based on the root cause of the fault is sent to a preset front end.

2. The fault location method according to claim 1, characterized in that, The process of receiving a service alarm event also includes: Determine whether the type of the service alarm event belongs to a preset alarm type; wherein, the preset alarm types include dial-up service alarms, cloud host threshold alarms, and database alarms; If so, the step of determining the target service corresponding to the service alarm event is allowed to be executed.

3. The fault location method according to claim 1, characterized in that, The process of establishing the resource ownership map includes: Query each database table in the current database; each database table records the configuration information of any resource object when it is created; the configuration information includes the attribute information of the resource object and its affiliation with other resource objects; The configuration information in each of the database tables is correlated to obtain the resource ownership graph.

4. The fault location method according to claim 1, characterized in that, The step of generating a corresponding inspection task for each resource object in the target resource ownership chain based on a preset inspection strategy includes: For each resource object in the target resource ownership chain, the indicator items to be checked are determined according to the resource type of the resource object; wherein, the resource type includes services, virtual machines, physical resources and virtual resources; Based on the indicators to be inspected, a corresponding inspection task is generated.

5. The fault location method according to claim 1, characterized in that, The execution of each of the aforementioned inspection tasks to obtain the corresponding inspection results includes: Each of the aforementioned inspection tasks is executed in parallel to obtain the corresponding inspection results; Alternatively, a directed acyclic graph (DAG) pre-constructed based on preset dependencies between resource objects can be obtained, and each of the aforementioned inspection tasks can be executed sequentially based on the DAG to obtain the corresponding inspection results.

6. The fault location method according to any one of claims 1 to 5, characterized in that, The preset root cause analysis rule is a pre-established root cause localization rule tree. Each node in the root cause localization rule tree corresponds to an anomaly detection rule, and the edges between nodes represent the causal relationship between the anomaly detection rules. Accordingly, the analysis of the inspection results based on preset root cause analysis rules to determine the root cause of the fault includes: Traverse the root cause localization rule tree to match each node in the root cause localization rule tree with the anomaly detection results in the inspection results; The matched nodes are sorted according to a preset sorting rule to obtain a sorting result, and the node that ranks first in the sorting result is taken as the root cause of the fault.

7. The fault location method according to claim 6, characterized in that, The step of sending the root cause link generated based on the root cause of the fault to a preset front end includes: Root cause links are constructed for the matched nodes based on the causal relationships between nodes; The root cause link is sent to a preset front end, so that the anomaly detection results corresponding to different nodes in the root cause link are classified and displayed in different boundary views of the preset front end; wherein, the boundary classification method is determined based on the boundary attributes of each node in the root cause localization rule tree.

8. The fault location method according to any one of claims 1 to 5, characterized in that, The preset root cause analysis rule is a pre-trained target classification model; wherein, the target classification model is obtained by training an initial classification model based on historical data samples, and each historical data sample includes the corresponding historical inspection results and fault root cause labels; Accordingly, the analysis of the inspection results based on preset root cause analysis rules to determine the root cause of the fault includes: The abnormal inspection results in the inspection results are input into the target classification model to output the probability distribution of each abnormal inspection result as the root cause of the fault, and the abnormal inspection result corresponding to the maximum probability is taken as the root cause of the fault.

9. A fault location device, characterized in that, include: The alarm receiving module is used to determine the target service corresponding to the service alarm event when a service alarm event is received. The query module is used to query the target resource ownership chain corresponding to the target business service from the current resource ownership relationship graph; The resource ownership map is used to record the hierarchical ownership relationships between business services, physical resources, and virtual resources. The inspection task generation module is used to generate corresponding inspection tasks for each resource object in the target resource ownership chain based on a preset inspection strategy, and execute each inspection task to obtain the corresponding inspection results. The fault root cause determination module is used to analyze the inspection results based on preset root cause analysis rules to determine the fault root cause, and send the root cause link generated based on the fault root cause to a preset front end.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein: The memory is used to store computer programs; The computer program is loaded and executed by the processor to implement the fault location method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Used to store computer-executable instructions, which, when loaded and executed by a processor, implement the fault location method as described in any one of claims 1 to 8.

Images