Monitoring alarm fault self-recovery methods, devices, equipment, media and program products

Abstract

The present application relates to a monitoring alarm fault self-healing method, apparatus, equipment, medium and program product, the method comprising: after receiving the alarm information of a first node, updating the fault weight of the first node according to the alarm event of the first node and the alarm events of each first right node that depends on the first node, and updating the fault weight of each first left node according to the alarm event of each first left node that the first node depends on; determining the target node of the alarm root cause from the first node, each first right node and each first left node according to the updated fault weight, and determining the fault event of the alarm root cause according to the alarm information of the target node; generating a self-healing plan according to the fault event, and performing fault repair processing based on the self-healing plan, thereby realizing self-healing processing for monitoring alarm faults and improving the processing efficiency of monitoring alarm faults.

Description

Technical Field

[0001] The present application relates to the field of data operation and maintenance technology, and in particular to a monitoring alarm fault self-healing method, device, equipment, medium and program product. Background Art

[0002] To ensure 24 / 7 online enterprise services, monitoring and alarming are crucial internet technology solutions. However, with the continuous growth of enterprise application systems, business expansion, and increasing infrastructure resources, the dependencies between services and batch jobs are becoming increasingly complex, requiring more and more monitoring content. Often, an anomaly in one service or job can affect related resources, services, or jobs, causing anomalies as well. This can trigger an alarm storm, where lower-level alarms are overwhelmed by higher-level alarms.

[0003] In traditional operation and maintenance methods, follow-up of problems after monitoring alarms usually requires manual analysis, and alarm root cause analysis often relies on the skills and experience of operation and maintenance engineers and development engineers.

[0004] However, the timeliness of manual participation will affect the speed of service recovery, and human unreliability may also cause the problem to escalate. Therefore, the traditional monitoring alarm fault handling process has the problem of low efficiency. Summary of the Invention

[0005] Based on this, it is necessary to provide a monitoring alarm fault self-healing method, device, equipment, medium and program product that can improve the efficiency of handling monitoring alarm faults in order to address the above technical problems.

[0006] In a first aspect, the present application provides a monitoring alarm fault self-healing method. The method includes: after receiving alarm information from a first node, updating the fault weight of the first node based on the alarm event of the first node and the alarm events of each first right node that depends on the first node, and updating the fault weight of each first left node based on the alarm event of each first left node that the first node depends on; determining the target node of the alarm root cause from the first node, each first right node, and each first left node based on the updated fault weight, and determining the fault event of the alarm root cause based on the alarm information of the target node; generating a self-healing plan based on the fault event, and performing fault repair processing based on the self-healing plan.

[0007] In one embodiment, the fault weight of the first node is updated based on the alarm event of the first node and the alarm events of each first right node that depends on the first node, including: performing a first update on the fault weight of the first node based on the alarm event of the first node; after the first update, performing a second update on the fault weight of the first node based on the alarm events of each first right node.

[0008] In one embodiment, the fault weight of the first node is updated for the first time based on the alarm event of the first node, including: if there is an unresolved alarm event for the first node within the target historical period, the fault weight of the first node is added to a preset value; if there is no unresolved alarm event for the first node within the target historical period, the fault weight of the first node is reset to the initial weight value.

[0009] In one embodiment, the fault weight of the first node is updated for the second time according to the alarm events of each first right node, including: if each first right node has an unresolved alarm event within the target historical period, the fault weight of the first node is updated for the second time.

[0010] In one embodiment, performing a second updating process on the fault weight of the first node includes: taking the sum of the fault weights of each first right node and the current fault weight of the first node as the fault weight of the first node after the second updating process.

[0011] In one embodiment, the fault weight of each first left node is updated based on the alarm events of each first left node that the first node depends on, including: if each first left node has an unresolved alarm event within the target historical period, the fault weight of each first left node is updated.

[0012] In one embodiment, the fault weight of each first left node is updated, including: for each first left node, taking the sum of the fault weight of the first left node and the current fault weight of the first node as the updated fault weight of the first left node.

[0013] In one embodiment, the target node of the root cause of the alarm is determined from the first node, each first right node and each first left node according to the updated fault weight, including: comparing the updated fault weights of the first node, each first right node and each first left node, and taking the node with the largest fault weight as the target node.

[0014] In one embodiment, generating a self-healing solution based on a fault event includes: extracting root cause information from the fault event and determining a self-healing strategy based on the root cause information; and generating a self-healing solution based on the self-healing strategy.

[0015] In one embodiment, the self-healing strategy includes a production patch repair strategy, and generating a self-healing plan based on the self-healing strategy includes: obtaining location information of the defect code and repair suggestion information; and using the location information of the defect code and repair suggestion information as the self-healing plan.

[0016] In one embodiment, the self-healing strategy includes a data change repair strategy, and a self-healing plan is generated according to the self-healing strategy, including: determining the target data that needs to be changed for fault repair; generating a first data change script for the target node, a second data change script for each second left node dependent on the target node, and a third data change script for each second right node dependent on the target node according to the target data; and generating a self-healing plan according to the first data change script, the second data change script, and the third data change script.

[0017] In one embodiment, fault repair processing is performed based on a self-healing solution, including: performing a data change operation on each second left node based on a second data change script; after performing the data change operation on each second left node, performing a data change operation on the target node based on a first data change script; after performing the data change operation on the target node, performing a data change operation on each second right node based on a third data change script.

[0018] In one embodiment, the self-healing solution also includes a backup script and a rollback script; fault repair processing based on the self-healing solution also includes: performing a data backup operation based on the backup script before the data change operation is performed; and performing a data rollback operation based on the rollback script after an execution error occurs in the data change operation.

[0019] In a second aspect, the present application also provides a monitoring alarm fault self-healing device. The device includes: an update module for, after receiving alarm information from a first node, updating the fault weight of the first node based on the alarm event of the first node and the alarm events of each first right node that depends on the first node, and updating the fault weight of each first left node based on the alarm event of each first left node that the first node depends on; a positioning module for determining the target node of the alarm root cause from the first node, each first right node, and each first left node based on the updated fault weight, and determining the fault event of the fault root cause based on the alarm information of the target node; and a self-healing module for generating a self-healing plan based on the fault event and performing fault repair processing based on the self-healing plan.

[0020] In one embodiment, the update module is specifically used to: perform a first update process on the fault weight of the first node according to the alarm event of the first node; after the first update process, perform a second update process on the fault weight of the first node according to the alarm events of each first right node.

[0021] In one embodiment, the update module is specifically used to: if the first node has an unresolved alarm event within the target historical period, add the fault weight of the first node to a preset value; if the first node has no unresolved alarm event within the target historical period, reset the fault weight of the first node to the initial weight value.

[0022] In one embodiment, the updating module is specifically configured to: if each first right node has an unresolved alarm event within a target historical period, perform a second updating process on the fault weight of the first node.

[0023] In one embodiment, the updating module is further configured to: use the sum of the fault weights of each first right node and the current fault weight of the first node as the fault weight of the first node after the second updating process.

[0024] In one embodiment, the updating module is further configured to update the fault weight of each first left node if there is an unresolved alarm event for each first left node within the target historical period.

[0025] In one embodiment, the updating module is further configured to: for each first left node, use the sum of the fault weight of the first left node and the current fault weight of the first node as the updated fault weight of the first left node.

[0026] In one embodiment, the positioning module is specifically configured to compare the updated fault weights of the first node, each first right node, and each first left node, and select the node with the largest fault weight as the target node.

[0027] In one embodiment, the self-healing module is specifically configured to: extract root cause information from a fault event, and determine a self-healing strategy based on the root cause information; and generate a self-healing solution according to the self-healing strategy.

[0028] In one embodiment, the self-healing strategy includes a production patch repair strategy and a self-healing module, which is specifically used to: obtain location information of defective code and repair suggestion information; and use the location information of defective code and repair suggestion information as a self-healing solution.

[0029] In one embodiment, the self-healing strategy includes a data change repair strategy and a self-healing module, which is specifically used to: determine the target data that needs to be changed for fault repair; generate a first data change script for the target node, a second data change script for each second left node dependent on the target node, and a third data change script for each second right node dependent on the target node based on the target data; and generate a self-healing plan based on the first data change script, the second data change script, and the third data change script.

[0030] In one embodiment, the self-healing module is specifically used to: perform a data change operation on each second left node based on the second data change script; after performing the data change operation on each second left node, perform a data change operation on the target node based on the first data change script; after performing the data change operation on the target node, perform a data change operation on each second right node based on the third data change script.

[0031] In one embodiment, the self-healing solution also includes a backup script and a rollback script; the self-healing module is further used to: perform a data backup operation based on the backup script before the data change operation is performed; and perform a data rollback operation based on the rollback script after an execution error occurs in the data change operation.

[0032] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect above.

[0034] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of any one of the methods in the first aspect.

[0035] The above-mentioned monitoring alarm fault self-healing method, device, equipment, medium and program product, after receiving the alarm information of the first node, updates the fault weight of the first node according to the alarm event of the first node and the alarm events of the first right nodes that depend on the first node, and updates the fault weight of each first left node according to the alarm event of each first left node that is dependent on the first node, and then determines the target node of the alarm root cause from the first node, each first right node and each first left node according to the updated fault weight, and determines the fault event of the alarm root cause according to the alarm information of the target node, and finally generates a self-healing plan according to the fault event, and performs fault repair processing based on the self-healing plan, thereby realizing self-healing processing of monitoring alarm faults and improving the processing efficiency of monitoring alarm faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a process for self-healing of a monitoring alarm fault in one embodiment;

[0037] Figure 2 A schematic diagram of a process for determining a target node and a fault event in one embodiment;

[0038] Figure 3 A schematic diagram of a process for generating a self-healing solution in one embodiment;

[0039] Figure 4 A schematic diagram of a process for performing fault repair processing based on a self-healing solution in one embodiment;

[0040] Figure 5 A structural diagram of a monitoring alarm fault self-healing device in one embodiment;

[0041] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] Monitoring and alarming are crucial technologies for ensuring 24 / 7 service availability. However, with the increasing number of enterprise application systems, business expansion, and growing infrastructure resources, the dependencies between services and batch jobs are becoming increasingly complex, requiring more and more monitoring content. Often, an anomaly in one service or batch job can impact related resources, services, or batch jobs, triggering an alarm storm where lower-level alarms are overwhelmed and drowned out by higher-level alarms. Traditionally, manual analysis is required to identify the root cause and address the fault. This not only relies on manual experience but also suffers from slow troubleshooting. Therefore, it is necessary to develop effective methods for efficiently handling monitoring and alarm failures.

[0044] In one embodiment, Figure 1 As shown, a monitoring alarm fault self-healing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server, wherein the terminal can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In the embodiment of the present application, the method includes the following steps:

[0045] Step 101: After receiving the alarm information of the first node, the fault weight of the first node is updated according to the alarm event of the first node and the alarm events of the first right nodes that depend on the first node, and the fault weight of each first left node is updated according to the alarm events of the first left nodes that are dependent on the first node.

[0046] Specifically, a node refers to a server, virtual machine, container, physical machine, online service, or batch job. The first node refers to the node that sends the alarm information, that is, the alarm node. The alarm information includes the on-site snapshot, alarm time, alarm node, and alarm content. The alarm content may include, for example, that a batch job of an application is interrupted due to a primary key conflict, or that a container has a memory overflow or CPU increase. There are node dependencies between nodes, such as the dependency between services and batch jobs and virtual machines, containers, and physical machines, as well as the dependency between services and services, and between batch jobs and batch jobs. Node dependencies are obtained through monitoring or configuration. The specific acquisition process is as follows: (1) First, obtain all nodes and generate a unique identifier for each node; (2) Take each node as the starting node, then obtain all dependency chains that depend on this node, and delete the sub-dependency chains contained in the large call chain, and use this as its right node chain. For example, if service A calls service B, and service B calls service C, only the dependency chain of service A, service B, and service C will be retained, and the dependency chain of service A, service B will be deleted; (3) Similarly, take this node as the end node, then obtain the dependent nodes of this node to form a dependency chain, and use the largest dependency chain with this node as the end node as its left node chain. That is, the node dependency relationship includes a left node chain and a right node chain, and each first right node is each node in the right node chain of the first node, and each first left node is each node in the left node chain of the first node.

[0047] Specifically, an alarm event refers to the occurrence of an alarm on a node. The act of a node sending an alarm message is called an alarm event. Alarm events on different nodes have unique alarm event numbers, fault weights, and processing statuses. The processing status refers to the processing result of the alarm event in the entire monitoring alarm fault self-healing process, including successful change, patch online, unprocessed, in-process, pending change, and failed change. Successful change and patch online can be considered resolved, while unprocessed, in-process, pending change, and failed change can be considered unresolved. Furthermore, the alarm information, alarm event number, fault weight, and processing status of each node are stored to form historical alarm information.

[0048] Optionally, after receiving the alarm information of the first section, first determine whether there is an alarm event of the first node with a processing status of unresolved in the historical alarm information. If so, the fault weight of the first node is increased by 1; if not, the fault weight of the first node is 1, a new alarm event number is generated, the processing status is unprocessed, and the alarm information, alarm event number, fault weight and processing status of the first node are stored. Secondly, determine whether the alarm events of each first right node in the right node chain of the first node exist in the historical alarm information and the processing status is unresolved, and then update the fault weight of the first node according to the fault weight of the first node and the fault weights of each existing first right node. Then determine whether the alarm events of each first left node in the left node chain of the first node exist in the historical alarm information and the processing status is unresolved. For the fault weights of each existing first left node, update them according to the updated fault weight of the first node and the fault weights of each existing first left node. In addition, it can be seen that the processing status of the alarm events corresponding to the updated fault weight of the first node and the fault weights of each first left node are both unresolved.

[0049] Step 102 : determining the target node of the alarm root cause from the first node, each first right node, and each first left node according to the updated fault weight, and determining the fault event of the alarm root cause according to the alarm information of the target node.

[0050] Optionally, after executing step 101, a new fault weight of the first node, a fault weight of each first right node, and a fault weight of each first left node are obtained. Then, based on the new fault weight of the first node, the fault weight of each first right node, and the fault weight of each first left node, the target node can be determined, and the alarm information of the target node is the fault event. Because the processing status of the alarm events corresponding to the updated fault weight of the first node and the fault weight of each first left node are both unresolved, the processing status of the fault event is also unresolved.

[0051] Optionally, after determining the target node and the fault event, the processing status of the alarm event of the first node is modified according to the processing status of the target node. For example, after executing step 101 on the alarm event of the first node, the stored processing status of the alarm event of the first node is unprocessed, and the processing status of the fault event of the target node obtained after step 102 is changing, then the processing status of the alarm event of the first node needs to be changed to changing.

[0052] Optionally, it is also necessary to determine whether the target node and fault event are in the monitoring alarm fault self-healing process. If so, the alarm and analysis details address and other information are sent to relevant operation and maintenance engineers and development engineers according to the configured alarm template; if not, the alarm information and analysis details address and other information are sent to relevant operation and maintenance engineers and development engineers according to the configured alarm template, and step 103 is executed, wherein the analysis details address can be a web page, and the analysis process of the target node and fault event can be understood through the analysis details address.

[0053] In addition, the enterprise has a large number of application systems, and the dependencies between nodes are very complex. Often, an abnormality in one node will cause abnormalities in the nodes that depend on it, resulting in the receipt of alarm information from many nodes. Therefore, it is necessary to find the target node and fault event that are the root cause of the alarm from the alarm information of many nodes in order to solve the problem from the source of the fault. The target node and fault event can be identified quickly and accurately based on the fault weight of the node.

[0054] Step 103: Generate a self-healing plan according to the fault event, and perform fault repair processing based on the self-healing plan.

[0055] Specifically, through analysis of historical events related to monitoring alarm fault handling, it was found that common fault handling methods are data modification and repair and production patch repair. Specifically, issues such as primary key conflicts, incorrect variable values, batch skipping, memory overflow, and high CPU utilization can be resolved through data modification and repair, while code rewriting and logical errors require production patch repair. The data modification and repair method can be completely automated, while the production patch repair method, which modifies the code program, can locate the specific defective code segment and provide optimization suggestions for reference by operation and maintenance engineers and development engineers. Therefore, based on the fault event, whether generating a self-healing solution through data modification or a self-healing solution through production patch repair, it can reduce manpower and time investment, quickly resolve faults, and ensure the quality and stability of enterprise services and batch operations.

[0056] Optionally, due to the large number of application systems in the enterprise and the very complex dependencies between the nodes, an abnormality in one node will often cause an abnormality in the nodes that depend on it, thereby receiving alarm information from many nodes. After receiving the alarm information from each node, steps 101 and 102 must be executed. Therefore, at least one target node and at least one fault event will be obtained, and then step 103 will be executed for each fault event to achieve the purpose of resolving all faults.

[0057] To sum up, after receiving the alarm information of the first node, the fault weight of the first node is updated according to the alarm event of the first node and the alarm events of the first right nodes that depend on the first node, and the fault weight of each first left node is updated according to the alarm event of each first left node that the first node depends on. Then, based on the updated fault weight, the target node of the root cause of the alarm is determined from the first node, each first right node and each first left node, and the fault event of the root cause of the alarm is determined based on the alarm information of the target node. Finally, a self-healing plan is generated according to the fault event, and fault repair processing is performed based on the self-healing plan, thereby realizing self-healing processing of monitoring alarm faults and improving the processing efficiency of monitoring alarm faults.

[0058] In one embodiment, the fault weight of the first node is updated based on the alarm event of the first node and the alarm events of each first right node that depends on the first node, including: performing a first update on the fault weight of the first node based on the alarm event of the first node; after the first update, performing a second update on the fault weight of the first node based on the alarm events of each first right node.

[0059] In one embodiment, the fault weight of the first node is updated for the first time based on the alarm event of the first node, including: if there is an unresolved alarm event for the first node within the target historical period, the fault weight of the first node is added to a preset value; if there is no unresolved alarm event for the first node within the target historical period, the fault weight of the first node is reset to the initial weight value.

[0060] In one embodiment, the fault weight of the first node is updated for the second time according to the alarm events of each first right node, including: if each first right node has an unresolved alarm event within the target historical period, the fault weight of the first node is updated for the second time.

[0061] In one embodiment, performing a second updating process on the fault weight of the first node includes: taking the sum of the fault weights of each first right node and the current fault weight of the first node as the fault weight of the first node after the second updating process.

[0062] Optionally, after receiving the alarm information of the first node, first determine whether there are any unresolved alarm events for the first node within the target historical period. If so, the sum of the fault weight of the first node within the target historical period and the preset value is the fault weight of the first node after the first update, where the preset value can be 1 or other set values, which is not limited here; if not, the initial weight value is the fault weight of the first node after the first update, where the initial weight value can be 1 or other set values, which is not limited here. When there are no unresolved alarm events for the first node within the target historical period, a new alarm event number needs to be generated, and the processing status of the alarm event of the first node is set to unprocessed, and stored together with the alarm content of the first node and the fault weight of the first node after the first update processing. Secondly, determine whether there is a right node chain of the first node, and determine whether each first right node in the right node chain has any unresolved alarm events within the target historical period. If so, the sum of the fault weights of each existing first right node and the fault weight of the first node after the first update is used as the fault weight of the first node after the second update, and then execute the step of updating the fault weight of each first left node according to the alarm events of each first left node dependent on the first node; if not, directly execute the step of updating the fault weight of each first left node according to the alarm events of each first left node dependent on the first node.

[0063] In addition, each first right node is a node that depends on the first node, and the alarm event of each first right node may be caused by the abnormality of the first node. Therefore, by adding the weight of each first right node to the first node, the first node can be identified as the alarm root cause node through the weight.

[0064] In one embodiment, the fault weight of each first left node is updated based on the alarm events of each first left node that the first node depends on, including: if each first left node has an unresolved alarm event within the target historical period, the fault weight of each first left node is updated.

[0065] In one embodiment, the fault weight of each first left node is updated, including: for each first left node, taking the sum of the fault weight of the first left node and the current fault weight of the first node as the updated fault weight of the first left node.

[0066] Optionally, if the right node chain of the first node does not exist or each first right node does not have an unresolved alarm event within the target historical period, then determine whether there is a left node chain of the first node, and determine whether each first left node in the left node chain has an unresolved alarm event within the target historical period. If so, for each existing first left node, the fault weight of the existing first left node and the fault weight of the first node after the first update are used as the fault weight of the first left node after the update processing, and then execute the step of determining the target node of the root cause of the alarm from the first node, each first right node and each first left node according to the fault weight after the update processing. Step: If the right node chain of the first node exists and each first right node has an unresolved alarm event within the target historical period, then determine whether there is a left node chain of the first node, and determine whether each first left node in the left node chain has an unresolved alarm event within the target historical period. If so, for each existing first left node, the fault weight of the existing first left node and the fault weight of the first node after the second update are used as the fault weight of the existing first left node after the update process, and then the step of determining the target node of the root cause of the alarm from the first node, each first right node and each first left node according to the fault weight after the update process is executed. If the left node chain of the first node does not exist or each first left node does not have an unresolved alarm event within the target historical period, then the step of determining the target node of the root cause of the alarm from the first node, each first right node and each first left node according to the fault weight after the update process is directly executed.

[0067] In addition, each first left node is a node that the first node depends on, and the alarm event of the first node may be caused by the abnormality of each first left node. Therefore, by adding the weight of the first node to each first left node, the weight can be used to identify the first left node as the alarm root cause node.

[0068] In one embodiment, the target node of the root cause of the alarm is determined from the first node, each first right node and each first left node according to the updated fault weight, including: comparing the updated fault weights of the first node, each first right node and each first left node, and taking the node with the largest fault weight as the target node.

[0069] Optionally, the node with the largest fault weight is found from the updated fault weight of the first node, the fault weights of each first right node, and the fault weights of each first left node. This node is the target node. It will be appreciated that, when executing the above method, the processing status of the alarm events of the first node, each first right node, and each first left node is all unresolved. Therefore, the processing status of the fault event of the target node derived from the first node, each first right node, and the first left node is also unresolved.

[0070] In summary, the flowchart for determining target nodes and fault events is as follows: Figure 2 shown.

[0071] In one embodiment, generating a self-healing solution based on a fault event includes: extracting root cause information from the fault event and determining a self-healing strategy based on the root cause information; and generating a self-healing solution based on the self-healing strategy.

[0072] Specifically, a fault event includes an alarm message, from which the root cause information can be extracted. For example, if the alarm message indicates a memory overflow in a container, the root cause is a memory overflow. If the alarm message indicates an infinite loop, the root cause is a code logic error. There are multiple root causes, but there are two possible self-healing strategies: a production patch repair strategy and a data modification repair strategy. If the root cause is a primary key conflict, incorrect variable value, batch skipping, memory overflow, or high CPU utilization, the self-healing solution is generated using the data modification repair strategy. If the root cause is a code write error or a logic error, the self-healing solution is generated using the production patch repair strategy.

[0073] Optionally, after extracting the root cause information, the system determines whether pre-maintained self-healing rules contain a solution that can resolve the root cause based on the root cause information. A self-healing rule is a system-supported self-healing solution configured in advance by operations and development engineers. It takes the form of a key-value pair, where the key represents the exception information and the value represents the solution. Examples include "Primary key conflict - delete duplicate primary key data and re-activate batch jobs" and "Server disk alarm - clean up binlog files." If the rule does not contain a solution, the target node and the fault event are sent to the operations and development engineers for analysis and review. If the rule does contain a solution, the system determines a self-healing strategy based on the self-healing rule, node dependencies, basic node information (including the application described by the node, whether it is an online node or a batch node, node name, run frequency, run events, and previous and subsequent dependent nodes), server information (including server IP address, server CPU and memory configuration), database username and password, table structure, SQL optimization suggestions, and the full set of SQL statements. A self-healing solution is then generated based on the self-healing strategy. Different self-healing strategies generate self-healing solutions in different ways, as detailed below.

[0074] In one embodiment, the self-healing strategy includes a production patch repair strategy, and generating a self-healing plan based on the self-healing strategy includes: obtaining location information of the defect code and repair suggestion information; and using the location information of the defect code and repair suggestion information as the self-healing plan.

[0075] Optionally, the defect code is located and its location information is obtained. Repair suggestions are then generated based on the query statement associated with the defect code. For example, if the query statement is a slow SQL statement, optimization suggestions are provided based on the table result index and execution plan. The defect code location information and repair suggestions are then sent to operations and maintenance engineers and development engineers as a self-healing solution.

[0076] In one embodiment, the self-healing strategy includes a data change repair strategy, and a self-healing plan is generated according to the self-healing strategy, including: determining the target data that needs to be changed for fault repair; generating a first data change script for the target node, a second data change script for each second left node dependent on the target node, and a third data change script for each second right node dependent on the target node according to the target data; and generating a self-healing plan according to the first data change script, the second data change script, and the third data change script.

[0077] Specifically, the target data that needs to be changed is determined, and the target data includes the first data for the target node, the second data for each second left node in the left node chain of the target node, and the third data for each second right node in the right node chain of the target node, wherein each node in the left node chain is the second left node that the target node depends on, and each node in the right node chain is the second right node that depends on the target node. The data change is executed through the change script, and therefore, a first data change script, a second data change script, and a third data change script are generated for the first data, the second data, and the third data, respectively, wherein the second data change script and the third data change script can also be referred to as the left dependency script and the right dependency script, respectively. Then, the first data change script, the second data change script, and the third data change script, as well as the backup script for data backup and the rollback script for data rollback, the database to be changed, the user name and password for logging into the database to be changed, the dependency check time, the change time, and the expected completion time are used to generate a self-healing solution according to a preset template. The alarm event number, alarm content, and corresponding self-healing plan are then sent to the operations and maintenance engineers and development engineers for review and approval. Approved self-healing plans are then executed to address the fault. If the operations and maintenance engineers or development engineers find any areas of the self-healing plan that they haven't analyzed, they can modify and improve the plan in the corresponding self-healing plan link.

[0078] In summary, the flowchart for generating a self-healing solution is as follows: Figure 3 shown.

[0079] In one embodiment, fault repair processing is performed based on a self-healing solution, including: performing a data change operation on each second left node based on a second data change script; after performing the data change operation on each second left node, performing a data change operation on the target node based on a first data change script; after performing the data change operation on the target node, performing a data change operation on each second right node based on a third data change script.

[0080] In one embodiment, the self-healing solution also includes a backup script and a rollback script; fault repair processing based on the self-healing solution also includes: performing a data backup operation based on the backup script before the data change operation is performed; and performing a data rollback operation based on the rollback script after an execution error occurs in the data change operation.

[0081] Optionally, a timer scanning component is used to periodically and automatically obtain self-healing plans with an approved status. The period can be set to 5 seconds. Then, it is determined whether the self-healing plan repairs the fault through data modification or production patch repair. If it is repaired through production patch repair, the processing status of the alarm event of the target node is changed to self-healing plan generated, and the generated self-healing plan is sent to the operation and maintenance engineer and development engineer as the change result; if it is repaired through data modification, the first check job, the second check job, and the third check job are called to execute the following method.

[0082] Optionally, first, according to the first data change script, the second data change script, the third data change script, the backup script and the rollback script in the self-healing scheme, the first data change job, the second data change job, the third data change job, the backup job and the rollback job are generated respectively, wherein the backup job is executed before the first data change job, the second data change job and the third data change job are executed, and is mainly used to back up the data before the change in advance; the rollback job is executed after the first data change job, the second data change job and the third data change job fail to change, and restores the data to the state before the change to eliminate the impact of the change. Secondly, the change time and the expected completion time in the self-healing scheme are configured to the first data change job, the dependent check time is configured to the second data change job, and the processing status of the alarm events of the target node, each second left node and each second right node is set to pending change. Then, execute the second data change operation, and use the first check operation to regularly check whether the dependent check time has arrived. If the dependent check time has arrived, start a periodic cycle to check whether the second data change operation is successfully executed. If the dependent check time has not arrived, continue to wait. If the second data change operation is successfully executed, start the backup operation, and execute the first data change operation according to the change time. If the second data change operation has not been completed or failed to execute at the change time, the processing status of the alarm event of the target node is set to change timeout. After the first data change operation starts to execute, the second check operation starts to regularly check whether the first data change operation is completed within the expected completion time. If the second data change operation is successfully executed, start the backup operation, and execute the first data change operation according to the change time. If the second data change operation has not been completed or failed to execute at the change time, set the processing status of the alarm event of the target node to change timeout. If the first data change job is successful, a third data change job is executed. A third check job periodically checks whether the third data change job has been successfully executed. If the third data change job is successful, the processing status of the alarm events for the target node, each second left node, and each second right node is set to "Change Success." This change success is sent as a change result to the operations and maintenance engineers and development engineers. If the third data change job fails, the cause is analyzed to determine whether it is caused by the change, primarily by the changed data, changed variables, batch jump failure, or expansion failure. If it is caused by the change, the third data change result is set to "Failure," a rollback job is executed, and the third data change job is reset. Finally, the processing status of the alarm events for the target node, each second left node, and each second right node is set to "Change Failure." The change failure result is sent to the operations and maintenance engineers and development engineers. If it is not caused by the change, this is considered a new fault anomaly, and steps 101-103 are executed. Furthermore, by updating the processing status of the alarm events for the target node, each second left node, and each second right node in real time, the fault repair process can be tracked.

[0083] To summarize, the flowchart for fault repair based on the self-healing solution is as follows: Figure 4 shown.

[0084] Based on the same inventive concept, embodiments of the present application also provide a monitoring alarm fault self-healing device for implementing the aforementioned monitoring alarm fault self-healing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the monitoring alarm fault self-healing device provided below can be found in the above-mentioned limitations of the monitoring alarm fault self-healing method and will not be further elaborated here.

[0085] In one embodiment, Figure 5 As shown, a monitoring alarm fault self-healing device is provided, and the monitoring alarm fault self-healing device 500 includes: an updating module 501, a positioning module 502, and a self-healing module 503, wherein:

[0086] The update module 501 is used to update the fault weight of the first node according to the alarm event of the first node and the alarm events of the first right nodes that depend on the first node after receiving the alarm information of the first node, and to update the fault weight of each first left node according to the alarm events of each first left node that the first node depends on.

[0087] The positioning module 502 is configured to determine the target node of the alarm root cause from the first node, each first right node and each first left node according to the updated fault weight, and determine the fault event of the fault root cause according to the alarm information of the target node.

[0088] The self-healing module 503 is configured to generate a self-healing solution according to a fault event and perform fault repair processing based on the self-healing solution.

[0089] In one embodiment, the update module 501 is specifically used to: perform a first update process on the fault weight of the first node according to the alarm event of the first node; after the first update process, perform a second update process on the fault weight of the first node according to the alarm events of each first right node.

[0090] In one embodiment, the update module 501 is specifically used to: if there is an unresolved alarm event for the first node within the target historical period, add the fault weight of the first node to the preset value; if there is no unresolved alarm event for the first node within the target historical period, reset the fault weight of the first node to the initial weight value.

[0091] In one embodiment, the updating module 501 is specifically configured to: if each first right node has an unresolved alarm event within the target historical period, perform a second updating process on the fault weight of the first node.

[0092] In one embodiment, the updating module 501 is further configured to: use the sum of the fault weights of each first right node and the current fault weight of the first node as the fault weight of the first node after the second update process.

[0093] In one embodiment, the updating module 501 is further configured to update the fault weight of each first left node if there is an unresolved alarm event for each first left node within the target historical period.

[0094] In one embodiment, the updating module is further configured to: for each first left node, use the sum of the fault weight of the first left node and the current fault weight of the first node as the updated fault weight of the first left node.

[0095] In one embodiment, the positioning module 502 is specifically configured to compare the updated fault weights of the first node, each first right node, and each first left node, and select the node with the largest fault weight as the target node.

[0096] In one embodiment, the self-healing module 503 is specifically configured to: extract root cause information from a fault event, determine a self-healing strategy based on the root cause information; and generate a self-healing solution according to the self-healing strategy.

[0097] In one embodiment, the self-healing strategy includes a production patch repair strategy, and the self-healing module 503 is specifically configured to: obtain location information of defective code and repair suggestion information; and use the location information of defective code and repair suggestion information as a self-healing solution.

[0098] In one embodiment, the self-healing strategy includes a data change repair strategy, and the self-healing module 503 is specifically used to: determine the target data that needs to be changed for fault repair; generate a first data change script for the target node, a second data change script for each second left node dependent on the target node, and a third data change script for each second right node dependent on the target node based on the target data; generate a self-healing plan based on the first data change script, the second data change script, and the third data change script.

[0099] In one embodiment, the self-healing module 503 is specifically used to: perform a data change operation on each second left node based on the second data change script; after performing the data change operation on each second left node, perform a data change operation on the target node based on the first data change script; after performing the data change operation on the target node, perform a data change operation on each second right node based on the third data change script.

[0100] In one embodiment, the self-healing solution also includes a backup script and a rollback script; the self-healing module 503 is further used to: perform a data backup operation based on the backup script before the data change operation is performed; and perform a data rollback operation based on the rollback script after an execution error occurs in the data change operation.

[0101] Each module in the aforementioned monitoring alarm fault self-healing device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in hardware form, or may be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0102] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store antenna posture information measurement data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an antenna posture information measurement method is implemented.

[0103] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0104] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0106] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0107] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0108] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A monitoring alarm fault self-healing method, characterized in that: The method comprises: After receiving the alarm information of the first node, the fault weight of the first node is updated for the first time according to the alarm event of the first node; after the first update, the fault weight of the first node is updated for the second time according to the alarm event of each first right node that depends on the first node, and the fault weight of each first left node is updated according to the fault weight of the first node after the second update and the alarm event of each first left node that the first node depends on; wherein, if the first node has an unresolved alarm event within the target historical period, the fault weight of the first node is added to the preset value; if the first node has no unresolved alarm event within the target historical period, the fault weight of the first node is reset to the initial weight value; if each first right node has an unresolved alarm event within the target historical period, the sum of the fault weight of each first right node and the current fault weight of the first node is used as the fault weight of the first node after the second update; Determining a target node of the alarm root cause from the first node, each of the first right nodes, and each of the first left nodes according to the updated fault weight, and determining a fault event of the alarm root cause according to the alarm information of the target node; A self-healing plan is generated according to the fault event, and fault repair processing is performed based on the self-healing plan.

2. The method according to claim 1, characterized in that The updating of the fault weight of each first left node according to the alarm event of each first left node on which the first node depends includes: If each of the first left nodes has an unresolved alarm event within the target historical period, the fault weight of each of the first left nodes is updated.

3. The method according to claim 2, characterized in that The updating of the fault weight of each first left node includes: For each first left node, the sum of the fault weight of the first left node and the current fault weight of the first node is used as the updated fault weight of the first left node.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining a target node of the alarm root cause from the first node, each of the first right nodes, and each of the first left nodes according to the updated fault weight includes: The updated fault weights of the first node, each of the first right nodes, and each of the first left nodes are compared, and the node with the largest fault weight is used as the target node.

5. The method according to claim 1, wherein Generating a self-healing solution according to the fault event includes: Extracting root cause information from the fault event and determining a self-healing strategy based on the root cause information; The self-healing solution is generated according to the self-healing strategy.

6. The method according to claim 5, characterized in that The self-healing strategy includes a production patch repair strategy, and generating the self-healing solution according to the self-healing strategy includes: Get the location information and repair suggestions of the defective code; The location information of the defect code and the repair suggestion information are used as the self-healing solution.

7. The method according to claim 5, characterized in that The self-healing strategy includes a data change repair strategy, and generating the self-healing solution according to the self-healing strategy includes: Determine the target data that needs to be changed to fix the problem; generating, according to the target data, a first data change script for the target node, a second data change script for each second left node that the target node depends on, and a third data change script for each second right node that depends on the target node; The self-healing solution is generated according to the first data change script, the second data change script, and the third data change script.

8. The method according to claim 7, characterized in that The fault repair process based on the self-healing solution includes: Performing a data change operation on each of the second left nodes based on the second data change script; After performing the data change operation on each of the second left nodes, performing the data change operation on the target node based on the first data change script; After performing the data change operation on the target node, performing the data change operation on each of the second right nodes based on the third data change script.

9. The method according to claim 8, characterized in that The self-recovery solution also includes a backup script and a rollback script; The fault repair process based on the self-healing solution further includes: Before the data modification operation is performed, performing a data backup operation based on the backup script; After an execution error occurs in the data change operation, a data rollback operation is performed based on the rollback script.

10. A monitoring alarm fault self-healing device, characterized in that: The device comprises: An updating module is used to, after receiving the alarm information of the first node, perform a first updating process on the fault weight of the first node according to the alarm event of the first node; after the first updating process, perform a second updating process on the fault weight of the first node according to the alarm event of each first right node that depends on the first node, and update the fault weight of each first left node according to the fault weight of the first node after the second update and the alarm event of each first left node that the first node depends on; wherein, if the first node has an unresolved alarm event within the target historical period, the fault weight of the first node is added to a preset value; if the first node does not have an unresolved alarm event within the target historical period, the fault weight of the first node is reset to an initial weight value; if each first right node has an unresolved alarm event within the target historical period, the sum of the fault weight of each first right node and the current fault weight of the first node is used as the fault weight of the first node after the second updating process; a positioning module, configured to determine a target node of the alarm root cause from the first node, each of the first right nodes, and each of the first left nodes according to the updated fault weight, and determine a fault event of the fault root cause according to the alarm information of the target node; The self-healing module is used to generate a self-healing plan according to the fault event and perform fault repair processing based on the self-healing plan.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images