Fault processing method and device, electronic equipment and storage medium

By preconfiguring the correspondence between the fault type and the self-healing program, and automatically orchestrating and executing the fault self-healing program, the problem of low fault processing efficiency in large-scale application systems is solved, and rapid fault processing and operation and maintenance efficiency is improved.

CN118916198APending Publication Date: 2024-11-08PEOPLE'S INSURANCE COMPANY OF CHINA

Patent Information

Application Number
CN202410954721.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In large-scale application systems, traditional fault handling methods require manual participation by operation and maintenance personnel, resulting in inefficient fault handling time, especially in a massive application node environment, it is difficult to quickly locate and deal with faults.

Method used

By preconfiguring the correspondence between the fault type and the fault self-healing program, detect the fault information and determine the fault type and arrange the corresponding self-healing program to automatically handle the fault quickly.

Benefits of technology

It improves the efficiency of fault handling, reduces the time and energy investment of operation and maintenance personnel, and can quickly process the fault information of application nodes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916198A_ABST
    Figure CN118916198A_ABST
Patent Text Reader

Abstract

The invention discloses a fault processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a fault type and a fault occurrence moment corresponding to the fault information of at least one application node when the fault information of the at least one application node is monitored; determining a fault self-healing program for processing the fault information of the at least one application node based on a fault type corresponding to the fault information of the at least one application node and a pre-configured corresponding relationship between the fault type and the fault self-healing program; and based on the fault occurrence time corresponding to the fault information of the at least one application node, arranging the execution sequence of a fault self-healing program corresponding to the fault information of the at least one application node, executing the arranged fault self-healing program, and processing the fault information of the at least one application node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault handling method, device, electronic device and storage medium. Background Art

[0002] At present, for failures in application nodes in application systems, the traditional operation and maintenance process usually includes: ① receiving a failure alarm, ② logging into the application node that reported the alarm, ③ troubleshooting the failure in the application node that reported the alarm, and ④ restoring the operating status of the faulty node. The entire process often requires manual participation by operation and maintenance personnel. When the number of application nodes in the application system being maintained is small, the timeliness of the entire system's failure handling is still within the controllable range of the operation and maintenance personnel.

[0003] With the explosive growth of business needs of application systems and the continuous evolution of IT technology, the number of application nodes in application systems has become increasingly large. When faced with an application system containing a large number of application nodes, how to quickly locate and handle faults in the system to improve fault handling efficiency still requires further solutions. Summary of the invention

[0004] Multiple aspects of the present application provide a fault handling method, device, electronic device and storable medium for improving fault handling efficiency.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows:

[0006] In the first aspect, a fault handling method is proposed, comprising:

[0007] When fault information of at least one application node is monitored, obtaining a fault type and a fault occurrence time corresponding to the fault information of at least one application node;

[0008] Determine a fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and the pre-configured correspondence between the fault type and the fault self-healing program;

[0009] Based on the fault occurrence time corresponding to the fault information of at least one of the application nodes, the fault self-healing program corresponding to the fault information of at least one of the application nodes is arranged in an execution order, and the arranged fault self-healing program is executed to process the fault information of at least one of the application nodes.

[0010] In a second aspect, a fault handling device is proposed, comprising:

[0011] A fault location module, configured to obtain a fault type and a fault occurrence time corresponding to the fault information of at least one application node when the fault information of at least one application node is monitored;

[0012] a program determination module, configured to determine a fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and a pre-configured correspondence between the fault type and the fault self-healing program;

[0013] A fault processing module is used to arrange the execution order of the fault self-healing program corresponding to the fault information of at least one application node based on the fault occurrence time corresponding to the fault information of at least one application node, and execute the arranged fault self-healing program to process the fault information of at least one application node.

[0014] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0015] The memory is used to store computer programs;

[0016] The processor, coupled to the memory, is configured to execute the computer program to:

[0017] When fault information of at least one application node is monitored, obtaining a fault type and a fault occurrence time corresponding to the fault information of at least one application node;

[0018] Determine a fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and the pre-configured correspondence between the fault type and the fault self-healing program;

[0019] Based on the fault occurrence time corresponding to the fault information of at least one of the application nodes, the fault self-healing program corresponding to the fault information of at least one of the application nodes is arranged in an execution order, and the arranged fault self-healing program is executed to process the fault information of at least one of the application nodes.

[0020] In a fourth aspect, a computer-readable storage medium storing a computer program is proposed. When the computer program is executed by a processor, the processor is caused to implement the steps of the fault handling method described in the first aspect.

[0021] In a fifth aspect, a computer program product is proposed. The computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the steps of the fault handling method described in the first aspect.

[0022] The above technical solution can at least achieve the following technical effects:

[0023] By adopting the fault handling method provided in the embodiment of the present application, when fault information is monitored, the fault type and fault occurrence time corresponding to the fault information of the application node can be determined. Since the correspondence between different fault types and fault self-healing programs is pre-established, after determining the fault type of at least one application node, the fault self-healing program matching it can be directly determined according to the fault type, and the execution order of the fault self-healing program corresponding to at least one application node can be arranged based on the fault occurrence time of each application node, so that the fault information of at least one application node can be quickly processed based on the arranged fault self-healing program, which effectively improves the fault handling efficiency and reduces the time and energy of operation and maintenance personnel consumed in the fault handling process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A schematic diagram of an implementation flow of a fault handling method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a fault self-healing program corresponding to insufficient disk space as the fault type in the fault handling method provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of a fault self-healing program corresponding to a fault type of application node downtime or pseudo-death in a fault handling method provided in an embodiment of the present application;

[0028] Figure 4 A flowchart of a fault self-healing program corresponding to a middleware node abnormality in a fault handling method provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of an interface for configuring fault self-healing parameters in a fault handling method provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of a process for starting a fault self-healing program in a fault handling method provided in an embodiment of the present application;

[0031] Figure 7 A flowchart of the fault handling method provided in the embodiment of the present application applied in an actual scenario;

[0032] Figure 8A schematic diagram of the structure of a fault handling device provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0035] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0037] First, the terms involved in one or more embodiments of the present application are explained.

[0038] Prometheus is an open source service monitoring system and time series database, which provides a common data model and fast data collection, storage and query interface. Its core component, Prometheus server, regularly pulls data from statically configured monitoring targets or self-labels automatically configured based on service discovery. When the newly pulled data is larger than the configured memory cache, the data will be persisted to the storage device.

[0039] Phantom death refers to a state where a node appears to be unresponsive but is actually still running.

[0040] Automated operation and maintenance refers to the use of operation and maintenance tools to solve large and complex tasks in production, monitor changes in the production environment, automatically respond to and handle events, etc., thereby improving operation and maintenance efficiency, reducing the probability of major problems, and more comprehensively protecting the production environment.

[0041] Middleware is a type of software that sits between application systems and system software. It uses the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, thereby achieving the purpose of resource sharing and function sharing.

[0042] Redis Manager, as a one-stop management platform for Redis, supports the management, monitoring, creation, and alarm functions of Redis clusters. It greatly reduces the operation and maintenance costs of Redis clusters, can monitor the cluster in real time, conduct accurate troubleshooting when Redis problems occur, or restore the scene of Redis failure by analyzing historical data. It also supports the alarm function of custom monitoring rules.

[0043] In order to solve the above-mentioned problems existing in the prior art, the embodiment of the present application provides a fault handling method. Since the correspondence between different fault types and fault self-healing programs is established in advance, after determining the fault type of at least one application node, the fault self-healing program matching it can be directly determined according to the fault type, and the execution order of the fault self-healing program corresponding to at least one application node can be arranged based on the time when the fault of each application node occurs, so that the fault information of at least one application node can be quickly processed based on the arranged fault self-healing program, which effectively improves the fault handling efficiency and reduces the time and energy of the operation and maintenance personnel consumed in the fault handling process. Specifically, in the present application, a fault handling method is provided, and the present application also involves a fault handling device for application software, an electronic device and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0044] Specifically, a schematic diagram of an implementation flow of a fault handling method provided by one or more embodiments of the present application is as follows: Figure 1 As shown, including:

[0045] Step 110: when fault information of at least one application node is monitored, the fault type and fault occurrence time corresponding to the fault information of at least one application node are obtained.

[0046] Optionally, the execution body of the fault handling method in the embodiment of the present application may include an API integration interface adapted to each monitoring platform, so as to connect to as many different monitoring platforms as possible, and capture the required data from different monitoring platforms according to the parameters configured on the fault self-healing configuration page, so as to realize fault self-healing requirements in multiple scenarios.

[0047] As an example, fault information can be monitored by fault monitoring platforms such as the operation and maintenance tool Prometheus. For example, the operation and maintenance tool Prometheus can periodically collect the CPU and memory utilization of each application node in the monitored application system cluster from the application system cluster, and capture the running status of the Redis node from the Redis Manager. When it is determined that the collected data is abnormal, the fault information of the alarm can be generated according to the preset alarm template. The fault information of the alarm may include information such as the source identifier of the alarm, the application system identifier of the alarm, the fault content of the alarm, the application node IP and port number of the alarm, the time when the fault occurred, and the ID of the fault information.

[0048] Among them, the alarm source identifier is used to indicate the operation and maintenance tool identifier of the source of the alarm fault information, such as the identifier of the operation and maintenance tool Prometheus. The application system identifier of the alarm is used to indicate the application system identifier of the source of the alarm fault information. For example, the application systems monitored by Prometheus include application system a, application system b and application system c, and the application system identifier of the alarm can be a, b or c. The host IP and port number of the alarm application system are used to indicate the IP and port number of the application node where the fault occurs in the alarmed application system. The time of fault occurrence is used to indicate the time when the fault information is generated. The ID of the fault information is used to uniquely indicate the identifier of the fault information. The fault type of each application node can be determined based on the fault code contained in the fault content of the alarm.

[0049] Each monitored host in Prometheus can provide an interface for outputting monitoring data through a dedicated exporter program. It will collect monitoring data at the target and expose an HTTP interface for Prometheus server to query. Prometheus periodically collects data through HTTP-based pull. All monitored targets need to be included in the monitoring system in advance for time series data collection, storage, alarm and display. Monitoring targets can be specified in a static form through configuration information, or Prometheus can be dynamically managed through a service discovery mechanism. Prometheus can directly use the API Server as a service discovery system to dynamically discover and monitor all monitored objects in the cluster.

[0050] It should be understood that in order to meet the requirements of implementing the fault handling method provided in the embodiment of the present application through an automated process, a series of standardized operations can be completed in advance for the application system that needs to perform fault handling: ① Application system deployment standardization: Specifically, the directory, log directory, startup script naming, etc. of the application system deployment can be standardized to ensure that the self-healing script does not need to be modified too much after the access fault is self-healed, but can be adapted through parameterized configuration; ② Monitoring alarm standardization: The alarm template used to generate fault information can be standardized in advance, so that both the operation and maintenance team and the self-healing platform can quickly locate the problem through the fault information notification of the alarm.

[0051] Step 120: Determine a fault self-healing program for processing the fault information of at least one application node based on the fault type corresponding to the fault information of at least one application node and the pre-configured correspondence between the fault type and the fault self-healing program.

[0052] Optionally, since different fault types have different degrees of impact on the continuity of services in the application system, the embodiments of the present application can pre-set the processing priorities corresponding to different fault types based on this, so that when arranging the execution order of multiple fault self-healing programs that need to be executed, the order of the fault self-healing programs corresponding to the fault types with higher processing priorities is set at a relatively front position to give priority to such fault self-healing programs. Specifically, based on the fault occurrence time corresponding to the fault information of at least one application node, the fault self-healing programs corresponding to the fault information of at least one application node are arranged in the execution order, including:

[0053] Determine, according to the preset processing priorities of the multiple fault types, a processing priority order of the fault types corresponding to the fault information of at least one application node;

[0054] Based on the fault occurrence time corresponding to the fault information of at least one application node and the processing priority order, the fault self-healing program corresponding to the fault information of at least one application node is arranged in an execution order.

[0055] For example, the fault types may include insufficient disk space, excessive CPU or memory usage, application node downtime or pseudo-death, middleware node abnormality, etc. The preset processing priorities of multiple fault types are: middleware node abnormality > application node downtime or pseudo-death > excessive CPU or memory usage > insufficient disk space.

[0056] Among them, when the fault type is insufficient disk space, when such a fault occurs, the disk space cleaning self-healing process is called to clean up according to the pre-configured cleaning strategy to achieve self-healing; and for the disk that cannot be cleaned, an alarm is sent to the operation and maintenance personnel for capacity expansion. Figure 2The fault type in the fault handling method provided in the embodiment of the present application is a fault self-healing process diagram corresponding to insufficient disk space. Specifically, expired files in the corresponding directory can be cleaned up according to the file retention period configured for each file in the file directory.

[0057] When the fault type is high CPU or memory usage, when such a fault occurs, the resource fault self-healing process is called, which includes checking whether there is an OutOfMemoryError or FullGC error in the log. If such an error exists, the application node can be restarted directly to release resources, and the fault handling result is sent to the operation and maintenance personnel after self-healing is achieved; otherwise, the alarm information is directly pushed to the operation and maintenance personnel.

[0058] When the fault type is application node downtime or pseudo-death, when such a fault occurs, the application node self-healing process is called, the application node is restarted to achieve self-healing and the results are sent to the operation and maintenance personnel. When multiple nodes in the same cluster are abnormal, the first-in-first-out queue algorithm is used to queue the abnormal nodes in the cluster and achieve self-healing of the faults one by one. Figure 3 A schematic diagram of a fault self-healing process corresponding to a fault type of application node downtime or suspended animation in a fault handling method provided in an embodiment of the present application. Figure 3 It may include: s31, determine whether the application node is a supervisor node, if so, execute s32, if not, execute s34; s32, determine whether the supervisor node is alive, if not, execute s33, if so, execute s34; s33, start the supervisor node; s34, determine whether there are multiple abnormal nodes in the cluster, if so, execute s35, if not, execute s36; s35, queue multiple abnormal nodes according to the time when the abnormal node fails; s36, restart the abnormal node.

[0059] When the fault type is a middleware node exception, the middleware currently used widely include redis, nginx, consul, kafka, etc. When such a fault occurs, the corresponding middleware self-healing process is matched according to the type of middleware to achieve middleware fault self-healing. When multiple machines are abnormal under the same business, the first-in-first-out queue algorithm is used to queue the abnormal nodes in the same business and achieve fault self-healing one by one. Figure 4 A schematic diagram of the process of self-healing of a middleware node abnormality in a fault handling method provided in an embodiment of the present application is provided. Figure 4It may include: s40, determine the middleware type, if it is redis, execute s41, if it is nginx, execute s42, if it is consul, execute s44, if it is kafka, execute s43; s41, restart the redis abnormal node; s42, restart the nginx abnormal node; s43, restart the kafka abnormal node; s44, determine whether the consul server node is normal, if so, execute s46, if not, execute s45; s45, restart the consul server abnormal node; s46, determine whether the consul client node is normal, if not, execute s47; s47, restart the consul client abnormal node.

[0060] As an implementation mode, if the number of fault types corresponding to the fault information of at least one application node is multiple, the execution order of the fault self-healing program corresponding to the fault information of at least one application node is first determined according to the processing priority of the fault type corresponding to the fault information of at least one application node. If the same fault type includes multiple faulty application nodes, the execution order of the fault self-healing program corresponding to each fault type is determined according to the order of the fault occurrence time of each application node in each fault type.

[0061] Step 130, based on the fault occurrence time corresponding to the fault information of at least one application node, arrange the execution order of the fault self-healing program corresponding to the fault information of at least one application node, and execute the arranged fault self-healing program to process the fault information of at least one application node.

[0062] Optionally, in order to avoid frequent execution of the fault self-healing program affecting the stability of the application system, the embodiment of the present application may also set a target time period without the need for fault self-healing and a specified fault type for which the fault self-healing program does not need to be executed. Specifically, based on the fault type corresponding to the fault information of at least one application node and the correspondence between the pre-configured fault type and the fault self-healing program, determining the fault self-healing program for processing the fault information of at least one application node includes:

[0063] Determine a pre-configured target time period for indicating that the application node does not need to be repaired, and a specified fault type for indicating that the fault self-healing program does not need to be executed; wherein the specified fault type includes a host downtime fault and a network connection abnormality fault;

[0064] When the time when the fault of each application node occurs does not belong to the target time period and the fault type corresponding to the fault information of each application node does not belong to the specified fault type, based on the fault type corresponding to the fault information of at least one application node and the correspondence between the pre-configured fault type and the fault self-healing program, determine the fault self-healing program for processing the fault information of at least one application node.

[0065] Since the execution of fault self-healing programs often involves restarting application nodes, and frequent execution of fault self-healing programs will affect the stability of the application system, you can set a target time period that does not require fault self-healing and a specified fault type that does not require the execution of fault self-healing programs to control the execution of fault self-healing programs. Among them, the target time period that does not require fault self-healing may include:

[0066] (1) The set time period T that does not require self-healing is when the application system undergoes routine changes or upgrades, and no fault handling is required.

[0067] (2) The expiration time t1 is set. Before executing the fault self-healing program, it can be determined whether the event has expired. Expired events will not be processed for faults.

[0068] (3) A silent period t2 is set. When the same fault information occurs multiple times in a short period of time, the fault information that does not exceed this silent period will not be processed.

[0069] Figure 5 This is a schematic diagram of an interface for configuring fault self-healing parameters in the fault handling method provided in an embodiment of the present application. Figure 5 In the process, you can set corresponding fault self-healing parameters for various fault types in different business systems, set the fault information that triggers the fault self-healing process, and set the target time period when the fault self-healing program does not need to be executed (including the time period without self-healing, expiration time and silent time), such as CPU usage warning value, memory usage warning value, etc.

[0070] Figure 6 A schematic diagram of a process for starting a fault self-healing program in a fault handling method provided in an embodiment of the present application. Figure 6 The process shown may include: s61, obtaining the alarm fault information; s62, determining whether the fault information is generated within the self-healing time T, if so, ending the process, otherwise executing s63; s63, determining whether the fault information exceeds the expiration time t1, if so, ending the process, otherwise executing s64; s64, determining whether the fault information is within the silent time t2, if so, ending the process, if otherwise executing s65; s65, determining whether the application node is in the startup state, if so, executing the fault self-healing program matching process, if otherwise executing s65.

[0071] Among them, the designated fault types include host downtime faults and network connection abnormality faults. Since the fault self-healing program will not be available when a host downtime fault occurs, it is necessary to detect the application node before executing the fault self-healing program. Only when the application node is available can the application fault self-healing program be started and executed. Moreover, when a network connection abnormality fault occurs, the application node itself is not abnormal and does not need to perform fault self-healing. Based on this, the embodiment of the present application divides host downtime faults and network connection abnormality faults into designated type faults, so that before executing the fault self-healing program, it is determined that the fault type of the application node to be executed by the fault self-healing program is not the designated fault type, and then the fault self-healing program is executed in the corresponding application node to improve the fault self-healing efficiency.

[0072] It should be noted that in the cluster of the same application system, usually only one application node is allowed to perform fault self-healing at the same time, so as to avoid multiple application nodes or even all application nodes in the cluster performing fault self-healing at the same time, which may cause the service of the application system to be unavailable. As an example, the application nodes in the cluster of the same application system can be serialized. For a cluster with a supervisor node, the availability of the supervisor node can usually be checked first. Only after the fault of the current application node is successfully self-healed can the fault self-healing program of the next application node be called up for execution.

[0073] Optionally, since the fault self-healing program will not be available when a host downtime occurs, the embodiment of the present application can perform a live operation on the application node that will execute the fault self-healing program before executing the fault self-healing program. Specifically, executing the arranged fault self-healing program and processing the fault information of at least one application node include:

[0074] Determine the application node currently to be processed in the arranged fault self-healing program;

[0075] Determine whether the current application node to be processed is in the startup state;

[0076] When it is determined that the current application node to be processed is in the startup state, a fault self-healing program corresponding to the current application node to be processed is executed to process the fault information of the current application node to be processed.

[0077] Optionally, to avoid infinite loop calls to the fault self-healing program in scenarios where the fault self-healing cannot be handled, the embodiment of the present application may set an upper limit value for the fault self-healing, that is, a preset threshold value described below. When the number of fault self-healing times of the application node exceeds the preset threshold value, the execution of the fault self-healing program may be stopped. Specifically, executing the arranged fault self-healing program to process the fault information of at least one application node includes:

[0078] Determine the number of self-healing failures of the application node currently to be processed in the scheduled self-healing failure program within a historical time period;

[0079] When the number of fault self-healing times is less than or equal to a preset threshold, a fault self-healing program corresponding to the application node currently to be processed is executed to process the fault information of the application node currently to be processed.

[0080] Optionally, in order to promptly send the fault handling result and the fault information that cannot be handled by the fault self-healing program to the operation and maintenance personnel so that the operation and maintenance personnel can understand the fault handling situation in a timely manner, the method provided in the embodiment of the present application also includes:

[0081] Obtaining a fault handling result of at least one application node; sending the fault handling result of at least one application node to an operation and maintenance personnel of at least one application node; or,

[0082] When the number of self-healing times of the fault is greater than a preset threshold, the fault information corresponding to the application node currently to be processed is sent to the operation and maintenance personnel of the application node currently to be processed.

[0083] As an example, the executor of the fault handling method can be connected to multiple notification channels, such as SMS, email, and WeChat, so that the fault handling results can be pushed to relevant operation and maintenance personnel in a timely manner.

[0084] Figure 7 A flowchart of the fault handling method provided in the embodiment of the present application applied in an actual scenario. Figure 7 It may include: s71, determining whether to use middleware, if yes, executing s72; s72, determining whether the middleware node is abnormal, if no, executing s73, if yes, executing the middleware node abnormal self-healing process, and determining whether the self-healing is successful, and if the self-healing is not successful, determining whether the number of fault self-healing times is less than the preset threshold, if yes, continuing to execute the middleware node abnormal self-healing process, otherwise sending a notification; s73, determining whether the application node is down or suspended, if no, executing s74, if no, executing the application node down or suspended self-healing process, and determining whether the self-healing is successful, and if the self-healing is not successful, determining whether the number of fault self-healing times is less than the preset threshold, if yes, continuing to execute the application node down or suspended self-healing process, otherwise a notification is sent; s74, determine whether the CPU or memory usage is too high, if not, execute s75, if it is a resource fault self-healing process, and determine whether the self-healing is successful, and if the self-healing is not successful, determine whether the number of fault self-healing times is less than the preset threshold, if it is less than the preset threshold, continue to execute the resource fault self-healing process, otherwise a notification is sent; s75, determine whether the disk space is insufficient, if not, execute s76, if so, execute the disk space cleaning self-healing process, and determine whether the self-healing is successful, and if the self-healing is not successful, determine whether the number of fault self-healing times is less than the preset threshold, if it is less than the preset threshold, continue to execute the disk space cleaning self-healing process, otherwise a notification is sent, s76 sends a notification to the operation and maintenance personnel.

[0085] By adopting the fault handling method provided in the embodiment of the present application, when fault information is monitored, the target application system from which the fault information comes and the target fault cause corresponding to the fault information can be determined based on the system identifier and the fault cause identifier in the fault information. Since the correspondence between different fault causes and fault self-healing programs is established in advance for the target application system, after determining the fault cause of the target application system, the target fault self-healing program that matches it can be directly determined based on the fault cause. Therefore, based on the target fault self-healing program, the fault information of the target application system can be quickly processed, which effectively improves the fault handling efficiency and reduces the time and energy of the operation and maintenance personnel consumed in the fault handling process.

[0086] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 110 to 230 can be device A; for another example, the execution subject of steps 110 to 120 can be device A, and the execution subject of step 130 can be device B; and so on.

[0087] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 110, 120, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0088] Figure 8 FIG. 8 is a schematic diagram of a fault handling device 800 provided in an exemplary embodiment of the present application. Figure 8 As shown, the device 800 includes: a fault location module 810, a program determination module 820 and a fault processing module 830, wherein:

[0089] The fault location module 810 is used to obtain the fault type and fault occurrence time corresponding to the fault information of at least one application node when the fault information of at least one application node is monitored;

[0090] A program determination module 820, configured to determine a fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and a pre-configured correspondence between the fault type and the fault self-healing program;

[0091] The fault processing module 830 is used to arrange the execution order of the fault self-healing program corresponding to the fault information of at least one of the application nodes based on the fault occurrence time corresponding to the fault information of at least one of the application nodes, and execute the arranged fault self-healing program to process the fault information of at least one of the application nodes.

[0092] The fault handling device provided in the embodiment of the present application can determine the fault type and fault occurrence time corresponding to the fault information of the application node when fault information is monitored. Since the corresponding relationship between different fault types and fault self-healing programs is pre-established, after determining the fault type of at least one application node, the fault self-healing program matching it can be directly determined according to the fault type, and the execution order of the fault self-healing program corresponding to at least one application node can be arranged based on the fault occurrence time of each application node, so that the fault information of at least one application node can be quickly processed based on the arranged fault self-healing program, thereby effectively improving the fault handling efficiency and reducing the time and energy of operation and maintenance personnel consumed in the fault handling process.

[0093] In some exemplary embodiments, the fault processing module 830 arranges the execution order of the fault self-healing program corresponding to the fault information of at least one application node based on the fault occurrence time corresponding to the fault information of at least one application node, and is specifically used to:

[0094] Determine, according to the preset processing priorities of the multiple fault types, a processing priority order of the fault types corresponding to the fault information of at least one of the application nodes;

[0095] Based on the fault occurrence time corresponding to the fault information of at least one of the application nodes and the processing priority order, the fault self-healing program corresponding to the fault information of at least one of the application nodes is arranged in an execution order.

[0096] In some exemplary embodiments, the program determination module 820 determines the fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and the pre-configured correspondence between the fault type and the fault self-healing program, specifically for:

[0097] Determine a pre-configured target time period for indicating that the application node does not need to repair the fault, and a specified fault type for indicating that the fault self-healing program does not need to be executed; wherein the specified fault type includes a host downtime fault and a network connection abnormality fault;

[0098] When the time when the fault of each application node occurs does not belong to the target time period, and the fault type corresponding to the fault information of each application node does not belong to the specified fault type, based on the fault type corresponding to the fault information of at least one application node and the correspondence between the pre-configured fault type and the fault self-healing program, determine the fault self-healing program for processing the fault information of at least one application node.

[0099] In some exemplary embodiments, when the fault processing module 830 executes the arranged fault self-healing program and processes the fault information of at least one of the application nodes, it is specifically used to:

[0100] Determine the application node currently to be processed in the arranged fault self-healing program;

[0101] Determine whether the current application node to be processed is in a startup state;

[0102] When it is determined that the application node currently to be processed is in the startup state, a fault self-healing program corresponding to the application node currently to be processed is executed to process the fault information of the application node currently to be processed.

[0103] In some exemplary embodiments, when the fault processing module 830 executes the arranged fault self-healing program and processes the fault information of at least one of the application nodes, it is specifically used to:

[0104] Determine the number of self-healing failures of the application node currently to be processed in the arranged self-healing failure program within a historical time period;

[0105] When the number of fault self-healing times is less than or equal to a preset threshold, a fault self-healing program corresponding to the application node currently to be processed is executed to process the fault information of the application node currently to be processed.

[0106] In some exemplary embodiments, the device further includes a fault information sending module, which is used to:

[0107] Obtaining a fault handling result of at least one of the application nodes; sending the fault handling result of at least one of the application nodes to an operation and maintenance personnel of at least one of the application nodes; or,

[0108] When the number of self-healing times of the fault is greater than the preset threshold, the fault information corresponding to the current application node to be processed is sent to the operation and maintenance personnel of the current application node to be processed.

[0109] The fault handling device 800 can achieve Figure 1 to Figure 7 For details, please refer to Figure 1 to Figure 7 The fault handling method of the illustrated embodiment will not be described in detail.

[0110] Fig. 9 The following is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Fig. 9 As shown, the device includes: a memory 91 and a processor 92.

[0111] The memory 91 is used to store computer programs and can be configured to store various other data to support operations on the computing device. Examples of such data include instructions for any application or method operating on the computing device, contact data, phone book data, messages, pictures, videos, etc.

[0112] The processor 92 is coupled to the memory 91 and is used to execute the computer program in the memory 91, so as to: when fault information of at least one application node is monitored, obtain the fault type and fault occurrence time corresponding to the fault information of at least one application node; determine the fault self-healing program for processing the fault information of at least one application node based on the fault type corresponding to the fault information of at least one application node and the pre-configured correspondence between the fault type and the fault self-healing program; arrange the execution order of the fault self-healing program corresponding to the fault information of at least one application node based on the fault occurrence time corresponding to the fault information of at least one application node, and execute the arranged fault self-healing program to process the fault information of at least one application node.

[0113] The electronic device provided in the embodiment of the present application can determine the fault type and fault occurrence time corresponding to the fault information of the application node when fault information is monitored. Since the corresponding relationship between different fault types and fault self-healing programs is pre-established, after determining the fault type of at least one application node, the fault self-healing program matching it can be directly determined according to the fault type, and the execution order of the fault self-healing program corresponding to at least one application node can be arranged based on the fault occurrence time of each application node, so that the fault information of at least one application node can be quickly processed based on the arranged fault self-healing program, which effectively improves the fault handling efficiency and reduces the time and energy of operation and maintenance personnel consumed in the fault handling process.

[0114] Further, if Fig. 9 As shown, the electronic device also includes: a communication component 93, a display 94, a power component 95, an audio component 96 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the electronic device only includes Fig. 9 In addition, depending on the implementation form of the traffic playback device, Fig. 9The components in the dashed box are optional components, not mandatory components. For example, when the electronic device is implemented as a terminal device such as a smartphone, tablet computer or desktop computer, it may include Fig. 9 Components in the dashed box; when the electronic device is implemented as a server-side device such as a conventional server, cloud server, data center or server array, it may not include Fig. 9 Components within the dashed box.

[0115] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above-mentioned fault handling method embodiment.

[0116] Accordingly, the embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed, can implement each step that can be performed by an electronic device in the processing method embodiment of the above-mentioned design diagram. Optionally, the computer program product, in addition to executing each step in the step embodiment of the above-mentioned fault handling method.

[0117] Above Fig. 9 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may also include a near field communication (NFC) module, a radio frequency identification (RFID) technology, an infrared data association (IrDA) technology, an ultra-wideband (UWB) technology, a Bluetooth (BT) technology, etc.

[0118] Above Fig. 9 The memory in can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0119] Above Fig. 9The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0120] Above Fig. 9 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0121] Above Fig. 9 The audio component in can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0128] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0129] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0131] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A fault handling method, characterized in that: include: When fault information of at least one application node is monitored, obtaining a fault type and a fault occurrence time corresponding to the fault information of at least one application node; Determine a fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and the pre-configured correspondence between the fault type and the fault self-healing program; Based on the fault occurrence time corresponding to the fault information of at least one of the application nodes, the fault self-healing program corresponding to the fault information of at least one of the application nodes is arranged in an execution order, and the arranged fault self-healing program is executed to process the fault information of at least one of the application nodes.

2. The method according to claim 1, characterized in that The arranging the execution order of the fault self-healing program corresponding to the fault information of at least one application node based on the fault occurrence time corresponding to the fault information of at least one application node includes: Determine, according to the preset processing priorities of the multiple fault types, a processing priority order of the fault types corresponding to the fault information of at least one of the application nodes; Based on the fault occurrence time corresponding to the fault information of at least one of the application nodes and the processing priority order, the fault self-healing program corresponding to the fault information of at least one of the application nodes is arranged in an execution order.

3. The method according to claim 1, characterized in that The determining, based on the fault type corresponding to the fault information of at least one of the application nodes and the pre-configured correspondence between the fault type and the fault self-healing program, a fault self-healing program for processing the fault information of at least one of the application nodes comprises: Determine a pre-configured target time period for indicating that the application node does not need to repair the fault, and a specified fault type for indicating that the fault self-healing program does not need to be executed; wherein the specified fault type includes a host downtime fault and a network connection abnormality fault; When the time when the fault of each application node occurs does not belong to the target time period, and the fault type corresponding to the fault information of each application node does not belong to the specified fault type, based on the fault type corresponding to the fault information of at least one application node and the correspondence between the pre-configured fault type and the fault self-healing program, determine the fault self-healing program for processing the fault information of at least one application node.

4. The method according to claim 1, characterized in that The executing the arranged fault self-healing program to process the fault information of at least one of the application nodes includes: Determine the application node currently to be processed in the arranged fault self-healing program; Determine whether the current application node to be processed is in a startup state; When it is determined that the application node currently to be processed is in the startup state, a fault self-healing program corresponding to the application node currently to be processed is executed to process the fault information of the application node currently to be processed.

5. The method according to any one of claims 1 to 4, characterized in that: The executing the arranged fault self-healing program to process the fault information of at least one of the application nodes includes: Determine the number of self-healing failures of the application node currently to be processed in the arranged self-healing failure program within a historical time period; When the number of fault self-healing times is less than or equal to a preset threshold, a fault self-healing program corresponding to the application node currently to be processed is executed to process the fault information of the application node currently to be processed.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining a fault handling result of at least one of the application nodes; sending the fault handling result of at least one of the application nodes to an operation and maintenance personnel of at least one of the application nodes; or, When the number of self-healing times of the fault is greater than the preset threshold, the fault information corresponding to the current application node to be processed is sent to the operation and maintenance personnel of the current application node to be processed.

7. A fault handling device, characterized in that: include: A fault location module, configured to obtain a fault type and a fault occurrence time corresponding to the fault information of at least one application node when the fault information of at least one application node is monitored; a program determination module, configured to determine a fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and a pre-configured correspondence between the fault type and the fault self-healing program; A fault processing module is used to arrange the execution order of the fault self-healing program corresponding to the fault information of at least one application node based on the fault occurrence time corresponding to the fault information of at least one application node, and execute the arranged fault self-healing program to process the fault information of at least one application node.

8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: When fault information of at least one application node is monitored, obtaining a fault type and a fault occurrence time corresponding to the fault information of at least one application node; Determine a fault self-healing program for processing the fault information of at least one of the application nodes based on the fault type corresponding to the fault information of at least one of the application nodes and the pre-configured correspondence between the fault type and the fault self-healing program; Based on the fault occurrence time corresponding to the fault information of at least one of the application nodes, the fault self-healing program corresponding to the fault information of at least one of the application nodes is arranged in an execution order, and the arranged fault self-healing program is executed to process the fault information of at least one of the application nodes.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the fault handling method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the steps of the fault handling method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Fault processing method and device, equipment and readable storage medium

    CN113467983A

  • Fault self-healing method and device based on whole process, electronic equipment and storage medium

    CN117312053A

  • Fault processing method and device, electronic equipment and storage medium

    CN117971542A

  • Fault processing method and device, nonvolatile storage medium and electronic equipment

    CN118014558A

  • Fault processing method and apparatus, device, and storage medium

    WO2023011160A1

Cited By

  • Fault processing method and device, electronic equipment and storage medium

    CN117971542A