Business system fault processing method and device

Abstract

The invention discloses a service system fault processing method and device. Comprising the following steps: extracting at least one first fault feature vector of a plurality of first fault state data of a plurality of first monitoring entities in a first service system in a first time period; for each first monitoring entity, analyzing a first fault type corresponding to the at least one first fault feature vector by using a fault classification model, and determining at least one selectable fault self-healing action corresponding to each first fault feature vector and the first fault type; analyzing at least one first fault feature vector of each first monitoring entity and an association relationship between the at least one first fault feature vector by using a fault conduction prediction model to obtain a first fault conduction chain; and analyzing the first fault conduction chain by using a fault self-healing action selection model, and determining first fault self-healing actions of the first monitoring entities and an execution sequence thereof. According to the method and the device, the technical problems of low efficiency and high cost of troubleshooting and repairing faults in a service system in a manual mode are solved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method and device for handling business system failures. Background Art

[0002] With the continuous upgrading and transformation of enterprise business, there are many business systems within the enterprise and the business correlation between systems is strong. Once the internal system architecture of the enterprise is transformed into a loosely coupled layered structure, it is often necessary to call on multiple professional lines and multiple systems to jointly participate in troubleshooting system failures. And relying solely on the past experience and knowledge of operation and maintenance personnel to troubleshoot and handle failures is inefficient and costly.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for handling business system faults, so as to at least solve the technical problems of low efficiency and high cost of manually troubleshooting and repairing faults in the business system.

[0005] According to one aspect of an embodiment of the present application, a business system fault processing is provided, including: obtaining a plurality of first fault status data corresponding to a plurality of first monitoring entities in a first business system within a first time period, and performing feature extraction on the plurality of first fault status data of each first monitoring entity to obtain at least one corresponding first fault feature vector; for each first monitoring entity, using a pre-trained fault classification model to analyze at least one first fault feature vector of the first monitoring entity to obtain a first fault type of a fault occurring in the first time period of the first monitoring entity, and determining at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity in the first time period; determining an association relationship between each first monitoring entity in the first business system, and using a pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain a first fault conduction chain of the first business system in the first time period; using a pre-trained fault self-healing action selection model to analyze the first fault conduction chain to determine a first fault self-healing action of each first monitoring entity and an execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and sequentially executing the first fault self-healing action of each first monitoring entity in the execution order.

[0006] Optionally, obtaining multiple first fault status data corresponding to multiple first monitoring entities in the business system within the first time period includes: obtaining multiple monitoring entity event information corresponding to multiple first monitoring entities in the business system within the first time period, wherein the monitoring entity event information includes status data of the first monitoring entity when at least one fault event occurs within the first time period; determining the fault status data of each first monitoring entity within the first time period based on the monitoring entity event information of each first monitoring entity; and preprocessing the fault status data of each first monitoring entity within the first time period to obtain the first fault status data of each first monitoring entity within the first time period.

[0007] Optionally, the training process of the fault classification model includes: obtaining multiple groups of first training samples, wherein the first training samples include: a second fault feature vector obtained by feature extraction of second fault state data of the second monitoring entity within a historical time period and a second fault type of the second monitoring entity; constructing a first neural network model; and iteratively training the first neural network model using the multiple groups of first training samples to obtain a fault classification model.

[0008] Optionally, the first neural network model is iteratively trained using multiple groups of first training samples to obtain a fault classification model, including: in each round of iteration, the second fault feature vector of each first training sample in the training sample batch corresponding to the current round of iteration is input into the first neural network model to obtain the predicted fault type output by the first neural network model; a first target loss function is constructed based on the second fault type of each first training sample in the training sample batch corresponding to the multiple rounds of iteration and the corresponding predicted fault type; model parameters of the first neural network model are adjusted based on the first target loss function until the model converges to obtain a fault classification model.

[0009] Optionally, determining at least one optional fault self-healing action corresponding to each first fault characteristic vector and first fault type of the first monitoring entity within the first time period includes: determining at least one group of operation and maintenance data corresponding to a first fault characteristic vector and a first fault type of the first monitoring entity within the first time period based on a preset operation and maintenance database, wherein the operation and maintenance data at least include: a fault handling strategy and self-healing parameters; for each group of operation and maintenance data, filling the operation and maintenance data into a preset fault self-healing action template, and constructing action instructions and execution scripts corresponding to the operation and maintenance data, to obtain at least one optional fault self-healing action corresponding to a first fault characteristic vector and a first fault type of the first monitoring entity within the first time period, wherein the optional fault self-healing action at least includes: a fault handling strategy, self-healing parameters, action instructions, and an execution script.

[0010] Optionally, the training process of the fault conduction prediction model includes: obtaining multiple groups of second training samples, wherein the second training samples include: second fault feature vectors obtained by respectively extracting features from second fault status data corresponding to at least two second monitoring entities within a historical time period and a connection relationship between at least two second monitoring entities, and a second fault conduction path between at least two second monitoring entities; constructing a second neural network model; and iteratively training the second neural network model using the multiple groups of second training samples to obtain a fault conduction prediction model.

[0011] Optionally, the second neural network model is iteratively trained using multiple groups of second training samples to obtain a fault conduction prediction model, including: in each round of iteration, the second fault feature vectors and connection relationships of at least two second monitoring entities in each second training sample in the training sample batch corresponding to the current round of iteration are input into the second neural network model to obtain a predicted fault conduction path output by the second neural network model; a second target loss function is constructed based on the second fault conduction paths of at least two second monitoring entities in each second training sample in the training sample batch corresponding to the multiple rounds of iteration and the corresponding predicted fault conduction paths; and model parameters of the second neural network model are adjusted based on the second target loss function until the model converges to obtain a fault conduction prediction model.

[0012] Optionally, the training process of the fault self-healing action selection model includes: obtaining multiple groups of third training samples, wherein the third training samples include: the second fault conduction chain of the second business system in a historical time period, the second fault self-healing actions of each second monitoring entity in the second business system, and the execution order of the second fault self-healing actions: constructing a third neural network model; using multiple groups of third training samples to iteratively train the third neural network model to obtain the fault self-healing action selection model.

[0013] Optionally, the third neural network model is iteratively trained using multiple groups of third training samples to obtain a fault self-healing action selection model, including: in each round of iteration, the second fault conduction chain of the second business system in each second training sample in the training sample batch corresponding to the current round of iteration within the historical time period is input into the third neural network model to obtain the predicted fault self-healing action of each second monitoring entity in the second business system and the predicted execution order of the predicted fault self-healing action output by the third neural network model; a third objective loss function is constructed based on the second fault self-healing action of each second monitoring entity in the second business system of each third training sample in the training sample batch corresponding to multiple rounds of iteration and the execution order of the second fault self-healing action and the predicted fault self-healing action of each second monitoring entity in the corresponding second business system; model parameters of the third neural network model are adjusted based on the third objective loss function until the model converges to obtain the fault self-healing action selection model.

[0014] According to another aspect of an embodiment of the present application, a business system fault processing device is also provided, including: an acquisition module, used to obtain multiple first fault status data corresponding to multiple first monitoring entities in a first business system within a first time period, and perform feature extraction on the multiple first fault status data of each first monitoring entity to obtain at least one corresponding first fault feature vector; a fault type prediction module, used to analyze at least one first fault feature vector of the first monitoring entity for each first monitoring entity using a pre-trained fault classification model, obtain the first fault type of the first monitoring entity that fails within the first time period, and determine the correspondence between each first fault feature vector and the first fault type of the first monitoring entity within the first time period. a fault conduction prediction module, used to determine the association relationship between each first monitoring entity in the first business system, and use the pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain a first fault conduction chain of the first business system in the first time period; a fault processing module, used to analyze the first fault conduction chain by using the pre-trained fault self-healing action selection model to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and execute the first fault self-healing action of each first monitoring entity in sequence according to the execution order.

[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned business system fault handling method by running the computer program.

[0016] According to another aspect of an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein the computer program implements the above-mentioned business system fault handling method when executed by a processor.

[0017] In an embodiment of the present application, at least one first fault feature vector of a single first monitoring entity is analyzed by a fault type classification model to predict the first fault type of the first monitoring entity that fails within a first time period, and to determine at least one optional fault self-healing action corresponding to the at least one first fault feature vector and the first fault type of the first monitoring entity; then, starting from the business system as a whole, the fault conduction prediction model is used to analyze the association relationship between the first monitoring entities in the first business system and at least one first fault feature vector of each first monitoring entity to predict the first fault conduction chain of the first business system that fails within the first time period; then, the first fault conduction chain is analyzed by a fault self-healing action selection model to determine the first fault self-healing action that is most suitable for the current scenario and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity; finally, the first fault self-healing actions of each first monitoring entity are executed in sequence according to the execution order, so as to efficiently and low-costly troubleshoot and repair the fault problem of the first business system, thereby solving the technical problem of low efficiency and high cost of troubleshooting and repairing faults in the business system due to manual methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a business system fault handling method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of an optional business system fault handling method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional business system knowledge graph according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of the structure of an optional business system fault processing device according to an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0027] Example 1

[0028] According to an embodiment of the present application, a method embodiment of a business system fault handling method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for handling a business system failure. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the business system fault handling method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the business system fault handling method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other monitoring entities through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0034] In the above operating environment, Figure 2 is a flow chart of an optional method for handling business system failures according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S208, wherein:

[0035] Step S201, obtaining a plurality of first fault status data corresponding to a plurality of first monitoring entities in a first business system within a first time period, and performing feature extraction on the plurality of first fault status data of each first monitoring entity to obtain at least one corresponding first fault feature vector.

[0036] In the technical solution provided in step S201, the first monitoring entity generally refers to all logical units in the first business system that can run and feedback operation status information, including but not limited to: software, hardware devices or other software and hardware hybrids (such as modules, components, database nodes, virtual machines, subsystems, network devices, etc.). The first fault status data is used to reflect the operation status data of the corresponding first monitoring entity when a fault occurs within the first time period, that is, the valid data related to the fault details fed back in the system log or probe information of the first monitoring entity. In addition, feature extraction is performed on the first fault status data of the first monitoring entity in order to obtain a first fault feature vector that can be processed by a model, so as to facilitate subsequent analysis using the model to obtain the corresponding fault type.

[0037] Step S204: For each first monitoring entity, use a pre-trained fault classification model to analyze at least one first fault feature vector of the first monitoring entity to obtain a first fault type of the first monitoring entity that fails within a first time period, and determine at least one optional fault self-healing action corresponding to each first fault feature vector and first fault type of the first monitoring entity within the first time period.

[0038] In the technical solution provided in step S204, each first entity in the first business system is traversed, and the following steps are first performed: at least one first fault feature vector of the first monitoring entity is input into a fault classification model, and the first fault type of the first monitoring entity that fails within a first time period output by the fault classification model is obtained, thereby obtaining at least one pairing consisting of at least one "first fault feature vector and first fault type" of the first monitoring entity; and then, determining at least one optional fault self-healing action corresponding to each pairing.

[0039] Step S206, determine the association relationship between each first monitoring entity in the first business system, and use the pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain the first fault conduction chain of the first business system in the first time period.

[0040] In the technical solution provided in step S206, the association relationship between each first monitoring entity in the first business system is determined; then, at least one first fault feature vector of each first monitoring entity in the first business system and the association relationship therebetween are analyzed to obtain a first fault conduction chain of the first business system in the first time period, wherein the first fault conduction chain can reflect the conduction path and conduction direction of the failure of each first monitoring entity in the first business system in the first time period.

[0041] Step S208, using a pre-trained fault self-healing action selection model to analyze the first fault conduction chain, so as to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and execute the first fault self-healing action of each first monitoring entity in sequence according to the execution order.

[0042] In the technical solution provided in step S208, since at least one "first fault feature vector and first fault type" pairing of a single first monitoring entity corresponds to at least one optional fault self-healing action, and in order to combine the specific scenario, select the fault self-healing action that best suits the current scenario from these optional fault self-healing actions, the present application proposes to input the first fault conduction chain obtained in the above step S206 into the fault self-healing action selection model, so as to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity based on historical experience and relevant operation and maintenance knowledge. It should be noted that the number of the first fault self-healing actions of the above first monitoring entity may be one or more, and the execution order between multiple first fault self-healing actions of the same first monitoring entity may not be adjacent to each other. Finally, the first fault self-healing actions of each first monitoring entity are executed in sequence according to the execution order, so as to quickly handle the fault of the business system and reduce the fault handling cost.

[0043] The above method of this embodiment is further introduced below.

[0044] As an optional implementation, in the technical solution provided in the above step S202, the method may include:

[0045] Step S2021, obtaining a plurality of monitoring entity event information corresponding to a plurality of first monitoring entities in the business system within a first time period;

[0046] Step S2022, determining the fault status data of each first monitoring entity within the first time period based on the monitoring entity event information of each first monitoring entity;

[0047] Step S2023: pre-process the fault status data of each first monitoring entity in the first time period to obtain the first fault status data of each first monitoring entity in the first time period.

[0048] In the above embodiment, for each first monitoring entity, first, multiple monitoring entity event information corresponding to the first monitoring entity in the first time period is obtained from multiple data sources, wherein the types of the monitoring entity event information include but are not limited to: log files, probe information, etc. When the monitoring entity event information is a log file, the corresponding data source may be a log server, a log management tool, a security situation awareness platform, etc.; when the monitoring entity event information is a probe information, the corresponding data source may be a network security device probe, a monitoring platform probe, etc. In addition, the acquired monitoring entity event information includes the state data of at least one fault event occurring in the first monitoring entity in the first time period. For example, if the first monitoring entity is a host or a virtual machine, its state data may include but is not limited to: CPU data (such as CPU usage, CPU core temperature, CPU load, etc.), memory data (such as used memory, free memory, memory usage, etc.), network connection data (such as the number of network connections, the data transmission rate of the network interface, the usage status of the network port, network delay and packet loss, etc.), etc. In addition, the above-mentioned fault events refer to events caused by human factors, software and hardware defects or failures, natural disasters, etc., and these events may cause harm to the data in the network or system. Then, the fault status data of the first monitoring entity in the first time period is extracted from the multiple monitoring entity event information of the first monitoring entity. In order to ensure the accuracy of subsequent fault judgment, the fault status data can also be preprocessed to obtain the corresponding first fault status data, wherein the preprocessing includes but is not limited to: data cleaning, formatting, denoising, etc.

[0049] As an optional implementation, in the technical solution provided in the above step S204, the training process of the above fault classification model may include:

[0050] Step S11, obtaining multiple groups of first training samples.

[0051] Specifically, the first training sample includes: a second fault feature vector obtained by extracting features from the second fault state data of the second monitoring entity in a historical time period and a second fault type of the second monitoring entity. The second monitoring entity may be a monitoring entity in any business system, such as the first monitoring entity in this application and / or other monitoring entities other than the first monitoring entity in multiple business systems. In addition, the second fault state data may be obtained by referring to steps S2021-S2023, which will not be described in detail here.

[0052] Step S12, constructing a first neural network model.

[0053] Among them, the model type of the above-mentioned first neural network model includes but is not limited to: machine learning model, deep learning model, model based on attention mechanism, etc. The embodiment of the present application does not impose specific restrictions on the model type, and it can be selected according to the actual application scenario.

[0054] Step S13, iteratively training the first neural network model using multiple groups of first training samples to obtain a fault classification model.

[0055] Specifically, the iterative training process in the above step S13 includes:

[0056] Step 1: In each round of iteration, the second fault feature vector of each first training sample in the training sample batch corresponding to the current round of iteration is input into the first neural network model to obtain the predicted fault type output by the first neural network model.

[0057] Step 2: Construct a first objective loss function based on the second fault type and predicted fault type of each first training sample in the training sample batch corresponding to the multiple rounds of iterations.

[0058] Step 3: Adjust the model parameters of the first neural network model based on the first objective loss function until the model converges to obtain a fault classification model.

[0059] That is to say, by predicting the predicted fault type corresponding to at least one second fault feature vector of each second monitoring entity, and constructing a first target loss function in combination with the corresponding second fault type, the gradient of the first target loss function based on the model parameters is calculated through the back propagation algorithm, and the model parameters are updated according to the calculated gradient values ​​through the gradient descent method. The above process is repeated until the model converges (the model performance no longer improves or reaches the predetermined number of iterations) to obtain the final fault classification model.

[0060] Further, for each first monitoring entity in the first business system, the above-mentioned trained fault classification model can be used to analyze at least one first fault feature vector of the first monitoring entity to obtain the first fault type of the first monitoring entity that fails in the first time period. For at least one pairing of "first fault feature vector and first fault type" of the first monitoring entity, at least one set of operation and maintenance data corresponding to a first fault feature vector and a first fault type of the first monitoring entity in the first time period is determined based on a preset operation and maintenance database (which can be constructed by expert historical experience and operation and maintenance knowledge), wherein the above-mentioned operation and maintenance data at least includes: fault handling strategy, self-healing parameters; and for each set of operation and maintenance data, the operation and maintenance data are respectively filled into a preset fault self-healing action template, and an action instruction and an execution script corresponding to the operation and maintenance data are constructed to obtain at least one optional fault self-healing action corresponding to a first fault feature vector and a first fault type of the first monitoring entity in the first time period, wherein the optional fault self-healing action at least includes: fault handling strategy, self-healing parameters, action instructions, and execution scripts.

[0061] Among them, the above-mentioned "action instructions, execution scripts" can be divided into the minimum granularity with processing independence and logical adaptability. The action instructions and execution scripts of this granularity are hereinafter referred to as "logically atomic action scripts". The action instructions and execution scripts in a single fault self-healing action are logically atomic action scripts.

[0062] In the above embodiment, for each first monitoring entity, the reason why a pairing of a "first fault feature vector and a first fault type" can correspond to at least one optional fault self-healing action is that there may be multiple possible fault handling solutions or fault handling combination solutions in different scenarios. For example, if the number of network connections of this monitoring entity reaches the maximum upper limit due to the cause of the other party's monitoring entity, the strategy for handling this type of fault may be ("self-recovery after the other party's connection is closed (strategy)", "several policy parameters", "this monitoring entity does not perform any action and waits for the other party to process (action instruction)"), or ("this monitoring entity forcibly closes several network connections (strategy)", "several policy parameters", "executes the script for closing the network connection (action instruction)"), wherein the self-healing parameter is used to associate and identify the dependent conditions or reference conditions for the execution of the self-healing action (such as: immediate execution, waiting for the completion of the previous action of some other associated monitoring entity before executing this action, targeting a specific object or range, etc.).

[0063] Furthermore, starting from the business system as a whole, to trace the root cause of the fault in each first monitoring entity in the entire first business system, it is necessary to perform correlation analysis on at least one fault feature of each first monitoring entity to obtain the corresponding fault transmission chain.

[0064] As an optional implementation, in the technical solution provided in the above step S206, the association relationship between each first monitoring entity can be determined through the knowledge graph of the first business system.

[0065] Specifically, the structure diagram of the knowledge graph of the first business system is as follows: Figure 3 As shown, A represents application-type monitoring entity, P represents PaaS-type monitoring entity, and L represents IaaS-type monitoring entity (such as network, storage, operating system, other hardware or underlying software, etc.), and there is a connection relationship between the three types of instance monitoring entities, and the connection relationship is reflected by the connection between the monitoring entities.

[0066] As an optional implementation, in the technical solution provided in the above step S206, the training process of the above fault conduction prediction model may include:

[0067] Step S21, obtaining multiple groups of second training samples.

[0068] The second training sample includes: a second fault feature vector obtained by extracting features from the second fault state data corresponding to at least two second monitoring entities in a historical time period, a connection relationship between at least two second monitoring entities, and a second fault conduction path between at least two second monitoring entities. The second connection relationship can be determined by the knowledge graph corresponding to the business system where the second monitoring entity is located.

[0069] Step S22, constructing a second neural network model.

[0070] Step S23, iteratively training the second neural network model using multiple groups of second training samples to obtain a fault conduction prediction model.

[0071] Specifically, the iterative training process in the above step S23 includes:

[0072] The first step is to input the second fault feature vectors and connection relationships of at least two second monitoring entities in each second training sample in the training sample batch corresponding to the current iteration process into the second neural network model in each iteration process to obtain the predicted fault conduction path output by the second neural network model.

[0073] In the second step, a second target loss function is constructed based on the second fault conduction paths of two second monitoring entities having a connection relationship in each second training sample in the training sample batch corresponding to the multiple rounds of iterations and the corresponding predicted fault conduction paths.

[0074] The third step is to adjust the model parameters of the second neural network model based on the second objective loss function until the model converges to obtain a fault conduction prediction model.

[0075] That is to say, by analyzing the second fault feature vectors and their connection relationships corresponding to at least two second monitoring entities in the historical time period, the corresponding fault conduction path is predicted, and the second target loss function is constructed with the corresponding second fault conduction path. The gradient of the second target loss function based on the model parameters is calculated by the back propagation algorithm, and the model parameters are updated according to the calculated gradient values ​​by the gradient descent method. The above process is repeated until the model converges (the model performance no longer improves or reaches the predetermined number of iterations) to obtain the final fault conduction prediction model.

[0076] As an optional implementation, in the technical solution provided in the above step S208, the training process of the above fault self-healing action selection model may include:

[0077] Step S31, obtaining multiple groups of third training samples.

[0078] The third training sample includes: the second fault transmission chain of the second business system in the historical time period, the second fault self-healing action of each second monitoring entity in the second business system, and the execution order of the second fault self-healing action.

[0079] Step S32, constructing a third neural network model.

[0080] Step S33: iteratively train the third neural network model using multiple groups of third training samples to obtain a fault self-healing action selection model.

[0081] Specifically, the iterative training process in the above step S33 includes:

[0082] Step 1: In each round of iteration, the second fault transmission chain of the second business system in each second training sample in the training sample batch corresponding to the current round of iteration in the historical time period is input into the third neural network model, and the predicted fault self-healing action of each second monitoring entity in the second business system and the predicted execution order of the predicted fault self-healing action output by the third neural network model are obtained;

[0083] Step 2: construct a third objective loss function based on the second fault self-healing actions of each second monitoring entity in the second business system of each third training sample in the training sample batch corresponding to the multiple iterations and the execution order of the second fault self-healing actions and the predicted fault self-healing actions of each second monitoring entity in the corresponding second business system and the predicted execution order of the predicted fault self-healing actions;

[0084] Step 3: Adjust the model parameters of the third neural network model based on the third objective loss function until the model converges to obtain the fault self-healing action selection model.

[0085] That is to say, in the above-mentioned iterative training process, by combining expert experience, operation and maintenance knowledge, and a self-healing strategy library (which may be composed of multiple historical self-healing strategies) to analyze different business systems in different scenarios, when each monitoring entity in the business system selects which fault self-healing action and in which order to perform the fault self-healing action, the system fault can be repaired efficiently and accurately to iteratively train the fault self-healing action model. In the training iteration process, the gradient of the third objective loss function based on the model parameters is calculated by the back propagation algorithm, and the model parameters are updated according to the calculated gradient value by the gradient descent method. The above process is repeated until the model converges (the model performance is no longer improved or reaches the predetermined number of iterations) to obtain the final fault self-healing action selection model.

[0086] It should be noted that the types of the first objective loss function of the above-mentioned fault classification model, the second objective loss function of the fault conduction prediction model, and the third objective loss function of the fault self-healing action selection model include but are not limited to: cross entropy objective loss function, hinge objective loss function, etc. This application does not impose any specific restrictions on the type of the first objective loss function.

[0087] When the above objective loss function is the cross entropy objective loss function, the corresponding expression can be written as:

[0088]

[0089] Where M is the number of fault types, y o,c is a binary indicator (1 if category c is the correct classification for sample o, 0 otherwise), p o,c is the probability that the model sample o belongs to category c.

[0090] When the above objective loss function is a hinge objective loss function, its corresponding expression can be written as:

[0091]

[0092] Among them, y i Represents the real category, Represents the predicted category, and N is the number of training samples. Therefore, the above hinge objective loss function maximizes the intervals of different types.

[0093] In addition, in the above steps S204-S208, the model (including the fault classification model, the fault conduction prediction model, and the fault self-healing action selection model) can also be loaded into the memory. For example, the original data of the model can be loaded from the non-volatile memory to the volatile memory so that the processor can run the model. The original data of the model refers to the unprocessed data, which usually includes the parameters and structural data of the model. The structural data can be a calculation relationship based on the parameters, such as the forward propagation calculation relationship between the intermediate layers and the neurons. Specifically, the structural data can include the code related to the structure of the model, such as the code for performing related calculations between the intermediate layers and the neurons.

[0094] In one embodiment, an area for loading a model can be divided in the memory, which may include a structure data storage area and a parameter storage area. The structure data storage area is used to store structure-related codes, and the parameters referenced by it can point to the address of a specific parameter in the parameter storage area through a pointer. During the training process of the model, it may be necessary to frequently update the parameters, and the parameter values ​​in the parameter storage area can be updated.

[0095] Furthermore, after executing step S208 to sequentially execute the fault self-healing actions of each first monitoring entity according to the fault transmission chain of the business system in the first time period, multiple log information or probe information corresponding to each first monitoring entity in the second time period can be re-acquired, wherein the second time period is any time period after each first monitoring entity completes executing the first fault self-healing action;

[0096] When it is determined whether there is corresponding monitoring entity event information in the log information or probe information of each first monitoring entity, and when there is corresponding monitoring entity event information in the log information or probe information of a preset number of first monitoring entities, it means that executing the first fault self-healing action of each first monitoring entity according to the execution order obtained in step S208 is not effective for processing the fault of the first business system or the success rate is low. At this time, the above-mentioned models can be optimized, iteratively trained and tuned again according to the actual situation.

[0097] Based on the scheme defined in the above steps S202 to S208, it can be learned that in the embodiment, at least one first fault feature vector of a single first monitoring entity is analyzed by a fault type classification model to predict the first fault type of the first monitoring entity that fails in the first time period, and to determine at least one optional fault self-healing action corresponding to the at least one first fault feature vector and the first fault type of the first monitoring entity; then, starting from the business system as a whole, the fault conduction prediction model is used to analyze the association relationship between the first monitoring entities in the first business system and at least one first fault feature vector of each first monitoring entity to predict the first fault conduction chain of the first business system that fails in the first time period; then, the first fault conduction chain is analyzed by a fault self-healing action selection model to determine the first fault self-healing action that is most suitable for the current scenario and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity; finally, the first fault self-healing actions of each first monitoring entity are executed in sequence according to the execution order, so as to efficiently and low-costly troubleshoot and repair the fault problem of the first business system, thereby solving the technical problem of low efficiency and high cost of troubleshooting and repairing faults in the business system due to manual methods.

[0098] Example 2

[0099] Based on Embodiment 1 of the present application, an embodiment of a business system fault processing device is also provided, and the business system fault processing method of the above embodiment is executed when the device is running. Figure 4 is a schematic diagram of the structure of an optional business system fault processing device according to an embodiment of the present application, such as Figure 4 As shown, the business system fault processing device at least includes an acquisition module 42, a fault type prediction module 44, a fault conduction prediction module 46 and a fault processing module 48, wherein:

[0100] An acquisition module 42 is used to acquire a plurality of first fault status data corresponding to a plurality of first monitoring entities in a first business system within a first time period, and perform feature extraction on the plurality of first fault status data of each first monitoring entity to obtain at least one corresponding first fault feature vector;

[0101] A fault type prediction module 44 is used to analyze at least one first fault feature vector of each first monitoring entity using a pre-trained fault classification model to obtain a first fault type of the first monitoring entity that fails in a first time period, and determine at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity in the first time period;

[0102] A fault conduction prediction module 46 is used to determine the association relationship between the first monitoring entities in the first business system, and analyze at least one first fault feature vector and the association relationship of each first monitoring entity using a pre-trained fault conduction prediction model to obtain a first fault conduction chain of the first business system in the first time period;

[0103] The fault processing module 48 is used to analyze the first fault conduction chain using a pre-trained fault self-healing action selection model to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and execute the first fault self-healing action of each first monitoring entity in sequence according to the execution order.

[0104] It should be noted that the various modules in the above-mentioned business system fault handling device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0105] Example 3

[0106] According to an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the business system fault handling method in Example 1.

[0107] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program: obtaining multiple first fault state data corresponding to multiple first monitoring entities in the first business system within a first time period, and extracting features from the multiple first fault state data of each first monitoring entity to obtain at least one corresponding first fault feature vector; for each first monitoring entity, using a pre-trained fault classification model to analyze at least one first fault feature vector of the first monitoring entity to obtain a first fault type of the first monitoring entity that fails within the first time period, and determining at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity within the first time period; determining an association relationship between each first monitoring entity in the first business system, and using a pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain a first fault conduction chain of the first business system within the first time period; using a pre-trained fault self-healing action selection model to analyze the first fault conduction chain to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and sequentially execute the first fault self-healing action of each first monitoring entity according to the execution order.

[0108] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the business system fault handling method in Example 1 is implemented.

[0109] Optionally, the computer program executes the following steps: obtaining a plurality of first fault status data corresponding to a plurality of first monitoring entities in the first business system within a first time period, and performing feature extraction on the plurality of first fault status data of each first monitoring entity to obtain at least one corresponding first fault feature vector; for each first monitoring entity, using a pre-trained fault classification model to analyze at least one first fault feature vector of the first monitoring entity to obtain a first fault type of the first monitoring entity that fails within the first time period, and determining at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity within the first time period; determining an association relationship between each first monitoring entity in the first business system, and using a pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain a first fault conduction chain of the first business system within the first time period; using a pre-trained fault self-healing action selection model to analyze the first fault conduction chain to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and sequentially executing the first fault self-healing action of each first monitoring entity in accordance with the execution order.

[0110] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the business system fault handling method in Example 1 is executed when the program is running.

[0111] Optionally, when the program is running, the following steps are executed: obtaining a plurality of first fault status data corresponding to a plurality of first monitoring entities in the first business system within a first time period, and extracting features from the plurality of first fault status data of each first monitoring entity to obtain at least one corresponding first fault feature vector; for each first monitoring entity, using a pre-trained fault classification model to analyze at least one first fault feature vector of the first monitoring entity to obtain a first fault type of the first monitoring entity that fails within the first time period, and determining at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity within the first time period; determining an association relationship between each first monitoring entity in the first business system, and using a pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain a first fault conduction chain of the first business system within the first time period; using a pre-trained fault self-healing action selection model to analyze the first fault conduction chain to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and sequentially executing the first fault self-healing action of each first monitoring entity in the execution order.

[0112] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 5 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 5 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the business system fault handling method in the above-mentioned embodiment 1 when running.

[0113] Optionally, the processor is configured to execute the following steps through a computer program: obtaining a plurality of first fault status data corresponding to a plurality of first monitoring entities in the first business system within a first time period, and performing feature extraction on the plurality of first fault status data of each first monitoring entity to obtain at least one corresponding first fault feature vector; for each first monitoring entity, using a pre-trained fault classification model to analyze at least one first fault feature vector of the first monitoring entity to obtain a first fault type of a fault occurring in the first time period of the first monitoring entity, and determining at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity within the first time period; determining an association relationship between each first monitoring entity in the first business system, and using a pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain a first fault conduction chain of the first business system within the first time period; using a pre-trained fault self-healing action selection model to analyze the first fault conduction chain to determine a first fault self-healing action of each first monitoring entity and an execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and sequentially executing the first fault self-healing action of each first monitoring entity in accordance with the execution order.

[0114] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0115] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a monitoring entity, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program code.

[0120] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for handling business system failures, It is characterized in that include: Acquire a plurality of first fault status data corresponding to a plurality of first monitoring entities in a first business system within a first time period, and perform feature extraction on the plurality of first fault status data of each of the first monitoring entities to obtain at least one corresponding first fault feature vector; For each of the first monitoring entities, use a pre-trained fault classification model to analyze at least one first fault feature vector of the first monitoring entity to obtain a first fault type of the first monitoring entity that fails in a first time period, and determine at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity in the first time period; Determine the association relationship between each first monitoring entity in the first business system, and use the pre-trained fault conduction prediction model to analyze at least one first fault feature vector and the association relationship of each first monitoring entity to obtain a first fault conduction chain of the first business system in a first time period; The first fault conduction chain is analyzed using a pre-trained fault self-healing action selection model to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and the first fault self-healing action of each first monitoring entity is executed sequentially according to the execution order.

2. The method according to claim 1, It is characterized in that Acquiring a plurality of first fault status data corresponding to a plurality of first monitoring entities in a business system within a first time period includes: Acquire multiple monitoring entity event information corresponding to multiple first monitoring entities in the business system within a first time period, wherein the monitoring entity event information includes state data of at least one fault event occurring in the first monitoring entity within the first time period; Determine fault status data of each of the first monitoring entities within a first time period based on monitoring entity event information of each of the first monitoring entities; Preprocessing is performed on the fault status data of each of the first monitoring entities in the first time period to obtain the first fault status data of each of the first monitoring entities in the first time period.

3. The method according to claim 1, It is characterized in that The training process of the fault classification model includes: Acquire multiple groups of first training samples, wherein the first training samples include: a second fault feature vector obtained by extracting features from second fault state data of a second monitoring entity in a historical time period and a second fault type of a fault of the second monitoring entity; Constructing the first neural network model; The first neural network model is iteratively trained using multiple groups of the first training samples to obtain the fault classification model.

4. The method according to claim 3, It is characterized in that Iteratively training the first neural network model using multiple groups of the first training samples to obtain the fault classification model includes: In each round of iteration, the second fault feature vector of each first training sample in the training sample batch corresponding to the current round of iteration is input into the first neural network model to obtain the predicted fault type output by the first neural network model; Constructing a first objective loss function based on the second fault type of each first training sample in the training sample batch corresponding to the multiple rounds of iterations and the corresponding predicted fault type; The model parameters of the first neural network model are adjusted based on the first objective loss function until the model converges to obtain the fault classification model.

5. The method according to claim 1, It is characterized in that Determining at least one optional fault self-healing action corresponding to each first fault feature vector and first fault type of the first monitoring entity within the first time period includes: Determine at least one set of operation and maintenance data corresponding to a first fault feature vector and a first fault type of the first monitoring entity within a first time period based on a preset operation and maintenance database, wherein the operation and maintenance data at least includes: a fault handling strategy and a self-healing parameter; For each group of the operation and maintenance data, the operation and maintenance data are respectively filled into a preset fault self-healing action template, and action instructions and execution scripts corresponding to the operation and maintenance data are constructed to obtain at least one optional fault self-healing action corresponding to a first fault feature vector and a first fault type of the first monitoring entity within a first time period, wherein the optional fault self-healing action includes at least: a fault handling strategy, self-healing parameters, action instructions, and an execution script.

6. The method according to claim 1, It is characterized in that The training process of the fault conduction prediction model includes: Acquire multiple groups of second training samples, wherein the second training samples include: second fault feature vectors obtained by extracting features from second fault state data corresponding to at least two second monitoring entities in a historical time period, a connection relationship between at least two second monitoring entities, and a second fault conduction path between at least two second monitoring entities; Constructing a second neural network model; The second neural network model is iteratively trained using multiple groups of the second training samples to obtain the fault conduction prediction model.

7. The method according to claim 6, It is characterized in that Iteratively training the second neural network model using multiple groups of the second training samples to obtain the fault conduction prediction model includes: In each round of iteration, the second fault feature vectors and connection relationships of at least two second monitoring entities in each second training sample in the training sample batch corresponding to the current round of iteration are input into the second neural network model to obtain the predicted fault conduction path output by the second neural network model; Constructing a second objective loss function based on the second fault conduction paths of at least two second monitoring entities of each second training sample in the training sample batch corresponding to the multiple rounds of iterations and the corresponding predicted fault conduction paths; The model parameters of the second neural network model are adjusted based on the second objective loss function until the model converges to obtain the fault conduction prediction model.

8. The method according to claim 1, It is characterized in that The training process of the fault self-healing action selection model includes: Acquire multiple groups of third training samples, wherein the third training samples include: a second fault transmission chain of the second business system in a historical time period, a second fault self-healing action of each second monitoring entity in the second business system, and an execution order of the second fault self-healing action; Constructing a third neural network model; The third neural network model is iteratively trained using multiple groups of the third training samples to obtain the fault self-healing action selection model.

9. The method according to claim 8, It is characterized in that Iteratively training the third neural network model using multiple groups of the third training samples to obtain the fault self-healing action selection model includes: In each round of iteration, the second fault conduction chain of the second business system in each second training sample in the training sample batch corresponding to the current round of iteration in the historical time period is input into the third neural network model, and the predicted fault self-healing action of each second monitoring entity in the second business system and the predicted execution order of the predicted fault self-healing action output by the third neural network model are obtained; Constructing a third objective loss function based on the second fault self-healing actions of each second monitoring entity in the second business system and the execution order of the second fault self-healing actions of each third training sample in the training sample batch corresponding to the multiple rounds of iterations and the predicted fault self-healing actions of each second monitoring entity in the corresponding second business system and the predicted execution order of the predicted fault self-healing actions; The model parameters of the third neural network model are adjusted based on the third objective loss function until the model converges to obtain the fault self-healing action selection model.

10. A business system fault processing device, It is characterized in that include: an acquisition module, configured to acquire a plurality of first fault status data corresponding to a plurality of first monitoring entities in a first business system within a first time period, and perform feature extraction on the plurality of first fault status data of each of the first monitoring entities to obtain at least one corresponding first fault feature vector; a fault type prediction module, configured to analyze, for each of the first monitoring entities, at least one first fault feature vector of the first monitoring entity using a pre-trained fault classification model, obtain a first fault type of a fault occurring in the first monitoring entity within a first time period, and determine at least one optional fault self-healing action corresponding to each first fault feature vector and the first fault type of the first monitoring entity within the first time period; a fault conduction prediction module, used to determine the association relationship between the first monitoring entities in the first business system, and analyze at least one first fault feature vector and the association relationship of each first monitoring entity using a pre-trained fault conduction prediction model to obtain a first fault conduction chain of the first business system in a first time period; A fault processing module is used to analyze the first fault conduction chain by using a pre-trained fault self-healing action selection model to determine the first fault self-healing action of each first monitoring entity and the execution order of the first fault self-healing action from at least one optional fault self-healing action of each first monitoring entity, and execute the first fault self-healing action of each first monitoring entity in sequence according to the execution order.

11. A non-volatile storage medium, It is characterized in that The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the business system fault handling method according to any one of claims 1 to 9 by running the computer program.

12. A computer program product, It is characterized in that include: A computer program, wherein when the computer program is executed by a processor, the business system fault handling method according to any one of claims 1 to 9 is implemented.

Images