Fault processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a fault processing method and device, computer equipment and a readable storage medium. The method comprises the following steps: under the condition that a preset system has a fault and the fault cannot be self-healed, performing plan searching in a pre-established plan database based on the fault; and under the condition that the target processing plan matched with the fault is found, processing the fault based on the target processing plan. By adopting the method, the fault processing time can be reduced.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a fault handling method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] During the operation of complex systems, the ability to quickly identify and handle faults directly determines the stability of the system and the continuity of business.

[0003] However, most systems currently still rely heavily on the experience of operation and maintenance engineers to analyze the source of the problem and manually select appropriate emergency plans after an alarm occurs.

[0004] This traditional processing method is not only time-consuming, but also susceptible to the limitations and subjectivity of human judgment, especially in major failure scenarios, when the time left for human intervention is very limited. If effective loss-stopping measures are not taken in time, it may lead to system paralysis, business interruption, and even serious economic losses. Summary of the invention

[0005] Based on this, it is necessary to provide a fault handling method, apparatus, computer device and readable storage medium that can reduce the time consumption of fault handling in response to the above technical problems.

[0006] In a first aspect, the present application provides a fault handling method, comprising:

[0007] In the event that a preset system fails and the failure cannot be self-healed, a plan search is performed in the pre-established plan database based on the failure;

[0008] When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0009] In one embodiment, the above-mentioned processing of the fault based on the target processing plan includes:

[0010] When the target processing plan is a full update, the fault is handled by a full one-time update;

[0011] When the target processing plan is grayscale execution, the fault is handled by using a step-by-step trial run.

[0012] In one embodiment, the above-mentioned searching for a plan in a pre-established plan database based on the fault includes:

[0013] Determine the alarm identifier corresponding to the fault based on the fault acquisition;

[0014] Search for plans in the plan database based on the alarm identifier.

[0015] In one embodiment, the method further comprises:

[0016] If no target processing plan matching the fault is found, a prompt message is output; the prompt message is used to prompt the operator to handle the fault.

[0017] In one embodiment, the method for establishing the above-mentioned plan database includes:

[0018] For each fault, establish a corresponding candidate processing plan based on the fault;

[0019] Conduct chaos drills on candidate processing plans to verify them;

[0020] If the verification is passed, the candidate processing plan will be stored in the plan database.

[0021] In one embodiment, the chaos training process includes:

[0022] Perform explosion radius analysis on the fault scenarios corresponding to the faults to obtain the scope of the faults;

[0023] Perform service dependency assessments on the fault scenarios corresponding to the faults to determine the services affected by the faults.

[0024] Based on the affected scope and affected services, a fault is injected into the test system, and the fault is processed based on the candidate processing plan to obtain a verification result indicating whether the verification is passed or not.

[0025] In one embodiment, the method further comprises:

[0026] Based on the fault, identify the personnel information involved in the fault and establish an emergency command group based on the personnel information;

[0027] If a preset system fails and the failure cannot be self-healed, the failure will be reported to the emergency command group;

[0028] After receiving the trigger operation input by the relevant personnel in the emergency command group, the step of searching for a plan in a pre-established plan database based on the fault is executed.

[0029] In a second aspect, the present application also provides a fault handling device, including:

[0030] The emergency plan search module is used to search for emergency plans in the pre-established emergency plan database based on the fault when the preset system fails and the fault cannot be self-healed;

[0031] The fault processing module is used to process the fault based on the target processing plan when a target processing plan matching the fault is found.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] In the event that a preset system fails and the failure cannot be self-healed, a plan search is performed in the pre-established plan database based on the failure;

[0034] When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0036] In the event that a preset system fails and the failure cannot be self-healed, a plan search is performed in the pre-established plan database based on the failure;

[0037] When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0038] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0039] In the event that a preset system fails and the failure cannot be self-healed, a plan search is performed in the pre-established plan database based on the failure;

[0040] When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0041] The above-mentioned fault handling method, device, computer equipment and readable storage medium, when a preset system fails and the fault cannot be self-healed, searches for a plan in a pre-established plan database based on the fault; when a target processing plan matching the fault is found, the fault is processed based on the target processing plan; in this method, when a system fails and cannot self-heal, the plan database can be used for searching, so as to quickly locate the target processing plan matching the fault, which can effectively shorten the time from fault identification to processing plan determination, avoid the process of manual step-by-step investigation and plan formulation, and improve fault handling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] FIG1 is a diagram showing the internal structure of a computer device in one embodiment;

[0044] Figure 2 A schematic diagram of a flow chart of a fault handling method in an embodiment;

[0045] Figure 3 is a flowchart of a fault handling method in another embodiment;

[0046] Figure 4 is a flowchart of a fault handling method in another embodiment;

[0047] Figure 5 is a flowchart of a fault handling method in another embodiment;

[0048] Figure 6 is a flowchart of a fault handling method in another embodiment;

[0049] Figure 7 FIG. 4 is a structural block diagram of a fault handling device in an embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG1. ​​The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the fault handling process. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a fault handling method is implemented.

[0052] Those skilled in the art will understand that the structure shown in FIG. 1 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0053] In an exemplary embodiment, Figure 2 As shown, a fault handling method is provided, which is described by taking the method applied to the computer device in FIG. 1 as an example, and includes the following steps 201 to 202. Among them:

[0054] Step 201 , when a preset system fails and the failure cannot be self-healed, a plan search is performed in a pre-established plan database based on the failure.

[0055] Among them, the preset system refers to a system that runs in a specific scenario, which can be industrial equipment, information management system, energy management system, etc.

[0056] A failure is an abnormal state or functional failure caused by internal or external reasons during system operation, which may manifest as hardware damage, software errors, network interruption or other unexpected abnormal phenomena.

[0057] Self-healing refers to the ability of a system to resolve partial failures by itself through built-in automatic recovery mechanisms. Self-healing usually relies on technologies such as redundant design, automatic restart function or parameter correction, and does not require human intervention.

[0058] The plan database is a pre-built repository for storing plans for handling different types of faults. Each plan usually includes a description of the fault characteristics, handling steps, required resources, and execution logic.

[0059] Each plan usually includes the following:

[0060] (1) Fault description: A detailed description of the specific fault phenomenon and impact scope.

[0061] (2) Processing steps: The specific operational process for troubleshooting, including the required tools, resources, and time estimates.

[0062] (3) Applicable conditions: Define the scenarios and limitations to which the plan applies, such as only applicable to specific hardware or software versions.

[0063] (4) Expected results: Describe the goals that should be achieved after implementing the plan, such as restoring normal functions or reducing the risk of failure.

[0064] In the embodiment of the present application, when the computer device detects that a system failure has occurred, the fault self-healing mechanism is first started. During this process, the computer device will quickly analyze the characteristics and impact range of the failure, and take measures according to the preset self-healing rules, such as reloading configuration files, adjusting system operating parameters, or triggering backup modules. If the self-healing mechanism is successful, the system will automatically recover, and the cause, time, and self-healing process of the failure will be recorded through the general event backtracking function for subsequent analysis and optimization.

[0065] If the self-healing mechanism fails to resolve the fault, the computer device will enter the contingency plan search phase. At this point, the computer device first collects detailed information related to the fault, including the fault code, device type, time and location of the fault, current system status, and related environmental parameters. After processing, these data will extract the core information describing the fault characteristics as the basis for subsequent searches. Next, the computer device converts the fault characteristics into standardized query conditions, including fault type, priority, impact range, and possible triggering causes.

[0066] Step 202: When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0067] Among them, plan search is the process of searching the plan database for the target processing plan that matches the fault based on the fault feature information. The search method may include keyword-based search or feature matching algorithm.

[0068] The target processing plan is the solution that is matched in the plan database, is closest to the current fault characteristics and can effectively solve the problem. The target processing plan usually contains specific operation steps, responsibility allocation and execution conditions.

[0069] Fault handling is the process of repairing or mitigating detected faults based on targeted handling plans. The handling methods may include adjusting system parameters, replacing hardware components, notifying maintenance personnel, or activating backup systems.

[0070] In an embodiment of the present application, the computer device searches a pre-established plan database based on the generated query conditions. The plan database contains various types of faults that the system may face and their corresponding processing solutions. During the search process, the computer device uses keyword matching and logical rule filtering methods to filter out candidate processing plans that match the current fault from the database. At the same time, the computer device will sort the candidate processing plans according to the adaptability and priority to ensure that the most appropriate target processing plan is selected.

[0071] After finding the target processing plan, the computer equipment will confirm its applicability. The confirmation process includes evaluating the resource requirements of the plan, the feasibility of the execution steps, and the possibility of solving the fault. If the target plan is confirmed, the computer equipment will start specific fault handling operations according to the instructions in the plan. For example, switching to backup equipment, adjusting system operating parameters, restarting some functional modules, or notifying manual intervention. During this process, the computer equipment will monitor the execution effect of each operation step in real time to ensure that the problem is effectively solved.

[0072] In some embodiments, after the fault handling is completed, the computer device will enter the evaluation phase to recheck the fault status. If the fault has been completely resolved, the system will resume normal operation and archive the records of the entire processing process, including fault information, the process of finding the plan, the execution steps and the processing results. These records will serve as an important reference for subsequent optimization plans. If the fault cannot be completely resolved, the device will try to call other candidate processing plans, or directly prompt manual intervention to avoid further expansion of the problem.

[0073] In the above-mentioned fault handling method, when a preset system fails and the fault cannot be self-healed, a plan search is performed in a pre-established plan database based on the fault; when a target processing plan matching the fault is found, the fault is handled based on the target processing plan; in this method, when a system fails and cannot self-heal, the plan database can be used for searching, so as to quickly locate the target processing plan matching the fault, which can effectively shorten the time from fault identification to processing plan determination, avoid the process of manual step-by-step investigation and plan formulation, and improve fault handling efficiency.

[0074] In an exemplary embodiment, the above-mentioned “processing the fault based on the target processing plan” includes:

[0075] In case 1, when the target processing plan is a full update, the fault is handled by using a full one-time update.

[0076] A full update is a one-time repair, replacement, or reset of all affected modules, components, or data in the system. A full update is usually applicable to the following scenarios:

[0077] (1) The fault has a wide impact range, and the consistency of all related modules or data must be ensured.

[0078] (2) The root cause of the failure is clear and can be solved through unified update measures.

[0079] (3) The system operation continuity requirements are low, allowing short-term comprehensive shutdown or resource restart.

[0080] When performing a full update, the computer device will adjust all affected modules at the same time according to the steps in the target processing plan, such as replacing all faulty modules at once or redeploying the entire system configuration. The advantages of a full update are high efficiency and thorough processing, but it usually requires a large amount of system resources and may affect the overall operation of the system.

[0081] In an embodiment of the present application, when the target processing plan is a full update, the computer device handles the fault in a one-time full update manner. First, the computer device loads the corresponding full update package or configuration file according to the update requirements provided in the plan database, and verifies its integrity and applicability. After confirming that the update content is correct, the computer device will back up the current data and configuration in advance so that it can be quickly rolled back in case of an accident during the update process. Subsequently, the computer device performs an update operation on all relevant modules, components or data at one time. After the update is completed, the computer device will restart and load the new version of the configuration or function module, and perform a comprehensive self-test and operation verification to ensure that the system state is stable and the fault is completely resolved.

[0082] In the second case, when the target processing plan is grayscale execution, the fault is handled by using a step-by-step trial run.

[0083] Grayscale execution refers to handling failures in a step-by-step trial run, that is, handling them in stages, regions, and modules to reduce risks and verify the handling effect. The main features of grayscale execution are gradualness and controllability, and are applicable to the following situations:

[0084] (1) The scope of the fault’s impact is limited, but there may be a risk of a chain reaction.

[0085] (2) The root cause of the failure has not yet been fully identified, and the effectiveness of the treatment plan needs to be verified through step-by-step testing.

[0086] (3) The system must maintain high availability and cannot be completely shut down.

[0087] During the grayscale execution process, computer equipment usually selects a portion of the affected modules or areas, and first test-runs the adjustment measures according to the requirements of the target processing plan to verify their effectiveness in resolving the fault. After confirming that the trial run is successful, the processing scope is gradually expanded until the fault is completely resolved. The advantage of grayscale execution is that it can effectively reduce processing risks and ensure the stable operation of the system during the adjustment period, but it may take longer than a full update.

[0088] Gradual trial operation means gradually verifying the effect of the treatment plan through small-scale experimental operations to avoid the uncertain risks that may be brought about by direct large-scale operations. During the trial operation stage, the computer equipment will dynamically monitor the operation effect after adjustment, evaluate the degree of improvement of the fault and the impact on system performance. If the trial operation fails, the system can be quickly rolled back to the original state to avoid expanding the scope of the problem.

[0089] In an embodiment of the present application, when the target processing plan is grayscale execution, a step-by-step trial operation is used to handle the fault. First, the computer device selects a part of the pilot modules or areas as the initial test range according to the grayscale execution strategy defined in the plan, and loads the corresponding update content. Subsequently, the computer device adjusts the modules within the pilot range or optimizes the operating parameters, and monitors the adjusted operating status in real time, including the effect of fault resolution, changes in system performance indicators, and whether new problems arise. If the trial operation effect meets expectations, the computer device will gradually expand the update scope, and promote the adjustment measures to more modules or areas in stages, and monitoring and verification are required before each expansion of the scope. After completing the adjustment of all modules, the computer device performs a global operating status assessment and a fault thoroughness check to confirm that the processing process is not omitted.

[0090] In the above embodiment, full update can quickly resolve low-risk faults and ensure business continuity; grayscale execution verifies the processing effect in a risk-controlled manner in complex scenarios. The combination of these two strategies can not only improve the efficiency and accuracy of fault handling, but also take into account the security, stability and user experience of the system.

[0091] In an exemplary embodiment, Figure 3 As shown, the above-mentioned “searching for a plan in a pre-established plan database based on a fault” includes:

[0092] Step 301: Determine an alarm identifier corresponding to a fault based on fault acquisition.

[0093] The alarm identifier is a descriptive code or label for a fault or abnormal state, which is used to quickly locate and identify specific problems. The alarm identifier usually contains the following information:

[0094] (1) Fault type: such as hardware failure, communication failure or software abnormality.

[0095] (2) Fault Level: Indicates the severity of the problem, such as general, important, or urgent.

[0096] (3) Trigger conditions: Describes the specific events or parameter thresholds that cause the alarm, such as excessive temperature, unstable voltage, etc.

[0097] (4) Location information: Indicates the location or device where the fault occurred, such as server number, module name, or circuit board number.

[0098] In the embodiment of the present application, the computer device acquires relevant data from the fault site, including the device status, operation log, and parameter information that triggers the fault. Then, the collected data is parsed and classified, the type and characteristics of the fault are determined through the built-in fault identification algorithm, and an alarm mark related to the fault is generated.

[0099] Step 302: search for a plan in the plan database based on the alarm identifier.

[0100] Among them, plan search refers to the process of matching the appropriate processing plan in the plan database through the alarm identification. The purpose of plan search is to quickly find a solution for the fault, shorten the system downtime, and improve the accuracy and efficiency of fault handling.

[0101] In an embodiment of the present application, after an alarm identifier is generated, the computer device searches for a plan in the plan database based on the identifier. First, the computer device converts the alarm identifier into standardized search conditions, such as extracting key fields in the alarm identifier (such as fault type, equipment number, level, etc.). Subsequently, the computer device performs a search operation in the plan database based on these search conditions. The plan database stores processing plans corresponding to various faults. The computer device uses a matching algorithm to filter out the plans that are most relevant to the current fault, and prioritizes them according to the severity of the fault and the scope of impact. The found plans need to be further verified by the system to ensure that they are applicable to the current fault environment.

[0102] After completing the search and confirming the applicable plan, the computer equipment will pass the selected plan to the processing module or operation and maintenance personnel as an operational guide for troubleshooting.

[0103] In the above embodiment, by quickly matching the alarm identifier corresponding to the fault with the emergency plan in the database, the time for fault diagnosis and processing plan selection can be significantly shortened.

[0104] In an exemplary embodiment, the method further includes:

[0105] If no target processing plan matching the fault is found, a prompt message is output; the prompt message is used to prompt the operator to handle the fault.

[0106] In an embodiment of the present application, if a target processing plan matching the fault is not found, the computer device will start a backup program for fault processing and output a prompt message to inform the operator to handle the fault. First, the computer device will determine whether an adapted processing solution can be found based on the previous plan search process. If the computer device fails to retrieve a plan matching the current fault from the plan database, it means that there is no predefined solution for the current fault, or the fault type is not covered by the plan library.

[0107] At this point, the computer device will generate a prompt message to the operator, explaining that the fault does not match the preset processing plan and recommends manual processing. The prompt message usually includes the following: First, the computer device will briefly describe the nature and triggering conditions of the fault to help the operator quickly understand the background of the fault; then, the computer device will point out the reason why the relevant plan cannot be found, which may be due to the particularity of the fault or the plan library has not been updated; finally, the computer device will prompt the operator to take manual intervention measures and provide possible preliminary processing suggestions or warnings based on the type and severity of the fault and the current system status.

[0108] In addition to basic fault information and handling suggestions, the prompt information may also include contact information or further support options. For example, the computer equipment can provide contact information for the technical support team, or guide the operator to access troubleshooting documents or online help to ensure that the operator can obtain the necessary resources to effectively handle the fault. In this way, the computer equipment can effectively make up for the shortcomings of the plan database and ensure that the operator can still intervene in a timely and accurate manner when the fault cannot be handled automatically, ensuring that the system resumes normal operation as soon as possible.

[0109] In the above embodiment, the output prompt information provides backup measures for the case where the fault scenario is not covered, so as to prevent the system from falling into an unresponsive state when there is no plan matching. The intervention and subsequent processing of the operator can reduce the risk of system operation and enhance the reliability of the system in the face of complex faults.

[0110] In an exemplary embodiment, Figure 4 As shown, the method for establishing the above-mentioned emergency plan database includes:

[0111] Step 401: for each fault, a corresponding candidate processing plan is established based on the fault.

[0112] Among them, candidate processing plans refer to a collection of different solutions prepared in advance to handle the fault when a fault occurs. These candidate processing plans are designed based on the different types of faults, the scope of impact, and the emergency processing requirements. Candidate processing plans are usually alternative plans before the final solution is determined, which are used for rapid response and preliminary fault repair. Each candidate processing plan usually contains information such as the steps to solve the fault, the required resources, and the execution conditions.

[0113] In the embodiment of the present application, for each fault, it is first necessary to establish a corresponding candidate processing plan based on the nature, cause and impact scope of the fault.

[0114] Step 402, performing chaos drills on the candidate processing plans to verify the candidate processing plans.

[0115] Step 403: If the verification is successful, the candidate processing plan is stored in the plan database.

[0116] Among them, chaos drills are a test method that simulates fault handling. By deliberately introducing various faults or anomalies into the test system, the test system's ability to cope with faults is verified. In chaos drills, candidate handling plans are simulated in actual or similar environments to observe their effectiveness and efficiency. The purpose of the drill is to verify the effectiveness of the plan under real conditions and identify potential loopholes or defects, thereby providing a more reliable basis for the actual handling of faults.

[0117] Verification processing refers to verifying the feasibility and effectiveness of candidate processing plans through testing and experiments. Verification processing usually includes a comprehensive evaluation of the results of the candidate processing plan execution to ensure that it can effectively repair the fault, restore system functions, and will not produce other side effects. Verification passing means that the candidate processing plan can successfully handle the fault in the simulated fault scenario and meet the system recovery goals.

[0118] In an embodiment of the present application, a chaos drill is performed on the candidate processing plan to verify the candidate processing plan. During the drill, different types of faults or abnormal conditions are deliberately introduced to verify whether the candidate processing plan can successfully restore the normal operation of the system. After the chaos drill successfully verifies the effectiveness of the candidate processing plan, if the candidate processing plan can achieve the expected fault handling effect and does not cause other potential problems, the computer device will store the plan in the plan database. The storage process means that the candidate processing plan has passed the verification and has been formally included in the plan database.

[0119] In the above embodiment, when similar failures occur in the future, it is possible to respond quickly and effectively, reduce system downtime, and improve the overall stability and reliability of the system. In addition, chaos drills can not only detect possible loopholes or inapplicability of the plan in actual operation, but also evaluate the execution efficiency and stability of the plan. In this way, the feasibility and reliability of the candidate processing plan can be fully verified, thereby ensuring that the plan can be effectively executed and the problem can be solved when a real failure occurs.

[0120] In an exemplary embodiment, Figure 5 As shown in the figure, the above chaos drill process includes:

[0121] Step 501 , performing explosion radius analysis on the fault scenarios corresponding to the fault to obtain the affected range of the fault.

[0122] Among them, the fault scenario is a descriptive definition of a specific fault that may occur in the system, including the conditions under which the fault occurs, the scope of impact, and the potential consequences. Each fault scenario is usually designed based on the system architecture, operating process, and historical fault data, aiming to simulate various possible abnormal scenarios in the system. The construction of the fault scenario provides a clear research object for subsequent fault impact assessment and processing.

[0123] The explosion radius is used to describe the scope of a fault, indicating the maximum physical or logical scope of a fault. In information systems, the explosion radius can refer to the size of the area of ​​hardware devices, network nodes, or data flows that may be affected by the fault. In complex systems, the analysis of the explosion radius helps to assess the severity and potential impact of the fault, and provides a scientific basis for fault handling.

[0124] The scope of impact refers to the area directly or indirectly affected by a certain fault, including devices, network connections, or service modules that may be affected by the fault. The size of the scope of impact is usually determined by analyzing the structural dependencies of the system, data transmission paths, etc., in order to accurately assess the propagation trend and influence of the fault.

[0125] In the embodiment of the present application, the explosion radius analysis is performed on the fault scenarios corresponding to the fault to obtain the scope of the fault. The core of the explosion radius analysis is to determine the scope of the fault, including directly affected modules, devices, network nodes, and related parts that may be indirectly affected.

[0126] Step 502: Perform service dependency assessment on the fault scenarios corresponding to the fault to obtain the affected services of the fault.

[0127] Among them, service dependency assessment is the process of analyzing and evaluating the dependency relationships between services in the system, aiming to determine the potential impact of a certain failure on other related services. In a complex distributed system, there may be direct calls and data interactions between services. Service dependency assessment determines the key business or functional modules that will be affected by the failure by building a dependency graph or running an impact chain analysis.

[0128] Affected services refer to specific services or service functions that are affected by the failure. Based on the service dependency assessment, the identification of affected services helps to clarify the impact of the failure on the end-user experience or the core functions of the system. For critical services or high-priority services, their impact may cause greater risks, so their recovery strategies need to be given priority.

[0129] In the embodiment of the present application, after the affected scope is determined, a service dependency assessment is performed for the fault scenario to obtain the affected services of the fault. The goal of the service dependency assessment is to determine the potential impact of the fault on related services in the system.

[0130] Step 503: Based on the affected scope and affected services, a fault is injected into the test system, and the fault is processed based on the candidate processing plan to obtain a verification result indicating whether the verification is passed.

[0131] The test system is an environment used to simulate and verify system behavior, which usually contains some or all functions of the actual system to ensure that the verification results have reference value. The test system can be a sandbox version of the real system or an independent environment dedicated to testing. The goal of the test system is to provide a safe and controllable experimental platform for fault simulation, processing plan verification and performance evaluation.

[0132] The verification result is an evaluation of the execution effect obtained by applying the candidate processing plan to the simulated fault scenario in the test system. The verification result usually includes information such as whether the fault is successfully resolved, potential problems during execution, and the impact on system performance. By analyzing the verification results, it can be determined whether the candidate processing plan can be included in the plan database.

[0133] In an embodiment of the present application, a fault is injected into the test system based on the affected scope and the affected business, and the fault is processed based on the candidate processing plan to obtain a verification result of whether the verification is passed. By injecting real or near-real fault events, the performance of the test system in a faulty state and its actual impact on business operations can be observed. Finally, based on the test operation of the injected fault, the effectiveness of the candidate processing plan is verified to obtain a verification result. The evaluation of the verification result not only includes whether the candidate processing plan successfully solves the fault, but also needs to pay attention to whether it causes new problems during the execution process, whether the impact on system performance is within an acceptable range, and whether it can meet the needs of the actual operating environment. If the verification passes, the candidate processing plan will be improved and stored in the plan database for use when the actual fault occurs; if it fails, it is necessary to further optimize the processing strategy and re-verify until the ideal result is obtained.

[0134] In the above embodiment, by analyzing the fault in detail through explosion radius analysis and service dependency assessment, and combining the process of verifying the handling plan with fault injection, it is not only possible to clearly identify the impact scope and service dependency of the fault, but also possible to verify the effectiveness and adaptability of the fault handling plan. While enhancing the system's fault recovery capability, this process helps to improve the pertinence and accuracy of the plan, as well as the system's adaptability and fault tolerance, thereby ensuring that the system can efficiently cope with complex fault scenarios.

[0135] In an exemplary embodiment, Figure 6 As shown, the above method also includes:

[0136] Step 601, based on the fault, identify the personnel information involved in the fault, and establish an emergency command group based on the personnel information.

[0137] Among them, fault delineation refers to accurately locating the fault location, scope involved, and affected modules by analyzing fault logs, system monitoring data, equipment operating status, etc. after a fault occurs.

[0138] Personnel information refers to the identity, contact information, and division of responsibilities of operators, technical support teams, decision makers, etc. who are related to troubleshooting. This information usually includes the role of the personnel (such as network administrator, database engineer, system architect, etc.), the system module they are responsible for, and the availability status in emergency situations.

[0139] The emergency command group is a temporary team formed for a specific fault or emergency event. Its members are composed of relevant personnel involved in the fault. The main tasks of this group are to communicate efficiently, assign tasks, execute plans and monitor the progress of processing to ensure that the problem can be solved in a timely manner. The emergency command group can communicate and command through instant messaging tools, telephone conferences or dedicated emergency command systems.

[0140] In an embodiment of the present application, when a computer device detects a fault, the computer device will quickly analyze and delineate the fault. Therefore, it is necessary to clarify the scope of the fault, the affected modules, and the relevant personnel who may need to be involved in the processing. By combining information such as computer device logs, device status, and service dependencies, the computer device can identify the personnel within the scope of responsibility affected and screen out the most relevant technical experts, operation and maintenance personnel, or management personnel.

[0141] Then, based on the identified personnel information, the computer equipment automatically establishes an emergency command group. The members of this group usually include the personnel directly responsible for the affected modules, key business leaders, and necessary decision-making support personnel. The emergency command group is created through instant messaging tools or professional emergency command computer equipment to ensure that all members can receive information in real time and participate in discussions.

[0142] Step 602: When a preset system fails and the failure cannot be self-healed, the failure is reported to the emergency command group.

[0143] Among them, fault notification means sending detailed information of the fault (such as fault description, scope of impact, preliminary analysis results, etc.) to the emergency command group after the fault is confirmed.

[0144] In the embodiment of the present application, when it is determined that the fault cannot be solved by the self-healing function, the computer device will report the fault information to the emergency command group. The reported information usually includes a basic description of the fault, the scope of impact, the preliminary analysis results, and the degree of urgency.

[0145] Subsequently, members of the emergency command group conduct preliminary discussions and judgments based on the information reported, and the relevant person in charge enters a trigger operation instruction after confirming the severity of the fault. The trigger operation can be in the form of an approval button in the computer device or a direct input instruction.

[0146] Step 603, after receiving the trigger operation input by the relevant personnel in the emergency command group, execute the step of searching for a plan in a pre-established plan database based on the fault.

[0147] The trigger operation is an operation instruction made by relevant personnel in the emergency command group based on the fault information, which is used to start the next processing step, such as calling the plan database for fault processing. The trigger operation can be in the form of clicking a button, sending an instruction, approving a plan, etc.

[0148] In the embodiment of the present application, after receiving the trigger operation, the computer device will call the plan database to search for plans according to the fault type and impact range. The computer device will filter the target processing plan that best matches the fault from the database in an efficient manner, and provide solutions or guidance steps based on the plan. The emergency command group coordinates the team to perform specific repair measures or optimization operations based on the processing plan provided by the computer device until the fault is properly handled.

[0149] In the above embodiment, based on the strategy of identifying the personnel involved in the fault and establishing an emergency command group, when a fault occurs, rapid information transmission and efficient collaboration can be used to ensure smooth and timely fault handling. Through the search and execution of emergency plans, system downtime can be reduced, business continuity can be ensured, and experience can be accumulated for future emergency responses, continuously optimizing emergency management capabilities.

[0150] According to some embodiments of the present application, a fault handling method is provided, which is described by taking the method applied to a computer device as an example, and may include the following steps:

[0151] Step 1: For each fault, establish a corresponding candidate processing plan based on the fault;

[0152] Step 2: Perform explosion radius analysis on the fault scenarios corresponding to the fault to obtain the affected range of the fault.

[0153] Step 3: Perform service dependency assessment on the fault scenarios corresponding to the fault to obtain the affected services.

[0154] Step 4: Based on the affected scope and affected services, a fault is injected into the test system, and the fault is processed based on the candidate processing plan to obtain a verification result indicating whether the verification is passed or not.

[0155] Step 5: If the verification is successful, the candidate processing plan is stored in the plan database.

[0156] Step 6: Based on the fault, identify the personnel information involved in the fault and establish an emergency command group based on the personnel information.

[0157] Step 7: When a preset system fails and the failure cannot be self-healed, the failure is reported to the emergency command group.

[0158] Step 8: After receiving the trigger operation input by relevant personnel in the emergency command group, the computer device determines the alarm identifier corresponding to the fault based on the fault acquisition.

[0159] Step 9: Search for a plan in the plan database based on the alarm identifier.

[0160] Step 10: When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0161] In some embodiments, when the target processing plan is a full update, the fault is processed by a full one-time update;

[0162] In some embodiments, when the target processing plan is grayscale execution, the fault is processed by a step-by-step trial run.

[0163] Step 11: if no target processing plan matching the fault is found, a prompt message is output; the prompt message is used to prompt the operator to process the fault.

[0164] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0165] Based on the same inventive concept, the embodiment of the present application also provides a fault handling device for implementing the fault handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more fault handling device embodiments provided below can refer to the limitations on the fault handling method above, and will not be repeated here.

[0166] In an exemplary embodiment, Figure 7 As shown, a fault handling device is provided, including: a plan search module 701 and a fault handling module 702, wherein:

[0167] The plan search module 701 is used to search for a plan in a pre-established plan database based on the fault when a preset system fails and the fault cannot be self-healed;

[0168] The fault processing module 702 is used to process the fault based on the target processing plan when a target processing plan matching the fault is found.

[0169] In an exemplary embodiment, the above-mentioned fault handling module 702 is specifically used to handle the fault by using a full one-time update method when the target processing plan is a full update; when the target processing plan is a grayscale execution, the fault is handled by using a step-by-step trial run method.

[0170] In an exemplary embodiment, the above-mentioned plan search module 701 is specifically used to obtain an alarm identifier corresponding to a fault based on the fault; and search for a plan in a plan database based on the alarm identifier.

[0171] In an exemplary embodiment, the above device further includes:

[0172] The prompt module 703 is used to output prompt information when no target processing plan matching the fault is found; the prompt information is used to prompt the operator to process the fault.

[0173] In an exemplary embodiment, the above device further includes:

[0174] Establishing module 704, for establishing corresponding candidate processing plans based on each fault;

[0175] Verification module 705, used to perform chaos drills on candidate processing plans to verify the candidate processing plans;

[0176] The storage module is used to store the candidate processing plan in the plan database when the verification is passed.

[0177] In an exemplary embodiment, the above-mentioned verification module 705 is specifically used to perform explosion radius analysis on the fault scenarios corresponding to the fault to obtain the scope of the fault; perform service dependency evaluation on the fault scenarios corresponding to the fault to obtain the affected business of the fault; based on the affected scope and the affected business, inject the fault into the test system, and handle the fault based on the candidate processing plan to obtain the verification result of whether the verification is passed.

[0178] In an exemplary embodiment, the above device further includes:

[0179] A delineation module 706 is used to delineate personnel information involved in the fault based on the fault, and establish an emergency command group based on the personnel information;

[0180] Notification module 707, used to notify the emergency command group of the failure when the preset system fails and the failure cannot be self-healed;

[0181] The trigger module 708 is used to execute the step of searching for a plan in a pre-established plan database based on the fault after receiving a trigger operation input by relevant personnel in the emergency command group.

[0182] Each module in the above fault handling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0183] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0184] In the event that a preset system fails and the failure cannot be self-healed, a plan search is performed in the pre-established plan database based on the failure;

[0185] When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0186] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0187] When the target processing plan is a full update, the fault is handled by a full one-time update;

[0188] When the target processing plan is grayscale execution, the fault is handled by using a step-by-step trial run.

[0189] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0190] Determine the alarm identifier corresponding to the fault based on the fault acquisition;

[0191] Search for plans in the plan database based on the alarm identifier.

[0192] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0193] If no target processing plan matching the fault is found, a prompt message is output; the prompt message is used to prompt the operator to handle the fault.

[0194] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0195] For each fault, establish a corresponding candidate processing plan based on the fault;

[0196] Conduct chaos drills on candidate processing plans to verify them;

[0197] If the verification is passed, the candidate processing plan will be stored in the plan database.

[0198] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0199] Perform explosion radius analysis on the fault scenarios corresponding to the faults to obtain the scope of the faults;

[0200] Perform service dependency assessments on the fault scenarios corresponding to the faults to determine the services affected by the faults.

[0201] Based on the affected scope and affected services, a fault is injected into the test system, and the fault is processed based on the candidate processing plan to obtain a verification result indicating whether the verification is passed or not.

[0202] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0203] Based on the fault, identify the personnel information involved in the fault and establish an emergency command group based on the personnel information;

[0204] If a preset system fails and the failure cannot be self-healed, the failure will be reported to the emergency command group;

[0205] After receiving the trigger operation input by the relevant personnel in the emergency command group, the step of searching for a plan in a pre-established plan database based on the fault is executed.

[0206] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0207] In the event that a preset system fails and the failure cannot be self-healed, a plan search is performed in the pre-established plan database based on the failure;

[0208] When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0210] When the target processing plan is a full update, the fault is handled by a full one-time update;

[0211] When the target processing plan is grayscale execution, the fault is handled by using a step-by-step trial run.

[0212] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0213] Determine the alarm identifier corresponding to the fault based on the fault acquisition;

[0214] Search for plans in the plan database based on the alarm identifier.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0216] If no target processing plan matching the fault is found, a prompt message is output; the prompt message is used to prompt the operator to handle the fault.

[0217] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0218] For each fault, establish a corresponding candidate processing plan based on the fault;

[0219] Conduct chaos drills on candidate processing plans to verify them;

[0220] If the verification is passed, the candidate processing plan will be stored in the plan database.

[0221] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0222] Perform explosion radius analysis on the fault scenarios corresponding to the faults to obtain the scope of the faults;

[0223] Perform service dependency assessments on the fault scenarios corresponding to the faults to determine the services affected by the faults.

[0224] Based on the affected scope and affected services, a fault is injected into the test system, and the fault is processed based on the candidate processing plan to obtain a verification result indicating whether the verification is passed or not.

[0225] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0226] Based on the fault, identify the personnel information involved in the fault and establish an emergency command group based on the personnel information;

[0227] If a preset system fails and the failure cannot be self-healed, the failure will be reported to the emergency command group;

[0228] After receiving the trigger operation input by the relevant personnel in the emergency command group, the step of searching for a plan in a pre-established plan database based on the fault is executed.

[0229] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0230] In the event that a preset system fails and the failure cannot be self-healed, a plan search is performed in the pre-established plan database based on the failure;

[0231] When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

[0232] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0233] When the target processing plan is a full update, the fault is handled by a full one-time update;

[0234] When the target processing plan is grayscale execution, the fault is handled by using a step-by-step trial run.

[0235] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0236] Determine the alarm identifier corresponding to the fault based on the fault acquisition;

[0237] Search for plans in the plan database based on the alarm identifier.

[0238] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0239] If no target processing plan matching the fault is found, a prompt message is output; the prompt message is used to prompt the operator to handle the fault.

[0240] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0241] For each fault, establish a corresponding candidate processing plan based on the fault;

[0242] Conduct chaos drills on candidate processing plans to verify them;

[0243] If the verification is passed, the candidate processing plan will be stored in the plan database.

[0244] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0245] Perform explosion radius analysis on the fault scenarios corresponding to the faults to obtain the scope of the faults;

[0246] Perform service dependency assessments on the fault scenarios corresponding to the faults to determine the services affected by the faults.

[0247] Based on the affected scope and affected services, a fault is injected into the test system, and the fault is processed based on the candidate processing plan to obtain a verification result indicating whether the verification is passed or not.

[0248] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0249] Based on the fault, identify the personnel information involved in the fault and establish an emergency command group based on the personnel information;

[0250] If a preset system fails and the failure cannot be self-healed, the failure will be reported to the emergency command group;

[0251] After receiving the trigger operation input by the relevant personnel in the emergency command group, the step of searching for a plan in a pre-established plan database based on the fault is executed.

[0252] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0253] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0254] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0255] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A fault handling method, characterized in that: The method comprises: In the event that a fault occurs in a preset system and the fault cannot be self-healed, searching for a plan in a pre-established plan database based on the fault; When a target processing plan matching the fault is found, the fault is processed based on the target processing plan.

2. The method according to claim 1, characterized in that The processing of the fault based on the target processing plan includes: When the target processing plan is a full update, the fault is processed by a full one-time update method; When the target processing plan is grayscale execution, the fault is processed by a step-by-step trial operation.

3. The method according to claim 1, characterized in that The searching for a plan in a pre-established plan database based on the fault includes: Determine an alarm indicator corresponding to the fault based on the fault acquisition; A plan search is performed in the plan database based on the alarm identifier.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In the case where no target processing plan matching the fault is found, a prompt message is output; the prompt message is used to prompt an operator to process the fault.

5. The method according to any one of claims 1 to 3, characterized in that: The method for establishing the emergency plan database comprises: For each fault, establishing a corresponding candidate processing plan based on the fault; Performing a chaos exercise on the candidate processing plan to verify the candidate processing plan; If the verification is successful, the candidate processing plan is stored in the plan database.

6. The method according to claim 5, characterized in that The chaos exercise process includes: Perform explosion radius analysis on the fault scenarios corresponding to the faults to obtain the impact range of the faults; Perform service dependency assessment on the fault scenarios corresponding to the faults to obtain the services affected by the faults; Based on the affected scope and the affected services, the fault is injected into the test system, and the fault is processed based on the candidate processing plan to obtain a verification result indicating whether the verification is passed.

7. The method according to claim 1, characterized in that The method further comprises: Based on the fault, identify the personnel information involved in the fault, and establish an emergency command group based on the personnel information; When the fault occurs in the preset system and the fault cannot be self-healed, the fault is reported to the emergency command group; After receiving the trigger operation input by the relevant personnel in the emergency command group, the step of searching for a plan in a pre-established plan database based on the fault is performed.

8. A fault handling device, characterized in that: The device comprises: A plan search module is used to search for a plan in a pre-established plan database based on a fault in a preset system and the fault cannot be self-healed; The fault processing module is used to process the fault based on the target processing plan when a target processing plan matching the fault is found.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images