Fault drilling method, device, equipment, medium and program product

Abstract

The invention provides a fault drilling method, device and equipment, a medium and a program product, which can be applied to the technical field of computers or the field of financial science and technology. The fault drilling method comprises the steps that a fault drilling scheme and drilling node identifiers are acquired, the fault drilling scheme comprises a plurality of fault drilling tasks, and the fault drilling tasks comprise execution sequence identifiers; issuing the fault drilling scheme to an operation node corresponding to the drilling node identifier; the task process sub-module allocates the fault drilling task to a plurality of processing threads based on the execution sequence identifier to obtain a plurality of to-be-executed tasks, wherein the to-be-executed tasks comprise task states; marking the states of the to-be-executed tasks as ready states and storing the plurality of to-be-executed tasks into a task pool; under the condition that the to-be-executed task in the ready state exists in the task pool, the task execution sub-module executes the to-be-executed task in the ready state to obtain an execution result; and monitoring operation state information of the operation nodes to obtain a fault drilling report.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology or financial technology, and more specifically to a fault drill method, device, equipment, medium and program product. Background Art

[0002] With the development of technology and the expansion of business, the components of the operation and maintenance system are increasing, and the configuration business is becoming more and more complicated. In addition, for complex business architectures, the business transaction chain is longer, and the requirements for transaction reliability and idempotence will be higher. The operation and maintenance scenarios are complex and diverse. In order to cope with complex operation and maintenance scenarios, it is necessary to conduct fault drills on the system regularly to analyze the weak links of the operation and maintenance system.

[0003] In related technologies, when conducting fault drills, the task engine needs to take into account both task execution and task scheduling. However, when the business transaction chain is long, the task dependencies are more complex. Relying on a single task engine to conduct fault drills will result in a longer drill time and lower efficiency. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a fault drill method, apparatus, device, medium and program product.

[0005] According to a first aspect of the present disclosure, a fault drill method is provided, which is applied to a server, wherein the server includes multiple systems, the systems include multiple running nodes, the server includes a task module, the task module includes a task flow submodule, a task execution submodule, a task pool and multiple processing threads, and the method includes: obtaining a fault drill plan and a drill node identifier, wherein the fault drill plan includes multiple fault drill tasks, and the fault drill tasks include an execution order identifier; issuing the fault drill plan to a running node corresponding to the drill node identifier; the task flow submodule assigns the fault drill tasks to multiple processing threads based on the execution order identifier to obtain multiple tasks to be executed, and the tasks to be executed include task states; the task flow submodule marks the states of the tasks to be executed as ready states and stores the multiple tasks to be executed in the task pool; when there are tasks to be executed in a ready state in the task pool, the task execution submodule executes the tasks to be executed in the ready state to obtain execution results; and generating a fault drill report based on the execution results and the operation monitoring data of the running nodes.

[0006] According to an embodiment of the present disclosure, the above-mentioned task module also includes an instance pool, the above-mentioned multiple processing threads are stored in the above-mentioned instance pool, and the above-mentioned instance pool also stores the task execution status of the multiple processing threads. The above-mentioned method also includes: the above-mentioned task flow sub-module updates the task execution status of the above-mentioned processing thread based on the above-mentioned execution result; when the task execution status of the above-mentioned processing thread is execution completed, the above-mentioned task flow sub-module returns to execute and assigns the above-mentioned fault drill task to multiple processing threads based on the above-mentioned execution order identifier to obtain multiple tasks to be executed.

[0007] According to an embodiment of the present disclosure, before the above-mentioned acquisition of the fault drill plan and the drill node identifier, it also includes: in response to the received fault plan configuration instruction, acquiring the fault identifier and preset configuration information in the above-mentioned fault plan configuration instruction, the above-mentioned preset configuration information includes fault execution process data, fault attribute data and drill node identifier; configuring the fault script based on the above-mentioned fault execution process data and the above-mentioned fault attribute data to obtain the fault drill plan, wherein the above-mentioned fault script is obtained by searching in the fault library based on the above-mentioned fault identifier.

[0008] According to an embodiment of the present disclosure, the above-mentioned server also stores preset parameter values. Before the above-mentioned task module executes the above-mentioned fault drill plan on the above-mentioned drill system running node, it also includes: verifying the parameter value of the above-mentioned fault script based on the above-mentioned preset parameter value; and verifying the destination path of the above-mentioned fault script based on the above-mentioned drill node identifier.

[0009] According to an embodiment of the present disclosure, the above-mentioned fault attribute data includes a fault whitelist, and the above-mentioned method also includes: for the above-mentioned running node, determining whether the above-mentioned running node is included in the above-mentioned fault whitelist; if the above-mentioned fault whitelist does not include the above-mentioned running node, stopping the execution of the above-mentioned fault drill plan.

[0010] According to an embodiment of the present disclosure, the above-mentioned fault attribute data also includes a fault blacklist, and the above-mentioned method also includes: for the above-mentioned running node, determining whether the above-mentioned fault blacklist includes the above-mentioned running node; in the case that the above-mentioned fault blacklist includes the above-mentioned drill system running node, stopping the execution of the above-mentioned fault drill plan.

[0011] The second aspect of the present disclosure provides a fault drill device, the device is installed on a server, the server includes multiple systems, the system includes multiple running nodes, the server includes a task module, the task module includes a task flow submodule, a task execution submodule, a task pool and multiple processing threads, including: an acquisition module, used to acquire a fault drill plan and a drill node identifier, the fault drill plan includes multiple fault drill tasks, and the fault drill task includes an execution order identifier; a delivery module, used to deliver the fault drill plan to the running node corresponding to the drill node identifier; an allocation module, used to The task flow submodule allocates the above-mentioned fault drill task to multiple processing threads based on the above-mentioned execution order identifier to obtain multiple tasks to be executed, and the above-mentioned tasks to be executed include task status; a marking module is used for the above-mentioned task flow submodule to mark the status of the above-mentioned tasks to be executed as ready status and store the above-mentioned multiple tasks to be executed into the above-mentioned task pool; an execution module is used for, when there are tasks to be executed in ready status in the above-mentioned task pool, the above-mentioned task execution submodule executes the above-mentioned tasks to be executed in ready status to obtain execution results; a generation module is used to generate a fault drill report based on the above-mentioned execution results and the operation monitoring data of the above-mentioned operation node.

[0012] According to an embodiment of the present disclosure, the above-mentioned task module also includes an instance pool, the above-mentioned multiple processing threads are stored in the above-mentioned instance pool, and the above-mentioned instance pool also stores the task execution status of the multiple processing threads. The above-mentioned device also includes: an update module, which is used for the above-mentioned task flow sub-module to update the task execution status of the above-mentioned processing thread based on the above-mentioned execution result; a return execution module, which is used for the above-mentioned task flow sub-module to return to execution when the task execution status of the above-mentioned processing thread is execution completion, and assign the above-mentioned fault drill task to multiple processing threads based on the above-mentioned execution order identifier to obtain multiple tasks to be executed.

[0013] According to an embodiment of the present disclosure, the above-mentioned device also includes: a response acquisition module, which is used to respond to the received fault solution configuration instruction and obtain the fault identification and preset configuration information in the above-mentioned fault solution configuration instruction, and the above-mentioned preset configuration information includes fault execution process data, fault attribute data and drill node identification; a solution configuration module, which is used to configure the fault script based on the above-mentioned fault execution process data and the above-mentioned fault attribute data to obtain a fault drill solution, wherein the above-mentioned fault script is obtained by searching in the fault library based on the above-mentioned fault identification.

[0014] According to an embodiment of the present disclosure, the above-mentioned server also stores preset parameter values, and the above-mentioned device also includes: a parameter value verification module, used to verify the parameter value of the above-mentioned fault script based on the above-mentioned preset parameter value; a destination path verification module, used to verify the destination path of the above-mentioned fault script based on the above-mentioned drill node identifier.

[0015] According to an embodiment of the present disclosure, the above-mentioned fault attribute data includes a fault whitelist, and the above-mentioned device also includes: a whitelist determination module, which is used to determine whether the above-mentioned running node is included in the above-mentioned fault whitelist for the above-mentioned running node; a first stop execution module, which is used to stop executing the above-mentioned fault drill plan when the above-mentioned fault whitelist does not include the above-mentioned running node.

[0016] According to an embodiment of the present disclosure, the above-mentioned fault attribute data also includes a fault blacklist, and the above-mentioned device also includes: a blacklist determination module, which is used to determine whether the above-mentioned fault blacklist includes the above-mentioned running node for the above-mentioned running node; a second stop execution module, which is used to stop executing the above-mentioned fault drill plan when the above-mentioned fault blacklist includes the above-mentioned running node of the drill system.

[0017] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0018] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0019] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0020] According to the embodiments of the present disclosure, the fault drill plan and the drill node identifier are obtained to send the fault drill plan to the running node, the task flow submodule allocates the execution thread of the task, and the task state is modified to the ready state and stored in the task pool, the task execution submodule executes the execution task in the ready state to obtain the execution result, and finally a fault drill report is generated according to the execution result and the operation monitoring data of the running node, the task flow process and the execution process are decoupled, and the efficiency of the fault drill is improved. At the same time, the task flow is carried out through the task pool, which solves the blocking problem of non-asynchronous execution tasks during the task execution process. Further, the fault drill report is generated through the operation monitoring data and the execution result, so as to evaluate the effect of the fault drill and find potential risks in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 The application scenario diagram of the fault drill method and device according to the embodiment of the present disclosure is schematically shown;

[0023] Figure 2 A flowchart of a fault drill method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 3 A schematic diagram of a task module according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 4 The following schematically shows a task execution diagram according to an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of a fault drill platform according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 6 The structure block diagram of the fault drill device according to the embodiment of the present disclosure is schematically shown;

[0028] Figure 7 A block diagram of an electronic device suitable for implementing a fault drill method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0033] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0034] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0036] At present, as the business scope of large-scale operation and maintenance systems expands, their applications and system components are becoming more and more numerous and complex. For the systems being operated and maintained, the business scenarios they face are becoming more and more complex. In order to cope with the evolution of complex system architectures, the system needs to have good stability. Chaos engineering can be used to conduct fault drills on distributed systems, and help the system find weak points by injecting faults, thereby improving the stability of the system. However, the fault drill methods based on chaos engineering in related technologies often use a task engine to take into account task execution and task scheduling. In the case of a long business transaction link, the task dependency is more complex. Relying on a single task engine for fault drills, the drill time is long and the efficiency is poor.

[0037] An embodiment of the present disclosure provides a fault drill method, which is applied to a server, wherein the server includes multiple systems, the systems include multiple running nodes, the server includes a task module, the task module includes a task flow submodule, a task execution submodule, a task pool and multiple processing threads, and the method includes: obtaining a fault drill plan and a drill node identifier, the fault drill plan includes multiple fault drill tasks, and the fault drill tasks include an execution order identifier; issuing the fault drill plan to the running node corresponding to the drill node identifier; the task flow submodule assigns the fault drill task to multiple processing threads based on the execution order identifier to obtain multiple tasks to be executed, and the tasks to be executed include task status; the task flow submodule marks the status of the tasks to be executed as a ready state and stores the multiple tasks to be executed in a task pool; when there are tasks to be executed in a ready state in the task pool, the task execution submodule executes the tasks to be executed in the ready state to obtain execution results; and generating a fault drill report based on the execution results and the operation monitoring data of the running node.

[0038] Figure 1 The application scenario diagram of the fault drill method and device according to the embodiments of the present disclosure is schematically shown.

[0039] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0042] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] It should be noted that the fault drill method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the fault drill device provided in the embodiment of the present disclosure can generally be set in the server 105. The fault drill method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the fault drill device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0045] The following will be based on Figure 1 The scene described by Figure 2~Figure 5 The fault drill method of the embodiment of the present disclosure is described in detail.

[0046] Figure 2 The flowchart of the fault drill method according to the embodiment of the present disclosure is schematically shown.

[0047] like Figure 2 As shown, the fault drill method of this embodiment includes operations S210 to S260.

[0048] According to an embodiment of the present disclosure, the above method is applied to a server, the above server includes multiple systems, the above system includes multiple running nodes, the above server includes a task module, the above task module includes a task flow sub-module, a task execution sub-module, a task pool and multiple processing threads.

[0049] In operation S210, a fault drill plan and a drill node identifier are obtained.

[0050] The fault drill plan includes multiple fault drill tasks, and the fault drill tasks include execution sequence identifiers.

[0051] According to the embodiments of the present disclosure, the above-mentioned fault drill plan may include data such as the purpose of the fault drill, the scope of the drill system, the drill window period, the drill announcement, the drill notification personnel, monitoring data, and the node high availability plan. The drill window period is the specific time when the system conducts a fault drill, and the node high availability plan is that when the target node is drilled, in order to ensure the normal operation of the system, the role of the target node in the system can be replaced by configuring redundant nodes and other methods.

[0052] According to the embodiments of the present disclosure, the above fault drill solution can be executed in each system by calling the automation of each system, and the execution of a specified script can be performed through the task scheduling of automation.

[0053] In operation S220, the fault drill plan is sent to the running node corresponding to the drill node identifier.

[0054] According to an embodiment of the present disclosure, before the fault drill plan is issued, the fault package of the atomic fault constituting the aforementioned fault drill plan may be issued to the running node, and the corresponding atomic fault is called when the running node executes the fault drill plan.

[0055] According to the embodiments of the present disclosure, the running nodes can be divided into multiple types of nodes according to the different functions of the running nodes in the fault drill scheme, for example, they can be divided into fault protection nodes, fault injection nodes, fault handling nodes, fault recovery nodes, manual nodes, delayed waiting nodes, etc. The above-mentioned fault protection nodes can be nodes that are set with protection to ensure the normal operation of the system during the fault drill, the fault injection nodes are nodes into which the fault is injected during the fault drill, the fault recovery nodes are nodes that clear the fault and revoke the impact on the target through operations in the fault drill scheme or node preset operations after the fault is injected, the manual nodes are nodes that can manually handle the fault during the fault drill, and the delayed waiting nodes are nodes that wait during the fault drill.

[0056] In operation S230 , the task flow submodule allocates the fault drill task to multiple processing threads based on the execution order identifier to obtain multiple tasks to be executed.

[0057] The tasks to be executed include task status.

[0058] According to the embodiment of the present disclosure, fault drill tasks with execution dependencies in the fault drill tasks will be executed through the task flow sub-module according to the execution order identifier, while for fault drill tasks without dependencies, the task flow sub-module will allocate threads and place the executable tasks in the task pool.

[0059] In operation S240 , the task flow submodule marks the status of the to-be-executed tasks as a ready status and stores the plurality of to-be-executed tasks in a task pool.

[0060] In operation S250 , when there are tasks to be executed in a ready state in the task pool, the task execution submodule executes the tasks to be executed in the ready state to obtain an execution result.

[0061] According to an embodiment of the present disclosure, the status of tasks to be executed stored in the task pool includes ready, executing, call failure, execution failure, successful execution, etc. The task execution submodule includes a task execution unit and a task query unit. The task query unit is used to monitor the status of tasks in the task pool. When there are tasks to be executed in the ready state in the task pool, the task execution unit will execute the tasks to be executed in the ready state.

[0062] In operation S260, a fault drill report is generated based on the execution result and the operation monitoring data of the operation node.

[0063] According to the embodiments of the present disclosure, the operating status of the running node during a fault drill can be monitored according to the pre-configured fault indicator observation configuration, the historical records of the fault drill tasks can be viewed, the drill situation and the weak links in the system can be analyzed, and the drill report can be exported.

[0064] For example, when the fault drill task is a host processor failure, a script may be executed to occupy a large number of host processors to simulate the operation of the target system when the host processor is over-occupied.

[0065] According to the embodiments of the present disclosure, the fault drill plan and the drill node identifier are obtained to send the fault drill plan to the running node, the task flow submodule allocates the execution thread of the task, and the task state is modified to the ready state and stored in the task pool, the task execution submodule executes the execution task in the ready state to obtain the execution result, and finally a fault drill report is generated according to the execution result and the operation monitoring data of the running node, the task flow process and the execution process are decoupled, and the efficiency of the fault drill is improved. At the same time, the task flow is carried out through the task pool, which solves the blocking problem of non-asynchronous execution tasks during the task execution process. Further, the fault drill report is generated through the operation monitoring data and the execution result, so as to evaluate the effect of the fault drill and find potential risks in the system.

[0066] According to the embodiment of the present disclosure, by subdividing the task module into a task flow submodule and a task execution submodule, the task flow submodule will not be involved in the subsequent modification or addition of the task execution logic, thereby reducing the complexity of the code and improving the work efficiency of relevant staff.

[0067] According to an embodiment of the present disclosure, the above-mentioned task module also includes an instance pool, multiple processing threads are stored in the instance pool, and the instance pool also stores the task execution status of the multiple processing threads. The method also includes: the task flow sub-module updates the task execution status of the processing thread based on the execution result; when the task execution status of the processing thread is completed, the task flow sub-module returns to execute and assigns the fault drill task to multiple processing threads based on the execution order identifier to obtain multiple tasks to be executed.

[0068] Figure 3 A schematic diagram of a task module according to an embodiment of the present disclosure is schematically shown.

[0069] like Figure 3 As shown, the task module includes a task flow submodule, a task execution submodule, an instance pool and a task pool, wherein the instance pool includes ready instances (threads), executing instances, task status and task dependencies, the task pool includes task status (ready, executing, call failure, execution failure, successful execution, etc.), the task flow submodule monitors the instance status, task status and task status in the instance pool; the task execution submodule monitors the task status in the task pool, the task query unit in the task execution submodule is used to query the task status in the task pool, and the task execution unit is used to execute the task.

[0070] According to an embodiment of the present disclosure, the above instance pool and task pool may be stored in a key-value database.

[0071] Figure 4 The following schematic diagram shows a task execution diagram according to an embodiment of the present disclosure.

[0072] like Figure 4 As shown, after receiving the execution instruction, the task flow submodule will assign tasks to the instances (threads) in the instance pool, and mark the assigned tasks as ready and store them in the task pool. The task execution unit executes the tasks in the task pool that are in the ready state, and asynchronously queries the task status in the task pool. After the task execution is completed, the task flow submodule updates the task status in the task pool, the instance status in the instance pool, and the task status, and performs a new round of thread allocation.

[0073] According to the embodiments of the present disclosure, the task execution status of the processing thread in the instance pool is updated through the task flow submodule after the task is executed, and the fault drill task is reallocated to the processing thread. The execution and scheduling of the task are respectively performed through two different submodules. The task processing status and the thread processing status are recorded through the task pool and the instance pool. This can improve the computing reversal efficiency and reduce the delay and waiting time in the task processing process. Furthermore, by decomposing the fault drill plan into multiple fault drill tasks, the computing resources of each thread can be fully utilized to improve the overall resource utilization.

[0074] According to an embodiment of the present disclosure, the above-mentioned server also stores preset parameter values, and before using the task module to execute the fault drill plan on the running node of the drill system, it also includes: verifying the parameter value of the fault script based on the preset parameter value; and verifying the destination path of the fault script based on the drill node identifier.

[0075] According to an embodiment of the present disclosure, before executing the fault drill plan, a green light verification can also be performed, that is, verifying whether the parameter values, names and destination paths in the script are correct based on the preset parameter values ​​stored in the server. This is a verification and confirmation process before the fault drill plan is executed.

[0076] According to the embodiment of the present disclosure, before executing the fault drill plan, multiple people are required to review the execution process, and at least two people are required to enter the correct password to execute the fault drill plan. In addition, permission control is performed at the system function menu level, and the viewing and editing functions are separated to prevent accidental touches.

[0077] According to the embodiments of the present disclosure, before executing the fault drill plan, the parameter values ​​in the script are verified using the stored preset parameter values, and the destination path in the script is verified using the drill node identifier. This allows error information in the fault drill plan to be discovered and corrected immediately, thereby avoiding invalid fault drills due to parameter errors and improving the accuracy and reliability of fault drills.

[0078] According to an embodiment of the present disclosure, before obtaining the fault drill plan and the drill node identifier, it also includes: in response to the received fault plan configuration instruction, obtaining the fault identifier and preset configuration information in the fault plan configuration instruction, the preset configuration information includes fault execution process data, fault attribute data and drill node identifier; configuring the fault script based on the fault execution process data and the fault attribute data to obtain the fault drill plan, wherein the fault script is obtained by searching in the fault library based on the fault identifier.

[0079] According to an embodiment of the present disclosure, the above-mentioned fault library includes multiple atomic faults (fault scripts). Atomic faults are the smallest fault units in the fault library. Atomic faults come from open source fault libraries and custom fault scripts. The fault scripts in the open source fault library and the faults in the custom fault scripts are entered into the fault library after testing and verification, which are called atomic faults. The atomic faults are sent down to the designated running node through an agent for running. The fault library may include atomic faults such as host faults, storage faults, network faults, application faults, container faults, and middleware faults.

[0080] According to the embodiment of the present disclosure, the entry of atomic faults into the fault library needs to be reviewed, and the atomic faults and related operations that pass the review can be configured and used in the fault drill plan. Users can also customize faults and upload them to the fault library.

[0081] According to the embodiments of the present disclosure, the fault execution process data includes data such as the execution order of each atomic fault, whether the atomic faults are run serially or in parallel, whether a delay is required during the execution process, and the execution trigger conditions of the atomic faults. The operation type of the node can also be configured according to the operation of the running node, including fault injection nodes, manual nodes, fault recovery nodes, fault handling nodes, fault protection nodes, etc. The configured fault drill plan can be reused, and users can choose to use the process template configuration or perform a customized configuration. It also supports the configuration of fault media packages, that is, the types of atomic faults sent to the running nodes.

[0082] According to an embodiment of the present disclosure, before responding to a received fault plan configuration instruction and obtaining the fault identification and preset configuration information in the fault plan configuration instruction, the configuration data of each system, the deployment unit in the system, and the host in the deployment unit can be obtained through the configuration management center platform. After obtaining the system configuration information, the parameter configuration in the atomic fault can be changed according to the system configuration information, and the atomic fault after the configuration change can be used to form a fault drill plan.

[0083] According to the embodiment of the present disclosure, fault attribute data includes explosion radius, high availability model, basic observation indicators of fault, basic information of drills, fault duration, fault task execution frequency, fault recovery time, fault severity, etc., wherein the basic information of drills includes drill date, drill system, responsible personnel, drill level, drill announcement, etc., and the basic observation indicators of faults include CPU utilization rate, memory utilization rate, etc.

[0084] According to the embodiment of the present disclosure, the high availability model configuration can be, for example, the unified entry and management of high availability model, resource model, and deployment model basic data. The high availability models, high availability objects, and high availability scenarios in each field are grouped to provide a field high availability view and an application high availability view.

[0085] According to the embodiments of the present disclosure, during a fault drill, the execution process of each running node can be viewed, and corresponding exception handling methods can be configured for execution failures. This supports global pause / resume, manual termination, execution log viewing, real-time display of execution progress and execution status, and viewing of the actual execution status of tasks.

[0086] Figure 5 A schematic diagram of a fault drill platform according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 5As shown, the fault drill platform includes a configuration module and a task module. The fault drill platform and the event intelligent handling platform, the application monitoring platform, and the automation platform operate in collaboration. The configuration module includes a fault injection configuration submodule for configuring data such as fault injection nodes; a drill protection configuration submodule for configuring a fault whitelist and a fault blacklist as well as target protection objects; a fault handling configuration submodule for configuring relevant data for automatic handling after a fault occurs; a high availability model configuration submodule for configuring alternative nodes or alternative solutions for running node failures during the fault drill process, and a fault indicator configuration submodule for configuring monitoring data for running nodes; and a drill plan configuration submodule for configuring a fault drill plan based on fault execution process data.

[0088] According to an embodiment of the present disclosure, the fault attribute data includes a fault whitelist, and the method further includes: for a running node, determining whether the fault whitelist includes the running node; if the fault whitelist does not include the running node, stopping execution of the fault drill plan.

[0089] According to an embodiment of the present disclosure, the fault attribute data also includes a fault blacklist, and the method also includes: for the running node, determining whether the fault blacklist includes the running node; if the fault blacklist includes the running node of the drill system, stopping the execution of the fault drill plan.

[0090] According to the embodiments of the present disclosure, the impact scope of the fault drill can be limited by configuring the fault whitelist and blacklist, and the target protection object can be isolated through script configuration, distribution, and execution, that is, the injected system information and IP information are limited. The two methods are combined to control the explosion radius of the fault drill.

[0091] According to the embodiments of the present disclosure, when the operation monitoring data of the operation node exceeds a preset threshold, an alarm may be triggered to notify relevant staff to handle the problem, or an automatic fault handling operation may be triggered to handle the problem.

[0092] According to the embodiments of the present disclosure, the impact range of the fault drill is configured by configuring a fault whitelist and a fault blacklist, and the fault drill plan is stopped after the impact range is exceeded. This can effectively control the explosion radius of the fault, exclude sensitive nodes to prevent damage to sensitive nodes, and improve the reliability of the system. Through the combined use of the fault whitelist and the blacklist, more refined control can be achieved to ensure that fault drills are only performed within the specified range.

[0093] Based on the above fault drill method, the present disclosure also provides a fault drill device. Figure 6 The device is described in detail.

[0094] Figure 6The structural block diagram of the fault drill device according to an embodiment of the present disclosure is schematically shown.

[0095] like Figure 6 As shown, the fault drill device 600 of this embodiment includes an acquisition module 610 , a delivery module 620 , an allocation module 630 , a marking module 640 , an execution module 650 and a generation module 660 .

[0096] The acquisition module 610 is used to acquire a fault drill scheme and a drill node identifier, the fault drill scheme includes multiple fault drill tasks, and the fault drill task includes an execution order identifier. In one embodiment, the acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0097] The sending module 620 is used to send the fault drill solution to the running node corresponding to the drill node identifier. In one embodiment, the sending module 620 can be used to perform the operation S220 described above, which will not be described in detail here.

[0098] The allocation module 630 is used in the task flow submodule to allocate the fault drill task to multiple processing threads based on the execution order identifier to obtain multiple tasks to be executed, and the tasks to be executed include task states. In one embodiment, the allocation module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0099] The marking module 640 is used to mark the state of the to-be-executed task as the ready state and store the multiple to-be-executed tasks in the task pool. In one embodiment, the marking module 640 can be used to perform the operation S240 described above, which will not be described in detail here.

[0100] The execution module 650 is used for executing the ready-to-be-executed tasks in the task pool, and the task execution submodule executes the ready-to-be-executed tasks to obtain execution results. In one embodiment, the execution module 650 can be used to execute the operation S250 described above, which will not be repeated here.

[0101] The generation module 660 is used to generate a fault drill report based on the execution result and the operation monitoring data of the operation node. In one embodiment, the generation module 660 can be used to perform the operation S260 described above, which will not be described in detail here.

[0102] According to an embodiment of the present disclosure, the above-mentioned task module also includes an instance pool, the above-mentioned multiple processing threads are stored in the above-mentioned instance pool, and the above-mentioned instance pool also stores the task execution status of the multiple processing threads. The above-mentioned device also includes: an update module, which is used for the above-mentioned task flow sub-module to update the task execution status of the above-mentioned processing thread based on the above-mentioned execution result; a return execution module, which is used for the above-mentioned task flow sub-module to return to execution when the task execution status of the above-mentioned processing thread is execution completion, and assign the above-mentioned fault drill task to multiple processing threads based on the above-mentioned execution order identifier to obtain multiple tasks to be executed.

[0103] According to an embodiment of the present disclosure, the above-mentioned device also includes: a response acquisition module, which is used to respond to the received fault solution configuration instruction and obtain the fault identification and preset configuration information in the above-mentioned fault solution configuration instruction, and the above-mentioned preset configuration information includes fault execution process data, fault attribute data and drill node identification; a solution configuration module, which is used to configure the fault script based on the above-mentioned fault execution process data and the above-mentioned fault attribute data to obtain a fault drill solution, wherein the above-mentioned fault script is obtained by searching in the fault library based on the above-mentioned fault identification.

[0104] According to an embodiment of the present disclosure, the above-mentioned server also stores preset parameter values, and the above-mentioned device also includes: a parameter value verification module, used to verify the parameter value of the above-mentioned fault script based on the above-mentioned preset parameter value; a destination path verification module, used to verify the destination path of the above-mentioned fault script based on the above-mentioned drill node identifier.

[0105] According to an embodiment of the present disclosure, the above-mentioned fault attribute data includes a fault whitelist, and the above-mentioned device also includes: a whitelist determination module, which is used to determine whether the above-mentioned running node is included in the above-mentioned fault whitelist for the above-mentioned running node; a first stop execution module, which is used to stop executing the above-mentioned fault drill plan when the above-mentioned fault whitelist does not include the above-mentioned running node.

[0106] According to an embodiment of the present disclosure, the above-mentioned fault attribute data also includes a fault blacklist, and the above-mentioned device also includes: a blacklist determination module, which is used to determine whether the above-mentioned fault blacklist includes the above-mentioned running node for the above-mentioned running node; a second stop execution module, which is used to stop executing the above-mentioned fault drill plan when the above-mentioned fault blacklist includes the above-mentioned running node of the drill system.

[0107] According to an embodiment of the present disclosure, any multiple modules among the acquisition module 610, the delivery module 620, the allocation module 630, the marking module 640, the execution module 650 and the generation module 660 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the delivery module 620, the allocation module 630, the marking module 640, the execution module 650 and the generation module 660 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the acquisition module 610, the delivery module 620, the allocation module 630, the marking module 640, the execution module 650 and the generation module 660 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0108] Figure 7 A block diagram of an electronic device suitable for implementing a fault drill method according to an embodiment of the present disclosure is schematically shown.

[0109] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0110] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0111] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0112] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0113] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0114] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the fault drill method provided by the embodiment of the present disclosure.

[0115] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0116] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0117] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0118] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0119] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0121] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A fault drill method, characterized in that: The method is applied to a server, the server includes multiple systems, the system includes multiple running nodes, the server includes a task module, the task module includes a task flow submodule, a task execution submodule, a task pool and multiple processing threads, and the method includes: Obtaining a fault drill plan and a drill node identifier, wherein the fault drill plan includes multiple fault drill tasks, and the fault drill tasks include an execution order identifier; Sending the fault drill plan to the running node corresponding to the drill node identifier; The task flow submodule allocates the fault drill task to multiple processing threads based on the execution order identifier to obtain multiple tasks to be executed, and the tasks to be executed include task status; The task flow submodule marks the status of the to-be-executed tasks as a ready status and stores the multiple to-be-executed tasks into the task pool; In the case that there are tasks to be executed in a ready state in the task pool, the task execution submodule executes the tasks to be executed in the ready state to obtain an execution result; A fault drill report is generated based on the execution result and the operation monitoring data of the operation node.

2. The method according to claim 1, characterized in that The task module further includes an instance pool, the multiple processing threads are stored in the instance pool, and the instance pool also stores the task execution status of the multiple processing threads. The method further includes: The task flow submodule updates the task execution status of the processing thread based on the execution result; When the task execution status of the processing thread is completed, the task flow submodule returns to execute and allocates the fault drill task to multiple processing threads based on the execution sequence identifier to obtain multiple tasks to be executed.

3. The method according to claim 1, characterized in that Before obtaining the fault drill plan and the drill node identifier, the method further includes: In response to the received fault solution configuration instruction, obtaining a fault identifier and preset configuration information in the fault solution configuration instruction, wherein the preset configuration information includes fault execution process data, fault attribute data and a drill node identifier; A fault script is configured based on the fault execution process data and the fault attribute data to obtain a fault drill solution, wherein the fault script is obtained by searching in a fault library based on the fault identifier.

4. The method according to claim 1, characterized in that: The server also stores preset parameter values, and before the task module is used to execute the fault drill solution on the exercise system running node, the method further includes: Verifying the parameter value of the fault script based on the preset parameter value; and The destination path of the fault script is verified based on the drill node identifier.

5. The method according to claim 3, characterized in that: The fault attribute data includes a fault whitelist, and the method further includes: For the running node, determining whether the fault whitelist includes the running node; When the fault whitelist does not include the running node, the execution of the fault drill plan is stopped.

6. The method according to claim 5, characterized in that The fault attribute data also includes a fault blacklist, and the method further includes: For the running node, determining whether the fault blacklist includes the running node; When the fault blacklist includes the exercise system running node, the execution of the fault exercise plan is stopped.

7. A fault drill device, characterized in that: The device is installed on a server, the server includes multiple systems, the system includes multiple running nodes, the server includes a task module, the task module includes a task flow submodule, a task execution submodule, a task pool and multiple processing threads, and the device includes: An acquisition module, used to acquire a fault drill plan and a drill node identifier, wherein the fault drill plan includes a plurality of fault drill tasks, and the fault drill tasks include an execution order identifier; A sending module, used for sending the fault drill plan to the running node corresponding to the drill node identifier; An allocation module, configured for the task flow submodule to allocate the fault drill task to multiple processing threads based on the execution order identifier to obtain multiple tasks to be executed, wherein the tasks to be executed include task states; A marking module, used for marking the status of the tasks to be executed as a ready state and storing the multiple tasks to be executed in the task pool; An execution module, configured to, when there are tasks to be executed in a ready state in the task pool, cause the task execution submodule to execute the tasks to be executed in the ready state to obtain execution results; A generation module is used to generate a fault drill report based on the execution result and the operation monitoring data of the operation node.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images