Emergency plan detection method and device

Abstract

The invention discloses an emergency plan detection method and device. The method comprises the following steps: determining a simulation object; when the simulation object is a first physical network card and has a fault, sending a ping command to a data node to which the first physical network card belongs; when the confirmation information is not received within the preset time, repairing and updating the first physical network card by using the emergency plan of the network card high-availability model; when the simulation objects are the first physical network card and the second physical network card and have faults, switching the main fragment and the standby fragment; and if the switching of the main and standby fragments fails within the preset time, switching and updating the main and standby fragments by using an emergency plan of the high-availability model of the distributed database. When the fault is actively injected to enable the simulation object to enter a fault state, the emergency plan is updated and optimized in time by detecting the performance of the emergency plan in the situation. The process is helpful for improving the recognition capability of the emergency plan for unknown fault scenes and the emergency disposal effect, thereby improving the detection efficiency of the emergency plan.

Description

Technical Field

[0001] The present application relates to the field of data detection technology, and in particular to a detection method and device for an emergency plan. Background Art

[0002] In order to meet regulatory requirements on business continuity of information systems, commercial banks will regularly test the emergency plans of the information systems under their jurisdiction in accordance with regulatory requirements, and continuously optimize and improve the emergency plans based on the test results to ensure the effectiveness of the application plans in failure scenarios.

[0003] At present, the existing emergency plans are generally manually checked and reviewed by operation and maintenance personnel, and the content and process are checked one by one. During the inspection process, personnel can find loopholes, deficiencies or areas that need to be updated in the emergency plan, record the fault scenarios and corresponding technical treatment measures found, and then add them to the original emergency plan. However, since the number of fault scenarios in the emergency plan that has been iterated for many years is usually very large, manual verification of each one is bound to consume a lot of time, resulting in low detection efficiency of the emergency plan.

[0004] Therefore, how to improve the detection efficiency of emergency plans has become an urgent problem to be solved in this field. Summary of the invention

[0005] The present application provides a method and device for detecting an emergency plan, the purpose of which is to improve the detection efficiency of the emergency plan.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A method for detecting an emergency plan, comprising:

[0008] Determine the simulation object;

[0009] When the simulation object is a first physical network card and a failure of the first physical network card is detected, a ping command is sent to a data node to which the first physical network card belongs;

[0010] When the confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is not received within the preset time, it is determined that the high availability model of the network card is invalid;

[0011] Repairing the first physical network card using the emergency plan of the network card high availability model to obtain a repair result;

[0012] Updating the emergency plan of the network card high availability model according to the repair result;

[0013] When the simulation objects are the first physical network card and the second physical network card, and it is detected that both the first physical network card and the second physical network card fail, switching the primary and backup shards corresponding to the first physical network card and the second physical network card;

[0014] If the switch between the primary and backup shards fails within the preset time, the emergency plan of the distributed database high availability model is used to switch the primary and backup shards to obtain the switching result;

[0015] The emergency plan of the distributed database high availability model is updated according to the switching result.

[0016] Optionally, updating the emergency plan of the network card high availability model according to the repair result includes:

[0017] When the repair result indicates that the repair time of the first physical network card exceeds a preset time, the emergency plan of the high availability model of the network card is tested to obtain the first fault cause;

[0018] The emergency plan of the network card high availability model is updated according to the first fault cause.

[0019] Optionally, also include:

[0020] When the repair result indicates that the repair of the first physical network card is complete, obtaining current network card status information of the first physical network card;

[0021] Obtaining normal network card status information of the first physical network card;

[0022] Comparing the current network card status information with the normal network card status information;

[0023] If the current network card status information is consistent with the normal network card status information, it is determined that the first physical network card is repaired successfully.

[0024] Optionally, updating the emergency plan of the distributed database high availability model according to the switching result includes:

[0025] If the switching result indicates that the primary and standby shard switching fails, the emergency plan of the distributed database high availability model is tested to obtain the second fault cause;

[0026] The emergency plan of the distributed database high availability model is updated according to the second failure cause.

[0027] Optionally, also include:

[0028] If the primary and backup shards are successfully switched within the preset time, then detecting whether the number of primary and backup shards meets the preset requirements; wherein the preset requirements include the number of primary shards and the number of backup shards;

[0029] If the number of the primary and backup shards does not meet the preset requirements, the emergency plan of the distributed database high availability model is used to reduce the number of the primary shards and the number of the backup shards.

[0030] Optionally, after determining the simulation object, the method further includes:

[0031] Performing fault handling on the simulation object using the ifdown command;

[0032] Sending an ifconfig command to the simulated object;

[0033] When the DOWN state fed back by the simulation object according to the ifconfig command is received, it is determined that the simulation object fails.

[0034] A detection device for an emergency plan, comprising:

[0035] A first determining unit, used to determine a simulation object;

[0036] A first detection unit is configured to send a ping command to a data node to which the first physical network card belongs when the simulation object is a first physical network card and a failure of the first physical network card is detected;

[0037] A second determining unit is configured to determine that the high availability model of the network card is invalid when confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is not received within a preset time;

[0038] A repair unit, configured to repair the first physical network card by using an emergency plan of a network card high availability model to obtain a repair result;

[0039] A first updating unit, configured to update the emergency plan of the network card high availability model according to the repair result;

[0040] A second detection unit is configured to switch the primary and backup shards corresponding to the first physical network card and the second physical network card when the simulation objects are the first physical network card and the second physical network card and it is detected that both the first physical network card and the second physical network card fail;

[0041] A switching unit is used to switch the primary and backup shards using an emergency plan of a distributed database high availability model to obtain a switching result if the primary and backup shards fail to switch within a preset time;

[0042] The second updating unit is used to update the emergency plan of the distributed database high availability model according to the switching result.

[0043] Optionally, the first updating unit is specifically configured to:

[0044] When the repair result indicates that the repair time of the first physical network card exceeds a preset time, the emergency plan of the high availability model of the network card is tested to obtain the first fault cause;

[0045] The emergency plan of the network card high availability model is updated according to the first fault cause.

[0046] Optionally, also include:

[0047] A first acquisition unit, configured to acquire current network card status information of the first physical network card when a repair result indicates that the repair of the first physical network card is complete;

[0048] A second acquisition unit, configured to acquire normal network card status information of the first physical network card;

[0049] A comparing unit, used for comparing the current network card state information with the normal network card state information;

[0050] The third determining unit is configured to determine that the first physical network card is repaired successfully if the current network card status information is consistent with the normal network card status information.

[0051] Optionally, the second updating unit is specifically used to:

[0052] If the switching result indicates that the primary and standby shard switching fails, the emergency plan of the distributed database high availability model is tested to obtain the second fault cause;

[0053] The emergency plan of the distributed database high availability model is updated according to the second failure cause.

[0054] The technical solution provided by the present application determines a simulation object; when the simulation object is a first physical network card and a fault is detected in the first physical network card, a ping command is sent to a data node to which the first physical network card belongs; when confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is not received within a preset time, it is determined that the high availability model of the network card has failed; the first physical network card is repaired by using the emergency plan of the network card high availability model to obtain a repair result; the emergency plan of the network card high availability model is updated according to the repair result; when the simulation objects are the first physical network card and the second physical network card and a fault is detected in both the first physical network card and the second physical network card, the primary and backup shards corresponding to the first physical network card and the second physical network card are switched; if the switching of the primary and backup shards fails within the preset time, the primary and backup shards are switched by using the emergency plan of the distributed database high availability model to obtain a switching result; the emergency plan of the distributed database high availability model is updated according to the switching result. In the present application, when the simulated object is in a fault state, the performance of the emergency plan in this situation is detected. If the emergency plan fails to resolve the fault, the cause of the fault can be located immediately, so that the plan can be effectively corrected and updated, thereby improving the detection efficiency of the emergency plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A flowchart of a detection method for an emergency plan provided in an embodiment of the present application;

[0057] Figure 2 A distributed database high availability architecture diagram provided for an embodiment of the present application;

[0058] Figure 3 A flowchart of a method for simulating object failure provided in an embodiment of the present application;

[0059] Figure 4 A flowchart of a method for verifying a first physical network card provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of the architecture of a detection device for an emergency plan provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of another emergency plan detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0064] like Figure 1 As shown, it is a flow chart of a detection method of an emergency plan provided in an embodiment of the present application, comprising the following steps:

[0065] S101: Determine a simulation object.

[0066] The simulation objects include but are not limited to the first physical network card and the second physical network card.

[0067] It should be noted that the first physical network card and the second physical network card are both arranged in each data node. In addition to the first physical network card and the second physical network card, each data node also has a primary shard and a backup shard.

[0068] For details, see Figure 2 A distributed database high availability architecture diagram is shown in Figure 2 It includes the management node cluster and the data node cluster. The management node cluster is responsible for management tasks such as database access resolution, access distribution, and shard replica master-slave role switching. The data node cluster deploys data node physical machines, provides database data storage, and responds to the operation instructions of the management node cluster. All data of an application is divided into three data shards. The primary and secondary shards of the three data shards are stored on the physical machines of the three data nodes respectively. Under normal circumstances, the primary and secondary shards provide read and write access services to the outside world.

[0069] Optionally, after step S101, in order to verify whether the emergency plan is available, the verification is performed by simulating the object failure, which can ensure that the steps and processes in the emergency plan can cope with the actual failure situation and avoid the plan being invalid or inapplicable when a real disaster or failure occurs. Therefore, another embodiment of the present application provides a method for simulating object failure, such as Figure 3 As shown, the following steps are included:

[0070] S301: Use the ifdown command to perform fault handling on the simulation object.

[0071] It is understandable that killing the simulation object through the ifdown command causes the simulation object to become abnormal and unable to work, that is, enter a fault state.

[0072] Optional, since the hardware failure of the physical network card hardware device includes many scenarios, simulation only through limited means cannot achieve full coverage of the failure scenarios. For example, in actual situations such as optical fiber module failure, sudden increase in packet loss rate, chip disturbance, etc., the network traffic passing through this network card will jitter or even the network link will be completely blocked. In order to further simulate the network card failure scenario in the database physical machine network card high availability model and improve the emergency plan's ability to identify unknown failure scenarios, it is also necessary to manually unplug the simulation object so that the simulation object enters a failure state and cannot work.

[0073] It should be emphasized that the methods for handling faults of simulation objects are not limited to killing the simulation objects by using the ifdown command and manually unplugging the simulation objects. Faults can also be actively injected by other means.

[0074] S302: Send an ifconfig command to the simulation object.

[0075] It is understandable that the ifconfig command is sent to the simulation object to confirm whether the simulation object is in a fault state so as to facilitate subsequent detection of the emergency plan.

[0076] S303: When receiving the DOWN state fed back by the simulation object according to the ifconfig command, it is determined that the simulation object fails.

[0077] It can be understood that when the simulation object is the first physical network card, the feedback from the first physical network card received at this time is a DOWN state, and other physical network cards should be in an UP state. When the simulation object is the first physical network card and the second physical network card, the feedback from the first physical network card and the second physical network card are both DOWN states, and it is determined that the simulation object has a fault.

[0078] S102: When the simulation object is the first physical network card and a failure of the first physical network card is detected, a ping command is sent to a data node to which the first physical network card belongs.

[0079] Among them, when the simulation object is the first physical network card and a failure of the first physical network card is detected, a ping command is sent to the data node to which the first physical network card belongs. If confirmation information is received from the data node to which the first physical network card belongs, it means that the data node to which the first physical network card belongs satisfies the network card high availability model, that is, when a physical network card fails, the remaining physical network cards work normally, and the physical machine does not have a situation where network communication is unavailable due to the hardware failure of a single physical network card, that is, the network card high availability model is satisfied.

[0080] It should be noted that if the network card high availability model is in effect, there is no need to start the emergency plan of the network card high availability model, nor is there any need to re-examine the emergency plan of the network card high availability model.

[0081] S103: When no confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is received within a preset time, it is determined that the high availability model of the network card is invalid.

[0082] Among them, when the confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is not received within the preset time, that is, other physical networks in the data node have abnormalities and are not working normally, then the physical machine of data node 1 does not meet the high-availability hardware redundant deployment specifications of the network card, and it is determined that the high-availability model of the network card has failed.

[0083] S104: Repair the first physical network card using the emergency plan of the network card high availability model to obtain a repair result.

[0084] The repair result includes a result that the first physical network card is successfully repaired and a result that the first physical network card is not successfully repaired.

[0085] Optionally, the emergency plan of the network card high availability model covers the high availability models at the cluster level, port level, board level, device level, cabinet level, computer room level, building level, campus level and city level. For details, see Table 1.

[0086] Table 1

[0087]

[0088]

[0089] It should be noted that the content of Table 1 above is for illustration only.

[0090] It is understandable that when the network card high availability model fails, the physical network card in the data node needs to be repaired. The first physical network card can be repaired, or other physical network cards can be repaired, or all physical network cards can be repaired at the same time. As long as one physical network card is repaired successfully, the data node can provide services normally.

[0091] Optionally, after step S104, after the first physical network card is repaired using the emergency plan, it is necessary to further determine whether the physical network card is repaired successfully, so as to determine whether the emergency plan is available, thereby avoiding further emergency response measures or repeated operations and improving the efficiency of the entire emergency response. Therefore, another embodiment of the present application provides a verification method for the first physical network card, such as Figure 4 As shown, the following steps are included:

[0092] S401: When the repair result indicates that the repair of the first physical network card is complete, current network card status information of the first physical network card is obtained.

[0093] The network card status information includes but is not limited to: network card interface status.

[0094] S402: Obtain normal network card status information of the first physical network card.

[0095] Among them, before simulating the failure of the first physical network card, a ping command is sent to the data node to which the first physical network card belongs, and the data node to which the first physical network card belongs returns normally. Then, an ifconfig command is sent to the data node to which the first physical node belongs to receive the UP status fed back by the data node to which the first physical node belongs, and the normal network card status information of the first physical network card is obtained.

[0096] S403: Compare the current network card status information with the normal network card status information.

[0097] It can be understood that by comparing the repaired current network card status information with the normal network card status information to see whether the status information of the two is consistent, if they are consistent, it is determined that the first physical network card is repaired successfully, if not, it is determined that the first physical network card is repaired unsuccessfully.

[0098] S404: If the current network card status information is consistent with the normal network card status information, it is determined that the first physical network card is repaired successfully.

[0099] S105: Update the emergency plan of the network card high availability model according to the repair result.

[0100] It is understandable that if the repair result indicates that the repair of the first physical network card failed or the repair time of the first physical network card is greater than the preset time, then the emergency plan of the network card high availability model needs to be rechecked to eliminate such problems and update the emergency plan of the network card high availability model.

[0101] Optionally, in another embodiment of the present application, the specific implementation of step S105 includes the following steps:

[0102] When the repair result indicates that the repair time of the first physical network card exceeds the preset time, the emergency plan of the high availability model of the network card is tested to obtain the first fault cause.

[0103] Among them, the causes of the failure include but are not limited to: the spare parts of the specific model required for the repair operation were not in place in time, the efficiency of the software layer in isolating the faulty network card was low, resulting in the failure to isolate the faulty network card in time, and the application was not sensitive to the response time of the physical machine, resulting in the failure to adjust the transaction in time and continue to send transaction requests to the physical machine.

[0104] Optionally, the preset duration can be set according to actual conditions and is not specifically limited here.

[0105] Update the emergency plan of the network card high availability model based on the first fault cause.

[0106] Among them, when updating the emergency plan of the network card high availability model, it is first necessary to adjust it according to the different causes of the failure. That is, in different situations, how to execute the emergency plan to solve the cause of the failure in this regard, so as to avoid the same problem from taking too long to repair next time.

[0107] S106: When the simulation objects are the first physical network card and the second physical network card, and it is detected that both the first physical network card and the second physical network card fail, the primary and backup shards corresponding to the first physical network card and the second physical network card are switched.

[0108] It can be understood that when the simulation objects are the first physical network card and the second physical network card, and it is detected that both the first physical network card and the second physical network card fail, that is, when the physical machines where the primary and backup shards of the distributed data shards are located lose network connection, according to the distributed database high availability model, the management node cluster of the distributed database should immediately switch the primary and backup shard roles for the failed shard, and restore the read and write access capabilities of the data shard as soon as possible.

[0109] It should be emphasized that before simulating the failure of the first physical network card and the second physical network card, the status information of all primary shards and backup shards is periodically determined through database management commands, and the status information of all primary shards and backup shards is used as a steady-state baseline.

[0110] S107: If the switch between the primary and backup shards fails within the preset time, the emergency plan of the distributed database high availability model is used to switch the primary and backup shards to obtain a switch result.

[0111] The switching result indicates the result of successful switching of the primary and backup shards or the result of failed switching of the primary and backup shards.

[0112] It can be understood that if the primary and backup shards are switched successfully within the preset time, it is determined that the distributed database high availability model is effective; if the primary and backup shards are switched unsuccessfully within the preset time, it is determined that the distributed database high availability model is invalid, and it is necessary to start the emergency plan of the distributed database high availability model and use the emergency plan of the distributed database high availability model to switch the primary and backup shards again.

[0113] S108: Update the emergency plan of the distributed database high availability model according to the switching result.

[0114] It is understandable that if the switching result indicates that the primary-backup shard switching has failed, then the emergency plan for the distributed database high availability model needs to be re-examined to eliminate such problems and update the emergency plan for the distributed database high availability model.

[0115] Optionally, in another embodiment of the present application, the specific implementation of step S108 includes:

[0116] If the switching result indicates that the primary and backup shards fail to switch, the emergency plan of the distributed database high availability model is tested to obtain the second fault cause.

[0117] If the switching result indicates that the primary and standby shards have failed to switch, the emergency plan of the distributed database high availability model is tested, the execution log of the emergency plan is analyzed, and the situation of each link and the execution result is analyzed to obtain the second cause of the failure. For example, some steps are executed incorrectly, and the execution time does not meet expectations, such as the design should be completed within 1 second, but the actual execution exceeds 1 second.

[0118] Optionally, after step S107, when the emergency plan of the distributed database high availability model is used to switch the primary and backup shards, it is necessary to further determine whether the switch is successful. If the switch is successful, it is necessary to perform a preset operation so that the data shard can provide services. Therefore, in another embodiment of the present application, a method for changing the number of data shards is provided, including:

[0119] If the primary and backup shards are switched successfully within the preset time, it is detected whether the number of primary and backup shards meets the preset requirements.

[0120] The preset requirements include the number of primary shards and the number of backup shards.

[0121] For example, the number of backup shards after switching is 1, and the number of backup shards in the preset requirement is 2. The number of primary and backup shards is checked to see if they meet the preset requirements. Obviously, the number of backup shards after switching is less than the number of backup shards in the preset requirements, and does not meet the preset requirements. At this time, the backup shard can only be in read-only state and cannot provide services normally.

[0122] If the number of primary and backup shards does not meet the preset requirements, the emergency plan of the distributed database high availability model is used to reduce the number of primary and backup shards.

[0123] Optionally, the modifyGroupConfig command can be used to reduce the number of primary shards and the number of backup shards so that the primary and backup shards can provide services normally.

[0124] For example, the preset requirement includes that the number of primary shards is 2 and the number of backup shards is 2. After switching, the number of primary shards is 1 and the number of backup shards is 1. The number of primary shards and the number of backup shards are reduced to the preset requirement including the number of primary shards is 1 and the number of backup shards is 1. At this time, the number of primary shards and the number of backup shards after switching meet the requirements and can provide services normally. At this time, there is no need to re-examine the emergency plan of the network card high availability model.

[0125] The emergency plan of the distributed database high availability model is updated according to the second failure cause.

[0126] It is understandable that when the primary-backup switch fails, or the reduced number of primary shards and backup shards still cannot enable the primary and backup shards to provide services normally, the emergency plan of the distributed database high availability model needs to be updated according to the second failure cause.

[0127] It should be noted that Figure 1 The content shown is to update the emergency plan through chaos engineering. Chaos engineering improves the reliability of software systems through controllable fault experiments. By purposefully injecting faults, illegal operations or extreme operations into information systems, potential problems in emergency plans are discovered to prevent future system failure risks. Chaos engineering experiments are implemented on production systems to build confidence in the production system's ability to resist production out of control. Chaos engineering can prepare emergency plans for faults, thereby building better systems and improving existing systems.

[0128] In summary, when active fault injection causes the simulated object to enter a fault state, the emergency plan can be updated and optimized in a timely manner by testing its performance in this situation. This process helps improve the emergency plan's ability to identify unknown fault scenarios and the emergency response effect, thereby improving the detection efficiency of the emergency plan.

[0129] like Figure 5As shown, it is a schematic diagram of the architecture of a detection device for an emergency plan provided in an embodiment of the present application, and the detection device includes: a first determination unit 100, a first detection unit 200, a second determination unit 300, a repair unit 400, a first update unit 500, a second detection unit 600, a switching unit 700 and a second update unit 800.

[0130] The first determining unit 100 is used to determine a simulation object.

[0131] The first detection unit 200 is configured to send a ping command to a data node to which the first physical network card belongs when the simulation object is the first physical network card and a failure of the first physical network card is detected.

[0132] The second determining unit 300 is configured to determine that the network card high availability model is invalid when confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is not received within a preset time.

[0133] The repair unit 400 is used to repair the first physical network card by using the emergency plan of the network card high availability model to obtain a repair result.

[0134] The first updating unit 500 is used to update the emergency plan of the network card high availability model according to the repair result.

[0135] The first update unit 500 is specifically used for: when the repair result indicates that the repair time of the first physical network card exceeds the preset time, detecting the emergency plan of the network card high availability model to obtain the first fault cause; updating the emergency plan of the network card high availability model according to the first fault cause.

[0136] The second detection unit 600 is used to switch the primary and backup shards corresponding to the first physical network card and the second physical network card when the simulation objects are the first physical network card and the second physical network card are detected to have failed.

[0137] The switching unit 700 is used to switch the primary and backup shards using the emergency plan of the distributed database high availability model to obtain a switching result if the primary and backup shards fail to switch within a preset time.

[0138] The second updating unit 800 is used to update the emergency plan of the distributed database high availability model according to the switching result.

[0139] The second update unit 800 is specifically used for: if the switching result indicates that the primary-slave shard switching fails, then the emergency plan of the distributed database high availability model is detected to obtain the second fault cause; and the emergency plan of the distributed database high availability model is updated according to the second fault cause.

[0140] In summary, when active fault injection causes the simulated object to enter a fault state, the emergency plan can be updated and optimized in a timely manner by testing its performance in this situation. This process helps improve the emergency plan's ability to identify unknown fault scenarios and the emergency response effect, thereby improving the detection efficiency of the emergency plan.

[0141] Preferably, combined Figure 5 According to the content shown, the detection device also includes: a first acquisition unit, a second acquisition unit, a comparison unit, and a third determination unit.

[0142] The first acquisition unit is configured to acquire current network card status information of the first physical network card when the repair result indicates that the repair of the first physical network card is complete.

[0143] The second acquisition unit is used to acquire normal network card status information of the first physical network card.

[0144] The comparison unit is used to compare the current network card status information with the normal network card status information.

[0145] The third determining unit is configured to determine that the first physical network card is repaired successfully if the current network card status information is consistent with the normal network card status information.

[0146] Preferably, combined Figure 5 As shown in the figure, the detection device also includes: a detection unit and a lowering unit.

[0147] The detection unit is used to detect whether the number of primary and backup shards meets preset requirements if the primary and backup shards are successfully switched within a preset time; wherein the preset requirements include the number of primary shards and the number of backup shards.

[0148] The reduction unit is used to reduce the number of primary shards and the number of backup shards by using the emergency plan of the distributed database high availability model if the number of primary and backup shards does not meet the preset requirements.

[0149] Preferably, combined Figure 5 According to the content shown, the detection device also includes: a processing unit, a sending unit, and a fourth determination unit.

[0150] The processing unit is used to perform fault processing on the simulation object by using the ifdown command.

[0151] The sending unit is used to send ifconfig commands to the simulation object.

[0152] The fourth determining unit is configured to determine that a fault occurs in the simulation object when a DOWN state fed back by the simulation object according to an ifconfig command is received.

[0153] like Figure 6As shown, it is a schematic diagram of another emergency plan detection device provided in an embodiment of the present application, and the detection device includes: a re-inspection case generation module, a re-inspection plan execution module, a system status monitoring module, a plan adjustment determination module, an emergency plan management module, an emergency plan storage module, a production system simulation and emergency disposal module.

[0154] The re-inspection case generation module is used to obtain the status information of the simulation object and perform fault handling on the simulation object.

[0155] The re-inspection plan execution module is used to perform various fault simulation operations.

[0156] The system status monitoring module is used to monitor the status of simulation objects, the effectiveness and failure of high-availability models, the activation of emergency plans for high-availability models, and the emergency response results of emergency plans.

[0157] The plan adjustment judgment module is used to determine whether the emergency plan of the high availability model meets the accuracy and timeliness of emergency response under the fault scenario based on the emergency response results of the emergency plan fed back by the system status monitoring module, and to give a modification judgment conclusion for the emergency plan that does not meet the expected high availability model.

[0158] The emergency plan management module is used to optimize and adjust emergency plans that do not meet the expected high availability model.

[0159] The emergency plan storage module is used to save the emergency plan before re-inspection and the emergency plan of the high-availability model after optimization, adjustment and update.

[0160] The production system module is used to provide banking information systems that provide business processing support services. It is the object of chaos engineering implementation and the object of emergency plan disposal.

[0161] The emergency response module is used to call the emergency plan of the high-availability model to perform emergency response and transaction recovery for failure scenarios that occur in the production system.

[0162] In summary, the emergency plan is rechecked according to the above devices, so that the emergency plan can quickly solve the faults in various fault scenarios, thereby avoiding serious losses.

[0163] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0164] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0165] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting an emergency plan, characterized in that: include: Determine the simulation object; When the simulation object is a first physical network card and a failure of the first physical network card is detected, a ping command is sent to a data node to which the first physical network card belongs; When the confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is not received within the preset time, it is determined that the high availability model of the network card is invalid; Repairing the first physical network card using the emergency plan of the network card high availability model to obtain a repair result; Updating the emergency plan of the network card high availability model according to the repair result; When the simulation objects are the first physical network card and the second physical network card, and it is detected that both the first physical network card and the second physical network card fail, switching the primary and backup shards corresponding to the first physical network card and the second physical network card; If the switch between the primary and backup shards fails within the preset time, the emergency plan of the distributed database high availability model is used to switch the primary and backup shards to obtain the switching result; The emergency plan of the distributed database high availability model is updated according to the switching result.

2. The method according to claim 1, characterized in that The updating of the emergency plan of the network card high availability model according to the repair result includes: When the repair result indicates that the repair time of the first physical network card exceeds a preset time, the emergency plan of the high availability model of the network card is tested to obtain the first fault cause; The emergency plan of the network card high availability model is updated according to the first fault cause.

3. The method according to claim 1, characterized in that Also includes: When the repair result indicates that the repair of the first physical network card is complete, obtaining current network card status information of the first physical network card; Obtaining normal network card status information of the first physical network card; Comparing the current network card status information with the normal network card status information; If the current network card status information is consistent with the normal network card status information, it is determined that the first physical network card is repaired successfully.

4. The method according to claim 1, characterized in that: The updating of the emergency plan of the distributed database high availability model according to the switching result includes: If the switching result indicates that the primary and standby shard switching fails, the emergency plan of the distributed database high availability model is tested to obtain the second fault cause; The emergency plan of the distributed database high availability model is updated according to the second failure cause.

5. The method according to claim 4, characterized in that Also includes: If the primary and backup shards are successfully switched within the preset time, then detecting whether the number of primary and backup shards meets the preset requirements; wherein the preset requirements include the number of primary shards and the number of backup shards; If the number of the primary and backup shards does not meet the preset requirements, the emergency plan of the distributed database high availability model is used to reduce the number of the primary shards and the number of the backup shards.

6. The method according to claim 1, characterized in that After the simulation object is determined, the method further includes: Performing fault handling on the simulation object using the ifdown command; Sending an ifconfig command to the simulated object; When the DOWN state fed back by the simulation object according to the ifconfig command is received, it is determined that the simulation object fails.

7. An emergency plan detection device, characterized in that: include: A first determining unit, used to determine a simulation object; A first detection unit is configured to send a ping command to a data node to which the first physical network card belongs when the simulation object is a first physical network card and a failure of the first physical network card is detected; A second determining unit is configured to determine that the high availability model of the network card is invalid when confirmation information fed back by the data node to which the first physical network card belongs according to the ping command is not received within a preset time; A repair unit, configured to repair the first physical network card by using an emergency plan of a network card high availability model to obtain a repair result; A first updating unit, configured to update the emergency plan of the network card high availability model according to the repair result; A second detection unit is configured to switch the primary and backup shards corresponding to the first physical network card and the second physical network card when the simulation objects are the first physical network card and the second physical network card and it is detected that both the first physical network card and the second physical network card fail; A switching unit is used to switch the primary and backup shards using an emergency plan of a distributed database high availability model to obtain a switching result if the primary and backup shards fail to switch within a preset time; The second updating unit is used to update the emergency plan of the distributed database high availability model according to the switching result.

8. The device according to claim 7, characterized in that The first updating unit is specifically used for: When the repair result indicates that the repair time of the first physical network card exceeds a preset time, the emergency plan of the high availability model of the network card is tested to obtain the first fault cause; The emergency plan of the network card high availability model is updated according to the first fault cause.

9. The device according to claim 7, characterized in that Also includes: A first acquisition unit, configured to acquire current network card status information of the first physical network card when a repair result indicates that the repair of the first physical network card is complete; A second acquisition unit, configured to acquire normal network card status information of the first physical network card; A comparing unit, used for comparing the current network card state information with the normal network card state information; The third determining unit is configured to determine that the first physical network card is repaired successfully if the current network card status information is consistent with the normal network card status information.

10. The device according to claim 7, characterized in that The second updating unit is specifically used for: If the switching result indicates that the primary and standby shard switching fails, the emergency plan of the distributed database high availability model is tested to obtain the second fault cause; The emergency plan of the distributed database high availability model is updated according to the second failure cause.

Images