Fault drill method, device, terminal equipment and medium based on chaos engineering

Abstract

The embodiment of the present invention discloses a fault drill method, device, terminal equipment and medium based on chaos engineering. Among them, a chaos engineering test plan matching the target system is obtained; the target fault is injected into the system according to the fault response influence factor, and the actual fault response result of the system to the target fault is monitored in real time; when it is determined that the difference between the expected fault and the actual fault response result meets the dynamic adjustment condition, the fault response influence factor is matched and adjusted; the operation of injecting the target fault into the system according to the fault response influence factor is returned to execute until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and the mapping relationship between the current fault response influence factor and the actual fault response result is recorded. The problem that the injection of faults causes the system to exceed the controllable range is solved, and the automation level, test efficiency, test scope and system robustness of chaos engineering itself are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer data processing technology, and in particular to a chaos engineering-based fault drill method, apparatus, terminal equipment and medium. Background Art

[0002] Chaos engineering is a complex technical means to improve the resilience of technical architecture. The main implementation method is to randomly introduce faults at all levels to the entire system under certain business background traffic, and observe the performance of the system through perfect monitoring methods, so as to discover the hidden dangers in the system, solve related problems in a targeted manner, and at the same time build the ability and confidence of the system to resist out-of-control conditions in the production environment.

[0003] The existing chaos engineering system mainly focuses on the multi-dimensional fault injection and the construction of the monitoring system for the entire system. In other words, the existing technology mainly focuses on what type of faults and what scope of action will be injected into the system under test, and what kind of fault response the system under test will make. However, once the injected fault exceeds the range that the system under test can withstand, the entire system may enter an out-of-control state and fail to respond to actual business needs. As a result, the existing chaos fault system cannot meet people's needs for system resilience and system robustness. Summary of the invention

[0004] The embodiments of the present invention provide a chaos engineering-based fault drill method, apparatus, terminal device, and medium to prevent fault injection from causing problems beyond the controllable range in the system, thereby improving the automation level of chaos engineering itself and the robustness of the system.

[0005] In a first aspect, an embodiment of the present invention provides a fault drill method based on chaos engineering, which includes:

[0006] Obtain a chaos engineering test plan that matches the target system. The chaos engineering test plan includes: fault response impact factors and expected fault response results corresponding to the target fault to be injected;

[0007] Injecting the target fault into the target system according to the fault response influencing factor, and monitoring the actual fault response result of the target system to the target fault in real time;

[0008] When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match;

[0009] Return to execute the operation of injecting the target fault into the target system according to the fault response influence factor, until the actual fault result matches the expected fault response result or the condition for ending the fault drill is met, and record the mapping relationship between the current fault response influence factor and the current actual fault response result.

[0010] In a second aspect, an embodiment of the present invention further provides a fault drill device based on chaos engineering, and the fault drill device based on chaos engineering includes:

[0011] A chaos engineering test plan acquisition module is used to acquire a chaos engineering test plan that matches the target system. The chaos engineering test plan includes: a fault response impact factor and an expected fault response result corresponding to the target fault to be injected;

[0012] An actual fault response result monitoring module, used to inject a target fault into a target system according to the fault response influencing factor, and monitor the actual fault response result of the target system to the target fault in real time;

[0013] A fault response impact factor adjustment module, used to adjust the fault response impact factor to match when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition;

[0014] A mapping relationship recording module is used to return the operation of injecting the target fault into the target system according to the fault response influence factor until the actual fault result matches the expected fault response result or the condition for ending the fault drill is met, and record the mapping relationship between the current fault response influence factor and the current actual fault response result.

[0015] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the chaos engineering-based fault drill method as described in any embodiment of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the chaos engineering-based fault drill method as described in any embodiment of the present invention is implemented.

[0017] The technical solution provided by the embodiment of the present invention obtains a chaos engineering test plan that matches the target system; injects the target fault into the target system according to the fault response influencing factor, and monitors the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match; returns to execute the operation of injecting the target fault into the target system according to the fault response influencing factor until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and records the mapping relationship between the current fault response influencing factor and the current actual fault response result. In addition to preventing the injection of faults from affecting the system beyond the controllable range, the chaos engineering system can also further improve the automation level of chaos engineering itself, trigger more random faults in a fully automatic manner, further improve the test efficiency and test scope, effectively cover a variety of scenarios, and improve the robustness of the chaos engineering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a failure drill method based on chaos engineering provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of another chaos engineering-based fault drill method provided in an embodiment of the present invention;

[0020] Figure 3 A flowchart of another chaos engineering-based fault drill method provided in an embodiment of the present invention;

[0021] Figure 4 A flowchart of another chaos engineering-based fault drill method provided in an embodiment of the present invention;

[0022] Figure 5 It is a structural schematic diagram of a failure drill device based on chaos engineering provided by an embodiment of the present invention;

[0023] Figure 6 It is a structural diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0025] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0027] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0030] The embodiments of the present invention can perform chaos engineering tests on computer systems based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0031] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0032] Figure 1The present invention provides a flowchart of a method for fault drill based on chaos engineering. This embodiment can be applied to the situation where a chaos engineering system injects faults during a production process or a test process. The method of this embodiment can be executed by a fault drill device based on chaos engineering, which can be implemented by software and / or hardware, and can be configured in a server or terminal device. Figure 1 As shown, the method specifically comprises the following steps:

[0033] S110. Obtain a chaos engineering test plan that matches the target system.

[0034] Among them, the chaos engineering test plan includes: the fault response impact factor and the expected fault response result corresponding to the target fault to be injected.

[0035] Among them, chaos engineering can be a complex technical means to improve the resilience of technical architecture. The main implementation method is to randomly introduce faults at all levels to the entire system under a certain business background flow, and observe the performance of the system through perfect monitoring means, so as to discover the hidden dangers of the system, solve related problems in a targeted manner, and at the same time establish the ability and confidence of the system to resist out-of-control conditions in the production environment. The chaos engineering test plan can specifically refer to the description information of the chaos engineering acting on the target system. The chaos engineering test plan is used to describe what kind of faults will be injected into the target system, and what kind of response the target system is expected to achieve for the injected faults.

[0036] In this embodiment, the target system refers to a business execution system for applying chaos engineering, and the business execution system may include: multiple system devices for performing business processing functions such as data measurement, data calculation or data transmission. Each of the above system devices may include multiple terminal devices, and a server or management node for unified management of multiple terminal devices, or may only include multiple terminal devices, or may only include multiple servers, etc. This embodiment does not limit this.

[0037] Optionally, the system device may be an actual physical machine, or a virtual machine installed on a physical machine, etc.

[0038] The target fault to be injected can be: various abnormal problems that may occur in the operation of the target system, such as system device disconnection, device downtime, load balancing service failure, service timeout, database connection timeout, storage space unreadable or memory disorder, etc. The target fault is generally associated with the actual operation scenario of the target system.

[0039] Among them, the fault response impact factor can be the degree of impact of the target fault on the chaos engineering, which can specifically include the fault effect degree of the target fault and the expected business traffic load degree. The expected fault response result can refer to what kind of response the target system is expected to achieve for the injected fault. For example, continue to execute the business normally, complete the switch of the primary and backup servers within the set time, or resume business processing after the set fault response time, etc. The expected fault response result generally reflects the user's tolerance for the target system to respond to a certain fault.

[0040] S120: Inject a target fault into a target system according to the fault response influencing factor, and monitor an actual fault response result of the target system to the target fault in real time.

[0041] The actual fault response result may refer to the response that the target system can achieve for the injected fault. For example, continuing to execute the business normally, switching between the primary and backup servers within a certain period of time, or resuming business processing after a set fault response period, etc. The actual fault response result refers to the response that the target system actually generates for the injected target fault.

[0042] Among them, after injecting a certain fault A into the target system, if the actual fault response result of the target system meets the expected fault response result, it means that the target system can normally cope with the occurrence of the fault A; if the actual fault response result of the target system does not meet the expected fault response result, it means that the target system cannot normally cope with the occurrence of the fault A, therefore, the target system needs to be optimized or upgraded in performance.

[0043] Among them, the actual fault response result of the target system meets the expected fault response result, which specifically means that the actual fault response result falls within the control range of the expected fault response result, and the actual fault response result of the target system does not meet the expected fault response result, which specifically means that the actual fault response result does not fall within the control range of the expected fault response result.

[0044] For example, if the expected fault response result corresponding to fault A injected into the target system is to resume business processing within 5 minutes, then when the business processing recovery time in the actual fault response result is less than or equal to 5 minutes, it means that the actual fault response result falls within the control range of the expected fault response result, that is, the target system can normally cope with the occurrence of fault A; and when the business processing recovery time in the actual fault response result is greater than 5 minutes, it means that the actual fault response result does not fall within the control range of the expected fault response result, that is, the target system cannot normally cope with the occurrence of fault A.

[0045] Optionally, the fault response influencing factor may include: a fault effect degree of a target fault; accordingly, injecting the target fault into the target system according to the fault response influencing factor may include:

[0046] According to the target fault and the fault effect degree of the target fault, a fault parameter pointing to at least one target system device in the target system is formed; and the fault parameter is sent to each of the target system devices to inject a target fault matching the fault effect degree into each of the target system devices.

[0047] Among them, the fault effect degree can be used to describe the impact of the fault to be injected into the target system on the target system, or the system equipment in the target system. Specifically, the fault effect degree may include the number of system equipment injected with the fault, and / or the size of the fault injected into a single system equipment. The fault parameters may include the size of the target fault and the fault effect degree, etc. The target system equipment may be an actual physical machine, or a virtual machine installed on a physical machine, etc.

[0048] In a specific example, if the fault you want to inject into the target system is to power off 5 system devices in the target system, you can set the target fault to power off, and set the number of system devices to inject faults to 5. In another specific example, if the fault you want to inject into the target system is to timeout a database request of a target server in the target system for 5 minutes, you can set the target fault to database request timeout, and the fault size injected into a single system device is a timeout of 5 minutes.

[0049] In this embodiment, firstly, a fault parameter pointing to at least one target system device in the target system is formed according to the target fault and the fault effect degree of the target fault. Further, the fault parameter is sent to each target system device to inject the target fault matching the fault effect degree into each target system device. Accordingly, the actual fault response result of the target system to the target fault is monitored in real time.

[0050] The advantage of this setting is that by sending fault parameters to each target system device and injecting a target fault that matches the fault effect, the actual fault response result of the target fault can be monitored in real time. In this way, the fault response result of the current target fault and the fault effect of the target fault can be obtained in real time, and the actual fault response result obtained is more accurate and timely, thereby improving the work efficiency and automation of chaos engineering.

[0051] S130, obtaining the difference type between the expected fault response result and the actual fault response result: if it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, execute S140; if it is determined that the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, execute S150.

[0052] In this embodiment, in response to the problem that the prior art does not pay enough attention to the correlation between fault injection and system monitoring, and the fault injection is not adjusted according to the degree of impact on the system, a technical solution is creatively proposed to dynamically adjust the fault effect degree of the target fault by monitoring the actual fault response results of the target system to the target fault.

[0053] The above settings are introduced mainly due to two considerations. One aspect is that the target fault can be injected into the target system in the production environment. Therefore, if the actual fault response result after the target fault is injected is far worse than the expected fault response result, it means that the user's normal business response cannot be met, which will greatly affect the user's experience of using the target system and the business processing performance of the target system. Therefore, when it is determined by monitoring the actual fault response result that the current target system cannot cope with the target fault normally, the actual fault response result can be alleviated by reducing the fault response impact factor, and then it can be obtained at what level of fault effect the target fault is under to achieve the expected fault response result, so as to dynamically obtain new test scenarios and test data.

[0054] On the other hand, if the actual fault response result after the target fault is injected is better than the expected fault response result, for example, the business recovery time is shorter, it means that the target system can cope with the target fault normally while still leaving a certain margin. Then, by increasing the fault response impact factor, the target system's response limit to the target fault can be explored, thereby achieving the technical effect of dynamically obtaining new test scenarios and test data.

[0055] The dynamic adjustment conditions may be some conditions that need to be adjusted accordingly due to the requirements of the fault response results. For example, when it is determined that the actual fault response result is much better than the expected fault response result, or when it is determined that the actual fault response result is much worse than the expected fault response result, it is determined that the difference between the expected fault response result and the actual fault response result satisfies the dynamic adjustment condition.

[0056] Among them, the difference between the expected fault response result and the actual fault response result of the target system meets the dynamic adjustment condition, specifically: the difference value (absolute value difference) between the expected fault response result and the actual fault response result is greater than or equal to the preset difference threshold value.

[0057] Correspondingly, it can be understood that the difference between the expected fault response result and the actual fault response result of the target system does not meet the dynamic adjustment condition, specifically referring to: the difference value between the expected fault response result and the actual fault response result is less than or equal to the preset difference threshold value (absolute value difference), that is, the expected fault response result matches the actual fault response result; or, through a preset number of adjustments, or an adjustment in a preset adjustment method, the difference value between the expected fault response result and the actual fault response result is greater than or equal to the preset difference threshold value, that is, the condition for ending the fault drill is met.

[0058] In a specific example, the expected fault response result is set to a business recovery processing time less than or equal to 5 minutes, and the preset difference threshold value is 1 minute. When the business recovery processing time in the actual fault response result is 2 minutes, it can be determined that the expected fault response result of the target system and the actual fault response result meet the dynamic adjustment condition; when the business recovery processing time in the actual fault response result is 4 minutes and 55 seconds, it can be determined that the expected fault response result of the target system matches the actual fault response result.

[0059] S140: Adjust the fault response influencing factor to match the fault response, and return to execute S120.

[0060] Optionally, when it is determined that the difference between the expected fault response result and the actual fault response result satisfies the dynamic adjustment condition, the matching adjustment of the fault response influencing factor may include: when it is determined that the actual fault response result falls within the control range of the expected fault response result, increasing the proportion according to a preset degree of effect, thereby increasing the fault effect degree of the target fault; when it is determined that the actual fault response result does not fall within the control range of the expected fault response result, reducing the proportion according to a preset degree of effect, thereby reducing the fault effect degree of the target fault.

[0061] As described above, when it is determined that the actual fault response result is much better than the expected fault response result, it is determined that the actual fault response result falls within the control range of the expected fault response result; when it is determined that the actual fault response result is much worse than the expected fault response result, it is determined that the actual fault response result does not fall within the control range of the expected fault response result.

[0062] For example, if the target fault A injected into the target system is a downtime fault, the fault effect level is that 3 out of 10 system devices experience downtime, the corresponding expected fault response result is to resume business processing within 5 minutes, and the preset difference threshold value is 1 minute, then when the business processing recovery time in the actual fault response result is less than or equal to 4 minutes, it means that the actual fault response result falls within the control range of the expected fault response result, that is, while the target system can normally cope with the occurrence of the target fault A, it also has a certain margin, and thus the fault effect level of the target fault can be correspondingly improved; and when the business processing recovery time in the actual fault response result is greater than 6 minutes, it means that the actual fault response result does not fall within the control range of the expected fault response result, that is, the target system cannot normally cope with the occurrence of the target fault A, and the entire target system is out of control. Therefore, it is necessary to reduce the fault effect level of the target fault accordingly.

[0063] Exemplarily, when the target system can normally cope with the occurrence of the target fault A, that is, the recovery service processing time in the actual fault response result is less than or equal to 4 minutes, the fault effect degree of the target fault can be increased at this time. Specifically, the fault effect degree can be increased from 3 out of 10 system devices having downtime to 4 out of 10 system devices having downtime. When the recovery service processing time in the actual fault response result at this time is still less than or equal to 4 minutes, the fault effect degree of the target fault can be further increased, that is, the fault effect degree can be increased from 4 out of 10 system devices having downtime to 5 out of 10 system devices having downtime. Correspondingly, if the recovery service processing time is greater than 6 minutes, the fault effect degree of the target fault can be further reduced, and the fault effect degree can be reduced from 3 out of 10 system devices having downtime to 2 out of 10 system devices having downtime, and so on, until the actual fault result matches the expected fault response result or the condition for ending the fault drill is met.

[0064] Specifically, since the target fault can be injected into the target system in the production environment, if the actual fault response result after the target fault is injected is far worse than the expected fault response result, it means that the user's normal business response cannot be met, which will greatly affect the user's usage experience of the target system and the business processing performance of the target system. Therefore, when it is determined by monitoring the actual fault response result that the current target system cannot cope with the target fault normally, the actual fault response result can be alleviated by reducing the fault effect degree of the target fault, and then it can be obtained at what fault effect degree the target fault is under to achieve the expected fault response result, so as to dynamically obtain new test scenarios and test data.

[0065] Furthermore, if the actual fault response result after the target fault is injected is better than the expected fault response result, and the service recovery time is shorter, it means that the target system can normally cope with the target fault while leaving a certain margin. Furthermore, by increasing the degree of the fault effect of the target fault, the target system's response limit to the target fault can be explored, thereby also achieving the technical effect of dynamically obtaining new test scenarios and test data.

[0066] In this embodiment, by determining whether the difference between the expected fault response result and the actual fault response result satisfies the dynamic adjustment conditions, the fault response influence factor is adjusted to match until the actual fault result matches the expected fault response result or the fault drill termination conditions are met (for example, the fault response influence factor has been adjusted a set number of times).

[0067] The advantage of this setting is that it can more accurately and reasonably calculate the target system's response to the target fault, thereby preventing the injection of faults from causing impacts on the system beyond the controllable range, triggering more random faults in a fully automatic manner, further improving the test efficiency and test scope, effectively covering multiple scenarios, and improving the robustness of the chaos engineering system.

[0068] S150: Record the mapping relationship between the current fault response influencing factor and the current actual fault response result.

[0069] The mapping relationship may refer to the corresponding relationship between the current fault response influencing factor and the matched current actual fault response result.

[0070] For example, if the target fault A injected into the target system corresponds to an actual fault response result of restoring business processing within 5 minutes, a mapping relationship such as "target fault A, restoring business processing within 5 minutes" can be established accordingly.

[0071] Optionally, the monitoring of the target system, the injection of target faults and the injection of simulated business traffic are all achieved through probes set on various system devices of the target system.

[0072] The probe may be a process or a plug-in running on a system device. Specifically, according to the different functions performed by the probe, the probe can be divided into monitoring probes, fault probes and flow probes, which are used to respectively implement operations such as monitoring the target system, injecting target faults and injecting simulated business traffic.

[0073] The benefit of this setting is that the target system can be monitored, target faults injected, and simulated business traffic injected by setting probes on each system device of the target system. This makes the monitoring of the target system, the injection of target faults, and the injection of simulated business traffic more specific and procedural, triggering more random faults in a fully automatic way, further improving the test efficiency and test scope, effectively covering multiple scenarios, and improving the robustness of the chaos engineering system.

[0074] The technical solution provided by the embodiment of the present invention obtains a chaos engineering test plan that matches the target system; injects the target fault into the target system according to the fault response influencing factor, and monitors the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match; returns to execute the operation of injecting the target fault into the target system according to the fault response influencing factor until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and records the mapping relationship between the current fault response influencing factor and the current actual fault response result. In addition to preventing the injection of faults from affecting the system beyond the controllable range, the chaos engineering system can also further improve the automation level of chaos engineering itself, trigger more random faults in a fully automatic manner, further improve the test efficiency and test scope, effectively cover a variety of scenarios, and improve the robustness of the chaos engineering system.

[0075] Figure 2 This is a flow chart of another fault drill method based on chaos engineering provided by an embodiment of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, the fault response influencing factor also includes: expected service traffic load level.

[0076] Accordingly, the method specifically comprises the following steps:

[0077] S210. Obtain a chaos engineering test plan that matches the target system.

[0078] The chaos engineering test plan includes: the fault response impact factor and the expected fault response result corresponding to the target fault to be injected. The fault response impact factor may include the fault effect degree of the target fault and the expected service traffic load degree.

[0079] S220: Inject the target fault into the target system according to the fault effect degree of the target fault and the expected service traffic load degree.

[0080] In this embodiment, the target fault is injected into the target system according to the fault response impact factor, which may specifically include: injecting the target fault into the target system according to the fault effect degree of the target fault and the expected service traffic load degree.

[0081] Injecting the target fault into the target system according to the fault effect degree of the target fault and the expected service traffic load degree may specifically include:

[0082] According to the target fault and the fault effect degree of the target fault, forming a fault parameter directed to at least one target system device in the target system;

[0083] Sending the fault parameter to each of the target system devices to inject a target fault matching the fault effect degree into each of the target system devices;

[0084] If it is determined that the target system is currently in a production environment, obtaining an actual service traffic load level in the target system;

[0085] If it is determined that the actual service traffic load level is less than the expected service traffic load level, calculating the traffic value to be supplemented;

[0086] Generate simulated business traffic that matches the traffic value to be supplemented, and inject it into the target system to control the target system to form an actual fault response result for the target fault under the expected business traffic load level.

[0087] In this embodiment, the target system may be in a production environment or a test environment, wherein the actual service traffic load level may be the actual service load level of the current target system in the production environment.

[0088] Exemplarily, the actual service flow load degree in the target system can be 80% or 100%. In particular, when the actual service flow load degree is 100%, the actual service flow load degree in the target system is full load. Accordingly, the actual service flow load degree in each target system generally has a minimum load (e.g., 20%) and a maximum load (i.e., full load).

[0089] The expected service traffic load level may refer to the service traffic load level, which is the load level that the target system is expected to achieve for the service traffic.

[0090] For example, for the target system, the expected service traffic load is 80%. If the actual service traffic load in the target system is 60%, the chaos engineering test that the user really needs cannot be met, and the matching traffic value can be supplemented to achieve an actual service traffic load of 80%.

[0091] It needs to be explained again that if it is determined that the actual business traffic load level is less than the expected business traffic load level, it means that the current actual generation environment can already meet (or exceed) the expected business traffic load level required by the user. At this time, there is no need to inject any simulated business traffic into the target system, and only the current actual business traffic load level can be used. Accordingly, when subsequently adjusting the fault response influencing factor to match, it is only necessary to make a matching adjustment to the fault effect level of the target fault.

[0092] In addition, if it is determined that the target system is currently in a production environment, a simulated business traffic matching the expected business traffic load level can be directly generated and injected into the target system to control the target system to form an actual fault response result for the target fault under the expected business traffic load level. S230, real-time monitoring of the actual fault response result of the target system for the target fault.

[0093] S240, obtaining the difference type between the expected fault response result and the actual fault response result: if it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, execute S250; if it is determined that the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, execute S260.

[0094] S250: Adjust the fault response influencing factor to match the fault response, and return to execute S220.

[0095] Optionally, when it is determined that the difference between the expected fault response result and the actual fault response result satisfies the dynamic adjustment condition, the fault response influencing factor is adjusted to match, including one of the following:

[0096] Only the fault effect degree is matched and adjusted; only the expected business traffic load degree is matched and adjusted; the fault effect degree is matched and adjusted first and then the expected business traffic load degree; and the expected business traffic load degree is matched and adjusted first and then the fault effect degree.

[0097] For example, if the target fault A injected into the target system has the expected fault response result of restoring business processing within 5 minutes, and for the target system, the actual business traffic load level in the production environment is 60%, and the expected business traffic load level is 80%, by injecting the corresponding simulated business traffic of 60M / s into the target system, the current business traffic load level in the target system reaches 80%. Accordingly, when the target fault A is injected into the target system, it can be measured that the corresponding actual fault response result is restoring business processing within 3 minutes. For the above situation, there can be four situations to match and adjust it.

[0098] First, only the fault effect degree can be adjusted to match. Since the target fault A is injected into the target system, the corresponding actual fault response result can be measured to be the restoration of business processing within 3 minutes. However, the expected fault response result is to restore business processing within 5 minutes, so the fault effect degree is increased, and the actual fault response result of the target system for the target fault is monitored in real time until the actual fault result matches the expected fault response result or the condition for ending the fault drill is met.

[0099] On the second aspect, only the expected business traffic load level can be adjusted to match. Since when the target fault A is injected into the target system, the expected business traffic load level is 80% by adding the simulated business traffic matching the traffic value to be supplemented, which does not reach the full load, the expected business traffic load level can be increased accordingly. For example, 90%, and then a new supplementary traffic value can be calculated, and a new simulated business traffic matching the traffic value to be supplemented can be generated accordingly and injected into the target system. The actual fault response result of the target system to the target fault is monitored in real time until the actual fault result matches the expected fault response result or the condition for ending the fault drill is met.

[0100] Third, make matching adjustments in the order of adjusting the fault effect first and then adjusting the expected service traffic load. Assuming that target fault A is injected into the target system, the corresponding actual fault response result can be measured to restore service processing within 3 minutes. However, the expected fault response result is to restore service processing within 5 minutes, so the fault effect needs to be increased.

[0101] In one case, when the fault effect reaches the preset maximum fault effect, the actual fault response result of the target system still falls within the control range of the expected fault response result. For example, 6 out of 10 system devices are down, and the actual fault response result is determined to be the restoration of business processing within 4 minutes. At this time, the injection environment of the target fault can be further deteriorated by correspondingly increasing the expected business traffic load level to detect the fault processing limit of the target system.

[0102] In another case, when the actual fault response result of the target system is adjusted from a situation falling within the control range of the expected fault response result to a situation not falling within the control range of the expected fault response result by increasing the fault effect degree, for example, when 4 out of 10 system devices are down, the actual fault response result is determined to be restoring business processing within 4 minutes; when the target fault is adjusted to 5 out of 10 system devices being down by increasing the fault effect degree, the actual fault response result is determined to be restoring business processing within 6 minutes. At this time, the expected business traffic load degree can be reduced on the basis that 5 out of 10 system devices are down, or the expected business traffic load degree can be increased on the basis that 4 out of 10 system devices are down, until the actual fault result matches the expected fault response result or the conditions for ending the fault drill are met.

[0103] In the fourth aspect, the matching adjustment is carried out in the order of adjusting the expected business traffic load level first and then adjusting the fault effect level. Assuming that the target fault A is injected into the target system, the corresponding actual fault response result can be measured to be the restoration of business processing within 3 minutes. However, the expected fault response result is to restore business processing within 5 minutes. At the same time, if it is determined that the expected business traffic load level is currently achieved by adding a simulated business traffic matching the traffic value to be supplemented, for example, 80%, it has not reached the full load, therefore, the expected business traffic load level can be increased accordingly. For example, 90%, and then a new supplementary traffic value can be calculated, and a new simulated business traffic matching the traffic value to be supplemented is generated accordingly and injected into the target system.

[0104] In one case, after the expected service traffic load level is adjusted to the maximum load of the target system, the actual fault response result of the target system still falls within the control range of the expected fault response result. For example, the current expected service traffic load level is adjusted to 100%. At this time, the injection environment of the target fault can be further deteriorated by correspondingly increasing the fault effect level to detect the fault processing limit of the target system.

[0105] In another case, by increasing the expected business traffic load level, the actual fault response result of the target system is adjusted from a situation falling within the control range of the expected fault response result to a situation not falling within the control range of the expected fault response result. For example, when the expected business traffic load level is 90%, the actual fault response result is determined to be restoring business processing within 4 minutes. When the expected business traffic load level is increased to 95%, the actual fault response result is determined to be restoring business processing within 6 minutes. At this time, the fault effect level can be reduced based on the expected business traffic load level of 95%, or, when the expected business traffic load level is 90%, the fault effect level can be increased, until the actual fault result matches the expected fault response result or the conditions for ending the fault drill are met.

[0106] Optionally, when it is determined that the difference between the expected fault response result and the actual fault response result satisfies the dynamic adjustment condition, the matching adjustment of the expected service traffic load degree may include:

[0107] When it is determined that the actual fault response result falls within the control range of the expected fault response result, the expected business traffic load level is increased by a ratio according to a preset load level; when it is determined that the actual fault response result does not fall within the control range of the expected fault response result, the expected business traffic load level is reduced by a ratio according to a preset load level.

[0108] For example, if the target fault A injected into the target system has an expected business traffic load level of 80%, which has not reached full load, and the actual fault response result falls within the control range of the expected fault response result, that is, the target system can normally cope with the occurrence of the target fault A. Therefore, the expected business traffic load level can be increased accordingly, for example, 90%, and then a new supplementary traffic value can be calculated, and a new simulated business traffic matching the traffic value to be supplemented can be generated and injected into the target system.

[0109] Correspondingly, the expected business traffic load level is 60%, which does not reach the minimum load, and the actual fault response result does not fall within the control range of the expected fault response result, that is, the target system cannot normally respond to the occurrence of the target fault A. Therefore, the expected business traffic load level can be reduced accordingly.

[0110] In this embodiment, by determining whether the difference between the expected fault response result and the actual fault response result satisfies the dynamic adjustment conditions, the expected business traffic load degree, the fault effect degree, or both the expected business traffic load degree and the fault effect degree in the fault response influencing factors are adjusted to match, until the actual fault result matches the expected fault response result or the conditions for ending the fault drill are met.

[0111] S260: Record the mapping relationship between the current fault response influencing factor and the current actual fault response result.

[0112] The benefit of this setting is that by adding a matching adjustment to the expected business traffic load in the fault response influencing factors, the target system's tolerance for target faults can be calculated more accurately and reasonably, thereby preventing the injection of faults from causing impacts beyond the controllable range on the system, triggering more random faults in a fully automatic manner, further improving test efficiency and test scope, effectively covering multiple scenarios, and improving the robustness of the chaos engineering system.

[0113] The technical solution provided by the embodiment of the present invention obtains a chaos engineering test plan that matches the target system; injects the target fault into the target system according to the expected business traffic load in the fault response influencing factor, and monitors the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match; returns to execute the operation of injecting the target fault into the target system according to the fault response influencing factor until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and records the mapping relationship between the current fault response influencing factor and the current actual fault response result. It can avoid the impact of fault injection on the system beyond the controllable range, trigger more random faults in a fully automatic manner, further improve the test efficiency and test scope, effectively cover a variety of scenarios, and improve the robustness of the chaos engineering system.

[0114] Figure 3 This is a flowchart of another fault drill method based on chaos engineering provided by an embodiment of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the matching adjustment is performed in the order of first adjusting the fault effect degree and then adjusting the expected business traffic load degree.

[0115] Accordingly, the method specifically comprises the following steps:

[0116] S310. Obtain a chaos engineering test plan that matches the target system.

[0117] The chaos engineering test plan includes: the fault response impact factor and the expected fault response result corresponding to the target fault to be injected. The fault response impact factor may include the fault effect degree of the target fault and the expected service traffic load degree.

[0118] S320: Inject the target fault into the target system according to the fault effect degree of the target fault and the expected service traffic load degree, and monitor the actual fault response result of the target system to the target fault in real time.

[0119] S330, obtaining the difference type between the expected fault response result and the actual fault response result: if it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, execute S340; if it is determined that the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, execute S350.

[0120] S340 , adjusting the fault response influencing factor in a matching manner in the order of first adjusting the fault effect degree and then adjusting the expected service flow load degree, and returning to execute S320 .

[0121] For example, if the target fault A injected into the target system has the expected fault response result of restoring business processing within 5 minutes, and for the target system, the actual business traffic load level in the production environment is 60%, and the expected business traffic load level is 80%, by injecting the corresponding simulated business traffic of 60M / s into the target system, the current business traffic load level in the target system reaches 80%. Accordingly, when the target fault A is injected into the target system, it can be measured that the corresponding actual fault response result is restoring business processing within 3 minutes. For the above situation, there can be four situations to match and adjust it.

[0122] First, if the expected fault response result corresponding to the target fault A injected into the target system is to resume business processing within 5 minutes, then when the business processing recovery time in the actual fault response result is less than or equal to 5 minutes, it means that the actual fault response result falls within the control range of the expected fault response result, that is, the target system can normally cope with the occurrence of the target fault A; and when the business processing recovery time in the actual fault response result is greater than 5 minutes, it means that the actual fault response result does not fall within the control range of the expected fault response result, that is, the target system cannot normally cope with the occurrence of the fault A.

[0123] Specifically, when the target fault A is injected into the target system, it can be measured that the corresponding actual fault response result is to restore business processing within 3 minutes, that is, when the recovery business processing time in the actual fault response result is less than or equal to 5 minutes, the fault effect degree of the target fault can be improved.

[0124] In one case, when the fault effect reaches a preset maximum fault effect, the actual fault response result of the target system still falls within the control range of the expected fault response result. For example, 6 of the 10 system devices are down, and the actual fault response result is determined to be restoring business processing within 4 minutes. At this time, the injection environment of the target fault can be further deteriorated by correspondingly increasing the expected business traffic load level to detect the fault processing limit of the target system.

[0125] In another case, when the actual fault response result of the target system is adjusted from a situation falling within the control range of the expected fault response result to a situation not falling within the control range of the expected fault response result by increasing the fault effect degree, for example, when 4 out of 10 system devices are down, the actual fault response result is determined to be restoring business processing within 4 minutes; when the target fault is adjusted to 5 out of 10 system devices being down by increasing the fault effect degree, the actual fault response result is determined to be restoring business processing within 6 minutes. At this time, the expected business traffic load degree can be reduced on the basis that 5 out of 10 system devices are down, or the expected business traffic load degree can be increased on the basis that 4 out of 10 system devices are down, until the actual fault result matches the expected fault response result or the conditions for ending the fault drill are met.

[0126] Specifically, the expected service traffic load level is adjusted. Since when the target fault A is injected into the target system, the expected service traffic load level is 80% by adding the simulated service traffic matching the traffic value to be supplemented, which does not reach the full load, the expected service traffic load level can be increased accordingly. For example, 90%, and then a new supplementary traffic value can be calculated, and a new simulated service traffic matching the traffic value to be supplemented is generated accordingly and injected into the target system. The actual fault response result of the target system to the target fault is monitored in real time until the actual fault result matches the expected fault response result or the condition for ending the fault drill is met.

[0127] S350: Record the mapping relationship between the current fault response influencing factor and the current actual fault response result.

[0128] The technical solution provided by the embodiment of the present invention obtains a chaos engineering test plan that matches the target system; injects the target fault into the target system according to the expected business traffic load level in the fault response influencing factor, and monitors the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the matching adjustment is performed in the order of adjusting the fault effect level first and then adjusting the expected business traffic load level; returns to execute the operation of injecting the target fault into the target system according to the fault response influencing factor until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and records the mapping relationship between the current fault response influencing factor and the current actual fault response result. It can avoid the impact of fault injection on the system beyond the controllable range, trigger more random faults in a fully automatic manner, further improve the test efficiency and test scope, effectively cover a variety of scenarios, and improve the robustness of the chaos engineering system.

[0129] Figure 4 This is a flowchart of another fault drill method based on chaos engineering provided by an embodiment of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the matching adjustment is performed in the order of first adjusting the expected service traffic load degree and then adjusting the fault effect degree.

[0130] Accordingly, the method specifically comprises the following steps:

[0131] S410. Obtain a chaos engineering test plan that matches the target system.

[0132] The chaos engineering test plan includes: the fault response impact factor and the expected fault response result corresponding to the target fault to be injected. The fault response impact factor may include the fault effect degree of the target fault and the expected service traffic load degree.

[0133] S420: Inject the target fault into the target system according to the fault effect degree of the target fault and the expected service traffic load degree, and monitor the actual fault response result of the target system to the target fault in real time.

[0134] S430, obtaining the difference type between the expected fault response result and the actual fault response result: if it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, execute S440; if it is determined that the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, execute S450.

[0135] S440 , adjusting the fault response influencing factor in a matching manner in the order of first adjusting the expected service flow load level and then adjusting the fault effect level, and returning to execute S420 .

[0136] For example, if the target fault A injected into the target system has the expected fault response result of restoring business processing within 5 minutes, and for the target system, the actual business traffic load level in the production environment is 60%, and the expected business traffic load level is 80%, by injecting the corresponding simulated business traffic of 60M / s into the target system, the current business traffic load level in the target system reaches 80%. Accordingly, when the target fault A is injected into the target system, it can be measured that the corresponding actual fault response result is restoring business processing within 3 minutes. For the above situation, there can be four situations to match and adjust it.

[0137] First, assuming that the target fault A is injected into the target system, the corresponding actual fault response result can be measured to be the restoration of business processing within 3 minutes. However, the expected fault response result is to restore business processing within 5 minutes. At the same time, if it is determined that the expected business traffic load level is currently reached by adding the simulated business traffic matching the traffic value to be supplemented, for example, 80%, it has not reached the full load, so the expected business traffic load level can be increased accordingly. For example, 90%, and then a new supplementary traffic value can be calculated, and a new simulated business traffic matching the traffic value to be supplemented is generated accordingly and injected into the target system.

[0138] In one case, after the expected service traffic load level is adjusted to the maximum load of the target system, the actual fault response result of the target system still falls within the control range of the expected fault response result. For example, the current expected service traffic load level is adjusted to 100%. At this time, the injection environment of the target fault can be further deteriorated by correspondingly increasing the fault effect level to detect the fault processing limit of the target system.

[0139] In another case, by increasing the expected business traffic load level, the actual fault response result of the target system is adjusted from a situation falling within the control range of the expected fault response result to a situation not falling within the control range of the expected fault response result. For example, when the expected business traffic load level is 90%, the actual fault response result is determined to be restoring business processing within 4 minutes. When the expected business traffic load level is increased to 95%, the actual fault response result is determined to be restoring business processing within 6 minutes. At this time, the fault effect level can be reduced based on the expected business traffic load level of 95%, or, when the expected business traffic load level is 90%, the fault effect level can be increased, until the actual fault result matches the expected fault response result or the conditions for ending the fault drill are met.

[0140] S450: Record the mapping relationship between the current fault response influencing factor and the current actual fault response result.

[0141] The technical solution provided by the embodiment of the present invention obtains a chaos engineering test plan that matches the target system; injects the target fault into the target system according to the expected business traffic load level in the fault response influencing factor, and monitors the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the matching adjustment is performed in the order of adjusting the expected business traffic load level first and then adjusting the fault effect level; returns to execute the operation of injecting the target fault into the target system according to the fault response influencing factor until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and records the mapping relationship between the current fault response influencing factor and the current actual fault response result. It can avoid the impact of fault injection on the system beyond the controllable range, trigger more random faults in a fully automatic manner, further improve the test efficiency and test scope, effectively cover a variety of scenarios, and improve the robustness of the chaos engineering system.

[0142] Figure 5 is a structural diagram of a chaos engineering-based fault drill device provided by another embodiment of the present invention. The chaos engineering-based fault drill device provided by this embodiment can be implemented by software and / or hardware and can be configured in a server or terminal device. Figure 5 As shown, the device may specifically include: a chaos engineering test plan acquisition module 510, an actual fault response result monitoring module 520, a fault response impact factor adjustment module 530 and a mapping relationship recording module 540.

[0143] Among them, the chaos engineering test plan acquisition module 510 is used to obtain a chaos engineering test plan that matches the target system; the actual fault response result monitoring module 520 is used to inject the target fault into the target system according to the fault response influence factor, and monitor the actual fault response result of the target system to the target fault in real time; the fault response influence factor adjustment module 530 is used to adjust the fault response influence factor to match when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition; the mapping relationship recording module 540 is used to return to execute the operation of injecting the target fault into the target system according to the fault response influence factor until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and records the mapping relationship between the current fault response influence factor and the current actual fault response result.

[0144] The technical solution provided by the embodiment of the present invention obtains a chaos engineering test plan that matches the target system; injects the target fault into the target system according to the fault response influencing factor, and monitors the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match; returns to execute the operation of injecting the target fault into the target system according to the fault response influencing factor until the actual fault result matches the expected fault response result or meets the condition for ending the fault drill, and records the mapping relationship between the current fault response influencing factor and the current actual fault response result. In addition to preventing the injection of faults from affecting the system beyond the controllable range, the chaos engineering system can also further improve the automation level of chaos engineering itself, trigger more random faults in a fully automatic manner, further improve the test efficiency and test scope, effectively cover a variety of scenarios, and improve the robustness of the chaos engineering system.

[0145] On the basis of the above embodiments, the fault response influencing factors include: the fault effect degree of the target fault; the actual fault response result monitoring module 520 can be specifically used to: form a fault parameter pointing to at least one target system device in the target system according to the target fault and the fault effect degree of the target fault; send the fault parameter to each of the target system devices to inject a target fault with a matching fault effect degree into each of the target system devices.

[0146] On the basis of the above embodiments, the fault response impact factor adjustment module 530 can be specifically used to: when it is determined that the actual fault response result falls within the control range of the expected fault response result, increase the proportion according to the preset effect degree to increase the fault effect degree of the target fault; when it is determined that the actual fault response result does not fall within the control range of the expected fault response result, reduce the proportion according to the preset effect degree to reduce the fault effect degree of the target fault.

[0147] On the basis of the above embodiments, the fault response influencing factor also includes: expected business traffic load level; the actual fault response result monitoring module 520 can be specifically used for: if it is determined that the target system is currently in a production environment, then obtaining the actual business traffic load level in the target system; if it is determined that the actual business traffic load level is less than the expected business traffic load level, then calculating the traffic value to be supplemented; generating simulated business traffic matching the traffic value to be supplemented, and injecting it into the target system.

[0148] On the basis of the above embodiments, the fault response impact factor adjustment module 530 can be specifically used to: when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, only the fault effect degree is matched and adjusted; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, only the expected business traffic load degree is matched and adjusted; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the matching adjustment is made in the order of first adjusting the fault effect degree and then adjusting the expected business traffic load degree; and when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the matching adjustment is made in the order of first adjusting the expected business traffic load degree and then adjusting the fault effect degree.

[0149] On the basis of the above embodiments, when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the expected business traffic load level is adjusted to match, which can be specifically used for: when it is determined that the actual fault response result falls within the control range of the expected fault response result, the expected business traffic load level is increased by a preset load level increase ratio; when it is determined that the actual fault response result does not fall within the control range of the expected fault response result, the expected business traffic load level is reduced by a preset load level decrease ratio.

[0150] On the basis of the above embodiments, the monitoring of the target system, the injection of target faults and the injection of simulated business traffic are all achieved through probes arranged on various system devices of the target system.

[0151] The above-mentioned chaos engineering-based fault drill device can execute the chaos engineering-based fault drill method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0152] Figure 6 FIG. 1 is a schematic diagram of the structure of a terminal device provided by another embodiment of the present invention. Figure 6 As shown, the device includes a processor 610, a memory 620, an input device 630, and an output device 640; the number of processors 610 in the device can be one or more. Figure 6 A processor 610 is taken as an example; the processor 610, memory 620, input device 630 and output device 640 in the device can be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0153] The memory 620, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the chaos engineering-based fault drill method in an embodiment of the present invention (for example, a chaos engineering test plan acquisition module 510, an actual fault response result monitoring module 520, a fault response influencing factor adjustment module 530 and a mapping relationship recording module 540). The processor 610 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 620, that is, implements the above-mentioned chaos engineering-based fault drill method, which includes: obtaining a chaos engineering test plan that matches the target system; injecting the target fault into the target system according to the fault response influence factor, and monitoring the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, adjusting the fault response influence factor to match; returning to execute the operation of injecting the target fault into the target system according to the fault response influence factor until the actual fault result matches the expected fault response result, and recording the mapping relationship between the current fault response influence factor and the expected fault response result.

[0154] The memory 620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 620 may further include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] The input device 630 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 640 may include a display device such as a display screen.

[0156] An embodiment of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to perform a chaos engineering-based fault drill method when executed by a computer processor, the method comprising: obtaining a chaos engineering test plan that matches the target system; injecting the target fault into the target system according to the fault response influence factor, and monitoring the actual fault response result of the target system to the target fault in real time; when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, adjusting the fault response influence factor to match; returning to execute the operation of injecting the target fault into the target system according to the fault response influence factor until the actual fault result matches the expected fault response result, and recording the mapping relationship between the current fault response influence factor and the expected fault response result.

[0157] Of course, the computer-readable storage medium provided in an embodiment of the present invention has computer-readable instructions that are not limited to the method operations described above, and can also execute related operations in the chaos engineering-based fault drill method provided in any embodiment of the present invention.

[0158] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0159] It is worth noting that in the above-mentioned embodiment of the chaos engineering-based fault drill device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0160] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A failure drill method based on chaos engineering, characterized in that: include: Obtain a chaos engineering test plan that matches the target system. The chaos engineering test plan includes: fault response impact factors and expected fault response results corresponding to the target fault to be injected; Injecting the target fault into the target system according to the fault response influencing factor, and monitoring the actual fault response result of the target system to the target fault in real time; When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match; Return to execute the operation of injecting the target fault into the target system according to the fault response influence factor, until the actual fault result matches the expected fault response result or the fault drill termination condition is met, and record the mapping relationship between the current fault response influence factor and the current actual fault response result; Wherein, the fault response influencing factors include: the fault effect degree of the target fault; Injecting the target fault into the target system according to the fault response influencing factor includes: According to the target fault and the fault effect degree of the target fault, forming a fault parameter directed to at least one target system device in the target system; Sending the fault parameter to each of the target system devices to inject a target fault matching the fault effect degree into each of the target system devices; The fault response influencing factors also include: expected service traffic load level; Injecting a target fault into a target system according to the fault response influencing factor also includes: If it is determined that the target system is currently in a production environment, obtaining an actual service traffic load level in the target system; If it is determined that the actual service traffic load level is less than the expected service traffic load level, calculating the traffic value to be supplemented; Generate simulated business traffic that matches the traffic value to be supplemented, and inject it into the target system.

2. The method according to claim 1, characterized in that When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match, including: When it is determined that the actual fault response result falls within the control range of the expected fault response result, the fault action degree of the target fault is increased according to a preset action degree increase ratio; When it is determined that the actual fault response result does not fall within the control range of the expected fault response result, the fault action degree of the target fault is reduced according to a preset action degree reduction ratio.

3. The method according to claim 1, characterized in that When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the fault response influencing factor is adjusted to match, including one of the following: When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, only the fault action degree is adjusted to match; When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, only the expected service traffic load degree is adjusted to match; When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the matching adjustment is performed in the order of first adjusting the fault effect degree and then adjusting the expected service traffic load degree; as well as When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, matching adjustment is performed in the order of first adjusting the expected service traffic load level and then adjusting the fault effect level.

4. The method according to claim 3, characterized in that When it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition, the expected service traffic load level is adjusted to match, including: When it is determined that the actual fault response result falls within the control range of the expected fault response result, increasing the expected service flow load level according to a preset load level increase ratio; When it is determined that the actual fault response result does not fall within the control range of the expected fault response result, the expected service flow load level is reduced according to a preset load level reduction ratio.

5. The method according to any one of claims 1 to 4, characterized in that: The monitoring of the target system, the injection of target faults and the injection of simulated business traffic are all achieved through probes set on various system devices of the target system.

6. A fault drill device based on chaos engineering, characterized in that: include: A chaos engineering test plan acquisition module is used to acquire a chaos engineering test plan that matches the target system. The chaos engineering test plan includes: a fault response impact factor and an expected fault response result corresponding to the target fault to be injected; An actual fault response result monitoring module, used to inject a target fault into a target system according to the fault response influencing factor, and monitor the actual fault response result of the target system to the target fault in real time; A fault response impact factor adjustment module, used to adjust the fault response impact factor to match when it is determined that the difference between the expected fault response result and the actual fault response result meets the dynamic adjustment condition; A mapping relationship recording module is used to return to execute the operation of injecting the target fault into the target system according to the fault response impact factor, until the actual fault result matches the expected fault response result or the fault drill end condition is met, and record the mapping relationship between the current fault response impact factor and the current actual fault response result; The fault response influencing factor includes: the fault effect degree of the target fault; an actual fault response result monitoring module, which is used to form a fault parameter pointing to at least one target system device in the target system according to the target fault and the fault effect degree of the target fault; and send the fault parameter to each of the target system devices to inject a target fault matching the fault effect degree into each of the target system devices; The fault response influencing factors also include: expected service traffic load level; The actual fault response result monitoring module is used to: If it is determined that the target system is currently in a production environment, obtaining an actual service traffic load level in the target system; If it is determined that the actual service traffic load level is less than the expected service traffic load level, calculating the traffic value to be supplemented; Generate simulated business traffic that matches the traffic value to be supplemented, and inject it into the target system.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the chaos engineering-based fault drill method as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a chaos engineering-based fault drill method as described in any one of claims 1 to 5 is implemented.

Images