Timeout mechanism verification method, device and medium based on chaos engineering

Abstract

This invention provides a method, apparatus, and medium for verifying timeout mechanisms based on chaos engineering, relating to the field of software testing. The method includes: acquiring network configuration information and fault configuration information of a target system, wherein the front-end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front-end and back-end of the target system; creating a chaos engineering event in the chaos engineering system based on the fault configuration information and network configuration information; triggering the chaos engineering event while the front-end of the target system is processing tasks normally; analyzing the target data to determine the event outcome of the chaos engineering event, wherein the target data consists of the runtime data generated by the target system before and after the chaos engineering event is triggered; and determining the operating status of the timeout mechanism by comparing the event outcome with the expected outcome. The solution of this invention can reduce the complexity and cost of timeout mechanism verification, and helps to improve the efficiency and accuracy of timeout mechanism verification.

Description

Technical Field

[0001] This invention relates to the field of software testing, and in particular to a timeout mechanism verification method, apparatus, and medium based on chaos engineering. Background Technology

[0002] A system's timeout mechanism is a crucial function for ensuring system reliability and stability, guaranteeing that applications or services can correctly handle requests in the face of latency or unresponsiveness. However, timeout mechanisms can fail both during system development and maintenance. Therefore, validating the timeout mechanism is essential to ensure system reliability and user experience.

[0003] Currently, the validity verification of existing timeout mechanisms is mainly achieved by manually entering commands into the target system. For example, entering `tc qdisc add dev eth0 root netem delay10000ms` in the target system's terminal operating system can simulate a network delay of 10 seconds. In this command, `tc` is a flow control tool in the terminal operating system used to control the working mode of `netem`, and `netem` is a network simulation function module provided by the terminal operating system kernel version. However, this method has the following drawbacks: First, as can be seen from the command input, this method requires a high level of coding ability from the verifier, resulting in low efficiency and a high error rate. Second, the verification process requires significant resource support, such as involving process approval and operation permission application issues, and the coordination costs across multiple departments and disciplines are high. Third, this verification method lacks a standard execution process, with chaotic steps or missing necessary steps, leading to inaccurate verification results and problems such as missed and false positives. Therefore, improvement is urgently needed. Summary of the Invention

[0004] This invention provides a timeout mechanism verification method, device, and medium based on chaos engineering, which solves the defects of high cost and low efficiency in the verification of timeout mechanisms in the prior art. By using chaos engineering events, it realizes the automatic injection of faults to create timeout trigger scenarios during system business processing, and automatically analyzes the event results and compares them with the expected results to verify whether the timeout mechanism is effective.

[0005] According to a first aspect of the present invention, the present invention provides a timeout mechanism verification method based on chaos engineering, the method comprising:

[0006] Obtain network configuration information and fault configuration information of the target system, wherein the front end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front end and back end of the target system;

[0007] Based on the fault configuration information and the network configuration information, a chaos engineering event is created in the chaos engineering system.

[0008] The chaos engineering event is triggered when the target system front-end is processing tasks normally.

[0009] The target data is analyzed to determine the event outcome of the chaotic engineering event, wherein the target data is the operational data generated by the target system before and after the chaotic engineering event is triggered;

[0010] The operational status of the timeout mechanism is determined by comparing the event outcome with the expected outcome.

[0011] According to the timeout mechanism verification method based on chaos engineering of the present invention, the step of obtaining the network configuration information and fault configuration information of the target system includes:

[0012] To obtain the network configuration information of the target system, follow these steps:

[0013] Install the Agent program on the backend of the target system;

[0014] The Agent program obtains the IP address and network card information of the target system.

[0015] Obtain the fault configuration information of the target system through the following steps:

[0016] A first fault scenario and a second fault scenario are set up, wherein the first fault scenario is used to simulate a fault between the front end and the back end, and the second fault scenario is used to simulate a fault between the back end and the database.

[0017] Perform the following operations on the first fault scenario and the second fault scenario respectively: obtain event parameters, the event parameters including event name, event duration, and event description; obtain fault event parameters, wherein the fault event parameters include network latency time, IP and network card information, and the network latency time is greater than the timeout mechanism trigger time.

[0018] According to the timeout mechanism verification method based on chaos engineering of the present invention, the step of creating a chaos engineering event in the chaos engineering system based on the fault configuration information and the network configuration information includes:

[0019] Perform the following operations on the first fault scenario and the second fault scenario respectively:

[0020] Based on the event description in the event parameters, search for the instruction in the fault instruction library of the chaos engineering system;

[0021] The found fault instructions are configured according to the event parameters and the fault event parameters to generate a chaotic engineering event consisting of the configured post-fault instructions.

[0022] According to the timeout mechanism verification method based on chaos engineering of the present invention, triggering the chaos engineering event when the target system front-end is processing the task normally includes:

[0023] Start the stress testing tool and use it to generate a preset stress task, which is then sent to the front end of the target system.

[0024] If the stress testing tool is detected to have run for a set time, the chaos engineering system is instructed to execute the chaos engineering event corresponding to either the first fault scenario or the second fault scenario.

[0025] If the end time of the chaotic engineering event corresponding to any of the fault scenarios reaches a set time, then the chaotic engineering system is instructed to execute the chaotic engineering event corresponding to another fault scenario in the first fault scenario and the second fault scenario.

[0026] If the time for the chaotic engineering event corresponding to another failure scenario to end reaches the set time, then stop generating the preset stress task and close the stress testing tool.

[0027] According to the timeout mechanism verification method based on chaos engineering of the present invention, the step of analyzing the target data to determine the event outcome of the chaos engineering event includes:

[0028] The system obtains the number of transactions per second of the target system before, during and after each chaotic engineering event, as well as the active status of the front-end threads, through a preset monitoring program.

[0029] Perform the following operations for each chaotic engineering event: determine the trend of the change in the transaction volume per second according to the order of before, during and after the execution of each chaotic engineering event, and determine whether there is a thread blocking situation in the monitored front-end thread activity state based on the preset thread blocking state.

[0030] The event results are obtained by summarizing the trends in transaction volume per second and thread blocking status corresponding to the first and second fault scenarios.

[0031] According to the timeout mechanism verification method based on chaos engineering of the present invention, the step of determining the operating state of the timeout mechanism by comparing the event result with the expected result includes:

[0032] If the change trend of the transaction processing volume per second corresponding to the first fault scenario and the second fault scenario conforms to the condition of first decreasing and then rising back to the state before the decrease, and there is no thread blocking situation, then the timeout mechanism is determined to be running normally.

[0033] If the trend of the transaction volume per second corresponding to the first fault scenario and / or the second fault scenario does not conform to the pattern of first decreasing and then rising back to the state before the decrease, or if thread blocking occurs in the first fault scenario and / or the second fault scenario, then the timeout mechanism is determined to be in an abnormal operating state.

[0034] According to the timeout mechanism verification method based on chaos engineering of the present invention, in the event that the timeout mechanism is in an abnormal operating state, the method further includes:

[0035] The target system after the timeout mechanism rectification is completed is taken as the new target system, and the step of triggering the chaos engineering event is returned to the execution of the task that is being processed normally at the front end of the target system.

[0036] According to a second aspect of the present invention, the present invention also provides a timeout mechanism verification device based on chaos engineering, the device comprising:

[0037] The acquisition module is used to acquire network configuration information and fault configuration information of the target system. The front end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front end and back end of the target system.

[0038] A creation module is used to create chaotic engineering events in the chaotic engineering system based on the fault configuration information and the network configuration information;

[0039] The triggering module is used to trigger the chaos engineering event when the target system front-end is processing tasks normally.

[0040] The analysis module is used to analyze the target data to determine the event outcome of the chaotic engineering event, wherein the target data is the operational data generated by the target system before and after the chaotic engineering event is triggered;

[0041] The comparison module is used to determine the operating status of the timeout mechanism by comparing the event result with the expected result.

[0042] According to a third aspect of the present invention, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the timeout mechanism verification method based on chaos engineering as described above.

[0043] According to a fourth aspect of the present invention, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the timeout mechanism verification method based on chaos engineering as described above.

[0044] This invention provides a timeout mechanism verification method based on chaos engineering. It utilizes the network configuration and fault configuration information of the target system to create chaotic engineering events in the chaotic engineering system. These events are triggered during normal operation of the target system, thus creating the conditions for the timeout mechanism to take effect. Next, the operating data generated by the target system before and after the chaotic engineering event is triggered is analyzed to obtain the event results. Finally, the event results are compared with the expected results to determine the operating status of the timeout mechanism. The entire verification process requires no multi-disciplinary collaboration, reducing the complexity of timeout mechanism verification. By automatically generating network faults using chaos engineering tools, it avoids manual input of any computer commands, program scripts, or modifications to program code. This not only reduces verification costs but also helps improve the efficiency and accuracy of timeout mechanism verification.

[0045] In addition, the timeout mechanism verification device based on chaos engineering, an electronic device, and a non-transitory computer-readable storage medium provided by the present invention can also achieve the above-mentioned technical effects, which will not be elaborated here. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is one of the flowcharts of the timeout mechanism verification method based on chaos engineering provided by the present invention.

[0048] Figure 2 This is a schematic diagram illustrating the principle of the application multi-server timeout mechanism verification provided by the present invention.

[0049] Figure 3 This is the second flowchart of the timeout mechanism verification method based on chaos engineering provided by this invention.

[0050] Figure 4 This is a schematic diagram of the timeout mechanism verification device based on chaos engineering provided by the present invention.

[0051] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] The following is combined with Figures 1 to 5 This invention describes a timeout mechanism verification method based on chaos engineering, a timeout mechanism verification device based on chaos engineering, an electronic device, and a non-transitory computer-readable storage medium.

[0054] Figure 1 This is one of the flowcharts illustrating the timeout mechanism verification method based on chaos engineering provided by this invention. Please refer to it. Figure 1 As shown, this embodiment provides a timeout mechanism verification method based on chaos engineering. This method can be implemented through steps S101 to S105, and each step will be described in detail below:

[0055] Step S101: Obtain the network configuration information and fault configuration information of the target system. The front end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front end and back end of the target system.

[0056] In this embodiment, the target system refers to the end-to-end system to be verified. This end-to-end system processes business through a front-end request-back-end architecture, such as an end-to-end system querying various data resources. Network configuration information includes, but is not limited to, the network configuration between the front-end and back-end, and the network configuration between the back-end and the data resources. Fault configuration information includes, but is not limited to, fault type, fault location, and fault time. Both fault configuration information and network configuration information can be obtained through specific commands.

[0057] In this embodiment, the timeout mechanism typically refers to a mechanism that sets a maximum waiting time limit for certain operations or requests in order to prevent the target system's backend resources from being occupied indefinitely. Once this time limit is reached, the target system will take some measures, such as canceling the request, releasing resources, or returning an error message. The present invention does not limit the specific implementation of the timeout mechanism.

[0058] Step S102: Based on the fault configuration information and network configuration information, create a chaotic engineering event in the chaotic engineering system.

[0059] In this embodiment, chaos engineering events refer to practical tasks that leverage chaos engineering to improve system resilience. Chaos engineering involves proactively injecting hardware or software faults into application systems to expose weaknesses in distributed systems and develop optimization strategies to continuously enhance the resilience of distributed systems. The principle of chaos engineering fault injection technology is as follows: First, an agent (with authorized system operation permissions) needs to be installed in the target application system. This agent is responsible for information exchange between the chaos engineering system and the target system. When the agent receives a specific fault injection instruction from the platform, it will execute the corresponding fault injection operation on its target system according to the instruction. Chaos engineering can simulate many types of faults, such as service interruption, process termination, network packet loss, and network latency. It should be noted that there can be multiple chaos engineering events, which are customized according to specific verification requirements and the network communication architecture of the target system during implementation.

[0060] Step S103: Trigger a chaos engineering event while the target system front-end is processing tasks normally.

[0061] In this embodiment, there are many ways to enable the front end of the target system to process tasks normally. For example, it can be achieved by controlling the load tool to generate tasks and sending them to the front end for execution. Alternatively, a large number of tasks can be collected in advance to form a task set, and then the tasks in the task set can be sent to the front end according to the set load pressure. For example, 200 tasks can be randomly selected from the task set every second and sent to the front end of the target system for processing.

[0062] Step S104: Analyze the target data to determine the event outcome of the chaotic engineering event. The target data refers to the operational data generated by the target system before and after the chaotic engineering event is triggered.

[0063] In this embodiment, there are many ways to obtain target data. For example, it can be obtained through interface listening by a preset monitoring program, or by collecting and parsing the target system's logs to find relevant data. It should be noted that the target system's operational data is continuously generated. Target data specifically refers to the target system's operational data before and after the chaos engineering event is triggered. This target data can characterize the target system's state before network faults are injected, after network faults are injected, and after the network faults are eliminated. The event results are obtained by analyzing the target data in a specific way.

[0064] Step S105: Determine the operating status of the timeout mechanism by comparing the event result with the expected result.

[0065] In this embodiment, the expected result is the expected state that the target system can achieve when the pre-set timeout mechanism is triggered and the system is processing business normally. The event result and the expected result can be compared by comparing whether the event result matches the expected result, and then the running state of the timeout mechanism can be identified based on whether the two results match.

[0066] This embodiment of the timeout mechanism verification method based on chaos engineering utilizes the network configuration information and fault configuration information of the target system to create a chaos engineering event in the chaos engineering system. The chaos engineering event is triggered during normal operation of the target system, thus creating the conditions for the timeout mechanism to take effect. Next, the running data generated by the target system before and after the chaos engineering event is triggered is analyzed to obtain the event result. Finally, the event result is compared with the expected result to obtain the running status of the timeout mechanism. The entire verification process does not require multi-disciplinary collaboration, reducing the complexity of timeout mechanism verification. By automatically generating network faults using chaos engineering tools, it avoids manual input of any computer commands, program scripts, or modifications to program code. This not only reduces verification costs but also helps improve the efficiency and accuracy of timeout mechanism verification.

[0067] In some possible implementations, step S101, obtaining the network configuration information and fault configuration information of the target system, specifically includes:

[0068] To obtain the network configuration information of the target system, follow these steps:

[0069] Install the Agent program on the backend of the target system;

[0070] Obtain the target system's IP address and network interface card information through an agent program;

[0071] Obtain the fault configuration information of the target system through the following steps:

[0072] Set up a first fault scenario and a second fault scenario, where the first fault scenario is used to simulate a fault between the front end and the back end, and the second fault scenario is used to simulate a fault between the back end and the database.

[0073] Perform the following operations for the first and second fault scenarios respectively: obtain event parameters, including event name, event duration, and event description; obtain fault event parameters, including network latency, IP address, and network card information, wherein the network latency is greater than the timeout mechanism trigger time.

[0074] It's important to note that setting the network latency greater than the timeout trigger time ensures the timeout mechanism is effective because it only triggers when the actual network latency exceeds the timeout threshold, thus terminating the resource allocation process and preventing process blocking and system crashes. Conversely, if the latency is less than the timeout threshold, it won't trigger, and the timeout mechanism's effectiveness cannot be verified.

[0075] The timeout mechanism verification method based on chaos engineering in this embodiment uses an agent program to obtain the network configuration information of the target system, which builds a communication bridge between the target system and the chaos engineering system, making it possible to automatically inject network faults into the target system. In addition, two fault scenarios are set to simulate network faults in the target system, making the fault types more diverse and improving the comprehensiveness and security of the verification.

[0076] In some possible implementations, step 102, based on fault configuration information and network configuration information, creates chaotic engineering events in the chaotic engineering system, specifically including:

[0077] Perform the following operations for the first and second fault scenarios respectively:

[0078] Search for instructions from the fault instruction library of the chaos engineering system based on the event description in the event parameters;

[0079] The found fault instructions are configured based on event parameters and fault event parameters to generate chaotic engineering events consisting of the configured post-fault instructions.

[0080] The timeout mechanism verification method based on chaos engineering in this embodiment generates corresponding chaos engineering events for each fault scenario based on event parameters and fault event parameters. It realizes the automatic generation of fault events by using the fault instruction library and configured fault information of the chaos engineering system. The generation of fault events is relatively flexible and only requires simple parameter configuration, which significantly reduces the business capability requirements of verification personnel.

[0081] In some possible implementations, step S103, which triggers a chaos engineering event while the target system front-end is processing tasks normally, specifically includes:

[0082] Start the stress testing tool and use it to generate a preset stress task, which is then sent to the front end of the target system.

[0083] If the stress testing tool is detected to have run for a set time, the chaos engineering system will execute the chaos engineering event corresponding to either the first or second failure scenario.

[0084] If the end time of the chaotic engineering event corresponding to any fault scenario reaches the set time, then the chaotic engineering system shall execute the chaotic engineering event corresponding to the other fault scenario in the first fault scenario and the second fault scenario.

[0085] If the time for the chaotic engineering event corresponding to another failure scenario to end reaches the set time, then stop generating the preset stress task and close the stress testing tool.

[0086] In this embodiment, the triggering order of the two fault scenarios can be pre-set. For example, the chaotic engineering event corresponding to the first fault scenario can be triggered first, and the chaotic engineering event corresponding to the second fault scenario can be triggered after the system has been running stably for a certain period of time after the event ends. Usually, the target system can be adjusted to a relatively stable state in about ten minutes.

[0087] The timeout mechanism verification method based on chaos engineering in this embodiment uses a stress testing tool to send tasks to the target system to simulate the normal business processing process. By detecting multiple specific time points, the chaotic engineering events corresponding to the fault scenario can be triggered sequentially during the normal business processing. At the same time, an interval time is set between two different events to ensure that different chaotic engineering events do not interfere with each other. The whole process does not require manual intervention and has a high degree of automation.

[0088] In some possible implementations, step S104, which involves analyzing the target data to determine the event outcome of the chaotic engineering event, specifically includes:

[0089] The system obtains the number of transactions per second of the target system before, during and after each chaotic engineering event, and monitors the activity status of the front-end threads by using a pre-set monitoring program.

[0090] Perform the following operations for each chaotic engineering event: determine the trend of the change in the transaction volume per second according to the order of before, during and after the execution of each chaotic engineering event, and determine whether there is a thread blocking situation in the monitored front-end thread activity state based on the preset thread blocking state.

[0091] The event results are obtained by summarizing the trends of transaction volume per second and thread blocking status for the first and second failure scenarios.

[0092] The timeout mechanism verification method based on chaos engineering in this embodiment monitors the thread status and business processing volume of the target system through a pre-deployed monitoring program. It analyzes the changing trend and blocking situation of business processing volume in units of chaos engineering events to obtain event results. These event results greatly simplify the comparison of data and save computing resources.

[0093] In some possible implementations, the preceding step S105 determines the running status of the timeout mechanism by comparing the event result with the expected result, specifically including:

[0094] If the trend of the transaction processing volume per second corresponding to the first and second failure scenarios is consistent with the pattern of first decreasing and then rising back to the state before the decrease, and there is no thread blocking in either case, then the timeout mechanism is judged to be running normally.

[0095] If the trend of the transaction volume per second corresponding to the first fault scenario and / or the second fault scenario does not conform to the pattern of first decreasing and then rising back to the state before the decrease, or if thread blocking occurs in the first fault scenario and / or the second fault scenario, then the timeout mechanism is determined to be in an abnormal operating state.

[0096] The timeout mechanism verification method based on chaos engineering in this embodiment sets expected results by combining two aspects: the change in the number of transactions per second and the thread blocking situation. It also provides a verification standard for whether the timeout mechanism is effectively triggered by comparing the expected results with the event results. It can automatically complete the verification of the effectiveness of the timeout mechanism without manual intervention and has high accuracy.

[0097] In some possible implementations, the method further includes the following in case the timeout mechanism malfunctions:

[0098] The target system after the timeout mechanism rectification is taken as the new target system, and the aforementioned step S103 is returned to be executed, that is, wait for the new target system's front-end to process the task normally and trigger the chaos engineering event again, so as to re-verify the new target system.

[0099] The timeout mechanism verification method based on chaos engineering in this embodiment addresses the situation where the timeout mechanism is not functioning properly. It achieves re-verification by injecting faults after rectification and returning to normal business operations. The results of this re-verification can assist in evaluating the rectification results and help improve the efficiency of timeout mechanism rectification.

[0100] In yet another embodiment, for ease of understanding of the present invention, the following is used... Figure 2 The method of the present invention is explained in detail using a multi-server architecture as an example. Specifically, three servers are pre-deployed, denoted as server A, server B, and server C. Server A deploys a chaos engineering system, server B deploys a monitoring system, and server C deploys a stress testing tool. This embodiment provides a timeout mechanism verification method based on chaos engineering applied to this architecture. Please refer to [reference needed]. Figure 3 As shown, the implementation process of this method is as follows:

[0101] Step S301: Determine the target system. The target system for verifying the effectiveness of the timeout mechanism is determined. The chaos engineering system obtains basic information about the target system through the agent program, including the IP addresses and network interface card information of the system modules.

[0102] Step S302: Determine the expected outcome of the event. The event requires fault injection into the backend system of the target system using the chaos engineering system. Expected outcome: After the chaos engineering system initiates the fault injection command, the timeout mechanism of the frontend system of the system receiving the fault injection will trigger a timeout handling mechanism after reaching a threshold. At this time, the overall system TPS (transactions per second) obtained by the monitoring system will decrease compared to before the fault injection, and no thread blocking information will be obtained during this period. After the chaos engineering event ends, the fault scenario is resolved, the backend system network returns to normal, and requests received by the frontend can be processed and returned normally by the backend. The overall system TPS returns to the level before the fault injection.

[0103] Step S303: Create a chaotic engineering fault scenario. This process includes: (1) Deploying a chaotic engineering system on server A, the chaotic engineering system receives parameters such as "event name, event duration, and event description"; (2) Obtaining the IP of the target event machine; (3) The chaotic engineering system receives fault events, such as event type and network latency parameters, and receives various parameter configurations, such as event start time, end time, event runtime, network card information, and specific configuration parameters of the event type; (4) The event case is set up and ready to be executed.

[0104] It should be noted that step S303 involves the creation of two chaotic engineering fault scenarios. These two scenarios simulate network latency fault events between the system front-end and back-end applications, and between the back-end application and the database, respectively. The only difference lies in configuring the fault event parameters, such as the target system IP and network card information; the rest of the creation process is identical. Furthermore, in step S303, when setting the specific configuration parameters for the event type, the actual configured network latency time value should be greater than the time threshold of the front-end timeout mechanism. For example, if the front-end timeout mechanism trigger threshold is 10 seconds, then when setting the network latency fault scenario, the network latency duration parameter can be set to 11 seconds.

[0105] Step S304: Pre-set the system monitoring system. The monitoring system is used to observe and verify the system status. Deploy the monitoring system on server B. The monitoring system (such as APM, ZABBIX, Prometheus, Grafana, etc.) receives various events related to the business processing metrics (TPS: transactions per second) of the target system. Simultaneously, it monitors application process blocking phenomena. For example, for Java applications, a JVM monitoring tool (JConsole, VisualVM, or JavaMission Control, etc.) can be pre-set to monitor the running status of Java applications. The JVM monitoring tool can obtain information such as thread status, deadlock, thread stack, and thread blocking.

[0106] Step S305: Pre-set up a stress testing scenario to simulate business processing in a production environment. Deploy stress testing tools (LoadRunner, JMeter, APTS, etc.) on server C to receive and send task instructions, apply load pressure to the front-end system of the target system, simulate concurrent access to the front-end system, and simulate multiple users simultaneously making requests to the back-end system according to different access actions (batch registration or login, query, modification, etc.) in a certain proportion (refer to the actual proportion in the production environment).

[0107] S306: Executing a chaos engineering fault scenario under background pressure. After receiving the task sending instruction, the stress testing tool executes the preset stress scenario to send tasks to the target server and runs stably for 10 minutes. During this time, the monitoring system obtains the average total TPS of the target system. Then, the chaos engineering system triggers a fault injection instruction, executing chaos engineering event cases in the backend system application cluster. The underlying chaos tool and the Agent program pre-installed on the target system execute chaos engineering events to inject faults into the target system. During this period, the stress scenario continues uninterrupted, and the stress testing tool continuously sends transaction requests to the target system cluster.

[0108] Step S307: Obtain the phenomena and data of the target system under timeout conditions. Obtain the timeout handling status of the front-end under the fault scenario. After the event fault injection, run again for 10 minutes. The monitoring system obtains the timeout handling status of the front-end system of the fault-injected system module during this period, and whether there are any thread blocking alarms. At the same time, obtain the total TPS data of the target system as a whole during this step.

[0109] Step S308: End the chaos engineering event. The chaos engineering system triggers the command to remove fault injection (in the network latency fault scenario, fault injection will be automatically removed and the target system service will be restored after the chaos engineering event case is stopped). The stress scenario continues to run for another 10 minutes, and the monitoring system obtains the total TPS data of the overall system.

[0110] Step S309: Complete all event logging, recording business data and monitoring information during the event.

[0111] Step S310: Determine whether all fault scenarios have been completed. If not, repeat steps S306 to S309 until all created chaotic engineering fault scenarios are completed. This implementation includes the execution of two fault scenarios: simulating network latency faults between the system front-end and back-end applications, and simulating network latency faults between the back-end application and the database.

[0112] Step S311: Determine whether the event results are consistent with the expected results. The chaos engineering system reads data from the monitoring system, compares and calculates the data acquired during the event, and determines whether the results of the two events are consistent with the expected results in step S302.

[0113] Step S312: If the results are consistent, it indicates that the event outcome meets the expected outcome, thus determining that the system's end-to-end timeout mechanism is effective, meaning the timeout mechanism is operating normally.

[0114] If there is no consistency in step S313, and any phenomenon or result that is inconsistent with step S302 occurs, then the end-to-end timeout mechanism of the system is determined to be invalid, that is, the timeout mechanism is in an abnormal operating state, and the system timeout mechanism needs to be rectified.

[0115] The timeout mechanism verification method based on chaos engineering in this embodiment can create fault events by using a chaos engineering system based on the target system's network configuration and fault event parameters. It can simulate the normal business processing of the target system with the help of stress tools and inject network latency faults with chaos engineering events to verify the effectiveness of system timeouts. It is suitable for various end-to-end systems and has good versatility.

[0116] The timeout mechanism verification device based on chaos engineering provided by the present invention will be described below. The timeout mechanism verification device based on chaos engineering described below and the timeout mechanism verification method based on chaos engineering described above can be referred to in correspondence.

[0117] Please refer to Figure 4 As shown, this embodiment provides a timeout mechanism verification device based on chaos engineering. Specifically, the device includes: an acquisition module 401, a creation module 402, a triggering module 403, an analysis module 404, and a comparison module 405. The above modules will be described in detail below:

[0118] The acquisition module 401 is used to acquire network configuration information and fault configuration information of the target system. The front end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front end and back end of the target system.

[0119] Create module 402 to create chaotic engineering events in the chaotic engineering system based on fault configuration information and network configuration information;

[0120] Trigger module 403 is used to trigger a chaos engineering event when the target system front-end is processing tasks normally.

[0121] Analysis module 404 is used to analyze the target data to determine the event outcome of the chaotic engineering event, wherein the target data is the running data generated by the target system before and after the chaotic engineering event is triggered;

[0122] The comparison module 405 is used to determine the running status of the timeout mechanism by comparing the event result with the expected result.

[0123] The timeout mechanism verification device based on chaos engineering in this embodiment creates chaotic engineering events in the chaotic engineering system using the network configuration information and fault configuration information of the target system. When the target system is performing normal tasks, the chaotic engineering events are triggered, thereby creating the conditions for the timeout mechanism of the target system to take effect. Then, the running data generated by the target system before and after the triggering of the chaotic engineering events are analyzed to obtain the event results. Finally, the event results are compared with the expected results to obtain the running status of the timeout mechanism. The entire verification process does not require multi-disciplinary cooperation, reducing the complexity of timeout mechanism verification. By automatically generating network faults with the help of chaos engineering tools, it avoids manual input of any computer instructions, program scripts or modification of program code, which not only reduces verification costs, but also helps to improve the efficiency and accuracy of timeout mechanism verification.

[0124] It should be noted that each module in the aforementioned timeout mechanism verification device based on chaos engineering can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independently of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0125] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a timeout mechanism verification method based on chaos engineering. This method includes: acquiring network configuration information and fault configuration information of the target system, wherein the front end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front end and back end of the target system; creating a chaos engineering event in the chaos engineering system based on the fault configuration information and the network configuration information; triggering the chaos engineering event when the front end of the target system is processing tasks normally; analyzing target data to determine the event result of the chaos engineering event, wherein the target data is the running data generated by the target system before and after the chaos engineering event is triggered; and determining the operating state of the timeout mechanism by comparing the event result with the expected result.

[0126] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the timeout mechanism verification method based on chaos engineering provided by the above methods. The method includes: acquiring network configuration information and fault configuration information of a target system, wherein the front end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front end and back end of the target system; creating a chaos engineering event in a chaos engineering system based on the fault configuration information and the network configuration information; triggering the chaos engineering event when the front end of the target system is processing tasks normally; analyzing target data to determine the event result of the chaos engineering event, wherein the target data is the running data generated by the target system before and after the chaos engineering event is triggered; and determining the running state of the timeout mechanism by comparing the event result with the expected result.

[0128] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the timeout mechanism verification method based on chaos engineering provided by the above methods. This method includes: acquiring network configuration information and fault configuration information of a target system, wherein the front end of the target system has a timeout mechanism, and the fault configuration information is used to cause network faults in the front end and back end of the target system; creating a chaos engineering event in a chaos engineering system based on the fault configuration information and the network configuration information; triggering the chaos engineering event when the front end of the target system is processing tasks normally; analyzing target data to determine the event result of the chaos engineering event, wherein the target data is the running data generated by the target system before and after the chaos engineering event is triggered; and determining the operating state of the timeout mechanism by comparing the event result with the expected result.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A timeout mechanism verification method based on chaos engineering, characterized in that, The method includes: The network configuration information and fault configuration information of the target system are obtained. The target system has a timeout mechanism in its front end, and the fault configuration information is used to cause network faults in both the front end and back end of the target system. The fault configuration information includes fault event parameters, which include event types. During the setting of specific configuration parameters for the event types, the actual configured network latency value is greater than the time threshold of the front end timeout mechanism. The timeout mechanism refers to the target system taking measures when an operation or request triggered by a chaos engineering event exceeds the maximum waiting time limit. Based on the fault configuration information and the network configuration information, a chaos engineering event is created in the chaos engineering system. The chaos engineering event is triggered when the target system front-end is processing tasks normally. The target data is analyzed to determine the event outcome of the chaotic engineering event, wherein the target data is the operational data generated by the target system before and after the chaotic engineering event is triggered; The operational status of the timeout mechanism is determined by comparing the event outcome with the expected outcome. The acquisition of network configuration information and fault configuration information of the target system includes: Obtain the fault configuration information of the target system through the following steps: A first fault scenario and a second fault scenario are set up, wherein the first fault scenario is used to simulate a fault between the front end and the back end, and the second fault scenario is used to simulate a fault between the back end and the database. Perform the following operations on the first fault scenario and the second fault scenario respectively: obtain event parameters and fault event parameters.

2. The timeout mechanism verification method based on chaos engineering according to claim 1, characterized in that, The acquisition of network configuration information and fault configuration information of the target system includes: To obtain the network configuration information of the target system, follow these steps: Install the Agent program on the backend of the target system; The Agent program obtains the IP address and network card information of the target system. The event parameters include the event name, event duration, and event description; the fault event parameters include network latency, IP address, and network card information, wherein the network latency is greater than the timeout mechanism trigger time.

3. The timeout mechanism verification method based on chaos engineering according to claim 2, characterized in that, The creation of chaotic engineering events in the chaotic engineering system based on the fault configuration information and the network configuration information includes: Perform the following operations on the first fault scenario and the second fault scenario respectively: Based on the event description in the event parameters, search for the instruction in the fault instruction library of the chaos engineering system; The found fault instructions are configured according to the event parameters and the fault event parameters to generate a chaotic engineering event consisting of the configured post-fault instructions.

4. The timeout mechanism verification method based on chaos engineering according to claim 3, characterized in that, When the target system is processing tasks normally at the front end, triggering the chaos engineering event includes: Start the stress testing tool and use it to generate a preset stress task, which is then sent to the front end of the target system. If the stress testing tool is detected to have run for a set time, the chaos engineering system is instructed to execute the chaos engineering event corresponding to either the first fault scenario or the second fault scenario. If the end time of the chaotic engineering event corresponding to any of the fault scenarios reaches a set time, then the chaotic engineering system is instructed to execute the chaotic engineering event corresponding to another fault scenario in the first fault scenario and the second fault scenario. If the time for the chaotic engineering event corresponding to another failure scenario to end reaches the set time, then stop generating the preset stress task and close the stress testing tool.

5. The timeout mechanism verification method based on chaos engineering according to claim 3, characterized in that, The analysis of the target data to determine the event outcome of the chaotic engineering event includes: The system obtains the number of transactions per second of the target system before, during and after each chaotic engineering event, as well as the active status of the front-end threads, through a preset monitoring program. Perform the following operations for each chaotic engineering event: determine the trend of the change in the transaction volume per second according to the order of before, during and after the execution of each chaotic engineering event, and determine whether there is a thread blocking situation in the monitored front-end thread activity state based on the preset thread blocking state. The event results are obtained by summarizing the trends in transaction volume per second and thread blocking status corresponding to the first and second fault scenarios.

6. The timeout mechanism verification method based on chaos engineering according to claim 5, characterized in that, The step of determining the operating status of the timeout mechanism by comparing the event outcome with the expected outcome includes: If the change trend of the transaction processing volume per second corresponding to the first fault scenario and the second fault scenario conforms to the condition of first decreasing and then rising back to the state before the decrease, and there is no thread blocking situation, then the timeout mechanism is determined to be running normally. If the trend of the transaction volume per second corresponding to the first fault scenario and / or the second fault scenario does not conform to the pattern of first decreasing and then rising back to the state before the decrease, or if thread blocking occurs in the first fault scenario and / or the second fault scenario, then the timeout mechanism is determined to be in an abnormal operating state.

7. The timeout mechanism verification method based on chaos engineering according to claim 6, characterized in that, In the event of an abnormal operation of the timeout mechanism, the method further includes: The target system after the timeout mechanism rectification is completed is taken as the new target system, and the step of triggering the chaos engineering event is returned to the execution of the task that is being processed normally at the front end of the target system.

8. A timeout mechanism verification device based on chaos engineering, characterized in that, The device includes: The acquisition module is used to acquire network configuration information and fault configuration information of the target system. The target system has a timeout mechanism in its front end, and the fault configuration information is used to cause network faults in both the front end and back end of the target system. The fault configuration information includes fault event parameters, which include event types. During the setting of specific configuration parameters for the event types, the actual configured network latency value is greater than the time threshold of the front end timeout mechanism. The timeout mechanism refers to the target system taking measures when an operation or request triggered by a chaos engineering event exceeds the maximum waiting time limit. A creation module is used to create chaotic engineering events in the chaotic engineering system based on the fault configuration information and the network configuration information; The triggering module is used to trigger the chaos engineering event when the target system front-end is processing tasks normally. The analysis module is used to analyze the target data to determine the event outcome of the chaotic engineering event, wherein the target data is the operational data generated by the target system before and after the chaotic engineering event is triggered; The comparison module is used to determine the operating status of the timeout mechanism by comparing the event result with the expected result; The acquisition of network configuration information and fault configuration information of the target system includes: Obtain the fault configuration information of the target system through the following steps: A first fault scenario and a second fault scenario are set up, wherein the first fault scenario is used to simulate a fault between the front end and the back end, and the second fault scenario is used to simulate a fault between the back end and the database. Perform the following operations on the first fault scenario and the second fault scenario respectively: obtain event parameters and fault event parameters.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the timeout mechanism verification method based on chaos engineering as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the timeout mechanism verification method based on chaos engineering as described in any one of claims 1 to 7.

Images