Automatic chaos test method, device and equipment of distributed system and medium

Abstract

The invention relates to the technical field of system testing, and discloses an automatic chaos testing method, device and equipment of a distributed system and a medium. The method comprises the steps that interface parameters are input through a target interface in an interface layer; interfaces of various chaotic test scenes are preset in the interface layer; the interface is provided with a corresponding execution file and a corresponding test suite; a corresponding execution script is arranged in the execution file and is used for calling the test suite; the interface parameters comprise a target to-be-tested object, identification information of a target execution file and information of a fault event; receiving and sending the interface parameters to the target execution file through the execution layer; calling the target test suite through the server to execute the target test suite on the target to-be-tested object to obtain a chaos test execution result; and receiving a returned result through the interface layer. According to the method, the test suite is distributed to the specified to-be-tested object for operation in a mode of calling the interface, so that batch and automatic testing can be realized, and the testing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of system testing, and in particular to an automated chaos testing method, device, equipment and medium for a distributed system. Background Art

[0002] In today's era, data and information are growing explosively, and major manufacturers have adopted highly available and highly complex distributed system solutions. Distributed systems can effectively handle large-scale data and business, and improve the scalability and reliability of the system. However, its high complexity also brings many challenges. During the online operation, various unpredictable sudden failures often occur. These failures are of various types, such as sudden downtime of services or machines, abnormal network connections, and forced exit of programs due to resource overload. Once such failures occur, they often cause immeasurable consequences to enterprises, such as business interruption, economic losses, and reputation damage. At the same time, they also greatly affect user experience and reduce users' trust in enterprise products and services. In order to cope with the risk of failure in distributed systems, chaos engineering catalog technology has gradually been introduced by major companies. In the product development stage and before the official launch, enterprises use chaos engineering catalog technology to simulate various common failure scenarios in distributed systems. In this way, potential problems are discovered in advance, thereby effectively avoiding failures from being left in the online production environment. In terms of traditional technologies for creating chaos experiments, there are currently some single-point failure simulation tools available on the market, among which Alibaba's ChaosBlade is more commonly used. Its operation is relatively simple. You only need to copy the toolkit to the server and enter the corresponding parameters during runtime to simulate failures at different levels, including but not limited to operating system level, application level, network level and other failure types, providing certain convenience and feasibility for failure simulation.

[0003] Although single-point fault simulation tools such as ChaosBlade have met some fault simulation needs to a certain extent, they still have serious limitations when facing large-scale distributed systems. General distributed systems are large in scale and complex in structure. Even for a cluster of only 50 machines, if the toolkit is copied to each machine one by one and simulated, it will take a lot of time and manpower. In actual enterprise application scenarios, the scale of distributed system clusters often reaches thousands or even more. In this case, using traditional single-point fault simulation tools to perform fault simulation becomes an almost impossible and arduous task, which seriously affects the efficiency of fault simulation.

[0004] Therefore, there is an urgent need for an automated chaos testing method that can quickly improve the efficiency of chaos testing. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem in the related art that a method of copying toolkits to each machine one by one and executing simulated failures is adopted, which consumes a lot of time, labor costs and has low efficiency.

[0006] In order to solve the above technical problems, in a first aspect, the present invention provides an automated chaos testing method for a distributed system, wherein the distributed system includes multiple groups of objects to be tested, and the automated chaos testing method for the distributed system includes:

[0007] Input interface parameters through the target interface in the interface layer;

[0008] The interface layer is pre-set with interfaces for multiple chaos test scenarios; the interface for each chaos test scenario is pre-set with a corresponding execution file and a corresponding test suite; the execution file is provided with a corresponding execution script, and the execution script is used to call the test suite, and the test suite includes an automated test script or a test tool; the target interface is the interface of the target chaos test scenario, and the interface parameters include identification information of the target object to be tested, identification information of the target execution file, and information of the fault event;

[0009] Receiving input interface parameters through the execution layer, and sending the interface parameters to the target execution file;

[0010] Based on the execution script corresponding to the target execution file, the target test suite is called through the server to execute the target test suite on the target object to be tested, and a chaos test execution result is obtained;

[0011] The chaotic test execution result is received through the interface layer.

[0012] In an optional implementation, before calling the target test suite through the server, it also includes:

[0013] The interface parameters are input into the target test suite as variables.

[0014] In an optional implementation manner, before the interface parameter is input as a variable into the target test suite, the method further includes:

[0015] Determining whether the target test suite exists in the server;

[0016] When the target test suite does not exist in the server, the target test suite is uploaded to the server.

[0017] In an optional implementation manner, before inputting the interface parameters through the target interface in the interface layer, the method further includes:

[0018] Pre-setting interfaces for multiple chaos test scenarios in the interface layer;

[0019] Pre-set the corresponding execution file and the corresponding test suite in the interface of each chaos test scenario;

[0020] An execution script is pre-set in the execution file.

[0021] In an optional embodiment, the method further includes:

[0022] Configure the component verification tool set directory and chaos engineering directory at the tool layer;

[0023] The component verification tool set directory includes automated test scripts corresponding to each chaos test scenario, and the chaos engineering directory includes test tools corresponding to each chaos test scenario.

[0024] In an optional implementation, a service test interface is pre-set in the interface layer, and after receiving the chaotic test execution result returned through the interface layer, the method further includes:

[0025] When the chaos test execution result is that the chaos test execution is successful, a business test is performed based on the business test interface to obtain a business test execution result.

[0026] In an optional implementation, when the chaos test execution result is that the chaos test execution is successful, performing a service test based on the service test interface to obtain the service test execution result includes:

[0027] When the chaos test execution result is that the chaos test execution is successful, a business test command sent by the server is received, the business test interface is called based on the business test command, and a business test is performed based on a business test script pre-set by the business test interface to obtain a business test execution result.

[0028] In a second aspect, the present invention provides an automated chaos testing device for a distributed system, wherein the distributed system includes multiple groups of objects to be tested, and the automated chaos testing device for the distributed system includes:

[0029] A first processing module, used for inputting interface parameters through a target interface in the interface layer;

[0030] The interface layer is pre-set with interfaces for multiple chaos test scenarios; the interface for each chaos test scenario is pre-set with a corresponding execution file and a corresponding test suite; the execution file is provided with a corresponding execution script, and the execution script is used to call the test suite, and the test suite includes an automated test script or a test tool; the target interface is the interface of the target chaos test scenario, and the interface parameters include identification information of the target object to be tested, identification information of the target execution file, and information of the fault event;

[0031] A second processing module, configured to receive input interface parameters through the execution layer, and send the interface parameters to the target execution file;

[0032] A third processing module is used to call a target test suite through a server based on an execution script corresponding to the target execution file, so as to execute the target test suite on the target object to be tested and obtain a chaos test execution result;

[0033] The fourth processing module is used to receive the chaotic test execution result returned through the interface layer. In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the automated chaos testing method for a distributed system of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a single computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the automated chaos testing method for a distributed system of the above-mentioned first aspect or any corresponding embodiment thereof.

[0035] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the automated chaos testing method for a distributed system of the above-mentioned first aspect or any corresponding embodiment thereof.

[0036] The technical solution provided by the present invention has the following technical effects:

[0037] The technical solution of the embodiment of the present invention pre-sets interfaces of multiple chaos test scenarios through the interface layer. The tester only needs to input interface parameters including identification information of the target object to be tested, identification information of the target execution file and information of the fault event in the target interface, and the corresponding chaos test can be started without complicated configuration operations. This greatly simplifies the test process, reduces manual intervention and cumbersome preparation work, improves the efficiency and convenience of test startup, and enables chaos testing to be carried out more quickly in distributed systems. Since interfaces of multiple chaos test scenarios are pre-set in the interface layer, and each interface corresponds to a specific execution file and test suite, it can cover different types of fault simulation requirements, such as network failure, service downtime, resource exhaustion, etc. The tester can flexibly select the target chaos test scenario according to different test purposes and the characteristics of the distributed system, and perform targeted tests on specific objects to be tested, thereby comprehensively detecting the performance of the distributed system under various fault conditions, and enhancing the comprehensiveness and effectiveness of the test. After receiving the interface parameters, the execution layer automatically calls the target test suite according to the execution script in the target execution file to perform the test on the target object to be tested, and returns the chaos test execution result through the interface layer. The whole process is automated, and there is no need to manually deploy and execute test tools or scripts on each group of target objects to be tested. It is especially suitable for testing scenarios with many objects to be tested in large-scale distributed systems. At the same time, this architectural design has good scalability. When a new chaos test scenario or object to be tested is added to the distributed system, it can be easily incorporated into the automated chaos test system by simply adding the corresponding interface configuration and execution files in the interface layer, without causing large-scale changes and impacts on the entire test framework. Finally, the chaos test execution results are received and centrally managed through the interface layer, which facilitates testers to uniformly collect, organize and analyze the test results. It can timely discover potential problems and weak links in the distributed system, provide a strong basis for subsequent system optimization and fault repair, help improve the stability and reliability of the distributed system, and reduce the system downtime and business loss risks caused by faults. Since automated testing can be easily performed on multiple groups of target test objects and multiple fault events, compared with traditional manual or limited testing methods, it can perform more extensive and in-depth testing on distributed systems in the same amount of time, thereby improving test efficiency and test coverage, reducing potential fault hazards that are missed due to insufficient testing, and improving the ability of distributed systems to cope with various complex situations, ensuring their stable operation in a production environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of an automated chaos testing method for a distributed system according to an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of an automated chaos testing framework for a distributed system according to an embodiment of the present invention;

[0041] Figure 3 It is a flowchart of the preliminary preparation stage of an embodiment of the present invention;

[0042] Figure 4 It is a structural schematic diagram of an automated chaos testing device for a distributed system according to an embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] In the implementation of traditional chaos experiments, there are many key issues that need to be solved, as follows:

[0046] When simulating single-point failures to accurately replicate the failure state of the production environment, there are common tools such as Alibaba's Chaos Blade. You only need to copy the toolkit to the server and enter the corresponding parameters during operation to carry out fault simulations at different levels. However, given that distributed systems are usually highly complex and large in scale, taking a cluster of only 50 machines as an example, if you copy the toolkit to each machine one by one to perform simulated failures, it will consume a lot of time and energy. If you are facing a cluster of thousands of machines, this traditional method is almost impossible to achieve the task objectives, and its low efficiency seriously restricts the effective implementation of chaos experiments in large-scale distributed systems.

[0047] After simulating a failure, verifying the operating status of the affected business modules and even the entire business system is an indispensable step. In addition, with the continuous version update and iteration of the system, the impact of chaos failures on the business needs to be re-evaluated each time. If this verification work is completed manually every time a version is updated, it will inevitably cause a huge waste of manpower and resource costs. Although there are some chaos tools on the market, such as ChaosMesh and ChaosbladeBox, which support fault orchestration functions, they have not successfully achieved the organic integration of fault simulation and automated business verification, resulting in the inability to accurately determine which fault point caused the business abnormality, which greatly limits the role of chaos experiments in ensuring business stability.

[0048] The embodiments of the present invention provide a method, device, equipment and medium for automated chaos testing of a distributed system to solve the above problems.

[0049] According to an embodiment of the present invention, an embodiment of an automated chaos testing method for a distributed system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] Figure 1 It is a flow chart of an automated chaos testing method for a distributed system according to an embodiment of the present invention.

[0051] like Figure 1 As shown, an automated chaos testing method for a distributed system is provided in an embodiment of the present invention, wherein the distributed system includes multiple groups of objects to be tested, each group of objects to be tested includes at least one machine to be tested, or a physical machine or a virtual machine, and the distributed system can call the Ansible module through the FastAPI framework to classify and group the machines to be tested in the distributed system cluster for secret-free management, thereby obtaining multiple groups of objects to be tested. The technical solution of the automated chaos testing method for a distributed system of the present invention is executed in parallel for each group of objects to be tested. In order to more clearly describe the technical solution of the present invention, the technical solution of the present invention is explained below by taking a group of objects to be tested (target objects to be tested) as an example. It should be understood that in the present invention, when performing automated chaos testing on a distributed system, each group of objects to be tested in the distributed system can be subjected to automated chaos testing according to the following technical solution of the present invention.

[0052] Taking a group of objects to be tested (target objects to be tested) as an example, the automated chaos testing method of the distributed system includes:

[0053] S101: Input interface parameters through the target interface in the interface layer.

[0054] In this embodiment, the interface layer is pre-set with interfaces for multiple chaos test scenarios. The interface of the chaos test scenario can be named as a chaos experiment trigger interface, and the interface of each chaos test scenario is pre-set with a corresponding execution file and a corresponding test suite. The execution file is provided with a corresponding execution script, and the execution script is used to call the test suite, and the test suite includes an automated test script or a test tool. The target interface is the interface of the target chaos test scenario, and the interface parameters include the identification information of the target object to be tested, the identification information of the target execution file, and the information of the fault event. The target interface is pre-set with a corresponding target execution file and a corresponding target test suite. The target execution file is provided with a corresponding execution script, and the execution script is used to call the target test suite, and the target test suite includes a target automated test script or a target test tool.

[0055] In this embodiment, the identification information of the target object to be tested may be a group name, a group number, the identification information (IP) of each machine to be tested in the target object to be tested, etc. The identification information of the target executable file may be the file name of the target executable file, etc. The information of the fault event is the specific information of the chaos experiment.

[0056] The fault event information refers to the detailed description of the chaotic fault to be simulated, including but not limited to the following aspects:

[0057] Fault type information: for example, network faults (such as network delays, network interruptions, packet loss, etc.), resource faults (such as memory leaks, excessive CPU usage, insufficient disk space, etc.), application faults (such as process crashes, thread deadlocks, service unavailability, etc.), database faults (such as database connection abnormalities, table corruption, data loss, etc.), and other specific fault categories.

[0058] Fault parameter information: If it is a network delay fault, it may include the specific duration of the delay. If it is a memory fault, there may be a target percentage value for memory usage. For CPU usage faults, there are set CPU usage values, etc. These specific parameters are used to accurately control the degree and effect of fault simulation.

[0059] Fault scope information: Whether the fault affects a single node, a specific node group, the entire cluster, or a specific application service module, database instance, etc., clarify the boundaries and objects of the fault.

[0060] Fault duration information: indicates the duration from the start of the fault simulation to the end, for example, whether it is a short-term instantaneous fault (such as a network outage of a few seconds) or a long-lasting fault (such as continuous high resource usage for several minutes or even hours).

[0061] In this embodiment, before inputting the interface parameters through the target interface in the interface layer, the automated chaos testing method of the distributed system further includes:

[0062] Preset interfaces for multiple chaos test scenarios in the interface layer. Preset corresponding execution files and corresponding test suites in the interface of each chaos test scenario. Preset execution scripts in the execution files. There is a one-to-one correspondence between the interface of the chaos test scenario, the execution file (e.g., yaml file), the test suite, and the execution script pre-set in the execution file.

[0063] In this embodiment, the automated chaos testing method for the distributed system further includes:

[0064] Configure the component verification tool set directory and chaos engineering directory at the tool layer. The component verification tool set directory includes the automated test scripts corresponding to each chaos test scenario, and the chaos engineering directory includes the test tools corresponding to each chaos test scenario.

[0065] The tool layer manages automated test scripts and test tools in different directories, mainly storing automated test scripts or test tools that are remotely called by yaml files. There is a one-to-one correspondence between yaml files and chaos experiment triggering interface APIs. The interface parameters include the corresponding yaml file name. When the interface is called, the task of the corresponding yaml file is executed, that is, the script is executed.

[0066] Directory 1: Component Verification Toolset Directory. There are many types of big data components and machines to be tested, and they are responsible for different business logics. Therefore, the test verification methods of different machines to be tested are different. Therefore, customized automated test scripts are needed, and they should be developed using different scripting languages ​​according to different test scenarios.

[0067] Directory 2: Chaos Engineering Directory. There are relatively mature tools on the market, such as ChaosMesh and ChaosBlade. After comparison, ChaosBlade is relatively more flexible and adaptable, so the ChaosBlade toolkit is stored here. It can inject faults on physical machines or in containers (K8S / Docker) by inputting parameters.

[0068] For example, when initiating a chaos experiment, it is necessary to simultaneously create memory overflow failure events for three machines under test in the target object under test (the target object under test includes three machines under test). When calling the chaos experiment trigger interface / component-autotest / chaosblade / basic-resource / memory, it is necessary to specify the machine, enter the interface parameters, and the identification information of the object under test: "machine": "192.168.0.1, 192.168.0.2, 192.168.0.3". At the same time, it is necessary to select physical memory or cache "mode": "ram", and occupancy ratio "percentage": 99 and other parameters (fault event information), identification information of the target executable file (file name of the yaml file). When calling the chaos experiment trigger interface, these interface parameters will be passed to the execution layer as variable values.

[0069] S102: receiving input interface parameters through the execution layer, and sending the interface parameters to the target execution file.

[0070] As an example, the input interface parameters are received by the Python Ansible library, and when the Ansibl e play function is executed, the interface parameters are passed as variables to the YAML file reference.

[0071] S103: Based on the execution script corresponding to the target execution file, the target test suite is called through the server to execute the target test suite on the target object to be tested, and a chaos test execution result is obtained.

[0072] In this embodiment, the effect achieved by the execution layer is that different interfaces will call different yaml files, and different interface parameter inputs will run different automated test scripts or test tools in batches or individually on different machines to be tested. For example, in the yaml file, task execution task shell: / <directory> / chaosbl ade-1.7.1 / blade{{operation}}{{param_combine}}, operation and param_combine are both passed from the interface.

[0073] In this embodiment, before calling the target test suite through the server, the automated chaos testing method for the distributed system further includes: inputting the interface parameters into the target test suite as variables.

[0074] In this embodiment, before inputting the interface parameters as variables into the target test suite, the automated chaos testing method for the distributed system further includes: determining whether the target test suite exists in the server. If the target test suite does not exist in the server, uploading the target test suite to the server.

[0075] When the target test suite exists in the server, the interface parameters are input to the target test suite as variables.

[0076] In this embodiment, the yaml file first checks whether the required test suite exists on the remote server. If it exists, it will not be uploaded repeatedly. Then, the interface parameters input by the interface are concatenated and passed as input parameters to the test suite to be executed remotely.

[0077] In this embodiment, the interface parameters are input as variables into the target test suite, which is equivalent to determining the specific experimental parameters and target test object of the test scenario. For example, after receiving the fault type parameter indicating the high memory surge and the corresponding memory occupancy ratio parameter, the automated test script can accurately perform the corresponding memory occupancy simulation operation on the specified machine to be tested, so that the test meets the requirements set in advance by the tester. To make the test suite universal and reusable, the test tool or automated test script itself is written according to a certain general logic, but it is necessary to perform different actual operations based on specific input parameters. Taking a test tool for simulating various resource failures as an example, it itself has a functional code framework for simulating failures of different resources (memory, CPU, etc.), but only when specific parameters are passed in (such as indicating that it is a memory failure and the corresponding memory parameters, or indicating that it is a CPU failure and related CPU parameters, etc.), it can know exactly which specific fault simulation behavior to perform on the target test object. In this way, the same test suite can be used repeatedly in different chaos test scenarios. As long as different interface parameters are passed in each time, different test effects can be achieved, which improves the reusability of the test suite and the flexibility of the entire test process. The interface parameters define the key details of the entire chaos test. Inputting the interface parameters as variables into the target test suite can ensure that the test tool or automated test script is executed accurately in the way the tester expects. For example, for an automated test script, if the correct identification information of the target object to be tested is not passed in, it may not know which machines to operate on. Without information about the fault event, it is unclear what kind of fault to simulate and how to simulate it. Only by accurately passing these interface parameters can the test behavior executed on the target object to be tested match the originally set chaos test scenario, and ultimately obtain the expected chaos test execution results.

[0078] In this embodiment, the technical solutions of S102-S103 are completed at the execution layer. Through the capabilities of Ansible, the test suites are uploaded to the test objects (physical machines or virtual machines) in batches or distributed for execution. The main function of Ansible is to transfer the test suites required for the chaos experiment to the corresponding test objects in batches or according to specific distribution rules, and run the test suites on the test objects to simulate various chaos failure scenarios. The main function of Ansible is achieved through the execution script corresponding to the execution file.

[0079] In this embodiment, conventional technical means may be used to write an execution script corresponding to an execution file. It should be noted that the writing of the execution script corresponding to an execution file mainly involves two aspects of logical content:

[0080] On the one hand, there is the logic related to file upload, which specifies how to upload the test suite required for the chaos experiment to the target object under test. For example, the execution script may specify the source location from which to obtain the test suite, such as a shared storage or a specific server directory (component verification tool set directory and chaos engineering directory), and then copy the test suite to the specific directory of the target object under test according to the rules.

[0081] On the one hand, there is the remote execution logic of the test suite: in addition to the file upload logic, the yaml file also writes the logic of how to remotely execute these uploaded test suites on the target object to be tested. For example, it defines which parameters need to be passed in when the automated test script is run (these parameters are the interface parameters passed from the interface layer earlier, embedded in the execution command in the form of variables), and in what order to execute different test suites. Taking the chaos experiment to simulate a high memory failure as an example, the yaml file will clearly state how to start the ChaosBlade test tool on the target object to be tested, and pass in interface parameters such as the memory usage ratio, so that the test tool can truly simulate a high memory failure event on the target object to be tested.

[0082] S104: Receive the returned chaos test execution result through the interface layer.

[0083] In this embodiment, the chaos test execution result can be returned to the interface layer through Ansible.

[0084] In this embodiment, a service test interface is pre-set in the interface layer. After receiving the returned chaos test execution result through the interface layer, the automated chaos test method of the distributed system further includes:

[0085] When the chaos test execution result is that the chaos test execution is successful, a business test is performed based on the business test interface to obtain the business test execution result. When the chaos test execution result is that the chaos test execution is successful, a business test is performed based on the business test interface to obtain the business test execution result, which specifically includes: when the chaos test execution result is that the chaos test execution is successful, a business test command sent by the server is received, the business test interface is called based on the business test command, and a business test is performed based on the business test script pre-set by the business test interface to obtain the business test execution result.

[0086] In this embodiment, service test interfaces for different service tests may be pre-set at the interface layer.

[0087] When the chaos test is completed on the target object and the result is successful, the server will generate a business test command according to the pre-set process and logic. This command contains some necessary information, such as informing the recipient that a business test is required, the possible business test scope (whether it is for a specific module or the entire business system), and some configuration information related to the business test. The recipient (which may be another system module or a component specifically responsible for business test execution) will obtain this business test command and start the subsequent business test process.

[0088] Calling the business test interface based on the business test command includes:

[0089] Parsing business test commands: After receiving the business test command, it is first necessary to parse it and extract key information related to calling the business test interface. For example, determine the specific name of the business test interface to be called, the version number (if there are multiple versions of the business test interface), the interface calling parameter format requirements and other information.

[0090] Initiate an interface call request: Based on the parsed information, construct an interface call request that meets the requirements. This request usually contains some necessary authentication information (if the business test interface requires identity authentication to ensure security) and parameter data organized according to the interface parameter format requirements (these parameters may come from the relevant information in the business test command, or may be some pre-configured default parameters in the system combined with the parameters in the command). Then, send this interface call request to the server or service endpoint where the corresponding business test interface is located to trigger the execution of the business test interface.

[0091] Perform business testing based on the business test scripts pre-set in the business test interface, including:

[0092] Loading and initialization of business test scripts: After receiving a call request, the business test interface will first load the pre-set business test script. These business test scripts are written according to different business requirements and business logic, and are used to simulate the business operations of real users or to check and verify the key functions of the business system. While loading the script, some initialization operations may be performed, such as establishing a connection with the business system database (if the business test needs to access the database) and initializing some test environment variables (such as setting test data generation rules, configuring test time limits, etc.).

[0093] Execute business test logic: Execute test operations step by step according to the logical sequence written in the business test script. For example, if it is an e-commerce business test script, it may first perform a user login operation (using a pre-set test account and password), and then browse the product list, add products to the shopping cart, place an order, simulate the payment process, and other operations related to the e-commerce business process. In the process of executing these operations, various relevant data and information will be collected, such as the response time of each operation, whether the returned result data meets expectations (such as whether the login successfully returns the correct user information, whether the order status is correctly updated after placing an order, etc.). These data and information will serve as the basis for generating business test execution results later.

[0094] Get the business test execution results, including:

[0095] Sorting and summarizing the result data: After the business test script is executed, the data and information collected during the entire test process will be sorted and summarized. The results of each test operation (success or failure), related performance data (such as the average, maximum, minimum, etc. of the response time), and possible error messages or abnormal situations will be recorded in detail.

[0096] Generate a business test execution result report: Generate a detailed business test execution result report based on the collated and summarized result data. This report usually adopts a structured format, such as a test overview (explaining the purpose, scope, time and other basic information of the test), test result details (listing the results, performance data, error information, etc. one by one according to the test operation steps), overall evaluation (based on the test results, judging whether the business functions of the business system are normal after the chaos test, whether the performance meets the requirements, etc.), and possible suggestions or improvement directions (if problems are found in the test, propose corresponding solutions or optimization suggestions). This business test execution result report is the final business test execution result, which will be used to evaluate the impact of chaos testing on the business system and determine whether the business system can still operate normally and meet business needs after experiencing chaos fault simulation.

[0097] For example, let's assume that this is a chaos test and business test process for a financial business system. After the chaos test successfully simulates the failure of some nodes in the database and recovers, the server sends a business test command to test the transfer business. After the receiver parses the command, it calls the transfer business test interface, which loads the transfer business test script. The script first initializes the test environment, sets the test transfer amount, receiving account and other information, and then performs a series of operations from logging in to the account, initiating a transfer request, to verifying the transfer result. The response time and result data of each step are collected, and the business test execution result report is generated after sorting and summarizing. The report shows the execution of the transfer business after the chaos test, such as whether the transfer is successful, whether the average response time is within an acceptable range, etc., so as to evaluate the stability of the distributed system and the integrity of the business functions.

[0098] Since the big data cluster of a distributed system is generally composed of a dozen machines for a small scale, and may be composed of hundreds or thousands of machines for a large scale, it is very time-consuming and labor-intensive to log in one by one to test and create a chaos experiment. Therefore, the technical solution of the present invention distributes the automated test scripts and test tools of different target objects to be tested to the target objects to be tested (specified cluster groups or specified machines) for operation by calling interfaces, and then returns the operation results to the interface, and then the interface is scenario-based orchestrated through the interface automation platform, gradually realizing full automation. As an example, the overall technical framework of the technical solution of the present invention can be a framework mainly including Python+FastAPI+Ansible+SQLite+ChaosBlade, which is divided into three layers in total: interface layer, execution layer and tool layer.

[0099] As an example, when the technical solution of the present invention is mainly implemented through the FastAPI Web service framework, the interface parameters are input through the chaos experiment trigger interface, and the interface parameters are passed to the execution layer (for example, Ansible-Playbook layer, referred to as AP layer). The execution layer is responsible for actually performing some specific operations, such as deploying relevant test tools and automated test scripts to the corresponding machines, and then running these automated test scripts and test tools to simulate chaos failures. This transmission process is like a chain. The interface layer collects requirements (interface parameters) and then passes them to the execution layer (performing operations). After the execution layer completes the execution of the corresponding automated test scripts and test tools, such as successfully simulating chaos failures on the specified machine, or there are errors during the execution process, these chaos test execution results will be returned through the interface. FastAPI is a high-performance framework for building Web application program interfaces (APIs). In the present invention, it is used to build the interface layer of the entire chaos experiment related operations. Testers can input various parameters through the chaos experiment trigger interface it provides to start the chaos test related processes.

[0100] In the present invention, a method can be designed based on the technical solution of the automated chaos testing method of the distributed system. Figure 2 The automated chaos test system (Chaos Component AutoTest Server, CCAS) of the distributed system shown in the figure. From top to bottom, it includes three layers: interface layer, execution layer (Ansible-Playbook layer) and tool set. The automated chaos test system of the distributed system is connected to the external interface automation platform. The interface automation platform belongs to the external system and is used to orchestrate the call of the chaos experiment trigger interface (the interface of the chaos test scenario pre-set in the interface layer) and the business verification interface. Interface automation uses open source, free or self-developed ones on the market. It mainly orchestrates and executes various chaos experiment trigger interfaces and business verification interfaces in series to achieve full automation of business verification in chaos scenarios.

[0101] In the process of conducting chaos experiments, some specific chaos test scenarios cannot be operated in the conventional way when they are ended (destroyed), but must be completed with the help of a unique identifier, that is, Task ResultID. For example, if a more complex and long-lasting chaos failure scenario is simulated, in order to end this scenario cleanly and neatly and restore the automated chaos test system to normal state, this specific ID is needed to accurately tell the automated chaos test system which specific chaos test scenario to destroy, including the relevant configuration (chaos experiment trigger interface, execution file, test suite, execution script) and related resource usage. Because the destruction of some chaos test scenarios has such a dependence on TaskResultID, when creating a chaos experiment, it is necessary to save the TaskResultID corresponding to this experiment. You can choose to use SQLite to save it. SQLite is a lightweight database that is very suitable for storing such key information simply and effectively in this scenario. When the subsequent call to destroy the experiment interface is needed, that is, when a chaos test scenario needs to be ended, the automated chaos test system can extract the corresponding TaskResultID previously saved from the SQLite database, and then accurately perform the destruction operation based on this ID to ensure the integrity and controllability of the entire chaos experiment process. In the SQLite database, TaskResultID will be stored as a field in the relevant data table. The data table may have multiple columns. In addition to the TaskResultID column, there may also be a creation time column (recording the start time of the experiment), a status column (indicating whether the experiment is in progress, completed or canceled, etc.), and an experiment parameter column (experimental parameters (failure event information) are stored in a certain format such as JSON format, including the previously mentioned servers, failure types, failure parameters, etc.).

[0102] The technical solution of the present invention can be summarized as uploading the ChaosBlade toolkit to the specified physical machine / virtual machine / machine group by calling the API interface, selecting the execution script in the yaml file to execute through the interface parameters, remotely running the ChaosBlade test tool or automated test script in single point / batch, simulating various chaos failure events on any cluster / node, and can be combined with the interface automation platform to ultimately achieve full automation of chaos experiment simulation and business verification orchestration.

[0103] The technical effects of the technical solution of the present invention are as follows:

[0104] Distributed triggering of chaos experiments through interface calls. Compared with previous methods, containerized environments can trigger chaos experiments in a workflow-like manner, but cannot be combined with business verification. In a physical machine environment, it is necessary to upload toolkits to each machine and inject faults in specific scenarios and environments through command lines, which is inefficient. The technical solution of the present invention can flexibly choose to inject various types of faults in batches and in parallel in a containerized or physical machine environment by calling interfaces, and can combine chaos experiment trigger interfaces and business verification interfaces to achieve full-scenario automated testing.

[0105] Chaos testing is fully automated. Generally, automated testing of components and machines is done by writing and executing scripts or tools for independent verification, and cannot be arranged serially or in parallel for scenario-based and associated verification. The technical solution of the present invention can integrate and classify automated test scripts or test tools, configure them in different chaos experiment trigger interfaces, and according to the test scenario requirements, scenario-based arrangement and call of corresponding chaos experiment trigger interfaces, remotely run automated test scripts or test tools, and combine with chaos experiment trigger interfaces to achieve full automation of chaos testing.

[0106] In daily iterations and online releases, chaos testing is basically a one-time step, and each execution of chaos testing is relatively labor-intensive. The technical solution of the present invention realizes full automation of chaos and business verification, allowing chaos testing to be a frequently executed step in daily iterations, while also greatly saving testing labor costs. The technical solution of the present invention is used for distributed systems with large-scale containerization and microservice hybrid deployment, and has high availability. In addition to the commonly used interface automation for business verification, the technical solution of the present invention is further developed to automate the verification of the functions and stability of components, and the three scenarios of chaos experiments, business verification, and component verification are combined and arranged. In key links such as daily online releases, changes, and related technical architecture upgrades, the robustness of the system is fully automatically verified. The above is a possible solution that is further developed based on the technical solution of the present invention, and is not specifically described in the present invention.

[0107] like Figure 3 As shown, before triggering the server chaos experiment, that is, before executing the technical solution of the present invention, some preliminary preparation work needs to be done:

[0108] For example, edit the Inventory file and fill in the identification information of the machine to be tested in the format required by Ansible, such as the machine IP, account password and port for accessing the machine. You can also group the machines to be tested to obtain multiple groups of objects to be tested, and then trigger the chaos experiment in parallel according to the groups.

[0109] Upload the inventory file to CCAS through the upload interface, and then call the password-free interface to access the machine settings filled in the inventory file from CCAS (physical machine or container) without password.

[0110] Select the corresponding interface according to the scenario where you need to trigger the chaos experiment in the physical machine or container. To trigger experiments in batches, you can fill in the machine group name of the Inventory in the machine parameter, or fill in multiple machine IPs separated by ",". If you only need to trigger the experiment on a single machine, fill in the corresponding IP in the machine parameter.

[0111] For business verification, according to the characteristics of the product, you can directly call the Web-side interface for verification. If it is the verification of the server-side component, you need to call the component interface corresponding to CCAS, fill in the component parameter with the corresponding component name, and the script parameter with the corresponding script name. If you want to execute multiple scripts, you can fill in multiple script names in the script parameter, separated by ",".

[0112] Scenario orchestration arranges the chaos experiment trigger interface, business verification interface and chaos experiment destruction interface as expected, and can be executed regularly and automatically on demand.

[0113] It should be noted that the contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0114] In this embodiment, an automated chaos testing device for a distributed system is also provided. A single device is used to implement the above-mentioned embodiments and optional implementation methods. Those that have been described will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0115] Figure 4 It is a structural diagram of an automated chaos testing device for a distributed system according to an embodiment of the present invention.

[0116] The present invention provides an automated chaos testing device for a distributed system, wherein the distributed system includes multiple groups of objects to be tested, such as Figure 4 As shown, the automated chaos testing device of the distributed system includes:

[0117] The first processing module 11 is used to input interface parameters through a target interface in the interface layer.

[0118] The interface layer is pre-set with interfaces for multiple chaos test scenarios. The interface of each chaos test scenario is pre-set with a corresponding execution file and a corresponding test suite. The execution file is set with a corresponding execution script, which is used to call the test suite. The test suite includes an automated test script or a test tool. The target interface is the interface of the target chaos test scenario. The interface parameters include the identification information of the target object to be tested, the identification information of the target execution file, and the information of the fault event.

[0119] The second processing module 12 is used to receive the input interface parameters through the execution layer and send the interface parameters to the target execution file.

[0120] The third processing module 13 is used to call the target test suite through the server based on the execution script corresponding to the target execution file, so as to execute the target test suite on the target object to be tested and obtain the chaos test execution result.

[0121] The fourth processing module 14 is used to receive the returned chaos test execution result through the interface layer. In an optional implementation, the automated chaos test device for a distributed system further includes: a configuration module, and the configuration module includes a first processing unit.

[0122] The first processing unit is used to pre-set interfaces of multiple chaos test scenarios in the interface layer before inputting interface parameters through the target interface in the interface layer. Pre-set corresponding execution files and corresponding test suites in the interface of each chaos test scenario. Pre-set execution scripts in the execution files.

[0123] In an optional implementation, the configuration module further includes a second processing unit.

[0124] The second processing unit is used to configure the component verification tool set directory and the chaos engineering directory at the tool layer.

[0125] The component verification toolset directory includes automated test scripts corresponding to each chaos test scenario, and the chaos engineering directory includes test tools corresponding to each chaos test scenario.

[0126] In an optional implementation, the automated chaos testing device for the distributed system further includes: a variable input module, configured to input the interface parameters as variables into the target test suite before calling the target test suite through the server.

[0127] In an optional embodiment, the automated chaos testing device for a distributed system further includes: a judgment module for determining whether a target test suite exists in the server before inputting the interface parameter as a variable into the target test suite. If the target test suite does not exist in the server, the target test suite is uploaded to the server.

[0128] In an optional embodiment, the automated chaos test device of the distributed system further includes: a business test module, which is used to perform a business test based on a business test interface to obtain a business test execution result after receiving the returned chaos test execution result through the interface layer, when the chaos test execution result is that the chaos test execution is successful. A business test interface is pre-set in the interface layer.

[0129] In an optional embodiment, the business testing module is specifically used to receive a business testing command sent by the server when the chaos test execution result is that the chaos test execution is successful, call the business testing interface based on the business testing command, perform business testing based on the business testing script pre-set in the business testing interface, and obtain the business testing execution result.

[0130] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0131] The automated chaos testing device for the distributed system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0132] The embodiment of the present invention also provides a computer device having the above Figure 4 The automated chaos testing apparatus for the distributed system shown.

[0133] See also Figure 5 , Figure 5 Schematic diagram of the hardware structure of the computer device according to the embodiment of the present invention. Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In an optional embodiment, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor device). Figure 5 A processor 10 is taken as an example.

[0134] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0135] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0136] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating device, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In an optional embodiment, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The memory 20 may include a volatile memory, such as a random access memory. The memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive. The memory 20 may also include a combination of the above-mentioned types of memory.

[0138] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0139] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be downloaded through a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc. Further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0140] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0141] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An automated chaos testing method for a distributed system, wherein the distributed system comprises a plurality of groups of objects to be tested, characterized in that: include: Input interface parameters through the target interface in the interface layer; The interface layer is pre-configured with interfaces for various chaos test scenarios; The interface of each chaos test scenario is pre-set with corresponding execution files and corresponding test suites; The execution file is provided with a corresponding execution script, and the execution script is used to call the test suite, and the test suite includes an automated test script or a test tool; the target interface is an interface of a target chaos test scenario, and the interface parameters include identification information of a target object to be tested, identification information of a target execution file, and information of a fault event; Receiving input interface parameters through the execution layer, and sending the interface parameters to the target execution file; Based on the execution script corresponding to the target execution file, the target test suite is called through the server to execute the target test suite on the target object to be tested, and a chaos test execution result is obtained; The chaotic test execution result is received through the interface layer.

2. The method according to claim 1, characterized in that Before invoking the target test suite via the server, also include: The interface parameters are input into the target test suite as variables.

3. The method according to claim 2, characterized in that Before inputting the interface parameters as variables into the target test suite, the method further includes: Determining whether the target test suite exists in the server; When the target test suite does not exist in the server, the target test suite is uploaded to the server.

4. The method according to claim 1, characterized in that: Before inputting interface parameters through the target interface in the interface layer, it also includes: Pre-setting interfaces for multiple chaos test scenarios in the interface layer; Pre-set the corresponding execution file and the corresponding test suite in the interface of each chaos test scenario; An execution script is pre-set in the execution file.

5. The method according to claim 1, characterized in that The method further comprises: Configure the component verification tool set directory and chaos engineering directory at the tool layer; The component verification tool set directory includes automated test scripts corresponding to each chaos test scenario, and the chaos engineering directory includes test tools corresponding to each chaos test scenario.

6. The method according to claim 1, characterized in that The interface layer is pre-set with a service test interface. After receiving the chaotic test execution result returned through the interface layer, the method further includes: When the chaos test execution result is that the chaos test execution is successful, a business test is performed based on the business test interface to obtain a business test execution result.

7. The method according to claim 6, characterized in that When the chaos test execution result is that the chaos test execution is successful, performing a service test based on the service test interface, and obtaining the service test execution result includes: When the chaos test execution result is that the chaos test execution is successful, a business test command sent by the server is received, the business test interface is called based on the business test command, and a business test is performed based on a business test script pre-set by the business test interface to obtain a business test execution result.

8. An automated chaos testing device for a distributed system, the distributed system comprising a plurality of groups of objects to be tested, characterized in that: include: A first processing module, used for inputting interface parameters through a target interface in the interface layer; The interface layer is pre-configured with interfaces for various chaos test scenarios; The interface of each chaos test scenario is pre-set with corresponding execution files and corresponding test suites; The execution file is provided with a corresponding execution script, and the execution script is used to call the test suite, and the test suite includes an automated test script or a test tool; the target interface is an interface of a target chaos test scenario, and the interface parameters include identification information of a target object to be tested, identification information of a target execution file, and information of a fault event; A second processing module, configured to receive input interface parameters through the execution layer, and send the interface parameters to the target execution file; A third processing module is used to call a target test suite through a server based on an execution script corresponding to the target execution file, so as to execute the target test suite on the target object to be tested and obtain a chaotic test execution result; The fourth processing module is used to receive the chaotic test execution result returned through the interface layer.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the automated chaos testing method for a distributed system according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the automated chaos testing method for a distributed system according to any one of claims 1 to 7.

Images