Chaotic test method, device and equipment based on system, medium and program product

Abstract

The invention relates to the technical field of software engineering, and discloses a system-based chaos test method, device and equipment, a medium and a program product. The method comprises the following steps: splitting the distributed metadata system into a plurality of subsystems to be tested; determining test data corresponding to each to-be-tested subsystem and a corresponding test scene: a fault type, a fault node and fault interval time; determining a test case set based on the test scene and the test data corresponding to each to-be-tested subsystem: a test case corresponding to each to-be-tested subsystem and a test case corresponding to the to-be-tested system; and executing the test case set, and determining whether the corresponding test data is correctly read and written and whether the service on the fault node is correctly operated so as to determine a test result. According to the invention, the complex distributed metadata system is decomposed into a plurality of to-be-tested subsystems, the test scene is customized according to the functional characteristics of different to-be-tested subsystems, and the targeted chaos test is carried out, so that the test efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software engineering, and in particular to a system-based chaos testing method, device, equipment, medium and program product. Background Art

[0002] In the context of the rapid development of the current Internet, the surge in enterprise data volume has made it difficult for a single database to meet storage needs, and distributed systems have therefore become an important development direction in the field of data storage. With the continuous increase in the number of users and the amount of stored data, the scale of distributed systems is also constantly expanding, and a scalable and extensible distributed system has become the goal pursued in the field of data storage. The corresponding metadata is also increasing in large quantities. How to efficiently, quickly, and accurately access massive storage data has become a key point in distributed systems.

[0003] Due to the large scale, wide distribution range, large geographical span, and various interdependencies of distributed systems, it is inevitable that various unexpected emergencies will occur, countless places where errors can occur, and various other unexpected abnormal behaviors will occur. In related technologies, distributed metadata systems are usually tested in the hope of identifying potential risks and problems, so as to solve these problems before actual deployment and operation, and reduce the impact on the business.

[0004] The distributed metadata system testing method in the related art has the problems that the test scenario is not targeted, the fault injection type is relatively blind and has no basis, and it is impossible to effectively test the distributed metadata system. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem in the related art that the test scenario is not targeted and cannot effectively test the distributed metadata system.

[0006] In order to solve the above technical problems, the present invention provides a system-based chaos testing method, which includes:

[0007] Splitting the system to be tested into multiple subsystems to be tested; the system to be tested is a distributed metadata system, and the multiple subsystems to be tested include a distributed coordination system, a distributed file data storage system, and a business system;

[0008] Determine the test data corresponding to each of the subsystems to be tested;

[0009] Determine a test scenario corresponding to each of the subsystems to be tested; the test scenario includes a fault type, a fault node, and a fault interval;

[0010] Determine a test case set based on the test scenario and the corresponding test data corresponding to each of the subsystems to be tested; the test case set includes test cases corresponding to each of the subsystems to be tested and test cases corresponding to the system to be tested; the test cases corresponding to each of the subsystems to be tested are respectively used to perform chaos testing on each of the subsystems to be tested, and the test cases corresponding to the system to be tested are used to perform chaos testing on the system to be tested;

[0011] Execute the test case set to determine whether the test data corresponding to each subsystem under test is read and written correctly, and whether the service on the faulty node corresponding to each subsystem under test is executed correctly, so as to determine the test results corresponding to each subsystem under test and the test results corresponding to the system under test.

[0012] In an optional implementation, before determining the test scenario corresponding to each of the subsystems to be tested, the method further includes:

[0013] A preset number of fault types are predetermined; the preset number of fault types include a first type, a second type, a third type and a fourth type; the fault event corresponding to the first type is a complete fault, the fault event corresponding to the second type is a partial communication fault, the fault event corresponding to the third type is a service interruption fault, and the fault event corresponding to the fourth type is an intermittent fault; each fault type includes at least one fault event.

[0014] In an optional implementation, determining the test scenario corresponding to each of the subsystems to be tested includes:

[0015] Determine at least one fault type corresponding to each of the subsystems to be tested; each of the at least one fault type includes at least one fault event;

[0016] Determine at least one fault node corresponding to each of the subsystems to be tested; each of the at least one fault node corresponds to at least one fault event;

[0017] At least one fault interval time corresponding to each of the subsystems to be tested is determined; each fault interval time in the at least one fault interval time corresponds to a fault event.

[0018] In an optional implementation, the determining at least one faulty node corresponding to each of the subsystems to be tested includes:

[0019] Determine the minimum number of processes of the distributed coordination system, and randomly select a preset number of server nodes as fault nodes in the distributed coordination system according to the minimum number of processes; the preset number of nodes is greater than or equal to 1 and less than or equal to the minimum number of processes;

[0020] Randomly select a server node in the distributed file data storage system as a fault node;

[0021] At least one server node is selected as a fault node in the service system according to an upper-layer service model.

[0022] In an optional implementation manner, determining at least one fault interval time corresponding to each of the subsystems to be tested includes:

[0023] For each of the subsystems to be tested, the fault interval time corresponding to each fault event in each fault type under the single subsystem to be tested is determined based on the timeout time of the client corresponding to the single subsystem to be tested and the maximum time interval for communication between the client and the server, so as to determine the fault interval time corresponding to each fault event in each fault type under each subsystem to be tested.

[0024] In an optional implementation, determining a test case set based on a test scenario and test data corresponding to each of the subsystems to be tested includes:

[0025] For each of the subsystems to be tested, a test process and a test data test process are designed to trigger a corresponding fault event at a corresponding fault interval time at each fault node corresponding to a single subsystem to be tested, so as to determine a test case corresponding to each of the subsystems to be tested;

[0026] The test case set is determined according to the test cases corresponding to each of the subsystems to be tested.

[0027] In a second aspect, the present invention provides a system-based chaos testing device, the system-based chaos testing device comprising:

[0028] A first processing module is used to split the system to be tested into multiple subsystems to be tested; the system to be tested is a distributed metadata system, and the multiple subsystems to be tested include a distributed coordination system, a distributed file data storage system and a business system;

[0029] A second processing module, used to determine the test data corresponding to each of the subsystems to be tested;

[0030] The third processing module is used to determine the test scenario corresponding to each of the subsystems to be tested; the test scenario includes the fault type, fault node and fault interval time;

[0031] The fourth processing module is used to determine a test case set based on the test scenario and the corresponding test data corresponding to each of the subsystems to be tested; the test case set includes a test case corresponding to each of the subsystems to be tested and a test case corresponding to the system to be tested; the test case corresponding to each of the subsystems to be tested is used to perform a chaos test on each of the subsystems to be tested, and the test case corresponding to the system to be tested is used to perform a chaos test on the system to be tested;

[0032] The fifth processing module is used to execute the test case set to determine whether the test data corresponding to each of the subsystems to be tested is read and written correctly, and whether the services on the faulty nodes corresponding to each of the subsystems to be tested are executed correctly, so as to determine the test results corresponding to each of the subsystems to be tested and the test results corresponding to the system to be tested.

[0033] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the system-based chaos testing method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a single computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the system-based chaos testing method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0035] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the system-based chaos testing method of the first aspect or any corresponding embodiment thereof.

[0036] The technical solution provided by the present invention has the following technical effects:

[0037] The technical solution of the embodiment of the present invention divides the system to be tested into multiple subsystems to be tested, and tests each subsystem to be tested, which can improve the pertinence of the test. And when determining the test case corresponding to each subsystem to be tested, by clarifying the test scenario corresponding to each subsystem to be tested, it can be ensured that the test case is directly targeted at specific fault events that may occur in the subsystem to be tested, thereby improving the pertinence of the test. Taking into account different fault types, fault nodes and fault intervals, a test case set with wider coverage can be constructed to reduce potential omissions, which helps to accurately simulate the situations that the system may encounter in actual operation, so that the test case can more accurately reflect the system behavior. The fault type and fault node are clarified, and the test case is easier to reproduce specific fault events, which is convenient for debugging and problem location. Targeted test cases can reduce unnecessary tests and improve test efficiency. Since the test case is designed based on a specific test scenario, the behavior and performance of the system can be more accurately evaluated. By simulating different fault types and node failures, potential risks and problems can be identified earlier. The technical solution of the present invention uses test cases corresponding to the subsystem to test whether the service on the faulty node runs correctly when a fault event occurs, and whether the test data corresponding to the subsystem to be tested can be read and written correctly, which can significantly improve the quality and effect of the test. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a flow chart of a system-based chaos testing method according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the split structure of the distributed metadata system according to an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a test case generation process according to an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of the deployment of nodes of the system to be tested in Example 1 of the embodiment of the present invention;

[0043] Figure 5 Schematic diagram of the deployment of nodes in the system to be tested according to the second embodiment of the present invention;

[0044] Figure 6is a schematic structural diagram of a system-based chaos testing device according to an embodiment of the present invention;

[0045] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] With the continuous development of the Internet, enterprises have accumulated more and more data. A single database can no longer meet the needs of storing massive data. Distributed systems have begun to enter people's field of vision. With the continuous increase in the number of users and the amount of stored data, the scale of distributed systems is also expanding. A scalable and extensible distributed system has become the goal pursued in the field of data storage. The corresponding metadata is also increasing in large quantities. How to efficiently, quickly and accurately access massive stored data has become a key point of distributed systems.

[0048] Distributed systems running in production environments are bound to have various unpredictable emergencies. Large-scale distributed systems are widely distributed, have a large geographical span, and are inherently interdependent. There are countless places where errors can occur, and it is also easy to have various unexpected abnormal behaviors. If the failures are not handled properly, it will cause business damage. Distributed metadata systems will also have problems such as reduced stability, reduced read and write performance, reduced availability, and even data loss. In complex distributed systems, it is impossible to prevent these failures from occurring. We should strive to identify as many risks as possible before these abnormal behaviors are triggered. Then, targeted reinforcement and prevention should be carried out to avoid the serious consequences of failures.

[0049] The traditional distributed metadata system testing method has the following problems: 1) The test environment is simple, the test data is single, and it is difficult to reproduce online faults. It is impossible to predict the type of online faults in advance, resulting in a large-scale impact. 2) The test scenario is single, and there is no targeted test plan for different system service models, which makes it impossible to quickly locate the problem when it occurs, resulting in a long time to restore to normal after the failure occurs. 3) The test scenario is not targeted, the fault injection type is relatively blind and has no basis, and it is not possible to effectively test the distributed metadata system.

[0050] To this end, the embodiments of the present invention provide a system-based chaos testing method, device, equipment, medium and program product to solve the above problems.

[0051] According to an embodiment of the present invention, a system-based chaos testing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] Figure 1 It is a flow chart of a system-based chaos testing method according to an embodiment of the present invention.

[0053] like Figure 1 As shown, an embodiment of the present invention provides a system-based chaos testing method, and the system-based chaos testing method includes:

[0054] S101: Split the system to be tested into multiple subsystems to be tested.

[0055] In this embodiment, the system to be tested is a distributed metadata system, and the multiple subsystems to be tested include a distributed coordination system, a distributed file data storage system, and a business system. The distributed coordination system is mainly used to solve the consistency problem of the application system in the distributed cluster. It is usually composed of multiple node services, one and only one of which provides the leader election function, and the rest are followers. A server node is determined as the leader through the election mechanism, which is responsible for coordinating other nodes (followers) in the cluster. The distributed file data storage system consists of multiple storage services, file management services of the master-slave structure, and services that provide status monitoring functions. The business system includes management services of the master-slave structure and related services that respond to user requests.

[0056] In this embodiment, the service deployment and configuration status of each server node in the system under test can be determined by pre-checking the system under test, and the deployment status can be collected. The system under test can be divided into multiple subsystems under test according to the service deployment and configuration status of each server node in the system under test, and the following information can be obtained: Figure 2 The distributed metadata system split structure diagram is shown. Specifically, the polling detection of the server nodes in the system to be tested can be implemented through scripts, the service deployment and configuration status on each server node can be detected, and the deployment status can be collected.

[0057] S102: Determine the test data corresponding to each subsystem to be tested.

[0058] In this embodiment, test data preparation can be performed according to the distributed metadata system split diagram, including automatic table creation, writing a certain amount of metadata, etc. Determine that the test data corresponding to each subsystem to be tested can use a script to automatically create the database table structure required for the test. This can be done through the HBase shell command or a programming interface (such as the Java API). For example, the HBase shell command can be used to automatically create a table, and after the table is created, the test data needs to be prepared. This can be done through the HBase shell or a programming interface. For example, the HBase shell command can be used to insert metadata.

[0059] S103: Determine the test scenario corresponding to each subsystem to be tested.

[0060] In this embodiment, the test scenario includes the fault type, fault node and fault interval time. In addition to the above fault type, fault node and fault interval time, the test scenario may also include an expected target.

[0061] In this embodiment, before determining the test scenario corresponding to each subsystem to be tested, the system-based chaos testing method also includes: predetermining a preset number of fault types. The preset number of fault types include a first type, a second type, a third type, and a fourth type. The fault event corresponding to the first type is a complete fault, the fault event corresponding to the second type is a partial communication fault, the fault event corresponding to the third type is a service interruption fault, and the fault event corresponding to the fourth type is an intermittent fault. Each fault type includes at least one fault event.

[0062] In this embodiment, in the distributed metadata system, the fault types can be divided into four categories according to the degree to which the process or service is affected by the fault: the first type (complete failure) is that after the fault event is injected, the process or service no longer executes any steps, nor receives or sends any messages. The second type (partial communication failure) is that after the fault event occurs, partial communication between processes or services is affected. The third type (service interruption failure) is that when the fault event occurs, the process and service are interrupted, resulting in communication interruption, and the communication can be restored after the fault is restored. The fourth type (intermittent failure) is that after the fault event occurs, the process or service can sometimes run normally, and sometimes there will be uncertain failures. According to this principle, the fault types are classified as shown in Table 1 below. When performing chaos test fault event injection, one to multiple fault events of the same fault type can be randomly selected, which can be determined according to the test cycle.

[0063] Table 1 Fault type classification table

[0064]

[0065] In this embodiment, when determining the test scenario corresponding to each subsystem to be tested, the availability of single-node failures, the system's tolerance under multi-node failures, and the self-healing of the system after a crash need to be considered. In addition, some failure events do not occur once, but may occur multiple times in succession, and the system's tolerance for continuous failures needs to be verified. To this end, determining the test scenario in the present invention can be divided into the following three steps: the first step is to determine the fault type, the second step is to determine the fault node, and the third step is to determine the fault execution interval.

[0066] The current chaos testing method does not target the service characteristics of distributed systems. The technical solution of the present invention splits the complex distributed metadata system into multiple subsystems to be tested according to the functional characteristics of each subsystem to be tested in the system, customizes the test scenarios according to the functional characteristics of different subsystems to be tested, and performs targeted chaos testing, thereby improving the testing efficiency.

[0067] In the present invention, the above S103 determines the test scenario corresponding to each subsystem to be tested, which specifically includes:

[0068] S1031: Determine at least one fault type corresponding to each subsystem to be tested.

[0069] In this embodiment, each fault type in the at least one fault type includes at least one fault event. In this embodiment, each subsystem to be tested may correspond to a preset number of fault types, that is, each subsystem to be tested corresponds to four fault types.

[0070] S1032: Determine at least one faulty node corresponding to each subsystem to be tested.

[0071] In this embodiment, each of the at least one faulty node corresponds to at least one fault event. Determining at least one faulty node corresponding to each subsystem to be tested in S1032 specifically includes:

[0072] Identify at least one failed node corresponding to the distributed coordination system:

[0073] The minimum number of processes of the distributed coordination system is determined, and a preset number of server nodes are randomly selected as fault nodes in the distributed coordination system according to the minimum number of processes. The preset number of nodes is greater than or equal to 1 and less than or equal to the minimum number of processes.

[0074] Determine at least one failed node corresponding to the distributed file data storage system:

[0075] In the distributed file data storage system, a server node is randomly selected as the fault node.

[0076] Determine at least one faulty node corresponding to the business system:

[0077] At least one server node is selected in the business system as a fault node according to the upper-layer business model.

[0078] In this embodiment, the distributed coordination system, the distributed file data storage system and the business system can all randomly select server nodes as fault nodes. However, the scope of random selection by the business system is the server nodes corresponding to the upper-layer business model. In addition to random selection, a fixed server node can also be specified as a fault node.

[0079] In this embodiment, each subsystem to be tested can be split in different ways according to the characteristics of each subsystem to be tested. Since the distributed coordination system is composed of multiple server nodes and has a one-master-multiple-standby structure, when determining the number of faulty nodes to be selected, in order to ensure that the cluster can normally select the master, it is necessary to consider the principle of more than half of the processes, which means that more than half of the surviving service nodes can meet the principle of normal master selection of the distributed coordination service. Assuming that the number of nodes in the distributed coordination system cluster is Q, the minimum number of cluster processes is k=Q / 2+1. When performing chaos testing, the value range of the number of faulty nodes is [1,Q / 2-1] (preset number of nodes).

[0080] The present invention proposes a minimum process number algorithm for the first time to effectively predict the scope of chaos test failures. The distributed file data storage system and the business system do not involve cluster master election. Only one faulty node can be created in the master and backup nodes of the distributed file data storage system, and the faulty node of the business system needs to be jointly determined according to the upper-level business model. Specifically, the faulty node can be determined according to the storage type of the upper-level business model, and a service node of the corresponding storage type can be randomly selected from the service nodes in the business system as the faulty node. For example, when the upper-level business model is a model for storage in a block structure, a service node for storage in a block structure can be randomly selected from the service nodes in the business system as the faulty node.

[0081] In this embodiment, at least one fault node corresponding to each subsystem to be tested may also be determined according to at least one fault type corresponding to each subsystem to be tested.

[0082] S1033: Determine at least one fault interval time corresponding to each subsystem to be tested.

[0083] In this embodiment, each fault interval in the at least one fault interval corresponds to a fault event.

[0084] In this embodiment, determining at least one fault interval time corresponding to each subsystem to be tested in S1033 specifically includes:

[0085] For each subsystem to be tested, the fault interval time corresponding to each fault event in each fault type under the single subsystem to be tested is determined according to the timeout time of the client corresponding to the single subsystem to be tested and the maximum time interval of communication between the client and the server, so as to determine the fault interval time corresponding to each fault event in each fault type under each subsystem to be tested. The fault interval time is the time interval between the occurrence of the fault event, that is, the time interval between the execution of the current fault event and the next fault event. It can be the interval between the execution of the same fault event or the interval between the execution of different fault events. For example, the execution order of the fault events corresponding to a certain fault node is: fault event A, fault event A, fault event B, fault event C. The fault interval time can be the time interval between fault event A and fault event A, the time interval between fault event A and fault event B, and the time interval between fault event B and fault event C.

[0086] In this embodiment, the distributed application usually relies on the notification mechanism of the distributed coordination service to implement functions such as Master election. When an exception occurs in the distributed coordination service, the communication with the distributed application will also be affected, thereby affecting the Master election of the application. Therefore, the test also needs to consider the failure interval time that the system can withstand. The failure interval time is related to the client timeout time of the distributed coordination service. The present invention proposes a method for calculating the failure interval time based on the distributed coordination service:

[0087] Get the client timeout sessionTimeout.

[0088] Calculate the blocking time of the client's internal call to the blocking method, BlockTime = readTimeout-idleRece, where idleRece is the time interval between the current time and the last listening time. According to experience, readTimeout is generally set to 2 / 3*sessionTimeout.

[0089] Calculate the maximum interval time of communication between the client and the server: pingInternal = BlockTime / 2-idleSend, where idleSend is the time interval between the current time and the last sending time.

[0090] Considering that the client and the server need to wait for at least one communication before communicating, the minimum fault interval time based on the distributed coordination service is T=sessionTimeout+pingInternal. Based on the above scheme, the fault interval time T corresponding to each fault event in each fault type under each subsystem to be tested is determined.

[0091] The present invention proposes a fault interval calculation algorithm for distributed coordination services, realizes fault interval determination, and formulates a more refined test plan. By executing the test case set calculated by the present invention, the fault tolerance and elasticity of the distributed metadata system can be verified within a controllable range to build a highly available distributed metadata system. According to the timeout time and communication interval of the client and server, the minimum fault interval is calculated, which can effectively detect the robustness of the distributed metadata system.

[0092] S104: Determine a test case set based on the test scenario and test data corresponding to each subsystem to be tested.

[0093] In this embodiment, the test case set includes test cases corresponding to each subsystem to be tested and test cases corresponding to the system to be tested. The test cases corresponding to each subsystem to be tested are used to perform chaos testing on each subsystem to be tested, and the test cases corresponding to the system to be tested are used to perform chaos testing on the system to be tested. The test scenario can simulate various fault conditions that may be encountered in actual use.

[0094] In this embodiment, the test case set is determined based on the test scenario and the test data corresponding to each subsystem to be tested in S104, specifically including: for each subsystem to be tested, at each fault node corresponding to a single subsystem to be tested, a test process and a test data test process are designed to trigger a corresponding fault event at a corresponding fault interval time, so as to determine the test case corresponding to each subsystem to be tested. The test case set is determined based on the test case corresponding to each subsystem to be tested.

[0095] In this embodiment, the test case corresponding to the subsystem to be tested implements the test of the test data corresponding to the subsystem to be tested (whether it can be read and written correctly) and the test of the service on the faulty node (whether it can run correctly). In this embodiment, obtaining the test case based on the test data and the test scenario is a key step in the software test design. The following are the detailed steps:

[0096] Design detailed test cases for each test scenario. You can set a test case ID: uniquely identify each test case. Test case name: describe the purpose of the test case. Precondition: the conditions that must be met before the test starts. Expected goal: the expected result after executing the test case. Priority: the importance and urgency of the test case. Test process: describe the specific steps of the test execution in detail, inject fault events into the fault nodes corresponding to the subsystem to be tested to simulate the occurrence of faults, monitor the response and recovery capabilities of the services on the fault nodes, verify whether the services on the fault nodes run correctly when the fault events occur, and verify whether the services on the fault nodes can be automatically recovered.

[0097] For example, the test scenario A corresponding to the distributed coordination system is: Fault type: the second type (network packet loss fault event) and the fourth type (network attack fault event), fault node B, fault interval time: the fault interval time corresponding to the network packet loss fault event is T1, and the fault interval time corresponding to the network attack fault event is T2.

[0098] The corresponding test process is to inject a network packet loss failure event into the faulty node B of the distributed coordination system. After T1 time, a network attack failure event is injected into the faulty node B of the distributed coordination system to monitor whether the service on the faulty node B runs correctly (the expected goal is: yes) and whether it can be automatically recovered (the expected goal is: yes).

[0099] Design test cases based on test data: Test data is an important part of test cases and is used to verify the behavior of the system when processing specific data. The test data in the present invention may include: Normal data: used to verify normal operation. Boundary data: used to test boundary conditions. Abnormal data: used to test the system's ability to handle abnormal inputs. Design test cases based on the test flow of test data.

[0100] Test process of test data: Taking the above example, after injecting network packet loss failure event and network attack failure event into the faulty node B of the distributed coordination system, read and write operations are performed on the test data on the faulty node B to verify whether the read and write operations of the test data can be correctly performed on the faulty node B when the network packet loss failure event and the network attack failure event occur at the same time (the expected goal is: yes).

[0101] In this embodiment, a script can be used to simulate each fault event, and the fault type can be labeled. The label is used to distinguish different fault types, and the label is combined with the fault node to determine the fault event on each fault node and inject the fault event corresponding to the label of the fault node into the fault node in sequence at the fault interval. A fault injection tool (such as Chaos Monkey) can be used to automate the fault injection process.

[0102] In this embodiment, the fault type corresponding to each subsystem under test can be determined according to S1031, the fault type can be tagged, and the fault node determined by S1032 can be combined with the tag to trigger a corresponding fault event at each fault node corresponding to the subsystem under test at the fault interval time.

[0103] The test case generation process diagram is as follows Figure 3As shown. In this embodiment, the structure of the test case includes: each test case should include a description of the test scenario, a verification test process for the service, a test data usage test process, a fault injection test process, expected goal setting and verification methods. The test case can perform read and write verification: verify whether the metadata can be correctly read and written before and after the fault event occurs. Service operation verification: check whether the service on the faulty node can be correctly restored or restarted.

[0104] S105: Execute the test case set to determine whether the test data corresponding to each subsystem under test is read and written correctly, and whether the service on the faulty node corresponding to each subsystem under test is running correctly, so as to determine the test results corresponding to each subsystem under test and the test results corresponding to the system under test.

[0105] In this embodiment, it is also possible to determine whether the service on the faulty node corresponding to each subsystem to be tested can be automatically restored, whether the test data corresponding to each subsystem to be tested is complete and consistent, and so on.

[0106] Test data read and write operation test: Verify whether the test data can be read and written correctly when a fault event occurs. Method: Perform read and write operations when a fault event occurs.

[0107] Test data integrity test: Verify that the test data remains intact during transmission and storage without unexpected changes or loss. Method: Use checksum or hash function to verify the integrity of the test data. Simulate interference during test data transmission (inject fault events).

[0108] Test data consistency test: Ensure that test data is consistent between different nodes or subsystems under test in a distributed environment. Method: Perform test data writing operations on multiple server nodes at the same time to check whether the test data is synchronized and consistent. Use distributed transactions to process test data consistency.

[0109] In this embodiment, the test case set can be executed by running the automated test script. After obtaining the test results corresponding to each subsystem to be tested and the test results corresponding to the system to be tested, the test results can be collected, and the relevant logs can be filtered, and the logs can be filtered and analyzed according to the relevant error keywords, and the test data can be analyzed to obtain the system test evaluation results.

[0110] In this embodiment, a suitable automated test tool, such as Selenium, JUnit, TestNG, etc., can be selected according to the test requirements. According to the test cases and test data, an automated test script is written. Specifically, the conventional technology in the art can be used to write the automated test script. How to generate the automated test script and the script's operating environment configuration can be implemented using the conventional technology in the art. The test script is written using the language and library of the selected tool, and the test case set is executed by running the automated test script, the test results are recorded, and the test results are analyzed to determine whether the test case achieves the expected goal, and the system test evaluation result is obtained.

[0111] Test results can include: Test pass / fail: record whether each test case passes. Execution time: record the execution time of each test case. Exception information: record any exceptions or error information that occurs during the execution process. Performance indicators: such as response time, throughput, etc.

[0112] A large amount of log information will be generated during system operation. In order to effectively analyze these logs, they need to be filtered and analyzed:

[0113] Log Collection: Collect log files from the system.

[0114] Log filtering: Use log analysis tools or scripts to filter logs based on predefined keywords or patterns. For example, you can filter log entries containing keywords such as "ERROR", "EXCEPTION", and "WARNING".

[0115] Log analysis: Analyze filtered logs to identify potential issues and anomalies.

[0116] Test data evaluation:

[0117] Evaluate test data to verify the system's data processing capabilities and accuracy:

[0118] Data integrity: Checks that the metadata is complete and not missing or corrupted.

[0119] Data consistency: Verify that metadata is consistent across all parts of the system.

[0120] Data accuracy: Check that the metadata is as expected and has no errors or biases.

[0121] Result analysis and evaluation:

[0122] Comprehensive test results, log analysis and test data evaluation give the system test evaluation results:

[0123] Functional evaluation: Evaluate whether the system meets all functional requirements.

[0124] Performance evaluation: Evaluate whether the system's performance indicators meet expectations.

[0125] Stability assessment: Evaluate the stability of the system under long-term operation.

[0126] Security assessment: Evaluate the security of the system and check for security vulnerabilities or risks.

[0127] Writing a test report:

[0128] Write detailed test reports summarizing test results and findings:

[0129] Test overview: including test environment, total number of test cases, pass rate, etc.

[0130] Detailed Results: Detailed results for each test case, including pass / fail, execution time, etc.

[0131] Problem Report: Lists all problems found during testing, including anomalies in logs and inconsistencies in test data.

[0132] Improvement suggestions: Make improvement suggestions based on the test results to improve the performance and stability of the system.

[0133] Issue tracking and fixing:

[0134] Log issues found in testing in the issue tracking system and collaborate with the development team to fix them:

[0135] Issue Tracking: Use issue tracking systems (such as JIRA, Bugzilla, etc.) to record and manage issues.

[0136] Fix and regression testing: After the development team fixes the issue, regression testing is performed to verify whether the issue has been resolved.

[0137] Suppose that when testing a distributed file storage system, the following problems are found:

[0138] Log analysis: Multiple "File write failure" errors were found.

[0139] Test data evaluation: It was found that some file data were inconsistent.

[0140] Result analysis: Analyze the logs and test data to determine that the problem may be caused by a storage node failure.

[0141] Based on this information, write test reports, make improvement suggestions, such as increasing the redundancy of storage nodes or improving fault recovery mechanisms, and collaborate with the development team to fix the problems.

[0142] By following these steps, you can ensure that the testing process comprehensively and systematically evaluates all aspects of the system, thereby improving the quality and reliability of the system. Specific embodiment one:

[0144] The following is a specific example of the technical solution of the present invention, in which the system to be tested is a distributed HBase system with three server clusters. When the system to be tested is a distributed HBase system with three server clusters, the specific process of performing chaos testing on the distributed HBase system based on the technical solution of the present invention is as follows:

[0145] Pre-check and split the system under test. The system under test can be split into a distributed coordination service zk cluster, hbas e business system and distributed file storage system hdfs. Obtain the node deployment status of the system under test, such as Figure 4 shown.

[0146] Prepare test data, build metadata tables, and use data generation scripts to write a certain amount of metadata.

[0147] Test scenario determination and test case generation:

[0148] According to the present invention, a preset number of fault types are determined in advance, and the fault types suitable for each subsystem to be tested are screened out for the distributed coordination system zk cluster, the distributed file data storage system hdfs and the business system hbase respectively.

[0149] Combined with the deployment of the system and the fault node selection method, determine the combination of the fault type of the system and each fault node. One-to-one combination, one fault type can correspond to one fault node, or one-to-many combination, one fault type can correspond to multiple fault nodes.

[0150] Determine the failure interval, obtain the sessionTimeout timeout in the NameNode, and calculate the minimum failure interval as 4 / 3*sessionTimeout.

[0151] Finally, the fault injection types and test scenarios of each subsystem under test are shown in Table 2, Table 3, and Table 4 (the labels / abbreviations corresponding to the fault types are ①, ②, ③, and ④, respectively):

[0152] Table 2 ZK cluster failure test scenarios

[0153]

[0154] Table 3 HBase and HDFS fault test scenarios

[0155]

[0156] Table 4 Overall system test scenarios

[0157]

[0158]

[0159] The test scenario is converted into test cases, the test cases are executed, and the test results are collected to ensure that each test result of the system under test meets the expectations of the test scenario, the test passes, and the test efficiency is effectively improved. Specific embodiment 2:

[0161] The following is a specific example of the technical solution of the present invention, in which the system to be tested is a simple HBase system in which the zk cluster and NN are deployed separately. When the system to be tested is a simple HBase system in which the zk cluster and NN are deployed separately, the specific process of performing chaos testing on the simple HBase system in which the zk cluster and NN are deployed separately based on the technical solution of the present invention is as follows:

[0162] Pre-check and split the system to be tested to obtain the node deployment status of the system to be tested, such as Figure 5 As shown, the system includes: a distributed coordination service zk cluster including 3 nodes, and a distributed business service master and standby Nanenode node.

[0163] Prepare test data, build metadata tables, and use data generation scripts to write a certain amount of metadata.

[0164] Test scenario determination and test case generation:

[0165] According to the preset number of fault types predetermined by the present invention, the fault types suitable for each subsystem to be tested are screened out for the distributed coordination system zk cluster and the distributed business system (including NameNode and ZKFC services).

[0166] Combined with the deployment of the system and the fault node selection method, determine the combination of the fault type of the system and each fault node. One-to-one combination, one fault type can correspond to one fault node, or one-to-many combination, one fault type can correspond to multiple fault nodes.

[0167] Determine the failure interval, obtain the sessionTimeout timeout in the NameNode, and calculate the minimum failure interval as 5 / 3*sessionTimeout.

[0168] Finally, the fault injection types and test scenarios of each subsystem under test are shown in Table 5, Table 6, and Table 7 (the labels / abbreviations corresponding to the fault types are ①, ②, ③, and ④, respectively):

[0169] Table 5 NameNode and ZKFC failure test scenarios

[0170]

[0171]

[0172] Table 6 zk cluster failure test scenarios

[0173]

[0174] Table 7 Continuous fault test scenarios

[0175]

[0176]

[0177] The technical solution after determining the test scenario can be found in the above description and will not be elaborated here.

[0178] The present invention can realize the automation of chaos testing under a limited test environment, and provide system test evaluation results through small-scale chaos testing, so as to provide a fault recovery plan for a large-scale online distributed metadata system. The detection and splitting method of the system to be tested proposed in the present invention can achieve more targeted testing for different subsystems to be tested, and effectively improve the test efficiency. The three-step method of fault type classification, fault node selection, and fault interval time calculation proposed in the present invention can formulate test scenarios more efficiently and comprehensively. The present invention has a wide range of applications, and is not only suitable for distributed metadata system testing, but also can be used for various self-developed and open source distributed storage system testing. Compared with traditional technologies, the main creativity lies in: proposing a distributed metadata system splitting strategy. Proposing a chaos test fault type classification method. Proposing a fault interval time prediction algorithm based on distributed coordination services. Proposing an automated chaos testing device, including automation such as detection and splitting of the system to be tested, test case scenario calculation, and test case generation and execution.

[0179] The present invention provides a set of automated chaos testing devices for distributed metadata systems, which realizes automation of system detection splitting, test scenario determination, test case generation and execution, and automated test result analysis and collection, and can effectively improve test efficiency. Only the deployment of the system to be tested needs to be given to realize the calculation and generation of chaos test cases. The device is not only applicable to open source distributed metadata systems, but also to self-developed distributed storage systems, and has a wide range of applications.

[0180] It should be noted that the contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0181] In this embodiment, a system-based chaos testing device is also provided, and a single device is used to implement the above-mentioned embodiments and optional implementation methods, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0182] Figure 6 It is a structural schematic diagram of a system-based chaos testing device according to an embodiment of the present invention.

[0183] The present invention provides a system-based chaos testing device, such as Figure 6 As shown, the system-based chaos testing device includes:

[0184] The first processing module 11 is used to split the system under test into multiple subsystems under test. The system under test is a distributed metadata system, and the multiple subsystems under test include a distributed coordination system, a distributed file data storage system, and a business system.

[0185] The second processing module 12 is used to determine the test data corresponding to each subsystem to be tested.

[0186] The third processing module 13 is used to determine the test scenario corresponding to each subsystem to be tested. The test scenario includes the fault type, fault node and fault interval time.

[0187] The fourth processing module 14 is used to determine a test case set based on the test scenario and the corresponding test data corresponding to each subsystem to be tested. The test case set includes test cases corresponding to each subsystem to be tested and test cases corresponding to the system to be tested. The test cases corresponding to each subsystem to be tested are used to perform chaos testing on each subsystem to be tested, and the test cases corresponding to the system to be tested are used to perform chaos testing on the system to be tested.

[0188] The fifth processing module 15 is used to execute the test case set to determine whether the test data corresponding to each subsystem under test is read and written correctly, and whether the service on the fault node corresponding to each subsystem under test is executed correctly, so as to determine the test results corresponding to each subsystem under test and the test results corresponding to the system under test.

[0189] In an optional embodiment, the system-based chaos testing device also includes: a fault type preset module, which is used to predetermine a preset number of fault types before determining the test scenario corresponding to each subsystem to be tested. The preset number of fault types include a first type, a second type, a third type, and a fourth type. The fault event corresponding to the first type is a complete fault, the fault event corresponding to the second type is a partial communication fault, the fault event corresponding to the third type is a service interruption fault, and the fault event corresponding to the fourth type is an intermittent fault. Each fault type includes at least one fault event.

[0190] In an optional embodiment, the second processing module 12 includes:

[0191] The first processing unit is used to determine at least one fault type corresponding to each subsystem to be tested. Each fault type in the at least one fault type includes at least one fault event.

[0192] The second processing unit is used to determine at least one faulty node corresponding to each subsystem to be tested, wherein each of the at least one faulty node corresponds to at least one fault event.

[0193] The third processing unit is used to determine at least one fault interval time corresponding to each subsystem to be tested, wherein each fault interval time in the at least one fault interval time corresponds to a fault event.

[0194] In an optional implementation, the second processing unit is specifically configured to determine a minimum number of processes of the distributed coordination system, and randomly select a preset number of server nodes as faulty nodes in the distributed coordination system according to the minimum number of processes. The preset number of nodes is greater than or equal to 1 and less than or equal to the minimum number of processes.

[0195] In the distributed file data storage system, a server node is randomly selected as the fault node.

[0196] At least one server node is selected in the business system as a fault node according to the upper-layer business model.

[0197] In an optional implementation manner, the third processing unit is specifically configured to determine at least one fault interval time corresponding to each subsystem to be tested, including:

[0198] For each subsystem to be tested, the fault interval time corresponding to each fault event in each fault type under the single subsystem to be tested is determined according to the timeout time of the client corresponding to the single subsystem to be tested and the maximum time interval for communication between the client and the server, so as to determine the fault interval time corresponding to each fault event in each fault type under each subsystem to be tested.

[0199] In an optional implementation, the fourth processing module 14 is specifically used to design a test process and a test process of test data for each fault event that triggers the corresponding fault interval time at each fault node corresponding to the single subsystem to be tested, so as to determine the test case corresponding to each subsystem to be tested. The test case set is determined according to the test case corresponding to each subsystem to be tested.

[0200] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0201] The system-based chaos testing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0202] The embodiment of the present invention also provides a computer device having the above Figure 6 The system-based chaos testing setup shown.

[0203] See also Figure 7 , Figure 7 Schematic diagram of the hardware structure of the computer device according to the embodiment of the present invention. Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In an optional embodiment, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor device). Figure 7 A processor 10 is taken as an example.

[0204] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0205] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0206] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating device, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In an optional embodiment, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0207] The memory 20 may include a volatile memory, such as a random access memory. The memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive. The memory 20 may also include a combination of the above-mentioned types of memory.

[0208] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0209] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0210] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0211] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A system-based chaos testing method, applied to the system characterized by: include: Split the system under test into multiple subsystems under test; The system to be tested is a distributed metadata system, and the plurality of subsystems to be tested include a distributed coordination system, a distributed file data storage system and a business system; Determine the test data corresponding to each of the subsystems to be tested; Determine a test scenario corresponding to each of the subsystems to be tested; the test scenario includes a fault type, a fault node, and a fault interval; Determine a test case set based on the test scenario and the corresponding test data corresponding to each of the subsystems to be tested; the test case set includes the test case corresponding to each of the subsystems to be tested and the test case corresponding to the system to be tested; The test cases corresponding to each of the subsystems to be tested are respectively used to perform chaos testing on each of the subsystems to be tested, and the test cases corresponding to the system to be tested are used to perform chaos testing on the system to be tested; Execute the test case set to determine whether the test data corresponding to each subsystem under test is read and written correctly, and whether the service on the faulty node corresponding to each subsystem under test is running correctly, so as to determine the test results corresponding to each subsystem under test and the test results corresponding to the system under test.

2. The method according to claim 1, characterized in that Before determining the test scenario corresponding to each of the subsystems to be tested, the method further includes: A preset number of fault types are predetermined; the preset number of fault types include a first type, a second type, a third type and a fourth type; the fault event corresponding to the first type is a complete fault, the fault event corresponding to the second type is a partial communication fault, the fault event corresponding to the third type is a service interruption fault, and the fault event corresponding to the fourth type is an intermittent fault; each fault type includes at least one fault event.

3. The method according to claim 2, characterized in that The determining of the test scenario corresponding to each of the subsystems to be tested includes: Determine at least one fault type corresponding to each of the subsystems to be tested; each of the at least one fault type includes at least one fault event; Determine at least one fault node corresponding to each of the subsystems to be tested; each of the at least one fault node corresponds to at least one fault event; At least one fault interval time corresponding to each of the subsystems to be tested is determined; each fault interval time in the at least one fault interval time corresponds to a fault event.

4. The method according to claim 3, characterized in that The determining of at least one fault node corresponding to each of the subsystems to be tested includes: Determine the minimum number of processes of the distributed coordination system, and randomly select a preset number of server nodes as fault nodes in the distributed coordination system according to the minimum number of processes; the preset number of nodes is greater than or equal to 1 and less than or equal to the minimum number of processes; Randomly select a server node in the distributed file data storage system as a fault node; At least one server node is selected as a fault node in the service system according to an upper-layer service model.

5. The method according to claim 3, characterized in that: The determining of at least one fault interval time corresponding to each of the subsystems to be tested comprises: For each of the subsystems to be tested, the fault interval time corresponding to each fault event in each fault type under the single subsystem to be tested is determined based on the timeout time of the client corresponding to the single subsystem to be tested and the maximum time interval for communication between the client and the server, so as to determine the fault interval time corresponding to each fault event in each fault type under each subsystem to be tested.

6. The method according to claim 3, characterized in that The determining of a test case set based on the test scenario and the test data corresponding to each of the subsystems to be tested includes: For each of the subsystems to be tested, a test process and a test data test process are designed to trigger a corresponding fault event at a corresponding fault interval time at each fault node corresponding to a single subsystem to be tested, so as to determine a test case corresponding to each of the subsystems to be tested; The test case set is determined according to the test case corresponding to each of the subsystems to be tested.

7. A system-based chaos testing device, characterized in that: include: A first processing module, used for splitting the system to be tested into multiple subsystems to be tested; The system to be tested is a distributed metadata system, and the plurality of subsystems to be tested include a distributed coordination system, a distributed file data storage system and a business system; A second processing module, used to determine the test data corresponding to each of the subsystems to be tested; A third processing module is used to determine the test scenario corresponding to each of the subsystems to be tested; The test scenario includes fault type, fault node and fault interval time; A fourth processing module is used to determine a test case set based on the test scenario and the corresponding test data corresponding to each of the subsystems to be tested; the test case set includes a test case corresponding to each of the subsystems to be tested and a test case corresponding to the system to be tested; The test cases corresponding to each of the subsystems to be tested are respectively used to perform chaos testing on each of the subsystems to be tested, and the test cases corresponding to the system to be tested are used to perform chaos testing on the system to be tested; The fifth processing module is used to execute the test case set to determine whether the test data corresponding to each of the subsystems to be tested is read and written correctly, and whether the services on the faulty nodes corresponding to each of the subsystems to be tested are executed correctly, so as to determine the test results corresponding to each of the subsystems to be tested and the test results corresponding to the system to be tested.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the system-based chaos testing method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the system-based chaos testing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the system-based chaos testing method according to any one of claims 1 to 6.

Images