Distributed system test method and device, computer equipment, medium and program product

Abstract

The invention relates to a distributed system testing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that a plurality of explosion factors are determined based on a target distributed system, and each explosion factor carries fault information; for the current test, screening out a current explosion factor from a plurality of explosion factors stored in a current explosion warehouse; on the basis of the fault information of the current explosion factor, under the condition that the current explosion factor passes safety field verification, the current explosion factor is injected into the target distributed system for current testing, and a testing result is obtained; and based on a test result, updating each explosion factor stored in the current explosion warehouse, taking the updated explosion warehouse as the current explosion warehouse corresponding to the next test, and returning to the step of screening out the current explosion factor from the plurality of explosion factors stored in the current explosion warehouse for continuous execution. And the test of the distributed system is ended until the stop condition is met, so that the test efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of system testing, and in particular to a distributed system testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of digital technology, the growing business needs and the increasing demand for system scalability have given rise to distributed systems. While providing benefits such as high availability, high reliability, and high scalability, distributed systems also face a series of problems such as complexity, security, and performance. How to test distributed systems has become a challenging issue.

[0003] In related technologies, faults injected into distributed systems are determined through manual experience or a large number of experiments to perform fault analysis on distributed systems. However, most faults are randomly generated or traversed. As the test progresses or the system version continues to iterate, the test value of faults determined manually or experimentally becomes lower and lower, resulting in more and more invalid tests and low distributed system test efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a distributed system testing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the distributed system testing efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a distributed system testing method, comprising:

[0006] Based on the target distributed system, multiple explosion factors are determined, each of which carries fault information;

[0007] For the current test, a current explosion warehouse corresponding to the current test is obtained, and a current explosion factor is screened out from a plurality of explosion factors stored in the current explosion warehouse;

[0008] In the case where it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system to perform a current test and obtain a test result;

[0009] Based on the test result, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue execution until the stop condition is met and the test of the distributed system is terminated.

[0010] In a second aspect, the present application also provides a distributed system testing device, including:

[0011] A determination module, used for determining a plurality of explosion factors based on a target distributed system, each explosion factor carrying fault information;

[0012] A screening module, used for obtaining a current explosion warehouse corresponding to the current test, and screening out a current explosion factor from a plurality of explosion factors stored in the current explosion warehouse;

[0013] A test module, for injecting the current explosion factor into the target distributed system for current testing to obtain a test result when it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification;

[0014] The end module is used to update the explosion factors stored in the current explosion warehouse based on the test results, and use the updated explosion warehouse as the current explosion warehouse corresponding to the next test, and return to the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse to continue executing until the stop condition is met to end the test of the distributed system.

[0015] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Based on the target distributed system, multiple explosion factors are determined, each of which carries fault information;

[0017] For the current test, a current explosion warehouse corresponding to the current test is obtained, and a current explosion factor is screened out from a plurality of explosion factors stored in the current explosion warehouse;

[0018] In the case where it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system to perform a current test and obtain a test result;

[0019] Based on the test result, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue execution until the stop condition is met and the test of the distributed system is terminated.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0021] Based on the target distributed system, multiple explosion factors are determined, each of which carries fault information;

[0022] For the current test, a current explosion warehouse corresponding to the current test is obtained, and a current explosion factor is screened out from a plurality of explosion factors stored in the current explosion warehouse;

[0023] In the case where it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system to perform a current test and obtain a test result;

[0024] Based on the test result, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue execution until the stop condition is met and the test of the distributed system is terminated.

[0025] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0026] Based on the target distributed system, multiple explosion factors are determined, each of which carries fault information;

[0027] For the current test, a current explosion warehouse corresponding to the current test is obtained, and a current explosion factor is screened out from a plurality of explosion factors stored in the current explosion warehouse;

[0028] In the case where it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system to perform a current test and obtain a test result;

[0029] Based on the test result, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue execution until the stop condition is met and the test of the distributed system is terminated.

[0030] The above-mentioned distributed system testing method, device, computer equipment, computer-readable storage medium and computer program product determine multiple explosion factors based on the target distributed system, each explosion factor carries fault information. That is, multiple faults for the target distributed system are predetermined. For the current test, the current explosion warehouse corresponding to the current test is obtained, and the current explosion factor is screened out from the multiple explosion factors stored in the current explosion warehouse; when it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system for the current test to obtain the test result. That is, before being injected into the target distributed system, the current explosion factor itself is verified in advance to ensure the effectiveness and accuracy of subsequent tests. Based on the test results, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue to execute until the test of the distributed system is terminated when the stop condition is met. That is, through multiple iterations to continuously update the explosion factor, it is possible to achieve more and more accurate testing of the distributed system, avoid invalid testing, and thus improve the testing efficiency of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 An application environment diagram of a distributed system testing method in one embodiment;

[0033] Figure 2 A schematic diagram of a flow chart of a distributed system testing method in one embodiment;

[0034] Figure 3 A schematic diagram of a test flow in an embodiment;

[0035] Figure 4 is a structural block diagram of a testing device for a distributed system in one embodiment;

[0036] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] Before introducing this application, the following terms are explained:

[0039] Blast factor (BF): The fault injected into a distributed system.

[0040] Blast radius (br): the impact of one or more blast factors on a distributed system.

[0041] Blast warehouse (BW for short): a warehouse that stores explosive factors and the explosion radius they cause.

[0042] Security area (SA): a set of rules for checking explosion factors. After one or more explosion factors that pass the check of these rules are injected into the distributed system, they will not theoretically have an unacceptable impact on the distributed system.

[0043] The distributed system testing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the computer device 102 communicates with the target distributed system 104 through a network.

[0044] In some embodiments, the computer device 102 determines the target distributed system 104 currently being tested, and based on the target distributed system, determines multiple explosion factors, each of which carries fault information; for the current test, obtains the current explosion warehouse corresponding to the current test, and selects the current explosion factor from the multiple explosion factors stored in the current explosion warehouse; in the case where it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system 104 to instruct the target distributed system 104 to perform the current test based on the injected current explosion factor. The computer device 102 obtains the test result of the current test, and based on the test result, the computer device 102 updates each explosion factor stored in the current explosion warehouse, and uses the updated explosion warehouse as the current explosion warehouse corresponding to the next test, and returns to the step of selecting the current explosion factor from the multiple explosion factors stored in the current explosion warehouse to continue execution until the test of the distributed system is terminated when the stop condition is met.

[0045] The computer device 102 may be a terminal or a server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0046] In an exemplary embodiment, Figure 2 As shown, a distributed system testing method is provided, which is applied to Figure 1 The computer device 102 in the example is used as an example to illustrate, including the following steps S202 to S208. Among them:

[0047] Step S202: determining a plurality of explosion factors based on the target distributed system, each explosion factor carrying fault information.

[0048] Among them, the target distributed system is the distributed system currently being tested. The explosion factor can be regarded as a fault. The fault information carried by each explosion factor can be understood as the attribute information of the fault, including but not limited to the fault type, fault object node, fault duration, fault damage degree, etc. Different explosion factors carry different fault information. Among them, the fault type includes basic resource type, intermediate type, and container type. Basic resource types include CPU (Central Processing Unit, central processing unit), network, memory, hard disk, etc., middleware types include JVM (a virtual machine), nginx (a proxy service), http (Hypertext Transfer Protocol), etc., and container types include Docker (open source application container engine), K8S (open source container deployment and management platform), etc.

[0049] Optionally, the computer device selects a plurality of matching explosion factors from pre-set explosion factors based on system characteristics of the target distributed system.

[0050] Exemplarily, the computer device obtains multiple explosion factors matching the system characteristics of the current system version from the explosion warehouse finally updated by the previous system version test according to the system characteristics of the target distributed system of the current system version.

[0051] Step S204: for the current test, obtain the current explosion warehouse corresponding to the current test, and select the current explosion factor from the multiple explosion factors stored in the current explosion warehouse.

[0052] Among them, the current explosion warehouse refers to the explosion warehouse used in the current test. The explosion warehouse includes multiple information rows, each of which includes explosion factor, explosion radius, number of impacts (number of times screened), impact range (quantifying the impact of the explosion factor on the target distributed system), etc. Each explosion factor is distinguished by fault information. For example, the fault information is used as the name of the explosion factor. The information row is used to record the explosion radius of a certain explosion factor when a certain explosion factor produces a certain impact range. The explosion warehouses corresponding to different test times are different. The current explosion warehouse is obtained by updating the warehouse of the previous explosion warehouse of the previous test.

[0053] Optionally, if the current test is the first test, the computer device generates an initial explosion warehouse based on the initial parameters of each explosion factor, and randomly selects one explosion factor from multiple explosion factors in the initial explosion warehouse as the current explosion factor for the current test.

[0054] For example, the initial explosion warehouse is shown in Table 1:

[0055] Table 1 Table corresponding to the initial explosion warehouse

[0056]

[0057] Table 1 mainly illustrates four explosion factors, namely, A node power failure (bf1), A node network failure (bf2), B node restart (bf3), C node disk occupancy (bf4). Among them, A node power failure (bf1), A node network failure (bf2), B node restart (bf3), C node disk occupancy (bf4) have the same initial parameters, that is, the influence range of each explosion factor is set to be the same, such as the value 1, and the influence times of each explosion factor are set to be the same, such as the value 1. In addition, due to the explosion radius The number of impacts and Scope of Influence about, that is For example, using formula (1) to visualize the explosion radius br:

[0058] (1)

[0059] In the above formula (1), n ​​is the total number of information rows in the explosion warehouse. and are weight factors, which can be set based on experience. For example, , , that is, the weight of the impact range is considered to be higher than the number of impacts. Based on this, in the first test, according to formula (1), it can be calculated that the radius of each explosion factor is equal, that is, they are all equal to the value r.

[0060] It should be noted that the larger the explosion radius, the greater the probability of being selected. Therefore, it can be understood that in the first test, the probability of each explosion factor being selected is the same.

[0061] Optionally, if the current test is not the first test, the computer device uses the explosion warehouse updated in the previous test as the current explosion warehouse for the current test, and randomly selects an explosion factor from multiple explosion factors as the current explosion factor according to the explosion radius of each explosion factor in the current explosion warehouse.

[0062] Step S206, when it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system to perform the current test and obtain the test result.

[0063] Among them, the safety field contains multiple preset verification rules, and the verification rules represent non-compliant fault information. The safety field can also be regarded as the inspection rules of the explosion factor. For example, taking the architecture of a 3-node distributed database HBase (a distributed, scalable database) + a distributed file system HDFS (a system used to solve the problem of massive data file storage) as an example, the architecture requires: a. At least 1 namenode service (responsible for managing the file system's namespace and storing metadata) and at least 1 zkfc service (used for preparation work selection and failover) to survive; b. At least 2 zk services (open source distributed coordination services) to survive; c. At least 2 journalnode services (used for sharing edit logs) to survive. Set the corresponding verification rules based on this requirement. The safety field is used to ensure that after one or more explosion factors are injected into the target distributed system, the target distributed system operates within an acceptable range of influence. It can be understood that after the current explosion factor passes the safety field verification, it can be considered that the current explosion factor can be fault-injected into the target distributed system and should not have unacceptable impact. Each test is a chaos test on the distributed system.

[0064] Exemplarily, the computer device obtains the security field, and traverses all verification rules in the security field to obtain a traversal result, and the traversal result is used to indicate whether the security field verification is passed.

[0065] Exemplarily, when the computer device determines that the current explosion factor does not pass the safety field check based on the fault information of the current explosion factor, it re-screens another explosion factor from the current explosion warehouse and verifies whether the other explosion factor satisfies the safety field check. If so, the other explosion factor is injected into the target distributed system for the current test to obtain the test result.

[0066] Step S208, based on the test results, the explosion factors stored in the current explosion warehouse are updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of selecting the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue execution until the test of the distributed system is terminated when the stop condition is met.

[0067] The test results include the type of impact of the injected current explosion factor on the target distributed system. The impact types include but are not limited to service availability, data persistence, performance, compatibility, security, usability, management functions, UI (User Interface) display, etc. Different impact types have corresponding impact ranges.

[0068] Optionally, after obtaining the test result, the computer device analyzes the test result to determine the impact type of the impact of the current explosion factor on the target distributed system after being injected into the target distributed system. The computer device updates the current explosion warehouse according to the impact type and the preset update rule to obtain an updated explosion warehouse. If the number of current tests does not reach the preset number of times, or the test duration does not reach the preset duration, it is determined that the stop condition is not met, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of selecting the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue execution until the test of the distributed system is terminated when the stop condition is met.

[0069] It should be noted that, according to formula (1), changes in the number of impacts and the impact range of an explosion factor will lead to changes in the explosion radius of all explosion factors in the explosion warehouse. Based on this, the current explosion warehouse is updated according to the impact range corresponding to the current explosion factor.

[0070] In the above distributed system testing method, multiple explosion factors are determined based on the target distributed system, and each explosion factor carries fault information. That is, multiple faults for the target distributed system are predetermined. For the current test, the current explosion warehouse corresponding to the current test is obtained, and the current explosion factor is screened out from the multiple explosion factors stored in the current explosion warehouse; when it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system for the current test to obtain the test result. That is, before being injected into the target distributed system, the current explosion factor itself is verified in advance to ensure the effectiveness and accuracy of subsequent tests. Based on the test results, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue to execute until the test of the distributed system is terminated when the stop condition is met. That is, through multiple iterations to continuously update the explosion factor, more and more accurate testing of the distributed system is achieved, invalid testing is avoided, and the test efficiency of the distributed system can be improved.

[0071] In some embodiments, selecting the current explosion factor from multiple explosion factors stored in the current explosion warehouse includes: obtaining the selection probability of each explosion factor from the current explosion warehouse; and selecting the current explosion factor from multiple explosion factors stored in the current explosion warehouse based on the selection probability of each explosion factor.

[0072] Each information row in the explosion warehouse also includes a corresponding selection probability. The larger the explosion radius of the information row, the greater the selection probability of the explosion factor of the information row.

[0073] Exemplarily, the computer device superimposes the sum of the explosion radii in the current explosion warehouse, and calculates the ratio between each explosion radius and the sum to obtain the selection probability of the corresponding explosion factor. The computer device randomly generates a random number, and based on the random number and the selection probability of each explosion factor, randomly selects a current explosion factor from the explosion factors of multiple information rows stored in the current explosion warehouse.

[0074] Of course, in other embodiments, multiple current explosion factors may be screened out and injected into the target distributed system together to obtain test results.

[0075] For example, the following steps are used to calculate the probability of selecting explosion factor i for information row i: :Step c. Calculate the total explosion radius ; Step d. Calculate the selection probability: .in, is the explosion radius of explosion factor i.

[0076] In this embodiment, according to the selection probability of each explosion factor in the current explosion warehouse, the corresponding current explosion factor can be quickly selected therefrom, thereby ensuring the test efficiency.

[0077] In some embodiments, based on the selection probability of each explosion factor, the current explosion factor is screened out from multiple explosion factors stored in the current explosion warehouse, including: randomly selecting a probability from a preset probability range as a target probability; based on the selection probability of each explosion factor, a first selection is made from multiple explosion factors stored in the current explosion warehouse, if the selection probability of the first selected explosion factor is less than the target probability, a second selection is made from multiple explosion factors in the current explosion warehouse; if the sum of the selection probability of the second selected explosion factor and the selection probability of the first selected explosion factor is greater than or equal to the target probability, the second selected explosion factor is used as the current explosion factor.

[0078] Exemplarily, a random number r in the interval [0,1) is randomly generated, and the random number r is used as the target probability.

[0079] Exemplarily, based on the selection probability of each explosion factor, a first selection is made from multiple explosion factors stored in the current explosion warehouse. If the selection probability of the first selected explosion factor is less than the target probability, a second selection is made from multiple explosion factors in the current explosion warehouse. If the sum of the selection probability of the second selected explosion factor and the selection probability of the first selected explosion factor (the sum obtained by the second selection) is greater than or equal to the target probability, the second selected explosion factor is used as the current explosion factor.

[0080] Exemplarily, if the sum obtained by the second selection is less than the target probability, a third selection is made from multiple explosion factors in the current explosion warehouse. The selection probability of the explosion factor selected for the third time is superimposed with the sum obtained by the second selection to obtain the sum obtained by the third selection. If the sum obtained by the third selection is greater than or equal to the target probability, the explosion factor selected for the third time is used as the current explosion factor. If the sum obtained by the third selection is less than the target probability, continue to traverse at least one selection from the current explosion warehouse until the sum of the selection probability of the last selection and the sum corresponding to the previous selection is greater than or equal to the target probability, and the explosion factor selected for the last selection is used as the current explosion factor.

[0081] In this embodiment, at least one explosion factor is selected from the current explosion warehouse through the various selection probabilities and target probabilities stored in the current explosion warehouse. In this way, it can be ensured that the explosion factor with a larger explosion radius has a greater probability of being selected, that is, the explosion factor with a greater impact is more easily selected and injected into the target distributed system for chaos testing, which can more effectively test the performance of the target distributed system.

[0082] In some embodiments, the method also includes: obtaining multiple verification rules contained in the safety place; for each verification rule, determining whether the fault information of the current explosion factor hits the verification rule, and obtaining the verification result of the verification rule; if the verification result of each verification rule indicates a miss, determining that the current explosion factor passes the safety field verification; if there is at least one verification rule whose verification result indicates a hit, determining that the current explosion factor does not pass the safety field verification.

[0083] For example, the target distributed system is a 3-node cluster distributed system. As shown in Table 2, the following explosion factors are configured:

[0084] Table 2 Explosion factor list

[0085]

[0086] The corresponding security field has the following verification rules: Verification rule 1: Node A and Node B cannot be rebooted (server restarted) at the same time; Verification rule 2: Node A and Node B cannot be ifdown (disconnected) at the same time.

[0087] Exemplarily, referring to Table 2 above, for a certain explosion factor: node ABCreboot (server restart occurs at node A, node B, and node C), the above two verification rules are traversed one by one, and none of them are hit, thereby confirming that the explosion factor passes the safety field verification.

[0088] In this embodiment, according to the verification rules set in the safety place, the explosion factor itself can be verified for compliance in advance to ensure the effectiveness and accuracy of subsequent tests.

[0089] In some embodiments, based on the test results, each explosion factor stored in the current explosion warehouse is updated, including: based on the test results, determining the current impact type of the current explosion factor on the target distributed system, and obtaining the mapping relationship between the impact type and the impact range; based on the mapping relationship and the current impact type, determining the current impact range caused by the current explosion factor; based on the current impact range caused by the current explosion factor, updating each explosion factor in the current explosion warehouse to obtain an updated explosion warehouse.

[0090] Exemplarily, the current impact range determined by the test result is determined as the target impact range, and the target impact range and the current explosion factor are used as the target combination. It is queried whether there is an information row containing the target combination in the current explosion warehouse. If so, the number of impacts in the queried information row is updated to complete the update of the information row. Based on the updated information row, the explosion radius in all information rows is updated to obtain an updated explosion warehouse. If not, a new information row is created, and the impact range and explosion factor in the new information row are set to the target impact range and the current explosion factor, and the number of impacts is set to 1. Based on the impact ranges and impact times of all information rows, the explosion radius of all information rows is updated to obtain an updated explosion warehouse.

[0091] Exemplarily, the impact type generated by the current explosion factor bfx is P1, and the corresponding current impact range is m. The following update rules can be used for updating: Based on m and bfx, query whether there is a corresponding information row in the current explosion warehouse (that is, the impact range and explosion factor in the information row are m and bfx respectively). If it exists, add 1 to the impact number in the queried information row to update the impact number, and update the explosion radius of all information rows in the current explosion warehouse based on the updated impact number and m. If it does not exist, create a new information row in the current explosion warehouse, fill in the explosion factor, impact range, and impact number of the new information row with bfx, m, and 1 respectively, and update the explosion radius of the explosion factor of all information rows based on the impact range and impact number of the new information row and other information rows, and obtain the updated explosion warehouse.

[0092] For example, the mapping relationship may be presented in a table, such as Table 3:

[0093] Table 3: Mapping table

[0094]

[0095] In this embodiment, the current impact type of the impact produced by the current explosion factor is determined according to the test results. Thus, according to the mapping relationship between the impact type and the impact range, the impact can be visualized to obtain the current impact range, thereby visualizing the explosion radius, achieving accurate testing, and greatly improving the efficiency of chaos testing.

[0096] In one embodiment, the method further includes: obtaining an updated explosion warehouse obtained when the stop condition is met, and determining a maximum explosion radius from the explosion radii corresponding to each explosion factor in the updated explosion warehouse; using the explosion factor corresponding to the maximum explosion radius as a target explosion factor, and the target explosion factor is the explosion factor with the greatest impact on the target distributed system.

[0097] Exemplarily, the updated explosion warehouse obtained when the stop condition is met is used as the target explosion warehouse, and the explosion radius of each information row in the target explosion warehouse is traversed to obtain the maximum explosion radius. The explosion factor of the information row where the maximum explosion radius is located is used as the target explosion factor.

[0098] For example, take Table 2 as an example to construct the explosion factor and obtain the initial explosion warehouse, as shown in Table 4:

[0099] Table 4 Initial explosion warehouse example table

[0100]

[0101] After multiple tests, the target explosion warehouse is obtained. It should be noted that, for the convenience of comparison, an additional column can be added to the target explosion warehouse to show the selection probability of each information row, as shown in Table 5:

[0102] Table 5 Target explosion warehouse

[0103]

[0104] Based on Table 5, the maximum selection probability is 0.097, and the corresponding explosion factor A reboot is used as the target explosion factor.

[0105] In this embodiment, the explosion warehouse is continuously updated through multiple iterations to optimize the explosion factor, so that the target distributed system can be tested more accurately, that is, a more accurate explosion radius can be obtained. Therefore, after completing the test of the target distributed system, according to the maximum explosion radius in the explosion warehouse obtained by the last update, the fault that has the greatest impact on the target distributed system can be more intuitively and accurately displayed, which is convenient for making accurate decisions on the target distributed system.

[0106] In a specific embodiment, the testing of distributed systems can be applied to automated testing and machine learning. In this way, existing test data can be used more effectively to optimize the testing and training process, and can also be integrated into the CI / CD process to achieve standardization, automation, and rapid iteration of testing and training. The following uses the application of automated testing and machine learning as an example. Figure 3 The following is a schematic diagram of a test process in an embodiment. The specific steps are as follows:

[0107] Step 1: After starting the test of the target distributed system, determine whether configuration is required, that is, whether there is an explosion factor, explosion radius, and safety field. If not, execute the following step 3; if required, execute step 2.

[0108] Step 2: Construct the explosion factor and the initial explosion radius to build the initial explosion warehouse. See the previous article for details.

[0109] Of course, in this application, more sophisticated explosion factors can be defined. For example, if the packet loss rate of a node A fails, multiple explosion factors can be defined. See Table 6 for details:

[0110] Table 6: Example table of refined explosion factors

[0111]

[0112] Of course, we can also quantify the impact range in a more detailed way, such as the active / standby switchover time, as shown in Table 7:

[0113] Table 7: Refined mapping table

[0114]

[0115] After determining the explosion factor and initial explosion radius, construct the initial explosion warehouse. Then proceed to step 3.

[0116] Step 4: Execute the test.

[0117] Specifically, step 4.1: for the current test, obtain the current explosion warehouse corresponding to the current test, and obtain the selection probability of each explosion factor from the current explosion warehouse; randomly select a probability from the preset probability range as the target probability; based on the selection probability of each explosion factor, make a first selection from multiple explosion factors stored in the current explosion warehouse, if the selection probability of the first selected explosion factor is less than the target probability, then make a second selection from multiple explosion factors in the current explosion warehouse; if the sum of the selection probability of the second selected explosion factor and the selection probability of the first selected explosion factor is greater than or equal to the target probability, then use the second selected explosion factor as the current explosion factor.

[0118] Step 4.2: Obtain multiple verification rules contained in the safety field; for each verification rule, determine whether the fault information of the current explosion factor hits the verification rule, and obtain the verification result of the verification rule; if the verification result of each verification rule indicates a miss, it is determined that the current explosion factor passes the safety field verification; if there is at least one verification rule whose verification result indicates a hit, it is determined that the current explosion factor does not pass the safety field verification.

[0119] Step 4.3: When the current explosion factor is determined to have passed the safety field verification based on the fault information of the current explosion factor, the current explosion factor is injected into the target distributed system for current testing to obtain the test results. Based on the test results, the current impact type of the current explosion factor on the target distributed system is determined, and the mapping relationship between the impact type and the impact range is obtained; based on the mapping relationship and the current impact type, the current impact range caused by the current explosion factor is determined; based on the current impact range caused by the current explosion factor, each explosion factor in the current explosion warehouse is updated to obtain an updated explosion warehouse.

[0120] Step 4.4: Determine whether to continue the test. If yes, use the updated explosion warehouse as the current explosion warehouse corresponding to the next test, and return to the step of selecting the current explosion factor from the multiple explosion factors stored in the current explosion warehouse to continue the execution. If no, end the test of the distributed system.

[0121] In this embodiment, multiple explosion factors are determined based on the target distributed system, and each explosion factor carries fault information. That is, multiple faults for the target distributed system are predetermined. For the current test, the current explosion warehouse corresponding to the current test is obtained, and the current explosion factor is screened out from the multiple explosion factors stored in the current explosion warehouse; when it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system for the current test to obtain the test result. That is, before being injected into the target distributed system, the current explosion factor itself is verified in advance to ensure the effectiveness and accuracy of subsequent tests. Based on the test results, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue to execute until the test of the distributed system is terminated when the stop condition is met. That is, through multiple iterations to continuously update the explosion factor, more and more accurate testing of the distributed system is achieved, invalid testing is avoided, and the test efficiency of the distributed system can be improved. In addition, the entire process does not require human intervention, avoiding human subjective factors and random collision factors, effectively utilizing data-driven testing, reducing time costs, optimizing product processes, and improving product quality.

[0122] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0123] Based on the same inventive concept, the embodiment of the present application also provides a distributed system testing device for implementing the distributed system testing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the testing device for one or more distributed systems provided below can refer to the limitations of the distributed system testing method above, and will not be repeated here.

[0124] In an exemplary embodiment, Figure 4 As shown, a distributed system testing device 400 is provided, including: a determination module 402, a screening module 404, a testing module 406 and an ending module 408, wherein:

[0125] A determination module 402 is used to determine a plurality of explosion factors based on a target distributed system, each explosion factor carrying fault information;

[0126] A screening module 404 is used to obtain a current explosion warehouse corresponding to the current test, and screen out a current explosion factor from a plurality of explosion factors stored in the current explosion warehouse;

[0127] A test module 406 is used to inject the current explosion factor into the target distributed system to perform a current test and obtain a test result when it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification;

[0128] End module 408 is used to update each explosion factor stored in the current explosion warehouse based on the test results, and use the updated explosion warehouse as the current explosion warehouse corresponding to the next test, and return to the step of filtering out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse to continue executing until the stop condition is met to end the test of the distributed system.

[0129] In some embodiments, the screening module 404 is used to obtain the selection probability of each explosion factor from the current explosion warehouse; based on the selection probability of each explosion factor, screen the current explosion factor from the multiple explosion factors stored in the current explosion warehouse.

[0130] In some embodiments, the screening module 404 is used to randomly select a probability from a preset probability range as a target probability; based on the selection probability of each explosion factor, a first selection is made from multiple explosion factors stored in the current explosion warehouse; if the selection probability of the first selected explosion factor is less than the target probability, a second selection is made from multiple explosion factors in the current explosion warehouse; if the sum of the selection probability of the second selected explosion factor and the selection probability of the first selected explosion factor is greater than or equal to the target probability, the second selected explosion factor is used as the current explosion factor.

[0131] In some embodiments, the device also includes a judgment module, which is used to obtain multiple verification rules contained in the safety field; for each verification rule, it is judged whether the fault information of the current explosion factor hits the verification rule to obtain the verification result of the verification rule; if the verification result of each verification rule indicates a miss, it is determined that the current explosion factor passes the safety field verification; if there is at least one verification rule whose verification result indicates a hit, it is determined that the current explosion factor does not pass the safety field verification.

[0132] In some embodiments, the end module 408 is also used to determine the current impact type of the current explosion factor on the target distributed system based on the test results, and obtain the mapping relationship between the impact type and the impact range; based on the mapping relationship and the current impact type, determine the current impact range caused by the current explosion factor; based on the current impact range caused by the current explosion factor, update each explosion factor in the current explosion warehouse to obtain an updated explosion warehouse.

[0133] In some embodiments, the determination module is also used to obtain an updated explosion warehouse obtained when the stop condition is met, and determine the maximum explosion radius from the explosion radii corresponding to each explosion factor in the updated explosion warehouse; the explosion factor corresponding to the maximum explosion radius is used as the target explosion factor, and the target explosion factor has the greatest impact on the target distributed system.

[0134] Each module in the test device of the above distributed system can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0135] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a test method for a distributed system is implemented.

[0136] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0138] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0139] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0141] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed system testing method, characterized in that: The method comprises: Based on the target distributed system, multiple explosion factors are determined, each of which carries fault information; For the current test, a current explosion warehouse corresponding to the current test is obtained, and a current explosion factor is screened out from a plurality of explosion factors stored in the current explosion warehouse; In the case where it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification, the current explosion factor is injected into the target distributed system to perform a current test and obtain a test result; Based on the test result, each explosion factor stored in the current explosion warehouse is updated, and the updated explosion warehouse is used as the current explosion warehouse corresponding to the next test, and the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse is returned to continue execution until the stop condition is met and the test of the distributed system is terminated.

2. The method according to claim 1, characterized in that The step of selecting the current explosion factor from the multiple explosion factors stored in the current explosion warehouse includes: Obtaining the selection probability of each explosion factor from the current explosion warehouse; Based on the selection probability of each explosion factor, the current explosion factor is screened out from the multiple explosion factors stored in the current explosion warehouse.

3. The method according to claim 2, characterized in that The step of selecting the current explosion factor from the multiple explosion factors stored in the current explosion warehouse based on the selection probability of each explosion factor includes: Randomly select a probability from the preset probability range as the target probability; Based on the selection probability of each explosion factor, a first selection is made from the multiple explosion factors stored in the current explosion warehouse, and if the selection probability of the first selected explosion factor is less than the target probability, a second selection is made from the multiple explosion factors in the current explosion warehouse; If the sum of the selection probability of the second selected explosion factor and the selection probability of the first selected explosion factor is greater than or equal to the target probability, the second selected explosion factor is used as the current explosion factor.

4. The method according to claim 1, characterized in that: The method further comprises: Get multiple validation rules contained in the security location; For each verification rule, determine whether the fault information of the current explosion factor matches the verification rule, and obtain the verification result of the verification rule; If the verification results of each verification rule indicate a miss, it is determined that the current explosion factor passes the safety field verification; If the verification result of at least one verification rule represents a hit, it is determined that the current explosion factor does not pass the safety field verification.

5. The method according to claim 1, characterized in that The updating of each explosion factor stored in the current explosion warehouse based on the test result includes: Based on the test results, determine the current impact type of the current explosion factor on the target distributed system, and obtain a mapping relationship between the impact type and the impact range; Based on the mapping relationship and the current impact type, determining a current impact range caused by the current explosion factor; Based on the current impact range caused by the current explosion factor, each explosion factor in the current explosion warehouse is updated to obtain an updated explosion warehouse.

6. The method according to claim 1, characterized in that The method further comprises: Acquire an updated explosion warehouse obtained when the stop condition is met, and determine a maximum explosion radius from the explosion radii corresponding to each explosion factor in the updated explosion warehouse; The explosion factor corresponding to the maximum explosion radius is used as the target explosion factor, and the target explosion factor has the greatest impact on the target distributed system.

7. A distributed system testing device, characterized in that: The device comprises: A determination module, used for determining a plurality of explosion factors based on a target distributed system, each explosion factor carrying fault information; A screening module, used for obtaining a current explosion warehouse corresponding to the current test, and screening out a current explosion factor from a plurality of explosion factors stored in the current explosion warehouse; A test module, for injecting the current explosion factor into the target distributed system for current testing to obtain a test result when it is determined based on the fault information of the current explosion factor that the current explosion factor passes the safety field verification; The end module is used to update the explosion factors stored in the current explosion warehouse based on the test results, and use the updated explosion warehouse as the current explosion warehouse corresponding to the next test, and return to the step of screening out the current explosion factor from the multiple explosion factors stored in the current explosion warehouse to continue executing until the stop condition is met to end the test of the distributed system.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images