Distributed system load balancing test method and device based on load difference degree guidance

Abstract

The invention provides a distributed system load balancing test method and device based on load difference degree guidance. The method comprises the steps that multi-dimensional input is converted into a one-dimensional initial input operation sequence; wherein the multi-dimensional input comprises one or a combination of more of a working load, a load configuration and a node fault; generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; controlling the distributed system to operate the new test case, and calculating the load difference degree of each pair of nodes in the distributed system according to the load data of the nodes in the distributed system during operation; and determining a target test case according to the load difference degree, and storing the target test case in a seed pool to guide next fuzzy test iteration. According to the method, three-dimensional input of user requests, system configuration changes, system fault conditions and the like which influence a load balancing mechanism is modeled into a standard one-dimensional input sequence, so that subsequent fuzzy testing is facilitated, and the testing efficiency is improved; meanwhile, the load difference degree between the distributed nodes is used as guide feedback to optimize the input space, high-quality test cases can be continuously generated, load imbalance between the nodes in the distributed system is revealed to the maximum extent, developers can be helped to more effectively recognize and solve potential performance bottlenecks, and the test efficiency is improved. The test coverage rate is improved, and the overall stability and response speed of the system are also improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer software security testing, and in particular to a distributed system load balancing testing method and device based on load difference guidance. Background Art

[0002] Distributed systems often need to handle a large number of requests, data, etc. from across interconnected nodes. Load balancing mechanisms can evenly distribute data and process loads, which is essential for optimizing resource utilization and ensuring high availability, scalability, and reliability. However, due to the complexity of load balancing mechanisms, it is difficult to avoid errors in the code implementation of these mechanisms. Given the critical role these mechanisms play in distributed systems, any load imbalance errors or node overheating issues may have serious consequences, including performance degradation and service unavailability, thus affecting the reliability, availability, and security of the system. For example, GitHub experienced a network partition interruption of more than 24 hours due to improper load distribution and subsequent system overload, affecting millions of developers worldwide. This type of uneven load distribution caused by load balancing mechanism errors, which in turn causes hot spots, hangs, or even crashes, is called a load imbalance vulnerability.

[0003] To ensure the correctness and security of the load logic of distributed systems, many testing methods have been applied to real-world applications. Among them, request load generation methods, such as SmallFil and Filebench, generate a large number of load request inputs through simulation, which are used to continuously test and evaluate the runtime performance of the system under test. Load balancing testing tools, such as ApacheJMeter and Gatling, continuously simulate and generate a large number of random concurrent accesses, monitor the data distribution of the distributed system under test and the utilization of various resources (CPU, memory, hard disk, etc.) of each distributed node in real time, and provide a user-friendly graphical interface in real time based on this information to assist testers in analyzing and locating the load imbalance state of the distributed system. In addition, distributed system fault injection methods, such as CrashFuzz and Mallory, mainly test the error handling capabilities of the distributed system under various abnormal input conditions. These testing methods frequently add or delete nodes to continuously test the load logic code of the system under node changes, and successfully detect many errors in actual programs.

[0004] However, existing distributed testing tools only focus on generating test cases with a single input dimension during testing. For example, request load generation methods and load balancing testing tools only focus on generating user request load inputs, and distributed fault injection methods only generate fault sequence inputs used to trigger distributed system faults, using fixed user loads during testing. In fact, many hidden distributed system load imbalance vulnerabilities require the combined influence of multi-dimensional inputs to be triggered, that is, the load changes of distributed systems will be affected by user request inputs, distributed node fault conditions, and changes in various resource configurations (hard disk, CPU, memory) at the same time. Existing distributed testing tools all ignore this point, resulting in low testing efficiency.

[0005] In addition, existing distributed load testing tools all use random methods to generate test cases, lacking the use of key runtime information of the distributed system to be tested as a guide to optimize the input space. Considering that the test of the load balancing mechanism involves multi-dimensional inputs, and the permutation and combination space between multi-dimensional inputs is huge, the existing random testing methods are even less applicable, resulting in low test coverage.

[0006] In addition, all current distributed testing tools lack accurate detectors for load imbalance vulnerabilities in distributed systems. This often results in a large number of load imbalance vulnerabilities being missed or ignored even if the vulnerability is triggered due to the lack of accurate vulnerability identifiers to identify it.

[0007] Therefore, how to solve the problems of low testing efficiency and test coverage of existing distributed testing methods, and the inability to accurately identify load imbalance vulnerabilities, is an important issue that needs to be urgently addressed in the field of computer software security testing. Summary of the invention

[0008] The present invention provides a distributed system load balancing testing method and device based on load difference guidance, which is used to overcome the defects of low testing efficiency and test coverage of existing distributed testing methods, and can continuously generate high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent, help developers more effectively identify and solve potential performance bottlenecks, and improve the overall stability and response speed of the system.

[0009] On the one hand, the present invention provides a distributed system load balancing test method guided by load difference, comprising: converting a multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; controlling the distributed system to run the new test case, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining a target test case according to the load difference, and storing the target test case in the seed pool to guide the next fuzzy test iteration.

[0010] Furthermore, the runtime load data includes computing load data, network load data and storage load data of the nodes in the distributed system, the computing load data includes the number of CPU cores used and the CPU core utilization rate of the nodes, the network load data includes the number of user request processing, the network read IO utilization rate and the network write IO utilization rate of the nodes, and the storage load data includes the storage status of the nodes; accordingly, the step of calculating the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system specifically includes: calculating the computing load difference of each pair of nodes based on the computing load data; calculating the network load difference of each pair of nodes based on the network load data; calculating the storage load difference of each pair of nodes based on the storage load data; and determining the load difference of each pair of nodes based on the computing load difference, the network load difference and the storage load difference.

[0011] Further, determining a target test case according to the load difference includes: when the load difference is greater than a historical load difference of a previous round of fuzzy testing, taking a new test case currently running in the distributed system as a target test case.

[0012] Furthermore, the target test case is determined according to the load difference, including: when one or more of the computing load difference, network load difference and storage load difference exceeds the corresponding difference threshold, a double check is initiated to detect a true imbalance failure, and a new test case currently running in the distributed system is used as the target test case.

[0013] Furthermore, the step of initiating a double check to detect a true imbalance fault specifically includes: when one or more of the computing load difference, the network load difference, and the storage load difference exceed the corresponding difference threshold, determining that the distributed system is in a load imbalance state and identifying a candidate imbalance fault; explicitly calling the re-load balancing API interface to control the distributed system to re-run the current new test case and check the load status; when it is determined that the distributed system is still in a load imbalance state, identifying the identified candidate imbalance fault as the true imbalance fault.

[0014] Furthermore, generating an initial test case according to the initial input operation sequence includes: determining the maximum length of the initial test case to be generated; randomly generating an operation sequence ranging from one to the maximum length, recorded as the initial test case; wherein the initial test case includes an operation sequence of at least one operation, the operation includes an operator and at least one operand, the operator is randomly generated with a set probability, and the operand is instantiated according to its category.

[0015] In a second aspect, the present invention also provides a distributed system load balancing test device guided by load difference, comprising: a standard test case unification module, used to convert multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; an initial test case creation module, used to generate an initial test case according to the initial input operation sequence, and mutate the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; a load difference calculation module, used to control the distributed system to run the new test case, and calculate the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; a target test case determination module, used to determine the target test case according to the load difference, and store the target test case in the seed pool to guide the next fuzzy test iteration.

[0016] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a distributed system load balancing test method guided by load difference as described in any one of the above.

[0017] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for testing load balancing of a distributed system guided by load differences.

[0018] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the distributed system load balancing test method guided by load difference as described in any one of the above described methods.

[0019] The distributed system load balancing test method based on load difference guidance provided by the present invention is by converting multi-dimensional input into a one-dimensional initial input operation sequence; wherein, the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein, the initial test case is stored in a seed pool; controlling the distributed system to run a new test case, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining the target test case according to the load difference, and storing the target test case in the seed pool to guide the next fuzzy test iteration. The method is convenient for subsequent fuzzy testing and improves the test efficiency by modeling the three-dimensional inputs that affect the load balancing mechanism such as user requests, system configuration changes, and system failure conditions into a standard one-dimensional input sequence; at the same time, by optimizing the input space by taking the load difference between distributed nodes as a guide feedback, high-quality test cases can be continuously generated to reveal the load imbalance between nodes in the distributed system to the greatest extent, which can help developers more effectively identify and solve potential performance bottlenecks, improve test coverage, and also improve the overall stability and response speed of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of a distributed system load balancing test method based on load difference guidance provided by an embodiment of the present invention.

[0022] Figure 2 It is a schematic diagram describing the unified standard test case input model provided by an embodiment of the present invention, that is, a description of the initial input operation sequence.

[0023] Figure 3 It is a distributed load difference model provided by an embodiment of the present invention for describing the load difference between nodes.

[0024] Figure 4It is a schematic diagram of the overall framework of a distributed system load balancing test method based on load difference guidance provided by an embodiment of the present invention.

[0025] Figure 5 It is a schematic diagram of the working process of the imbalance detector provided by an embodiment of the present invention.

[0026] Figure 6 It is a structural schematic diagram of a distributed system load balancing test device based on load difference guidance provided by an embodiment of the present invention.

[0027] Figure 7 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that existing distributed testing tools only focus on generating test cases with a single input dimension during testing. In fact, many hidden distributed system load imbalance vulnerabilities require the combined influence of multi-dimensional inputs to be triggered, that is, the load changes of distributed systems will be affected by user request input, distributed node failure conditions, and various resource configuration (hard disk, CPU, memory) changes at the same time. Existing distributed testing tools all ignore this point, resulting in low testing efficiency.

[0030] In addition, existing distributed load testing tools all use random methods to generate test cases, and lack the use of key runtime information of the distributed system to be tested as a guide to optimize the input space. Considering that the test of the load balancing mechanism involves multi-dimensional inputs, and the permutation and combination space between multi-dimensional inputs is huge, the existing random testing methods are even less applicable, resulting in low test coverage. In addition, all current distributed testing tools lack accurate detectors for load imbalance vulnerabilities in distributed systems, which often leads to the fact that even if the vulnerability is executed, it is often missed and ignored because there is no accurate vulnerability identifier to identify it.

[0031] In view of this, the present invention proposes a distributed system load balancing test method based on load difference guidance, specifically, Figure 1 The flowchart of the distributed system load balancing test method based on load difference guidance provided by an embodiment of the present invention is shown.

[0032] like Figure 1 As shown, the method includes steps S110-S140, and steps S110-S140 and related steps will be described in detail below.

[0033] S110, converting a multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration, and node failure.

[0034] It is easy to understand that the load change of a distributed system will be affected by user request input, distributed node failure conditions, and changes in various resource configurations (hard disk, CPU, memory) at the same time, rather than changes in a single input dimension. Considering this, this embodiment creates a unified standard test case input model for modeling and describing operations related to the load of a distributed system, converting the received multi-dimensional input that affects the system load balancing mechanism into a one-dimensional initial input operation sequence to achieve standardization / unification of multi-dimensional inputs.

[0035] The multi-dimensional input includes one or more of workload, load configuration, and node failure. Workload includes task type, expected running time, etc., load configuration includes CPU usage, memory usage, etc., and node failure includes failure type, occurrence time, etc.

[0036] Figure 2 A schematic diagram of describing a unified standard test case input model provided by an embodiment of the present invention is shown, that is, a description of an initial input operation sequence.

[0037] like Figure 2 As shown, each test case used for fuzz testing is an operation sequence containing at least one operation, and each operation consists of an operator opt and at least one operand opd.

[0038] Operators can be divided into three categories: file_op, node_op, and volume_op. File_op is used to describe client request input, such as file creation, appending, and deletion; node_op and volume_op are used to describe system configuration input, such as node addition / deletion and volume expansion / reduction.

[0039] The operation "add_MN nodea" indicates adding a metadata management node "nodea", while the operation "remove_storage nodeb" indicates deleting a storage node "nodeb".

[0040] The number and content of operands opd are determined by the operator opt. For example, if the operator opt is "create", at least two operands are required: opd1 is the "file name" and opd2 is the size of the new file.

[0041] It should be noted that, since input data of different dimensions may have different dimensions and ranges, this embodiment performs standardized processing on multi-dimensional inputs, which can effectively ensure the consistency and effectiveness of the subsequent fuzzy testing process.

[0042] On the basis of converting the multi-dimensional input into a standard one-dimensional initial input operation sequence in step S110, further, step S120 is performed.

[0043] S120, generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool.

[0044] It is easy to understand that before starting the testing phase, initial test cases need to be created to drive the testing process.

[0045] First, determine the maximum length max of the initial test case / operation sequence opSeq to be generated n According to historical experience, the steps required for triggering a distributed system fault tolerance exception are usually no more than 8. Therefore, this embodiment sets the maximum length max n Set to 8.

[0046] Next, randomly generate a value from 1 to the maximum length max n The unequal operation sequences are recorded as initial test cases and stored in the seed pool. The initial test case includes an operation sequence of at least one operation, and the operation includes an operator and at least one operand. The operator is randomly generated with a set probability, and the operand is instantiated according to its category.

[0047] For example, in a specific embodiment, the probability is set to a value of 1 / t, where t is the number of all different load-related operations.

[0048] The categories of operands include file name category (Category FileName), node ID category (CategoryNodeId), and size category (Category Size).

[0049] About the file name category. This category intends to use the file tree Treefiles to manage and record all file names and their storage topology relationships in the distributed file system. When instantiating a file name, either select an existing file name from the file tree in a uniform random distribution, or create a new file name and add it to the file tree.

[0050] About the node ID category. This category intends to use two lists, listMN and listS, to track all management nodes and data storage nodes in the distributed system respectively. When instantiating a node ID, a node is randomly selected from these lists according to the specific operator opt. For example, if opt is add_MN, nodeId is randomly selected from listMN; similarly, if opt is add_storage, nodeId is selected from listS.

[0051] About the size category. This category is used to specify the size of the data being operated, and it tracks the remaining storage capacity freespace of the distributed file system. In order to more effectively test the fault-tolerant logic, you can create some boundary scenarios of data size. For example, when instantiating the "create filename size" operation, the file size will be randomly assigned a value between 0 and freespace.

[0052] After the initial test cases are created, the initial test cases are taken out from the seed pool and mutated to obtain new test cases to explore more program states and paths, thereby discovering potential vulnerabilities or errors.

[0053] Among them, mutation operations include but are not limited to bit flipping, byte flipping, insertion, deletion, replacement, copying, rearrangement, arithmetic mutation, memory mutation, format string mutation, type confusion, logical mutation, etc.

[0054] After the initial test case is created and mutated to obtain a new test case in step S120, step S130 is further performed.

[0055] S130, controlling the distributed system to run the new test case, and calculating the load difference between each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system.

[0056] It is easy to understand that the new test cases obtained by mutation are used to test the distributed system, and the distributed system is controlled to run the new test cases to check whether new errors or vulnerabilities are triggered.

[0057] When a distributed system runs a new test case, the status of distributed nodes (i.e., nodes in the distributed system) is updated and analyzed in real time, and the load difference between each pair of nodes is calculated based on the runtime load data of the distributed nodes.

[0058] The runtime load data of the distributed nodes in this embodiment include but are not limited to computing load data, network load data and storage load data. The computing load data includes the number of CPU cores used and the CPU core utilization rate of the distributed nodes, the network load data includes the number of user request processing, the network read IO utilization rate and the network write IO utilization rate of the distributed nodes, and the storage load data includes the storage status of the distributed nodes.

[0059] Correspondingly, the load difference between each pair of nodes is obtained based on the differences in the nodes' calculation load data, network load data, and storage load data, which is not specifically limited here.

[0060] In a specific embodiment, a distributed load difference model is constructed, which is specifically used to calculate the load difference between every two nodes in the distributed system.

[0061] Based on the calculation of the load difference between each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system in step S130, further, step S140 is performed.

[0062] S140, determining a target test case according to the load difference, and storing the target test case in a seed pool to guide the next fuzzy test iteration.

[0063] It is easy to understand that the load state of the distributed system, such as the load imbalance state or the load balance state, can be determined according to the load difference between each pair of nodes in the distributed system.

[0064] Specifically, when the load difference between nodes in the distributed system becomes larger or any new imbalance fault is discovered, the new test case currently running in the distributed system will be regarded as a valuable seed (target test case) and stored in the seed pool to guide the next fuzz testing iteration. In this way, the present embodiment can continuously generate high-quality operation sequences as test cases, aiming to maximize the load imbalance between distributed nodes.

[0065] In this embodiment, by converting the multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; controlling the distributed system to run the new test case, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining the target test case according to the load difference, and storing the target test case in the seed pool to guide the next fuzzy test iteration. The method models the three-dimensional inputs that affect the load balancing mechanism, such as user requests, system configuration changes, and system failure conditions, into a standard one-dimensional input sequence, which is convenient for subsequent fuzzy testing and improves the test efficiency; at the same time, by using the load difference between distributed nodes as a guide feedback to optimize the input space, high-quality test cases can be continuously generated to reveal the load imbalance between nodes in the distributed system to the greatest extent, which can help developers more effectively identify and solve potential performance bottlenecks, improve test coverage, and also improve the overall stability and response speed of the system.

[0066] On the basis of the above-mentioned embodiment, the load difference calculation process of each pair of nodes in the distributed system is further described in detail below.

[0067] It is easy to understand that in most distributed systems, the goal of the load balancing mechanism is to achieve load distribution balance between nodes as much as possible, rather than maintaining absolute uniformity. Therefore, short-term and slight load differences between nodes are considered normal and acceptable. However, the core insight of this embodiment is that the final load imbalance state in the system is caused by the continuous accumulation of small load differences in many intermediate stages. Therefore, in order to effectively touch the deep code logic and efficiently trigger imbalance faults, this embodiment introduces a load difference model to guide test case generation to ensure that the nodes in the system experience as many load changes as possible.

[0068] Specifically, in the process of running new test cases in a distributed system, the status of distributed nodes (i.e., nodes in a distributed system) is updated and analyzed in real time. The status monitor collects the runtime load data of the distributed nodes in real time and calculates the load difference between each pair of nodes based on the runtime load data of the distributed nodes.

[0069] Figure 3 The distributed load difference model provided by the embodiment of the present invention for describing the load difference between nodes is shown.

[0070] like Figure 3As shown in the figure, the runtime load data of the node can be divided into three types, including the computing load data of the node in the distributed system, the network load data and the storage load data. The computing load data includes the distributed node Number of CPU cores used , No. CPU core usage , network load data includes distributed nodes Number of user requests processed , Network read IO usage , Network write IO usage , storing load data including distributed nodes Storage status .

[0071] Accordingly, the step of calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system specifically includes: calculating the calculation load difference of each pair of nodes according to the calculation load data; ; Based on the network load data, calculate the network load difference for each pair of nodes ; Based on the storage load data, calculate the storage load difference of each pair of nodes ; Determine the difference between each pair of nodes based on the difference in computing load, network load, and storage load. Load Difference .

[0072] After calculating the load difference of each pair of nodes in the distributed system, the load difference between the distributed nodes is used as a guide feedback to optimize the input space, which can continuously generate high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent. This can help developers more effectively identify and solve potential performance bottlenecks, improve test coverage, and improve the overall stability and response speed of the system.

[0073] On the basis of the above-mentioned embodiment, the process of determining the target test case according to the load difference to guide the next fuzzy test will be described in detail below.

[0074] It is easy to understand that this embodiment provides two possibilities for determining the target test case, namely, the load difference becomes larger or any new imbalance fault is found.

[0075] For example, in a specific embodiment, the target test case is determined based on the load difference, including: when the load difference is greater than the historical load difference of the previous round of fuzz testing, the new test case currently running in the distributed system is used as the target test case, and the target test case is stored in a seed pool to guide the next fuzz testing iteration.

[0076] Specifically, the load difference calculated in this round of fuzz testing is compared with the load difference of the previous round of fuzz testing (recorded as historical load difference). If the load difference of this round of fuzz testing becomes larger than the historical load difference of the previous round of fuzz testing, the new test case currently being run will be used as the target test case. The target test case will be regarded as a valuable seed and stored in the seed pool to guide the next fuzz testing iteration.

[0077] It should be noted that in addition to the load difference of this round of fuzzy testing being larger than the historical load difference of the previous round of fuzzy testing, other constraints can also be set according to actual conditions. For example, the difference between the load difference of this round of fuzzy testing and the historical load difference of the previous round of fuzzy testing exceeds a set threshold, etc., which is not specifically limited here.

[0078] In another specific embodiment, the target test case is determined according to the load difference, including: when one or more of the computing load difference, the network load difference and the storage load difference exceeds the corresponding difference threshold, a double check is initiated to detect a true imbalance failure, and a new test case currently running in the distributed system is used as the target test case, and the target test case is stored in a seed pool to guide the next fuzzy test iteration.

[0079] Specifically, in the process of calculating the load difference, multiple different load difference values ​​are calculated, including but not limited to the computing load difference, the network load difference and the storage load difference. The computing load difference, the network load difference and the storage load difference are respectively compared with the corresponding difference thresholds. If one or more of the computing load difference, the network load difference and the storage load difference exceeds the corresponding difference threshold, it means that the distributed system is currently in an unbalanced state, and a candidate unbalanced fault is identified. Among them, the candidate unbalanced fault can be determined according to the type of load difference exceeding the corresponding difference threshold, which is not specifically limited here.

[0080] The difference threshold corresponding to each load difference type can be set according to actual conditions and is not specifically limited here. For example, preferably, in this embodiment, the product of the average node load and the set difference threshold is used as the difference threshold of the current load type, and the average node load corresponds to each load difference type one by one.

[0081] In order to confirm its effectiveness and minimize the probability of false positives, this embodiment designs a double check process to detect real imbalance faults. In the case of detecting real imbalance faults, the new test case currently running is used as the target test case, and the target test case will be regarded as a valuable seed and stored in the seed pool to guide the next fuzz test iteration.

[0082] In yet another specific embodiment, the step of initiating a double check to detect a true imbalance fault comprises: when one or more of the computational load difference, the network load difference, and the storage load difference exceed corresponding difference thresholds, determining that the distributed system is in a load imbalance state, and identifying a candidate imbalance fault; explicitly calling a re-load balancing API interface to control the distributed system to re-run the current new test case and check the load status; and when determining that the distributed system is still in a load imbalance state, identifying the identified candidate imbalance fault as a true imbalance fault.

[0083] It is easy to understand that in practice, different distributed systems use different load balancing mechanisms. For example, CephFS calculates and balances the load in real time, while GlusterFS performs load balancing through periodic scheduled tasks. Therefore, different distributed systems take different amounts of time to complete the test case, activate the load balancing mechanism, and reach a load-balanced state. This difference makes it difficult for the test tool to accurately determine when the detector should check the system load status to identify imbalances while avoiding false positives. The appropriate check time varies significantly depending on the specific distributed system implementation and factors in its distributed operating environment (such as latency, bandwidth, etc.).

[0084] Fortunately, most distributed systems provide API interfaces such as rebalancing, allowing users to directly trigger and execute their load balancing mechanisms. In order to filter out false positives, the test framework of an embodiment of the present invention introduces a double-check process. Specifically, when a candidate imbalance vulnerability is identified, this embodiment will explicitly call the re-load balancing API interface. Specifically, this embodiment will poll and call the "rebalance()" API to confirm whether the rebalancing operation has been completed. Once the API returns "rebalance completed", this test framework will immediately re-execute the current new test case and check the load status of the system. If the distributed system is still in a load imbalance state, the identified candidate imbalance fault is confirmed as a true imbalance fault.

[0085] In this embodiment, when the load difference is greater than the historical load difference of the previous round of fuzz testing, the new test case currently running in the distributed system is used as the target test case, or when one or more of the calculation load difference, network load difference and storage load difference exceed the corresponding difference threshold, a double check is started to detect the real imbalance fault, and the new test case currently running in the distributed system is used as the target test case, and the target test case is stored in the seed pool to guide the next fuzz test iteration. This method models the three-dimensional inputs that affect the load balancing mechanism, such as user requests, system configuration changes, and system failure conditions, into a standard one-dimensional input sequence, which is convenient for subsequent fuzz testing and improves the test efficiency; at the same time, by using the load difference between distributed nodes as a guide feedback to optimize the input space, high-quality test cases can be continuously generated to reveal the load imbalance between nodes in the distributed system to the greatest extent, which can help developers more effectively identify and solve potential performance bottlenecks, improve test coverage, and improve the overall stability and response speed of the system; in addition, the load imbalance vulnerability of the distributed system can be detected in real time and accurately.

[0086] In some other embodiments, Figure 4 The overall framework schematic diagram of the distributed system load balancing test method based on load difference guidance provided by an embodiment of the present invention is shown.

[0087] like Figure 4 As shown, the fuzzy testing framework of the distributed system load balancing testing method guided by load difference provided by the embodiment of the present invention is mainly composed of two key components: a test case generator for generating high-quality test cases; and an imbalance detector for identifying imbalance faults.

[0088] First, an embodiment of the present invention constructs a unified standard test case input model. Based on this model, multi-dimensional inputs such as client requests, system configurations, and node failures are converted into a unified one-dimensional initial input operation sequence to facilitate the subsequent fuzzy testing process.

[0089] Then, the initial test cases are created based on the initial input operation sequence and stored in the seedspool, which is a collection used to generate different test cases through mutation. New test cases are generated by selecting and mutating existing test cases in the seedspool.

[0090] Subsequently, the tested distributed file system is controlled to execute these new test cases. The imbalance detector monitors and collects runtime load data (such as CPU / IO usage, storage distribution, etc.) in real time, calculates the load difference between distributed nodes, and updates the load difference model in real time.

[0091] At the same time, the imbalance detector identifies the imbalance state of the system and reports it when an imbalance failure is detected, so that test cases that cause new imbalance failures or larger load differences are preferentially added to the seed pool to guide subsequent test case generation.

[0092] This test framework enters the next round of test iteration until the load imbalance exceeds the predefined threshold. At this time, it is determined that the system under test has entered a fault state, the distributed system is reset to the initial state and the test process is restarted.

[0093] It should be noted that, in this embodiment, the steps of converting multi-dimensional input into a one-dimensional initial input operation sequence, generating initial test cases based on the initial input operation sequence, and mutating the selected initial test cases to obtain new test cases are all implemented by the test case generator.

[0094] In this embodiment, the distributed system is controlled to run new test cases, and the load difference of each pair of nodes in the distributed system is calculated based on the runtime load data of the nodes in the distributed system. The target test cases are determined based on the load difference, and the target test cases are stored in a seed pool to guide the next fuzzy test iteration process, which is implemented by the imbalance detector. Figure 4 The anomaly detector and the abnormal load analyzer in are both included in the imbalance detector.

[0095] Further, Figure 5 A schematic diagram of the working process of the imbalance detector provided in an embodiment of the present invention is shown.

[0096] like Figure 5 As shown, after executing the new test case generated by the embodiment of the present invention, the state monitor collects runtime load data and calculates the load difference model (LVMij) of each pair of nodes (nodei and nodej) in the distributed system. Then, three anomaly detectors are used to evaluate the differences in computing load, network load, and storage load respectively. These detectors determine whether the distributed system to be tested has entered a load imbalance state by verifying whether the maximum load in each node exceeds the average node load multiplied by the difference threshold 𝑡. If an imbalanced state is detected, a candidate imbalance fault is identified. In order to confirm its effectiveness and minimize false alarms, this embodiment will also perform a double check process to ultimately determine whether the candidate imbalance fault is a true imbalance fault.

[0097] Corresponding to the distributed system load balancing test method based on load difference guidance provided by the embodiment of the present invention, the present invention also proposes a distributed system load balancing test method based on load difference guidance and a distributed system load balancing test device based on load difference guidance.

[0098] Specifically, Figure 6 The schematic diagram of the structure of the distributed system load balancing test device based on load difference guidance provided by an embodiment of the present invention is shown.

[0099] like Figure 6 As shown, the device includes: a standard test case unification module 610, which is used to convert a multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; an initial test case creation module 620, which is used to generate an initial test case according to the initial input operation sequence, and mutate the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; a load difference calculation module 630, which is used to control the distributed system to run the new test case, and calculate the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; a target test case determination module 640, which is used to determine the target test case according to the load difference, and store the target test case in the seed pool to guide the next fuzzy test iteration.

[0100] In this embodiment, a multi-dimensional input is converted into a one-dimensional initial input operation sequence through a standard test case unification module 610; wherein the multi-dimensional input includes one or more combinations of workload, load configuration, and node failure; an initial test case creation module 620 generates an initial test case according to the initial input operation sequence, and mutates the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; a load difference calculation module 630 controls the distributed system to run a new test case, and calculates the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; a target test case determination module 640 determines the target test case according to the load difference, and stores the target test case in the seed pool to guide the next fuzzy test iteration. The device models the three-dimensional inputs that affect the load balancing mechanism, such as user requests, system configuration changes, and system failure conditions, into a standard one-dimensional input sequence, which facilitates subsequent fuzzy testing and improves testing efficiency. At the same time, by using the load differences between distributed nodes as guided feedback to optimize the input space, it can continuously generate high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent, which can help developers more effectively identify and solve potential performance bottlenecks, improve test coverage, and also improve the overall stability and response speed of the system.

[0101] It should be noted that the distributed system load balancing test device based on load difference guidance provided in the embodiment of the present invention and the distributed system load balancing test method based on load difference guidance described in the above embodiments can correspond to each other.

[0102] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a distributed system load balancing test method guided by load difference, the method comprising: converting a multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; controlling the distributed system to run the new test case, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining a target test case according to the load difference, and storing the target test case in the seed pool to guide the next fuzzy test iteration.

[0103] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the distributed system load balancing test method guided by load difference provided by the above methods, the method including: converting a multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; controlling the distributed system to run the new test case, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining the target test case according to the load difference, and storing the target test case in the seed pool to guide the next fuzzy test iteration.

[0105] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the load difference-guided distributed system load balancing test method provided by the above-mentioned methods, the method comprising: converting a multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration, and node failure; generating an initial test case according to the initial input operation sequence, and mutating the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; controlling the distributed system to run the new test case, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining a target test case according to the load difference, and storing the target test case in the seed pool to guide the next fuzzy test iteration.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed system load balancing test method based on load difference guidance, characterized in that: include: Converting a multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration, and node failure; Generate an initial test case according to the initial input operation sequence, and mutate the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; Controlling the distributed system to run the new test case, and calculating the load difference between each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; A target test case is determined according to the load difference, and the target test case is stored in a seed pool to guide the next fuzzy test iteration.

2. The distributed system load balancing test method based on load difference guidance according to claim 1 is characterized in that: The runtime load data includes computing load data, network load data and storage load data of the nodes in the distributed system. The computing load data includes the number of CPU cores used and the CPU core utilization rate of the nodes. The network load data includes the number of user request processing, network read IO utilization rate and network write IO utilization rate of the nodes. The storage load data includes the storage status of the nodes. Accordingly, the step of calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system specifically includes: Calculating the computational load difference of each pair of nodes according to the computational load data; Calculating the network load difference of each pair of nodes according to the network load data; Calculating the storage load difference of each pair of nodes according to the storage load data; The load difference degree of each pair of nodes is determined according to the computing load difference, the network load difference and the storage load difference.

3. The distributed system load balancing test method based on load difference guidance according to claim 2 is characterized in that: Determining a target test case according to the load difference includes: When the load difference is greater than the historical load difference of the previous round of fuzzy testing, the new test case currently running in the distributed system is used as the target test case.

4. The distributed system load balancing test method based on load difference guidance according to claim 2 is characterized in that: Determining a target test case according to the load difference includes: When one or more of the computing load difference, the network load difference, and the storage load difference exceeds the corresponding difference threshold, a double check is initiated to detect a true imbalance failure, and a new test case currently running on the distributed system is used as a target test case.

5. The distributed system load balancing test method based on load difference guidance according to claim 4 is characterized in that: The step of starting the double check to detect a real imbalance fault specifically includes: In the case where one or more of the computing load difference, the network load difference, and the storage load difference exceeds the corresponding difference threshold, determining that the distributed system is in a load imbalance state, and identifying a candidate imbalance fault; Explicitly call the re-load balancing API interface to control the distributed system to re-run the current new test case and check the load status; When it is determined that the distributed system is still in a load imbalance state, the identified candidate imbalance fault is identified as the real imbalance fault.

6. The distributed system load balancing test method based on load difference guidance according to any one of claims 1 to 5, characterized in that: Generating an initial test case according to the initial input operation sequence includes: Determine the maximum length of the initial test cases to be generated; Randomly generate operation sequences ranging from one to the maximum length, recorded as initial test cases; The initial test case includes an operation sequence of at least one operation, wherein the operation includes an operator and at least one operand, the operator is randomly generated with a set probability, and the operand is instantiated according to its category.

7. A distributed system load balancing test device based on load difference guidance, characterized in that: include: A standard test case unification module, used to convert multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more combinations of workload, load configuration and node failure; An initial test case creation module, used to generate an initial test case according to the initial input operation sequence, and mutate the selected initial test case to obtain a new test case; wherein the initial test case is stored in a seed pool; A load difference calculation module, used to control the distributed system to run the new test case, and calculate the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; The target test case determination module is used to determine the target test case according to the load difference, and store the target test case in a seed pool to guide the next fuzzy test iteration.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the distributed system load balancing test method based on load difference guidance according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed system load balancing test method based on load difference guidance as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the distributed system load balancing test method based on load difference guidance as claimed in any one of claims 1 to 6 is implemented.

Images