A load balancing test method and apparatus for distributed systems based on load difference guidance

Abstract

This invention provides a method and apparatus for load balancing testing of distributed systems based on load difference. The method includes: converting multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failures; generating initial test cases based on the initial input operation sequence, and mutating the selected initial test cases to obtain new test cases; wherein the initial test cases are stored in a seed pool; controlling the distributed system to run the new test cases, and calculating the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system; determining target test cases based on the load difference, and storing the target test cases in the seed pool to guide the next fuzzing iteration. This method models the three dimensions of input affecting the load balancing mechanism—user requests, system configuration changes, and system failures—into a standard one-dimensional input sequence, facilitating subsequent fuzz testing and improving testing efficiency. Simultaneously, by using the load disparity between distributed nodes as guiding feedback to optimize the input space, it continuously generates high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent possible. This helps developers more effectively identify and resolve potential performance bottlenecks, improves test coverage, and enhances the overall stability and response speed of the system.

Description

Technical Field

[0001] This invention relates to the field of computer software security testing technology, and in particular to a distributed system load balancing testing method and apparatus based on load difference guidance. Background Technology

[0002] Distributed systems often need to handle large volumes of requests and data from interconnected nodes. Load balancing mechanisms, which distribute data and processing load evenly, are crucial for optimizing resource utilization and ensuring high availability, scalability, and reliability. However, due to the complexity of load balancing mechanisms, errors in their code implementation are difficult to avoid. Given the critical role these mechanisms play in distributed systems, any load imbalance errors or node overheating can have serious consequences, including performance degradation and service unavailability, thus impacting system reliability, availability, and security. For example, GitHub experienced a network partition outage of over 24 hours due to improper load balancing and subsequent system overload, affecting millions of developers worldwide. This type of load imbalance caused by errors in load balancing mechanisms, leading to hotspots, hangs, or even crashes, is known as a load imbalance vulnerability.

[0003] To ensure the correctness and security of load balancing logic in distributed systems, numerous testing methods have been applied in real-world applications. Among these, request load generation methods, such as SmallFil and Filebench, simulate a large number of load request inputs for continuous testing and evaluation of the runtime performance of the system under test. Load balancing testing tools, such as Apache JMeter and Gatling, continuously simulate a large number of random concurrent accesses, monitor the data distribution of the distributed system under test in real time, and assess the utilization of various resources (CPU, memory, disk, etc.) of each distributed node. Based on this information, they provide a user-friendly graphical interface in real time to assist testers in analyzing and locating load imbalances in the distributed system. Furthermore, distributed system fault injection methods, such as CrashFuzz and Mallory, primarily assess the error handling capabilities of the distributed system under various abnormal input conditions. These testing methods frequently add or remove nodes to continuously test the load balancing logic code under node changes, successfully detecting numerous errors in actual programs.

[0004] However, existing distributed testing tools focus solely on generating test cases from a single input dimension. For example, request load generation methods and load balancing testing tools only generate user request load inputs, while distributed fault injection methods only generate fault sequence inputs to trigger distributed system failures, using a fixed user load during testing. In reality, many deeply hidden distributed system load imbalance vulnerabilities require the combined influence of multiple input dimensions to be triggered. That is, changes in the load of a distributed system are simultaneously affected by user request inputs, distributed node failures, and changes in various resource configurations (disk, CPU, memory). Existing distributed testing tools overlook this, leading to inefficient testing.

[0005] Furthermore, existing distributed load testing tools all use random methods to generate test cases, lacking the use of key runtime information of the distributed system under test as guiding feedback to optimize the input space. Considering that load balancing mechanism testing requires multi-dimensional inputs, and the permutation and combination space between these multi-dimensional inputs is enormous, existing random testing methods are even more unsuitable, resulting in low test coverage.

[0006] Furthermore, all current distributed testing tools lack accurate detectors for load imbalance vulnerabilities in distributed systems. This often leads to the omission or neglect of a large number of load imbalance vulnerabilities, even when they are exploited.

[0007] Therefore, how to solve the problems of low testing efficiency and coverage of existing distributed testing methods, and the inability to accurately identify load imbalance vulnerabilities, is an important issue that urgently needs to be addressed in the field of computer software security testing. Summary of the Invention

[0008] This invention provides a load balancing testing method and apparatus for distributed systems based on load difference guidance, which overcomes the shortcomings of low testing efficiency and low test coverage of existing distributed testing methods. It can continuously generate high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent, helping developers to more effectively identify and solve potential performance bottlenecks and improve the overall stability and response speed of the system.

[0009] On one hand, this invention provides a distributed system load balancing testing method based on load difference guidance, comprising: converting multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failures; generating initial test cases according to the initial input operation sequence, and mutating the selected initial test cases to obtain new test cases; wherein the initial test cases are stored in a seed pool; controlling the distributed system to run the new test cases, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining target test cases according to the load difference, and storing the target test cases in the seed pool to guide the next fuzzing iteration.

[0010] Furthermore, the runtime load data includes computational load data, network load data, and storage load data of nodes in the distributed system. The computational load data includes the number of CPU cores used and the CPU core utilization rate of the nodes. The network load data includes the number of user requests processed by the nodes, the network read IO utilization rate, and the network write IO utilization rate. The storage load data includes the storage status of the nodes. Accordingly, the step of calculating the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes specifically includes: calculating the computational load difference of each pair of nodes based on the computational load data; calculating the network load difference of each pair of nodes based on the network load data; calculating the storage load difference of each pair of nodes based on the storage load data; and determining the load difference of each pair of nodes based on the computational load difference, network load difference, and storage load difference.

[0011] Further, determining the target test case based on the load difference includes: if the load difference is greater than the historical load difference of the previous round of fuzz testing, taking the new test case currently running in the distributed system as the target test case.

[0012] Further, determining the target test case based on the load difference includes: if one or more of the computational load difference, network load difference, and storage load difference exceed the corresponding difference threshold, initiating a double check to detect the real imbalance fault, and using the new test case currently running in the distributed system as the target test case.

[0013] Furthermore, the step of initiating dual checks to detect genuine load imbalance faults specifically includes: if one or more of the computational load difference, network load difference, and storage load difference exceed the corresponding difference threshold, determining that the distributed system is in a load imbalance state and identifying candidate load imbalance faults; explicitly calling the reload balancing API interface to control the distributed system to rerun the current new test case and check the load status; if it is determined that the distributed system is still in a load imbalance state, identifying the identified candidate load imbalance faults as the genuine load imbalance faults.

[0014] Further, generating initial test cases based on the initial input operation sequence includes: determining the maximum length of the initial test cases to be generated; randomly generating operation sequences ranging from one to the maximum length, denoted as initial test cases; wherein, the initial test cases include operation sequences of at least one operation, the operation including an operator and at least one operand, the operator being randomly generated with a set probability, and the operand being instantiated according to its category.

[0015] Secondly, the present invention also provides a distributed system load balancing testing device based on load difference guidance, comprising: a standard test case unification module, used to convert multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failure; an initial test case creation module, used to generate initial test cases according to the initial input operation sequence, and mutate the selected initial test cases to obtain new test cases; wherein the initial test cases are stored in a seed pool; a load difference calculation module, used to control the distributed system to run the new test cases, and calculate the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; and a target test case determination module, used to determine target test cases according to the load difference, and store the target test cases in the seed pool to guide the next fuzz test iteration.

[0016] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the distributed system load balancing test method based on load difference as described above.

[0017] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed system load balancing test method based on load difference guidance as described above.

[0018] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the distributed system load balancing test method based on load difference as described above.

[0019] This invention provides a distributed system load balancing testing method based on load difference guidance. It transforms multi-dimensional inputs into a one-dimensional initial input operation sequence. The multi-dimensional inputs include one or more combinations of workload, load configuration, and node failures. Initial test cases are generated based on the initial input operation sequence, and the selected initial test cases are mutated to obtain new test cases. These initial test cases are stored in a seed pool. The distributed system is controlled to run the new test cases, and the load difference between each pair of nodes in the distributed system is calculated based on the runtime load data of the nodes. Target test cases are determined based on the load difference and stored in the seed pool to guide the next fuzzing iteration. This method models the three-dimensional inputs affecting the load balancing mechanism—user requests, system configuration changes, and system failures—into a standard one-dimensional input sequence, facilitating subsequent fuzzing and improving testing efficiency. Simultaneously, by using the load difference between distributed nodes as guiding feedback to optimize the input space, it can continuously generate high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent. This helps developers more effectively identify and resolve potential performance bottlenecks, improves test coverage, and enhances the overall stability and response speed of the system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the distributed system load balancing test method based on load difference guidance provided in this embodiment of the invention.

[0022] Figure 2 This is a schematic diagram describing the unified standard test case input model provided in the embodiments of the present invention, that is, a description of the initial input operation sequence.

[0023] Figure 3 This is a distributed load difference model provided in this embodiment of the invention for describing the load difference between nodes.

[0024] Figure 4This is a schematic diagram of the overall framework of the distributed system load balancing test method based on load difference guidance provided in the embodiments of the present invention.

[0025] Figure 5 This is a schematic diagram of the workflow of the imbalance detector provided in an embodiment of the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of a distributed system load balancing test device based on load difference guidance provided in an embodiment of the present invention.

[0027] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] It's important to note that existing distributed testing tools focus solely on generating test cases from a single input dimension. However, many deeply hidden distributed system load imbalance vulnerabilities require the combined influence of multiple input dimensions to be triggered. This means that load changes in a distributed system are simultaneously affected by user requests, distributed node failures, and variations in resource configurations (disk, CPU, memory). Existing distributed testing tools overlook this, leading to inefficient testing.

[0030] Furthermore, existing distributed load testing tools all use random methods to generate test cases, lacking the use of key runtime information of the distributed system under test as guiding feedback to optimize the input space. Considering that testing load balancing mechanisms requires multi-dimensional inputs, and the permutation and combination space between these multi-dimensional inputs is enormous, existing random testing methods are even more unsuitable, resulting in low test coverage. In addition, all current distributed testing tools lack accurate detectors for load imbalance vulnerabilities in distributed systems. This often leads to situations where, even if vulnerabilities are exploited, they are often missed or ignored because there is no accurate vulnerability identifier.

[0031] In view of this, the present invention proposes a load balancing test method for distributed systems based on load difference guidance. Specifically, Figure 1 The diagram illustrates a flowchart of a distributed system load balancing test method based on load difference guidance provided in an embodiment of the present invention.

[0032] like Figure 1 As shown, the method includes steps S110-S140, and steps S110-S140 and related steps will be described in detail below.

[0033] S110, convert the multi-dimensional input into a one-dimensional initial input operation sequence; wherein the multi-dimensional input includes one or more of the following: workload, load configuration, and node faults.

[0034] It is easy to understand that load changes in a distributed system are simultaneously affected by user request inputs, distributed node failures, and changes in various resource configurations (hard disk, CPU, memory), rather than by changes in a single input dimension. Considering this, this embodiment creates a unified standard test case input model to model and describe load-related operations in a distributed system. It transforms the received multi-dimensional inputs that affect the system's load balancing mechanism into a one-dimensional initial input operation sequence, thereby achieving standardization / unification of multi-dimensional inputs.

[0035] The multi-dimensional inputs include one or more of the following: workload, load configuration, and node failures. Workload includes, for example, task type and estimated runtime; load configuration includes, for example, CPU utilization and memory usage; and node failures include, for example, failure type and occurrence time.

[0036] Figure 2 This diagram illustrates a description of the unified standard test case input model provided in an embodiment of the present invention, that is, a description of the initial input operation sequence.

[0037] like Figure 2 As shown, each test case used for fuzz testing is a sequence of operations containing at least one operation, each operation consisting of an operator opt and at least one operand opd.

[0038] The operators can be divided into three categories: file_op, node_op, and volume_op. file_op is used to describe client request input, such as file creation, appending, and deletion; node_op and volume_op are used to describe system configuration input, such as adding / deleting nodes and expanding / shrinking volumes.

[0039] The operation "add_MN nodea" means adding a metadata management node "nodea", while the operation "remove_storage nodeb" means deleting a storage node "nodeb".

[0040] The number and content of operands opd are determined by the operator opt. For example, if the operator opt is "create", then at least two operands are required: opd1 is the "filename", and opd2 is the size of the new file.

[0041] It should be noted that since input data of different dimensions may have different units and ranges, this embodiment performs standardization processing on multi-dimensional inputs, which can effectively ensure the consistency and effectiveness of the subsequent fuzzing process.

[0042] Based on the conversion of multi-dimensional inputs into a standard one-dimensional initial input operation sequence in step S110, step S120 is further executed.

[0043] S120, generate initial test cases based on the initial input operation sequence, and mutate the selected initial test cases to obtain new test cases; wherein, the initial test cases are stored in a seed pool.

[0044] It is easy to understand that before starting the testing phase, initial test cases need to be created to drive the testing process.

[0045] First, determine the maximum length (max) of the initial test case / operation sequence opSeq to be generated. n Based on historical experience, the steps required to trigger fault tolerance exceptions in a distributed system typically do not exceed 8. Therefore, this embodiment sets the maximum length max... n Set it to 8.

[0046] Next, randomly generate numbers from 1 to the maximum length max. n The unequal sequence of operations is designated as the initial test case and stored in a seed pool. Each initial test case includes a sequence of operations with at least one operation. Each operation comprises an operator and at least one operand. The operator is randomly generated with a set probability, and the operand is instantiated based on its category.

[0047] For example, in one specific embodiment, the probability is set to 1 / t, where t is the number of all different load-related operations.

[0048] Operand categories include filename category (Category FileName), node ID category (CategoryNodeId), and size category (Category Size).

[0049] Regarding the filename category: This category intends to use Treefiles to manage and record all filenames and their storage topology in the distributed file system. When instantiating a filename, either an existing filename is selected from the file tree in a uniformly random distribution, or a new filename is created and added to the file tree.

[0050] Regarding the Node ID category: This category intends to use two lists, listMN and listS, to track all management nodes and data storage nodes in the distributed system, respectively. When instantiating a Node ID, a node will be randomly selected from these lists based on a specific operator, opt. For example, if opt is add_MN, then nodeId will be randomly selected from listMN; similarly, if opt is add_storage, then nodeId will be selected from listS.

[0051] Regarding the size category: This category specifies the size of the data being operated on and tracks the remaining free space of the distributed file system. To more effectively test fault tolerance logic, boundary scenarios with varying data sizes can be created. For example, when instantiating the "create filename size" operation, the file size can be randomly assigned a value between 0 and free space.

[0052] After creating the initial test cases, the initial test cases are taken from the seed pool and mutated to obtain new test cases, in order to explore more program states and paths, thereby discovering potential vulnerabilities or errors.

[0053] Mutation operations include, but are not limited to, bit flipping, byte flipping, insertion, deletion, replacement, copying, rearrangement, arithmetic mutation, memory mutation, format string mutation, type obfuscation, and logical mutation.

[0054] After creating the initial test case and mutating it to obtain the new test case in step S120, step S130 is further executed.

[0055] S130, control the distributed system to run the new test case, and calculate the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system.

[0056] The easily understood approach is to use the mutated new test cases to test the distributed system, and control the distributed system to run the new test cases to check for new errors or vulnerabilities.

[0057] During the execution of new test cases in the distributed system, the status of the distributed nodes (i.e., nodes in the distributed system) is updated and analyzed in real time, and the load difference between each pair of nodes is calculated based on the runtime load data of the distributed nodes.

[0058] In this embodiment, the runtime load data of the distributed nodes includes, but is not limited to, computation load data, network load data, and storage load data. Specifically, computation load data includes the number of CPU cores used and the CPU core utilization rate of the distributed nodes; network load data includes the number of user requests processed, network read I / O utilization rate, and network write I / O utilization rate of the distributed nodes; and storage load data includes the storage status of the distributed nodes.

[0059] Correspondingly, the load difference between each pair of nodes is obtained by combining the differences in computing load data, network load data, and storage load data of the nodes, without making specific limitations here.

[0060] In one specific embodiment, a distributed load difference model was constructed specifically for calculating the load difference between every two nodes in a distributed system.

[0061] Based on the calculation of the load difference between each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system in step S130, step S140 is further executed.

[0062] S140, determine the target test case based on the load difference degree, and store the target test case in the seed pool to guide the next fuzz test iteration.

[0063] It is easy to understand that the load state of a distributed system can be determined based on the load difference between each pair of nodes, such as an unbalanced load state or a balanced load state.

[0064] Specifically, when the load disparity between nodes in the distributed system increases or any new imbalance faults are discovered, the new test cases currently running in the distributed system are considered valuable seeds (target test cases) and stored in the seed pool to guide the next fuzzing iteration. In this way, this embodiment can continuously generate high-quality operation sequences as test cases, aiming to maximize the load imbalance between distributed nodes.

[0065] In this embodiment, multi-dimensional inputs are converted into a one-dimensional initial input operation sequence. The multi-dimensional inputs include one or more combinations of workload, load configuration, and node failures. Initial test cases are generated based on the initial input operation sequence, and the selected initial test cases are mutated to obtain new test cases. These initial test cases are stored in a seed pool. The distributed system is controlled to run the new test cases, and the load difference between each pair of nodes in the distributed system is calculated based on the runtime load data of the nodes. Target test cases are determined based on the load difference and stored in the seed pool to guide the next fuzzing iteration. This method models the three-dimensional inputs affecting the load balancing mechanism—user requests, system configuration changes, and system failures—into a standard one-dimensional input sequence, facilitating subsequent fuzzing and improving testing efficiency. Simultaneously, by using the load difference between distributed nodes as guiding feedback to optimize the input space, high-quality test cases can be continuously generated to reveal the load imbalance between nodes in the distributed system to the greatest extent. This helps developers more effectively identify and resolve potential performance bottlenecks, improves test coverage, and enhances the overall stability and response speed of the system.

[0066] Based on the above embodiments, the following detailed description of the load difference calculation process for each pair of nodes in the distributed system is further provided.

[0067] It's easy to understand that in most distributed systems, the goal of load balancing mechanisms is to achieve a balanced load distribution among nodes, rather than maintaining absolute uniformity. Therefore, brief and slight load differences between nodes are considered normal and acceptable. However, the core insight of this embodiment is that the eventual load imbalance in the system is caused by the cumulative effect of many intermediate small load differences. Therefore, to effectively access deep code logic and efficiently trigger imbalance failures, this embodiment introduces a load difference model to guide test case generation, ensuring that nodes in the system experience as many load variations as possible.

[0068] Specifically, during the process of running new test cases in the distributed system, the status of the distributed nodes (i.e., nodes in the distributed system) is updated and analyzed in real time. The status monitor collects the runtime load data of the distributed nodes in real time and calculates the load difference between each pair of nodes based on the runtime load data of the distributed nodes.

[0069] Figure 3 This invention illustrates a distributed load difference model for describing the load difference between nodes, provided by an embodiment of the present invention.

[0070] like Figure 3As shown, the runtime load data of a node can be divided into three types: computational load data, network load data, and storage load data of nodes in a distributed system. Computational load data includes distributed node... CPU core usage , No. CPU core utilization Network load data includes distributed nodes User request processing count Network read I / O utilization Network write I / O utilization The storage load data includes distributed nodes. storage status .

[0071] Accordingly, the step of calculating the load difference between each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system specifically includes: calculating the computational load difference between each pair of nodes based on the computational load data. ; Calculate the network load difference for each pair of nodes based on network load data. Based on the storage load data, calculate the storage load difference for each pair of nodes. Based on the differences in computational load, network load, and storage load, determine the nodes for each pair. Load variability .

[0072] After calculating the load difference between each pair of nodes in the distributed system, the load difference between the distributed nodes is used as the guiding feedback to optimize the input space. This can continuously generate high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent. This can help developers more effectively identify and solve potential performance bottlenecks, improve test coverage, and also improve the overall stability and response speed of the system.

[0073] Based on the above embodiments, the following will further describe in detail the process of determining target test cases based on load difference to guide the next fuzz test.

[0074] It is easy to understand that this embodiment provides two possibilities for determining the target test cases: either the load difference increases or any new imbalance fault is discovered.

[0075] For example, in one specific embodiment, determining the target test case based on the load difference includes: if the load difference is greater than the historical load difference of the previous round of fuzzing, taking the new test case currently running in the distributed system as the target test case, and storing the target test case in the seed pool to guide the next fuzzing iteration.

[0076] Specifically, the load difference calculated in this round of fuzzing is compared with the load difference in the previous round of fuzzing (referred to as the historical load difference). If the load difference in this round of fuzzing is larger than the historical load difference in the previous round of fuzzing, the new test case currently running is taken as the target test case. The target test case is regarded as a valuable seed and stored in the seed pool to guide the next fuzzing iteration.

[0077] It should be noted that, in addition to the load difference in this round of fuzzing being greater than the historical load difference in the previous round of fuzzing, other constraints can also be set according to the actual situation, such as the difference between the load difference in this round of fuzzing and the historical load difference in the previous round of fuzzing exceeding a set threshold, etc., which are not specifically limited here.

[0078] In another specific embodiment, the target test case is determined based on the load difference, including: if one or more of the calculated load difference, network load difference, and storage load difference exceed the corresponding difference threshold, a double check is initiated to detect the real imbalance fault, and the new test case currently running in the distributed system is used as the target test case, and the target test case is stored in the seed pool to guide the next fuzz test iteration.

[0079] Specifically, the calculation of load disparity involves calculating multiple different load disparity values, including but not limited to computational load disparity, network load disparity, and storage load disparity. These values ​​are then compared to their corresponding disparity thresholds. If one or more of these thresholds exceed the specified threshold, the distributed system is considered to be in an unbalanced state, and candidate imbalance faults are identified. The candidate imbalance faults can be determined based on the type of load disparity exceeding the corresponding threshold; no specific limitations are specified here.

[0080] The difference thresholds corresponding to each load difference type can be set according to the actual situation, and are not specifically limited here. For example, preferably, in this embodiment, the product of the average node load and the set difference threshold is used as the difference threshold for the current load type, and the average node load corresponds one-to-one with each load difference type.

[0081] To confirm its effectiveness and minimize the probability of false positives, this embodiment employs a dual-check process to detect genuine imbalance faults. Upon detecting a genuine imbalance fault, the currently running new test case is designated as the target test case. This target test case is then stored as a valuable seed in the seed pool to guide the next fuzzing iteration.

[0082] In another specific embodiment, the step of initiating a double check to detect a true imbalance fault includes: determining that the distributed system is in a load imbalance state and identifying candidate imbalance faults when one or more of the calculated load difference, network load difference, and storage load difference exceed the corresponding difference threshold; explicitly calling the reload balancing API interface to control the distributed system to rerun the current new test case and check the load status; if it is determined that the distributed system is still in a load imbalance state, the identified candidate imbalance faults are identified as true imbalance faults.

[0083] It's easy to understand that different distributed systems use different load balancing mechanisms in practice. For example, CephFS computes and balances the load in real time, while GlusterFS performs load balancing through periodic scheduled tasks. Therefore, the time required for different distributed systems to complete test cases, activate the load balancing mechanism, and reach a balanced state varies. This difference makes it difficult for testing tools to accurately determine when the detector should check the system load status to identify imbalances while avoiding false alarms. The appropriate check time varies significantly depending on the specific distributed system implementation and factors in its distributed operating environment, such as latency and bandwidth.

[0084] Fortunately, most distributed systems provide API interfaces for rebalancing, allowing users to directly trigger and execute their load balancing mechanisms. To filter out false positives, the testing framework of this embodiment introduces a double-checking process. Specifically, when a candidate imbalance vulnerability is identified, this embodiment will explicitly call the rebalance API interface. Specifically, this embodiment will poll the "rebalance()" API to confirm whether the rebalancing operation has been completed. Once the API returns "rebalance completed", this testing framework will immediately re-execute the current new test case and check the system's load status. If the distributed system is still in a state of load imbalance, the identified candidate imbalance fault is confirmed as a real imbalance fault.

[0085] In this embodiment, if the load difference is greater than the historical load difference from the previous round of fuzzing, the newly running test case of the distributed system is used as the target test case. Alternatively, if one or more of the calculated load difference, network load difference, and storage load difference exceed the corresponding difference threshold, a double check is initiated to detect the true imbalance fault, and the newly running test case of the distributed system is used as the target test case. The target test case is stored in a seed pool to guide the next fuzzing iteration. This method models the three dimensions of input affecting the load balancing mechanism—user requests, system configuration changes, and system faults—into a standard one-dimensional input sequence, facilitating subsequent fuzzing and improving testing efficiency. Simultaneously, by using the load difference between distributed nodes as guiding feedback to optimize the input space, high-quality test cases can be continuously generated to reveal the load imbalance between nodes in the distributed system to the greatest extent. This helps developers more effectively identify and resolve potential performance bottlenecks, improves test coverage, and enhances the overall stability and response speed of the system. Furthermore, it can accurately detect load imbalance vulnerabilities in the distributed system in real time.

[0086] In some other embodiments, Figure 4 This diagram illustrates the overall framework of the distributed system load balancing test method based on load difference guidance provided in an embodiment of the present invention.

[0087] like Figure 4 As shown in the embodiments of the present invention, the fuzzy testing framework of the distributed system load balancing testing method based on load difference guidance mainly consists of two key components: a test case generator for generating high-quality test cases; and an imbalance detector for identifying imbalance faults.

[0088] First, this embodiment of the invention constructs a unified standard test case input model. Based on this model, multi-dimensional inputs such as client requests, system configurations, and node faults are converted into a unified one-dimensional initial input operation sequence to facilitate the subsequent fuzzing process.

[0089] Then, initial test cases are created based on the initial input operation sequence and stored in a seed pool, which is a collection used to generate different test cases through mutation. New test cases are generated by selecting and mutating existing test cases in the seed pool.

[0090] Subsequently, the distributed file system under test is controlled to execute these new test cases. The load imbalance detector monitors and collects runtime load data (such as CPU / IO utilization, storage distribution, etc.) in real time, calculates the load difference between distributed nodes, and updates the load difference model in real time.

[0091] At the same time, the imbalance detector identifies the imbalance state of the system and reports it when an imbalance fault is detected. This allows test cases that lead to new imbalance faults or larger load differences to be added to the seed pool first to guide the generation of subsequent test cases.

[0092] This testing framework will proceed to the next round of testing iterations until the load imbalance exceeds a predefined threshold. At this point, the system under test is considered to have entered a fault state, the distributed system is reset to its initial state, and the testing process restarts.

[0093] It should be noted that in this embodiment, the steps of converting multi-dimensional inputs into a one-dimensional initial input operation sequence, generating initial test cases based on the initial input operation sequence, and mutating the selected initial test cases to obtain new test cases are all implemented by the test case generator.

[0094] In this embodiment, the process of controlling the distributed system to run new test cases, calculating the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system, determining the target test cases based on the load difference, and storing the target test cases in the seed pool to guide the next fuzz test iteration is all implemented by the imbalance detector. Figure 4 The anomaly detector and anomaly load analyzer are both included in the imbalance detector.

[0095] Furthermore, Figure 5 A schematic diagram illustrating the workflow of the imbalance detector provided in an embodiment of the present invention is shown.

[0096] like Figure 5 As shown, after executing the new test cases generated in this embodiment of the invention, the state monitor collects runtime load data and calculates the load difference model (LVMij) for each pair of nodes (nodei and nodej) in the distributed system. Then, three anomaly detectors are used to evaluate the differences in computational load, network load, and storage load, respectively. These detectors determine whether the distributed system under test has entered a load imbalance state by verifying whether the maximum load in each node exceeds the average node load multiplied by the difference threshold . If an imbalance state is detected, a candidate imbalance fault is identified. To confirm its validity and minimize false positives, this embodiment also performs a double-check process to ultimately determine whether the candidate imbalance fault is a true imbalance fault.

[0097] Corresponding to the load balancing test method for distributed systems based on load difference guidance provided in the embodiments of the present invention, the present invention also proposes a load balancing test method for distributed systems based on load difference guidance and a load balancing test device for distributed systems based on load difference guidance.

[0098] Specifically, Figure 6 A schematic diagram of the structure of the distributed system load balancing test device based on load difference guidance provided in an embodiment of the present invention is shown.

[0099] like Figure 6 As shown, the device includes: a standard test case unification module 610, used to convert multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failure; an initial test case creation module 620, used to generate initial test cases according to the initial input operation sequence, and mutate the selected initial test cases to obtain new test cases; wherein the initial test cases are stored in a seed pool; a load difference calculation module 630, used to control the distributed system to run the new test cases, and calculate the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; and a target test case determination module 640, used to determine target test cases according to the load difference, and store the target test cases in the seed pool to guide the next fuzzing iteration.

[0100] In this embodiment, the standard test case unification module 610 converts multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein, the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failure; the initial test case creation module 620 generates initial test cases based on the initial input operation sequence and mutates the selected initial test cases to obtain new test cases; wherein, the initial test cases are stored in a seed pool; the load difference calculation module 630 controls the distributed system to run the new test cases and calculates the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system; the target test case determination module 640 determines the target test cases based on the load difference and stores the target test cases in the seed pool to guide the next fuzzing iteration. This device models the three dimensions of input affecting the load balancing mechanism—user requests, system configuration changes, and system failures—into a standard one-dimensional input sequence, facilitating subsequent fuzz testing and improving testing efficiency. Simultaneously, by using the load disparity between distributed nodes as guiding feedback to optimize the input space, it can continuously generate high-quality test cases to reveal the load imbalance between nodes in the distributed system to the greatest extent possible. This helps developers more effectively identify and resolve potential performance bottlenecks, improving test coverage and enhancing the overall stability and response speed of the system.

[0101] It should be noted that the distributed system load balancing test device based on load difference guidance provided in the embodiments of the present invention can be referred to in correspondence with the distributed system load balancing test methods based on load difference guidance described in the above embodiments.

[0102] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a distributed system load balancing test method guided by load difference. This method includes: converting multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failures; generating initial test cases based on the initial input operation sequence, and mutating selected initial test cases to obtain new test cases; wherein the initial test cases are stored in a seed pool; controlling the distributed system to run the new test cases, and calculating the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system; determining target test cases based on the load difference, and storing the target test cases in the seed pool to guide the next fuzzing iteration.

[0103] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the distributed system load balancing test method based on load difference guidance provided by the above methods. The method includes: converting multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failures; generating initial test cases according to the initial input operation sequence, and mutating the selected initial test cases to obtain new test cases; wherein the initial test cases are stored in a seed pool; controlling the distributed system to run the new test cases, and calculating the load difference of each pair of nodes in the distributed system according to the runtime load data of the nodes in the distributed system; determining target test cases according to the load difference, and storing the target test cases in the seed pool to guide the next fuzz test iteration.

[0105] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a distributed system load balancing testing method based on load difference guidance provided by the above methods. This method includes: converting multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node failures; generating initial test cases based on the initial input operation sequence, and mutating selected initial test cases to obtain new test cases; wherein the initial test cases are stored in a seed pool; controlling the distributed system to run the new test cases, and calculating the load difference degree for each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system; determining target test cases based on the load difference degree, and storing the target test cases in the seed pool to guide the next fuzzing iteration.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A load balancing test method for distributed systems based on load difference guidance, characterized in that, include: Transform multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein the multi-dimensional inputs include one or more combinations of workload, load configuration, and node faults; Initial test cases are generated based on the initial input operation sequence, and the selected initial test cases are mutated to obtain new test cases; wherein, the initial test cases are stored in a seed pool; Control the distributed system to run the new test cases, and calculate the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system; The target test cases are determined based on the load difference and stored in the seed pool to guide the next fuzz test iteration.

2. The distributed system load balancing test method based on load difference guidance according to claim 1, characterized in that, The runtime load data includes the computation load data, network load data, and storage load data of the nodes in the distributed system. The computation load data includes the number of CPU cores used by the node and the CPU core utilization rate. The network load data includes the number of user requests processed by the node, the network read IO utilization rate, and the network write IO utilization rate. The storage load data includes the storage status of the node. Accordingly, the step of calculating the load difference between each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system specifically includes: Based on the computational load data, calculate the computational load difference for each pair of nodes; Based on the network load data, calculate the network load difference for each pair of nodes; Based on the storage load data, calculate the storage load difference for each pair of nodes; The load difference degree for each pair of nodes is determined based on the calculated load difference, network load difference, and storage load difference.

3. The distributed system load balancing test method based on load difference guidance according to claim 2, characterized in that, Determining target test cases based on the load difference includes: If the load difference is greater than the historical load difference of the previous round of fuzz testing, the new test case currently running in the distributed system will be used as the target test case.

4. The distributed system load balancing test method based on load difference guidance according to claim 2, characterized in that, Determining target test cases based on the load difference includes: If one or more of the computational load difference, network load difference, and storage load difference exceed the corresponding difference threshold, a double check is initiated to detect the true imbalance fault, and the new test case currently running in the distributed system is used as the target test case.

5. The distributed system load balancing test method based on load difference guidance according to claim 4, characterized in that, The step of initiating a dual check to detect a true imbalance fault specifically includes: If one or more of the computational load difference, network load difference, and storage load difference exceed the corresponding difference threshold, the distributed system is determined to be in a state of load imbalance, and candidate imbalance faults are identified. Explicitly call the reload balancing API interface to control the distributed system to rerun the current new test cases and check the load status; If it is determined that the distributed system is still in a state of unbalanced load, the identified candidate unbalanced faults will be identified as the real unbalanced faults.

6. The distributed system load balancing test method based on load difference guidance according to any one of claims 1-5, characterized in that, Generate initial test cases based on the initial input operation sequence, including: Determine the maximum length of the initial test cases to be generated; Randomly generate operation sequences ranging from one to a maximum length, denoted as initial test cases; The initial test case includes an operation sequence of at least one operation, the operation including an operator and at least one operand, the operator being randomly generated with a set probability, and the operand being instantiated according to its category.

7. A load balancing test device for distributed systems based on load difference guidance, characterized in that, include: A standard test case unification module is used to convert multi-dimensional inputs into a one-dimensional initial input operation sequence; wherein, the multi-dimensional inputs include one or more combinations of workload, load configuration, and node faults; An initial test case creation module is used to generate initial test cases based on the initial input operation sequence, and mutate the selected initial test cases to obtain new test cases; wherein, the initial test cases are stored in a seed pool; The load difference calculation module is used to control the distributed system to run the new test case and calculate the load difference of each pair of nodes in the distributed system based on the runtime load data of the nodes in the distributed system. The target test case determination module is used to determine target test cases based on the load difference and store the target test cases in a seed pool to guide the next fuzz test iteration.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the distributed system load balancing test method based on load difference as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed system load balancing test method based on load difference guidance as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the distributed system load balancing test method based on load difference guidance as described in any one of claims 1 to 6.

Images