Method and system for realizing multi-node cloud hard disk performance test

By adopting a master-slave architecture and multi-node testing method with optimized communication protocols in cloud hard disk performance testing, the challenge of performance testing in complex cloud hard disk environments is solved, efficient and accurate test results are achieved, and user experience and service quality are improved. .

CN119049531APending Publication Date: 2024-11-29SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411049429.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the field of cloud computing, ensuring the stability and high performance of cloud hard disks in complex environments is a major challenge. It is difficult for existing technology to effectively implement distributed performance testing of parallel operation of multi-cloud hard disks, affecting the optimization of storage services and user experience.

Method used

Using a multi-node cloud hard disk performance testing method, the master-slave architecture of the master-management node and the work execution node is used to utilize persistent TCP connections or optimized HTTP communication protocols, combined with configuration file parsing, CountDownLatch synchronization mechanism and load generator selection, Achieve efficient testing tasks allocation and execution, ensure test synchronization and data consistency, and improve testing efficiency and accuracy.

Benefits of technology

It significantly improves the efficiency and accuracy of cloud hard disk performance testing, enhances system flexibility and scalability, ensures the accuracy and consistency of test results, and improves the user experience and service quality of end users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049531A_ABST
    Figure CN119049531A_ABST
Patent Text Reader

Abstract

The invention discloses a method and system for realizing a multi-node cloud hard disk performance test, and belongs to the technical field of data storage, and the system comprises at least one main management node which is responsible for configuring a test environment, distributing a test task, and collecting and analyzing a test result; the plurality of work execution nodes are distributed on virtual machines associated with different cloud hard disks, execute the test tasks distributed by the main management node and report test data; instruction transmission and data exchange are carried out between the main management node and the work execution node through persistent TCP (Transmission Control Protocol) connection or an optimized HTTP (Hyper Text Transport Protocol) communication protocol; and analyzing the configuration file in the YAML format through a configuration file analysis module, and guiding the system to automatically configure a test environment and parameters. According to the method, the test efficiency and precision can be improved, resource optimization and cost control are realized, the flexibility and expansibility of the system are enhanced, test synchronization and data consistency are ensured, and finally the use experience and service quality of a terminal user are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a method and system for implementing multi-node cloud hard disk performance testing. Background Art

[0002] In the field of cloud computing, cloud hard disks, as an advanced form of block storage, use virtualization technology to achieve on-demand allocation and dynamic expansion of storage resources, meeting the needs of diverse business scenarios. However, ensuring the stability and high performance of cloud hard disks in complex environments has become a major challenge for cloud service providers. Therefore, building a distributed performance test system that can simulate the parallel operation of multiple cloud hard disks under real usage conditions is crucial for verifying and optimizing storage services. Summary of the invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a method and system for implementing multi-node cloud hard disk performance testing, which can improve test efficiency and accuracy, achieve resource optimization and cost control, enhance system flexibility and scalability, and ensure test synchronization and data consistency, improve the versatility and adaptability of test tasks, and ultimately improve the user experience and service quality of end users.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A method for implementing a multi-node cloud hard disk performance test, comprising:

[0006] At least one master management node, responsible for configuring the test environment, assigning test tasks, and collecting and analyzing test results;

[0007] Multiple work execution nodes are distributed on virtual machines associated with different cloud disks, executing test tasks assigned by the main management node and reporting test data;

[0008] The main management node and the work execution node transmit instructions and exchange data through a persistent TCP connection or an optimized HTTP communication protocol to reduce network resource consumption and ensure efficient communication;

[0009] Equipped with a configuration file parsing module, which is used to parse the configuration files in YAML format and guide the system to automatically configure the test environment and parameters;

[0010] The synchronization mechanism based on CountDownLatch ensures the synchronous execution of multi-node test tasks and improves the accuracy and consistency of test results;

[0011] Based on the load generator selection mechanism, a variety of test tools are supported through a simple factory mode, which enhances the flexibility and compatibility of test tasks;

[0012] IO multiplexing is implemented using Java and Netty frameworks, which optimizes concurrent processing capabilities and supports efficient test scheduling under large-scale concurrent connections.

[0013] This performance testing method adopts a classic master-slave architecture design, which consists of a centrally commanded master management node and a group of work execution nodes distributed on each cloud disk-associated virtual machine. As the nerve center of the system, the master management node is responsible for comprehensively supervising the test process. Its core functions include: issuing test files to each work execution node, finely allocating test tasks, aggregating and integrating test results and test data analysis, etc., to ensure the orderly progress of test activities. In contrast, the work execution nodes focus on the actual execution level of tasks. They follow the instructions of the master management node and are responsible for receiving test tasks, preparing the test environment, executing test scripts, reporting test results, and necessary resource cleanup and recycling work, so as to ensure the efficient execution of test jobs and maximize the utilization of resources. Such an architectural design not only realizes the centralization and efficiency of test management, but also ensures the distribution and flexibility of test execution. It is the key to supporting the concurrent performance evaluation of large-scale cloud disks.

[0014] Furthermore, the main management node includes:

[0015] The test task scheduling module dynamically allocates test tasks to each work execution node according to preset strategies.

[0016] Furthermore, the work execution node includes:

[0017] The environment preparation module is responsible for receiving instructions from the main management node and completing the construction and configuration of the test environment;

[0018] Test execution module, executes test scripts, simulates user operations, and generates performance test data;

[0019] The result reporting module directly feeds back the test results to the main management node through the established connection.

[0020] Furthermore, the content defined in the configuration file includes: target file system information, worker node configuration, workload generator parameters, and supports one-click deployment and flexible adjustment of test parameters.

[0021] Furthermore, a synchronization control mechanism is implemented based on Java concurrent programming to ensure that in multi-node concurrent testing, all nodes start and end test tasks synchronously, improving the reliability of test results and the controllability of the test process.

[0022] Furthermore, the specific process of implementing this method is as follows:

[0023] 1) Configuration and startup of the main management node: Configure the test environment on the main management node, including loading the configuration file containing the work execution node details, the target cloud disk file system type (such as ext4, xfs), and the selected test tools and scripts; then, by executing the java-jar command, officially start the main management node test management system, laying the foundation for the global control of the test;

[0024] 2) Configuration and deployment to work execution nodes: With the master management node ready, it uses the scp command to distribute the test system to each work execution node based on the work execution node information recorded in the configuration file; then, it executes the same java-jar command on each work execution node to activate their respective test execution environments, paving the way for parallel processing of test tasks;

[0025] 3) Test task distribution: After confirming that all work execution nodes have been successfully deployed and are on standby, the main management node issues specific test task instructions to each work execution node based on the pre-planned test plan, starting the substantive operation phase of the test;

[0026] 4) Execution and simulation testing: After receiving the instruction, the work execution node immediately starts the test task and strictly follows the preset test procedures;

[0027] 5) Result collection and analysis: After the test is completed, the work execution node quickly feeds back the test data to the main management node, cleans up the test residues through the rm-rf and umount commands, and restores the initial state; the main management node collects all reported test results and carefully integrates them into the summary report, providing detailed data support for the technical team and operation and maintenance personnel, and facilitating in-depth analysis.

[0028] Furthermore, the execution and simulation test involves creating a test directory (mkdir), formatting and establishing a file system as required (mkfs.X), mounting the new file system to the test directory (mount), and finally performing performance testing in the directory using specified test tools and scripts to fully simulate user interaction and system load in real application scenarios.

[0029] The present invention also claims a system for implementing multi-node cloud hard disk performance testing, comprising:

[0030] At least one master management node, responsible for configuring the test environment, assigning test tasks, and collecting and analyzing test results;

[0031] Multiple work execution nodes are distributed on virtual machines associated with different cloud disks, executing test tasks assigned by the main management node and reporting test data;

[0032] The main management node and the work execution node transmit instructions and exchange data through a persistent TCP connection or an optimized HTTP communication protocol to reduce network resource consumption and ensure efficient communication;

[0033] Configuration file parsing module, used to parse YAML format configuration files and guide the system to automatically configure the test environment and parameters;

[0034] The synchronization module based on CountDownLatch is used to ensure the synchronous execution of multi-node test tasks and improve the accuracy and consistency of test results;

[0035] A load generator-based selection module is used to support multiple test tools through a simple factory mode, which enhances the flexibility and compatibility of test tasks;

[0036] Implemented IO multiplexing based on Java and Netty framework, optimized concurrent processing capabilities, and supported efficient test scheduling under large-scale concurrent connections;

[0037] The system can implement the above method.

[0038] The present invention also claims a device for implementing a multi-node cloud hard disk performance test, comprising: at least one memory and at least one processor;

[0039] The at least one memory is used to store a machine-readable program;

[0040] The at least one processor is used to call the machine-readable program to implement the above method.

[0041] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which, when executed by a processor, causes the processor to perform the above-mentioned method.

[0042] Compared with the prior art, the method and system for implementing multi-node cloud hard disk performance testing of the present invention has the following beneficial effects:

[0043] 1. Significantly improve test efficiency: This invention significantly improves the efficiency and accuracy of cloud hard disk performance testing by adopting advanced distributed architecture and optimization algorithms. It can complete large-scale concurrent testing in a short time, quickly locate system bottlenecks, provide real-time feedback for optimizing storage services, and accelerate product iteration and service quality upgrades;

[0044] 2. Enhance system flexibility and scalability: The modular design and highly configurable features enable the system to flexibly adapt to changing business needs and technical environments. Whether it is the expansion of test scale or the introduction of new storage technology, it can be quickly adjusted and deployed to ensure the long-term value and adaptability of the test platform;

[0045] 3. Ensure data accuracy and consistency: Through rigorous concurrency control strategies, the data synchronization problem in multi-threaded or multi-node testing is effectively solved, ensuring the high reliability of test results, providing a solid data foundation for decision-making, and avoiding misjudgment and waste of resources due to inconsistent data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram showing the principle of a method for implementing multi-node cloud hard disk performance testing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0048] An embodiment of the present invention provides a method for implementing a multi-node cloud hard disk performance test, including:

[0049] At least one master management node, responsible for configuring the test environment, assigning test tasks, and collecting and analyzing test results;

[0050] Multiple work execution nodes are distributed on virtual machines associated with different cloud disks, executing test tasks assigned by the main management node and reporting test data;

[0051] The main management node and the work execution node transmit instructions and exchange data through a persistent TCP connection or an optimized HTTP communication protocol to reduce network resource consumption and ensure efficient communication;

[0052] Equipped with a configuration file parsing module, which is used to parse the configuration files in YAML format and guide the system to automatically configure the test environment and parameters;

[0053] The synchronization mechanism based on CountDownLatch ensures the synchronous execution of multi-node test tasks and improves the accuracy and consistency of test results;

[0054] Based on the load generator selection mechanism, a variety of test tools are supported through a simple factory mode, which enhances the flexibility and compatibility of test tasks;

[0055] IO multiplexing is implemented using Java and Netty frameworks, which optimizes concurrent processing capabilities and supports efficient test scheduling under large-scale concurrent connections.

[0056] Wherein, the main management node includes:

[0057] The test task scheduling module dynamically allocates test tasks to each work execution node according to preset strategies.

[0058] The work execution node includes:

[0059] The environment preparation module is responsible for receiving instructions from the main management node and completing the construction and configuration of the test environment;

[0060] Test execution module, executes test scripts, simulates user operations, and generates performance test data;

[0061] The result reporting module directly feeds back the test results to the main management node through the established connection.

[0062] The persistent TCP connection communication solution adopted realizes two-way data transmission between the main management node and the work execution node, eliminating the need for frequent handshakes and disconnections, thereby improving communication efficiency.

[0063] The content defined in the configuration file includes: target file system information, worker node configuration, workload generator parameters, and supports one-click deployment and flexible adjustment of test parameters.

[0064] The synchronization control mechanism implemented based on Java concurrent programming ensures that in multi-node concurrent testing, all nodes start and end test tasks synchronously, improving the reliability of test results and the controllability of the test process.

[0065] It supports the integration of multiple load generation tools. Through a simple factory mode, it dynamically instantiates the corresponding load generator according to the configuration, thus achieving the pluggability of the test tools and the diversity of the test scenarios.

[0066] In general, this performance testing method adopts a classic master-slave architecture design, which consists of a centrally commanded master management node (referred to as the master node) and a group of work execution nodes (called worker nodes) distributed on each cloud disk-associated virtual machine. As the nerve center of the system, the master node is responsible for the overall supervision of the test process. Its core functions include: sending test files to each worker node, finely allocating test tasks, aggregating and integrating test results and test data analysis, etc., to ensure the orderly progress of test activities. In contrast, the worker nodes focus on the actual execution of tasks. They follow the instructions of the master node, and are responsible for receiving test tasks, preparing the test environment, executing test scripts, reporting test results, and necessary resource cleanup and recycling work, so as to ensure the efficient execution of test jobs and maximize the utilization of resources. Such an architectural design not only realizes the centralization and efficiency of test management, but also ensures the distribution and flexibility of test execution. It is the key to supporting the concurrent performance evaluation of large-scale cloud disks.

[0067] Combined with Figure 1 As shown in the figure, the sequence of steps in the performance testing process is as follows:

[0068] 1. Master node configuration and startup: The first step is to carefully configure the test environment on the master node, including loading the configuration file containing worker node details, target cloud disk file system type (such as ext4, xfs), selected test tools and scripts. Then, by executing the java-jar command, the test management system of the master node is officially started, laying the foundation for the global control of the test.

[0069] 2. Configuration and deployment to worker nodes: With the master node ready, it distributes the test system to each worker node using the scp command based on the worker node information recorded in the configuration file. Immediately afterwards, the same java-jar command is executed on each worker node to activate their respective test execution environments, paving the way for parallel processing of test tasks.

[0070] 3. Test task distribution: After confirming that all worker nodes have been successfully deployed and are on standby, the master node issues specific test task instructions to each worker node based on the pre-planned test plan to start the substantive operation phase of the test.

[0071] 4. Execution and simulation test: After receiving the command, the worker node immediately starts the test task and strictly follows the preset test procedures. This stage involves creating a test directory (mkdir), formatting and establishing a file system according to requirements (mkfs.X), mounting the new file system to the test directory (mount), and finally using the specified test tools and scripts in the directory to carry out performance testing, fully simulating user interaction and system load in real application scenarios.

[0072] 5. Result collection and analysis: After the test is completed, the worker node quickly feeds back the test data to the master node, cleans up the test residues through the rm-rf and umount commands, and restores the initial state; the master node collects all reported test results and carefully integrates them into the summary report, providing detailed data support for the technical team and operation and maintenance personnel, and facilitating in-depth analysis.

[0073] The master and worker nodes in the performance testing framework are deployed in independent virtual machine environments, and smooth communication between them depends on a robust network infrastructure. There are three strategies worth considering for the design of the communication mechanism between the two:

[0074] Solution 1: The worker nodes receive and execute test instructions via the HTTP protocol, while the master node uses a regular polling mechanism to continuously send HTTP requests to each worker node to collect test progress and results. This method is easy to implement, but frequent polling activities can easily lead to unnecessary network traffic and system resource consumption.

[0075] Solution 2: This solution uses the HTTP protocol in the task distribution phase, which is similar to the previous solution. The difference is that in the test result reporting phase, the master plays the role of the server and waits for the worker node to actively upload the test results via HTTP. Although this strategy reduces the polling burden, it requires both parties to have the dual functions of client and server, making the system architecture relatively complex.

[0076] Solution 3: Different from the previous two, this solution establishes a persistent TCP connection between the master and worker nodes at the beginning of task delivery. The test data is directly transmitted through these pre-established connections, which not only saves the overhead of repeatedly establishing connections, but also avoids the waste of resources caused by polling. In addition, it clearly defines the master as the command initiator (client), and the worker focuses on receiving commands and feedback results (server), which simplifies the system architecture and improves efficiency.

[0077] In summary, although Solution 1 is easy to implement quickly, its continuous polling mechanism causes unnecessary occupation of resources that cannot be ignored. Solution 2 is an improvement, but its architecture complexity has increased. In contrast, Solution 3 uses TCP long connection technology, which not only effectively reduces resource waste, but also simplifies system logic, thus showing significant advantages in comprehensive performance and system simplicity.

[0078] At the initial stage of system startup, the master node needs to parse and load the core configuration file. The configuration file is written in YAML format and defines the relevant settings of the file system to be tested and the specific parameters of the performance test task. The following is an example of the configuration file:

[0079]

[0080] Among them, the target configuration section specifies the target file system for this performance test. Each client in mounts is the information of a worker node, including its IP address (address) and the corresponding drive letter (mountPoints). These configurations instruct the master node on how to send software packages to worker nodes and pull up applications and assist each worker node in building the infrastructure of the distributed test environment. The workLoad configuration section defines the selection and configuration of the workload generator. The provider indicates the type of workload, which can be fio, iozone or other load generation tools. The script indicates the test script that needs to be run for the performance test. It can define specific test modes, parameter sets, execution cycles and other details to ensure that the test tool can simulate actual application scenarios and impose expected loads on the target file system, thereby accurately evaluating its performance.

[0081] The traditional BIO (Blocking I / O) model was widely used in Java language in early network communication. This model works in a blocking manner, which means that when a thread processes a TCP connection, it will focus entirely on the operation of the connection until the read and write operations on the connection are completed. Under this mechanism, a thread can only correspond to one TCP connection and cannot handle multiple connection requests at the same time. As the number of worker nodes increases, the master node faces a surge in the number of threads. To solve this problem, the performance testing system uses IO multiplexing technology, which allows a single thread to efficiently monitor and process multiple TCP connections at the same time. Netty is a popular Java language RPC (Remote Procedure Call) framework. When initialized, the server configures two dedicated thread pools: one for managing the establishment and closing of connections (parent thread pool), and the other for processing business logic (child thread pool). First, set the parent thread pool size of the worker node to 1, because the master node only needs a single thread to efficiently accept and manage the connection requests of all worker nodes without additional thread overhead. Secondly, considering the serial execution requirements of test tasks, in order to avoid mutual interference between parallel tasks affecting the test results, the child thread pool of the worker node is also limited to contain only one thread. This configuration ensures that only one test task is executed in the system at any time, which not only prevents over-allocation of resources, but also maintains the consistency and reliability of test data.

[0082] During the execution of the test task, considering that the worker nodes may take different time to execute the test commands due to load differences, this may lead to test synchronization issues. For example, some nodes are performing read operations while other nodes are performing write operations, which may pose a challenge to the accuracy of the test results. To deal with this problem, the master node adopts a batch and synchronous execution strategy when distributing test instructions. The master node first divides all test commands into multiple groups according to their responsibilities, and serializes these groups into queues. During initialization, the total number of currently active worker nodes is recorded, and then a group of commands is extracted from the queue in sequence and distributed to each worker. Whenever a worker completes its task group, it decrements the counter and temporarily blocks the execution thread corresponding to the worker until the counters of all workers are zero. At this time, the counter is reset to the total number of workers, and the next batch of commands are extracted from the queue for a new round of distribution. This process repeats until the queue is cleared, and finally all test results are summarized and recorded in a file. The CountDownLatch synchronization auxiliary class introduced in Java 5 appropriately meets this synchronization requirement. It allows an initial count value to be set, and each call to the countDown() method will reduce the count by 1. Other threads can use the await() method to block themselves until the count drops to zero, thereby ensuring that all worker nodes can synchronously start the execution of each set of commands, effectively maintaining the accuracy and consistency of the test data, and improving the rigor and effectiveness of the test.

[0083] The performance test program is written in Java. As an object-oriented language, the native polymorphism can support multiple load generators more elegantly. Specifically, a simple factory pattern is used to implement the selection of multiple load generators. First, an interface class is defined, which contains three methods: buildScriptQueue(), run(), and exportResult(). The buildScriptQueue() method is responsible for grouping test scripts into a queue; the run() method is responsible for executing the performance test script and formatting the test results; the exportResult() method is responsible for exporting the test results to a file for the technical team and operation and maintenance personnel to view. Then, the specific load generator is implemented. For each supported load generator (such as FIO, iozone, etc.), the corresponding Java class is written to implement the above interface. These implementation classes need to rewrite the buildScriptQueue(), run(), and exportResult() methods defined in the interface to adapt to the characteristics and interfaces of each tool, ensuring that when executing specific performance testing tasks, the functions of the underlying tools can be correctly called and the unified data output standards can be followed. Finally, implement a factory class that contains a static factory method that can dynamically instantiate and return the corresponding interface implementation object based on the load generation tool in the configuration.

[0084] As a mature data storage solution, block storage presents storage resources to users in refined "data block" units, each of which represents an independent and continuous hard disk space. Cloud hard disk is a virtual block storage service tailored for virtual machine environments. It creates a low-cost, highly reliable and elastic storage solution by flexibly allocating reserved capacity. On this basis, users can freely build file systems such as ext4 and xfs to fully tap the potential of cloud hard disks. In actual deployment, block storage systems can efficiently virtualize and differentiate multiple cloud hard disks from a single storage pool to support diverse business operation requirements. In view of this, carefully designing and accurately simulating complex scenarios of multiple cloud hard disks operating in parallel under this architecture is particularly critical to ensure the quality of service and the rigor of data processing. Therefore, it is inevitable to build a distributed performance testing system that can run across multiple clients, which can accurately reflect the actual performance of multiple cloud hard disks in complex network environments and high load conditions.

[0085] This method adopts a master-slave architecture, with the master node coordinating global test management and data analysis, and the worker node executing test tasks in parallel. Through carefully planned processes and strategies, the concurrent performance evaluation of large-scale cloud hard disks is realized; Performance test system design: adopts a master-slave architecture, the master node is responsible for test process control and data analysis, and the worker node executes specific tests to ensure centralized control and distributed execution efficiency of the test; Communication mechanism comparison: three communication strategies (HTTP polling, HTTP upload results, TCP long connection) are discussed, and TCP long connection is finally adopted to reduce resource consumption and improve efficiency; Configuration and deployment: The system initialization is guided by the configuration file in YAML format, the test objectives and parameters are clarified, different load generators and test scripts are supported, and various test requirements are adapted; Technical optimization: The Netty framework and IO multiplexing technology are used to improve communication efficiency, and CountDownLatch is used to ensure the synchronous execution of test commands and maintain test data consistency; Polymorphism and simple factory mode: The object-oriented characteristics of Java are used to implement polymorphic support and dynamic instantiation of load generators, which enhances the flexibility and scalability of the system.

[0086] The embodiment of the present invention further provides a system for implementing multi-node cloud hard disk performance testing, which can implement the method for implementing multi-node cloud hard disk performance testing described in the above embodiment; the system includes:

[0087] At least one master management node, responsible for configuring the test environment, assigning test tasks, and collecting and analyzing test results;

[0088] Multiple work execution nodes are distributed on virtual machines associated with different cloud disks, executing test tasks assigned by the main management node and reporting test data;

[0089] The main management node and the work execution node transmit instructions and exchange data through a persistent TCP connection or an optimized HTTP communication protocol to reduce network resource consumption and ensure efficient communication;

[0090] Configuration file parsing module, used to parse YAML format configuration files and guide the system to automatically configure the test environment and parameters;

[0091] The synchronization module based on CountDownLatch is used to ensure the synchronous execution of multi-node test tasks and improve the accuracy and consistency of test results;

[0092] A load generator-based selection module is used to support multiple test tools through a simple factory mode, which enhances the flexibility and compatibility of test tasks;

[0093] IO multiplexing is implemented based on Java and Netty framework, which optimizes concurrent processing capabilities and supports efficient test scheduling under large-scale concurrent connections.

[0094] Wherein, the main management node includes:

[0095] The test task scheduling module dynamically allocates test tasks to each work execution node according to preset strategies.

[0096] The work execution node includes:

[0097] The environment preparation module is responsible for receiving instructions from the main management node and completing the construction and configuration of the test environment;

[0098] Test execution module, executes test scripts, simulates user operations, and generates performance test data;

[0099] The result reporting module directly feeds back the test results to the main management node through the established connection.

[0100] The persistent TCP connection communication solution adopted realizes two-way data transmission between the main management node and the work execution node, eliminating the need for frequent handshakes and disconnections, thereby improving communication efficiency.

[0101] The content defined in the configuration file includes: target file system information, worker node configuration, workload generator parameters, and supports one-click deployment and flexible adjustment of test parameters.

[0102] The synchronization control mechanism implemented based on Java concurrent programming ensures that in multi-node concurrent testing, all nodes start and end test tasks synchronously, improving the reliability of test results and the controllability of the test process.

[0103] It supports the integration of multiple load generation tools. Through a simple factory mode, it dynamically instantiates the corresponding load generator according to the configuration, thus achieving the pluggability of the test tools and the diversity of the test scenarios.

[0104] The system finally achieved the following: 1. Improve test efficiency and accuracy: Design and implement a highly integrated distributed performance test system. Through the optimized master-slave architecture, ensure that the test tasks can efficiently and accurately simulate the complex scenarios of multiple cloud hard disks operating in parallel in the real world, and improve the representativeness and reliability of the test results. 2. Resource optimization and cost control: Introduce innovative communication mechanisms, such as TCP long connections, to reduce network communication overhead and system resource consumption. Compared with the traditional polling method, this design aims to reduce operating costs while enhancing the system's response speed and processing capabilities. 3. Enhance system flexibility and scalability: By adopting Java and Netty frameworks, combined with IO multiplexing technology, effective management of large-scale concurrent connections is achieved, ensuring that the system can easily cope with the expansion of cloud hard disk scale and the growth of testing needs in the future, while maintaining the simplicity and maintainability of the architecture. 4. Ensure test synchronization and data consistency: Innovatively apply concurrency control mechanisms, such as CountDownLatch, to solve the test synchronization problem caused by asynchronous operations of multiple nodes, ensure the consistent execution order of test steps, and maintain the integrity and accuracy of test data. 5. Improve the versatility and adaptability of test tasks: Design a configurable and modular load generator selection mechanism, and use a simple factory mode to support the seamless access of multiple test tools (such as FIO, iozone), so that the test system can flexibly adapt to changes in different file system types and business needs, and enhance the wide applicability and deep customization capabilities of the test solution. 6. Simplify the configuration and management process: Through the carefully designed YAML format configuration file, one-click deployment and parameterized configuration of the test environment are achieved, simplifying the system initialization and task distribution process, reducing the complexity of operations, and improving the efficiency of test preparation.

[0105] An embodiment of the present invention further provides a device for implementing a multi-node cloud hard disk performance test, comprising: at least one memory and at least one processor;

[0106] The at least one memory is used to store a machine-readable program;

[0107] The at least one processor is used to call the machine-readable program to implement the method for implementing multi-node cloud hard disk performance testing described in the above embodiment.

[0108] The embodiment of the present invention also provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by the processor, the processor executes the method for implementing the multi-node cloud hard disk performance test described in the above embodiment. Specifically, a system or device equipped with a storage medium can be provided, on which the software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0109] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0110] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0111] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0112] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0113] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for implementing multi-node cloud hard disk performance testing, characterized in that: include: At least one master management node, responsible for configuring the test environment, assigning test tasks, and collecting and analyzing test results; Multiple work execution nodes are distributed on virtual machines associated with different cloud disks, executing test tasks assigned by the main management node and reporting test data; The main management node and the work execution node perform instruction transmission and data exchange via a persistent TCP connection or an optimized HTTP communication protocol; The configuration file parsing module parses the YAML format configuration file and guides the system to automatically configure the test environment and parameters; The synchronization mechanism based on CountDownLatch ensures the synchronous execution of multi-node test tasks; Based on the load generator selection mechanism, it supports multiple testing tools through a simple factory mode; Use Java and Netty framework to implement IO multiplexing and support efficient test scheduling under large-scale concurrent connections.

2. A method for implementing multi-node cloud hard disk performance testing according to claim 1, characterized in that: The main management node includes: The test task scheduling module dynamically allocates test tasks to each work execution node according to preset strategies.

3. A method for implementing multi-node cloud hard disk performance testing according to claim 1 or 2, characterized in that: The work execution node includes: The environment preparation module is responsible for receiving instructions from the main management node and completing the construction and configuration of the test environment; Test execution module, executes test scripts, simulates user operations, and generates performance test data; The result reporting module directly feeds back the test results to the main management node through the established connection.

4. A method for implementing multi-node cloud hard disk performance testing according to claim 1, characterized in that: The content defined in the configuration file includes: target file system information, worker node configuration, workload generator parameters, and supports one-click deployment and flexible adjustment of test parameters.

5. A method for implementing multi-node cloud hard disk performance testing according to claim 1, characterized in that: A synchronization control mechanism is implemented based on Java concurrent programming to ensure that in multi-node concurrent testing, all nodes start and end test tasks synchronously.

6. A method for implementing multi-node cloud hard disk performance testing according to claim 1, characterized in that: The specific process of implementing this method is as follows: 1) Configuration and startup of the main management node: Configure the test environment on the main management node, including loading the configuration file containing the work execution node details, the target cloud disk file system type, and the selected test tools and scripts; Then, by executing the java-jar command, the main management node test management system is officially started; 2) Configuration and deployment to work execution nodes: As the master management node is ready, it uses the scp command to distribute the test system to each work execution node based on the work execution node information recorded in the configuration file; then, it executes the same java-jar command on each work execution node to activate their respective test execution environments; 3) Test task distribution: After confirming that all work execution nodes have been successfully deployed and are on standby, the main management node issues specific test task instructions to each work execution node based on the pre-planned test plan, starting the substantive operation phase of the test; 4) Execution and simulation testing: After receiving the instruction, the work execution node immediately starts the test task and strictly follows the preset test procedures; 5) Result collection and analysis: After the test is completed, the working execution node quickly feeds back the test data to the main management node, and cleans up the test residues through the rm-rf and umount commands to restore the initial state; the main management node collects all reported test results and integrates them into a summary report.

7. A method for implementing multi-node cloud hard disk performance testing according to claim 6, characterized in that: The execution and simulation test involves creating a test directory, formatting and establishing a file system as required, mounting the new file system to the test directory, and finally performing performance testing in the directory using specified test tools and scripts to fully simulate user interaction and system load in real application scenarios.

8. A system for implementing multi-node cloud hard disk performance testing, characterized in that: include: At least one master management node, responsible for configuring the test environment, assigning test tasks, and collecting and analyzing test results; Multiple work execution nodes are distributed on virtual machines associated with different cloud disks, executing test tasks assigned by the main management node and reporting test data; The main management node and the work execution node perform instruction transmission and data exchange via a persistent TCP connection or an optimized HTTP communication protocol; Configuration file parsing module, used to parse YAML format configuration files and guide the system to automatically configure the test environment and parameters; A synchronization module based on CountDownLatch to ensure the synchronous execution of multi-node test tasks; Load generator-based selection module to support multiple testing tools through a simple factory pattern; Implement IO multiplexing based on Java and Netty framework, and support efficient test scheduling under large-scale concurrent connections; The system can implement the method described in any one of claims 1 to 7.

9. A device for implementing multi-node cloud hard disk performance testing, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.

10. A computer readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Multi-channel extended hard disk automatic testing device and method

    CN119851745A

  • Method and device for testing data transmission rate of solid state disk

    CN120066869A

  • Solid State Drive Data Transfer Rate Testing Method and Apparatus

    CN120066869B

  • Storage management method and device, equipment, storage medium and program product

    CN120523412A

  • Storage management method, device, equipment, storage medium and program product

    CN120523412B