Performance test method and device for distributed storage system

Abstract

The invention relates to the technical field of distributed storage, and discloses a performance testing method and device for a distributed storage system. The method comprises the following steps: building a test environment for simulating a real application scene; automatically executing a test method on the distributed storage system based on the test environment, and collecting, monitoring and storing performance data in real time in the test process; analyzing and processing the performance data to obtain a test result; and optimizing the performance of the distributed storage system according to the test result. According to the method, through a multi-dimensional testing method and an automatic testing process, the technical problem of low efficiency in existing traditional testing is effectively solved.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and for example to a performance testing method and apparatus for a distributed storage system. Background Technology

[0002] With the advent of the big data era, distributed storage systems, with their advantages of high scalability and high reliability, have been widely used in finance, the internet, scientific research, and many other fields. The performance of distributed storage systems directly affects business operational efficiency and user experience; therefore, comprehensive and accurate performance testing of distributed storage systems is crucial.

[0003] Currently, existing performance testing methods for distributed storage systems have several shortcomings. Firstly, most methods only test a single performance metric (such as throughput or response time), failing to comprehensively reflect the system's overall performance under complex business scenarios. This results in limited test results that are difficult to meet real-world application needs. Secondly, traditional testing methods do not fully consider the complexities of real-world operation, such as dynamic changes in nodes (e.g., node failures, new node additions) and uneven data distribution. This leads to significant differences between the test environment and the real-world application environment, resulting in lower accuracy and reliability of the test results. Furthermore, some testing methods have low levels of automation, requiring substantial manual intervention. This not only increases the workload of testers but also increases the risk of human error affecting the accuracy of test results, reducing testing efficiency and hindering the rapid iteration and updates required for distributed storage systems.

[0004] Therefore, a method is needed to comprehensively, accurately, and efficiently test the performance of distributed storage systems in order to solve the aforementioned problems in existing technologies.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This disclosure provides a performance testing method and apparatus for a distributed storage system. The method effectively solves the technical problem of low efficiency in existing traditional testing by using multi-dimensional testing methods and automated testing processes.

[0008] In some embodiments, a performance testing method for a distributed storage system is provided, including: Build a test environment that simulates real-world application scenarios; Based on the aforementioned test environment, the distributed storage system is automatically tested using a test method, and performance data is collected, monitored, and stored in real time during the test. The performance data is analyzed and processed to obtain test results; Based on the test results, the performance of the distributed storage system is optimized.

[0009] In some embodiments, a performance testing apparatus for a distributed storage system is provided, comprising: The module is used to build a test environment that simulates real-world application scenarios; The testing module is used to automatically execute test methods on the distributed storage system based on the test environment, and to collect, monitor and store performance data in real time during the test. The analysis module is used to analyze and process the performance data to obtain test results; An optimization module is used to optimize the performance of the distributed storage system based on the test results.

[0010] The performance testing method and apparatus for distributed storage systems provided in this disclosure can achieve the following technical effects: By building a test environment highly consistent with real-world application scenarios, the reliability of test results is ensured; based on this environment, multi-dimensional testing is automatically executed, generating loads through FIO tools and achieving second-level monitoring, comprehensively collecting metrics such as throughput, response time, and IOPS; data is cleaned, outliers are removed, and the data is smoothed using a moving average method; after that, performance bottlenecks are accurately identified through multi-dimensional comparative analysis; finally, a suggested test report is generated. This method, through a closed-loop design of environment simulation, automated testing, and intelligent analysis, systematically solves the three major pain points of traditional testing: single scenario, large error, and low efficiency.

[0011] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0012] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of a performance testing method for a distributed storage system provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the concurrent performance testing process provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram of the dynamic scene performance testing process provided in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the data characteristic performance testing process provided in the embodiments of this disclosure; Figure 5 This is a schematic diagram of the performance data analysis and processing flow provided in the embodiments of this disclosure; Figure 6 This is a schematic diagram of the performance optimization process of the distributed storage system provided in this embodiment of the disclosure; Figure 7 This is a schematic diagram of a performance testing device for a distributed storage system provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of a performance testing device for a distributed storage system provided in an embodiment of this disclosure.

[0013] 100: Building modules; 200: Test module; 300: Analysis module; 400: Optimization module. Detailed Implementation

[0014] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0015] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate to implement embodiments of the present disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0016] Unless otherwise stated, the term "multiple" means two or more.

[0017] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0018] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0019] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0020] Combination Figure 1 As shown in the embodiments of this disclosure, a performance testing method for a distributed storage system is provided, including: S101: Set up a test environment that simulates real-world application scenarios; S102: Based on the test environment, automatically execute test methods on the distributed storage system, and collect, monitor and store performance data in real time during the test; S103: Analyze and process the performance data to obtain test results; S104: Optimize the performance of the distributed storage system based on the test results.

[0021] The test environment includes hardware, software, and network environments. The hardware environment consists of at least three identical servers selected as storage nodes, and one server selected as the test control node, based on the actual deployment scale of the distributed storage system. The storage nodes must have a CPU of at least Intel Xeon E5-2670, at least 64GB of memory, a RAID 5 array of ten 4TB SATA hard drives, and dual 10GbE network cards. The test control node must have a CPU of at least Intel Xeon E3-1230, at least 32GB of memory, a RAID 1 array of two 1TB SSDs, and a 10GbE network card.

[0022] The distributed storage system itself only includes a number of storage nodes, such as the three storage nodes mentioned above. The test control node is a test environment component and is independent of the distributed storage system.

[0023] The software environment consists of CentOS 7.9 operating system installed on all storage nodes, and Ceph distributed storage system (version 16.2.10) deployed. The number of replicas in the storage pool is configured to be 3, and the number of PGs (Placement Groups) is set according to the principle of 1 PG per 100GB of storage pool capacity. CentOS 7.9 operating system is installed on the test control node, and a test toolset is deployed, including FIO (version 3.33) for generating storage I / O load, Prometheus (version 2.45.0) and Grafana (version 10.1.0) for performance data collection and visualization, and Python 3.9 environment for writing automated test scripts.

[0024] FIO (Flexible I / O Tester) is an open-source, powerful storage performance testing tool specifically designed to evaluate and verify the I / O performance of storage devices (such as hard drives, SSDs, and storage arrays) and distributed storage systems.

[0025] The network environment is designed to connect all storage nodes and test control nodes within the same local area network via a 10GbE switch. Static IP addresses are configured for each node to ensure normal communication between them. At the same time, port aggregation is configured on the switch to bind the dual 10GbE network cards of the storage nodes into a single logical port to improve network bandwidth and reliability.

[0026] The testing methods include basic performance testing, concurrency performance testing, dynamic scenario performance testing, data characteristic performance testing, and data compression scenario performance testing.

[0027] The basic performance test further includes: Throughput test, response time test, and IOPS test, each test is executed repeatedly a set number of times.

[0028] Throughput testing, for example, involves setting different I / O block sizes (including 4KB, 8KB, 16KB, 32KB, 64KB, 128KB, 256KB, 512KB, and 1MB). For each I / O block, under four I / O modes—sequential read, sequential write, random read, and random write—the FIO tool is used to initiate I / O requests to the storage pool of the distributed storage system for 30 minutes as a test case. Each test case is executed three times, and the average value is taken as the throughput test result for that test case.

[0029] Throughput represents the amount of data successfully transmitted per unit of time, usually measured in MB / s (megabytes per second) or GB / s (gigabytes per second).

[0030] For example, response time testing uses the same I / O block size and I / O mode as throughput testing. The FIO tool records the time consumed from the initiation to completion of each I / O request. The average, maximum, minimum, and 95th percentile of the response time are calculated for each test case. Each test case is executed three times, and the average value is taken as the final result.

[0031] Response time refers to the time taken from initiation to completion of a single I / O request, usually measured in microseconds (μs) or milliseconds (ms).

[0032] The 95th percentile of response time refers to the response time at the 95th percentile after arranging all response times of the test cases in ascending order.

[0033] For example, IOPS testing uses the same I / O block size and I / O mode as throughput testing. The FIO tool is used to count the number of I / O operations completed per second. Each test case is executed for 30 minutes and repeated 3 times. The average value is taken as the IOPS test result.

[0034] IOPS (Input / Output Operations Per Second) represents the number of I / O operations that can be processed per second, measured in times per second.

[0035] Reference Figure 2 The concurrent performance test further includes: S201: Set different numbers of concurrent users to simulate multiple users simultaneously initiating I / O requests to the distributed storage system and continuously test for a set time. S202: Record the throughput, response time, and IOPS of the distributed storage system under different numbers of concurrent users, and repeat each test a set number of times.

[0036] Set different numbers of concurrent users (including 10, 20, 50, 100, 200, and 500), and simulate multiple users simultaneously initiating I / O requests to the distributed storage system using the FIO tool under an I / O block size of 64KB and random read I / O mode. The test lasts for 60 minutes, and the system throughput, response time, and IOPS are recorded under different numbers of concurrent users. Each test case is executed three times.

[0037] Reference Figure 3 The dynamic scene performance test further includes: Node failure test, S301: During the basic performance test of the distributed storage system, a storage node is shut down remotely to simulate node failure, and the test continues for a first set time. S302: Record the performance metrics of the distributed storage system before, during, and after a node failure; Node expansion test, S303: During the concurrent performance test of the system, a new storage node identical to the original node is added to the distributed storage system, and the test continues for a second set time. S304: Record the performance indicators of the distributed storage system before node expansion, during the expansion process, and after the expansion is completed, as well as the data volume of the original nodes and the newly added nodes after the expansion is completed.

[0038] For example, during the basic performance test of normal system operation (I / O block size 64KB, random read / write mixed mode, read / write ratio of 7:3), one of the storage nodes is forcibly shut down via remote command to simulate node failure. The test is conducted for 60 minutes, and the throughput, response time, IOPS, and data recovery time are recorded before the node failure, during the failure, and after the failure recovery (restarting the failed node and waiting for the system to rebalance the data).

[0039] For example, during a concurrent performance test of the system (100 concurrent users, 64KB I / O block size, random read mode), a new server with the same configuration as the original storage node is added as a new storage node and added to the distributed storage system. The test is conducted continuously for 120 minutes, and the system performance indicators are recorded before, during, and after the node expansion. At the same time, the amount of data on the original node and the new node is monitored.

[0040] The data volume includes: the number of PlacementGroups (PGs) carried by each node (including existing nodes and new nodes), and the actual amount of data stored by each node.

[0041] Reference Figure 4 The data characteristic performance test further includes: Performance tests with different data volumes: S401: Write different amounts of data to the distributed storage system respectively; S402: After the data is written, the distributed storage system corresponding to different data volumes will be tested for performance. The throughput, response time and IOPS of the distributed storage system will be recorded continuously for a set time. Data compression scenario performance test, S403: Enable the data compression function of the distributed storage system and write the set text type test data into the distributed storage system; S404: During the data writing process and during the random read test after the write is completed, record the throughput, response time and IOPS of the distributed storage system.

[0042] Performance tests for different data volumes include: writing 100GB, 500GB, 1TB, 2TB, and 5TB of test data to the distributed storage system respectively. After the data is written, the performance of the storage pool with different data volumes is tested for 30 minutes using an I / O block size of 64KB and random read mode. The system throughput, response time, and IOPS are recorded.

[0043] For example, in a data compression scenario performance test, enable the data compression function of the distributed storage system (using the Snappy compression algorithm), write 100GB of text-type test data to the storage system (compression ratio of approximately 3:1), and record the throughput, response time, IOPS, and CPU utilization of the distributed storage system during the data writing process and during random read tests after the writing is completed.

[0044] The real-time acquisition, monitoring, and storage of performance data during the testing process further includes: Real-time collection, monitoring and storage of performance data for each storage node: CPU utilization, memory utilization, disk I / O rate, network bandwidth utilization, OSD status, PG status and data read / write latency data.

[0045] Real-time data collection includes, for example, using the paramiko library in a Python 3.9 environment to remotely control each storage node and test node, using the subprocess library to call the FIO tool to execute test methods, and using the requests library to interact with the Prometheus API to obtain performance data. The script sets parameters such as the execution order, execution duration, and number of repetitions for each test method, and implements exception handling during the test process (such as automatically reconnecting when a test node disconnects, and restarting the test when the FIO test process terminates abnormally).

[0046] Automated test scripts are run on the test control node. The scripts automatically execute test tasks in the order of the designed test methods. During test execution, the Prometheus tool collects real-time performance metrics such as CPU utilization, memory utilization, disk I / O rate, network bandwidth utilization, OSD (Object Storage Daemon) status, PG status, and data read / write latency of the distributed storage system through the node_exporter and ceph_exporter plugins deployed on each storage node. The collection frequency is set to 1 time / second.

[0047] Monitor and store performance data from each storage node. For example, connect to the Prometheus data source using Grafana to create a performance monitoring dashboard that displays the performance data change curves in real time during the test. Meanwhile, Prometheus stores the collected performance data as time-series data on the local disk with a retention period of 30 days for subsequent performance data analysis.

[0048] Reference Figure 5 The analysis and processing of the performance data to obtain the test results further includes: S501: After the test is completed, the performance data under the test method is cleaned, outliers are removed, and the data is smoothed. S502: Perform statistical analysis on the processed performance data and generate a performance test report.

[0049] After the test task is completed, the Prometheus API is called through a Python script to extract performance data for each test method according to the execution time period of the test method. The extracted performance data is converted into Excel format, and the performance data is cleaned to remove outliers (such as extremely high or low performance data caused by network jitter). The moving average method is used to smooth the performance data to reduce the impact of performance data fluctuations on the analysis results.

[0050] The performance analysis is multi-dimensional, including basic performance analysis, concurrency performance analysis, dynamic scenario performance analysis, and data characteristic impact analysis.

[0051] Basic performance analysis includes, for example, comparing throughput, response time, and IOPS data under different I / O block sizes and I / O modes, creating performance data comparison charts, analyzing the impact of I / O block size and I / O mode on system performance, and determining the optimal I / O parameter configuration for the system in different application scenarios.

[0052] Generally speaking, throughput, response time, and IPOS can be used as performance metrics for distributed storage systems.

[0053] For each performance metric, taking throughput as an example, the throughput corresponding to each I / O block size is taken as a feature point. Multiple feature points corresponding to different I / O block sizes are fitted to obtain the trend of throughput changing with I / O block size, and thus the trend of the influence of I / O block size on throughput. Other performance metrics are determined in the same way as throughput, so they will not be elaborated here. The influence trend of I / O mode on performance metrics can also be referred to the influence law of I / O block size on performance metrics, so they will not be elaborated here.

[0054] Select the I / O block size and I / O mode corresponding to the conditions of high throughput, short response time and high IOPS as the optimal I / O parameter configuration.

[0055] I / O block size is positively correlated with throughput; as block size increases, throughput rises. It is negatively correlated with IOPS; as block size increases, IOPS decreases. It exhibits a non-linear relationship with response time, changing in two phases: when block size (BS) ≤ 64KB, the response time is 50μs + 0.5 × BS; when block size (BS) > 64KB, the response time is 82μs + 0.8 × (BS - 64). For example, in a video processing scenario with an I / O block size of 1MB and sequential read I / O mode, under high throughput requirements (>2GB / s), the I / O parameters can be set to 95% read and 5% write.

[0056] Concurrency performance analysis, for example: Based on performance data under different numbers of concurrent users, plot the relationship curves between the number of concurrent users and throughput, response time, and IOPS, determine the maximum concurrent processing capacity of the distributed storage system and the impact trend of the increase in the number of concurrent users on the performance of the distributed storage system, and identify the performance bottleneck of the system in high-concurrency scenarios.

[0057] For each performance metric, taking throughput as an example, the throughput corresponding to each number of concurrent users is taken as a feature point. Multiple feature points corresponding to different numbers of concurrent users are fitted to obtain the trend of throughput changing with the number of concurrent users, and thus obtain the trend of the impact of the number of concurrent users on throughput. The judgment method for other performance metrics is the same as that for throughput, so it will not be elaborated here.

[0058] The maximum concurrent user capacity of the system is determined by selecting the number of concurrent users corresponding to the conditions of high throughput, short response time, and high IOPS.

[0059] The performance bottleneck in high-concurrency scenarios is the system's ability to maintain a stable or declining performance metric when it is close to its maximum concurrent processing capacity.

[0060] Generally speaking, throughput is positively correlated with the maximum concurrent processing capacity of a distributed storage system, response time is negatively correlated with the maximum concurrent processing capacity of a distributed storage system, and IOPS is positively correlated with the maximum concurrent processing capacity of a distributed storage system.

[0061] As the number of concurrent users increases, the performance metrics of distributed storage systems exhibit distinct phased changes.

[0062] Throughput: In the low-concurrency phase (typically <50 concurrent users), throughput increases linearly with the number of concurrent users, resulting in low system resource utilization. When the number of concurrent users reaches the 50-100 range, the throughput growth rate gradually slows down, and the system enters its optimal working state. After exceeding the critical point (e.g., >150 concurrent users), the throughput tends to stabilize or even begin to decline. The throughput curve exhibits an "S"-shaped growth pattern, eventually reaching a plateau.

[0063] Response time: During low concurrency periods, the response time remains stable, basically maintaining the baseline level. When the number of concurrent requests exceeds 50% of the system's processing capacity, the response time begins to rise slowly. After reaching a critical point, the response time exhibits an exponential growth trend. The response time curve is shaped like a hockey stick, rising sharply after the critical point.

[0064] IOPS: Initially, it increases linearly with the number of concurrent connections; the growth rate slows down when approaching the system's maximum processing capacity; it may decline after overload. The IOPS curve initially increases linearly and then tends to plateau.

[0065] Dynamic scenario performance analysis includes, for example, analyzing the changes in performance data at different stages for storage node failure and storage node expansion scenarios, evaluating the stability and reliability of the system under dynamic changing scenarios; calculating the business interruption time and data recovery time after storage node failure, as well as the time required for data balancing after storage node expansion, to verify the system's fault tolerance and scalability.

[0066] For each performance metric, taking throughput as an example, determine the changes in throughput before, during, and after a storage node failure. If the overall rate of change remains within the preset rate of change, it indicates that the throughput changes smoothly. Other performance metrics are determined in the same way as throughput. If all performance metrics change smoothly, it indicates that the system is stable and reliable; otherwise, it indicates that the system is unstable and unreliable.

[0067] If the service interruption time and data recovery time after a storage node failure are both less than the preset fault tolerance time, it indicates that the system has strong fault tolerance; otherwise, it has poor fault tolerance.

[0068] If the time required for data balancing after storage node expansion is less than the preset expansion time, it indicates that the system has strong expansion capabilities; otherwise, the expansion capabilities are weak.

[0069] Data characteristic impact analysis, for example: comparing performance data with different data volumes and with and without data compression, analyzing the impact of data volume and data compression on the performance of distributed storage systems, and providing a basis for capacity planning and enabling data compression in practical applications of distributed storage systems.

[0070] For each performance metric, taking throughput as an example, we enable data compression with different data volumes, take the throughput under each data volume as a feature point, fit multiple feature points corresponding to different data volumes to obtain the trend of throughput changing with data volume, and then obtain the trend of the impact of different data volume compression on throughput. The judgment method for other performance metrics is the same as that for throughput, so it will not be elaborated here.

[0071] Alternatively, by using different data volumes without enabling data compression, the throughput for each data volume can be used as a feature point. By fitting multiple feature points corresponding to different data volumes, the trend of throughput changing with data volume can be obtained. The impact trends of different data volumes with compression and no compression on throughput can be compared. Other performance indicators are judged in the same way as throughput, so they will not be elaborated here.

[0072] For example, in capacity planning of distributed storage systems, if the number of files is large (millions) and the individual files are small (KB), prioritize expanding metadata nodes rather than storage space; if the individual files are large (GB) and the total amount is huge (PB), estimate the actual demand based on the compression ratio (e.g., 1PB of original data → 600TB after compression).

[0073] Data compression is enabled for reasons such as high redundancy data (logs, text), cold storage (Zstd / Gzip high compression ratio), and cross-data center synchronization with limited network bandwidth (such as MinIO gateway compression).

[0074] Generate a performance test report, for example: Based on the above data analysis results, generate a performance test report that includes test environment configuration, test method design, test execution process, performance data statistics and analysis, and test conclusions and recommendations.

[0075] Reference Figure 6 The optimization of the performance of the distributed storage system based on the test results further includes: S601: Optimize and adjust the distributed storage system based on the performance test report; S602: Optimize and adjust the distributed storage system based on the performance test report; S603: If the optimized performance data meets the expected target, then the optimization is complete; S604: If the optimized performance data does not meet the expected target, repeat the above steps until the performance data meets the expected target.

[0076] Before performing the above steps S601-S604, the following is also included: Re-execute the test method and compare the performance data before and after; If the performance data from two consecutive tests exceed the specified deviation, the test environment or test method shall be adjusted. After adjustment, re-execute the test method and repeat the above steps until the performance data of the two tests are within the specified deviation value.

[0077] Specifically, select at least two test methods (such as a random read method with 100 concurrent users and a node failure method), re-execute the test according to the test process, and compare the performance data of the two tests. If the deviation between the two test results is within 5%, the test results are considered to be repeatable and accurate; if the deviation exceeds 5%, investigate the problems in the test environment, test cases, or test tools, and re-perform the test verification.

[0078] Steps S601-S604 further involve: optimizing and adjusting the distributed storage system according to the performance test report (such as replacing the hard drive with a higher performance one, optimizing the configuration parameters of the distributed storage system, increasing the memory capacity of the storage nodes, etc.); after optimization, re-executing the relevant performance test methods and comparing the performance data before and after optimization; if the optimized performance indicators meet the expected goals, the performance test and optimization are completed; if the expected goals are not met, the above optimization and testing process is repeated until the performance indicators meet the requirements.

[0079] If the test report shows a problem after the above process, corresponding optimizations can be made based on that problem. For example, if the system has any of the following problems: performance bottlenecks in high-concurrency scenarios, poor system fault tolerance, poor system scalability, or performance degradation after data compression, the system should be optimized in the following ways: Performance bottlenecks in high-concurrency scenarios can be addressed by adding nodes or isolating hot nodes. Poor system fault tolerance can be mitigated by deploying nodes across multiple data centers. Poor system scalability can be addressed by read / write separation or background rate limiting. Performance degradation after data compression can be addressed by parallel compression I / O.

[0080] Optimization adjustments can include hardware-level optimization based on performance test reports, software configuration tuning, and architecture and strategy optimization. For example, hardware-level optimization could involve upgrading storage media, replacing SATA HDDs with NVMe SSDs, and reducing I / O latency (e.g., from 10ms to 0.1ms), suitable for metadata services or hot data storage. Software configuration tuning could involve adjusting distributed protocol parameters and dynamically adjusting the number of replicas to balance performance and reliability. Architecture and strategy optimization could involve tiered data storage, storing frequently accessed data on SSDs and migrating less frequently accessed data to HDDs or object storage.

[0081] The expected performance targets are as follows: Latency: The average / tail latency of read and write operations is reduced to a certain threshold (e.g., optimized from 100ms to 20ms).

[0082] Throughput: The increase in the amount of data processed per unit of time (e.g., from 500MB / s to 1.2GB / s).

[0083] Concurrency capability: The maximum number of client connections or concurrent threads supported (e.g., increasing from 1000 to 5000 concurrent connections).

[0084] Resource efficiency: Reduce CPU / memory / disk utilization (e.g., reduce disk I / O utilization from 90% to 60%).

[0085] Scalability: The linear performance improvement after adding nodes (e.g., the throughput increase after expansion is proportional to the number of nodes).

[0086] The distributed storage system performance testing method provided in this specification ensures the reliability of test results by building a test environment highly consistent with real-world application scenarios. Based on this environment, multi-dimensional testing is automatically executed. Load is generated using the FIO tool and monitored at the second level, comprehensively collecting metrics such as throughput, response time, and IOPS. Data is cleaned, outliers are removed, and the data is smoothed using a moving average method. Multi-dimensional comparative analysis is then used to accurately identify performance bottlenecks. Finally, a recommended test report is generated. This method, through a closed-loop design of environment simulation, automated testing, and intelligent analysis, systematically solves the three major pain points of traditional testing: single-scenario testing, large errors, and low efficiency.

[0087] Combination Figure 7 Based on the above-described performance testing method for distributed storage systems, this disclosure provides a performance testing apparatus for distributed storage systems, comprising: Module 100 is used to build a test environment that simulates real-world application scenarios. Test module 200 is used to automatically execute test methods on the distributed storage system based on the test environment, and to collect, monitor and store performance data in real time during the test. Analysis module 300 is used to analyze and process the performance data to obtain test results; The optimization module 400 is used to optimize the performance of the distributed storage system based on the test results.

[0088] The performance testing apparatus and method for distributed storage systems provided in this disclosure, based on embodiments of the present invention, ensure the reliability of test results by building a test environment highly consistent with real-world application scenarios. Multi-dimensional testing is automatically executed within this environment, generating loads and achieving second-level monitoring through the FIO tool to comprehensively collect metrics such as throughput, response time, and IOPS. Data is cleaned, outliers are removed, and the data is smoothed using a moving average method. Performance bottlenecks are then accurately identified through multi-dimensional comparative analysis. Finally, a recommended test report is generated. This method, through a closed-loop design of environment simulation, automated testing, and intelligent analysis, systematically solves the three major pain points of traditional testing: single-scenario testing, large errors, and low efficiency.

[0089] Given that the solutions and methods for solving the problem are similar, the specific implementation of the device in the embodiments of this specification can be referred to the foregoing method embodiments, and will not be repeated here.

[0090] Combination Figure 8 As shown, this disclosure provides a performance testing device for a distributed storage system, including a processor 804 and a memory 801. Optionally, the device may further include a communication interface 802 and a bus 803. The processor 804, communication interface 802, and memory 801 can communicate with each other via the bus 803. The communication interface 802 can be used for information transmission. The processor 804 can invoke logical instructions in the memory 801 to execute the performance testing method for the distributed storage system described in the above embodiment.

[0091] Furthermore, the logic instructions in the aforementioned memory 801 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0092] The memory 801, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 804 executes functional applications and data processing by running the program instructions / modules stored in the memory 801, thereby implementing the performance testing method of the distributed storage system in the above embodiments.

[0093] The memory 801 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 801 may include high-speed random access memory and may also include non-volatile memory.

[0094] This disclosure provides a computer program product, which is a software product that mainly implements the methods described in this specification through a computer program.

[0095] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described method.

[0096] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0097] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0098] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0099] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0100] The methods and products disclosed in the embodiments herein (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks, and the operations or steps corresponding to different blocks in the flowchart and block diagram descriptions, may occur in a different order than indicated in the drawings or descriptions. For example, two consecutive blocks or operation steps may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. Sometimes there is no specific order between different operations or steps. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A performance testing method for a distributed storage system, characterized in that, include: Build a test environment that simulates real-world application scenarios; Based on the aforementioned test environment, the distributed storage system is automatically tested using a test method, and performance data is collected, monitored, and stored in real time during the test. The performance data is analyzed and processed to obtain test results; Based on the test results, the performance of the distributed storage system is optimized.

2. The performance testing method for a distributed storage system according to claim 1, characterized in that, The testing methods include basic performance testing, concurrency performance testing, dynamic scenario performance testing, data characteristic performance testing, and data compression scenario performance testing.

3. The performance testing method for a distributed storage system according to claim 2, characterized in that, The basic performance test further includes: Throughput test, response time test, and IOPS test, each test is executed repeatedly a set number of times.

4. The performance testing method for a distributed storage system according to claim 2, characterized in that, The concurrent performance test further includes: Set different numbers of concurrent users to simulate multiple users simultaneously initiating I / O requests to the distributed storage system, and continuously test for a set time. Record the throughput, response time, and IOPS of the distributed storage system under different numbers of concurrent users, and repeat each test a set number of times.

5. The performance testing method for a distributed storage system according to claim 2, characterized in that, The distributed storage system includes several storage nodes; The dynamic scene performance test further includes: Node failure testing involves shutting down a storage node remotely during basic performance testing of a distributed storage system to simulate node failure and continuously test for a set period of time. Record the performance metrics of the distributed storage system before, during, and after node failures. Node expansion test: During the concurrent performance test of the system, a new storage node identical to the original node is added to the distributed storage system, and the test continues for a second set time. Record the performance metrics of the distributed storage system before, during, and after node expansion, as well as the data volume of the original and newly added nodes after expansion.

6. The performance testing method for a distributed storage system according to claim 2, characterized in that, The data characteristic performance test further includes: Performance tests were conducted with different data volumes, by writing different amounts of data to the distributed storage system. After the data is written, the distributed storage system with different data volumes will be tested for performance. The throughput, response time and IOPS of the distributed storage system will be recorded continuously for a set time. Data compression scenario performance test: Enable the data compression function of the distributed storage system and write test data of the specified text type into the distributed storage system; During the data writing process and during random read tests after the write is completed, the throughput, response time, and IOPS of the distributed storage system are recorded.

7. The performance testing method for a distributed storage system according to claim 1, characterized in that, The distributed storage system includes several storage nodes; The real-time acquisition, monitoring, and storage of performance data during the testing process further includes: Real-time collection, monitoring and storage of performance data for each storage node: CPU utilization, memory utilization, disk I / O rate, network bandwidth utilization, OSD status, PG status and data read / write latency data.

8. The performance testing method for a distributed storage system according to claim 1, characterized in that, The analysis and processing of the performance data to obtain the test results further includes: After the test is completed, the performance data under the test method is cleaned, outliers are removed, and the data is smoothed. The processed performance data is statistically analyzed to generate a performance test report.

9. The performance testing method for a distributed storage system according to claim 8, characterized in that, The optimization of the performance of the distributed storage system based on the test results further includes: Based on the performance test report, optimize and adjust the distributed storage system. After optimization, re-execute the test method and compare the performance data before and after optimization; If the optimized performance data meets the expected target, then this optimization is complete; If the optimized performance data does not meet the expected target, repeat the above steps until the performance data meets the expected target.

10. A performance testing device for a distributed storage system, characterized in that, include: The module is used to build a test environment that simulates real-world application scenarios; The testing module is used to automatically execute test methods on the distributed storage system based on the test environment, and to collect, monitor and store performance data in real time during the test. The analysis module is used to analyze and process the performance data to obtain test results; An optimization module is used to optimize the performance of the distributed storage system based on the test results.

Images