Storage test-oriented large-scale data efficient generation method and device

Abstract

The invention provides a large-scale data efficient generation method and device oriented to storage testing. Relates to the field of distributed storage system testing, and discloses a high-efficiency large-scale data generation method for storage testing, which can remarkably improve the large-scale data generation efficiency in a distributed storage system testing scene, realize high-speed generation of TB-PB-level data, and improve the test efficiency of the distributed storage system. Meanwhile, parallel processing efficiency and task management overhead are balanced through a dynamic fragmentation algorithm, resource occupation is reduced, and test authenticity is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of distributed storage system testing, and particularly relates to a large-scale data efficient generation method and device for storage testing. BACKGROUND

[0002] As the core infrastructure of modern data centers, distributed storage systems are widely used in cloud computing, big data processing, and enterprise storage management. In related technologies, a multi-layer task scheduling system for storage testing is constructed through the collaborative work of the control layer (MGR), coordination layer, and execution layer (OSS cluster). Specifically, this system covers the entire process from test request analysis, resource dynamic allocation, task mapping, to local data generation and batch metadata submission, including key links such as PG distribution strategy, parallelism control, and heat map data simulation. With the continuous expansion of storage scale and the increasing complexity of testing requirements, the test data generation technology has evolved from traditional single-point writing to a distributed collaborative generation mode based on cluster state awareness, to meet the requirements of high throughput, low resource occupation, and business feature simulation.

[0003] However, in existing data generation methods, centralized tools such as dd / fio are directly used for single-point writing, without dynamic sharding and resource isolation based on cluster load state and node performance, which may lead to low generation efficiency, insufficient data authenticity, or severe resource occupation, thereby affecting the testing accuracy of the storage system and the stability of the production environment. Specifically, the traditional method is limited by network bandwidth and protocol stack performance, with a single-node throughput usually below 200MB / s, and it takes several days to generate 1PB of data. At the same time, the generated data is mostly uniformly distributed, lacking simulation of hot / warm / cold data access characteristics, leading to deviations between test results and actual business scenarios. In addition, the centralized architecture often occupies more than 30% of the cluster resources when performing large-scale metadata operations, making it difficult to complete efficient test data generation without affecting production services. SUMMARY

[0004] The present disclosure aims to at least partially solve one of the technical problems in the related art.

[0005] The present disclosure proposes a large-scale data efficient generation method for storage testing, which can significantly improve the generation efficiency of large-scale data in distributed storage system testing scenarios, achieve high-speed generation of TB-PB level data, and balance parallel processing efficiency and task management overhead through a dynamic sharding algorithm, reducing resource occupation and improving test authenticity.

[0006] Another object of the present disclosure is to propose a large-scale data efficient generation device for storage testing.

[0007] To achieve the above object, the first aspect of the present application provides a large-scale data efficient generation method for storage testing, comprising: S1, receiving a test data generation request sent by a client, wherein the request contains a data scale, a data type template, a target storage pool identifier and optional parameters; S2, dynamically calculating a shard size and a shard number based on the request parameters and real-time state information of the cluster; S3, mapping the shard tasks to corresponding OSS nodes according to a PG distribution strategy, and distributing the tasks according to a parallelism control strategy, while monitoring the task execution state in real time and performing dynamic adjustment; S4, each OSS node generates test data conforming to the data type template according to the distributed tasks, and writes the data into the local storage through a memory direct writing method, and submits the metadata to the Monitor node in batches after generating a preset number of objects.

[0008] In an embodiment of the present application, S2, dynamically calculating a shard size and a shard number based on the request parameters and real-time state information of the cluster, further comprises: S21, in the calculation of the shard factor scale_factor, the result of log 10 (healthy_pg_count)+1 is used to adapt to clusters of different scales, wherein +1 is used to prevent scale_factor from being 0 when the number of healthy PGs is 1, so as to ensure that a minimum-scale cluster can still generate reasonable shards; S22, in the calculation of the dynamic scaling factor k, the result of max(throughput_ratio×10,5) is used to ensure that each node is allocated at least 5 shards to maintain the minimum parallelism, and min(available_nodes,k) is used to prevent the number of shards from exceeding the number of available nodes to avoid task distribution failure.

[0009] In an embodiment of the present application, S3, mapping the shard tasks to corresponding OSS nodes according to a PG distribution strategy, and distributing the tasks according to a parallelism control strategy, further comprises: S31, the PG distribution strategy comprises mapping each shard task to the OSS node where the primary OSD corresponding to the healthy PG in the storage pool is located, so as to ensure the locality of data writing and reduce cross-node communication overhead; S32, the parallelism control strategy comprises dynamically adjusting the task distribution rate according to the current node load, and if it is detected that the task execution speed of an OSS node is lower than a preset threshold, part of the tasks of the node are redistributed to other OSS nodes with lower load.

[0010] In one embodiment of the present application, the S4, each OSS node generates test data conforming to the data type template locally according to the assigned task, and further comprises: S41, the local data generator adopts a double-engine mode, wherein a pseudo-random stream engine is based on an improved linear congruential generator (LCG) algorithm and is optimized through a SIMD instruction set to improve data generation efficiency; and S42, the heat map simulation engine generates hot data, warm data and cold data at a specified storage location according to the data type template, wherein the access frequency of the hot data is higher than that of the warm data, and the access frequency of the warm data is higher than that of the cold data, so as to simulate real business load characteristics.

[0011] In one embodiment of the present application, further comprising: S5, after the task allocation is completed, the coordination layer continuously collects the execution state information of each OSS node, and feeds back the state information to the control layer after the state information is summarized, and the control layer dynamically adjusts the task allocation strategy according to the state information, including re-allocating tasks, adjusting parallelism or pausing tasks to avoid resource overload.

[0012] To achieve the above purpose, another aspect of the present application provides a large-scale data efficient generation device for storage testing, comprising:

[0013] A request receiving module is configured to receive a test data generation request sent by a client, wherein the request contains a data scale, a data type template, a target storage pool identifier and optional parameters;

[0014] A sharding calculation module is configured to dynamically calculate a sharding size and a sharding number based on request parameters and real-time state information of a cluster;

[0015] A task mapping and allocation module is configured to map the sharding tasks to corresponding OSS nodes according to a PG distribution strategy, and allocate tasks according to a parallelism control strategy, while monitoring a task execution state in real time and dynamically adjusting;

[0016] A data generation and writing module is configured to generate test data conforming to the data type template locally by each OSS node according to the allocated task, and write the data into a local storage through a memory direct writing mode, and batch-submit metadata to a Monitor node after a preset number of objects are generated.

[0017] The large-scale data efficient generation method and device for storage testing according to the embodiments of the present application realize efficient and stable generation of massive test data through a hierarchical architecture and fine task decomposition, while ensuring the service quality of a production environment.

[0018] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of exemplary embodiments of the present application, wherein:

[0020] Figure 1 is a flow chart of a large-scale data efficient generation method for storage test according to an embodiment of the present application;

[0021] Figure 2 is an architecture diagram of a large-scale data efficient generation method for storage test according to an embodiment of the present application;

[0022] Figure 3 is a sharding flow chart according to an embodiment of the present application;

[0023] Figure 4 is a device structure diagram of a large-scale data efficient generation method for storage test according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0025] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should belong to the protection scope of the present application.

[0026] A large-scale data efficient generation method and device for storage test according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0027] Technical terms that can be used in the present application:

[0028] Ceph: a distributed storage system using object storage architecture;

[0029] PG (Placement Group): a data sharding logical unit for managing object distribution;

[0030] OSS (Object Storage Service): an object storage service provided by a physical node;

[0031] MGR (Manager): a cluster management component responsible for monitoring and coordination;

[0032] High water level capacity expansion: capacity expansion test scenario when the storage pool capacity reaches a threshold (e.g., 85%);

[0033] Disk replacement test: simulates the data reconstruction performance verification in the hard disk failure replacement process;

[0034] Pseudo-random data: non-real data generated by an algorithm to meet the test requirements.

[0035] Figure 1 The flowchart of the large-scale data efficient generation method for storage test according to the embodiments of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0036] S1, receiving a test data generation request sent by a client, the request containing data size, data type template, target storage pool identifier and optional parameters.

[0037] Specifically, the step of "receiving a test data generation request sent by a client, the request containing data size, data type template, target storage pool identifier and optional parameters" is a key entry link in the entire distributed storage test data generation system. Its technical implementation is based on the control layer (MGR) module of the Ceph architecture, which communicates with the client through a standardized interface to realize precise configuration and efficient scheduling of test tasks.

[0038] At the technical implementation level, the client sends a structured request to the MGR module through RESTful API or a special communication protocol (such as gRPC). The request message needs to conform to the pre-defined Schema format, containing mandatory parameters such as `total_size` (unit: GB), `data_template` (such as hot / warm / cold data distribution ratio), `target_pool_id` (unique identifier of the target storage pool), and optional parameters such as `priority` (task priority), `callback_url` (task completion callback address), etc. After receiving the request, the MGR module first performs protocol analysis and parameter verification to ensure the legality and integrity of the input parameters, such as verifying whether `total_size` is within the system support range (e.g., 1 GB to 1 PB), whether `data_template` conforms to the pre-set heat map template format (JSON or YAML), and whether the target storage pool exists and is in a writable state.

[0039] ​At the parameter index level, `total_size` as the core input parameter determines the total amount of data of the entire test task, directly affecting the subsequent development of the fragmentation strategy. `data_template` defines the access feature distribution of the test data, such as 30% hot data, 50% warm data, and 20% cold data. This parameter will be used by the heat map simulation engine to generate test objects that meet the business characteristics. `target_pool_id` is used to specify the target storage pool for data writing, and the system will map the task according to the PG distribution strategy of the pool. Optional parameters such as `priority` (value range 1-5, the higher the value, the higher the priority) are used for task scheduling priority sorting, and `callback_url` is used for asynchronous notification of task status.

[0040] At the application scenario level, this step is widely used in performance testing of distributed storage systems, high-water-level capacity expansion verification, EC erasure code recovery testing, and other scenarios. For example, when simulating hard disk failure reconstruction, testers can specify the generation of cold data as the main test set to verify the reconstruction efficiency of the system in the low access frequency data scenario. In high-water-level capacity expansion testing, a high proportion of hot data can be set to simulate real business hotspots and evaluate the stability and response capability of the system under high load.

[0041] At the technical effect level, this step realizes the standardization of test task input through the structured request mechanism, laying the foundation for subsequent dynamic fragmentation, resource scheduling, and parallel execution. Its innovation lies in parameterizing and templating test requirements, combining the PG distribution characteristics of Ceph, and ensuring the reasonable distribution and efficient execution of tasks at the storage layer. At the same time, through the support of optional parameters, the system has good extensibility and flexibility, can adapt to the needs of different test scenarios, and improves the automation level and execution efficiency of test tasks.

[0042] S2, based on request parameters and real-time state information of the cluster, dynamically calculates the shard size and the number of shards.

[0043] Specifically, this step involves dynamically calculating the shard size (shard_size) and the number of shards (shard_count) in a distributed storage system based on request parameters and real-time state information of the cluster to achieve efficient and balanced generation of test data. This algorithm introduces a shard factor (scale_factor) and a dynamic scaling factor (k), combines cluster size, node performance, and total data volume, and realizes adaptive adjustment of the fragmentation strategy, thereby optimizing system throughput and resource utilization.

[0044] In terms of technical implementation, the calculation formula of the shard size is shard_size = max(min_threshold, min(max_threshold, total_size / (k x scale_factor))), wherein min_threshold and max_threshold are set to 1 GB and 10 GB respectively, for preventing the shard from being too small or too large. total_size is the total data size (unit: GB) input by the user, and k and scale_factor are dynamically calculated according to the cluster state. scale_factor is calculated by the formula scale_factor = log 10 (healthy_pg_count) + 1, wherein healthy_pg_count is the current number of healthy PGs, log 10 The function is used to suppress the excessive amplification of the number of PGs to the number of shards in a large-scale cluster, and the +1 operation ensures that reasonable shards can still be generated under the minimum cluster size. The calculation formula of k is k = min(available_nodes, max(throughput_ratio x 10, 5)), wherein available_nodes is the current number of available nodes, throughput_ratio is the ratio of the actual throughput of the node to the benchmark throughput (100 MB / s), and the maximum value of 10 is taken after multiplication, and the minimum value is taken with the available number of nodes, so as to ensure the parallelism while avoiding task overload.

[0045] In application scenarios, this step is suitable for high-water-level expansion testing, fault reconstruction testing and EC erasure code performance verification of distributed storage systems such as Ceph. Through dynamic sharding, the system can reasonably allocate tasks according to the current cluster load and node performance, and ensure that high throughput (> 10 GB / s) and low resource occupation (< 5% CPU / memory overhead) can be maintained under different scales and load conditions.

[0046] The technical effect of this step is to effectively balance the contradiction between shard management overhead and parallel efficiency, and to avoid frequent task scheduling and large metadata operation overhead due to too small shards, or uneven load and decreased parallelism due to too large shards. By real-time sensing of the cluster state and dynamically adjusting the shard strategy, the system realizes efficient and stable data distribution and processing in TB to PB level data generation tasks, and significantly improves the test efficiency and system resource utilization.

[0047] Further, S2 further comprises:

[0048] In the calculation of the shard factor scale_factor, log 10The result of (healthy_pg_count) + 1 is used to adapt to clusters of different sizes, wherein + 1 is used to prevent scale_factor being 0 when the number of healthy PGs is 1, so as to ensure that a minimum size cluster can still generate a reasonable number of shards.

[0049] Specifically, in the dynamic sharding algorithm of the application, the calculation formula of the shard factor `scale_factor` is `scale_factor = log 10 (healthy_pg_count) + 1`, which is designed to achieve reasonable sharding of data generation tasks in Ceph clusters of different sizes, so as to ensure system load balancing while avoiding resource management overhead and parallel efficiency problems caused by too small or too large shards. This step is implemented in the dynamic partition mapper of the coordination layer and is one of the key control parameters for the entire data generation task scheduling and resource allocation.

[0050] In terms of technical implementation, `healthy_pg_count` represents the number of PGs (Placement Groups) in the current storage pool that are in a healthy state and can participate in data generation tasks. PG is a basic logical unit for data distribution and replica management in Ceph, and its number directly affects the parallel processing capability of the cluster. By using the logarithmic function `log 10 ()` with base 10, the exponential growth of the number of PGs can be converted into a linear scaling factor, so as to avoid the surge in the number of shards caused by too many PGs in a large-scale cluster, thereby causing performance bottlenecks in task scheduling and metadata management. Further, the design of + 1 is to prevent `log 10 (1) = 0` when `healthy_pg_count = 1`, resulting in `scale_factor = 0`, so that the shard size calculation result is infinite, causing the task to be unable to be reasonably split. This correction term ensures that a minimum size cluster (such as a single PG configuration) can still generate a number of shards that meet the system load balancing requirements.

[0051] From the parameter index level, the value range of `scale_factor` is usually between 1 and 3, depending on the number of healthy PGs in the cluster. For example, when `healthy_pg_count = 10`, `scale_factor = log 10 (10) + 1 = 2`; when `healthy_pg_count = 1000`, `scale_factor = log 10(1000) + 1 = 4`. This factor is closely related to the subsequent shard size calculation formula `shard_size = max(min_threshold, min(max_threshold, total_size / (k x scale_factor)))`, where `min_threshold` and `max_threshold` are set to 1GB and 10GB respectively, to limit the minimum and maximum size of the shard, to balance the task management overhead and parallel efficiency.

[0052] In practical application scenarios, this step is suitable for high-water-level expansion testing, disk replacement testing, and EC erasure code performance stress testing of Ceph storage systems. In these scenarios, test tasks need to generate TB to PB level test data quickly without affecting production services. By dynamically calculating `scale_factor`, the system can adaptively adjust the shard strategy according to the current cluster health status and storage capacity, ensuring reasonable load on each OSS node and avoiding resource contention and task accumulation.

[0053] The technical effect of this step is that by introducing a logarithmic function and a correction term, the shard factor is adaptively adjusted, so that the number of shards is reasonably distributed in clusters of different sizes. This not only improves the parallel efficiency of data generation, but also effectively reduces the system overhead of task scheduling and metadata operations, enhancing the stability and scalability of the entire test process. This design embodies the innovation of the present application in resource dynamic perception and task intelligent allocation, providing key support for efficient testing of distributed storage systems.

[0054] In the calculation of the dynamic scaling factor k, the result of max(throughput_ratio x 10, 5) is used to ensure that each node is allocated at least 5 shards to maintain the minimum parallelism, while min(available_nodes, k) is used to prevent the number of shards from exceeding the number of available nodes, avoiding task allocation failure.

[0055] Specifically, in the dynamic shard algorithm of the present application, the calculation of the dynamic scaling factor k is one of the key steps to achieve efficient parallel data generation. Its calculation formula is: k = min(\text{available\_nodes}, max(\text{throughput\_ratio} x 10, 5)), which dynamically adjusts the upper limit of the number of shards by combining node availability and performance status, thereby balancing resource utilization and task execution stability.

[0056] At the technical implementation level, this step first obtains the number of available nodes \text{available\_nodes} in the current cluster, i.e., the number of OSS nodes that are currently in a healthy state and can participate in data generation tasks, through the monitoring module. Then, the system collects the actual IO throughput \text{actual\_throughput} of each node and compares it with the preset baseline throughput value \text{baseline\_throughput} = 100 MB / s to obtain the throughput performance ratio \text{throughput\_ratio}. This ratio reflects the relative processing capacity of the node and is used to guide the dynamic adjustment of the number of shards. In some implementations, \text{throughput\_ratio} can be updated periodically (e.g., every 30 seconds) to ensure the real-time and accuracy of the scaling factor.

[0057] At the parameter index level, the purpose of designing \text{throughput\_ratio} × 10 in the formula is to amplify the performance difference by 10 times, so that more shard tasks can be allocated when the node performance is high, improving the overall parallel efficiency. The max(·,5) sets the minimum value of the scaling factor to 5, ensuring that even in low-performance or small-scale clusters, basic parallelism can be maintained, and the task execution efficiency will not be reduced due to too few shards. Further, min(·, \text{available\_nodes}) is used to limit the maximum scaling factor to not exceed the number of available nodes, preventing task allocation failures or node overload.

[0058] At the application scenario level, this step is widely used in test scenarios of distributed storage systems such as Ceph, such as high-water-level capacity expansion testing, EC erasure code performance stress testing, and node fault reconstruction simulation. In these scenarios, the system needs to dynamically adjust the task allocation strategy according to the real-time load and node state to ensure the efficiency and stability of test data generation. For example, in the disk replacement test, if some nodes are isolated due to faults, \text{available\_nodes} will decrease accordingly, and k will also be automatically adjusted to avoid task allocation to unavailable nodes.

[0059] At the technical effect level, the dynamic scaling factor mechanism effectively solves the problem of resource waste or task blocking caused by fixed parallelism in traditional test data generation. By introducing dual constraints of performance ratio and node availability, the system can adaptively adjust the task allocation strategy under different scales and load conditions, thereby improving the overall throughput capacity, reducing the task failure rate, and ensuring the stability of the production environment. This step, as a core component of the dynamic partitioning algorithm, provides key inputs for the calculation of shard size and number, and is an important guarantee for efficient, secure, and scalable test data generation.

[0060] S3, mapping the shard task to the corresponding OSS node according to the PG distribution strategy, and distributing the task according to the parallelism control strategy, while monitoring the task execution state in real time and making dynamic adjustments.

[0061] Specifically, this step involves mapping shard tasks to corresponding OSS (Object Storage Service) nodes according to the PG (Placement Group) distribution strategy, and distributing tasks according to the parallelism control strategy, while monitoring the task execution state in real time and making dynamic adjustments. This process is the core link of the coordination layer and the execution layer working together in this invention, aiming to achieve efficient and balanced generation of test data in distributed storage systems, while ensuring the rational use of system resources and the stability of task execution.

[0062] At the technical implementation level, this step first relies on the PG distribution strategy, which maps data generation tasks to specific PGs according to the PG distribution of the target storage pool. Each PG corresponds to a primary OSD (Object Storage Daemon) in the Ceph architecture, and each OSD runs on a specific OSS node. Therefore, the dynamic partition mapper binds each shard task to the corresponding PG and its primary OSD OSS node by querying the PG distribution table of the storage pool, thereby achieving distributed writing of data. The shard size is calculated by the dynamic partitioning algorithm, and the formula is: shard_size = max(min_threshold, min(max_threshold, total_size / (k × scale_factor))), where min_threshold is set to 1GB, max_threshold is set to 10GB, scale_factor = log 10 (healthy_pg_count) + 1, k = min(available_nodes, max(throughput_ratio × 10, 5)), through dynamic adjustment of these parameters, the shard size is ensured to be within a reasonable range, balancing parallel efficiency and management overhead.

[0063] Further, the parallelism controller allocates the shard tasks to each OSS node according to the parallelism factor issued by the MGR. The controller uses a token bucket algorithm or a sliding window mechanism to control the task distribution rate, preventing a surge in concurrent tasks from causing system overload. At the same time, each OSS node feeds back task progress, throughput, error code, and other key indicators to the coordination layer in real time through a heartbeat mechanism or a state reporting interface during task execution. After aggregating the states of each node, the coordination layer feeds back abnormal information to the MGR, which triggers dynamic adjustment strategies, such as migrating tasks from inefficient nodes to high-performance nodes or redistributing tasks according to load balancing strategies, thereby achieving adaptive optimization of task execution.

[0064] In application scenarios, this step is widely applicable to high-water-level capacity expansion testing, disk replacement testing, and EC erasure code performance stress testing of distributed storage systems such as Ceph. Through PG-level task mapping and dynamic parallelism control, the system can generate TB to PB-level test data quickly without affecting production business, meeting the performance verification needs of large-scale storage systems under extreme loads.

[0065] The technical effect of this step lies in significantly improving data generation efficiency and system stability. Through the combination of PG distribution strategies and dynamic sharding algorithms, task allocation is more balanced, avoiding resource contention and hotspot problems. At the same time, real-time monitoring and dynamic adjustment mechanisms effectively respond to node performance fluctuations or failures, ensuring the high availability and recoverability of tasks, providing reliable support for the testing and verification of storage systems.

[0066] Further, S3 further comprises:

[0067] S31, the PG distribution strategy includes mapping each shard task to the OSS node where the main OSD corresponding to the healthy PG in the storage pool is located, to ensure the locality of data writing and reduce cross-node communication overhead.

[0068] Specifically, the PG distribution strategy is one of the key technical steps for efficient generation of large-scale test data in the present application, and its core lies in mapping each shard task to the OSS node where the main OSD corresponding to the healthy PG in the storage pool is located, thereby ensuring the locality of data writing, reducing cross-node communication overhead, and improving overall generation efficiency.

[0069] In terms of technical implementation, the strategy is based on the mapping relationship between PG and OSD of the Ceph distributed storage system. Each PG (Placement Group) is composed of multiple OSDs, one of which is responsible for coordinating data writing. The invention, through a dynamic partition mapper module, first obtains the list of all healthy PGs in the target storage pool and queries the OSS node where the primary OSD of each PG is located. Then, the total data volume is dynamically sharded according to the number of healthy PGs, and each shard task is bound to a PG and further mapped to the OSS node where its primary OSD is located. This mapping method ensures that data generation tasks are executed as locally as possible at the physical layer, avoiding network latency and bandwidth consumption caused by cross-node data transmission.

[0070] In terms of parameter indicators, the strategy relies on key parameters such as the number of healthy PGs (healthy_pg_count) and the number of available nodes (available_nodes). The shard factor (scale_factor) is dynamically calculated by the formula scale_factor = log 10 (healthy_pg_count) + 1 to adapt to clusters of different sizes and prevent excessive sharding in large-scale clusters, increasing scheduling overhead. At the same time, the size of each shard task is determined by shard_size = max(min_threshold, min(max_threshold, total_size / (k x scale_factor))), where min_threshold and max_threshold are set to 1GB and 10GB respectively to balance task management overhead and parallel efficiency. This strategy can support efficient generation of TB to PB level data in practical applications and is suitable for typical storage testing scenarios such as high-water-level expansion, disk replacement testing, and EC erasure code performance testing.

[0071] In application scenarios, the strategy is integrated into the dynamic partition mapper module of the coordination layer and works in conjunction with the parallelism controller. When the control layer (MGR) issues a task, the coordination layer intelligently allocates the task to the most suitable OSS node based on PG distribution and node state, ensuring stable execution of the generation task even in high-load or partial node failure conditions.

[0072] The technical effect of this step is to significantly reduce cross-node communication overhead and improve the localization rate of data writing, thereby improving overall generation throughput. By binding tasks to the primary OSD node of a PG, the system can reduce network IO pressure, optimize resource utilization, and enhance the reliability and scalability of task execution, providing efficient and stable data generation support for testing and verification of distributed storage systems.

[0073] S32, the parallelism control strategy includes dynamically adjusting the task distribution rate according to the current node load, and if it is detected that the task execution speed of a certain OSS node is lower than a preset threshold, part of the tasks of the node are redistributed to other OSS nodes with lower load.

[0074] Specifically, the parallelism control strategy in the application is to dynamically adjust the task distribution rate according to the load state of the current OSS node to achieve load balancing and efficient use of system resources. Specifically, the parallelism controller in the coordination layer collects the task execution speed indicators of each OSS node in real time, and compares them with the preset performance threshold. If it is detected that the task execution speed of a certain OSS node is lower than the set reference value (such as 100 MB / s), the task redistribution mechanism is triggered to migrate part of the tasks of the node to a node with lower load, thereby avoiding the decline of the overall performance caused by the overload of the local node.

[0075] In terms of technical implementation, the parallelism controller uses a sliding average algorithm based on a time window (such as a 10-second window) to evaluate the real-time throughput of each node and calculate the ratio of the throughput to the baseline throughput (throughput_ratio). When the throughput_ratio is lower than the preset threshold (such as 0.7), it is determined that the node is in an inefficient state. At this time, the controller marks the tasks with a completed proportion lower than the set threshold (such as 30%) in the current processed shard tasks of the node as migratable tasks, and redistributes these tasks to OSS nodes with lighter load through a task queue scheduling algorithm (such as priority queue or FIFO). During the task migration process, a lightweight metadata synchronization mechanism is used to ensure task state consistency and avoid data duplication or loss.

[0076] In terms of parameter setting, the preset baseline throughput (baseline_throughput) can be dynamically adjusted according to the hardware configuration of the cluster, and is usually set to 100 MB / s to adapt to storage nodes of different sizes. The trigger threshold for task migration (throughput_ratio<0.7) and the migration proportion (the proportion of completed tasks<30%) can be configured according to actual test scenarios to balance the system response speed and task migration overhead. In addition, to prevent system jitter caused by frequent migration, a migration cooling time (such as 30 seconds) and a migration frequency upper limit (such as no more than 5 times per minute) can be set.

[0077] This step is suitable for distributed storage systems such as Ceph in high-water level expansion, fault reconstruction, EC erasure code verification, and other test scenarios. In these scenarios, the system needs to generate TB to PB level test data quickly without affecting the production service. By dynamically adjusting the parallelism, the task execution efficiency can be effectively improved, the task completion time can be reduced, and the reasonable allocation of cluster resources and system stability can be ensured.

[0078] From the technical effect, the parallelism control strategy significantly improves the execution efficiency and fault tolerance of distributed test tasks. In the process of generating large-scale data, it avoids resource waste and task blocking caused by node performance fluctuations, realizes adaptive load adjustment, and thus improves the overall throughput and reduces the task failure rate. The introduction of this strategy provides a more stable and efficient task scheduling mechanism for storage system testing, which has important engineering practical value.

[0079] S4, each OSS node generates test data according to the assigned task, and writes the data into the local storage through the memory direct writing method. After generating a preset number of objects, the metadata is batch submitted to the Monitor node.

[0080] Specifically, in the execution layer, each OSS node generates test data according to the task assigned by the coordination layer, and writes the data into the local storage through the memory direct writing method. Then, after generating a preset number of objects, the metadata is batch submitted to the Monitor node. This step is the core link for efficient and low-resource test data generation of distributed storage systems, and its technical implementation involves data generation engine, storage writing mechanism, and metadata management strategy.

[0081] In some implementations, after the OSS node receives the shard task issued by the coordination layer, it first calls the local data generator, which adopts a dual-engine architecture: a pseudo-random stream engine and a heat map simulation engine. The pseudo-random stream engine is based on an improved linear congruential generator (LCG) algorithm combined with SIMD (Single Instruction Multiple Data) instruction set optimization to generate high-speed pseudo-random data stream with a data generation rate of over 1 GB / s. The heat map simulation engine generates objects with different access frequencies at specified storage locations according to the data type template provided by the user (such as hot / cold data distribution ratio), to simulate real business load characteristics and improve the authenticity of the test scenario.

[0082] Further, the generated data is directly written into the local storage device through a memory mapping (Memory Mapped I / O) mode, skipping the file system cache layer, significantly reducing memory occupation and improving writing performance. This mode usually uses the POSIX `mmap()` interface or the Linux `O_DIRECT` flag to ensure that data is directly written from the user space to the disk, reducing context switching and cache pollution, and is suitable for high-performance storage media such as SSD or NVMe, which can achieve a writing throughput close to the hardware limit.

[0083] Optionally, after generating a preset number of objects (such as 1000 objects), the OSS node sends the metadata (including object name, size, storage location, verification information, etc.) of the batch of objects to the Monitor node through a batch submission mechanism. This mechanism uses a batch aggregation strategy to avoid network and system overhead caused by frequent metadata interaction, improving overall task execution efficiency. Metadata submission is usually completed through gRPC or a custom lightweight protocol to ensure low latency and high reliability.

[0084] This step plays a connecting role in the system, on the one hand ensuring efficient generation and local storage of test data, and on the other hand reducing the load pressure of the Monitor node through a batch submission mechanism to improve the overall stability of the system. Its technical value lies in the realization of TB-PB level test data localization and parallel generation, meeting the test requirements of high throughput (>10GB / s), low resource occupation (CPU occupation <15%, memory occupation <5GB) and reproducibility, while ensuring the normal operation of the production environment.

[0085] Further, S4 comprises:

[0086] S41, the local data generator adopts a dual-engine mode, in which the pseudo-random streaming engine is based on an improved linear congruential generator (LCG) algorithm and is optimized through a SIMD instruction set to improve data generation efficiency.

[0087] Specifically, in the execution layer of the present application, the local data generator adopts a dual-engine mode, in which the pseudo-random streaming engine is based on an improved linear congruential generator (LCG) algorithm and is optimized through a SIMD (Single Instruction, Multiple Data) instruction set to significantly improve data generation efficiency. This step is the core execution unit in the entire test data generation process, and its technical implementation directly determines whether the system can complete the rapid generation of TB to PB level data under the premise of high throughput and low resource occupation.

[0088] In some implementations, the pseudo-random stream engine generates a pseudo-random data stream through an improved LCG algorithm. A traditional LCG algorithm is based on a recursive formula X n+1 aX n where a, c, m are constant parameters. The present application optimizes the parameters and extends the state of the LCG, uses a 64-bit integer as a state variable, selects a modulus m (such as 2^61-1) that satisfies the Mersenne Prime characteristic, and introduces a nonlinear disturbance term to enhance randomness, thereby generating pseudo-random data that meets the storage test requirements. The algorithm has good periodicity (the period can reach 10^18 level), which can ensure that the data is not repeated during a long generation process, and meets the reproducibility requirements of test data.

[0089] Further, to improve the throughput of data generation, the present application introduces SIMD instruction set optimization in the implementation of the LCG algorithm. In the x86 architecture, the AVX2 instruction set is used to vectorize the recursive process of LCG, and multiple random numbers (such as 8 64-bit random numbers) can be generated in parallel in each clock cycle, thereby increasing the data generation speed to 3-5 times that of traditional scalar implementation. In the ARM architecture, similar optimization can be performed using the NEON instruction set. Through memory prefetch and data alignment (16 / 32 byte alignment) strategies, memory access delay is further reduced, and data stream continuity and stability are improved.

[0090] This step is mainly deployed in the local execution layer of the OSS node in actual application, and is used to quickly generate test data conforming to the heat map distribution after the test task is fragmented. In high-water-level capacity expansion testing, pseudo-random data can simulate the distribution characteristics of real business data; in disk replacement testing, its non-repetitive nature can avoid data conflicts and improve reconstruction efficiency. By generating locally rather than remotely, network bandwidth bottlenecks are effectively avoided, and combined with the memory direct write mechanism, the file system cache is skipped, significantly reducing memory usage and CPU overhead.

[0091] This technical implementation has significant advantages in performance indicators. The single-node pseudo-random data generation throughput can reach 1.2 GB / s or more, and the CPU occupancy rate is controlled at 15% or less, and the peak memory occupancy is not more than 512 MB. Through algorithm improvement and SIMD optimization, the present application realizes the efficiency and resource-friendliness of test data generation under the premise of ensuring data quality, providing reliable support for performance verification of distributed storage systems.

[0092] S42, the heat map simulation engine generates hot data, warm data and cold data in the specified storage location according to the data type template, wherein the access frequency of the hot data is higher than that of the warm data, and the access frequency of the warm data is higher than that of the cold data, to simulate real business load characteristics.

[0093] Specifically, the heat map simulation engine generates hot data, warm data and cold data according to the data type template in the specified storage location, which is one of the core functional modules of the local data generator in the execution layer of the application, and its technical implementation is based on the deep modeling of business load characteristics and the simulation of data access patterns. In some implementations, the engine controls the generation ratio and distribution strategy of different types of data through a preset data type template (such as 30% hot data, 50% warm data, and 20% cold data), thereby constructing a data access heat map that conforms to the real business scenario.

[0094] From the technical implementation level, the heat map simulation engine first parses the data type template issued by the control layer (MGR) to determine the access frequency distribution model of hot, warm and cold data. Hot data is usually simulated as high-frequency small objects (such as 1KB-100KB), warm data as medium-frequency medium-sized objects (such as 100KB-10MB), and cold data as low-frequency large objects (such as 10MB-1GB). During data generation, the engine writes different categories of data into the PG (Placement Group) of the specified storage pool through object naming strategies, access timestamp injection, data lifecycle marking, etc. For example, hot data objects can be assigned a shorter TTL (Time To Live) or frequent timestamp update to simulate high-frequency access behavior; cold data uses long TTL or infrequent update strategy to simulate low-frequency access characteristics.

[0095] At the parameter index level, the heat map simulation engine supports user-defined access frequency distribution models, such as setting the hot data access frequency to ≥100 times / sec, the warm data to 10-100 times / sec, and the cold data to ≤10 times / sec. At the same time, the engine supports object size range configuration to ensure that the generated data has reasonable storage density and access pressure in the storage system. In addition, the generation ratio of hot data and warm data needs to meet the load demand of the test scenario, for example, in QoS policy verification, a high proportion of hot data can effectively test the response capability of the system under high concurrency access.

[0096] In application scenarios, this step is widely used in performance stress testing, fault recovery testing and capacity planning verification of distributed storage systems. For example, in high-water-level capacity expansion testing, frequent access of hot data can simulate the storage pressure during peak business hours; in disk replacement testing, the low-frequency access characteristics of cold data help to evaluate the resource consumption and recovery efficiency during data reconstruction.

[0097] The technical effect of this step is that, by simulating the characteristics of real business load, the representativeness of test data and the reliability of test results are significantly improved. At the same time, the engine works with the pseudo-random stream engine to ensure high throughput (>10 GB / s) and low resource occupation (CPU usage <10%) of the data generation process, so as to realize efficient generation of large-scale test data without affecting the production environment.

[0098] The large-scale data efficient generation method for storage test of the embodiment of the application significantly improves the test data generation efficiency of the distributed storage system, reduces resource consumption, supports real business load simulation, and ensures parallel operation of the test process and the production environment.

[0099] In an embodiment of the application, as shown in Figure 2 The application relates to the technical field of distributed storage testing, and in particular to a TB-PB level test data high-speed generation method for distributed storage systems such as Ceph in high-water-level expansion, fault reconstruction, EC erasure code verification and other scenarios. The application is composed of three layers of architecture: a control layer (MGR), a coordination layer, and an execution layer (OSS cluster). A client initiates a test request, and the whole system cooperates to complete the data generation task. Distributed collaborative computing and resource isolation technology are mainly used to solve the industry pain points of low test data generation efficiency, insufficient authenticity, and serious resource occupation. The overall scheme is as follows:

[0100] 1. Client test request: a user (tester) initiates a test data request through a client, and the request contains but is not limited to the following key parameters:

[0101] Data size (for example: 100TB), data type template (for example: mixed heat map, in which hot data accounts for 30%, warm data accounts for 50%, and cold data accounts for 20%), target storage pool (for example: storage pool ID "ec_pool_1"), and other optional parameters: priority, callback address, etc.

[0102] 2. The control layer (MGR) module is mainly responsible for the following tasks:

[0103] 1) The MGR receives the client request, performs request analysis and verification.

[0104] 2) The task scheduling engine formulates a data generation strategy according to the global resource status (current cluster load, health status of each node) and request parameters:

[0105] ① Determine the data distribution strategy: according to the PG distribution of the storage pool, map the data generation task to a specific PG.

[0106] ② Determine the resource allocation strategy: allocate resources (CPU, memory, network bandwidth) for the test task to ensure that the production business is not affected.

[0107] ③Generate task execution plan: including shard size, parallelism, task execution order, etc.

[0108] 3) MGR issues policies to the coordination layer through RPC instructions.

[0109] 3. The coordination layer mainly includes the following two modules:

[0110] 1) Dynamic partition mapper:

[0111] ① According to the policy issued by MGR, the total data volume is dynamically sharded according to PG distribution. Each shard corresponds to a PG, and the shard size calculation formula is: shard size = total data volume / (healthy PG number x shard coefficient), where the shard coefficient is dynamically adjusted according to the cluster size (such as log10(node number)).

[0112] ② Establish the mapping of PG to OSS node (because the primary OSD of each PG is located in a certain OSS node).

[0113] 2) Parallelism controller:

[0114] ① According to the parallelism indicated by MGR, shard tasks are assigned to each OSS node. At the same time, control the task distribution rate to avoid too many tasks being started at the same time, causing system overload.

[0115] ② Real-time monitoring of task execution status, if a node task fails, automatically reassign.

[0116] 4. Execution layer (OSS cluster):

[0117] ① Each OSS node receives the task assigned by the coordination layer (usually a node handles multiple shard tasks at the same time).

[0118] ② Local data generator:

[0119] Generate data locally according to task requirements. Use dual-engine mode:

[0120] Pseudo-random streaming engine: use improved LCG algorithm (SIMD optimization) to generate pseudo-random data.

[0121] Thermal map simulation engine: generate hot data (frequent access), warm data (moderate access), and cold data (rare access) at specified locations according to templates.

[0122] Use memory direct write engine: write generated data directly to local storage (skip file system cache to avoid high memory usage).

[0123] ③ Metadata batch submission:

[0124] Metadata of each batch of data (e.g. 1000 objects) is submitted to the Monitor node in bulk, avoiding frequent interactions. After submission, the data is marked as accessible.

[0125] 5. Process monitoring and feedback:

[0126] Each OSS node reports progress and status (e.g. generation speed, error information) to the coordination layer in real time during task execution. The coordination layer aggregates node information and feeds back to the control layer (MGR), which performs global monitoring and dynamic adjustment (e.g. if a node slows down, it can be reassigned to other nodes). When all tasks are completed, the coordination layer notifies the MGR, which then notifies the client (through the callback address) that the task is completed.

[0127] Further, as shown in Figure 3 the present application needs to consider two factors when designing dynamic sharding: 1) Lower limit of shard size: to avoid excessive management overhead (such as metadata operations, task scheduling overhead) due to small shards; 2) Upper limit of shard size: to avoid load imbalance and parallelism decline due to large shards. To ensure that each shard size is moderate, both fully utilizing parallelism and not causing excessive task management overhead due to small shards, a dynamic sharding algorithm is proposed, which dynamically calculates shard size based on cluster size, total data size and node performance.

[0128] Shard size calculation formula: shard_size = max(min_threshold, min(max_threshold, total_size / (k x scale_factor))), the dynamic nature of this formula is that k and factor are calculated according to real-time cluster state.

[0129] Shard count calculation formula: shard_count = ceil(total_size / shard_size)

[0130] Parameter definition: shard_size: shard size, dynamically calculated; min_threshold: minimum shard threshold, configured to 1 GB, to prevent small shards from causing excessive management overhead; max_threshold: maximum shard threshold, configured to 10 GB, to prevent large shards from reducing parallel efficiency; total_size: total data size, i.e., the total data size (unit: GB) to be processed, which needs to be input by the user; k: dynamic scaling factor, i.e., to control the range of the number of shards, dynamically adjusted based on the cluster state, dynamically calculated; scale_factor: shard factor, to adapt to the change of cluster size, dynamically calculated; shard_count: total number of shards, i.e., the number of shards finally generated, dynamically calculated.

[0131] wherein the values of the parameters k and scale_factor are calculated by the following formulas, respectively:

[0132] 1) Sharding factor calculation formula: scale_factor = log 10 (healthy_pg_count) + 1;

[0133] (1) healthy_pg_count: number of healthy PGs: the number of PGs (Placement Group) in a healthy state (can participate in tasks) in the cluster, representing the storage size of the cluster. (2) log 10 (): the logarithm function with base 10, to prevent excessive shards from large-scale clusters; (3) +1: to prevent the factor from being 0 when the PG is 1, to ensure that a minimum-size cluster still has reasonable shards;

[0134] 2) Dynamic scaling factor formula: k = min(available_nodes, max(throughput_ratio x 10, 5));

[0135] (1) available_nodes: available nodes, the number of nodes currently available to participate in processing tasks; (2) throughput_ratio: throughput performance ratio, actual_throughput / baseline_throughput, that is, the actual throughput ratio of the node, which can truly reflect the processing capacity of the node, and baseline_throughput represents the baseline throughput value, which is preset to 100MB / s here; (3) x10: performance multiplier, capacity expansion throughput ratio impact, enhance the impact of high-performance nodes on shard division; (4) max(..., 5): lower limit constraint, ensure the minimum scaling factor, prevent too few shards from limiting parallelism; (5) min(): upper limit constraint, limit the maximum scaling factor, avoid the number of shards exceeding the number of nodes;

[0136] The algorithm can effectively solve the contradiction between "parallel efficiency of large-scale data processing" and "shard management overhead" in an actual distributed system, and provides an optimal shard strategy for various data operations.

[0137] The application realizes efficient, accurate and safe generation of super-large-scale test data in a distributed storage environment through three levels of division, cooperation and innovation algorithm, and provides strong support for quality verification of storage products. It also solves the problems of low efficiency, insufficient authenticity and serious resource occupation in traditional industry test data generation, greatly improves the test efficiency in the version iteration period, and guarantees the service quality of the production environment. And through the hierarchical architecture and fine task decomposition, the efficient and stable generation of massive test data is realized, and the service quality of the production environment is guaranteed. It also ensures that each shard size is moderate, which can fully utilize parallelism and will not cause excessive task management overhead due to too small shards.

[0138] To realize the above-mentioned embodiments, as shown in Figure 4 The large-scale data efficient generation device 10 for storage test is provided in the embodiment, which comprises:

[0139] The request receiving module 100 is configured to receive a test data generation request sent by a client, wherein the request contains a data scale, a data type template, a target storage pool identifier and optional parameters;

[0140] The shard calculation module 200 is configured to dynamically calculate a shard size and a shard number based on request parameters and real-time state information of a cluster;

[0141] The task mapping and distribution module 300 is configured to map the shard tasks to corresponding OSS nodes according to a PG distribution strategy, distribute the tasks according to a parallelism control strategy, and simultaneously monitor a task execution state and perform dynamic adjustment.

[0142] The data generation and writing module 400 is configured to generate test data conforming to the data type template according to the assigned task at each OSS node, and write the data into the local storage through a memory direct writing manner, and batch submit the metadata to the Monitor node after a preset number of objects are generated.

[0143] Further, the shard size is determined by the formula shard_size = max(min_threshold, min(max_threshold, total_size / (k×scale_factor))), and the shard factor scale_factor is calculated by the formula scale_factor = log 10 (healthy_pg_count)+1, and the dynamic scaling factor k is calculated by the formula k = min(available_nodes, max(throughput_ratio×10, 5)); the shard calculation module is further configured to:

[0144] In the calculation of the shard factor scale_factor, the result of log 10 The result of (healthy_pg_count)+1 is used to adapt to clusters of different sizes, wherein +1 is used to prevent scale_factor from being 0 when the number of healthy PGs is 1, so as to ensure that a minimum size cluster can still generate reasonable shards;

[0145] In the calculation of the dynamic scaling factor k, the result of max(throughput_ratio×10, 5) is used to ensure that each node is allocated at least 5 shards to maintain the minimum parallelism, and min(available_nodes, k) is used to prevent the number of shards from exceeding the number of available nodes to avoid task allocation failure.

[0146] Further, the task mapping and allocation module is further configured to:

[0147] Map each shard task to the OSS node where the main OSD corresponding to the healthy PG in the storage pool is located, so as to ensure the locality of data writing and reduce cross-node communication overhead;

[0148] According to the current node load, the task distribution rate is dynamically adjusted, and if it is detected that the task execution speed of an OSS node is lower than a preset threshold, part of the tasks of the node are re-allocated to other OSS nodes with lower load.

[0149] Further, the data generation and writing module is further configured to:

[0150] Adopt a dual-engine mode, in which a pseudo-random stream engine is based on an improved linear congruential generator (LCG) algorithm and is optimized through a SIMD instruction set to improve data generation efficiency.

[0151] According to the data type template, hot data, warm data and cold data are generated in a designated storage location, wherein the access frequency of the hot data is higher than that of the warm data, and the access frequency of the warm data is higher than that of the cold data, so as to simulate the real business load characteristics.

[0152] Further, the method further comprises:

[0153] A state feedback module is configured to continuously collect execution state information of each OSS node after task allocation is completed, and feed back the state information to the control layer after the state information is summarized.

[0154] A strategy adjustment module is configured to dynamically adjust the task allocation strategy according to the state information, including re-allocating tasks, adjusting parallelism or pausing tasks to avoid resource overload.

[0155] The large-scale data efficient generation device for storage test-oriented according to the embodiment of the application significantly improves the distributed storage system test data generation efficiency, reduces resource consumption, supports real business load simulation, and guarantees parallel operation of the test process and the production environment.

[0156] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0157] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one feature. In the description of the application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

Claims

1. A method for efficiently generating large-scale data for storage testing, characterized in that, include: S1, Receive test data generation request sent by client, the request includes data size, data type template, target storage pool identifier and optional parameters; S2 dynamically calculates the shard size and number of shards based on request parameters and real-time cluster status information; S3 maps the sharded tasks to the corresponding OSS nodes according to the PG distribution strategy, allocates tasks according to the parallelism control strategy, and monitors the task execution status in real time and makes dynamic adjustments. S4, each OSS node generates test data that conforms to the data type template locally according to the assigned task, and writes the data to local storage through memory direct write method. After generating a preset number of objects, the metadata is submitted to the Monitor node in batches.

2. The method as described in claim 1, characterized in that, The shard size is determined by the preset formula shard_size = max(min_threshold, min(max_threshold, total_size / (k×scale_factor))). The shard factor scale_factor is determined by the formula scale_factor = log 10 The dynamic scaling factor k is calculated as (healthy_pg_count) + 1, and is calculated using the formula k = min(available_nodes, max(throughput_ratio × 10, 5)). S2, which dynamically calculates the shard size and number of shards based on request parameters and real-time cluster status information, also includes: S21, in the calculation of the fragmentation factor scale_factor, log 10 The result of (healthy_pg_count)+1 is used to adapt to clusters of different sizes. The +1 is used to prevent the scale_factor from being 0 when the number of healthy PGs is 1, thus ensuring that even the smallest cluster can still generate reasonable shards. S22, in the calculation of the dynamic scaling factor k, the result of max(throughput_ratio×10,5) is used to ensure that each node is allocated at least 5 shards to maintain the minimum parallelism, while min(available_nodes,k) is used to prevent the number of shards from exceeding the number of available nodes and avoid task allocation failure.

3. The method as described in claim 1, characterized in that, S3, which maps the sharded tasks to the corresponding OSS nodes according to the PG distribution strategy and allocates tasks according to the parallelism control strategy, further includes: S31, the PG distribution strategy includes mapping each shard task to the OSS node where the primary OSD corresponding to the healthy PG in the storage pool is located, so as to ensure the locality of data writing and reduce cross-node communication overhead; S32, the parallelism control strategy includes dynamically adjusting the task distribution rate according to the current node load. If the task execution speed of an OSS node is detected to be lower than a preset threshold, some tasks of that node will be redistributed to other OSS nodes with lower load.

4. The method as described in claim 1, characterized in that, In step S4, each OSS node generates test data conforming to the data type template locally according to the assigned task, and also includes: S41, the local data generator adopts a dual-engine mode, wherein the pseudo-random streaming engine is based on the improved linear congruent generator LCG algorithm and is optimized by the SIMD instruction set to improve data generation efficiency. S42, the heatmap simulation engine generates hot data, warm data and cold data in a specified storage location according to the data type template, wherein the access frequency of hot data is higher than that of warm data, and the access frequency of warm data is higher than that of cold data, in order to simulate the characteristics of real business load.

5. The method as described in claim 1, characterized in that, Also includes: S5, after the task allocation is completed, the coordination layer continuously collects the execution status information of each OSS node, and feeds back the status information to the control layer. The control layer dynamically adjusts the task allocation strategy according to the status information, including reallocating tasks, adjusting the parallelism, or pausing tasks to avoid resource overload.

6. A large-scale data high-efficiency generation device for storage testing, characterized in that, include: The request receiving module is used to receive test data generation requests sent by the client. The request includes data size, data type template, target storage pool identifier, and optional parameters. The sharding calculation module is used to dynamically calculate the shard size and number of shards based on request parameters and real-time cluster status information. The task mapping and allocation module is used to map the sharded tasks to the corresponding OSS nodes according to the PG distribution strategy, allocate tasks according to the parallelism control strategy, and monitor the task execution status in real time and make dynamic adjustments. The data generation and writing module is used by each OSS node to generate test data that conforms to the data type template locally according to the assigned task, and write the data to local storage through memory direct write method. After generating a preset number of objects, the metadata is submitted to the Monitor node in batches.

7. The apparatus as claimed in claim 6, characterized in that, The shard size is determined by the formula shard_size = max(min_threshold, min(max_threshold, total_size / (k×scale_factor))). The shard factor scale_factor is determined by the formula scale_factor = log 10 The dynamic scaling factor k is calculated as (healthy_pg_count) + 1, and the sharding calculation module is also used for: In the calculation of the fragmentation factor scale_factor, log 10 The result of (healthy_pg_count)+1 is used to adapt to clusters of different sizes. The +1 is used to prevent the scale_factor from being 0 when the number of healthy PGs is 1, thus ensuring that even the smallest cluster can still generate reasonable shards. In the calculation of the dynamic scaling factor k, the result of max(throughput_ratio×10,5) is used to ensure that each node is allocated at least 5 shards to maintain the minimum parallelism, while min(available_nodes,k) is used to prevent the number of shards from exceeding the number of available nodes and avoid task allocation failure.

8. The apparatus as claimed in claim 6, characterized in that, The task mapping and allocation module is also used for: Each shard task is mapped to the OSS node where the primary OSD corresponding to the healthy PG in the storage pool resides, to ensure the locality of data writing and reduce cross-node communication overhead; The task distribution rate is dynamically adjusted based on the current node load. If the task execution speed of an OSS node is detected to be lower than the preset threshold, some tasks of that node will be redistributed to other OSS nodes with lower load.

9. The apparatus as claimed in claim 6, characterized in that, The data generation and writing module is also used for: A dual-engine mode is adopted, in which the pseudo-random streaming engine is based on the improved linear congruent generator (LCG) algorithm and optimized by the SIMD instruction set to improve data generation efficiency. Based on the data type template, hot data, warm data, and cold data are generated in the specified storage location. Hot data is accessed more frequently than warm data, and warm data is accessed more frequently than cold data, in order to simulate the characteristics of real business load.

10. The apparatus as claimed in claim 6, characterized in that, Also includes: The status feedback module is used to continuously collect the execution status information of each OSS node after the task allocation is completed, and to summarize the status information and feed it back to the control layer. The strategy adjustment module is used to dynamically adjust the task allocation strategy based on the status information, including reallocating tasks, adjusting the parallelism, or pausing tasks to avoid resource overload.

Images