Pressure test method for memory engine, server, product and storage medium

Abstract

The present disclosure provides a pressure test method for a memory engine, a server, a product and a storage medium. In the pressure test method for the memory engine, after pressure test parameters of a stand-alone memory engine under test are obtained, test data can be generated on the basis of data generation parameters and a data format corresponding to a workload of said stand-alone memory engine; then, the test data matches the data format corresponding to the workload of said stand-alone memory engine, better simulating the storage scenario of the workload; and a read-write interface provided by said stand-alone memory engine performs a data processing operation on at least some of data blocks among a plurality of data blocks, such that a pressure device can directly act on said stand-alone memory engine, and the requirements for performing pressure test on said stand-alone memory engine are met, wherein the type of the data processing operation matches the operation type of the workload of said stand-alone memory engine.

Description

[0001]Storage Engine Stress Testing Method, Server, Product, and Storage Medium Cross-Reference This disclosure claims priority to Chinese patent application No. 202410244837.4, filed with the Patent Office of China on March 4, 2024, entitled "Storage Engine Stress Testing Method, Server, Product, and Storage Medium," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of computer technology, and more particularly to a storage engine stress testing method, server, product, and storage medium. Background: Storage services such as elastic block storage services and object storage services can utilize distributed storage systems as their storage base. A distributed storage system may include multiple stand-alone storage engines, each responsible for storing data on a single server. In some scenarios, it is necessary to stress test a stand-alone storage engine to evaluate its performance under a given workload. However, there is currently a lack of a stressor compatible with stand-alone storage engines, making it difficult to accurately stress test stand-alone storage engines. Therefore, a new solution is needed. SUMMARY OF THE INVENTION Various aspects of the present disclosure provide a storage engine stress testing method, server, product, and storage medium, for providing a stressor that matches the interface of a stand-alone storage engine and the operation type of its workload, thereby more accurately stress testing the stand-alone storage engine. Embodiments of the present disclosure provide a storage engine stress testing method, comprising: obtaining stress testing parameters for a stand-alone storage engine to be tested; the stress testing parameters including at least: data generation parameters; the stand-alone storage engine is configured to manage multiple data blocks on a hard disk; generating test data based on the data generation parameters and a data format corresponding to the workload of the stand-alone storage engine; performing a data processing operation on at least some of the multiple data blocks through a read / write interface provided by the stand-alone storage engine; the type of the data processing operation matching the operation type of the workload of the stand-alone storage engine; and calculating a write amplification factor for the stand-alone storage engine based on a first amount of data actually written to the hard disk by the data processing operation and a second amount of data corresponding to the test data. Optionally, data processing operations are performed on at least some of the multiple data blocks through the read and write interface provided by the stand-alone storage engine, including: writing the test data to at least some of the multiple data blocks; performing an update operation on part of the test data written to the data blocks; and performing a data compression operation on the data blocks on which data has been written on the hard disk when the occupancy of the hard disk meets a set threshold condition.Optionally, the workload of the stand-alone storage engine is an LSM tree storage engine; and the data format of the test data is a key-value pair format. Optionally, the test data is a key-value pair. Optionally, generating the test data based on the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine includes: determining a target key length and / or a target value length based on the data generation parameters and the data format; and generating multiple key-value pairs based on the target key length and / or the target value length using a set data generation algorithm. Optionally, the stress test parameters also include: a target data volume threshold for the test data that has been written; the target data volume threshold is used to describe the upper limit of the data volume generated by the upper-layer application of the stand-alone storage engine during stable operation; performing an update operation on part of the test data written into the data block, including: if the data volume of the test data that has been written is less than the target data volume threshold, then during the process of writing the test data, performing an update operation on part of the test data that has been written into the data block according to the update frequency in the stress test parameters; if the data volume of the test data that has been written is greater than or equal to the target data volume threshold, stopping the test data writing operation and performing an update operation on part of the test data that has been written into the data block until the occupancy of the hard disk meets the set threshold condition. Optionally, performing an update operation on a portion of the test data already written into the data block based on the update frequency in the stress test parameter includes: during any data processing operation, determining whether the data processing operation is an update type based on the update frequency; if so, performing an update operation on the portion of the test data already written into the data block, wherein the update operation includes: selecting a target key from the already written test data, generating a new value corresponding to the target key, appending the target key and its corresponding new value to the data block in a write state, and marking the target key and its old value as invalid data. Optionally, the method further includes: if the determination is not correct, generating a key-value pair; and in response to the already written data block not exceeding the target data volume threshold, writing the key-value pair into the already written data block.Optionally, when the occupancy of the hard disk meets a set threshold condition, a data compression operation is performed on the data blocks on the hard disk where data has been written, including: when the occupancy of the hard disk is greater than or equal to a first hard disk occupancy threshold, determining a first data block from the data blocks on the hard disk where test data has been written; the number of invalid data contained in the first data block meets a set condition; migrating valid data other than the invalid data in the first data block to a second data block in a write state in the stand-alone storage engine, until the occupancy of the hard disk is less than or equal to the second hard disk occupancy threshold, stopping the migration operation on the valid data in the first data block; and performing a garbage collection operation on the remaining invalid data in the first data block. Optionally, calculating the write amplification factor of the stand-alone storage engine based on a first amount of data actually written to the hard disk by the data processing operation and a second amount of test data includes: determining the first amount of data actually written to the hard disk based on the second amount of data corresponding to the test data and the amount of data migrated during the data compression operation; and calculating the write amplification factor of the stand-alone storage engine under the stress test parameters based on a ratio of the first amount of data to the second amount of data. This embodiment of the present disclosure also provides a server comprising: a memory and a processor; the memory being configured to store one or more computer instructions; and the processor being configured to execute the one or more computer instructions to: perform the steps of the method provided in this embodiment of the present disclosure. This embodiment of the present disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the method provided in this embodiment of the present disclosure. This embodiment of the present disclosure also provides a computer program product comprising a computer program / instructions, which, when executed by a processor, can implement the steps of the method provided in this embodiment of the present disclosure. In a storage engine stress testing method provided by embodiments of the present disclosure, after obtaining stress test parameters for a stand-alone storage engine to be tested, test data can be generated based on the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine. Furthermore, the test data can be made to match the data format corresponding to the workload of the stand-alone storage engine, better simulating the storage scenario of the workload. Data processing operations are performed on at least a portion of multiple data blocks through the read / write interface provided by the stand-alone storage engine, allowing the stressor to directly act on the stand-alone storage engine, meeting the requirements of stress testing the stand-alone storage engine. Furthermore, the type of data processing operation matches the operation type of the workload of the stand-alone storage engine, simulating the real load pressure of the stand-alone storage engine, thereby more accurately stress testing the stand-alone storage engine.BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings described herein are intended to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are intended to explain the present disclosure and do not constitute undue limitations thereon. In the accompanying drawings: Figure 1 is a flowchart of a storage engine stress testing method provided in an exemplary embodiment of the present disclosure; Figure 2 is a schematic diagram of data blocks and mapping relationships managed by a stressor provided in an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram of write and update operations of a stressor provided in an exemplary embodiment of the present disclosure; Figure 4 is a schematic diagram of a compression operation of a stressor provided in an exemplary embodiment of the present disclosure; and Figure 5 is a schematic diagram of the server structure provided in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION To further clarify the objectives, technical solutions, and advantages of the present disclosure, the technical solutions of the present disclosure will be described clearly and completely below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. It should be understood that the described embodiments are only some of the embodiments of the present disclosure, and are not exhaustive. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without inventive effort are within the scope of protection of the present disclosure. The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used in the embodiments of the present invention and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. "A plurality" generally includes at least two, but does not exclude the inclusion of at least one. It should be understood that the term "and / or" as used herein merely describes an associative relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent: A alone, A and B simultaneously, or B alone. Furthermore, the character " / " herein generally indicates that the associated objects are in an "or" relationship. It should also be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such product or system. Without further limitation, the elements specified by the phrase "comprising a..." do not exclude the presence of additional identical elements in the product or system comprising the elements. In a distributed storage system, Chunk Server (stand-alone storage engine) is an important component, mainly responsible for storing and managing data blocks.A standalone storage engine receives data write requests from clients or other servers and persistently stores data blocks on local disks. Typically, a standalone storage engine uses a specific index organization scheme to persist data in a specific format on storage media such as mechanical hard drives or solid-state drives. Many standalone storage engines in distributed systems employ append-only write semantics, which simplifies data consistency and concurrency control while also facilitating data backup and recovery. Under append-only write semantics, the file read and write interfaces of standalone storage engines are incompatible with the read and write interfaces of traditional POSIX (Portable Operating System Interface)-based kernel file systems and the read and write interfaces of user-mode file systems based on the SPDK (Storage Performance Development Kit). In some scenarios, stress testing of standalone storage engines is necessary to evaluate their performance under a given load. A typical load model is that of an LSM-tree-based storage engine. The LSM tree storage engine is a data format and storage engine design pattern based on a log structure and merge sort. It improves write performance and data compression efficiency through append-write and merge sort. However, there is currently a lack of a stressor that matches the specific workload types of standalone storage engines, making it difficult to accurately stress test standalone storage engines in specific scenarios. Currently, there is an open-source testing tool, FIO (Flexible I / O Tester), which is compatible with the read and write interfaces of POSIX-based kernel file systems and SPDK-based user-space file systems. It can test queue depths, data block sizes, and other parameters, and provides performance reports on throughput and latency. However, this FIO testing tool does not match the interfaces of standalone storage engines and cannot simulate some workload types of standalone storage engine workloads. Furthermore, such tools cannot simulate specific workloads to provide stress testing with KV (Key-Value) semantics, and therefore cannot provide write amplification factor assessments. Another commonly used testing tool is db_bench. This testing tool is compatible with the read and write interfaces of POSIX-based kernel file systems and SPDK-based user-mode file systems. It can provide KV semantic stress testing for storage engines. However, this testing tool does not match the interfaces of stand-alone storage engines and cannot be directly connected to them.To address the above technical issues, some embodiments of the present disclosure provide a solution. The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. FIG1 is a flow diagram of a storage engine stress testing method provided by an exemplary embodiment of the present disclosure. The method may include the steps shown in FIG1: Step 101: Obtain stress test parameters for a stand-alone storage engine to be tested; the stress test parameters include at least data generation parameters; the stand-alone storage engine is configured to manage multiple data blocks on a hard disk. Step 102: Generate test data based on the data generation parameters and a data format corresponding to the workload of the stand-alone storage engine. Step 103: Perform a data processing operation on at least some of the multiple data blocks using a read / write interface provided by the stand-alone storage engine; the type of the data processing operation matches the operation type of the workload of the stand-alone storage engine. Step 104: Calculate the write amplification factor of the stand-alone storage engine based on a first amount of data actually written to the hard disk by the data processing operation and a second amount of data corresponding to the test data. This embodiment can be performed by a stress tester, which can be configured to simulate the upper-layer workload of a stand-alone storage engine. The stress tester is developed based on the interface specifications or protocols for the read / write interfaces provided by any stand-alone storage engine in the distributed storage system. The stress tester can then read and write multiple data blocks managed by any stand-alone storage engine through the read / write interfaces provided by that stand-alone storage engine. During the test, the stress tester can obtain stress test parameters for the stand-alone storage engine. These stress test parameters are used to describe different test scenarios to simulate different load pressures. These stress test parameters include at least data generation parameters. In addition to the data generation parameters, these stress test parameters may also include at least one of the update frequency of data update operations, a target data volume threshold for written test data, and a hard disk usage threshold. These parameters will be described in detail in subsequent embodiments. The data generation parameters describe at least one of the test data generation method, format, and data length. Optionally, these data generation methods may include random generation, user input generation, data integration generation, or generation based on historical data, which is not limited in this embodiment. The data length of the test data describes the amount of storage space occupied by the test data. This length can be 2 KB, 4 KB, or other lengths, and is not limited in this embodiment. The stressor can generate test data based on data generation parameters and the data format corresponding to the workload of the stand-alone storage engine. The data format refers to the method used to store and organize data in a computer.For a standalone storage engine, the data format corresponding to its workload refers to the data storage structure used by the upper-layer application of the standalone storage engine. For example, the data format corresponding to the workload of a standalone storage engine can be a key-value (KV) structure, an array, a stack, a queue, a tree structure (such as a binary tree, a balanced binary tree, a search tree, etc.), or a graph structure (such as a graph, a hash table, a heap, etc.). In some embodiments, the workload of the standalone storage engine is an LSM tree storage engine, and the data format of the test data generated by the stressor is a KV format. For example, when the data format corresponding to the workload of the standalone storage engine is a KV format, the stressor can randomly generate a large number of KV pairs as test data. Taking the KV format of the test data as an example, the data generation parameters can optionally include a target key length and / or a target value length. Accordingly, when generating test data, the stressor can determine the target key length and / or target value length based on the data generation parameters. Optionally, when creating any key-value pair, a key is randomly generated, and a value corresponding to the key is randomly generated based on the target length of the value. After generating test data, the stressor can perform data processing operations on at least some of the multiple data blocks managed by the stand-alone storage engine through the read / write interface provided by the stand-alone storage engine. The data processing operations performed by the stressor on at least some of the multiple data blocks match the operation type of the workload of the stand-alone storage engine. The workload operation type describes the type of data operation performed by the upper-layer application of the stand-alone storage engine. The upper-layer application of the stand-alone storage engine can be a client or another storage engine. The workload operation type may include at least one of: data read operations, data write operations, data update operations, data delete operations, concurrent data processing operations, and metadata operations. In some optional embodiments, the data processing operation includes: a data write operation (input), a data update operation (update), and a data compression operation (compact). Accordingly, when performing the data processing operation on multiple data blocks, the stressor may write the test data to at least some of the multiple data blocks. In some optional embodiments, the stress test parameters also include the number of data blocks. Before writing the test data to at least some of the multiple data blocks, the stressor may create a corresponding number of data blocks in the stand-alone storage engine based on the number of data blocks. For example, if the number of data blocks to be created is set to 100 in the stress test parameters, the stressor may create 100 data blocks on the hard disk using the read / write interface of the stand-alone storage engine.As shown in Figure 2, the stressor can create multiple data blocks and maintain a collection of these data blocks through the read / write interface of a standalone storage engine. After writing any KV data pair to a data block, the stressor can use a mapping table in memory to maintain the mapping relationship between the key and the corresponding value storage location on the data block, and can use a reverse mapping table to maintain the reverse mapping relationship between the data block identifier and the collection of keys corresponding to the values ​​stored therein. Optionally, the stress test parameter may also include concurrency. Accordingly, when performing any write operation, the stressor may concurrently write test data to a corresponding number of data blocks based on the concurrency. Optionally, during the test data writing process, the stressor may perform an update operation on a portion of the test data already written to the data block. In some optional embodiments, the stress test parameter may also include an update frequency for the data update operation. Accordingly, when performing an update operation on a portion of the test data already written to the data block, the stressor may perform an update operation on the portion of the test data already written to the data block based on the update frequency specified in the stress test parameter. For example, the update frequency can be: after every five write operations, an update operation is performed; or after every ten write operations, an update operation is performed. Optionally, when the usage of the hard disk corresponding to the stand-alone storage engine meets a set threshold, the stressor can perform data compression on the data blocks on the hard disk where data has already been written. In this embodiment, if the stand-alone storage engine does not support update operations, the stressor can convert the update operation on the test data into an append operation on the data block and an invalid data marking operation. An example explanation is provided below. Optionally, when executing any data processing operation, the stressor can determine whether the processing operation is an update based on the update frequency. If not, the stressor can randomly generate a key-value pair and determine whether the data block currently being written exceeds the length limit. If the length limit is not exceeded, the key-value pair is written to the data block currently being written. If the length limit is exceeded, an empty data block is retrieved from the empty data block set and the key-value pair is written to the empty data block. If the processing operation type is update, the stressor selects a target key from the written test data and generates a new value corresponding to the target key. The stressor appends the target key and its corresponding new value to the data block in the write state and marks the target key and its old value as invalid data. The additionally written target key and its new value are marked as valid data. Furthermore, the target key and its old value are updated to the target key and its new value.Figure 3 is a schematic diagram of a write operation and an update operation of a stressor provided by an exemplary embodiment of the present disclosure. As shown in Figure 3, the update operation may include the following steps: Step S301: Checking whether the run time has been exceeded. In this embodiment, a run time may be set for the stressor. After the stressor begins running, it may determine whether the run time has been exceeded. If so, the test may be terminated. If not, step S302 may be executed. Step S302: Determine whether the operation is a write operation based on the frequency. The stressor may determine whether the current operation is a write operation based on the update frequency. If so, step S303 may be executed. If not, an update operation may be executed. This update operation may include randomly selecting K and regenerating a KV data pair. Step S303: Randomly generating a KV data pair. The stressor may randomly generate KV data and determine whether the data block currently being written exceeds the length limit. Step S304: Determine whether the data block currently being written exceeds the length limit. In this embodiment, it is determined whether the data block currently being written exceeds the length limit. If so, step S305 is executed; if not, step S306 is executed. In step S305, an empty data block is selected from a set of empty data blocks for writing. In this embodiment, if the length limit is exceeded, the stressor may select an empty data block from a maintained set of empty data blocks for writing. In step S306, a KV data pair is written. In this embodiment, if the length limit is not exceeded, the generated KV data is written to the data block currently being written. Optionally, if the current operation is determined to be an update based on the update frequency, the stressor may randomly select a previously written K, regenerate the KV data corresponding to the K, and perform a write operation based on the regenerated KV data. The newly generated KV data is written to the data block currently being written. The stressor may simultaneously update the maintained mapping table and reverse mapping table for subsequent table lookup. In some optional embodiments, stress test parameters may include a target data volume threshold for written test data. The target data volume threshold can be used to describe the upper limit of the data volume generated by the upper-layer application (i.e., workload) of a standalone storage engine during stable operation. Optionally, when performing an update operation on a portion of test data already written to a data block, if the amount of the written test data is less than the target data volume threshold, the stressor may perform an update operation on the portion of the test data already written to the data block according to the update frequency in the stress test parameter during the test data writing process.Optionally, if the amount of written test data is greater than or equal to the target data volume threshold, the stress test parameter may stop the test data write operation and perform the aforementioned update operation on the portion of the test data already written to the data block until the hard disk usage meets the set threshold. In some optional embodiments, the stress test parameter also includes: a first hard disk usage threshold, which is used to describe the upper limit of the hard disk usage for the data compression operation. The hard disk usage meeting the set threshold condition may include: the hard disk usage being greater than or equal to the first hard disk usage threshold. For example, the first hard disk usage threshold may be 85%, 90%, or another value. Accordingly, when the hard disk usage is greater than or equal to the first hard disk usage threshold, the stressor may perform a data compression operation on the data block in the stand-alone storage engine. In an LSM tree storage engine, a data compression operation generally refers to the process of storing data from the Ln layer of the LSM tree to the Ln+1 layer of the LSM tree. During the process of storing data from the Ln layer of the LSM tree to the Ln+1 layer of the LSM tree, a data deduplication operation may be performed. In an embodiment of the present disclosure, to simplify the operational complexity of stress testing, data compression operations can be performed at the same level in the LSM tree. That is, the source data block corresponding to the data and the destination data block corresponding to the data can be located at the same level in the LSM tree. Based on this, in this embodiment, the multiple data blocks created by the stressor can be located at the same level in the LSM tree. When performing data compression on a data block with written data, the stressor can migrate valid data in the written data block to another data block and delete the data block and its invalid data, thereby freeing up storage space for the data block. This is described below with an example. Optionally, when the hard disk usage corresponding to the standalone storage engine is greater than or equal to a first hard disk usage threshold, the stressor can determine a first data block from the data blocks on the hard disk where the test data has been written. The amount of valid data contained in the first data block satisfies a set condition. The set condition may include: the first data block having the largest amount of invalid data among the multiple data blocks where the test data has been written, or the first data block having the smallest amount of data (i.e., data write length) among the multiple data blocks where the test data has been written, etc., although this embodiment does not impose any limitation. The stressor can migrate valid data in the first data block to a second data block in the standalone storage engine, which is in a write state. The stressor can read valid data from the first data block and write the read valid data to the second data block. The second data block can be a blank data block among the multiple data blocks created during the test, or any data block that is not fully written among the multiple data blocks, and this embodiment does not impose any limitation.In some optional embodiments, the stress test parameters may further include a second hard disk usage threshold, which is used to trigger the cessation of data compression. For example, the second hard disk usage threshold may be 75%, 80%, or another value. Accordingly, during the data compression process, if the hard disk usage is less than or equal to the second hard disk usage threshold, the stressor may stop migrating valid data in the first data block. For the first data block, after the stressor migrates the valid data in the first data block, it may delete the remaining invalid data in the first data block to reclaim storage space, thereby achieving compression of the data stored in the first data block. After all valid data in the first data block has been migrated, the stressor may delete the first data block and create a new data block to replace it, ensuring that the total number of data blocks remains unchanged during the test. The following will provide an exemplary description of the above-mentioned compression process with reference to FIG4 . Step S401: Whether to stop the compression operation. As shown in FIG4 , the stressor may determine whether to stop the compression operation. For example, the compression device may determine whether the hard disk usage is less than or equal to a second hard disk usage threshold; if so, the compression operation is stopped; if not, the compression operation as shown in FIG4 is performed. If the compression operation is stopped, the compression process may be terminated; if not, step S402 may be performed. In step S402, the data block containing the least valid KV data pairs is selected. As shown in FIG4 , if the compression operation is not stopped, the compression device may select a target data block containing the least valid data. In step S403, valid KV data pairs are read from the selected data block. In this embodiment, the data block containing the least valid KV data pairs may be selected, and valid KV data may be read from the target data block. The read valid KV data may be rewritten into another data block. The other data block may be the data block currently being written. In step S404, whether the data block currently being written exceeds the length limit is determined. In this embodiment, the stressor can determine whether the data block currently being written exceeds the length limit. As shown in Figure 4, if the length limit is not exceeded, step S406 is executed; if the length limit is exceeded, step S405 is executed. In step S405, an empty data block is extracted from the empty data block set for writing. In this embodiment, if the length does not exceed the length threshold, the read valid KV data is written to the data block currently being written. Step S406 writes the KV data pair. In this embodiment, if the length limit is exceeded, an empty data block is extracted from the maintained empty data block set for writing.After all valid data in the target data block is written to other data blocks, invalid data in the target data block can be deleted, thereby reclaiming available space. After performing a data compression operation based on the above-described embodiment, the stressor can calculate the write amplification factor of the stand-alone storage engine based on the amount of data actually written to the hard disk of the stand-alone storage engine during the data processing operation and the amount of test data. Optionally, the stressor can determine the first amount of data actually written to the hard disk based on the second amount of data corresponding to the test data and the amount of data migrated during the data compression operation, and calculate the write amplification factor of the stand-alone storage engine under the stress test parameters based on the ratio of the first to second amounts of data. In this embodiment, after obtaining the stress test parameters for the stand-alone storage engine to be tested, test data can be generated based on the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine. This ensures that the test data matches the data format corresponding to the workload of the stand-alone storage engine, better simulating the storage scenario of the workload. By performing data processing operations on at least some of the multiple data blocks through the read / write interface provided by the stand-alone storage engine, the stressor can be directly applied to the stand-alone storage engine, meeting the requirements for stress testing the stand-alone storage engine. The type of data processing operation matches the workload type of the stand-alone storage engine, simulating the actual load pressure of the stand-alone storage engine, thereby more accurately stress testing the stand-alone storage engine. It is worth noting that in some optional embodiments, some parameters in the stress test parameters can be modified to simulate different load pressures. For example, the length of the values ​​in the stress test parameters can be modified to test the write amplification factor of the stand-alone storage engine under load pressures corresponding to values ​​of different lengths. For another example, the update frequency in the stress test parameters can be modified to test the write amplification factor of the stand-alone storage engine under different update frequencies. This will not be described further. It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, steps 101 to 104 may be performed by device A; for another example, steps 101 and 102 may be performed by device A, and step 103 may be performed by device B; and so on. Furthermore, some of the processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be understood that these operations may not be performed in the order in which they appear herein or may be performed in parallel. Operation sequence numbers, such as 101 and 102, are merely used to distinguish between different operations and do not represent any execution order.Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be noted that the terms "first" and "second" herein are used to distinguish between different messages, devices, components, and the like, and do not represent a sequential order, nor do they limit "first" and "second" to different types. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) involved in this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of these data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or deny. Figure 5 illustrates a schematic diagram of the server structure provided by an exemplary embodiment of the present disclosure. As shown in Figure 5, the server includes: a memory 501, a processor 502, and a communication component 503. Memory 501 is configured to store computer programs and can be configured to store various other data to support operations on the server. Examples of such data include instructions for any application or method configured to operate on the server. The processor 502 is coupled to the memory 501 and is configured to execute a computer program in the memory 501, so as to: obtain stress test parameters of a stand-alone storage engine to be tested; the stress test parameters include at least: data generation parameters; the stand-alone storage engine is configured to manage multiple data blocks on a hard disk; generate test data according to the data generation parameters and a data format corresponding to the workload of the stand-alone storage engine; perform a data processing operation on at least some of the multiple data blocks through a read / write interface provided by the stand-alone storage engine; the type of the data processing operation matches the operation type of the workload of the stand-alone storage engine; and calculate a write amplification factor of the stand-alone storage engine based on a first amount of data actually written to the hard disk by the data processing operation and a second amount of data corresponding to the test data. Optionally, when the processor 502 performs data processing operations on at least some of the multiple data blocks through the read / write interface provided by the stand-alone storage engine, the processor 502 is specifically configured to: write the test data to at least some of the multiple data blocks; perform an update operation on the test data already written to the data blocks; and, when the hard disk usage meets a set threshold, perform a data compression operation on the data blocks already written to the hard disk. Optionally, the workload of the stand-alone storage engine is an LSM tree storage engine; and the test data is in a key-value pair format.Optionally, when the processor 502 generates test data according to the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine, it is specifically configured to: determine the target length of the key and / or the target length of the value according to the data generation parameters and the data format; and use the set data generation algorithm to generate multiple key-value pairs according to the target length of the key and / or the target length of the value. Optionally, the stress test parameters further include: a target data volume threshold of the test data that has been written; the target data volume threshold is used to describe the upper limit of the data volume generated by the upper-layer application of the stand-alone storage engine during stable operation; accordingly, when the processor 502 performs an update operation on part of the test data written into the data block, it is specifically used to: if the data volume of the test data that has been written is less than the target data volume threshold, then in the process of writing the test data, according to the update frequency in the stress test parameters, perform an update operation on part of the test data that has been written into the data block; if the data volume of the test data that has been written is greater than or equal to the target data volume threshold, stop the test data writing operation, and perform an update operation on part of the test data that has been written into the data block, until the occupancy of the hard disk meets the set threshold condition. Optionally, when the processor 502 performs an update operation on part of the test data written into the data block according to the update frequency in the stress test parameter, the specific configuration is as follows: when performing any data processing operation, determine whether the type of the data processing operation is an update type according to the update frequency; if so, select a target key from the written test data; generate a new value corresponding to the target key; append the target key and its corresponding new value to the data block in the write state, and mark the target key and its old value as invalid data. Optionally, when the occupancy of the hard disk meets a set threshold condition, the processor 502 performs a data compression operation on the data blocks on which data has been written on the hard disk, and is specifically configured to: when the occupancy of the hard disk is greater than or equal to a first hard disk occupancy threshold, determine a first data block from the data blocks on which test data has been written on the hard disk; the number of invalid data contained in the first data block meets a set condition; migrate valid data other than the invalid data in the first data block to a second data block in a write state in the stand-alone storage engine, until the occupancy of the hard disk is less than or equal to the second hard disk occupancy threshold, stop migrating the valid data in the first data block; and perform a garbage collection operation on the remaining invalid data in the first data block.Optionally, when calculating the write amplification factor of the stand-alone storage engine based on the first amount of data actually written to the hard disk by the data processing operation and the second amount of test data, the processor 502 is specifically configured to: determine the first amount of data actually written to the hard disk based on the second amount of data corresponding to the test data and the amount of data migrated during the data compression operation; and calculate the write amplification factor of the stand-alone storage engine under the stress test parameters based on the ratio of the first amount of data to the second amount of data. Furthermore, as shown in FIG5 , the server also includes other components, such as a power supply component 504. FIG5 schematically illustrates only some components and does not imply that the server includes only the components shown in FIG5 . Memory 501 may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Communication component 503 is configured to facilitate wired or wireless communication between the device in which the communication component resides and other devices. The device containing the communication component can access a wireless network based on communication standards, such as Wi-Fi (wireless network communication technology), 2G (such as Global System for Mobile Communications (GSM)), 3G (such as Wideband Code Division Multiple Access (WCDMA), 4G (such as Long Term Evolution (LTE)), 4G+ (such as upgraded Long Term Evolution (LTE-Advanced, LTE-A)), or 5G (5th Generation Mobile Communication Technology), or a combination of them.In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component can be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, or other technologies. The power supply component 504 is configured to provide power to various components of the device where the power supply component resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component resides. In this embodiment, after obtaining stress test parameters for the stand-alone storage engine to be tested, test data can be generated based on the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine. Furthermore, the test data can be aligned with the data format corresponding to the workload of the stand-alone storage engine, thereby better simulating the storage scenario of the workload. By performing data processing operations on at least some of the multiple data blocks through the read / write interface provided by the stand-alone storage engine, the stressor can be directly applied to the stand-alone storage engine, meeting the requirements for stress testing the stand-alone storage engine. The type of data processing operation matches the workload type of the stand-alone storage engine, simulating the actual load pressure of the stand-alone storage engine, thereby more accurately stress testing the stand-alone storage engine. Accordingly, embodiments of the present disclosure also provide a computer-readable storage medium storing a computer program. When executed, the computer program can implement the steps of the above-described method embodiment that can be performed by the server. Accordingly, embodiments of the present disclosure also provide a computer program product, including a computer program / instructions. When executed by a processor, the computer program / instructions can implement the steps of the above-described method embodiment that can be performed by the server. Those skilled in the art will appreciate that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware.Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code. The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing device, produce means for implementing the functions specified in one or more processes in the flowcharts and / or one or more blocks in the block diagrams. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing the computer or other programmable device to execute a series of operational steps to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram. In a typical configuration, a computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium. Computer-readable media, including permanent and non-permanent, removable and non-removable media, can be implemented using any method or technology to store information. The information can be computer-readable instructions, data structures, program components, or other data.Examples of computer storage media include, but are not limited to, Parallel Random Access Machine (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be configured to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves. It should also be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, product, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, product, or apparatus. Without further limitation, the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus comprising the recited elements. The foregoing description is merely an example of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations of the present disclosure are possible. Any modifications, equivalent substitutions, improvements, and the like made within the spirit and principles of the present disclosure are intended to be encompassed by the claims of the present disclosure.Industrial Applicability The solution provided by the embodiment of the present disclosure can be applied to a distributed storage system, and stress test parameters of a stand-alone storage engine to be tested can be obtained; the stress test parameters include at least: data generation parameters; the stand-alone storage engine is configured to manage multiple data blocks on a hard disk; test data is generated according to the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine; data processing operations are performed on at least some of the multiple data blocks through the read and write interface provided by the stand-alone storage engine; the type of the data processing operation matches the operation type of the workload of the stand-alone storage engine; the write amplification factor of the stand-alone storage engine is calculated based on a first data volume actually written to the hard disk by the data processing operation and a second data volume corresponding to the test data, thereby solving the technical effect of the inability to accurately perform stress testing on the stand-alone storage engine in a specific scenario.

Claims

Claims 1. A storage engine stress testing method, comprising: Obtain the stress test parameters of the stand-alone storage engine to be tested; The stress test parameters include at least: data generation parameters; the stand-alone storage engine is configured to manage multiple data blocks on the hard disk; test data is generated according to the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine; data processing operations are performed on at least some of the multiple data blocks through the read and write interface provided by the stand-alone storage engine; the type of the data processing operation matches the operation type of the workload of the stand-alone storage engine; and the write amplification factor of the stand-alone storage engine is calculated based on a first amount of data actually written to the hard disk by the data processing operation and a second amount of data corresponding to the test data.

2. The method according to claim 1, wherein: Performing data processing operations on at least some of the multiple data blocks through the read-write interface provided by the stand-alone storage engine, including: writing the test data to at least some of the multiple data blocks; performing an update operation on some of the test data written to the data blocks; and performing a data compression operation on the data blocks on the hard disk that have data written when the hard disk usage meets a set threshold condition.

3. The method according to claim 2, wherein: The workload of the stand-alone storage engine is a log structure merge LSM tree storage engine; the data format of the test data is a key-value pair format.

4. The method according to claim 3, wherein: The test data is a key-value pair.

5. The method according to claim 4, wherein: Generate test data according to the data generation parameters and the data format corresponding to the workload of the stand-alone storage engine, including: determining a target length of the key and / or a target length of the value according to the data generation parameters and the data format; and using a set data generation algorithm to generate a plurality of key-value pairs according to the target length of the key and / or the target length of the value.

6. The method according to claim 2, wherein: The stress test parameters also include: a target data volume threshold of the written test data, wherein the target data volume threshold is used to describe the upper limit of the data volume generated by the upper-layer application of the stand-alone storage engine during stable operation.

7. The method according to claim 6, wherein: The performing an update operation on the portion of the test data written into the data block includes: if the amount of the written test data is less than the target data amount threshold, performing an update operation on the portion of the test data written into the data block according to the update frequency in the stress test parameter during the process of writing the test data.

8. The method according to claim 7, wherein: Performing an update operation on a portion of the test data written into the data block according to the update frequency in the stress test parameter includes: when performing any data processing operation, determining whether the data processing operation is an update type according to the update frequency; and if so, performing an update operation on the portion of the test data written into the data block, wherein the update operation includes: A target key is selected from the written test data, a new value corresponding to the target key is generated, the target key and its corresponding new value are appended to a data block in a written state, and the target key and its old value are marked as invalid data.

9. The method according to claim 8, wherein: The method further includes: if the answer is no, generating a key-value pair; and in response to the written data block not exceeding the target data amount threshold, writing the key-value pair into the written data block.

10. The method according to claim 8, wherein: Calculating the write amplification coefficient of the stand-alone storage engine based on the first data volume actually written on the hard disk by the data processing operation and the second data volume of the test data, including: determining the first data volume actually written to the hard disk based on the second data volume corresponding to the test data and the amount of data migrated during the data compression operation; and calculating the write amplification coefficient of the stand-alone storage engine under the stress test parameters based on the ratio of the first data volume to the second data volume.

11. The method according to claim 6, wherein: The performing an update operation on a portion of the test data written into the data block includes: in response to a data volume of the written test data being greater than or equal to a target data volume threshold, stopping the write operation on the test data, and performing an update operation on a portion of the test data written into the data block until the hard disk usage satisfies the set threshold condition.

12. The method according to claim 3, wherein: When the occupancy of the hard disk meets a set threshold condition, a data compression operation is performed on the data blocks on which data has been written on the hard disk, including: when the occupancy of the hard disk is greater than or equal to a first hard disk occupancy threshold, determining a first data block from the data blocks on which test data has been written on the hard disk; the number of invalid data contained in the first data block meets a set condition; migrating valid data other than the invalid data in the first data block to a second data block in a writing state in the stand-alone storage engine, until the occupancy of the hard disk is less than or equal to the second hard disk occupancy threshold, stopping the migration operation on the valid data in the first data block; and performing a garbage collection operation on the remaining invalid data in the first data block.

13. A server comprising: A memory and a processor; the memory is configured to store one or more computer instructions; the processor is configured to execute the one or more computer instructions to: perform the steps in the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the storage engine stress testing method according to any one of claims 1 to 12 can be implemented.

15. A computer program product, comprising a computer program / instruction, wherein when executed by a processor, the computer program / instruction can implement the storage engine stress testing method according to any one of claims 1 to 12.

Images