Data storage method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data storage method and device, computer equipment, a storage medium and a program product. The method comprises the steps of obtaining batch key value pairs corresponding to batch data in response to a processing request of the batch data; the batch key value pair comprises multiple groups of data and request information corresponding to each group of data; adding an end mark behind the last key value pair in the batch key value pairs, and recording the batch key value pairs added with the mark into a log file of a storage system; and writing the marked batch key value pair into a first cache region of the storage system, and solidifying cache content in the first cache region into the solid state disk at one time under the condition that the current offset of the first cache region reaches a preset offset or an end mark of the batch key value pair is monitored. By adopting the method, the data storage efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of storage system technology, and in particular to a data storage method, apparatus, computer equipment, storage medium, and program product. Background Technology

[0002] In a storage system, the system receives a large number of data processing requests from users and stores the data processing requests and their corresponding data in the solid-state drives (SSDs) of the storage system.

[0003] In related technologies, data storage typically involves first recording the data to be stored in batches in a log, then parsing the data in the log sequentially and storing it on a solid-state drive (SSD). However, in the case of batch data storage, these technologies require moving the data from the log to the SSD multiple times, resulting in low data storage efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a data storage method, apparatus, computer equipment, storage medium, and program product that can improve data storage efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a data storage method, the method comprising:

[0006] In response to a batch data processing request, retrieve the batch key-value pairs corresponding to the batch data; the batch key-value pairs include multiple sets of data and the request information corresponding to each set of data;

[0007] Add an end marker to the last key-value pair in the batch, and record the batch key-value pairs after adding the marker to the storage system's log file;

[0008] The batch key-value pairs with added tags are written to the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach the preset offset, or when the end tag of the batch key-value pairs is detected, the cached content in the first cache area is permanently saved to the solid-state drive.

[0009] In the data storage method provided in this application embodiment, in response to a batch data processing request, batch key-value pairs corresponding to the batch data are obtained; the batch key-value pairs include multiple sets of data and request information corresponding to each set of data; then, an end marker is added to the end of the last key-value pair in the batch key-value pairs, and the batch key-value pairs with the added marker are recorded in the log file of the storage system; then, the batch key-value pairs with the added marker are written to the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach a preset offset, or when the end marker of the batch key-value pairs is detected, the cached content in the first cache area is permanently saved to the solid-state drive. In this method, for batch data processing requests, after obtaining the batch key-value pairs, an end marker is first added to the batch key-value pairs and recorded in the log file. Then, the marked batch key-value pairs are written to the first cache area to ensure the transactional nature of batch data processing. Then, when the first cache area reaches a preset offset, such as when the cache area is full, or when all batch key-value pairs have been written, i.e. when the batch key-value pairs are cached, the cached content in the first cache area is solidified to the solid-state drive in one go. This reduces the number of interactions with the solid-state drive, reduces system overhead, shortens storage latency, and improves storage efficiency.

[0010] In one embodiment, before persisting the storage content in the first cache region to the solid-state drive, the method further includes:

[0011] Get the number of reference key-value pairs and the version number in a batch of key-value pairs;

[0012] Accordingly, the storage content in the first cache area is persisted to the solid-state drive, including:

[0013] Based on the version number, count the number of cache key-value pairs that match the version number in the first cache region;

[0014] If the number of cached key-value pairs matches the number of reference key-value pairs, the storage content in the first cache area will be persisted to the solid-state drive.

[0015] In the data storage method provided in this application embodiment, based on the version number, the number of cache key-value pairs matching the version number in the first cache area is counted. When the number of cache key-value pairs matches the number of reference key-value pairs, the storage content in the first cache area is solidified to the solid-state drive to ensure the consistency and integrity of batch key-value pairs, so as to facilitate the subsequent solidification of batch data according to the version number and simplify the subsequent data recovery logic.

[0016] In one embodiment, the method further includes:

[0017] If the number of cached key-value pairs does not match the number of reference key-value pairs, discard the batch of key-value pairs that match the version number in the first cache area.

[0018] In the data storage method provided in this application embodiment, when the number of cached key-value pairs does not match the number of reference key-value pairs, a batch of key-value pairs matching the version number in the first cache area are discarded to ensure the transactional nature of the batch data. At the same time, there is no need to interact with the solid-state drive to avoid invalid data storage.

[0019] In one embodiment, the method further includes:

[0020] Determine the system load level based on the storage system's load quantification metrics;

[0021] Based on the system load level, the maximum capacity of the first cache area and the time interval for writing the batch key-value pairs to the first cache area with added tags are adjusted.

[0022] In the data storage method provided in this application embodiment, the system load level is determined based on the load quantification index of the storage system. From the two dimensions of performance and reliability, the upper limit of the capacity of the first cache area and the time interval for writing the batch key-value pairs with added tags to the first cache area are adjusted to balance the performance of the write interface and the stability of the system.

[0023] In one embodiment, based on the system load level, the upper limit of the capacity of the first cache region and the time interval corresponding to the writing of the batch key-value pairs to be marked into the first cache region are adjusted, including:

[0024] When the system load level is greater than the preset load level, reduce the upper limit of the cache capacity of the first cache area and increase the time interval for writing the batch key-value pairs with added tags to the first cache area.

[0025] When the system load level is less than or equal to the preset load level, increase the upper limit of the cache capacity of the first cache area and reduce the time interval for writing the batch key-value pairs with added tags to the first cache area.

[0026] In the data storage method provided in this application embodiment, when the system load level is greater than the preset load level, the upper limit of the cache capacity of the first cache area is reduced, and the time interval for writing the batch key-value pairs with added tags to the first cache area is increased to ensure the write success rate and system stability; when the system load level is less than or equal to the preset load level, the upper limit of the cache capacity of the first cache area is increased, and the time interval for writing the batch key-value pairs with added tags to the first cache area is reduced to make full use of system resources, improve data processing throughput, and thus improve system write performance.

[0027] In one embodiment, the method further includes:

[0028] Upon detecting a storage system failure and recovery, retrieve multiple key-value pairs from the log file;

[0029] Multiple key-value pairs are written into the second cache area of ​​the storage system, and when the current offset of the second cache area reaches the preset offset, or when the end marker of multiple key-value pairs is detected, the cached content in the second cache area is permanently saved to the solid-state drive.

[0030] In the data storage method provided in this application embodiment, when a storage system fault recovery is detected, multiple key-value pairs in the log file are obtained, and the multiple key-value pairs are written into the second cache area of ​​the storage system. When the current offset of the second cache area reaches a preset offset, or when the end marker of the multiple key-value pairs is detected, the cached content in the second cache area is solidified to the solid-state drive at once, thereby realizing the batch and efficient migration of fault recovery data, reducing the loss of system performance caused by frequent disk writing, and saving system resources.

[0031] Secondly, this application also provides a data storage device, comprising:

[0032] The response module is used to respond to batch data processing requests and obtain batch key-value pairs corresponding to the batch data; the batch key-value pairs include multiple sets of data and the request information corresponding to each set of data;

[0033] The storage module is used to add an end marker to the last key-value pair in a batch of key-value pairs and record the batch of key-value pairs after adding the marker to the log file of the storage system;

[0034] The solidification module is used to write the batch key-value pairs with added tags into the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach the preset offset, or when the end mark of the batch key-value pairs is detected, the cached content in the first cache area is solidified to the solid-state drive in one go.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the embodiments of the first aspect described above.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above.

[0037] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above.

[0038] The aforementioned data storage method, apparatus, computer equipment, storage medium, and program product, in response to a batch data processing request, acquire batch key-value pairs corresponding to the batch data; the batch key-value pairs include multiple sets of data and request information corresponding to each set of data; then, an end marker is added to the end of the last key-value pair in the batch key-value pairs, and the batch key-value pairs with the added marker are recorded in the log file of the storage system; then, the batch key-value pairs with the added marker are written to the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach a preset offset, or when the end marker of the batch key-value pairs is detected, the cached content in the first cache area is permanently saved to the solid-state drive. In this method, for batch data processing requests, after obtaining the batch key-value pairs, an end marker is first added to the batch key-value pairs and recorded in the log file. Then, the marked batch key-value pairs are written to the first cache area to ensure the transactional nature of batch data processing. Then, when the first cache area reaches a preset offset, such as when the cache area is full, or when all batch key-value pairs have been written, that is, when the batch key-value pairs are cached, the cached content in the first cache area is solidified to the solid-state drive in one go. This reduces the number of interactions with the solid-state drive, reduces system overhead, shortens storage latency, and improves storage efficiency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is an internal structural diagram of a computer device in one embodiment;

[0041] Figure 2 This is a flowchart illustrating a data storage method in one embodiment;

[0042] Figure 3 This is a flowchart illustrating the cache content persistence step in one embodiment;

[0043] Figure 4 This is a flowchart illustrating the log adjustment steps based on load level in one embodiment;

[0044] Figure 5This is a flowchart illustrating the log adjustment steps based on load level in another embodiment;

[0045] Figure 6 This is a flowchart illustrating the data storage method in another embodiment;

[0046] Figure 7 This is a flowchart illustrating a log management method in one embodiment;

[0047] Figure 8 This is a flowchart illustrating a log playback method in one embodiment;

[0048] Figure 9 This is a structural block diagram of a data storage device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0051] The data storage method provided in this application can be applied to storage systems, such as flash memory systems. The storage system can be a complete dedicated storage device with its own processor, memory, and software. In this case, the data storage method is executed by the storage system's own processor. Alternatively, the storage system can be a storage device deployed on a computer device, which can be a server. Its internal structure diagram can be as follows: Figure 1As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data required for a data storage method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data storage method.

[0052] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0053] Taking a flash storage system as an example, when submitting data from the front-end management page to the back-end storage service, there are multiple requirements, including transactional requirements, write interface performance, and data recovery latency. The transactional requirement means that for write or delete requests of the same batch of data, either all requests must succeed, or the entire batch must fail; there cannot be a situation where some succeed and some fail. Therefore, to ensure transactional performance, the batch data log information needs to be recorded and persistently stored before processing the batch data.

[0054] In related technologies, when storing log data, the usual approach is to first record the log data to be stored in batches into log files, and then parse and solidify the data in the logs sequentially onto a solid-state drive (SSD). However, in the case of batch data storage, these technologies require moving the log data to the SSD multiple times. Due to the multiple interactions with the SSD, this wastes system resources and results in low log data storage efficiency.

[0055] Based on this, this application provides a data storage method that, in response to a batch data processing request, obtains batch key-value pairs corresponding to the batch data; then, adds an end marker to the last key-value pair in the batch key-value pairs and records the marked batch key-value pairs to the log file of the storage system; then, writes the marked batch key-value pairs into the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach a preset offset, or when the end marker of the batch key-value pairs is detected, the cached content in the first cache area is permanently stored in the solid-state drive (SSD) at once, reducing the number of interactions with the SSD, reducing system overhead, shortening storage latency, and improving storage efficiency.

[0056] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0057] In one exemplary embodiment, such as Figure 2 As shown, a data storage method is provided, including the following steps:

[0058] S201, in response to a batch data processing request, retrieve the batch key-value pairs corresponding to the batch data; the batch key-value pairs include multiple sets of data and the request information corresponding to each set of data.

[0059] Batch data processing requests include multiple data processing requests within the same batch, such as batch data processing requests at the same time or within the same time period. Each processing request can be a data deletion request, a data modification request, a data storage request, etc. To ensure the transactional nature of batch data processing requests, the corresponding log data needs to be stored and persisted before executing the batch data processing requests, so that data can be recovered in a timely manner in case of subsequent batch data processing interruption or system failure.

[0060] In response to a batch data processing request, each set of data and its corresponding request information (e.g., request type) are obtained. Then, each set of data and its corresponding request information are integrated in key-value pair format to obtain batch key-value pairs corresponding to the batch data. In this embodiment, the batch key-value pairs corresponding to the batch data represent the batch data processing request log information.

[0061] Optionally, for each set of data, the request information corresponding to that set of data can be used as the key, and the set of data itself can be used as the value to obtain key-value pairs for each set of data.

[0062] Optionally, for each set of data, extract the key information from the request information corresponding to that set of data, use the key information as the key, and use that set of data as the value to obtain key-value pairs corresponding to each set of data.

[0063] S202, add an end marker to the last key-value pair in the batch key-value pairs, and record the batch key-value pairs after adding the marker to the storage system's log file.

[0064] The batch of key-value pairs is sorted, and the last key-value pair in the sorting result is taken as the last group of log information in the batch. An end marker, such as an END marker, is added to the end of this key-value pair. In this embodiment, the sorting method of the batch key-value pairs is not limited. For example, the batch key-value pairs can be sorted according to the key pairs in the key-value pairs, or according to the storage amount of the key-value pairs.

[0065] When an end marker is added, the batch of tagged key-value pairs are recorded to the storage system's log file. It is important to emphasize that in this embodiment, the tagged batch of key-value pairs are recorded to the storage system's log file all at once to ensure the transactional nature of batch data processing.

[0066] In practical application scenarios, in addition to adding an end marker after the last key-value pair in the batch, intermediate markers, such as the PROCESS marker, are added to the other key-value pairs in the batch, except for the last one. The same version number is also added to each key-value pair in the batch. This is to define the logical boundaries of the batch requests through the END marker (batch end identifier), the PROCESS marker (processing status), and the batchCnt version number (batch unique identifier).

[0067] S203, write the batch key-value pairs with added tags into the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach the preset offset, or when the end mark of the batch key-value pairs is detected, the cached content in the first cache area is permanently saved to the solid-state drive.

[0068] The first cache area is a pre-defined initial memory region used to store batch key-value pairs from log files, providing a transit point for subsequent data persistence. Furthermore, the size of the first cache area can be flexibly adjusted based on the actual system load during subsequent log information storage.

[0069] The batch key-value pairs with added tags are written to the first cache area of ​​the storage system, and the offset of the first cache area is monitored in real time, as well as the written content in the first cache area is identified. Then, based on the monitoring results and identification results, the timing of the first cache area is determined.

[0070] The preset offset can be a preset storage amount or a value calculated based on the capacity of the first cache region and a preset ratio. If the current offset of the first cache region reaches the preset offset, it means that the cache amount of the first cache region is approaching saturation, and the cached content in the first cache region will be permanently saved to the solid-state drive.

[0071] If the end marker of a batch of key-value pairs is detected in the first cache area, it means that the log information of the batch data is stored in the first cache area. Then the cached content in the first cache area will be solidified to the solid-state drive in one go.

[0072] In this embodiment, in response to a batch data processing request, batch key-value pairs corresponding to the batch data are obtained; the batch key-value pairs include multiple sets of data and request information corresponding to each set of data; then, an end marker is added to the end of the last key-value pair in the batch key-value pairs, and the batch key-value pairs with the added marker are recorded in the log file of the storage system; then, the batch key-value pairs with the added marker are written to the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach a preset offset, or when the end marker of the batch key-value pairs is detected, the cached content in the first cache area is permanently saved to the solid-state drive. In this method, for batch data processing requests, after obtaining the batch key-value pairs, an end marker is first added to the batch key-value pairs and recorded in the log file. Then, the marked batch key-value pairs are written to the first cache area to ensure the transactional nature of batch data processing. Then, when the first cache area reaches a preset offset, such as when the cache area is full, or when all batch key-value pairs have been written, i.e. when the batch key-value pairs are cached, the cached content in the first cache area is solidified to the solid-state drive in one go. This reduces the number of interactions with the solid-state drive, reduces system overhead, shortens storage latency, and improves storage efficiency.

[0073] When processing the cached content in the first cache area, the process can proceed directly if a preset offset is met or an end marker is detected. Alternatively, the integrity of the content in the first cache area can be verified before or after the above conditions are met, and the content can be verified only after the verification is successful. Based on this, the integrity verification of the first cache area will be explained below through an embodiment.

[0074] In one exemplary embodiment, before persisting the storage content in the first cache region to the solid-state drive, the method further includes: obtaining the number of reference key-value pairs and the version number in the batch of key-value pairs.

[0075] When adding an end marker to the last key-value pair in a batch, you can also add intermediate markers to the other key-value pairs in the batch, and add the same version number to each key-value pair in the batch. Based on this, you can record the version number and count the number of key-value pairs with the same version number in the batch, as a reference number of key-value pairs.

[0076] Accordingly, such as Figure 3 As shown, the storage content in the first cache area is burned to the solid-state drive, including:

[0077] S301, based on the version number, count the number of cache key-value pairs in the first cache region that match the version number.

[0078] Query the cache key-value pairs corresponding to the version number in the first cache area, and count the number of each cache key-value pair in the first cache area to obtain the number of cache key-value pairs.

[0079] S302, when the number of cached key-value pairs matches the number of reference key-value pairs, the storage content in the first cache area is persisted to the solid-state drive.

[0080] The number of cached key-value pairs matches the number of reference key-value pairs. This means that the number of cached key-value pairs is equal to the number of reference key-value pairs. It means that no omissions occurred during the process of writing batch key-value pairs from the log file to the first cache area. In other words, the batch key-value pairs in the first cache area are complete. Then, the storage content in the first cache area will be solidified to the solid-state drive.

[0081] It should be noted that the step of matching the number of cached key-value pairs with the number of reference key-value pairs in this embodiment can be performed before the current offset of the first cache region reaches the preset offset, or it can be performed after the end marker of a batch of key-value pairs appears in the first cache region.

[0082] Optionally, if the number of cached key-value pairs matches the number of reference key-value pairs, and the current offset of the first cache region is detected to have reached a preset offset, then the storage content in the first cache region is saved to the solid-state drive; otherwise, no saving is performed.

[0083] Optionally, if the end marker of the batch of key-value pairs is detected, the number of cached key-value pairs is matched with the number of reference key-value pairs. If they match, the storage content in the first cache area is saved to the solid-state drive; if they do not match, they are not saved.

[0084] In this embodiment, based on the version number, the number of cache key-value pairs matching the version number in the first cache area is counted. When the number of cache key-value pairs matches the number of reference key-value pairs, the storage content in the first cache area is solidified to the solid-state drive to ensure the consistency and integrity of batch key-value pairs, so as to facilitate the subsequent solidification of batch data according to the version number and simplify the subsequent data recovery logic.

[0085] Next, the data storage method for scenarios where batch key-value pair validation fails will be further explained. In an exemplary embodiment, the method further includes:

[0086] If the number of cached key-value pairs does not match the number of reference key-value pairs, discard the batch of key-value pairs that match the version number in the first cache area.

[0087] If the number of cached key-value pairs does not match the number of reference key-value pairs, it means that the cached content in the first cache area is not a complete batch key-value pair. In this case, the batch key-value pairs that match the version number in the first cache area can be stopped from being persisted and the batch key-value pairs that match the version number can be discarded. At the same time, a rollback operation is triggered to retrieve the marked batch key-value pairs from the log file again and rewrite them into the first cache area.

[0088] In another scenario, the batch key-value pairs can be re-tagged, and the tagged batch key-value pairs can be re-recorded to the log file and written to the first cache area.

[0089] In this embodiment, when the number of cached key-value pairs does not match the number of reference key-value pairs, a batch of key-value pairs matching the version number in the first cache area are discarded to ensure the transactional nature of the batch data. At the same time, there is no need to interact with the solid-state drive, thus avoiding invalid data persistence.

[0090] It should be noted that the capacity of the first cache area is preset and can be flexibly adjusted according to the system load during subsequent processing. In an exemplary embodiment, such as... Figure 4 As shown, the method also includes:

[0091] S401 determines the system load level based on the storage system's load quantification metrics.

[0092] Load quantification metrics are quantitative values ​​characterizing the storage capacity of a storage system, including but not limited to request queue depth, request processing latency, and interface failure rate. The load quantification metrics of the storage system are preprocessed, such as through standardization and normalization. Then, the preprocessed load quantification metrics are merged, for example, through weighted summation, to obtain the current system load. Finally, the current system load is compared with the load ranges of various standard load levels, and the load level corresponding to the range to which the current system load belongs is determined as the current system load level.

[0093] S402, based on the system load level, adjusts the upper limit of the capacity of the first cache area and the time interval for writing the batch key-value pairs to be marked into the first cache area.

[0094] The upper limit of the capacity of the first cache region represents the number of batch key-value pairs that the first cache region can hold. The larger the upper limit of the capacity of the first cache region, the more batch key-value pairs the first cache region can hold. The smaller the upper limit of the capacity of the first cache region, the fewer batch key-value pairs the first cache region can hold.

[0095] For example, based on the mapping relationship between system load level, capacity of the first cache region, and time interval, the target capacity of the first cache region corresponding to the system load level, and the time interval corresponding to the system load level are determined. Further, the current capacity of the first cache region is adjusted to the target capacity, and the time interval for writing tagged batch key-value pairs to the first cache region is adjusted to the target time interval.

[0096] In this embodiment, based on the system load level determined by the load quantification index of the storage system, the capacity limit of the first cache area and the time interval for writing the batch key-value pairs with added tags to the first cache area are adjusted from the two dimensions of performance and reliability, taking into account both write interface performance and system stability.

[0097] The following example illustrates another implementation of S402, which adjusts the upper limit of the capacity of the first cache region and the time interval for writing the batch key-value pairs with added tags to the first cache region based on the system load level. Figure 5 As shown, it includes:

[0098] S501, when the system load level is greater than the preset load level, reduces the upper limit of the cache capacity of the first cache area and increases the time interval for writing the batch key-value pairs with added tags to the first cache area.

[0099] If the system load level is greater than the preset load level, it means that the storage system is currently under high load. In this case, the upper limit of the cache capacity of the first cache area is reduced, and the time interval for submitting to the first cache area is increased, so as to reduce the concurrency of writing to the first cache area and ensure the write success rate.

[0100] S502, when the system load level is less than or equal to the preset load level, increases the upper limit of the cache capacity of the first cache area and reduces the time interval for writing the batch key-value pairs with added tags to the corresponding first cache area.

[0101] If the system load level is less than or equal to the preset load level, it means that the storage system is currently under low load. In this case, the upper limit of the cache capacity of the first cache area is increased, and the time interval for submitting to the first cache area is reduced to speed up the write speed and ensure the performance of the system write interface.

[0102] It should be noted that when adjusting the upper limit of the cache capacity of the first cache area, such as shrinking or increasing it, it can be done gradually by a percentage of the current upper limit or a preset capacity value. When adjusting the time interval, it can be done gradually by a preset time period.

[0103] In this embodiment, when the system load level is greater than the preset load level, the upper limit of the cache capacity of the first cache area is reduced, and the time interval for writing the batch key-value pairs with added tags to the first cache area is increased to ensure the write success rate and system stability. When the system load level is less than or equal to the preset load level, the upper limit of the cache capacity of the first cache area is increased, and the time interval for writing the batch key-value pairs with added tags to the first cache area is reduced to make full use of system resources, improve data processing throughput, and thus improve system write performance.

[0104] The above embodiments describe the steps for storing batch key-value pairs. In practical applications, storage systems may experience interruptions due to unforeseen factors such as power outages, resulting in complete storage failure. In such cases, rapid log replay is required after system recovery. In an exemplary embodiment, as follows... Figure 6 As shown, the method also includes:

[0105] S601 retrieves multiple key-value pairs from the log file upon detecting a storage system failure recovery.

[0106] Taking a storage system failure caused by a power outage as an example, once the storage system is detected to have been powered back on, and the failure is confirmed to have been resolved, multiple key-value pairs with the same version number are retrieved from the log file. These represent a batch of log data that failed to be persisted during the system failure. It's important to note that in the case of storage system recovery, the key-value pairs stored in the log file are tagged; therefore, the key-value pairs corresponding to the specified version number can be directly retrieved from the log file.

[0107] S602 writes multiple key-value pairs into the second cache area of ​​the storage system, and when the current offset of the second cache area reaches the preset offset, or when the end marker of multiple key-value pairs is detected, the cached content in the second cache area is permanently saved to the solid-state drive at once.

[0108] The second cache area is a pre-defined initial memory region used to store batch key-value pairs in the log files, providing a transit point for subsequent data solidification. In practical applications, the second cache area in this embodiment can be the same area as the first cache area in the aforementioned embodiment, or it can be a different area.

[0109] In this embodiment, when a storage system failure recovery is detected, multiple key-value pairs in the log file are obtained, and these pairs are written into the second cache area of ​​the storage system. When the current offset of the second cache area reaches a preset offset, or when the end marker of the multiple key-value pairs is detected, the cached content in the second cache area is permanently saved to the solid-state drive in one go. This achieves batch and efficient migration of failure recovery data, reduces the loss of system performance caused by frequent disk writes, and saves system resources.

[0110] In an exemplary embodiment, taking log management in a flash memory system as an example, a data storage method is provided, such as... Figure 7 As shown, Figure 7 This is a diagram illustrating the log management process in a scenario involving batch request submissions. Figure 7 The flowchart shown includes the following steps:

[0111] (1) In response to the batch data processing request, obtain the batch key-value pairs corresponding to the batch data.

[0112] (2) Add an end marker after the last key-value pair in the batch key-value pairs, add an intermediate marker to the other key-value pairs in the batch key-value pairs except the last key-value pair, and add the same version number to each key-value pair in the batch key-value pairs.

[0113] (3) Record the batch key-value pairs with added tags to the log file of the storage system.

[0114] (4) Write the tagged batch key-value pairs into the first cache area of ​​the storage system.

[0115] (5) If the current offset of the first cache area reaches the preset offset, or if the first cache area has a batch of key-value pairs end markers, the cached content in the first cache area is solidified to the solid-state drive in one go.

[0116] (6) Initialize the offset of the first cache region to facilitate the next flush.

[0117] During the log management process described above, the load level of the flash memory system is monitored in real time. When the system load level is greater than the preset load level, the upper limit of the cache capacity of the first cache area is reduced, and the time interval for writing the batch key-value pairs marked with tags to the corresponding first cache area is increased, thereby reducing the write concurrency and ensuring the write success rate. When the system load level is less than or equal to the preset load level, the upper limit of the cache capacity of the first cache area is increased, and the time interval for writing the batch key-value pairs marked with tags to the corresponding first cache area is reduced, thereby improving the hardening speed, making the overall disk write traffic of the system stable, and the overall failure retry rate controllable.

[0118] In this embodiment, the logical boundaries of batch requests are defined by end markers, intermediate markers, and version numbers to ensure that a batch of write / delete requests either succeeds or fails entirely at the transaction level, avoiding data inconsistency caused by partial operations. A fixed-size cache block is also used, and after the accumulated log volume reaches a threshold, it is batch-flushed back to the key-value pair data block, reducing random I / O requests.

[0119] In an exemplary embodiment, taking log management in a flash memory system as an example, a data storage method is provided, such as... Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the log storage process in a log flushing scenario. Figure 8 The flowchart shown includes the following steps:

[0120] (1) In the event that the storage system has recovered from a failure, obtain multiple key-value pairs from the log file.

[0121] (2) Write multiple key-value pairs into the second cache area of ​​the storage system, and when the current offset of the second cache area is detected to reach the preset offset, or when the end marker of multiple key-value pairs is detected, compare the number of cached key-value pairs in the second cache area with the number of reference key-value pairs.

[0122] (3) When the number of cached key-value pairs matches the number of reference key-value pairs, the storage content in the first cache area is solidified to the solid-state drive.

[0123] (4) If the number of cached key-value pairs does not match the number of reference key-value pairs, discard the batch of key-value pairs that match the version number in the first cache area.

[0124] In this embodiment of the application, in the scenario of storage system failure recovery, the method of filtering valid log segments by end marker and version number shortens the log recovery latency.

[0125] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0126] Based on the same inventive concept, this application also provides a data storage device for implementing the data storage method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more data storage device embodiments provided below can be found in the limitations of the data storage method described above, and will not be repeated here.

[0127] In one exemplary embodiment, such as Figure 9 As shown, a data storage device is provided, including: a response module 901, a storage module 902, and a persistence module 903, wherein:

[0128] The response module 901 is used to respond to batch data processing requests and obtain batch key-value pairs corresponding to the batch data; the batch key-value pairs include multiple sets of data and request information corresponding to each set of data;

[0129] Storage module 902 is used to add an end marker to the last key-value pair in a batch of key-value pairs and record the batch of key-value pairs after adding the marker to the log file of the storage system;

[0130] The solidification module 903 is used to write the batch key-value pairs with added tags into the first cache area of ​​the storage system, and when it detects that the current offset of the first cache area reaches the preset offset, or when the end mark of the batch key-value pairs is detected, it solidifies the cached content in the first cache area to the solid-state drive in one go.

[0131] In one exemplary embodiment, the data storage device includes: an information acquisition module, which is used to acquire the number of reference key-value pairs and the version number in a batch of key-value pairs;

[0132] Accordingly, the solidification module 903 is used to count the number of cache key-value pairs that match the version number in the first cache area based on the version number; and when the number of cache key-value pairs matches the number of reference key-value pairs, the storage content in the first cache area is solidified to the solid-state drive.

[0133] In one exemplary embodiment, the data storage device includes a discard module for discarding a batch of key-value pairs matching a version number in a first cache region when the number of cached key-value pairs does not match the number of reference key-value pairs.

[0134] In one exemplary embodiment, the data storage device includes: a level determination module and an adjustment module, wherein:

[0135] The load level determination module is used to determine the system load level based on the load quantification indicators of the storage system.

[0136] The adjustment module is used to adjust the upper limit of the capacity of the first cache area and the time interval for writing the batch key-value pairs to the first cache area with added tags, based on the system load level.

[0137] In an exemplary embodiment, the adjustment module includes a first adjustment unit and a second adjustment unit, wherein:

[0138] The first adjustment unit is used to reduce the upper limit of the cache capacity of the first cache area and increase the time interval for writing the batch key-value pairs with added tags to the first cache area when the system load level is greater than the preset load level.

[0139] The second adjustment unit is used to increase the upper limit of the cache capacity of the first cache area and reduce the time interval for writing the batch key-value pairs with added tags to the first cache area when the system load level is less than or equal to the preset load level.

[0140] In an exemplary embodiment, the data storage device includes: a log flushing module, configured to acquire multiple key-value pairs from a log file when a storage system failure recovery is detected; write the multiple key-value pairs into a second cache area of ​​the storage system; and, when the current offset of the second cache area reaches a preset offset, or when the end marker of the multiple key-value pairs is detected, permanently save the cached content in the second cache area to the solid-state drive in one go.

[0141] Each module in the aforementioned data storage device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0142] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0144] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0145] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data storage method, characterized in that, The method includes: In response to a batch data processing request, obtain the batch key-value pairs corresponding to the batch data; the batch key-value pairs include multiple sets of data and request information corresponding to each set of data; Add an end marker to the end of the last key-value pair in the batch, and record the batch key-value pairs after adding the marker to the log file of the storage system; The batch key-value pairs with added tags are written into the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach a preset offset, or when the end mark of the batch key-value pairs is detected, the cached content in the first cache area is permanently saved to the solid-state drive.

2. The method according to claim 1, characterized in that, Before the step of burning the storage content in the first cache area to the solid-state drive, the method further includes: Obtain the number of reference key-value pairs and the version number in the batch of key-value pairs; Accordingly, the step of burning the storage content in the first cache area to the solid-state drive includes: Based on the version number, count the number of cache key-value pairs in the first cache region that match the version number; If the number of cached key-value pairs matches the number of reference key-value pairs, the storage content in the first cache area is saved to the solid-state drive.

3. The method according to claim 2, characterized in that, The method further includes: If the number of cached key-value pairs does not match the number of reference key-value pairs, discard the batch of key-value pairs in the first cache area that match the version number.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The system load level is determined based on the load quantification indicators of the storage system. Based on the system load level, the upper limit of the capacity of the first cache area and the time interval for writing the batch key-value pairs to be marked into the first cache area are adjusted.

5. The method according to claim 4, characterized in that, The adjustment of the capacity limit of the first cache region and the time interval for writing the batch key-value pairs to be marked into the first cache region based on the system load level includes: When the system load level is greater than the preset load level, the upper limit of the cache capacity of the first cache area is reduced, and the time interval for writing the batch key-value pairs with added tags to the first cache area is increased. If the system load level is less than or equal to the preset load level, increase the upper limit of the cache capacity of the first cache area and reduce the time interval between writing the batch key-value pairs to be marked into the first cache area.

6. The method according to any one of claims 1-3, characterized in that, The method further includes: Upon detecting that the storage system has recovered from a failure, multiple key-value pairs are retrieved from the log file; The multiple key-value pairs are written into the second cache area of ​​the storage system, and when the current offset of the second cache area is detected to reach the preset offset, or when the end marker of the multiple key-value pairs is detected, the cached content in the second cache area is permanently saved to the solid-state drive.

7. A data storage device, characterized in that, The device includes: The response module is used to respond to batch data processing requests and obtain batch key-value pairs corresponding to the batch data; the batch key-value pairs include multiple sets of data and request information corresponding to each set of data; The storage module is used to add an end marker to the last key-value pair in the batch key-value pairs and record the batch key-value pairs after adding the marker to the log file of the storage system; The solidification module is used to write batch key-value pairs with added tags into the first cache area of ​​the storage system, and when the current offset of the first cache area is detected to reach a preset offset, or when the end mark of the batch key-value pairs is detected, the cached content in the first cache area is solidified to the solid-state drive in one go.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images