Data processing method, computing device, storage medium, and program product

Abstract

Embodiments of the present application provide a data processing method, a computing device, a storage medium and a program product. In the method, a first storage area and a second storage area in a storage device are determined; at least one valid data segment in the first storage area is divided into multiple data segments, and multiple request tasks corresponding to the multiple data segments are generated; according to a task execution order, at least two read requests corresponding to the request tasks are triggered in parallel, and in a case that read request responses corresponding to multiple request tasks belonging to a same valid data segment are ended, write requests corresponding to the multiple request tasks are triggered in sequence to ensure sequential writing. The technical solution provided by the embodiments of the present application improves the data processing efficiency, ensures the garbage collection efficiency, and thus ensures the performance and availability of the storage device.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data processing method, computing device, storage medium, and program product. Background Technology

[0002] To ensure the storage performance of storage devices, it is necessary to perform garbage collection on the storage devices to clean up junk data, i.e. invalid data, in order to free up storage space.

[0003] One type of current storage device typically divides data into multiple fixed-size, contiguous storage areas. Each storage area only supports strict sequential writes, and erasure is performed on a per-storage-area basis. Garbage collection is usually performed by a storage engine running in user space. The process involves first identifying the first storage area requiring garbage collection, then writing valid data from the first storage area to a second storage area, and finally erasing the first storage area.

[0004] However, there is currently a problem of low garbage collection efficiency, which cannot keep up with the rate of storage space consumption, resulting in less available storage space and poor performance of storage devices. Summary of the Invention

[0005] This application provides a data processing method, computing device, storage medium, and program product to solve the technical problem of low waste recycling efficiency in the prior art.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] Determine the first storage region and the second storage region in the storage device;

[0008] Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments;

[0009] According to the task execution order, read requests corresponding to at least two untriggered request tasks are triggered in parallel; the read requests are used to read the corresponding data segments and write them into memory; the at least two request tasks correspond to one or more valid data segments;

[0010] When the read request responses corresponding to multiple request tasks belonging to the same valid data segment have ended, the write requests corresponding to the multiple request tasks are triggered sequentially; the write requests are used to read the corresponding data segments from the memory and write them into the second storage area.

[0011] Optionally, the step of sequentially triggering the write requests corresponding to the plurality of request tasks includes:

[0012] If the write request response for any triggered request task has ended, then the write request for the next triggered request task will be triggered.

[0013] Optionally, the step of triggering the read requests corresponding to at least two untriggered request tasks in parallel according to the task execution order includes:

[0014] Based on the number of parallel requests and the order of task execution, the read requests corresponding to at least two untriggered request tasks are triggered in parallel.

[0015] The method further includes:

[0016] If the write request response for any triggered request task has ended, then the read request for the next untriggered request task will be triggered.

[0017] Optionally, it also includes:

[0018] If the read request response for any triggered request task ends, then the read request for the next untriggered request task will be triggered.

[0019] Optionally, triggering the read request corresponding to the next untriggered request task after the read request response for any triggered request task has ended includes:

[0020] If the read request response for any triggered request task ends, and the number of request tasks that have triggered read requests and have not triggered write requests is less than a preset value, then the read request for the next untriggered request task will be triggered.

[0021] Optionally, the step of triggering the read requests corresponding to at least two untriggered request tasks in parallel according to the task execution order includes:

[0022] Based on the number of parallel requests and the order of task execution, the read requests corresponding to at least two untriggered request tasks are triggered in parallel.

[0023] The method further includes:

[0024] Count the number of currently triggered read request tasks;

[0025] If the read request response for any triggered request task ends, update the number of read request tasks.

[0026] Detect whether the number of read request tasks has reached the number of parallel requests;

[0027] If the number of read request tasks has not reached the number of parallel requests, the next read request corresponding to the untriggered request task will be triggered according to the task execution order.

[0028] Optionally, it also includes:

[0029] Count the number of currently triggered write request tasks;

[0030] If the write request response for any triggered request task ends, update the number of write request tasks.

[0031] Check if the number of write request tasks is equal to zero;

[0032] If the number of write request tasks is zero, the next write request corresponding to the already triggered request task will be triggered according to the task execution order.

[0033] Optionally, it also includes:

[0034] When the write request response for any triggered request task ends, the resources occupied by the request task are released.

[0035] Optionally, determining the first storage region and the second storage region in the storage device includes:

[0036] Identify the first storage area where the proportion of invalid data reaches a predetermined proportion;

[0037] Identify the second storage area that is currently idle.

[0038] Optionally, the step of dividing at least one valid data segment in the first storage area into multiple data segments and generating request tasks corresponding to each of the multiple data segments includes:

[0039] Identify at least one valid data segment in the first storage area;

[0040] The at least one valid data segment is divided into multiple data segments according to a predetermined size.

[0041] Generate a task list corresponding to the multiple data segments; the task list includes the request tasks corresponding to each of the multiple data segments.

[0042] Optionally, it also includes:

[0043] If the request tasks corresponding to the multiple data segments are successfully processed, the first storage area is switched to a garbage state in order to perform an erasure operation.

[0044] Secondly, this application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the data processing method as described in the first aspect above.

[0045] Thirdly, this application provides a computer storage medium storing a computer program, which, when executed by a computer, implements the data processing method described in the first aspect above.

[0046] Fourthly, this application provides a computer program product, including a computer program / instruction, which, when executed by a computer, implements the data processing method described in the first aspect above.

[0047] This embodiment of the application, following the task execution order, can trigger read requests corresponding to at least two request tasks in parallel without waiting for the completion of the previous request task. This allows for the parallel processing of at least two read requests. Furthermore, after the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, write requests corresponding to multiple request tasks are triggered sequentially to ensure sequential writing. Here, at least two request tasks can correspond to one or more valid data segments. This embodiment of the application improves data processing efficiency through the parallel processing of at least two read requests, thereby ensuring garbage collection efficiency and the performance of the storage device.

[0048] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of one embodiment of a data processing method provided in this application is shown;

[0051] Figure 2 A flowchart of yet another embodiment of a data processing method provided in this application is shown;

[0052] Figure 3 This illustration shows a read / write diagram of an embodiment of this application in a practical application;

[0053] Figure 4A flowchart of yet another embodiment of a data processing method provided in this application is shown;

[0054] Figure 5 This illustration shows a scenario interaction diagram of an embodiment of this application in a practical application;

[0055] Figure 6 This invention provides a schematic diagram of the structure of one embodiment of a data processing apparatus.

[0056] Figure 7 A schematic diagram of one embodiment of a computing device provided in this application is shown. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0058] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0059] The technical solution of this application embodiment is applied to the garbage collection scenario of storage devices, especially the garbage collection scenario of distributed storage systems. A distributed storage system can be composed of multiple storage servers, and each storage server can deploy multiple storage devices and can deploy a storage engine to manage the multiple storage devices, such as being responsible for garbage collection in the multiple storage devices.

[0060] For ease of understanding, the technical concepts that may be involved in the embodiments of this application are explained below:

[0061] Distributed storage system: refers to a storage method that distributes data across multiple storage servers in a network. This storage solution can provide high availability, fault tolerance, and scalability.

[0062] Storage devices: Physical hardware devices used to store data. These devices can be, for example, traditional hard disk drives (HDDs), solid-state drives (SSDs), or other forms of non-volatile storage media. In a distributed storage system, multiple storage devices can be deployed in each storage server to collectively provide storage services.

[0063] Zoned storage: A type of storage device whose address space is divided into zones with write constraints different from those of conventional storage devices. Typical zoned storage devices can include two types: ZNS SSD (Zoned Namespace Solid State Drive) and SMR HDD (Shingled Magnetic Recording Hard Disk Drive).

[0064] Zone: The smallest unit of allocation and erasure on zoned storage.

[0065] Storage area: A storage space on a storage device that is divided into fixed-size, contiguous blocks of memory. Each storage area only supports strict sequential writes and erasure on a per-storage-area basis. In Zoned Storage, a storage area is also known as a Zone space.

[0066] Garbage collection (GC) is the operation of reading valid data from the first storage area and writing it to the second storage area, while releasing the first storage area.

[0067] User mode and kernel mode: Kernel mode refers to the environment in which the operating system kernel runs; user mode refers to the environment in which applications and most user-level programs run. Storage devices run in kernel mode and can be garbage collected by the storage engine running in user mode, thus reducing the burden on the operating system kernel and improving the overall system efficiency.

[0068] Storage engine: An application that runs in user space.

[0069] As described in the background section, there is currently a technical problem of low garbage collection efficiency, which makes it impossible for garbage collection efficiency to keep up with the rate of storage area consumption, resulting in less available storage area and poor performance of storage devices.

[0070] The inventors discovered that due to limitations in garbage collection efficiency, garbage collection bandwidth can easily reach a bottleneck. Meanwhile, front-end throughput continuously consumes storage space, causing the garbage collection speed to be slower than the storage space consumption rate, resulting in less available storage space. Currently, the only solution is to limit the front-end throughput, which affects storage performance and reduces the storage experience.

[0071] Through a series of studies, the inventors discovered that in traditional methods, when a storage program performs garbage collection, valid data is first divided into multiple data segments, and corresponding request tasks are generated. For each data segment, a read request is triggered first to read the valid data from the first storage area and write it into memory. After the read request is completed, a write request is triggered to write the data from memory to the second storage area. Then, the next read request is triggered, and so on, until all valid data has been read into the second storage area, at which point the first storage area is erased. The inventors found that this implementation uses a serial processing method for reading valid data. If a request has a high latency, it will cause the entire process to stall, affecting the final garbage collection efficiency.

[0072] To improve garbage collection efficiency and storage performance, the inventors have proposed the technical solution of this application after a series of studies. In the embodiments of this application, at least two read requests corresponding to different request tasks can be triggered in parallel according to the task execution order. After the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, write requests corresponding to multiple request tasks are triggered sequentially to ensure sequential writing. The embodiments of this application realize the parallel processing of at least two read requests. Since at least two request tasks can correspond to one or more valid data segments, parallel processing of request tasks across data segments can be realized. Compared with the serial processing method, the data processing efficiency of valid data is improved, the throughput can be increased, and the processing resources can be fully utilized, thereby ensuring garbage collection efficiency and improving the utilization rate, performance, and availability of the storage device.

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0075] The implementation details of the technical solutions in the embodiments of this application are described in detail below.

[0076] Figure 1 A flowchart illustrating an embodiment of a data processing method provided in this application. The technical solution of this embodiment can be executed by a storage engine, which can run in user mode. The method may include the following steps:

[0077] 101: Determine the first storage area and the second storage area in the storage device.

[0078] Storage devices can be, for example, zoned storage, such as ZNS SSDs or SMR HDDs.

[0079] Storage devices can divide storage space into multiple fixed-size, contiguous storage regions. Each storage region only supports writing data strictly in sequence. Once a region is full, no new data can be written unless all data in that region is erased. Storage regions have four states: Free, Open, Closed, and Garbage. When a storage region is in the Free state, it means it is unused, contains no written data, and is fully available to receive write operations immediately. Once data is written to a storage region, it transitions from the Free state to the Open state. In the Open state, new data can continue to be appended to the region until its maximum capacity is reached or it is closed. When a storage region is full or closed via command, it transitions from the Open state to the Closed state. In the Closed state, the region no longer accepts new write requests but can still be read from. The storage area can perform garbage collection operations, which means migrating valid data to other storage areas. When valid data in a storage area has been deleted or marked as invalid, the storage area may be marked as garbage. Then, the entire storage area can be erased to free up space, and the storage area will be reset to the free state and become available again.

[0080] In this embodiment of the application, a first storage area and a second storage area in the storage device that meet the garbage collection requirements may be determined. The garbage collection requirements may be, for example, that the proportion of invalid data in the first storage area reaches a predetermined proportion and the second storage area is in an idle state. That is, a storage area with a predetermined proportion of invalid data may be determined as the first storage area and an idle storage area may be determined as the second storage area.

[0081] 102: Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments.

[0082] Here, a request task can refer to a task that reads data from a first storage area and writes it to a second storage area. Therefore, each request task can correspond to a read request that triggers the reading of data from the first storage area and the writing of it to memory, and a write request that triggers the reading of data from memory and the writing of it to the second storage area. Since executing a request task involves data reading and writing, a request task can also refer to an I / O (Input / Output) task.

[0083] The first storage area may include at least one valid data segment. If the valid data in the first storage area is contiguous in address, then the first storage area may contain one valid data segment. If the valid data in the first storage area is non-contiguous in address, then the first storage area may contain multiple valid data segments.

[0084] To avoid network congestion and significant memory pressure caused by directly transmitting excessively large valid data segments, which would negatively impact performance, this embodiment of the application can divide at least one valid data segment into multiple data segments. Optionally, each valid data segment can be divided into multiple data segments according to a predetermined size. Each valid data segment can be divided into one or more data segments based on its size. The predetermined size can be determined based on actual IO requirements, such as 64KB (kilobytes).

[0085] For each data segment, a corresponding request task can be generated. Each request task can include a read request and a write request. The read request is used to trigger the reading of the corresponding data segment and writing it into memory. The write request is used to trigger the reading of the corresponding data segment from memory and writing it into the second storage area, thereby realizing effective data transfer.

[0086] The generated multiple request tasks can be saved to a task list. The task list stores the request information for each request task, which may include, for example, the identifier of the second storage area, the offset of the data segment, and the length.

[0087] 103: In accordance with the task execution order, trigger the read requests corresponding to at least two untriggered request tasks in parallel.

[0088] The read request is used to read the corresponding data segment and write it into memory.

[0089] At least two request tasks can correspond to one or more valid data segments.

[0090] For example, since data is written sequentially to the storage area, the execution order of the request tasks corresponding to multiple data segments can be determined based on the storage addresses of the data segments. The storage addresses of the data segments can be determined based on the offset and length of the data segments. Specifically, triggering the read requests corresponding to at least two untriggered request tasks in parallel according to the task execution order can be: triggering the read requests corresponding to at least two untriggered request tasks in parallel according to the number of parallel requests and the task execution order. The number of parallel requests indicates the maximum number of read requests that can be triggered simultaneously. This number of parallel requests can, for example, refer to the IO Queue Depth (input / output queue depth), which indicates the maximum number of read requests that can be triggered simultaneously. Optionally, this number of parallel requests is greater than 1.

[0091] In this embodiment of the application, an untriggered request task refers to a request task that has not triggered a read request or a write request, while a triggered request task refers to a request task that has triggered a read request or a write request.

[0092] 104: When multiple request tasks belonging to the same valid data segment have completed their respective read request responses, the write requests corresponding to the multiple request tasks are triggered sequentially.

[0093] Write requests are used to read corresponding data segments from memory and write them to the second storage area.

[0094] The write requests corresponding to multiple request tasks belonging to the same valid data segment can be triggered sequentially according to the task execution order. Optionally, sequential triggering can mean that after the write request response of any triggered request task has ended, the write request corresponding to the next triggered request task is triggered to ensure that the data is written sequentially in the second storage area according to the task execution order.

[0095] In this method, the parallel triggering of at least two read requests corresponding to untriggered request tasks can achieve parallel processing of at least two read requests. Since the response completion order of at least two read requests may differ from the task execution order, the write requests corresponding to multiple request tasks belonging to the same valid data segment can be triggered sequentially after all read requests corresponding to multiple request tasks belonging to the same valid data segment have completed their responses. This ensures that the valid data segment is complete, so that even if the completion order of read requests corresponding to multiple tasks belonging to the same valid data segment triggered in parallel differs from the task execution order, it can still be guaranteed that after the valid data segment is completely read into memory, at least one data segment included in it can be written sequentially to the second storage area according to the task execution order.

[0096] In this embodiment, according to the task execution order, at least two read requests corresponding to the request tasks can be triggered in parallel without waiting for the previous request task to finish, enabling multiple read requests to be processed in parallel. After the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, the write requests corresponding to multiple request tasks are triggered sequentially to ensure data integrity and sequential writing. This embodiment of the application realizes the parallel processing of multiple read requests. Since at least two request tasks can correspond to one or more valid data segments, parallel processing of request tasks across data segments can be realized. Compared with the serial processing method, throughput can be improved, processing resources can be fully utilized, thereby improving garbage collection efficiency, improving the utilization rate of storage devices, and improving the performance and availability of storage devices.

[0097] In some embodiments, dividing at least one valid data segment in the first storage area into multiple data segments and generating request tasks corresponding to each of the multiple data segments may include: determining at least one valid data segment in the first storage area; dividing the at least one valid data segment into multiple data segments according to a predetermined size; generating a task list corresponding to the multiple data segments; the task list includes request tasks corresponding to each of the multiple data segments.

[0098] In this embodiment, the valid data in the first storage area may be distributed across one or more valid data segments. However, a single valid data segment might be too large. Directly transmitting such a large segment could lead to network congestion and significant memory pressure. Furthermore, an excessively large valid data segment might not be fully loaded into memory, causing program crashes. This embodiment divides the valid data segments in the first storage area into multiple fixed-length data segments, each corresponding to an I / O request task. This ensures that the amount of data involved in each I / O operation is the same, resulting in a more even I / O load. The read / write operations for each I / O request task will have relatively consistent time requirements. This consistency helps reduce operation time fluctuations caused by varying valid data segment sizes. Moreover, compared to transmitting larger valid data segments, this reduces latency and improves overall system performance.

[0099] In some embodiments, the method may further include: releasing the resources occupied by the request task when the write request response corresponding to any triggered request task ends.

[0100] For example, since the request information of a request task is stored in memory, the end of the write request response of any triggered request task indicates that the execution of the request task has ended. At this time, the memory resources occupied by the request information of the request task, as well as the memory resources occupied by the data segments read by the corresponding read request, can be released.

[0101] In this application embodiment, there are multiple ways to implement parallel processing of read requests.

[0102] As an optional implementation, as described above, triggering the write requests corresponding to multiple request tasks in sequence can be done by: triggering the write request corresponding to the next triggered request task after the write request response of any triggered request task has ended.

[0103] Optionally, the above-mentioned parallel triggering of read requests corresponding to at least two untriggered request tasks according to the task execution order can be: triggering read requests corresponding to at least two untriggered request tasks in parallel according to the number of parallel requests and the task execution order. The method may further include: triggering the read request corresponding to the next untriggered request task after the write request response for any triggered request task has ended. Here, the next untriggered request task is the first request task among the remaining untriggered request tasks determined according to the task execution order.

[0104] like Figure 2 The illustrated embodiment shows another data processing method provided in this application, which may include the following steps:

[0105] 201: Determine the first storage region and the second storage region in the storage device.

[0106] 202: Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments.

[0107] For detailed instructions on steps 201 to 202, please refer to [link / reference]. Figure 1 Steps 101 to 102 in the illustrated embodiment will not be repeated here.

[0108] 203: Based on the number of parallel requests and the order of task execution, trigger the read requests corresponding to at least two untriggered request tasks in parallel.

[0109] Based on the number of parallel requests, it can refer to the read requests corresponding to at least two request tasks that have not yet been triggered, which can be triggered simultaneously. The number of parallel requests also indicates the maximum number of read requests that can be triggered at the same time.

[0110] 204: If multiple request tasks belonging to the same valid data segment have completed their respective read request responses, write requests corresponding to the multiple request tasks will be triggered sequentially according to the task execution order.

[0111] In other words, when all read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, the write requests corresponding to each request task are triggered one by one according to the task execution order, starting from the first request task among the multiple request tasks. That is, when the write request corresponding to any one of the triggered request tasks finishes responding, the write request corresponding to the next triggered request task is triggered.

[0112] 205: If the write request response for any triggered request task has ended, trigger the read request for the next untriggered request task.

[0113] In this optional implementation, after dividing the valid data segment into multiple data segments, at least two read requests corresponding to each request task can be triggered in parallel according to the number of parallel requests. Once all read requests corresponding to multiple request tasks belonging to the same valid data segment have responded and completed, write requests corresponding to multiple request tasks are then triggered sequentially. That is, if the write request corresponding to any one of the triggered request tasks has finished responding, the write request corresponding to the next triggered request task is triggered. Furthermore, to further improve processing efficiency and performance, while triggering the write request corresponding to the next triggered request task, a read request corresponding to the next untriggered request task can also be triggered simultaneously, thus achieving parallel processing of read and write requests.

[0114] For ease of understanding, see [link to relevant documentation]. Figure 3 In the read / write diagram shown, as Figure 3As shown, assuming the first storage area includes three valid data segments (valid data segment 0, valid data segment 1, and valid data segment 2), the three valid data segments are divided into multiple data segments according to a predetermined size, generating a task list corresponding to each data segment. The task list includes the request tasks corresponding to each data segment. For example, valid data segment 0 can be divided into four data segments, which correspond to request tasks IO0, IO1, IO2, and IO3, respectively. Valid data segment 1 and valid data segment 2 can be divided into multiple data segments, which correspond to request tasks IO4, IO5, etc. For example, the number of parallel requests can be set to 5. Then, according to the number of parallel requests and the task execution order, the number of parallel requests can be triggered in parallel for the first time. For example, the read requests corresponding to IO0-IO3 of valid data segment 0 and IO4 of valid data segment 1 can be triggered simultaneously. After all the read requests corresponding to IO0-IO3 of valid data segment 0 have finished responding, the write request corresponding to IO0 can be triggered. After the write request corresponding to IO0 has finished responding, the write request corresponding to the next triggered request task IO1 and the read request corresponding to the next untriggered request task IO5 can be triggered simultaneously.

[0115] As an alternative implementation, the method may further include: triggering a read request for the next untriggered request task when the read request response of any triggered request task has ended.

[0116] In this optional implementation, once the read request response of any triggered request task ends, the read request corresponding to the next untriggered request task can be triggered, so that the triggering of the read request is not affected by the write request, further ensuring the parallel processing of read requests.

[0117] In this optional implementation, if a write request from any triggered request task is responded to, a read request corresponding to the next untriggered request task can also be triggered to ensure that read and write requests are processed in parallel.

[0118] Optionally, if the read request response of any triggered request task ends, triggering the read request corresponding to the next untriggered request task can be done as follows: if the read request response of any triggered request task ends, and the number of request tasks with triggered read requests and untriggered write requests is less than a preset value, then trigger the read request corresponding to the next untriggered request task.

[0119] For example, if the read requests corresponding to IO1-IO9 have been triggered, but the write requests corresponding to IO1-IO9 have not been triggered, then the number of requests with triggered read requests and those with untriggered write requests is 9. If the preset value is 6, then the condition of being less than the preset value is not met, and the next untriggered request task cannot be triggered.

[0120] Since processing read requests requires memory resources to store the data segments read from the first storage area, setting a preset value can prevent the number of tasks that have triggered read requests and those that have not triggered write requests from becoming too large, thus avoiding excessive memory consumption and performance impact.

[0121] According to the number of parallel requests and the task execution order, this embodiment can trigger read requests corresponding to at least two request tasks in parallel without waiting for the previous request task to finish, enabling the parallel processing of multiple read requests. After the read request responses corresponding to multiple request tasks have finished, write requests corresponding to multiple request tasks are triggered sequentially to ensure sequential writing. This embodiment, through the parallel processing of multiple read requests, since at least two request tasks can correspond to one or more valid data segments, enables parallel processing of request tasks across data segments, improving throughput, making full use of processing resources, thereby improving data processing efficiency, ensuring garbage collection efficiency, and ensuring the performance of the storage device.

[0122] As another optional implementation, parallel processing can be achieved by counting the number of request tasks. Therefore, such as Figure 4 In the illustrated embodiment, another data processing method provided in this application may include the following steps:

[0123] 401: Determine the first storage region and the second storage region in the storage device.

[0124] 402: Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments.

[0125] Step 401 can be found in [reference]. Figure 1 Steps 101 and 402 of the illustrated embodiment can be found in [reference needed]. Figure 1 Step 102 of the illustrated embodiment will not be described in detail here.

[0126] 403: Based on the number of parallel requests and the order of task execution, at least two read requests that have not yet been triggered are triggered in parallel.

[0127] 404: When multiple request tasks belonging to the same valid data segment have completed their respective read request responses, write requests corresponding to the multiple request tasks are triggered sequentially.

[0128] 405: Counts the number of currently triggered read request tasks.

[0129] 406: Update the number of read request tasks if the read request response for any triggered request task has ended.

[0130] 407: Check if the number of read request tasks has reached the number of concurrent requests. If not, proceed to step 408; if yes, continue checking.

[0131] 408: Following the task execution order, trigger the read request corresponding to the next untriggered request task, and return to step 405 to continue execution.

[0132] For example, still assuming that the first storage area includes three valid data segments (valid data segment 0, valid data segment 1, and valid data segment 2), the three valid data segments are divided into multiple data segments according to a predetermined size, and a task list corresponding to the multiple data segments is generated. The task list includes the request tasks corresponding to the multiple data segments. For example, valid data segment 0 can be divided into four data segments, which correspond to request tasks IO0, IO1, IO2, and IO3, respectively. Valid data segment 1 and valid data segment 2 can be divided into multiple data segments, which correspond to request tasks IO4, IO5, etc., respectively. For example, the number of parallel requests can be set to 5. Initially, 5 read requests can be triggered in parallel: IO0-IO3 corresponding to valid data segment 0 and IO4 corresponding to valid data segment 1. The number of currently triggered read request tasks is counted as the number of parallel requests, which is 5. The system continues to monitor. When the read request corresponding to IO0 finishes responding, the number of currently triggered read request tasks is decremented by 1, and the number of request tasks is updated to 4. If it is detected that the number of read request tasks 4 cannot reach the number of parallel requests 5, then the next untriggered request task corresponding to read request IO5 is triggered. The number of currently triggered read request tasks will be increased from 4 to 5. The number of currently triggered read request tasks is counted as 5. If it is detected that the number of read request tasks has reached the number of parallel requests 5, the system continues to monitor until it is detected that the number of read request tasks cannot reach the number of parallel requests, at which point the next untriggered request task corresponding to read request is triggered.

[0133] This embodiment, based on the number of parallel requests and the task execution order, can trigger read requests corresponding to at least two request tasks in parallel without waiting for the completion of the previous request task. This allows for the parallel processing of multiple read requests. Furthermore, after the read request responses for multiple request tasks belonging to the same valid data segment have ended, the write requests corresponding to those tasks are triggered sequentially to ensure sequential writing. This achieves parallel processing of multiple read requests, which, compared to serial processing, improves throughput, fully utilizes processing resources, and thus improves garbage collection efficiency, storage device utilization, and overall storage device performance and availability. In addition, based on the statistics and comparison of the number of read request tasks and the number of parallel requests, after the read request response for any triggered request task has ended, the next untriggered request task's read request can be triggered, ensuring the number of read request tasks reaches the number of parallel requests, thereby fully utilizing processing resources and improving throughput.

[0134] In some embodiments, the method may further include: counting the number of currently triggered write request tasks; updating the number of write request tasks when the write request response corresponding to any triggered request task has ended; detecting whether the number of write request tasks is equal to zero; and, if the number of write request tasks is equal to zero, triggering the write request corresponding to the next triggered request task according to the task execution order.

[0135] For example, by counting the number of currently triggered write request tasks, the number of write request tasks can be limited to no more than 1.

[0136] By counting the number of write request tasks, the system can trigger the next write request task after any given task completes. Read request triggering is independent of write request triggering, allowing for parallel processing of read and write requests. This avoids the problems of read / write sequential processing that can cause significant bandwidth fluctuations and latency due to mutual waiting, improving data processing efficiency, ensuring efficient garbage collection, and ultimately increasing storage device utilization, reducing write latency, and enhancing storage device performance and availability.

[0137] In some embodiments, as described above, the storage engine runs in user mode, and the read or write requests triggered by the storage engine need to be sent to the storage device for response. Therefore, the above-mentioned parallel triggering of at least two untriggered request tasks corresponding to read requests may include: sending the read requests corresponding to at least two untriggered request tasks to the storage device in parallel, so that the storage device can read the data segments corresponding to the read requests and write them into memory.

[0138] The storage device can read the data segment corresponding to the read request and write it into memory based on the read request information. The read request information may include, for example, storage address information such as a second storage area identifier, offset, and length. The storage device can read the corresponding data segment based on the storage address information and write it into memory. The storage device can then feed back the memory address of the data segment in memory to the storage engine.

[0139] The above-mentioned sequential triggering of at least two write requests corresponding to the request tasks may include: sequentially sending the write requests corresponding to at least two request tasks to the storage device, so that the storage device can read the data segments corresponding to the write requests from memory and write them into the second storage area.

[0140] The storage device can read the data segment corresponding to the write request and write it to the second storage area based on the write request information. The write request information may include, for example, the second storage area identifier and memory address. The storage device can determine the data segment to be retrieved from memory based on the memory address, and determine the corresponding second storage area to write the data segment based on the second storage area identifier.

[0141] In this embodiment, the storage device can be managed by a kernel-mode device driver, while the technical solution can be executed by a storage engine running in user mode. When the user-mode storage engine needs to read or write data to the storage device, it can send read / write requests to the kernel. After receiving these read / write requests, the kernel can interact with the storage device through the corresponding device driver to enable the storage device to perform actual read / write operations.

[0142] In some embodiments, the method may further include: if the request tasks corresponding to the multiple data segments are successfully processed, switching the first storage area to a garbage state in order to perform an erasure operation.

[0143] In this scenario, once the request tasks corresponding to multiple data segments have been processed, all valid data in the first storage area is migrated to the second storage area. The first storage area contains only invalid data. In the garbage state, all invalid data in the first storage area can be erased, and the first storage area can be reset to the Free state, becoming a writable storage space. By reclaiming the space that stores invalid data, more available space is provided for new data writing, which can improve the utilization of the storage device, reduce write latency, and improve the performance and availability of the storage device.

[0144] For ease of understanding, Figure 5This diagram illustrates a scenario interaction diagram of an embodiment of this application in a practical application. The technical solution of this embodiment can be applied to garbage collection of storage devices involved in a distributed storage system. The distributed storage system can consist of multiple storage servers, and each storage server can deploy multiple storage devices, and a storage engine can be deployed to manage these multiple storage devices. Figure 5 As shown, storage server 500 can deploy storage engine 501 and multiple storage devices. The storage engine can run in user space, while the storage devices run in kernel space. In practical applications, the storage server can be implemented as a physical machine.

[0145] For ease of description, let's take storage device 502, one of multiple storage devices, as an example. See [link to documentation]. Figure 4 The processing flowchart shown illustrates that storage engine 501 can determine a first storage area in storage device 502 where the proportion of invalid data reaches a predetermined percentage, and select a second storage area that is in an idle state. It then divides at least one valid data segment in the first storage area into multiple data segments and generates request tasks corresponding to each data segment, thereby obtaining a task list consisting of multiple request tasks. Storage engine 501 can trigger read requests from at least two request tasks according to the number of parallel requests and the task execution order. For example, storage engine 501 can initially trigger a number of parallel read requests. Specifically, the read requests are sent to storage device 502, which can simultaneously respond to multiple read requests and read the corresponding data segments and write them into memory.

[0146] Storage engine 501 can sequentially send write requests corresponding to multiple request tasks to storage device 502 after the read request responses for multiple request tasks belonging to the same valid data segment have ended. Storage device 502 can read the corresponding data segment from memory and write it into the second storage area according to the request information of the write request.

[0147] Storage engine 501 can initially trigger a number of concurrent read requests. It can detect when the number of read request tasks reaches the number of concurrent requests and continue detecting. If the response to a read request corresponding to any triggered request task ends, the number of read request tasks can be decremented by 1. If the number of read request tasks cannot reach the number of concurrent requests, the next untriggered request task's corresponding read request can be triggered, and the number of read request tasks can be incremented by 1. If the number of read request tasks reaches the number of concurrent requests, the detection can continue.

[0148] Storage engine 501 can trigger a write request corresponding to the first triggered request task among multiple triggered request tasks when the read request responses for multiple request tasks belonging to the same valid data segment have ended. Storage engine 501 can count the number of currently triggered write request tasks as 1. When the write request response for any triggered request task ends, it updates the number of write request tasks to 0. If the number of write request tasks is 0, it will trigger the write request corresponding to the next triggered request task according to the task execution order.

[0149] By counting the number of read request tasks, the number of read request tasks can reach the number of parallel requests, allowing multiple read requests to be processed in parallel to fully utilize processing resources and improve throughput. By counting the number of write request tasks, the next triggered write request task can be triggered after the completion of the write request corresponding to any triggered request task. The triggering of read requests and write requests is independent, enabling parallel processing of read and write requests. This avoids the problems of read / write serial processing that can cause drastic fluctuations in throughput bandwidth and latency due to mutual waiting between read and write operations, thus improving data processing efficiency, ensuring garbage collection efficiency, improving storage device utilization, reducing write latency, and ultimately improving the performance and availability of the storage device.

[0150] In this scenario, if the request tasks corresponding to multiple data segments in the first storage area are successfully processed (meaning all valid data in the first storage area is read into the second storage area), the first storage area can be switched to a garbage state for erasure. After erasing all invalid data in the first storage area, it can be reset to a free state, becoming writable storage space. By reclaiming the space previously used for storing invalid data, more available space is provided for new data writing, improving storage device utilization, reducing write latency, and enhancing the performance and availability of the storage device.

[0151] This embodiment implements parallel processing of multiple read requests and parallel processing of read and write requests. Compared with serial processing, it can improve throughput, make full use of processing resources, thereby improving garbage collection efficiency, improving storage device utilization, and improving storage device performance and availability.

[0152] Figure 6 This application provides a schematic diagram of the structure of a data processing apparatus according to one embodiment. The apparatus includes:

[0153] The first determining module 601 is used to determine the first storage area and the second storage area in the storage device;

[0154] The first generation module 602 is used to divide at least one valid data segment in the first storage area into multiple data segments, and to generate request tasks corresponding to the multiple data segments respectively.

[0155] The first triggering module 603 is used to trigger the read requests corresponding to at least two untriggered request tasks in parallel according to the task execution order.

[0156] The second triggering module 604 is used to sequentially trigger the write requests corresponding to the multiple request tasks when the read request responses corresponding to the multiple request tasks belonging to the same valid data segment have ended.

[0157] In some embodiments, the first generation module may divide at least one valid data segment in the first storage area into multiple data segments and generate request tasks corresponding to the multiple data segments respectively, including: determining at least one valid data segment in the first storage area; dividing the at least one valid data segment into multiple data segments according to a predetermined size; generating a task list corresponding to the multiple data segments; the task list includes request tasks corresponding to the multiple data segments respectively.

[0158] In some embodiments, the device may further include:

[0159] The resource release module is used to release the resources occupied by a request task when the write request response for any triggered request task has ended.

[0160] In some embodiments, the first triggering module's parallel triggering of read requests corresponding to at least two untriggered request tasks may include: simultaneously sending the read requests corresponding to at least two untriggered request tasks to the storage device, so that the storage device can read the data segments corresponding to the read requests and write them into memory.

[0161] The second triggering module sequentially triggers write requests corresponding to multiple request tasks, including: sequentially sending the write requests corresponding to the multiple request tasks to the storage device, so that the storage device can read the data segments corresponding to the write requests from the memory and write them into the second storage area.

[0162] In some embodiments, the device may further include:

[0163] The state switching module is used to switch the first storage area to the garbage state when the request tasks corresponding to multiple data segments are successfully processed, so as to perform the erase operation.

[0164] In some embodiments, the second triggering module sequentially triggers the write requests corresponding to the at least two request tasks, including: triggering the write request corresponding to the next triggered request task when the write request response for any triggered request task has ended.

[0165] In some embodiments, the first triggering module may trigger the read requests corresponding to the untriggered request tasks in parallel according to the task execution order. This may involve triggering the read requests corresponding to at least two untriggered request tasks in parallel according to the number of parallel requests and the task execution order.

[0166] The first triggering module can also be used to trigger a read request for the next untriggered request task when the write request response for any triggered request task has ended.

[0167] In some embodiments, the device may further include:

[0168] The third triggering module is used to trigger the read request corresponding to the next untriggered request task when the read request response of any triggered request task has ended.

[0169] If the read request response of any triggered request task ends, the read request corresponding to the next untriggered request task can be triggered as follows: if the read request response of any triggered request task ends, and the number of request tasks with triggered read requests and untriggered write requests is less than a preset value, then the read request corresponding to the next untriggered request task can be triggered.

[0170] In some embodiments, the first triggering module triggers the read requests corresponding to at least two untriggered request tasks in parallel according to the task execution order, including: triggering the read requests corresponding to at least two untriggered request tasks in parallel according to the number of parallel requests and the task execution order;

[0171] The device may also include:

[0172] The data statistics module is used to count the number of currently triggered read request tasks; and update the number of request tasks when the write request response corresponding to any triggered request task ends.

[0173] The data detection module is used to detect whether the number of read request tasks has reached the number of parallel requests; if the number of read request tasks has not reached the number of parallel requests, the next read request corresponding to the untriggered request task is triggered according to the task execution order.

[0174] Optionally, the data statistics module can also count the number of currently triggered write request tasks; and update the number of write request tasks when the write request response for any triggered request task has ended.

[0175] The data detection module can also detect whether the number of write request tasks is equal to zero; if the number of write request tasks is equal to zero, the next write request corresponding to the triggered request task will be triggered according to the task execution order.

[0176] Figure 6 The data processing device can perform Figure 1 , Figure 2 ,or Figure 4 The implementation principle and technical effects of the data processing method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the data processing device in the above embodiments performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0177] This application also provides a computing device, such as... Figure 7 As shown, the device may include a storage component 701 and a processing component 702;

[0178] The storage component 701 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component to achieve, for example... Figure 1 or Figure 2 or Figure 4 The data processing method described in the illustrated embodiment.

[0179] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0180] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0181] The processing component 702 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0182] Storage component 701 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0183] It should be noted that the aforementioned computing devices can be physical devices or elastic computing hosts provided by cloud computing platforms. They can be implemented as a distributed cluster of multiple servers or terminal devices, or as a single server or a single terminal device.

[0184] It should be noted that the above-mentioned computing devices implement Figure 1 or Figure 2 or Figure 4 In the case of the data processing method described in the illustrated embodiment, it can be a physical device or an elastic computing host provided by a cloud computing platform. It can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device.

[0185] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 or Figure 2 or Figure 4 The data processing method described in the illustrated embodiment. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0186] This application also provides a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program, when executed by a computer, can perform the above-described functions. Figure 1 or Figure 2 or Figure 4 The data processing method described in the illustrated embodiment. In such an embodiment, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the various functions defined in the system of this application.

[0187] In the foregoing embodiments, the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0188] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Determine the first storage region and the second storage region in the storage device; Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments; According to the task execution order, the read requests corresponding to at least two untriggered request tasks are triggered in parallel. The read request is used to read the corresponding data segment and write it into memory; the at least two request tasks correspond to one or more valid data segments; When the read request responses corresponding to multiple request tasks belonging to the same valid data segment have ended, the write requests corresponding to the multiple request tasks are triggered sequentially; the write requests are used to read the corresponding data segments from the memory and write them into the second storage area.

2. The method according to claim 1, characterized in that, The sequential triggering of the write requests corresponding to the plurality of request tasks includes: If the write request response for any triggered request task has ended, then the write request for the next triggered request task will be triggered.

3. The method according to claim 2, characterized in that, The step of triggering read requests corresponding to at least two untriggered request tasks in parallel according to the task execution order includes: Based on the number of parallel requests and the order of task execution, the read requests corresponding to at least two untriggered request tasks are triggered in parallel. The method further includes: If the write request response for any triggered request task has ended, then the read request for the next untriggered request task will be triggered.

4. The method according to claim 1, characterized in that, Also includes: If the read request response for any triggered request task ends, then the read request for the next untriggered request task will be triggered.

5. The method according to claim 4, characterized in that, The step of triggering a read request for the next untriggered request task after the read request response for any triggered request task has ended includes: If the read request response for any triggered request task ends, and the number of request tasks that have triggered read requests and have not triggered write requests is less than a preset value, then the read request for the next untriggered request task will be triggered.

6. The method according to claim 1, characterized in that, The step of triggering read requests corresponding to at least two untriggered request tasks in parallel according to the task execution order includes: Based on the number of parallel requests and the order of task execution, the read requests corresponding to at least two untriggered request tasks are triggered in parallel. The method further includes: Count the number of currently triggered read request tasks; If the read request response for any triggered request task ends, update the number of read request tasks. Detect whether the number of read request tasks has reached the number of parallel requests; If the number of read request tasks has not reached the number of parallel requests, the next read request corresponding to the untriggered request task will be triggered according to the task execution order.

7. The method according to claim 1, characterized in that, Also includes: Count the number of write request tasks that have been triggered. If the write request response for any triggered request task ends, update the number of write request tasks. Check if the number of write request tasks is equal to zero; If the number of write request tasks is zero, the next write request corresponding to the already triggered request task will be triggered according to the task execution order.

8. The method according to claim 1, characterized in that, Also includes: When the write request response for any triggered request task ends, the resources occupied by the request task are released.

9. The method according to claim 1, characterized in that, The determination of the first storage region and the second storage region in the storage device includes: Identify the first storage area where the proportion of invalid data reaches a predetermined proportion; Identify the second storage area that is currently idle.

10. The method according to claim 1, characterized in that, The step of dividing at least one valid data segment in the first storage area into multiple data segments and generating request tasks corresponding to each of the multiple data segments includes: Identify at least one valid data segment in the first storage area; The at least one valid data segment is divided into multiple data segments according to a predetermined size. Generate a task list corresponding to the multiple data segments; the task list includes the request tasks corresponding to each of the multiple data segments.

11. The method according to claim 1, characterized in that, Also includes: If the request tasks corresponding to the multiple data segments are successfully processed, the first storage area is switched to a garbage state in order to perform an erasure operation.

12. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the data processing method as described in any one of claims 1 to 11.

13. A computer storage medium, characterized in that, The device contains a computer program that, when executed by a computer, implements the data processing method as described in any one of claims 1 to 11.

14. A computer program product, characterized in that, It includes a computer program / instruction, which, when executed by a computer, implements the data processing method as described in any one of claims 1 to 11.

Images