Task processing method and device, program product and storage medium

Abstract

The invention provides a task processing method and device, a program product and a storage medium device, the method is applied to a main thread of a storage engine, the storage engine further comprises an auxiliary thread, the storage engine is used for accessing a memory based on partition namespace, and the memory comprises at least one partition; the method comprises the following steps: in response to a received data read-write request, executing a data read-write task aiming at the data read-write request according to metadata related to a partition in the memory; in the process of executing the data read-write task, if a partition recovery task of the memory is triggered, processing a first type of subtasks related to the metadata in the partition recovery task, and submitting a second type of subtasks, which are not related to the metadata, in the partition recovery task to the auxiliary thread for processing. According to the embodiment, the processing efficiency of the storage engine on the data read-write task can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of cloud computing technology, and in particular to task processing methods, devices, program products and storage media. Background Technology

[0002] SSD (Solid State Disk) is a type of computer storage that primarily uses NAND flash memory as its permanent storage. Currently, some SSDs employ an interface technology specification based on ZNS (Zoned Namespace). In ZNS technology, the memory's address space is abstracted into contiguous logical partitions (Zones), and this partition information is exposed to upper-layer applications, allowing applications to partition storage according to their specific projects.

[0003] The user-space storage engine built on ZNS SSD provides foreground read / write functionality for user data. Data is written to storage partitions, and a partition reclamation process needs to be implemented to release partitions for user use in a timely manner. In existing technologies, foreground data read / write tasks and partition reclamation tasks are executed in the same thread, which leads to competition for hardware resources. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides task processing methods, devices, systems, program products, and storage media.

[0005] According to a first aspect of the embodiments of this specification, a task processing method is provided, the method being applied in the main thread of a storage engine, the storage engine further comprising auxiliary threads, the storage engine being used to access a partition-based namespace-based memory, the memory comprising at least one partition; the method comprising:

[0006] In response to receiving a data read / write request, the system performs a data read / write task for the data read / write request based on the partition-related metadata in the memory.

[0007] During the execution of the data read / write task, if the partition reclamation task of the memory is triggered, the first type of subtasks related to the metadata in the partition reclamation task are processed, and the second type of subtasks not related to the metadata in the partition reclamation task are submitted to the auxiliary thread for processing.

[0008] According to a second aspect of the embodiments of this specification, a task processing method is provided, the method being applied to an auxiliary thread of a storage engine, the storage engine further comprising a main thread, the storage engine being used to access a partition-based namespace-based memory, the memory comprising at least one partition; the method comprising:

[0009] The main thread receives a second type of subtask submitted by the main thread. The main thread, upon receiving a data read / write request, executes a data read / write task for the data read / write request based on the metadata related to the partition in the memory. During the execution of the data read / write task, if the partition reclamation task of the memory is triggered, the main thread processes the first type of subtask related to the metadata in the partition reclamation task and submits the second type of subtask unrelated to the metadata in the partition reclamation task to the auxiliary thread.

[0010] Process the second type of subtask submitted by the main thread.

[0011] According to a third aspect of the embodiments of this specification, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method embodiments described in the first or second aspects above.

[0012] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method embodiments described in the first or second aspect above.

[0013] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method embodiments described in the first or second aspect above.

[0014] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0015] In this embodiment of the specification, for a storage engine based on Partition Storage (ZNS), the task processing method introduces an auxiliary thread and splits the partition reclamation task. For the main thread, upon receiving a data read / write request, it executes a data read / write task based on the partition-related metadata in the storage. During the execution of the data read / write task by the main thread, in response to the triggering of a partition reclamation task, the main thread processes the first type of subtasks related to the metadata within the partition reclamation task, and submits the second type of subtasks unrelated to the metadata to the auxiliary thread for processing. Therefore, this embodiment can significantly reduce the performance impact of partition reclamation task processing on data read / write tasks and improve the storage engine's processing efficiency for data read / write tasks.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this specification and, together with the specification, serve to explain the principles of this disclosure.

[0018] Figure 1 This is a schematic diagram of a Zone according to an exemplary embodiment of this specification.

[0019] Figure 2A This is a flowchart illustrating a task processing method according to an exemplary embodiment of this specification.

[0020] Figure 2B This is a flowchart illustrating another task processing method according to an exemplary embodiment of this specification.

[0021] Figure 2C This is a flowchart illustrating another task processing method according to an exemplary embodiment of this specification.

[0022] Figure 3 This specification is a hardware structure diagram of a computer device containing a task processing apparatus, according to an exemplary embodiment. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0024] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0027] Solid-state drives (SSDs) consist of a control unit and storage units. The storage units contain flash memory chips or DRAM chips (Dynamic Random Access Memory), and they are widely used in many fields. Compared to hard disk drives (HDDs), SSDs offer higher read and write speeds, but are also more expensive. Currently, in some scenarios, service providers also use user-space storage engines based on ZNS SSDs to provide storage services to users. A user-space storage engine is a storage system that runs in user space rather than kernel space. It can directly access hardware resources without the intervention of the traditional operating system kernel layer.

[0028] The ZNS SSD-based storage engine is suitable for applications requiring a large number of sequential writes and a small number of random writes, enabling high-speed data access. In providing storage services to users, the ZNS SSD-based storage engine needs to provide data read and write functionality to the user in the foreground.

[0029] The storage engine allows external users to read and write data. The granularity of data read and write operations that the storage engine passes to users can be called a data block (Chunk), while the granularity of the storage engine managing ZNS SSDs is a partition (Zone). The size of a Chunk and the size of a Zone can be different. The foreground can write to the storage engine and can also read the contents of a written Chunk. In addition to providing Chunk read and write functionality, the storage engine also needs to support Chunk deletion functionality.

[0030] Because ZNS SSDs only support append writes, the storage engine needs to introduce a Recycle function to support functions such as releasing space after data deletion and Zone space resource management while providing foreground read and write capabilities. By properly managing the storage Zones, Zones can be released in a timely manner for users to use.

[0031] The ZNS SSD interface to the storage engine abstracts the storage device address space into a series of contiguous logical partitions (Zones). Its operational characteristics are as follows: a partition needs to be opened before it can be written to, becoming an open / active partition; logical pages within an open partition can only be written sequentially and cannot be updated in place; if data needs to be updated, the entire partition must be reset before writing can begin again from the beginning.

[0032] For example, such as Figure 1 As shown, in ZNS SSDs, the Logical Block Addressing (LBA) is divided into multiple zone granularities according to the configuration. The figure shows X partitions, from Zone1 to ZoneX. In ZNS SSDs, the state of a zone can include several types:

[0033] This indicates the free state of a Zone where no data has been written.

[0034] This indicates that the Zone is in an open state, meaning it can be written to.

[0035] This indicates that the Zone is full and can only support a read-only Closed state.

[0036] This indicates that the data in the Zone is in a garbage state, meaning it contains invalid data.

[0037] The state transition process for a Zone can be:

[0038] A Zone in the Free state can be transformed into the Open state through the OpenZone operation;

[0039] Within an Open Zone, data can only be appended to and written, with random reads supported; that is, data can only be written sequentially to the Zone and cannot be updated in-place. Within each Zone, data writing must follow a sequential appending method; existing data cannot be randomly overwritten. After each sequential write operation, a "Write Pointer" is created to record the LBA address of the last written data within the Zone.

[0040] After writing to an Open Zone, it is converted to a Closed Zone through the CloseZone operation. A Closed Zone is read-only.

[0041] After the Recycle operation, a Zone in the Closed state is transformed into a Garbage state after the valid data within the Zone is migrated out.

[0042] After a Zone in Garbage state is reset by the WritePtr operation via Reset, it transitions to Free state.

[0043] Storage engines built on high-speed ZNS SSD media typically employ a RunToCompletion (RTC) thread model to provide high-speed data read and write capabilities, and because ZNS SSDs require data to be written sequentially within each zone, they generally use this model. The core idea is that each thread is responsible for one I / O (Input / Output) task, and this thread executes the task continuously until it is completed—that is, "run to completion"—without being interrupted by other tasks. This means that even if other tasks are waiting in the queue, the current task will continue to occupy CPU resources until it completes. Only after the current task is completed will the next task be executed. This model avoids the context switching overhead during task execution because all operations of the same task are executed sequentially by the same thread, thus reducing lock contention and context switching problems common in multithreading.

[0044] Currently, user-space storage engines using the RTC thread model process for both foreground I / O tasks and background tasks (such as Recycle tasks) within a single I / O thread. Since switching between foreground and background tasks within this single I / O thread incurs time overhead, some solutions allow background tasks to execute as user-space coroutines within the I / O thread, replacing heavy thread switching with lightweight coroutine switching.

[0045] However, in the RTC thread model, it is inevitable that CPU resources and disk I / O resources will be contested with foreground I / O tasks, which will have two effects:

[0046] It affects the average latency and long-tail latency of foreground I / O tasks, which in turn affects the storage engine's foreground IOPS (Input / Output Operations Per Second) capability;

[0047] When the foreground traffic is high, the throughput of Recycle I / O is suppressed and cannot reach the threshold calculated by the system's rate limiting strategy, thus reducing the efficiency of Zone space resource release.

[0048] Based on actual testing, the runtime trends of foreground IOPS under different Recycle throughput are as follows:

[0049] 1. Without recycle tasks, the foreground write IOPS can reach 180,000.

[0050] 2. After the Recycle throughput reaches 320 megabits per second, the IOPS drops to a minimum of 130,000. If the Recycle throughput is further increased, the average IOPS will remain at around 130,000.

[0051] 3. Under this foreground pressure, the Recycle runtime bandwidth reached a maximum of 380 Mbps, and the Recycle throughput limit after 400 Mbps did not take effect;

[0052] 4. After 120 terabytes of Recycle throughput, the fluctuation of foreground IOPS increased significantly. This is because the average write latency and long tail increase as the throughput of Recycle tasks increases.

[0053] Based on this, the embodiments of this specification provide a task processing scheme, which introduces an auxiliary thread and splits the partition recycling task Recycle. Some subtasks are unloaded from the main thread to the auxiliary thread, thereby reducing the impact of the Recycle task on the foreground read and write tasks. The embodiments of this specification will be described in detail below.

[0054] like Figure 2A As shown, Figure 2A This is a flowchart illustrating a task processing method according to an exemplary embodiment, comprising the following steps:

[0055] In step 202, in response to receiving a data read / write request for a partition in the memory, a data read / write task is executed for the data read / write request based on the metadata related to the partition in the memory.

[0056] In step 204, during the execution of the data read / write task, if the partition reclamation task of the memory is triggered, the first type of subtask related to the metadata in the partition reclamation task is processed, and the second type of subtask unrelated to the metadata in the partition reclamation task is submitted to the auxiliary thread for processing.

[0057] The method in this embodiment can be applied to the main thread of a storage engine based on ZNS SSD, which can be used to access memory based on partition namespaces.

[0058] As an example, the storage engine in this embodiment can be a user-space storage engine, such as a user-space standalone storage engine. A standalone storage engine refers to a functional node in a distributed storage system responsible for managing the storage of user data on the storage medium. Generally, a single computer device (such as a storage server) can run one standalone storage engine process, responsible for managing one or more storage devices on the computer device.

[0059] In some examples, a storage engine process may run in a computer device, and the storage engine process may include one or more threads; for example, a computer device may include multiple memories, and each memory may correspond to a main thread, with each main thread responsible for handling the data read and write tasks of the corresponding memory.

[0060] In some examples, the storage can be one or more ZNS-based solid-state drives (SSDs), each divided into multiple zones. As an example, a zone can support append writes, random reads, and other operations, but the data within the zone must be completely erased; partial erasure and overwriting are not allowed. Therefore, if reclaimable data exists within a zone, it cannot be erased, preventing the zone from being released and thus unusable, resulting in idle and wasted space.

[0061] In practical applications, the data read / write requests in step 202 can be initiated by various requesters, including but not limited to other applications on the computer device where the storage engine resides, other servers on the network, or virtual machines on the computer device. As an example, in a cloud computing scenario, the data read / write request can be initiated by a user's virtual machine. The machine where the user's virtual machine resides can be the same as or different from the computer device where the storage engine in this embodiment resides. Furthermore, a storage device can be used by a single user or shared by multiple tenants; this embodiment does not impose any limitations on this.

[0062] When the storage engine receives a data read / write request for a partition in the memory, the request is first processed by the main thread. The storage engine can manage each zone through the metadata of the zones in the memory. The metadata related to a zone can include the zone's own metadata, including but not limited to a combination of one or more of the following: zone status information, zone address, write pointer, used space or erase count, etc., which are not limited in this embodiment.

[0063] In other examples, the Zone-related metadata in step 202 may also include other types of metadata, such as metadata related to the data stored in the Zone. For instance, when a storage engine provides data storage services to a caller, it may provide services at a different management granularity than that of a Zone. In this embodiment, this granularity is referred to as a Chunk. As an example, the size of a Chunk can be larger than the size of a Zone. Therefore, a data read / write request can be a request for one or more Chunks; the storage engine can also combine the Chunk's metadata to perform data read / write tasks. The Chunk's metadata may include, but is not limited to, a combination of one or more of the following information: the Chunk's address, timestamp, owner, or permissions, etc., which are not limited in this embodiment.

[0064] The main thread needs to rely on metadata to perform data read and write tasks. For example, when processing data read and write requests, the main thread needs to query relevant metadata and then perform specific read and write operations based on this information. These operations may include partition location, write pointer management, partition status tracking, data integrity checks, or chunk permission checks. After completing the read and write operations, the metadata may also need to be updated.

[0065] Since the main thread needs to rely on metadata to perform data read and write tasks, it needs to manage the metadata. This can be achieved through locking mechanisms or other means to ensure the consistency of the metadata and prevent concurrent operations.

[0066] As mentioned in the previous embodiments, in related technologies, during the main thread's data read / write tasks, a partition reclamation task is also required for the partitions in the memory. This embodiment designs an auxiliary thread to assist in completing the partition reclamation task. The storage engine can contain one or more auxiliary threads. The timing of the auxiliary thread's creation is not limited in this embodiment; it can be created after the storage engine starts or after the partition reclamation task is triggered. The auxiliary thread's role is to cooperate with the main thread to complete some sub-tasks in the partition reclamation task, which refers to erasing garbage partitions for reuse. Since the main thread manages metadata, the partition reclamation task may involve metadata-related processing. Therefore, this embodiment breaks down the partition reclamation task into two types of sub-tasks:

[0067] ① The first type of subtasks related to the metadata; since the main thread manages the metadata, the first type of subtasks are still executed by the main thread. If they are assigned to auxiliary threads, it means that the main thread and auxiliary threads need to access shared resources, which could lead to potential thread safety issues such as data inconsistency, race conditions, and deadlocks due to multiple threads accessing shared resources simultaneously. Therefore, this embodiment can avoid the thread safety issues introduced by multi-threaded access.

[0068] ② A second type of subtask that is not related to the metadata; this type of subtask is submitted by the main thread to the auxiliary thread for processing. Therefore, when the auxiliary thread processes the second type of subtask, it will not affect the data read and write tasks being processed by the main thread, and at the same time, it can reduce the load on the main thread.

[0069] In step 204, the triggering of the partition reclamation task can be determined by the storage engine through the set reclamation strategy. For example, it can be determined based on the usage of partitions in the memory. For example, it can be determined whether to trigger the reclamation of certain garbage partitions based on the proportion of used partitions in the memory to the total number of partitions in the memory. This embodiment does not limit this.

[0070] In some examples, the process of determining whether to trigger a partition reclamation task can be executed by certain threads within the storage engine. For instance, one thread might determine whether each storage device needs to trigger a partition reclamation task, and then notify the main thread for that storage device. In other examples, the main thread can execute the process, with each main thread independently determining whether its corresponding storage device needs to trigger a partition reclamation task.

[0071] In some examples, the storage engine runs on a computer device, which may include a multi-core processor; the main thread and the auxiliary thread run on different processor cores.

[0072] In this embodiment, the multi-core processor of the computer device can be multiple cores on a single processor. These cores can share some processor resources, such as cache memory and input / output interfaces, but each core has its own execution unit and control logic. Most existing central processing units (CPUs) adopt this design, such as dual-core, quad-core, octa-core, or even more core CPUs. In other examples, it can also be multiple cores on multiple processors. For instance, in some scenarios, a computer device can be configured with multiple physical processors, each of which can include one or more cores.

[0073] In this embodiment, by running the main thread and the auxiliary thread on different processor cores, the execution of the auxiliary thread will not compete for the processor resources of the main thread, thereby improving the efficiency of the main thread in processing data read and write tasks.

[0074] In practice, specific APIs or tools can be used to bind different threads to different CPU cores for execution. For example, in Linux systems, functions such as `pthread_setaffinity_np` can be used to bind threads to a specific CPU core. In the storage engine implementation, the corresponding thread binding function can be called for the started thread as needed to bind the thread to a specific CPU core. In practice, other methods can also be used to bind specific threads to specific CPU cores; this embodiment does not limit this approach.

[0075] In some examples, a computer device may have multiple memories, and the memory engine may include a main thread corresponding to each memory, and the number of processor cores in the computer device is greater than the number of main threads;

[0076] Each memory location corresponds to a main thread that runs on a different processor core.

[0077] The main thread is associated with one or more auxiliary threads; submitting the second type of subtasks in the partition reclamation task that are not related to the metadata to the auxiliary threads for processing includes:

[0078] Submit the second type of subtasks in the partition reclamation task that are not related to the metadata to the associated auxiliary thread for processing.

[0079] In this embodiment, some computer devices may have multiple processor cores, and the number of processor cores is greater than the number of memory. When the computer device includes multiple memory, the storage engine can allocate a main thread to each memory. Based on this, this embodiment designs that the main thread of each memory runs on a different processor core. This one-to-one allocation can reduce resource competition between different main threads on processor cores and improve the processing speed of read and write tasks of each memory.

[0080] Furthermore, after allocating a corresponding processor core to each main thread, the remaining processor cores can be used in whole or in part to run auxiliary threads. In practical applications, the number of auxiliary threads can be flexibly configured based on the number of processor cores and the number of main threads; this embodiment does not impose any limitations on this.

[0081] In some examples, when there are enough processor cores, each main thread can be associated with a corresponding auxiliary thread, and these auxiliary threads, as well as the auxiliary threads and the main threads, can run on different processing cores. In other examples, multiple (at least two) main threads can be associated with the same auxiliary thread. Therefore, the main thread can submit second-type subtasks in the partition reclamation task that are not related to the metadata to the associated auxiliary thread for processing.

[0082] In some examples, the auxiliary thread can have a task queue, and the main thread can submit the second type of subtask to the task queue corresponding to the auxiliary thread. As an example, the task queue can be a first-in, first-out queue, and the auxiliary thread can retrieve the second type of subtask from its corresponding task queue and process it.

[0083] In some examples, there can be multiple auxiliary threads, each corresponding to a task queue. Submitting the second type of subtasks unrelated to the metadata in the partition reclamation task to the auxiliary threads for processing includes:

[0084] The priority of each task queue is determined based on the number of second-type subtasks to be executed in each task queue and / or the resource utilization of the processor core where the auxiliary thread corresponding to the task queue resides. The priority is negatively correlated with the number of subtasks and negatively correlated with the resource utilization.

[0085] The second type of subtasks in the partition reclamation task that are not related to the metadata are submitted to the highest priority task queue. These second type of subtasks are then processed by the auxiliary thread corresponding to that task queue.

[0086] In this embodiment, each auxiliary thread has a corresponding task queue. One auxiliary thread can provide partition recycling services for one or more main threads. The main thread can use multiple auxiliary threads, and the main thread can submit the second type of subtasks to any task queue. The task queue stores the second type of subtasks submitted by the main thread, and the auxiliary thread can retrieve the second type of subtasks from its corresponding task queue and process them.

[0087] The priority of the task queue can be dynamically adjusted based on the number of second-type subtasks to be executed in the queue and the resource utilization of the processor core where the auxiliary thread resides. The more tasks there are or the higher the resource utilization, the lower the priority of the task queue. The main thread can submit second-type subtasks to be executed to the highest-priority task queue. The highest-priority task queue means that it currently has fewer tasks and the corresponding processor core resource utilization is low, so it may have more computing resources to process new second-type subtasks.

[0088] Based on this, this embodiment can dynamically allocate new second-type subtasks to the most suitable task queue according to the priority of the task queue. This allows for more efficient use of processor resources and avoids situations where some cores are overloaded while others are idle.

[0089] In some examples, submitting the second type of subtask in the partition reclamation task that is not related to the metadata to an auxiliary thread for processing includes:

[0090] After determining the second type of subtask in the partition reclamation task that is not related to the metadata, and after determining the partition to be reclaimed and the free partition in the memory, the second type of subtask carrying the identifier of the partition to be reclaimed and the identifier of the free partition is submitted to the auxiliary thread;

[0091] After receiving the message that the second type of subtask has been successfully processed, the metadata of the partition to be reclaimed and the metadata of the free partition are updated. The message that the second type of subtask has been successfully processed is sent by the auxiliary thread to the main thread after updating the status of the partition to be reclaimed to the closed state, updating the free partition to the open state, writing the valid data in the partition to be reclaimed to the free partition, and performing an erase operation on the partition to be reclaimed, based on the identifier carried in the received second type of subtask.

[0092] In this embodiment, the main thread can construct a second type of subtask and submit it to the auxiliary thread. For example, when a memory partition reclamation task is triggered, the main thread can determine the partition to be reclaimed (SourceZone) and a free partition (DstZone) that needs to have valid data reclaimed. The main thread can construct a second type of subtask containing the identifiers of the partition to be reclaimed and the free partition and submit it to the auxiliary thread, so that the auxiliary thread can receive and process the second type of subtask.

[0093] In this embodiment, because the partition reclamation task is split into multiple subtasks, with the main thread executing the subtask to determine the partitions to be reclaimed and the free partitions, while the subtasks that interact with the memory and consume CPU resources are executed by auxiliary threads, the load on the main thread is reduced. Furthermore, these second-type subtasks handled by the auxiliary threads do not raise multi-threading safety issues.

[0094] In some examples, the storage engine divides the storage space of the memory into data blocks; the storage engine provides storage services based on the granularity of the data blocks, the data read / write request is a data read / write request for the data block, the partition stores data block fragments of the data block; the main thread is also responsible for managing the metadata of the data block fragments;

[0095] The message indicating successful processing of the second type of subtask is sent by the auxiliary thread after writing the valid data in the partition to be reclaimed to the free partition, updating the metadata of the data block fragment corresponding to the valid data in the free partition, and then performing an erase operation on the partition to be reclaimed after the update is successful.

[0096] In this embodiment, the data block segments (ChunkSegment) stored by the partition can also have their own metadata. The main thread does not need to rely on the metadata of the ChunkSegment when processing data read and write tasks. Therefore, before the Recycle task ends, the data block segments ChunkSegment on DstZone can be exclusively occupied by the partition recycling task. Thus, the update of the metadata of the ChunkSegment can also be offloaded to the auxiliary thread.

[0097] In the RTC thread model, the introduction of auxiliary threads also raises the issue of sharing runtime resources. In some examples, the memory region corresponding to the main thread is isolated from the memory region of the auxiliary thread. In this embodiment, the memory resources used by the Recycle task can also be isolated separately from the memory resources of the main thread. As an example, after the auxiliary thread starts, a corresponding memory storage region can be allocated to the auxiliary thread. This memory storage region is different from the memory storage region allocated to the main thread. The two memory storage regions can be isolated from each other. The memory storage region can store the resources needed by the thread when processing the task. In this way, the two are isolated from each other, which can avoid the mutual exclusion / thread switching overhead caused by the main thread and auxiliary thread accessing resources at the same time.

[0098] like Figure 2B The figure shown is a schematic diagram illustrating another task processing method according to an exemplary embodiment of this specification.

[0099] 1. The main thread can create Recycle tasks and submit them to the auxiliary thread RecycleWorker;

[0100] 2. Based on the Recycle task, the RecycleWorker closes a partition (Close Zone) or opens a partition (Open Zone) as needed. That is, it updates the Source Zone to the closed state and updates the DstZone (which was in a free state) to the Open state. Alternatively, it closes the Old DstZone (used for writing data in the previous Recycle task) and opens the New DstZone (a free partition that needs to be written data in the current Recycle task).

[0101] 3. RecycleWorker prepares Recycle I / O Tasks according to the configured I / OSize;

[0102] 4. The RecycleWorker performs the actual Recycle I / O, reading data from the Source Zone and writing it to the DstZone; it also erases the SourceZone.

[0103] 5. When a ChunkSegment Recycle in the Source Zone ends, the RecycleWorker updates its metadata;

[0104] 6. Once the recycling of valid data in the SourceZone is complete, the RecycleWorker notifies the main I / O thread that this round of recycling tasks has ended;

[0105] 7. The main I / O thread updates the Zone / Chunk metadata based on the results of the Recycle task;

[0106] Of the steps above, only steps 1 and 7 are executed in the main I / O thread. While RecycleWoker is processing Recycle tasks, the main I / O thread can handle the foreground I / O process normally.

[0107] In this embodiment, the subtasks unloaded to the auxiliary thread include:

[0108] The logic for interacting with the hardware, including closing / opening the DstZone and actual I / O;

[0109] Recycle I / O Tasks prepare CPU-intensive tasks;

[0110] Before the Recycle task ends, the ChunkSegment on DstZone can be exclusively accessed by the Recycle thread, and its corresponding metadata updates can also be offloaded to the Worker thread.

[0111] The subtasks retained in the main thread are mainly the subprocesses of selecting SourceZone and updating data block metadata ChunkMeta / partition metadata ZoneMeta after the Recycle task is completed. This is done mainly to avoid thread safety issues introduced by multi-threaded access.

[0112] like Figure 2C As shown, if the solution in this embodiment is not used, the partition recycling task is executed in the main thread. Through Figure 2B and Figure 2CThe differences between this embodiment and traditional solutions are evident. Compared to existing Recycle processes, this approach significantly reduces the impact of Recycle tasks on the average I / O latency and long-tail latency of foreground read / write tasks, and also significantly improves the foreground IOPS capability of the storage engine.

[0113] In the following test scenario:

[0114] 1. I / O size 4160, 4QD Write, Recycle throughput limits = 200M, 400M, 600M, 800M

[0115] 2. I / O size 4160, 6QD Read, Recycle throughput limits = 200M, 400M, 600M, 800M

[0116] Here, I / O Size refers to the physical block size of each input / output operation. In this example, the physical block size of each operation is 4160 bytes.

[0117] QD (Queue Depth) refers to the number of read and write operations that can be performed simultaneously. 4QD Write means that 4 write operations can wait in the queue for execution at the same time, while 6QD Read means that 6 read operations can wait in the queue for execution at the same time.

[0118] In the Recycle throughput limit, M refers to MB / s, which is megabytes per second.

[0119] In the first test scenario, this embodiment can result in: a 5% to 38% increase in I / OPS; a 30% reduction in average write latency; and a 55% reduction in maximum write latency.

[0120] In the second test scenario, this embodiment can result in: a 6% to 35% increase in I / OPS; a 25% reduction in average read latency; and a 30% reduction in maximum read latency.

[0121] As can be seen from the above embodiments, this embodiment proposes an optimization for offloading the Recycle process in the ZNS SSD user-space storage engine. In this embodiment, for sub-processes with high overhead in the Recycle process, such as Recycle I / O, they are distributed to auxiliary worker threads that specifically handle Recycle tasks, thereby offloading this overhead from the main thread. Therefore, the impact of the Recycle process on the foreground read and write functions can be reduced.

[0122] By introducing a dedicated RecycleWorker thread to handle Recycle tasks, and by splitting Recycle tasks and offloading more expensive subtasks to the RecycleWorker thread for execution, resource contention between Recycle tasks and foreground I / O tasks is avoided. Therefore, the impact of the Recycle process on the average latency of foreground I / O is significantly reduced, the long-tail latency impact of the Recycle process on foreground I / O is significantly reduced, and the foreground IOPS capability of the storage engine is significantly improved.

[0123] Corresponding to the aforementioned task processing method embodiments, this specification also provides another task processing method, which is applied to an auxiliary thread of a storage engine, the storage engine further including a main thread, the storage engine being used to access memory based on a partition namespace, the memory including at least one partition; the method includes:

[0124] The main thread receives a second type of subtask submitted by the main thread. The main thread, upon receiving a data read / write request, executes a data read / write task for the data read / write request based on the metadata related to the partition in the memory. During the execution of the data read / write task, if the partition reclamation task of the memory is triggered, the main thread processes the first type of subtask related to the metadata in the partition reclamation task and submits the second type of subtask unrelated to the metadata in the partition reclamation task to the auxiliary thread.

[0125] Process the second type of subtask submitted by the main thread.

[0126] In some examples, the second type of subtask is a task submitted by the main thread after determining the second type of subtask that is not related to the metadata in the partition reclamation task, and after determining the partition to be reclaimed and the free partition in the memory, carrying the identifier of the partition to be reclaimed and the identifier of the free partition.

[0127] The processing of the second type of subtask submitted by the main thread includes:

[0128] Based on the identifier carried in the second type of subtask, the status of the partition to be reclaimed is updated to closed, the free partition is updated to open, the valid data in the partition to be reclaimed is written to the free partition, the partition to be reclaimed is erased, and the message that the second type of subtask has been successfully processed is sent to the main thread.

[0129] The task processing method in this embodiment can be found in the description of the foregoing embodiments, and is not limited thereto.

[0130] Corresponding to the embodiments of the aforementioned task processing method, this specification also provides embodiments of a task processing apparatus and the computer equipment on which it is applied.

[0131] The embodiments of the task processing device described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by its processor reading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 As shown, Figure 3 This is a hardware structure diagram of a computer device containing a task processing apparatus according to an exemplary embodiment, except... Figure 3 In addition to the processor 310, network interface 320, memory 330, and non-volatile memory 340 shown, the computer device in which the task processing device is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0132] A task processing apparatus, the apparatus being used in the main thread of a storage engine, the storage engine further including auxiliary threads, the storage engine being used to access partition-namespace-based memory, the memory including at least one partition; the apparatus comprising:

[0133] The data read / write processing module is configured to: in response to receiving a data read / write request, execute a data read / write task for the data read / write request based on the metadata related to the partition in the memory;

[0134] The partition reclamation processing module is used to: during the execution of the data read and write task, if the partition reclamation task of the memory is triggered, process the first type of subtasks related to the metadata in the partition reclamation task, and submit the second type of subtasks unrelated to the metadata in the partition reclamation task to the auxiliary thread for processing.

[0135] In some examples, the storage engine runs on a computer device that includes a multi-core processor; the main thread and the auxiliary thread run on different processor cores.

[0136] In some examples, there are multiple memories, the memory engine includes a main thread corresponding to each memory, and the number of processor cores in the computer device is greater than the number of main threads;

[0137] Each memory location corresponds to a main thread that runs on a different processor core.

[0138] The main thread is associated with one or more auxiliary threads; the partition reclamation processing module is used for:

[0139] Submit the second type of subtasks in the partition reclamation task that are not related to the metadata to the associated auxiliary thread for processing.

[0140] In some examples, there are multiple auxiliary threads, each corresponding to a task queue. The partition recycling processing module is used for:

[0141] The priority of each task queue is determined based on the number of second-type subtasks to be executed in each task queue and / or the resource utilization rate of the processor core where the auxiliary thread corresponding to the task queue is located; wherein the priority is negatively correlated with the number and the priority is negatively correlated with the resource utilization rate.

[0142] Submit the second type of subtasks in the partition reclamation task that are not related to the metadata to the highest priority task queue; wherein, the second type of subtasks in the task queue are processed by the auxiliary thread corresponding to the task queue.

[0143] In some examples, the partition reclamation processing module is used for:

[0144] After determining the second type of subtask in the partition reclamation task that is not related to the metadata, and after determining the partition to be reclaimed and the free partition in the memory, the second type of subtask carrying the identifier of the partition to be reclaimed and the identifier of the free partition is submitted to the auxiliary thread;

[0145] After receiving the message that the second type of subtask has been successfully processed, the metadata of the partition to be reclaimed and the metadata of the free partition are updated. The message that the second type of subtask has been successfully processed is sent by the auxiliary thread to the main thread after updating the status of the partition to be reclaimed to the closed state, updating the free partition to the open state, writing the valid data in the partition to be reclaimed to the free partition, and performing an erase operation on the partition to be reclaimed, based on the identifier carried in the received second type of subtask.

[0146] In some examples, the storage engine divides the storage space of the memory into data blocks; the storage engine provides storage services based on the granularity of the data blocks, the data read / write request is a data read / write request for the data block, the partition stores data block fragments of the data block; the main thread is also responsible for managing the metadata of the data block fragments;

[0147] The message indicating successful processing of the second type of subtask is sent by the auxiliary thread after writing the valid data in the partition to be reclaimed to the free partition, updating the metadata of the data block fragment corresponding to the valid data in the free partition, and then performing an erase operation on the partition to be reclaimed after the update is successful.

[0148] In some examples, the memory region corresponding to the main thread is isolated from the memory region of the auxiliary thread.

[0149] The specific implementation process of the functions and roles of each module in the above task processing device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0150] This specification also provides another task processing apparatus in its embodiments, the apparatus being applied to an auxiliary thread of a storage engine, the storage engine further including a main thread, the storage engine being used to access partition-based namespace-based memory, the memory including at least one partition; the apparatus comprising:

[0151] The receiving module is used to receive the second type of subtasks submitted by the main thread; the main thread, upon receiving a data read / write request, executes a data read / write task for the data read / write request based on the metadata related to the partition in the memory, and during the execution of the data read / write task, if the partition reclamation task of the memory is triggered, processes the first type of subtasks related to the metadata in the partition reclamation task, and submits the second type of subtasks unrelated to the metadata in the partition reclamation task to the auxiliary thread;

[0152] The processing module is used to process the second type of subtasks submitted by the main thread.

[0153] In some examples, the second type of subtask is a task submitted by the main thread after determining the second type of subtask that is not related to the metadata in the partition reclamation task, and after determining the partition to be reclaimed and the free partition in the memory, carrying the identifier of the partition to be reclaimed and the identifier of the free partition.

[0154] The processing module is used for:

[0155] Based on the identifier carried in the second type of subtask, the status of the partition to be reclaimed is updated to closed, the free partition is updated to open, the valid data in the partition to be reclaimed is written to the free partition, the partition to be reclaimed is erased, and the message that the second type of subtask has been successfully processed is sent to the main thread.

[0156] The task processing method in this embodiment can be found in the description of the foregoing embodiments, and is not limited thereto.

[0157] Accordingly, this specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned task processing method embodiments.

[0158] Accordingly, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the task processing method embodiment.

[0159] Accordingly, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the task processing method embodiments.

[0160] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0161] The above embodiments can be applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0162] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0163] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0164] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0165] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0166] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0167] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0168] The terms "specific example" or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the embodiments or examples, which are included in at least one embodiment or example of this specification. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0169] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0170] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0171] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A task processing method, the method being applied to the main thread of a storage engine, the storage engine further comprising auxiliary threads, the storage engine being used to access partition-based namespace-based memory, the memory comprising at least one partition; the method comprising: In response to receiving a data read / write request, the system performs a data read / write task for the data read / write request based on the partition-related metadata in the memory. During the execution of the data read / write task, if the partition reclamation task of the memory is triggered, the first type of subtasks related to the metadata in the partition reclamation task are processed, and the second type of subtasks not related to the metadata in the partition reclamation task are submitted to the auxiliary thread for processing.

2. The method according to claim 1, wherein the storage engine runs on a computer device, the computer device including a multi-core processor; the main thread and the auxiliary thread run on different processor cores respectively.

3. The method according to claim 2, wherein there are multiple memories, the memory engine includes a main thread corresponding to each memory, and the number of processor cores in the computer device is greater than the number of main threads; Each memory location corresponds to a main thread that runs on a different processor core. The main thread is associated with one or more auxiliary threads; The step of submitting the second type of subtasks in the partition reclamation task that are not related to the metadata to the auxiliary thread for processing includes: The second type of subtasks in the partition reclamation task that are not related to the metadata are submitted to the associated auxiliary thread for processing.

4. The method according to claim 2, wherein there are multiple auxiliary threads, each auxiliary thread corresponding to a task queue, and the step of submitting the second type of subtasks in the partition reclamation task that are not related to the metadata to the auxiliary thread for processing includes: The priority of each task queue is determined based on the number of second-type subtasks to be executed in each task queue and / or the resource utilization of the processor core where the auxiliary thread corresponding to the task queue is located. Submit the second type of subtasks in the partition reclamation task that are not related to the metadata to the highest priority task queue; wherein, the second type of subtasks in the task queue are processed by the auxiliary thread corresponding to the task queue.

5. The method according to claim 1, wherein submitting the second type of subtask in the partition reclamation task that is not related to the metadata to the auxiliary thread for processing includes: After determining the second type of subtask in the partition reclamation task that is not related to the metadata, and after determining the partition to be reclaimed and the free partition in the memory, the second type of subtask carrying the identifier of the partition to be reclaimed and the identifier of the free partition is submitted to the auxiliary thread; After receiving the message that the second type of subtask has been successfully processed, the metadata of the partition to be reclaimed and the metadata of the free partition are updated. The message that the second type of subtask has been successfully processed is sent by the auxiliary thread to the main thread after updating the status of the partition to be reclaimed to the closed state, updating the free partition to the open state, writing the valid data in the partition to be reclaimed to the free partition, and performing an erase operation on the partition to be reclaimed, based on the identifier carried in the received second type of subtask.

6. The method according to claim 5, wherein the storage engine divides the storage space of the memory according to data blocks; the storage engine provides storage services based on the granularity of the data blocks; the data read / write request is a data read / write request for the data block; the partition stores data block fragments of the data block; and the main thread is also responsible for managing the metadata of the data block fragments. The message indicating successful processing of the second type of subtask is sent by the auxiliary thread after writing the valid data in the partition to be reclaimed to the free partition, updating the metadata of the data block fragment corresponding to the valid data in the free partition, and then performing an erase operation on the partition to be reclaimed after the update is successful.

7. The method according to claim 1, wherein the memory region corresponding to the main thread is isolated from the memory region of the auxiliary thread.

8. A task processing method, the method being applied to an auxiliary thread of a storage engine, the storage engine further comprising a main thread, the storage engine being used to access memory based on a partition namespace, the memory comprising at least one partition; the method comprising: Receive the second type of subtask submitted by the main thread; The main thread receives a data read / write request, executes a data read / write task for the request based on the metadata related to the partition in the memory, and if the partition reclamation task of the memory is triggered during the execution of the data read / write task, it processes the first type of subtask related to the metadata in the partition reclamation task, and submits the second type of subtask unrelated to the metadata in the partition reclamation task to the auxiliary thread. Process the second type of subtask submitted by the main thread.

9. The method according to claim 8, wherein the second type of subtask is a task carrying the identifier of the partition to be reclaimed and the identifier of the free partition submitted by the main thread after determining the second type of subtask that is not related to the metadata in determining the partition reclamation task, and after determining the partition to be reclaimed and the free partition in the memory; The processing of the second type of subtask submitted by the main thread includes: Based on the identifier carried in the second type of subtask, the status of the partition to be reclaimed is updated to closed, the free partition is updated to open, the valid data in the partition to be reclaimed is written to the free partition, the partition to be reclaimed is erased, and the message that the second type of subtask has been successfully processed is sent to the main thread.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

11. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method of any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 9.

Images