A method and system for unloading key-value database tasks based on computing storage devices

Abstract

The application relates to the technical field of key value database, and discloses a key value database task offloading method and system based on a computing storage device, which comprises the following steps: dynamically generating an offloading judgment standard according to metadata information of a host terminal historical merging task; identifying a merging task suitable for offloading to the computing storage device based on the offloading judgment standard; in response to a merging task triggering event, scheduling selection of the computing storage device or the host as an execution device of the merging task suitable for offloading; and screening a sorting string table file participating in the merging task and checking execution resources for the selected execution device, so as to complete task offloading. The application effectively solves the problem of system performance decline caused by simple offloading, realizes efficient utilization of computing storage resources, and improves the overall throughput rate of the system.

Description

Technical Field

[0001] This application relates to the field of key-value database technology, and for example to a method and system for unloading key-value database tasks based on computing storage devices. Background Technology

[0002] Databases play a crucial role in modern computing systems, providing data persistence and efficient retrieval. Their performance directly determines the response speed and throughput of upper-layer applications. Key-value databases based on the Log-Structured Merge-tree (LSM-tree) structure have become the mainstream high-performance storage engine due to their advantages such as sequential write friendliness, low write amplification, and high throughput. Representative implementations include LevelDB, RocksDB, Cassandra, and HBase.

[0003] LSM-trees convert random writes to sequential append writes, and utilize a background compaction mechanism to merge multi-level sorted string tables (SSTables), deduplicate, and perform garbage collection, thus ensuring read-write consistency while reducing write amplification and improving space utilization. However, as data volume and write speed increase, merging becomes a major bottleneck for system performance. Its execution requires scanning and merging a large number of sorted string tables, generating intensive disk I / O and CPU overhead. Frequent triggering can lead to resource contention in the foreground write path, resulting in "write thrashing." Computational Storage Devices (CSDs) offer a new approach to overcoming this bottleneck. They integrate programmable computing units within SSDs, supporting the offloading of some data processing logic to the storage end, achieving "near-data computation." Currently, research on accelerating key-value databases using CSDs employs a simple offloading scheme, either by offloading all merge tasks or by offloading them to the CSD according to the merge workload ratio. However, due to insufficient task adaptation, indiscriminate offloading can easily lead to storage resource overload, and increased bandwidth competition due to host-storage data interaction. This simple offloading method results in a decline in overall system performance. Therefore, a task offloading method is urgently needed to solve these problems.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method and system for unloading key-value database tasks based on computing storage devices, in order to solve the technical problem that simply unloading merged tasks in an LSM-tree to computing storage devices leads to a decrease in the overall system performance.

[0007] In some embodiments, a key-value database task offloading method based on a computing storage device includes the following steps:

[0008] Based on the metadata information of historical merge tasks on the host side, the unload judgment criteria are dynamically generated;

[0009] Identify merge tasks suitable for unloading to compute storage devices based on unloading criteria;

[0010] In response to a merge task trigger event, the scheduler selects either a compute storage device or a host as the appropriate execution device for the merge task to be offloaded.

[0011] For the selected execution device, filter the sorted string table file participating in the merging task and verify the execution resources to complete the task unloading.

[0012] In some embodiments, metadata information includes the execution time of the merging task on the host side, the level to which it belongs, and the overlap of key-value ranges.

[0013] In some embodiments, uninstallation criteria are dynamically generated based on metadata information of historical merge tasks on the host side, including:

[0014] Maintain a circular buffer to cache metadata information;

[0015] Based on the execution time and the level to which it belongs, the average execution speed of each level is calculated and updated using a sliding window. The average execution speed is compared with the preset execution speed of the computing storage device to determine the offloading effectiveness of each level.

[0016] Based on the key value range overlap and the level to which it belongs, in any level, when the speed of the merge task executed by the host falls within the execution speed of the k tasks in the history of the level that are closest to the speed of the computing storage device, the key value range overlap is collected and the average value is calculated to obtain the overlap threshold of the corresponding level.

[0017] In some embodiments, identifying merge tasks suitable for offloading to compute storage devices based on offloading determination criteria includes:

[0018] When the average execution speed of the task's hierarchy is less than the preset execution speed of the computing and storage device, or when the overlap of the task's key value range is less than the overlap threshold of its hierarchy, the merged task will be marked as an unloadable task.

[0019] In some embodiments, in response to a merge task triggering event, scheduling selects a compute storage device or host as the execution device for a suitable merge task to be offloaded, including:

[0020] When the merge thread is triggered, obtain the usage status of the compute storage device;

[0021] An out-of-order selection strategy is used to select an idle device from the computing storage device as the execution device;

[0022] If all computing and storage devices are busy, the host is selected as the execution device.

[0023] In some embodiments, the out-of-order selection strategy includes:

[0024] Randomly generate device numbers within the range of the number of storage devices, and check whether the corresponding devices are idle;

[0025] If the device is available, select the computing storage device as the execution device; otherwise, continue to select randomly until the maximum number of selections is reached.

[0026] In some embodiments, filtering the sorted string table file participating in the merging task includes:

[0027] Traverse each level sequentially from the lowest to the highest level;

[0028] If the current level is a valid offload level, then all sorted string table files located on the selected computing storage device are directly selected. The valid offload level is the level where the host's average execution speed is lower than the preset execution speed of the computing storage device.

[0029] If the current level is not a valid unloading level, select the sorted string table file located on the selected computing storage device and whose key value range overlap meets the overlap threshold.

[0030] In some embodiments, verifying execution resources includes:

[0031] Check whether the selected computing storage device meets the memory resources required for this merge task;

[0032] If the conditions are met, the task is pushed down to the compute storage device; otherwise, it is rolled back to the host for execution.

[0033] In some embodiments, the sliding window size is 10, and the average execution speed is the average execution speed of 10 merged tasks.

[0034] In some embodiments, a key-value database task offloading system based on a computing storage device includes an analyzer, an allocator, and a picker connected via communication links, wherein...

[0035] The analyzer is used to dynamically generate unloading criteria based on the metadata information of historical merge tasks on the host side, and to identify merge tasks that are suitable for unloading to compute storage devices based on the unloading criteria.

[0036] In response to a merge task triggering event, the allocator schedules either a compute storage device or a host as the appropriate execution device for the merge task to be offloaded.

[0037] The selector filters the sorted string table files participating in the merge task for the selected execution device and verifies the execution resources to complete the task unloading.

[0038] The key-value database task unloading method and system based on computing storage devices provided in this disclosure can achieve the following technical effects:

[0039] This system dynamically generates unloading criteria based on metadata information from historical merge tasks on the host side, enabling precise selection of unloading tasks and avoiding the blind approach of simply unloading proportionally or all tasks. This addresses the performance degradation caused by indiscriminate unloading at its source. Based on these criteria, suitable merge tasks for unloading are identified, ensuring that only tasks truly compatible with the computing and storage devices are selected, preventing unsuitable tasks from consuming device resources. After a task is triggered, the system schedules and selects the execution device based on the trigger event, flexibly allocating task load between the host and computing / storage devices. This avoids task blocking when devices are busy and improves the smoothness of task execution. Finally, the system filters and sorts string table files and verifies resources to ensure that tasks unloaded to the device can run efficiently matching hardware conditions, reducing performance loss caused by resource mismatch. Therefore, this application effectively solves the system performance degradation problem caused by simple unloading, achieving efficient utilization of computing and storage resources and improving overall system throughput.

[0040] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0041] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0042] Figure 1 This is a flowchart of a key-value database task unloading method based on a computing storage device, provided in an embodiment of this disclosure;

[0043] Figure 2 This is a flowchart of an execution device scheduling and selection provided in an embodiment of this disclosure;

[0044] Figure 3This is a flowchart of file filtering and resource verification for a sorted string table file provided in an embodiment of this disclosure;

[0045] Figure 4 This is a timing diagram of a key-value database task unloading method based on a computing storage device provided in an embodiment of this disclosure;

[0046] Figure 5 This is an architecture diagram of a key-value database task unloading system based on computing storage devices provided in this embodiment of the disclosure. Detailed Implementation

[0047] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0048] The terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0049] Unless otherwise stated, the term "multiple" means two or more.

[0050] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0051] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0052] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0053] LSM-tree integrates programmable computing units within its SSDs via compute storage devices (CSDs), enabling the offloading of some data processing logic to the storage end, achieving "near-data computation." Since the compute accelerator and storage medium of the compute storage device are integrated within the same device, data transfer between them enjoys advantages such as high bandwidth, low latency, and low power consumption. This significantly reduces data transfer overhead and minimizes the consumption of host resources. Therefore, offloading compaction tasks from key-value databases to compute storage devices improves overall database throughput; compaction can also be understood as compression and reorganization. Currently, research using CSDs to accelerate key-value databases uses simple offloading schemes, either by offloading all compaction tasks or by simply offloading a proportion of the compaction workload to the compute storage device. However, due to insufficient task adaptation, indiscriminate offloading can easily lead to storage resource overload, intensified bandwidth competition between host and storage data interaction, and other issues. Such simple offloading methods result in a decline in overall system performance.

[0054] To address the aforementioned issues, this disclosure provides a key-value database task offloading method and system based on computing storage devices. The hardware architecture of this application consists of three types of devices: host CPU, host memory, and computing storage device arrays. While maintaining similarity to the original LSM tree structure, this architecture replaces the traditional disk array with computing storage device arrays, and each computing storage device has a merging task computation acceleration core within its acceleration device. Based on this hardware architecture, this application designs a preferred offloading task strategy based on merging semantics, based on host-side merging performance and metadata correlation analysis. The final implementation is a key-value database task offloading system, also known as a semantically aware offloader, which mainly includes a communication-connected analyzer, allocator, and selector. The analyzer, allocator, and selector are used to execute the key-value database task offloading method. The key-value database task offloading method provided by this disclosure will be described in detail below with reference to the accompanying drawings.

[0055] Figure 1 This is a flowchart of a key-value database task unloading method based on a computing storage device, provided in an embodiment of this disclosure. Figure 1 As shown, the key-value database task unloading method includes the following steps:

[0056] S101: Dynamically generate unload judgment criteria based on the metadata information of historical merge tasks on the host side.

[0057] In some embodiments, the metadata information includes the execution time of the merge task on the host side, the hierarchical level to which it belongs, and the key-value range overlap. Here, key-value range overlap refers to the extent to which the ranges of keys contained in the multiple sorted string table files participating in this merge task overlap. Low overlap means that these files contain mostly different keys, while high overlap means that there is significant overlap in key ranges, potentially including many new and old versions with the same keys that need to be compared and merged.

[0058] In some embodiments, the analyzer needs to dynamically generate unloading criteria based on the metadata information of historical merged tasks on the host side, ensuring that only truly suitable tasks are unloaded to the CSD. The specific implementation is as follows:

[0059] In some embodiments, the analyzer maintains a circular buffer in host memory specifically for caching core metadata for each merge task executed on the host side in real time. This metadata includes: the execution time of the task, such as a Level 2 merge taking 80ms; the level to which the task belongs, such as the level identifier of an LSM-tree like Level 1 or Level 3; and the overlap of key-value ranges involved in the task, such as the overlap of key-value ranges between two sorted string table files being 200. Through the circular buffer, the system can continuously track the characteristics of the most recent merge tasks, providing a data foundation for the generation of subsequent judgment criteria.

[0060] In some embodiments, the average execution speed of each level is dynamically updated using a sliding window algorithm, serving as the core criterion for determining the effectiveness of level unloading. A sliding window with a fixed window size of 10 is used, meaning the average speed of the most recent 10 merge tasks is calculated each time. Whenever a host segment completes a merge task, its execution time is immediately added to the statistical queue of the corresponding level. The calculated average execution speed of the level is compared with a preset computing and storage device execution speed. For example, if the preset benchmark speed for the computing and storage device to process a certain level of task is 250MB / s, and the average speed of the most recent 10 merges for Level 3 is lower than 250MB / s (i.e., the average execution speed is lower than the computing and storage device execution speed), then that level is marked as an effective unloading level. If the average speed of the most recent 10 merges for Level 1 is higher than 250MB / s (i.e., the average execution speed is higher than the computing and storage device execution speed), then that level is marked as an ineffective unloading level. Thus, the effectiveness of unloading at each level can be determined based on the execution time and the level to which it belongs.

[0061] In some embodiments, for each level, a specific key-value overlap threshold is derived by filtering key historical data. Specifically, based on the key-value range overlap and the level to which it belongs, in any level, when the speed of the merge task executed on the host side falls within the execution speed of the k tasks in the history of that level that are closest to the speed of the computing and storage device, the key-value range overlap is collected and the average value is calculated to obtain the overlap threshold for the corresponding level. Specifically, in any level, the k historical tasks whose host-side execution speed is closest to a preset computing and storage device execution speed are selected; for example, k=10, meaning the 10 tasks in that level whose execution time is closest to the CSD benchmark speed of 250MB / s are selected. The key-value range overlap data of these 10 tasks is collected, and their average value is the overlap threshold.

[0062] S102: Identify merged tasks suitable for unloading to computing storage devices based on the unloading determination criteria.

[0063] In some embodiments, when identifying merged tasks suitable for offloading to compute storage devices based on the offloading determination criteria, if the average execution speed of the task's level is less than the preset compute storage device execution speed, that level is marked as a valid offloading level. Tasks at this level are more suitable for offloading to compute storage devices to improve performance due to lower host processing efficiency. For example, if the average host execution speed of Level 3 is 150MB / s, while the preset compute storage device execution speed is 250MB / s, meaning the compute storage device processes tasks at this level faster, then Level 3 is determined to be a valid offloading level, and tasks in this level have the basis for offloading.

[0064] Alternatively, when the key-value range overlap of a task is lower than the overlap threshold of its level, the merged task is marked as an unloadable task. For ineffective unloading levels (i.e., average host execution speed not lower than the preset computing and storage device execution speed), if the key-value range overlap of a task is lower than the overlap threshold derived for that level, the task is still considered suitable for unloading. For example, Level 2 is an ineffective unloading level (average host execution speed 300MB / s ≥ and preset computing and storage device execution speed 250MB / s), but its derived overlap threshold is 400. When the overlap of a task is 200 (lower than the overlap threshold), the task is considered suitable for unloading.

[0065] The above, from the perspectives of hierarchy and task, constitutes the core of the unloading judgment criteria. For tasks at the effective unloading level, they are directly included in the unloading range by comparing execution speed. For tasks at the ineffective unloading level, individual tasks suitable for unloading are selected through an overlap threshold. This dual threshold strategy of hierarchy and task dimensions takes into account both computational intensity and I / O characteristics, ensuring that only tasks truly suitable for computing and storage devices are selected.

[0066] S103: In response to the merge task trigger event, the scheduler selects either a compute storage device or a host as the execution device for the appropriate merge task to be offloaded.

[0067] In some embodiments, when the merge thread is triggered, the allocator obtains the usage status of the compute storage devices and selects an idle device from the compute storage devices as the execution device using a random selection strategy. If all compute storage devices are busy, the host is selected as the execution device. The random selection strategy includes: randomly generating device numbers within the range of the number of compute storage devices, checking whether the corresponding device is idle, selecting the idle compute storage device as the execution device if it is, otherwise continuing random selection until the maximum number of selections is reached.

[0068] Figure 2 This is a flowchart illustrating the scheduling and selection of an execution device according to an embodiment of this disclosure. (In conjunction with...) Figure 2 When the system responds to the compaction task trigger event, such as operations like merging and sorting string table files in LevelDB and RocksDB, it first obtains the usage status of the compute storage devices, actively querying whether the compute storage devices are currently busy. Then, it initializes the selection count to zero and sets the default execution device to the host. Next, it enters a loop: if the selection count is less than the total number of devices, it randomly generates a device number and checks if the device is idle. If idle, it selects the device and exits the loop; if not idle, it increments the selection count and returns to re-evaluate the difference between the selection count and the total number of devices. If the selection count reaches the total number of devices (the selection count limit), it maintains the default execution device as the host and finally outputs the selected execution device. When all compute storage devices are busy, the host is executed by default; when an idle compute storage device is selected, it is used as the execution device, thus achieving load balancing.

[0069] S104: For the selected execution device, filter the sorted string table file participating in the merging task and verify the execution resources, and complete the task unloading.

[0070] In some embodiments, filtering sorted string table files participating in the merging task includes: a picker traversing each level sequentially from the lowest to the highest level; if the current level is a valid offload level, then all sorted string table files located on the selected computing storage device are directly selected, wherein the valid offload level is a level where the host's average execution speed is lower than a preset computing storage device execution speed; if the current level is not a valid offload level, then sorted string table files located on the selected computing storage device and whose key value range overlap meets the overlap threshold are selected.

[0071] In some embodiments, verifying execution resources includes: a selector checking whether the selected compute storage device meets the memory resources required for this merge task; if it does, the task is pushed down to the compute storage device; otherwise, it is rolled back to the host for execution.

[0072] Figure 3 This is a flowchart of a sorted string table file filtering and resource verification process provided in an embodiment of this disclosure. Figure 3 As shown, taking the selected execution device, effective unloading level, and overlap threshold as input, the process traverses the levels sequentially from the bottom to the top. If the current level is an effective unloading level, all sorted string table files under the device path are selected. If the file is not empty, the device memory resource requirements are checked. If they are met, the task is pushed down to the execution device; otherwise, it is rolled back to the host. If the current level is not an effective unloading level, the default execution device is kept as the host. Finally, the task unloading is completed. This process further filters and places the task on the execution device based on the determination that it can be unloaded.

[0073] The key-value database task unloading method in this application will be further explained below with reference to the accompanying drawings.

[0074] Figure 4 This is a timing diagram of a key-value database task unloading method based on a computing storage device provided in an embodiment of this disclosure. Combined with... Figure 4 When the host database is running continuously, background merge task requests will be generated. The analyzer continuously monitors and analyzes, obtaining the task's metadata after each merge task is executed by the host, then calculating the average execution speed of each level (one of the criteria for determining whether to unload); deriving the overlap threshold for each level (another criterion for determining whether a task is suitable for unloading); marking the effective unloading levels, and dynamically updating this information. When the host triggers a merge thread start request, the allocator first queries the CSD array status, i.e., whether all compute storage devices are busy. If an idle CSD is found (e.g., CSD 0), it is selected to execute the task; if all CSDs are busy, execution is returned to the host CPU. After the allocator determines the CSD execution device, the selector traverses the levels from the bottom to the top (e.g., Level 0 → Level 1 → ...) based on the overlap threshold calculated by the analyzer and the effective unloading level. If the traversed level is an effective unloading level, the sorted string table file to be merged is directly selected. If the traversed level is not an effective unloading level, the task with an overlap lower than the overlap threshold is selected. After task selection, the resources of the corresponding CSD devices are further verified. When resources are sufficient, the task can be offloaded to the corresponding CSD device for execution, where the device's internal accelerated kernel processes the merge task and updates its status upon completion. When resources are insufficient, execution is deferred to the host CPU to avoid device overload. If all CSDs are busy, the system allocates the task to the host CPU for execution. The host CPU handles merge tasks that are not suitable for offloading.

[0075] Figure 5 This is an architecture diagram of a key-value database task offloading system based on computing storage devices provided in this embodiment of the disclosure. Combined with... Figure 5 When the host machine runs the merge thread (handling operations such as merging sorted string table files), the analyzer collects metadata information from the thread and stores it in a circular buffer. Based on this data, it analyzes suitable merge tasks to be offloaded to compute storage devices. Upon receiving instructions to schedule the merge thread, the allocator selects compute storage devices in random order to achieve load balancing and avoid overload caused by consistently selecting a particular device. If all compute storage devices are busy, execution returns to the host machine's merge thread. The selector, combining the results calculated by the analyzer, filters tasks from specific paths to determine which merges can actually be offloaded to compute storage devices.

[0076] The following section provides a detailed explanation of the analyzer, allocator, and picker in the key-value database task unloading system, using pseudocode as an example.

[0077] First, the analyzer analyzes the relevant metadata information from the host-side key-value database to determine the appropriate levels for offloading to the compute storage device and the corresponding overlap thresholds. The algorithm strategy process can be formally represented as follows:

[0078]

[0079] Through the above process, the analyzer can determine the effective unloading level and the overlap threshold of the key values ​​corresponding to each level. When the merged task meets one of the two threshold conditions, it is identified as a task suitable for unloading.

[0080] Next is the allocator, which automatically schedules and selects hosts and compute storage devices when the merge thread is triggered. The allocator achieves workload balancing by selecting compute storage devices out of order. The algorithm strategy can be formally represented as follows:

[0081]

[0082] In summary, by selecting computing and storage devices out of order, load balancing among computing and storage devices is achieved, thereby improving device utilization.

[0083] Finally, the selector, based on the effective offloading level and key-value overlap threshold obtained from the analyzer, will select the preferred merging task for a thread's compute storage device after the thread is scheduled to a compute storage device. The algorithm strategy process can be formally represented as follows:

[0084]

[0085] This application identifies suitable unloading levels and key-value overlap thresholds based on information such as the execution time of the merge operation on the host side, the merge target level, and the overlap of key-value ranges. Merge tasks with deeper levels and lower key-value range overlap are more suitable for unloading. Based on these identification results, this application optimizes the unloading of merge tasks by using an analyzer, allocator, and selector to achieve optimized task selection and coordinated scheduling with the host and computing / storage devices, thereby improving the overall system throughput.

[0086] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions executed by a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0087] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for descriptive purposes only and is not intended to limit the scope of protection. As used in the description herein, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0089] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims

1. A method for unloading key-value database tasks based on computing storage devices, characterized in that, The method comprises the following steps: According to the metadata information of the host end history merging task, the offloading judgment standard is dynamically generated, including: maintaining a circular buffer to cache the metadata information, wherein the metadata information includes the execution time consumption of the host end executing the merging task, the corresponding level, and the key value range overlap degree; according to the execution time consumption and the corresponding level, the average execution speed of each level is calculated and updated by using a sliding window, wherein the average execution speed is used for comparison with the preset computing storage device execution speed to determine the offloading effectiveness of each level; according to the key value range overlap degree and the corresponding level, in any level, when the speed of the merging task executed by the host falls within the execution speed of the k tasks in the history of the corresponding level that is closest to the computing storage device speed, the key value range overlap degree is collected and the average value is calculated to obtain the overlap threshold of the corresponding level; Based on the offloading judgment standard, the merging task suitable for offloading to the computing storage device is identified, including: when the average execution speed of the task corresponding level is less than the preset computing storage device execution speed, or when the key value range overlap degree of the task itself is lower than the overlap threshold of its corresponding level, the merging task is marked as a task that can be offloaded; In response to a merging task triggering event, the computing storage device or the host is selected as the execution device of the merging task suitable for offloading; For the selected execution device, the sorting string table file participating in the merging task is screened and the execution resource is checked to complete the task offloading.

2. The compute storage device based key-value database task offloading method according to claim 1, characterized in that, The response to the merging task triggering event includes: When the merging thread is triggered, the usage state of the computing storage device is obtained; An out-of-order selection strategy is used to select an idle device from the computing storage device as the execution device; If all computing storage devices are in a busy state, the host is selected as the execution device.

3. The compute storage device based key-value database task offloading method according to claim 2, characterized in that, The out-of-order selection strategy includes: A device number is randomly generated within the range of the number of computing storage devices, and it is checked whether the corresponding device is idle; If it is idle, the computing storage device is selected as the execution device, otherwise the random selection is continued until the upper limit of the selection number is reached.

4. The compute storage device based key-value database task offloading method of claim 1, wherein, The screening of the sorting string table file participating in the merging task includes: The levels are traversed in order from low to high; If the current level is an effective offloading level, all sorting string table files located in the selected computing storage device are directly selected, wherein the effective offloading level is a level whose host average execution speed is lower than the preset computing storage device execution speed; If the current level is not an effective offloading level, the sorting string table file located in the selected computing storage device and satisfying the overlap threshold of the key value range overlap degree is selected.

5. The compute storage device based key-value database task offloading method according to claim 1, wherein, The checking of the execution resource includes: It is checked whether the selected computing storage device meets the memory resource required by the current merging task; If it meets, the task is pushed down to the computing storage device, otherwise it is rolled back to the host for execution.

6. The compute storage device based key-value database task offloading method according to claim 1, wherein, The size of the sliding window is 10, and the average execution speed is the average execution speed of 10 merging tasks.

7. A system for task offloading based on a compute storage device based key value database, the system comprising: The system includes an analyzer, a distributor and a selector connected in communication, wherein The analyzer is configured to maintain a circular buffer to cache metadata information of the merge tasks executed by the host, wherein the metadata information comprises execution time, hierarchy level and key-value range overlap degree of the merge tasks executed by the host; calculate and update average execution speed of each hierarchy level according to the execution time and the hierarchy level in a sliding window, wherein the average execution speed is used to compare with preset computing storage device execution speed to determine the offloading effectiveness of each hierarchy level; according to the key-value range overlap degree and the hierarchy level, when the speed of the merge task executed by the host falls in the execution speed of the k tasks closest to the computing storage device speed in the history of the hierarchy level, the key-value range overlap degree is collected and an average value is calculated to obtain the overlap degree threshold of the corresponding hierarchy level. The analyzer is further configured to identify the merge tasks suitable for offloading to the computing storage device based on the offloading determination standard, and mark the merge task as a task that can be offloaded when the average execution speed of the hierarchy level to which the task belongs is less than the preset computing storage device execution speed, or when the key-value range overlap degree of the task itself is lower than the overlap degree threshold of the hierarchy level to which the task belongs. The allocator schedules the selected computing storage device or host as the execution device of the merge task suitable for offloading in response to a merge task triggering event. The selector filters the sorted string table file participating in the merge task and checks the execution resource for the selected execution device to complete the task offloading.

Images