Calculation task management method and device of database, electronic equipment, storage medium, system and computer program product

Abstract

The invention provides a calculation task management method and device of a database, electronic equipment, a storage medium, a system and a computer program product, and is applied to near-storage calculation equipment, the method can comprise the following steps: in response to an execution instruction about a preset calculation task received from a host, obtaining data required by the preset calculation task from a memory of the host, the data required by a preset computing task is stored in a memory corresponding to the near-memory computing device, and the preset computing task is a computing task which needs to be executed between a memory and a CPU (Central Processing Unit), the data exchange frequency of the computing task is higher than a preset degree, and the computing power demand of the computing task is lower than a preset computing power when the computing task is executed by the host; executing the preset calculation task through data exchange with the memory corresponding to the near-memory calculation equipment so as to obtain a calculation result corresponding to the preset calculation task; and transmitting a calculation result to the host. A calculation framework of a database storage model can be reconstructed, and the execution performance and efficiency of a database system are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of storage, and in particular to a method, device, electronic device, storage medium, system and computer program product for managing computing tasks of a database. Background Art

[0002] RocksDB is a KV (Key-Value) storage engine database with the following main features: (1) It uses the LSM tree (Log-Structured Merge Tree) structure to store data; (2) It is an embedded persistent storage system that supports high-throughput IO reading and writing; (3) The storage process involves a large amount of calculations, which require data interaction between the CPU and memory.

[0003] In the R / W (read / write) process of RocksDB, there are many scenarios of massive data movement and simple calculations, which will cause a large amount of data copying between the memory and the CPU, which is time-consuming and consumes a lot of CPU resources. Researchers have made a lot of efforts to improve the performance of RocksDB, but they only consider internal details and there is no macro-architecture solution to solve the above problems. Summary of the invention

[0004] The computing task management method, device, electronic device, storage medium, system and computer program product of a database provided by the exemplary embodiments of the present disclosure can at least solve the above-mentioned technical problems and other technical problems not mentioned above.

[0005] According to one aspect of the present disclosure, a method for managing computing tasks of a database is provided, which is applied to a near-memory computing device, the method comprising: in response to receiving an execution instruction regarding a preset computing task from a host, obtaining data required for the preset computing task from the memory of the host, and storing the data required for the preset computing task in the memory corresponding to the near-memory computing device, wherein the preset computing task is: when executed by the host, it is necessary to perform data exchange between the memory and the CPU at a frequency higher than a preset degree and the computing power requirement is lower than the preset computing power; the preset computing task is executed by exchanging data with the memory corresponding to the near-memory computing device to obtain a computing result corresponding to the preset computing task; and the computing result is transmitted to the host.

[0006] Optionally, the preset computing task includes computing tasks involved in at least one of the following items: range query and file merging for database space optimization.

[0007] Optionally, in the case where the preset computing task is a computing task involved in the file merge, the response to receiving an execution instruction regarding the preset computing task from the host, obtaining data required for the preset computing task from the memory of the host includes: when the host detects that the trigger condition for the file merge is met, receiving the execution instruction regarding the preset computing task from the host, and in response to the execution instruction of the preset computing task, obtaining the data required for the preset computing task from the memory of the host, wherein the trigger condition includes a threshold condition.

[0008] Optionally, in the case where the preset computing task is a computing task involved in the file merge, the preset computing task is executed by exchanging data with the memory corresponding to the near storage computing device to obtain the computing result corresponding to the preset computing task, including: determining the file to be merged for the file merge; reading each key-value pair in the file to be merged from the memory corresponding to the near storage computing device; comparing the keys of each key-value pair, exchanging the key-value pairs to sort the key-value pairs in the order of the keys; and obtaining a sorted key-value pair sequence in the memory corresponding to the near storage computing device.

[0009] Optionally, in the case where the preset computing task is the computing task involved in the range query, the response to receiving an execution instruction regarding the preset computing task from the host, obtaining the data required for the preset computing task from the memory of the host includes: when the condition of the range query has been confirmed by the host, receiving the execution instruction regarding the preset computing task from the host, and in response to the execution instruction of the preset computing task, obtaining the data required for the preset computing task from the memory of the host, wherein the condition of the range query includes the starting value and the ending value of the query key.

[0010] Optionally, when the preset computing task is the computing task involved in the range query, the preset computing task is executed by exchanging data with the memory corresponding to the near memory computing device to obtain the computing result corresponding to the preset computing task, including: reading a memory table from the memory corresponding to the near memory computing device; traversing the key-value pairs in the memory table, and when the conditions of the range query are not completely covered by the memory table, reading the ordered string table that meets the conditions of the range query from the memory into the memory corresponding to the near memory computing device through a Bloom filter; and collecting all key-value pairs that meet the conditions of the range query from the memory table and the ordered string table.

[0011] According to another aspect of the present disclosure, a computing task management device for a database is also provided, which is applied to a near-memory computing device, and the device includes: a data acquisition module, configured to: in response to receiving an execution instruction regarding a preset computing task from a host, acquire data required for the preset computing task from the memory of the host, and store the data required for the preset computing task in the memory corresponding to the near-memory computing device, wherein the preset computing task is: when executed by the host, it is necessary to perform data exchange between the memory and the CPU at a frequency higher than a preset degree, and the computing power requirement is lower than the preset computing power; the task execution module, configured to: execute the preset computing task by exchanging data with the memory corresponding to the near-memory computing device to obtain the computing result corresponding to the preset computing task; the computing result transmission module, configured to: transmit the computing result to the host.

[0012] According to another aspect of the present disclosure, a computing task management system for a database is also provided, the system comprising a host and a near-memory computing device, the system being configured as follows: the host sends an execution instruction regarding a preset computing task to the near-memory computing device; the near-memory computing device, in response to receiving the execution instruction for the preset computing task from the host, obtains data required for the preset computing task from the host's memory, and stores the data required for the preset computing task into the memory corresponding to the near-memory computing device, wherein the preset computing task is a computing task that, when executed by the host, requires data exchange between the memory and the CPU at a frequency higher than a preset degree and has a computing power requirement lower than a preset computing power; the near-memory computing device executes the preset computing task and obtains a computing result corresponding to the preset computing task by exchanging data with the memory corresponding to the near-memory computing device; the near-memory computing device transmits the computing result to the host; and the host receives the computing result transmitted from the near-memory computing device.

[0013] According to another aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the database computing task management methods described above.

[0014] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the database computing task management methods described above.

[0015] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the database computing task management methods described above.

[0016] According to another aspect of an embodiment of the present disclosure, a computer program product is further provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method for managing computing tasks of a database as described in any one of the above items is implemented.

[0017] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0018] According to the computing task management method, device, electronic device, storage medium, system and computer program product of the database disclosed in the present invention, the near memory processing (Process-Near-Memory, PNM) characteristics of the near memory computing device can be used to rebuild the computing architecture of storage models such as RocksDB, so as to offload the computing tasks involved in data operations such as file merging (Compaction) and range query (Range Query) used for database space optimization in the database, which require frequent data copying and moving between the CPU and the memory but are only for the CPU to perform simple calculations, from the CPU to the near memory computing device, so as to significantly reduce the amount of data read into the memory, reduce the memory burden, release the resource utilization of the CPU, reduce the data interaction between the CPU and the memory, reduce the impact of a large amount of data calculations on the database performance, increase the memory bandwidth, and thus improve the execution performance and efficiency of the database system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0020] Figure 1 A schematic diagram showing the overall architecture and data processing flow of RocksDB in the related art;

[0021] Figure 2 A schematic diagram showing file merging by RocksDB in the related art;

[0022] Figure 3 A schematic diagram showing a range query performed by RocksDB in the related art;

[0023] Figure 4 A schematic diagram showing a memory expansion based on CXL and a memory expansion based on CXL-PNM in the related art;

[0024] Figure 5 A flowchart showing a computing task management method for a database of a near-storage computing device in an exemplary embodiment of the present disclosure is shown;

[0025] Figure 6 A schematic diagram showing system function deployment for performing file merging in an exemplary embodiment of the present disclosure is shown;

[0026] Figure 7 A schematic diagram showing system function deployment for executing range query in an exemplary embodiment of the present disclosure is shown;

[0027] Figure 8 A schematic diagram showing a process of performing file merging in an exemplary embodiment of the present disclosure is shown;

[0028] Fig. 9 A flowchart showing the execution of file merging in an exemplary embodiment of the present disclosure is shown;

[0029] Fig.10 A schematic diagram showing a flow chart of executing a range query in an exemplary embodiment of the present disclosure;

[0030] Fig.11 A flowchart showing execution of a range query in an exemplary embodiment of the present disclosure is shown;

[0031] Fig.12 A schematic diagram of a system architecture for extending PNM in an exemplary embodiment of the present disclosure is shown;

[0032] Fig.13 A schematic diagram showing the differences between the file merging and range query process in the related art and the exemplary embodiment of the present disclosure;

[0033] Fig.14 A schematic diagram showing the difference between using PNM assistance in an exemplary embodiment of the present disclosure;

[0034] Fig.15 A block diagram showing a computing task management apparatus for a database of a near-storage computing device in an exemplary embodiment of the present disclosure is shown;

[0035] Fig.16 A block diagram showing a computing task management system of a database in an exemplary embodiment of the present disclosure;

[0036] Fig.17 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0039] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0040] RocksDB can store any binary KV data, and it can organize all data in order. The underlying implementation in RocksDB uses the LSM tree structure to improve write performance, so its application scenario is write-intensive. Before analyzing RocksDB's high-computing application scenarios, you can first understand RocksDB's overall architecture and data processing flow.

[0041] Figure 1 A schematic diagram showing the overall architecture and data processing flow of RocksDB in the related art.

[0042] Reference Figure 1 RocksDB is a key-value storage system, in which the LSM tree structure is a storage engine for fast storage and access of key-value data. It adopts a multi-level structure, including the memory table (MemTable) in memory and the Sorted String Table (SST) on disk. The design of LSM tree enables RocksDB to efficiently handle a large number of write operations and provide excellent read performance.

[0043] In RocksDB, the information log on the disk, the manifest file, the current file or current pointer (current), the write-ahead log (WAL) and SST are several key components that jointly maintain the consistency and performance of the database.

[0044] The data processing flow of RocksDB can mainly include the following steps: (1) Data can be first written to the memory table in the memory, which is an ordered data structure; (2) When the data in the memory table reaches a certain size, it will be converted to an immutable memory table (read-only). At the same time, a new memory table will be generated for writing data later. (3) RocksDB can flush the data in the immutable memory table to the SST according to the strategy, and then merge the SST level by level. Each level of file merging operation in the SST has a threshold. If the threshold trigger is reached, the file merge operation will be performed.

[0045] In the LSM tree structure, SST is an important component that can be used to store sorted key-value pairs for fast search based on the key. Each SST stores a portion of data, and the data is sorted by key. This design enables RocksDB to perform range queries and sequential reads efficiently. When the memory table is filled, it will be flushed to the SST and stored, thereby achieving persistent storage of data.

[0046] In general, LSM tree structures and SST can work together in RocksDB to achieve high-performance key-value storage and access functions.

[0047] In order to strike a balance between read amplification, space amplification, and write amplification, RocksDB needs to perform refreshes and file merges. When using LSM trees to manage core key-value data, compaction is responsible for merging and organizing data to optimize read performance and reduce storage space usage. RocksDB uses compaction as a merge strategy, mainly to avoid the impact of excessive redundancy in LSM trees on read performance. There are many methods for file merging, and the following is a more commonly used one.

[0048] Figure 2 A schematic diagram showing file merging by RocksDB in the related art.

[0049] Reference Figure 2, RocksDB can include multiple layers of data, for example, level 0 to level 3, etc. The size of each layer of data can be different, and the next layer can be several times larger than the previous layer. Level 0 is not globally ordered, but only the file is ordered internally, and the rest of the layers are ordered globally. Each SST stores a portion of data, and these data are sorted by key. For example, each SST of level 3 stores key-value pairs of data with keys Key0 to Key25, Key26 to Key50, Key52 to Key60, Key61 to Key85, Key88 to Key97, Key98 to Key108, etc. File merging is to use some simple rules to select SSTs with overlapping keys and use merge sort to generate new SSTs. In this process, a large amount of data will be copied between the memory (such as DRAM (Dynamic Random Access Memory)) and the CPU, just to allow the CPU to perform simple repetitive operations. It can be seen that file merging will take up a lot of CPU resources, mainly when RocksDB reaches the trigger condition for file merging and the CPU is idle, it will always run.

[0050] As for range queries, compared with the write performance of RocksDB, the performance of range queries is poor, and it often needs to be queried in the memory and the SST of each layer.

[0051] Figure 3 A schematic diagram showing a range query performed by RocksDB in the related art.

[0052] Reference Figure 3 The black block indicates that the range query condition overlaps with the data in the area, and the area needs to be read, that is, the data pool is read. Range query mainly involves two processes: (1) searching for files that overlap with the range query condition in each level of SST in memory and disk; (2) aggregating the results and returning them to the upper-level application. The range query process also involves a large amount of data copying between memory and CPU, but only to allow the CPU to do a large number of simple repeated calculations, such as binary search and Bloom filter file matching.

[0053] In the related technical solutions, the optimization solutions for file merging can be: tuning parameters for file merging, or dividing the thread of file merging into multiple sub-threads to run the file merging in parallel, etc. However, improper optimization in these solutions may cause write stalls, greatly reduce the performance of RocksDB, and may still occupy a large amount of CPU resources.

[0054] At the same time, in the related technical solutions, the optimization solution for range query is to use Prefix Bloom Filter instead of Bloom filter for file matching. However, this usually leads to increased misjudgment and the CPU still performs multiple Bloom filter comparisons on the file.

[0055] It can be seen that the relevant technical solutions all have the following key problems: (1) they consume a large amount of CPU resources, and some performance optimization solutions may even produce side effects in some extreme cases; (2) they cannot fundamentally release CPU resources to improve performance; (3) the CPU spends a lot of time executing simple and frequent tasks.

[0056] In short, none of the related technical solutions fundamentally solves the problem of large amounts of data movement and copying between the CPU and memory involved in file merging and range queries in RocksDB, just to allow the CPU to perform simple calculations.

[0057] In order to solve the above problems, the present disclosure provides a computing task management method, device, electronic device, storage medium, system and computer program product for a database, which can utilize the near memory processing characteristics of the near memory computing device to rebuild the computing architecture of storage models such as RocksDB, so as to offload the computing tasks of data operations such as file merging and range query in storage engines such as RocksDB, which require frequent data copying and movement between the CPU and the memory but are only for the CPU to perform simple calculations, from the CPU to the near memory computing device, so as to significantly reduce the amount of data read into the memory, reduce the memory burden, release the resource utilization of the CPU, reduce the data interaction between the CPU and the memory, reduce the impact of a large amount of data calculation on the performance of the storage engine, increase the memory bandwidth, and thus improve the execution performance and efficiency of the storage engine system.

[0058] It is understandable that the database computing task management method, device, electronic device, storage medium, system and computer program product provided by the present disclosure can also be applied to other key-value storage databases other than RocksDB, or column storage databases, document databases, graph databases and other databases, as well as databases using a structure similar to LSM tree, such as LevelDB, Cassandra, HBase, BadgerDB, etc. The commonality of these databases may include the need to process data operations that require frequent data copy and movement between the CPU and the memory only for the CPU to perform simple calculations, such as file merging, range query and other data operations. The present disclosure uses the RocksDB database as an example.

[0059] PNM is a technology that integrates memory and logic chips into advanced integrated circuit packages. It can reduce data movement between the CPU and memory by utilizing logic chips for data calculations.

[0060] In the case of PNM, computational functions can be performed closer to the memory to reduce the bottleneck of data transfer between the CPU and memory.

[0061] CXL TM (Compute Express Link TM ) is an open standard for high-speed processor-to-device and processor-to-memory interfaces that enable more efficient use of memory and accelerators used with processors. CXL technology can interconnect disparate computing and storage resources to improve system performance and efficiency. TM Can be used in conjunction with other technologies, such as PNM, to help facilitate memory capacity expansion.

[0062] Figure 4 A schematic diagram showing memory expansion based on CXL and memory expansion based on CXL-PNM in the related art.

[0063] Reference Figure 4 (a) in the figure shows a typical CXL-based memory expansion in the related art. The CPU and the CXL device memory can be connected through a CXL controller (CXL CTRL) to achieve high-speed, low-latency, high-bandwidth data transmission and sharing between computing resources such as the CPU and storage resources such as the CXL device memory.

[0064] Reference Figure 4 (b) in the figure shows that in the related art, CXL and PNM technologies are combined through a CXL controller to realize CXL-PNM memory expansion based on a CXL controller. CXL technology can provide a high-speed, low-latency data transmission channel, so that the CPU and the accelerator in the PNM can efficiently share and access memory resources.

[0065] In CXL technology, there are several key components and concepts:

[0066] CXL CTRL: This is the control part of CXL, which is responsible for managing and controlling CXL connections and communications. It handles CXL-related protocols and signals to ensure that communication between devices is correct.

[0067] DOR (Device On-Ramp): DOR is a device access point that allows devices (such as accelerators, memory buffers, etc.) to connect to the CXL network. It provides a standardized interface for devices to communicate with other components in the CXL network.

[0068] MC (Memory Controller): is the component responsible for managing memory access. In the CXL environment, MC can ensure that access to shared memory is efficient and consistent. In CXL technology, MC can work with the host CPU, with accelerator devices (such as PNM, etc.), or move to the memory buffer chip.

[0069] CXL.Jo: This is the joint interface of CXL, which allows two or more CXL devices to work together and share memory and other resources. Through CXL.Jo, the devices can operate as a whole, thereby improving overall performance and efficiency.

[0070] CXL.mem: This is the memory protocol of CXL, which defines the transmission interface between CPU and memory to ensure that CPU can efficiently access and use shared memory in the CXL network.

[0071] Device Memory: This refers to the memory on the accelerator or other devices (such as the host, etc.). In the CXL architecture, this memory can be shared and accessed by the CPU and other devices (such as PNM, etc.), thereby improving overall performance and resource utilization.

[0072] PNM solutions based on the CXL interface can significantly improve performance in applications that require high memory bandwidth.

[0073] Next, we will refer to Figures 5 to 17 The computing task management method, device, electronic device, storage medium, system and computer program product of the database disclosed in the present invention are specifically described.

[0074] According to an exemplary embodiment of the present disclosure, the architecture system of the present disclosure can be a new functional module directly added based on CXL on the original RocksDB design architecture to realize the hot-swappable function of computing units and memory expansion, so that CXL devices can be added / deleted online on the CXL server as needed without destroying the original system. Specifically, the near memory processing characteristics of the CXL-PNM storage device can be used to offload the computing tasks of file merging and range queries in RocksDB from the CPU to the CXL-PNM device to obtain an accelerated computing architecture of the present disclosure. The near memory computing solution based on CXL can significantly reduce the data interaction between the memory and the CPU in RocksDB, further improving the overall performance of RocksDB.

[0075] Figure 5 A flowchart of a computing task management method for a database of a nearby computing device in an exemplary embodiment of the present disclosure is shown.

[0076] Reference Figure 5 In step 501, in response to receiving an execution instruction regarding a preset computing task from the host, data required for the preset computing task can be obtained from the host's memory, and the data required for the preset computing task can be stored in the memory corresponding to the near-storage computing device, wherein the preset computing task can be: when executed by the host, it is necessary to perform data exchange between the memory and the CPU at a frequency higher than a preset degree, and the computing power requirement is lower than the preset computing power.

[0077] According to an exemplary embodiment of the present disclosure, the memory of the host is persistent storage, and specifically, it may be a disk, a hard disk, etc. of the host.

[0078] According to an exemplary embodiment of the present disclosure, the memory corresponding to the near-storage computing device may be the memory of a CXL-PNM extended through a CXL interface, or may be the memory of a host, and the like.

[0079] According to an exemplary embodiment of the present disclosure, the preset computing task may be: a computing task involved in data operations in which the frequency of data exchange between the memory and the CPU and the computing power requirement of the computing task respectively meet preset conditions during data exchange between the memory and the CPU of the host so as to execute the computing task through the CPU, that is, a computing task that requires frequent data copying and movement between the CPU and the memory but only for the CPU to perform simple calculations on the data, such as a computing task involved in at least one of the following data operations: range query, file merging for database space optimization, etc.

[0080] According to an exemplary embodiment of the present disclosure, the host may send execution instructions of computing tasks involved in data operations such as file merging and range queries to the near storage computing device. The near storage computing device may receive execution instructions of computing tasks involved in data operations such as file merging and range queries from the host. It is understandable that the execution instructions may include execution parameters of the corresponding computing tasks. The execution parameters of file merging may include, but are not limited to, the location and size of the data blocks or files that need to be merged, the strategy and level of file merging, etc. The execution parameters of range queries may include, but are not limited to, the key range of the query, iterator configuration (such as traversal direction, whether to include boundary keys, etc.), etc.

[0081] According to an exemplary embodiment of the present disclosure, after the near storage computing device receives the execution instruction of the computing task involved in the data operation, the near storage computing device can read the data required for the computing task from the persistent storage such as the disk into the memory corresponding to the near storage computing device, so as to facilitate the data exchange between the near storage computing device and the memory corresponding to the near storage computing device during the execution of the computing task.

[0082] In step 502, the preset computing task may be executed by exchanging data with the memory corresponding to the near-storage computing device to obtain a computing result corresponding to the preset computing task.

[0083] According to an exemplary embodiment of the present disclosure, the near-memory computing device can perform the computing tasks involved in the data operation and obtain the corresponding computing results by exchanging data with the memory corresponding to the near-memory computing device. According to an exemplary embodiment of the present disclosure, the host can include a controller corresponding to the preset computing task and a task dispatching module corresponding to the preset computing task; the controller can be used to coordinate the execution process of the data operation corresponding to the preset computing task and generate the result of the data operation; the task dispatching module can be used to perform at least one of the following items: interface interaction with the near-memory computing device, access to the memory corresponding to the near-memory computing device, transmission of execution parameters of the preset computing task, etc.

[0084] According to an exemplary embodiment of the present disclosure, a controller for file merging and / or a controller for range query can be deployed on the host side of RocksDB to be responsible for the overall process of file merging and range query respectively. A task dispatching module for file merging and / or a task dispatching module for range query can also be deployed on the host side of RocksDB to dispatch computing tasks involved in corresponding data operations, interface interaction with near-storage computing devices, access memory corresponding to near-storage computing devices, transmit execution parameters of corresponding computing tasks, etc.

[0085] According to an exemplary embodiment of the present disclosure, the near-storage computing device may include a computing unit corresponding to the data operation; the computing unit may be used to execute computing tasks involved in the data operation.

[0086] According to an exemplary embodiment of the present disclosure, computing units corresponding to data operations, such as computing units for file merging and / or computing units for range queries, may be deployed in a near-storage computing device, such as a CXL-PNM device, so that the near-storage computing devices can process their respective computing tasks through their respective computing units for data operations.

[0087] Figure 6 A schematic diagram of system function deployment for performing file merging in an exemplary embodiment of the present disclosure is shown.

[0088] Reference Figure 6 According to an exemplary embodiment of the present disclosure, on the host side of RocksDB, the controller 601 of file merging can coordinate the entire process of file merging, and the task dispatching module 602 of file merging can be responsible for interface interaction to access the memory of the CXL-PNM device, transmit the parameters of the computing tasks involved in the file merging, receive the computing results, etc. On the CXL-PNM device side, the computing unit 604 of file merging can be deployed in the computing core 603 of the near-memory computing device to perform the computing process of the computing tasks involved in the file merging. In the process of executing the computing process of the file merging, data exchange can be performed between the computing unit 604 of the file merging and the CXL memory 605. For example, the computing unit 604 of the file merging can read the data required for the computing task of the file merging from the CXL memory 605. After the calculation is completed, the computing unit 604 of the file merging can return the calculation result to the task dispatching module 602 of the file merging on the host side.

[0089] Figure 7 A schematic diagram of system function deployment for executing range query in an exemplary embodiment of the present disclosure is shown.

[0090] Reference Figure 7According to an exemplary embodiment of the present disclosure, on the host side of RocksDB, the range query controller 701 may coordinate the entire range query process, and the range query task dispatching module 702 may be responsible for interface interaction to access the memory of the CXL-PNM device, transmit the parameters of the computing task involved in the range query, receive the computing results, etc. On the CXL-PNM device side, the range query computing unit 703 may be deployed in the computing core 603 of the near memory computing device to perform the computing process involved in the range query. During the computing process of executing the range query, data exchange may be performed between the range query computing unit 703 and the CXL memory 605. For example, the range query computing unit 703 may read the data required for the computing task of the range query from the CXL memory 605. After the calculation is completed, the range query computing unit 703 may return the calculation result to the host-side range query task dispatching module 702.

[0091] According to an exemplary embodiment of the present disclosure, in a case where the preset computing task is a computing task involved in file merging, in response to receiving an execution instruction regarding the preset computing task from a host, obtaining data required for the preset computing task from a memory of the host may include but is not limited to the following steps: in a case where the host detects that a trigger condition for file merging is met, an execution instruction regarding the preset computing task may be received from the host, and in response to the execution instruction of the preset computing task, obtaining data required for the preset computing task from the memory of the host, wherein the trigger condition may include but is not limited to a threshold condition.

[0092] According to an exemplary embodiment of the present disclosure, the host side can detect whether the triggering conditions for file merging are met. In RocksDB, the triggering conditions for file merging include but are not limited to the following: (1) Size threshold: When the size of a MemTable or SST exceeds a preset threshold, file merging can be triggered, which helps to ensure that the size of the database does not grow indefinitely; (2) Time threshold: Even if the size threshold is not reached, if the existence time of a MemTable or SST exceeds the preset time threshold, file merging may also be triggered, which helps to prevent long-term retention of data and ensure data persistence; (3) Manual triggering: Users or applications can also manually trigger file merging; (4) Background thread: RocksDB's background thread can regularly check and trigger the required file merging to maintain the performance and efficiency of the database. When the host detects that the triggering conditions for file merging are met, the host can send execution instructions for computing tasks involved in file merging to the near storage computing device, and the near storage computing device can receive the execution instructions for computing tasks involved in file merging from the host and respond.

[0093] According to an exemplary embodiment of the present disclosure, in a case where the preset computing task is a computing task involved in a range query, in response to receiving an execution instruction regarding the preset computing task from a host, obtaining data required for the preset computing task from a memory of the host may include but is not limited to the following steps: when a condition for the range query has been confirmed by the host, receiving an execution instruction regarding the preset computing task from the host, and in response to the execution instruction of the preset computing task, obtaining data required for the preset computing task from the memory of the host, wherein the condition for the range query may include but is not limited to a start value and an end value of a query key.

[0094] According to an exemplary embodiment of the present disclosure, when a range query is required, the host side can confirm the conditions of the range query. When performing a range query in RocksDB, it is necessary to specify the start key and end key of the query, that is, the start value and end value of the query key, to determine the range of the query; this range can be a closed interval, an open interval, or a half-open and half-closed interval, depending on the requirements of the query. After the host side confirms the conditions of the range query, the host side can send execution instructions for the computing tasks involved in the range query to the near-storage computing device, so that the near-storage computing device can respond immediately after receiving the execution instructions sent by the host.

[0095] Return to reference Figure 5 In step 503, the calculation result may be transmitted to the host.

[0096] According to an exemplary embodiment of the present disclosure, after the near-storage computing device executes a computing task and obtains a computing result, the computing result can be transmitted from the near-storage computing device to the host side, so that the host side obtains the result of the corresponding data operation according to the received computing result. This process can significantly release CPU resources, reduce frequent data copying and exchange between the CPU and the memory, etc.

[0097] According to an exemplary embodiment of the present disclosure, in a case where the preset computing task is a computing task involved in file merging, the preset computing task is executed by exchanging data with the memory corresponding to the near storage computing device to obtain the computing result corresponding to the preset computing task, which may include but is not limited to the following steps: the file to be merged targeted by the file merge may be determined; each key-value pair in the file to be merged may be read from the memory corresponding to the near storage computing device; the keys of each key-value pair may be compared, and the key-value pairs may be exchanged to sort the key-value pairs in the order of the keys; and the sorted key-value pair sequence may be obtained in the memory corresponding to the near storage computing device.

[0098] Figure 8 A schematic diagram of a process of performing file merging in an exemplary embodiment of the present disclosure is shown.

[0099] Reference Figure 8(a), according to an exemplary embodiment of the present disclosure, the process of file merging can be composed of the four steps shown in the figure. In step 801, the host can check whether it is necessary to perform file merging at present. If necessary, in step 802, the file to be merged can be read, and the file can be selected to be read into the computing core (including computing unit) of the near storage computing device. In step 803, the file merge can be performed on the file to be merged in the computing core of the near storage computing device. In step 804, the result of the file merge can be recorded on the host side. Steps 801 and 804 processed on the host side can mainly use the above-mentioned file merging controller 601 and task dispatching module 602. Steps 802 and 803 can be processed by the CXL-PNM side, which can mainly include receiving parameters and processing the computing tasks involved in data operations and returning the computing results.

[0100] Reference Figure 8 (b), according to an exemplary embodiment of the present disclosure, in the process of the near-storage computing device executing the computing tasks involved in the file merge, it can first be determined which level or levels of the files to be merged (SST) need to be merged, and these files to be merged can be read from the disk into the memory, and then the near-storage computing device can perform a merge sort involving many key comparisons: read each key-value pair in the file to be merged from the memory (data movement from the memory to the near-storage computing device), and compare the keys of each key-value pair, exchange each key-value pair to sort each key-value pair in the key order. At this time, the sorted key-value pair sequence (the calculation result corresponding to the file merge) can be obtained in the memory, and the key-value pair sequence can be transmitted to the host through the data movement from the near-storage computing device to the memory, so as to generate a merged file (the result of the file merge) on the host side based on the sorted key-value pair sequence.

[0101] According to an exemplary embodiment of the present disclosure, in the process of the near-memory computing device executing the computing task of file merging, a massive amount of data will be copied between the computing core of the near-memory computing device and the CXL memory, which can save the transmission distance compared to the massive data copying between the CPU and the memory; the computing core of the near-memory computing device executing a large number of but simple calculations can significantly release CPU resources compared to executing them by the CPU.

[0102] Fig. 9 A flowchart of performing file merging in an exemplary embodiment of the present disclosure is shown.

[0103] Reference Fig. 9According to an exemplary embodiment of the present disclosure, in the process of executing file merging, in step 901, the state of the file merging trigger can be judged, and the judgment can be made by any one or more of the above-mentioned thresholds and other trigger conditions. If the trigger condition of the corresponding threshold is not met, the process ends. If the trigger condition of the corresponding threshold is met, in step 902, the data required for file merging can be read from a persistent storage such as a disk or SSD (Solid State Drive) into the CXL memory. Then in step 903, the file to be merged can be selected from the memory to a near storage computing device. In step 904, the logical operation of file merging can be performed in the near storage computing device, such as key comparison, key sorting, merging key-value pair sequences (merge sort) logical calculation operations. In step 905, the sorted key-value pair sequence can be further merged according to the result of the logical operation to generate a new file as a result. In this process, the computing core of CXI-PNM can be used instead of the CPU to perform the compression operation in steps 902 to 904 to avoid not performing the compression operation when the CPU is busy. And it can bring the following advantages: it can save data transmission time, reduce data transmission distance, not consume a lot of CPU resources, and ensure that file merging is always in progress.

[0104] According to an exemplary embodiment of the present disclosure, in a case where the preset computing task is a computing task involved in a range query, the preset computing task is executed by exchanging data with the memory corresponding to the near memory computing device to obtain the computing result corresponding to the preset computing task, which may include but is not limited to the following steps: a memory table may be read from the memory corresponding to the near memory computing device; the key-value pairs in the memory table may be traversed, and when the conditions of the range query are not completely covered by the memory table, an ordered string table that meets the conditions of the range query may be read from the memory through a Bloom filter into the memory corresponding to the near memory computing device; all key-value pairs that meet the conditions of the range query are collected from the memory table and the ordered string table.

[0105] Fig.10 A schematic diagram of a flow chart for executing a range query in an exemplary embodiment of the present disclosure is shown.

[0106] Reference Fig.10(a), according to an exemplary embodiment of the present disclosure, the range query process can be composed of the four steps shown in the figure. In step 1001, the current range query conditions can be confirmed in the host. In step 1002, the MemTable can be read from the memory by the near storage computing device and confirmed whether the range query conditions are fully covered. If not fully covered, in step 1003, the near storage computing device can read each layer of SST that meets the range query conditions from the disk or the like into the memory. In this step, a Bloom filter can be used for key comparison and key matching to read each layer of SST that meets the range query conditions. In step 1004, the result of the range query can be recorded on the host side. Steps 1001 and 1004 can be processed on the host side, mainly using the above-mentioned range query controller 701 and task dispatching module 702. Steps 1002 and 1003 can be processed by the CXL-PNM side, which can mainly include receiving parameters, processing corresponding computing tasks, and returning computing results.

[0107] Reference Fig.10 (b), according to an exemplary embodiment of the present disclosure, in the process of executing a computing task of a range query, the following steps may be performed by a near-memory computing device: the MemTable may be first read from the memory, and the key-value pairs in the MemTable may be traversed to determine whether the conditions of the range query are fully covered by the MemTable; when the conditions of the range query are not fully covered by the memory table, the SST that meets the conditions of the range query may be read into the memory from a persistent storage such as a disk through a Bloom filter; finally, all key-value pairs that meet the conditions of the range query may be collected from the read MemTable and all SSTs (the calculation results corresponding to the range query), and returned to the host to obtain the results of the range query through the host. Since RocksDB uses a hierarchical (multi-level) storage structure, each layer contains multiple SST files. When querying, it is necessary to start reading from the bottom SST file and go up layer by layer. When the query range spans multiple and / or multiple layers of SST, the data in these files needs to be read and merged. Bloom filters can help quickly determine whether a key exists in a specific SST through key comparison, thereby avoiding unnecessary disk I / O operations on files; in this way, the SST containing the target key can be located more efficiently and range queries can be performed.

[0108] According to an exemplary embodiment of the present disclosure, in the process of executing the computing task of range query by the near-memory computing device, since a large number of Bloom filters may be involved in the process of range query for key comparison and key matching, a large amount of data can be copied between the computing core of the near-memory computing device and the CXL memory to complete the Bloom filtering, which can save data transmission loss and transmission distance compared with the massive data copy between the CPU and the memory; the computing core of the near-memory computing device performs a large number of simple calculations, which can significantly release CPU resources compared with the CPU. In addition, when performing range query, the query can be started from the lowest component to merge and process the data layer by layer; the data can be effectively filtered and screened at an earlier level or stage as much as possible, so that only the data that meets the query conditions will be passed to the upper layer for further processing, which can avoid unnecessary transmission to the upper layer; therefore, the earlier the data is processed, the less data will be returned upward, and the amount of data transferred upward at each layer is reduced, and the query results returned in the end will also be more accurate and efficient.

[0109] Fig.11 A flowchart for executing a range query in an exemplary embodiment of the present disclosure is shown.

[0110] Reference Fig.11 According to an exemplary embodiment of the present disclosure, in the process of executing a range query, in step 1101, the conditions of the range query can be checked and relevant parameters can be prepared and verified. For example, it is possible to check whether the start key and end key of the range query are legal, such as whether they are empty or the format is correct; it is possible to check whether the parameters passed to the range query function or method meet expectations, such as the type, quantity, value range, etc. of the parameters; for some optional parameters, such as parameters that limit the number of returned results, validity checks can also be performed. In step 1102, the MemTable can be read from the memory, and it can be checked whether the conditions of the range query are fully covered by the MemTable. If they are fully covered, the process ends; if the conditions of the range query are not fully covered by the MemTable, step 1103 can be executed to read the SST data of each layer in turn to cover the conditions of the range query. In step 1104, files that meet the range query conditions can be selected and read. This step is the core step of the range query, which can be executed in a near-storage computing device, and can mainly include file matching based on Bloom filters (key comparison and key matching). In step 1105, the results can be summarized and the results read by the range query are returned to the upper-layer application. In the process of executing range query, the CXL-PNM computing core can be used instead of the CPU to perform range query, so as to avoid a large amount of data transmission between the CPU and DRAM. In addition, by pre-calculating, it can be ensured that less useful data needs to be transmitted to the upper layer and less data redundancy, so as to obtain the following advantages: some useless data does not need to be transmitted, reducing data transmission loss.

[0111] Fig.12 A schematic diagram of a system architecture for extending PNM in an exemplary embodiment of the present disclosure is shown.

[0112] Reference Fig.12 According to an exemplary embodiment of the present disclosure, the upper-layer application and RocksDB can exchange information about read operations and write operations through requests. After the host side of RocksDB detects that the trigger of file merging is triggered, and / or receives the instruction about range query from the upper-layer application, the SST data of file merging and / or range query can be read by the file reader respectively, and the computing tasks of file merging and / or range query are dispatched to the computing core (e.g., computing unit) in the CXL-based near memory computing device (CXL-PNM) through the task dispatching module (task dispatcher) for processing. The computing core of PNM can deploy the computing core of file merging and the computing core of range query respectively to process the computing tasks of file merging and range query respectively; in the process of executing the computing tasks, the computing core of file merging and the computing core of range query can interact with the CXL memory respectively. Fig.12 The solid line arrow in the middle box indicates the part that does not require the assistance of PNM. The process of this part can be the same as that in the related art. All data can be read into DRAM (memory), and data exchange can be performed between CPU and DRAM. Fig.12 The dotted arrows in the middle are the parts that need PNM assistance. The processes in these parts can read the data required for calculation into CXL memory, and the data exchange is carried out between CXL-PNM and CXL memory. As for RocksDB, the calculation process of merging the control file and the execution process of range query can be completed in the CXL-PNM device through appropriate application modification to adapt to this new architecture.

[0113] Fig.13 A schematic diagram showing the differences between the file merging and range query processes in the related art and the exemplary embodiment of the present disclosure is shown.

[0114] Reference Fig.13 In (a), in the related art, when executing file merging and range query, a large amount of data will be ① read from DRAM to the CPU, then ② the CPU will perform calculations on the read data, and finally ③ a large amount of calculation results will be returned to DRAM. For the process of file merging and range query, the process overhead is mainly in these two parts: ①③, which causes a large amount of data migration and serious consumption of CPU resources. It takes a lot of time to transfer data and also causes additional CPU time overhead.

[0115] Reference Fig.13In (b), according to an exemplary embodiment of the present disclosure, a CXL-based PNM device can be extended so that when performing file merging and range queries, the required data can be ① read from the CXL memory to the PNM device, so that the data calculation process can be ② performed in the PNM device, and the calculation result can be ③ returned to the CXL memory, so that less data (for example, the calculation result data rather than the data required for the calculation process) can be read from the CXL memory to the CPU at ④, ⑤ only scheduling work and integrating the final results can be performed in the CPU instead of executing the calculation process, and finally, ⑥ less data can be returned to CXL.

[0116] from Fig.13 It can be found that all data interactions in related technologies are carried out between DRAM and CPU, which will increase the resource overhead of CPU and DRAM. For the new computing architecture, a large amount of data copy movement occurs inside the CXL-PNM device. Internal data migration can save time and space overhead. In addition, transferring a large number of simple calculations to PNM can unload the task burden of the CPU, making the CPU's workload very small and improving the overall execution efficiency.

[0117] Fig.14 A schematic diagram showing the difference between using PNM assistance and not using PNM assistance in an exemplary embodiment of the present disclosure.

[0118] Reference Fig.14 In the related art, when PNM assistance is not used, the solid line marked ① is to transfer the data in DRAM to the persistent storage device and / or read the persistent data to DRAM; the solid line marked ② is for the CPU to read a large amount of data from DRAM, perform the calculation process by the CPU, and write the data to DRAM by the CPU. In the solid line marked ②, the calculation process performed by the CPU is a simple calculation, but the data interaction between the CPU and DRAM will consume a lot of time. If the calculation process performed by the CPU requires a large amount of data, this will cause a huge waste of CPU resources. It can be seen that the time spent in this process is not worth it because there is a serious mismatch between the time overhead and the calculation overhead.

[0119] Still refer to Fig.14According to an exemplary embodiment of the present disclosure, when PNM is used for assistance, at ① marked by the dotted line, data in the CXL-PNM memory can be transferred to a persistent storage device, and / or persistently stored data can be read from the persistent storage device to the CXL-PNM memory; at ② marked by the dotted line, a large amount of data movement is only for simple calculations, so the PNM can replace the CPU to undertake the calculation task, and the data can be moved only inside the CXL-PNM device (between the calculation unit of the PNM and the CXL memory); at ④, the CPU can only read the result data of the calculation task from the CXL memory, which is greatly reduced compared to reading all the data involved in the calculation, and the CPU can only perform the dispatch of calculation tasks and the processing of calculation results. The CPU can write less final result data to the CXL memory instead of exchanging all the data required for the calculation task with the memory.

[0120] It can be seen that by executing computing tasks by PNM, data copying, movement, and simple calculations can be offloaded from the CPU to the PNM device. PNM devices can perform simple calculations by using a closer physical space between the memory and the computing core to reduce the distance and time of data transmission, thereby improving system execution efficiency and memory bandwidth.

[0121] Fig.15 A block diagram of a computing task management apparatus for a database of a nearby computing device in an exemplary embodiment of the present disclosure is shown.

[0122] Reference Fig.15 The exemplary embodiment of the present disclosure also provides a computing task management device 1500 for a database of a near-storage computing device, which may include but is not limited to a data acquisition module 1501, a task execution module 1502, and a computing result transmission module 1503.

[0123] The data acquisition module 1501 can obtain the data required for the preset computing task from the host's memory in response to receiving an execution instruction regarding the preset computing task from the host, and store the data required for the preset computing task in the memory corresponding to the near-storage computing device, wherein the preset computing task may be a computing task that, when executed by the host, requires data exchange between the memory and the CPU at a frequency higher than a preset level and requires computing power lower than a preset computing power.

[0124] The task execution module 1502 can execute the preset computing task by exchanging data with the memory corresponding to the near-storage computing device to obtain the computing result corresponding to the preset computing task.

[0125] The calculation result transmission module 1503 can transmit the calculation result to the host.

[0126] It can be understood that the modules in the computing task management device 1500 for a database applied to a near-stored computing device are used to implement the computing task management method for a database applied to a near-stored computing device. The specific implementation process in the exemplary embodiment is roughly the same as the exemplary embodiment in the computing task management method for a database applied to a near-stored computing device, and will not be described repeatedly here. The computing task management device 1500 for a database applied to a near-stored computing device can be configured as software, hardware, firmware, or any combination of the above items to perform specific functions. For example, these modules in the device may correspond to pure software code, or to a dedicated integrated circuit, or to a module that combines software and hardware. In addition, one or more functions implemented by these modules in the device may also be uniformly executed by components in a physical entity device (e.g., a processor, a client, or a server, etc.).

[0127] Fig.16 A block diagram showing a computing task management system of a database in an exemplary embodiment of the present disclosure is shown.

[0128] Reference Fig.16 The exemplary embodiment of the present disclosure further provides a database computing task management system 1600 , which may include a host 1601 and a near storage computing device 1602 .

[0129] The host 1601 may send an execution instruction regarding a preset computing task to the nearby computing device 1602 .

[0130] The near-memory computing device 1602 can obtain the data required for the preset computing task from the memory of the host 1601 in response to receiving an execution instruction of the preset computing task from the host 1601, and store the data required for the preset computing task in the memory corresponding to the near-memory computing device 1602, wherein the preset computing task may be a computing task that, when executed by the host 1601, requires data exchange between the memory and the CPU at a frequency higher than a preset level and requires computing power lower than the preset computing power.

[0131] The near-storage computing device 1602 can execute the preset computing task and obtain the computing result corresponding to the preset computing task by exchanging data with the memory corresponding to the near-storage computing device 1602 .

[0132] The local storage computing device 1602 may transmit the computing result to the host 1601 .

[0133] The host 1601 may receive the calculation results transmitted from the near storage computing device 1602 .

[0134] It can be understood that the computing task management system 1600 of the above-mentioned database is used to implement the computing task management method of the database applied to the near-storage computing device and the host. The specific implementation process in the exemplary embodiment is roughly the same as the exemplary embodiment in the computing task management method of the database applied to the near-storage computing device or the host, and will not be repeated here.

[0135] Fig.17 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0136] Reference Fig.17 The electronic device 1700 includes at least one memory 1701 and at least one processor 1702, wherein the at least one memory 1701 stores a set of computer executable instructions. When the set of computer executable instructions is executed by the at least one processor 1702, a computing task management method for a database according to an exemplary embodiment of the present disclosure is executed.

[0137] As an example, the electronic device 1700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1700 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0138] In the electronic device 1700, the processor 1702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0139] The processor 1702 may execute instructions or codes stored in the memory 1701, wherein the memory 1701 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0140] The memory 1701 may be integrated with the processor 1702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1701 and the processor 1702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1702 can read files stored in the memory.

[0141] In addition, the electronic device 1700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1700 may be connected to each other via a bus and / or a network.

[0142] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above-mentioned database computing task management method.

[0143] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.

[0144] Another embodiment of the present disclosure relates to a system including at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above-mentioned database computing task management method.

[0145] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.

[0146] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0147] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.

[0148] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which, when executed by a processor, implements the computing task management method of a database as described in any one of the above.

[0149] According to the database computing task management method, device, electronic device, storage medium, system and computer program product provided by the present disclosure, the near memory processing characteristics of the near memory computing device can be used to rebuild the computing architecture of storage models such as RocksDB, so as to offload the computing tasks involved in data operations such as file merging and range query in the storage engine that require frequent data copying and moving between the CPU and the memory but are only for the CPU to perform simple calculations from the CPU to the near memory computing device, so as to significantly reduce the amount of data read into the memory, reduce the memory burden, release the resource utilization of the CPU, reduce the data interaction between the CPU and the memory, reduce the impact of a large amount of data calculations on the performance of the storage engine, increase the memory bandwidth, and thus improve the execution performance and efficiency of the database system.

[0150] In addition, the architecture system disclosed in the present invention can be a new functional module directly added based on CXL on the design architecture of the original storage engine such as RocksDB to realize the hot-swappable function of computing units and memory expansion, so that CXL devices can be added / deleted online on the CXL server as needed without destroying the original system. Specifically, the near memory processing characteristics of the CXL-PNM storage device can be used to offload the computing tasks of file merging and range queries in the storage engine from the CPU to the CXL-PNM device to obtain an accelerated computing architecture disclosed in the present invention. The near memory computing solution based on CXL can significantly reduce the data interaction between the memory and the CPU in the storage engine, and further improve the performance of the overall storage engine.

[0151] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A method for managing computing tasks of a database, characterized in that: Applied to a near-storage computing device, the method comprises: In response to receiving an execution instruction for a preset computing task from a host, obtaining data required for the preset computing task from a memory of the host, and storing the data required for the preset computing task in a memory corresponding to the near-storage computing device, wherein the preset computing task is a computing task that, when executed by the host, requires data exchange between the memory and the CPU at a frequency higher than a preset degree and requires computing power lower than a preset computing power; Executing the preset computing task by exchanging data with the memory corresponding to the near-storage computing device to obtain a computing result corresponding to the preset computing task; The calculation result is transmitted to the host.

2. The computing task management method according to claim 1, characterized in that: The preset computing tasks include computing tasks involving at least one of the following items: range query and file merging for database space optimization.

3. The computing task management method according to claim 2, characterized in that: In a case where the preset computing task is a computing task involved in the file merging, in response to receiving an execution instruction on the preset computing task from the host, obtaining data required for the preset computing task from the memory of the host includes: When the host detects that the trigger condition for merging the files is met, an execution instruction for the preset computing task is received from the host, and in response to the execution instruction for the preset computing task, data required for the preset computing task is obtained from the host's memory, wherein the trigger condition includes a threshold condition.

4. The computing task management method according to claim 2, characterized in that: In the case where the preset computing task is a computing task involved in the file merging, the step of executing the preset computing task by exchanging data with the memory corresponding to the near storage computing device to obtain a computing result corresponding to the preset computing task includes: Determine the file to be merged for the file merge; Reading each key-value pair in the file to be merged from the memory corresponding to the near-storage computing device; Comparing the keys of the key-value pairs, and exchanging the key-value pairs to sort the key-value pairs in order of the keys; A sorted key-value pair sequence is obtained in the memory corresponding to the near-storage computing device.

5. The computing task management method according to claim 2, characterized in that: In a case where the preset computing task is a computing task involved in the range query, in response to receiving an execution instruction for the preset computing task from a host, obtaining data required for the preset computing task from a memory of the host includes: When the condition of the range query has been confirmed by the host, an execution instruction regarding the preset computing task is received from the host, and in response to the execution instruction of the preset computing task, data required for the preset computing task is obtained from the memory of the host, wherein the condition of the range query includes the starting value and the ending value of the query key.

6. The computing task management method according to claim 2, characterized in that: In the case where the preset computing task is a computing task involved in the range query, executing the preset computing task by exchanging data with the memory corresponding to the near storage computing device to obtain a computing result corresponding to the preset computing task includes: Reading a memory table from a memory corresponding to the near-storage computing device; Traversing the key-value pairs in the memory table, and when the condition of the range query is not completely covered by the memory table, reading the ordered string table that meets the condition of the range query from the memory into the memory corresponding to the near-memory computing device through the Bloom filter; All key-value pairs that meet the conditions of the range query are collected from the memory table and the ordered string table.

7. A computing task management device for a database, characterized in that: Applied to a near-storage computing device, the device comprises: The data acquisition module is configured to: in response to receiving an execution instruction for a preset computing task from a host, acquire data required for the preset computing task from a memory of the host, and store the data required for the preset computing task in a memory corresponding to the near-storage computing device, wherein the preset computing task is a computing task that, when executed by the host, requires data exchange between the memory and the CPU at a frequency higher than a preset degree and requires a computing power lower than a preset computing power; The task execution module is configured to: execute the preset computing task by exchanging data with the memory corresponding to the near-storage computing device to obtain a computing result corresponding to the preset computing task; The calculation result transmission module is configured to: transmit the calculation result to the host.

8. A database computing task management system, characterized in that: The system includes a host and a near-storage computing device, wherein: The host sends an execution instruction about a preset computing task to the near-storage computing device; The near-memory computing device, in response to receiving an execution instruction of the preset computing task from the host, obtains data required for the preset computing task from the memory of the host, and stores the data required for the preset computing task in a memory corresponding to the near-memory computing device, wherein the preset computing task is a computing task that, when executed by the host, requires data exchange between the memory and the CPU at a frequency higher than a preset degree and a computing power requirement lower than a preset computing power; The near-storage computing device executes the preset computing task and obtains the computing result corresponding to the preset computing task by exchanging data with the memory corresponding to the near-storage computing device; The near storage computing device transmits the computing result to the host; The host receives the calculation result transmitted from the near-storage computing device.

9. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the database computing task management method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to execute the database computing task management method according to any one of claims 1 to 6.

11. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the database computing task management method as described in any one of claims 1-6.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the computing task management method of a database as described in any one of claims 1 to 6 is implemented.

Images