LSM tree remote merging and adaptive resource unloading scheduling method based on serverless cloud function

Abstract

The invention discloses an LSM tree remote merging and self-adaptive resource unloading scheduling method based on a serverless cloud function, which is characterized in that the problems of chained compaction, resource competition and the like due to data repeated reading and writing in the traditional compaction operation are solved, and the reading and writing performance and merging efficiency of LSM tree storage in a large-scale writing scene are remarkably improved. The method comprises the following steps: when an LSM tree triggers a compaction, dividing SST files with overlapped key values in multiple layers into sub-tasks through a parallel sub-compaction algorithm, submitting the sub-tasks to a remote monitoring mechanism, and judging whether the sub-tasks are executed locally or unloaded to a cloud to execute the compaction in real time according to the current resource utilization rate and task requirements. And if local resources are insufficient or read-write competition exists, merging is carried out by utilizing a cloud compaction service realized based on serverless, and a merging result is stored in a temporary storage and is cleared after the client writes back. The method is adaptive to resource unloading and parallel task division, the write amplification problem is reduced, the merging efficiency and the system read-write performance are considered at the same time, and the method is suitable for optimization of the cloud key value storage system.

Description

Technical Field

[0001] The present invention belongs to the technical field of storage systems, and in particular to a method for remote merging and adaptive resource unloading scheduling of LSM trees based on Serverless cloud functions. Background Art

[0002] Modern data storage is facing three fundamental trends: first, the amount of stored data is increasing; second, in the data life cycle, the number of insert operations is significantly greater than the number of query operations; third, data management is gradually migrating to cloud-based solutions. Since most workloads are write-intensive, many applications have chosen the write-optimized log-structured merge tree (LSM tree for short) as the key-value storage system.

[0003] The LSM tree adopts a hierarchical structure design, usually including a memory component and multiple disk components. The basic idea is to write data in batches in memory and then write sorted large blocks of data to disk to achieve high write throughput. In this structure, the files on the disk are not modified in place, but the data at different layers are sorted and expired data is cleaned up through compaction operations, thereby reducing space amplification and query overhead. Usually, when calling a merge operation, the system needs to read all the files involved into memory, reorder the key-value pairs in order, and then write the results back to disk.

[0004] In application scenarios with heavy write loads, as new data continues to pour in, the LSM tree will periodically trigger compaction operations. However, compaction operations often generate a large amount of I / O and high CPU overhead, and compete fiercely with the continuously arriving read and write requests for system resources, which may cause a sharp drop in system performance and unstable throughput and response delay. Therefore, how to avoid temporary resource competition caused by periodic merging and ensure stable system throughput has become a key issue that needs to be solved urgently. The key-value storage system based on the cloud platform can dynamically adjust computing resources according to actual needs and provide sufficient CPU and memory resources when necessary, but there is still a resource competition problem when the LSM tree is directly deployed in the cloud environment. This is mainly because the triggering frequency of compaction operations in different time periods is not the same, and the elastic computing of the cloud platform usually expands resources at a coarse granularity, making it difficult to completely avoid conflicts between read and write operations and compaction while ensuring resource utilization efficiency.

[0005] In recent years, the emerging Serverless Computing has provided new ideas and opportunities for solving the above problems. Serverless architectures based on the Function-as-a-Service model, such as Lambda, Cloud Function, Azure Function, and OpenWhisk, can automatically scale services according to user needs and adopt a pay-as-you-go model. In Serverless computing, the most basic execution unit is the function. The client can register the corresponding function with the service provider and then call these functions through events or user requests. This architecture, with its high elasticity, high scalability, and precise resource utilization capabilities, has brought new technical breakthroughs to solving the resource competition problem of traditional LSM trees in cloud environments. However, how to reasonably combine Serverless cloud functions with LSM trees and fully utilize the advantages of Serverless cloud functions without reducing the performance of LSM trees is still a key challenge. Summary of the invention

[0006] In order to solve the above technical problems, the purpose of the present invention is to provide an LSM tree remote merging and adaptive resource unloading scheduling method based on serverless cloud functions. By subdividing the compaction operation into multiple subtasks and intelligently determining the execution location of the tasks, the coordinated scheduling of local and remote cloud resources is realized, thereby improving the throughput rate and response stability of the entire system.

[0007] The specific technical solution for achieving the purpose of the present invention is: A method for remote merging and adaptive resource unloading scheduling of LSM trees based on serverless cloud functions. The method combines the traditional LSM tree structure with serverless cloud function technology to realize dynamic unloading and parallel merging of compaction operations, including the following steps: (1) When the LSM tree triggers a compaction operation, the parallel sub-compaction algorithm selects SST files with overlapping key values ​​from the multi-layer data to be merged, and divides the SST files into several compaction subtasks according to the key range and submits them to the remote monitoring mechanism; (2) The LSM tree cloud storage sends the merge task to the remote monitoring mechanism, which dynamically determines whether the compaction operation is executed on the local node based on the real-time utilization of the current system CPU, memory, and network resources and the calculated resources required for the compaction task; (3) When the evaluation results show that local resources are sufficient, the compaction operation is performed directly locally; otherwise, the compaction task is offloaded to the cloud compaction service built on serverless cloud functions for execution; (4) After receiving the compaction task, the cloud compaction service starts an equal number of cloud function instances based on the same merge function as the local one, reads the corresponding data from the LSM tree data storage, performs key-value pair merge, and writes the merge result to the temporary storage; (5) After completing the compaction task, the cloud compaction service notifies the client of the execution result of the merge task; (6) After receiving the merge status signal, the client reads the merge result from the temporary storage, writes it back to the SST storage area, and clears the intermediate data in the temporary storage to complete the compaction operation.

[0008] Furthermore, the parallel sub-compaction algorithm specifically includes: (1) Select the SST file to be merged based on the set compaction trigger threshold; (2) Check the continuity of key values ​​between the SST to be merged and the SST files in the adjacent layer. If there are continuous overlapping keys, the adjacent SST files are included in the merging task at the same time; (3) Sort and deduplicate by key range, and divide the merge boundary into multiple non-overlapping areas based on data size, thereby forming several compaction subtasks to achieve parallel merging.

[0009] Furthermore, the remote monitoring mechanism specifically includes: (1) Collect the CPU, memory, and network bandwidth computing resource utilization of the current LSM tree node; (2) Dynamically evaluate the feasibility of executing tasks locally based on compaction task resource consumption, storage resource usage, and read and write operation growth rates; (3) When the resources required by the compaction task are lower than the remaining local computing resources and will not cause subsequent computing competition, it is executed locally; otherwise, it is offloaded to the cloud.

[0010] Furthermore, the cloud compaction service is implemented based on the serverless cloud function architecture and specifically includes: (1) Develop the required compaction functions in the local environment according to the platform provider's rules and deploy them to the serverless platform using command line tools. (2) The platform returns the URL corresponding to the function to the LSM tree client; (3) When the compaction task in the LSM tree needs to call the cloud compaction service, the client initiates the call through the URL of the function; (4) The serverless platform builds a compaction function based on the call request and performs the merge function.

[0011] Furthermore, the cloud compaction service has the following specific tasks: (1) Receive compaction offloading tasks from the remote monitoring mechanism; (2) Start the corresponding cloud merge function to perform key-value merge operations on the passed LSM tree data; (3) The merged results are temporarily stored in temporary storage, and the task execution status is fed back to the client in real time through the communication module.

[0012] The beneficial effects of the present invention include: By fine-grained division and intelligent scheduling of compaction tasks, the present invention utilizes the automatic expansion and parallel execution capabilities of serverless cloud functions to achieve effective separation of remote merge operations and local resources. This not only reduces local resource competition caused by compaction operations, but also avoids unnecessary waste of cloud resources, thereby significantly improving the overall throughput rate and response stability of the LSM tree system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the present invention; Figure 2 This is an example diagram of the data storage architecture of the LSM tree of the present invention; Figure 3 The figure is an example diagram of the execution process of the compaction operation proposed in the present invention. DETAILED DESCRIPTION

[0014] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0015] The first technical solution adopted by the present invention is: a remote merging method of LSM trees based on serverless cloud functions, comprising the following steps: When the LSM tree triggers a compaction operation, the parallel sub-compaction algorithm is called to scan the SST files at each layer.

[0016] Select the target SST file that meets the threshold condition and detect whether there is an SST file with overlapping key values ​​in the adjacent layer.

[0017] The selected SST file and its overlapping adjacent SST files are divided into merging tasks, and the merging tasks are split into several non-overlapping sub-merging tasks according to the key value interval.

[0018] Each sub-merge task is sent to the cloud compaction service, which automatically expands and starts the corresponding cloud function instance based on the serverless cloud function.

[0019] After receiving the task, the cloud function instance reads the target data from the LSM tree storage system according to the task range and performs a key-value pair merge operation.

[0020] During the merging process, the newly generated merged result is first written to temporary storage, and the task execution status is fed back to the client in real time.

[0021] After receiving the feedback signal, the client reads the merged result from the temporary storage, writes the data back to the original SST file storage area, and cleans up the intermediate data in the temporary storage.

[0022] The second technical solution adopted by the present invention is: an LSM tree adaptive resource unloading scheduling method based on serverless cloud function, comprising the following steps: Real-time monitoring of various resource usage of LSM tree nodes, including CPU, memory, disk I / O, and network bandwidth.

[0023] Estimate the computing and storage resources that may be occupied by the current compaction task based on the preset compaction task resource consumption model.

[0024] Compare the estimated resource requirements of the compaction task with the remaining local resources to determine whether it is possible to ensure that resource contention will not occur when the compaction operation is performed locally.

[0025] When local resources are sufficient and the read and write loads are stable, choose to perform compaction operations locally.

[0026] When local resources are insufficient or the resource demand of the compaction task is estimated to be high, the compaction task is offloaded to the cloud service based on serverless cloud functions.

[0027] After receiving the offload task, the cloud service automatically expands and starts the cloud function instance, improving the efficiency of the merge operation through parallel execution.

[0028] Based on real-time monitoring data and task feedback, the allocation ratio of local and cloud tasks is dynamically adjusted to ensure that the LSM tree maintains stable and efficient read and write performance during compaction. Example

[0029] This embodiment designs and implements a remote merging and adaptive resource offloading scheduling scheme for LSM trees based on serverless cloud functions, named SCCompaction. SCCompaction uses the elastic scalability of cloud functions on the serverless platform. To address the high computational overhead of compaction operations in LSM trees, it splits large-scale merging tasks into multiple subtasks through a parallel sub-compaction algorithm to achieve parallel processing. At the same time, it monitors the system and resource usage in real time through a remote monitoring mechanism, and dynamically determines whether the merge task meets the local execution conditions. When local CPU or memory resources are insufficient, or the task resource overhead is expected to be large, SCCompaction will offload some compaction tasks to the remote cloud for execution, thereby effectively reducing the local resource load and avoiding resource competition with foreground read and write tasks. The entire solution uses temporary storage of cloud merging results and a subsequent write-back mechanism to make the data merging process more flexible and efficient, significantly improving the overall write performance and system throughput of the LSM tree, while reducing unnecessary data transmission and resource waste.

[0030] See also Figure 1 This embodiment implements a remote merging method of LSM trees based on serverless cloud functions, including: When the LSM tree triggers a compaction operation, this embodiment uses a parallel sub-compaction algorithm to divide the SST files with overlapping key value ranges at different levels into subtasks. Specifically, the system first selects a number of SST files with overlapping key values ​​through the algorithm, and divides the overall compaction task into multiple compaction subtasks. These subtasks are then sent to the remote monitoring mechanism, which determines in real time whether to perform a compaction operation locally based on the current system resource utilization and the estimated computing resources that each task may consume.

[0031] If the judgment result shows that the current local resources are insufficient to meet the efficient execution requirements of the compaction task, the remote monitoring mechanism will offload the detected task to the cloud compaction service. On the remote node, the service starts the same cloud merge function as the local environment based on the passed task parameters, directly accesses the data in the corresponding LSM tree storage, and completes the merge operation with key-value pairs. After the merge process is completed, the system stores the compression results in temporary storage and sends the task execution status to the client through the notification mechanism. When the client receives the execution completion signal, it will immediately extract the merged data from the temporary storage, write it back to the location of the original storage SST file, and clear the data in the temporary storage after the write-back is completed, thereby completing a complete remote compaction operation.

[0032] An example diagram of an LSM tree data storage architecture for the serverless cloud function-based system is shown in Figure 2 , including read and write operations: All incoming write operations of the LSM tree are first added to the memory component, which includes two in-memory sorted jump tables, namely memtable and immutable memtable, which are used to store data in batches and complete sorting. When writing, the system first records the data in the write-ahead log (WAL) to ensure that the system can recover lost data after a crash; then, the data is written to the memtable in memory to achieve fast insert, update and delete operations, which only require adding a new key-value pair record to reflect the latest changes.

[0033] When the memtable reaches the preset capacity limit, the system converts it to an immutable memtable that cannot be modified anymore, and writes the data in it to the disk through the flush operation to generate an SST file at the L0 layer. Since the data refreshed directly from the memtable may have overlapping key values ​​in the SST file in L0, the modern LSM tree simplifies the operation by merging multiple components into new data blocks for the merge operation. In the disk storage layer, except for the L0 layer, the SST files of each layer are stored in sorted order, and the key value ranges of each file are ensured not to overlap through the compaction operation, which greatly improves the query efficiency. There is a preset limit on the total size of files in each layer on the disk, and its size threshold usually increases tenfold with the increase of the layer; when the total size of SST files at a certain level (for example, the Li layer) exceeds its limit, the system adopts the compaction strategy, selects an SST file in the layer, and merges and sorts it with all SST files with overlapping key value ranges in the next layer (Li+1 layer), and generates a new Li+1 layer SST file to maintain the specification of the storage structure.

[0034] In addition, if Figure 2 As shown in the figure, the LSM tree is organized into a tree structure, which not only improves the efficiency of point search and range query, but also ensures the reliability of the system through write ahead log. During the search operation, the system first searches in the memtable; if the target key is not found, it continues to search in the immutable memtable; if there is no hit in the memory component, it searches from the L0 layer of the disk to the Ln layer in sequence, and terminates the search when the first matching key is found. For range query (scan) operations, you can search for key values ​​in all ranges in parallel and merge the found data, thereby making full use of parallel computing to improve query efficiency.

[0035] The LSM tree architecture provided in this embodiment not only maintains the efficient writing and data recovery capabilities of the original design, but also implements key value sorting, non-overlapping key value ranges, and efficient query through tiered storage and compaction mechanisms, taking into account the collaborative work of memory and disk data, and meeting the requirements of modern storage systems for throughput and query response speed.

[0036] Figure 3 The specific execution process of the compaction operation is illustrated by example. The basic idea of ​​this embodiment is to merge the upper layer data blocks that reach the preset storage threshold with the data blocks with overlapping key value ranges in the lower layer to reduce the number of overall data blocks. During the merging process, the key values ​​in each data block are sorted and outdated data is deleted, thereby improving the utilization of disk space and query efficiency.

[0037] In this embodiment, Figure 3 The execution process shown specifically includes two steps: (a) When a layer (such as layer Li) reaches the merge trigger condition, the system selects a suitable SST file in the layer and retrieves all SST files in the next layer (layer Li+1) that overlap with the key value range of the selected SST file; in this process, the number of SST files expected to be generated in the next layer is also estimated. If the sum of the number of newly generated SST files and the number of original SST files exceeds the threshold of the layer, the SST files with overlapping key values ​​in the next layer are also added to the merge task until the number of SST files in the new layer after the merge meets the condition that compaction will not be triggered again. After determining all the SST files that need to be merged, the algorithm further extracts the corresponding merge boundaries, sorts and deduplicates these boundaries, and evenly divides them according to the overall data size, thereby splitting the original compaction task into multiple non-overlapping subtasks.

[0038] (b) Then, the selected SST files are merged and sorted to ensure that the new SST files generated after the merge maintain continuity and order in the key value order, and the merged results are stored in the Li+2 layer. This operation can effectively optimize the organizational structure of data blocks in the LSM tree, thereby reducing the additional data copy overhead caused by repeated writing while ensuring query performance. However, as the number of written key-value pairs increases, the data re-written in the upper layer will frequently trigger the merge operation of the next level, which may occupy a large amount of computing resources and cause chain merges.

[0039] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for remote merging and adaptive resource unloading scheduling of LSM trees based on serverless cloud functions, characterized in that: The steps include: (1) When the LSM tree triggers a compaction operation, the parallel sub-compaction algorithm selects SST files with overlapping key values ​​from the multi-layer data to be merged, and divides the SST files into several compaction subtasks according to the key range and submits them to the remote monitoring mechanism; (2) The LSM tree cloud storage sends the merge task to the remote monitoring mechanism, which dynamically determines whether the compaction operation is executed on the local node based on the real-time utilization of the current system CPU, memory, and network resources and the calculated resources required for the compaction task; (3) When the evaluation results show that local resources are sufficient, the compaction operation is performed directly locally; otherwise, the compaction task is offloaded to the cloud compaction service built on serverless cloud functions for execution; (4) After receiving the compaction task, the cloud compaction service starts an equal number of cloud function instances based on the same merge function as the local one, reads the corresponding data from the LSM tree data storage, performs key-value pair merge, and writes the merge result to the temporary storage; (5) After completing the compaction task, the cloud compaction service notifies the client of the execution result of the merge task; (6) After receiving the merge status signal, the client reads the merge result from the temporary storage, writes it back to the SST storage area, and clears the intermediate data in the temporary storage to complete the compaction operation.

2. The method according to claim 1, characterized in that: The parallel sub-compaction algorithm specifically includes: (1) Select the SST file to be merged based on the set compaction trigger threshold; (2) Check the continuity of key values ​​between the SST to be merged and the SST files in the adjacent layer. If there are continuous overlapping keys, the adjacent SST files are included in the merging task at the same time; (3) Sort and deduplicate by key range, and divide the merge boundary into multiple non-overlapping areas based on data size, thereby forming several compaction subtasks to achieve parallel merging.

3. The method according to claim 1, characterized in that: The remote monitoring mechanism specifically includes: (1) Collect the CPU, memory, and network bandwidth computing resource utilization of the current LSM tree node; (2) Dynamically evaluate the feasibility of executing tasks locally based on compaction task resource consumption, storage resource usage, and read and write operation growth rates; (3) When the resources required by the compaction task are lower than the remaining local computing resources and will not cause subsequent computing competition, it is executed locally; otherwise, it is offloaded to the cloud.

4. The method according to claim 1, characterized in that: The cloud compaction service is implemented based on the serverless cloud function architecture and specifically includes: (1) Develop the required compaction functions in the local environment according to the platform provider's rules and deploy them to the serverless platform using command line tools. (2) The platform returns the URL corresponding to the function to the LSM tree client; (3) When the compaction task in the LSM tree needs to call the cloud compaction service, the client initiates the call through the URL of the function; (4) The serverless platform builds a compaction function based on the call request and performs the merge function.

5. The method according to claim 1, characterized in that: The specific tasks of the cloud compaction service include: (1) Receive compaction offloading tasks from the remote monitoring mechanism; (2) Start the corresponding cloud merge function to perform key-value merge operations on the passed LSM tree data; (3) The merged results are temporarily stored in temporary storage, and the task execution status is fed back to the client in real time through the communication module.

Images