Disk buffer setting method based on LSM tree

By setting up buffer areas and dynamically adjusting storage space in the LSM tree, the problems of low disk utilization and slowing throughput rates caused by massive data writing in traditional LSM technology are solved, and load balancing of data processing and efficiency of storage system are realized. promote.

CN119002808APending Publication Date: 2024-11-22CHONGQING UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411030162.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When traditional LSM technology instantly writes massive data, storage delays between different levels of disks lead to uneven data processing, resulting in low disk utilization and slowing down data processing throughput.

Method used

Set up a buffer area during the data storage process, and extract the margin data of the previous layer through data abstraction processing and preliminary sorting, and dynamically adjust the storage space to ensure that the data can be quickly written to the next layer of disk, avoid data blockage and improve data merging speed.

Benefits of technology

Real-time monitoring of storage resource usage, dynamic allocation of data storage, ensure load balancing, improve storage system efficiency and response speed, and prevent data blockage and loss.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002808A_ABST
    Figure CN119002808A_ABST
Patent Text Reader

Abstract

The invention relates to a disk buffer setting method based on an LSM tree, and belongs to the technical field of computer data processing. According to the method, aiming at the situation that a large amount of instantaneous data inrush possibly occurs in the data storage process, a buffer area is particularly arranged between disk storage of different hierarchies. The buffer areas can effectively extract the remaining data of the upper layer, so that the phenomenon of data congestion is relieved, and the data is subjected to numerical abstraction processing and preliminary sorting, so that the data can be written into a disk of the lower layer more quickly. According to the method, data blockage and loss are prevented, and meanwhile, the data merging speed of each layer under the condition of data overload is increased. According to the method, the congestion condition of overload data in disk storage is considered, the buffer areas are set by using different hierarchical blocks in the disk to relieve data overload and optimize data flow and storage efficiency, and the stability and efficiency of a data processing system are greatly enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of computer data processing and relates to a disk buffer setting method based on an LSM tree. Background Art

[0002] LSM (Log-Structured Merge-tree) is an ordered off-site update structure that optimizes data storage and management, and is particularly suitable for high-frequency write scenarios. This structure can effectively handle the instantaneous writing of massive data and ensure the carrying capacity during data processing. Traditional LSM technology reduces data redundancy by continuously writing data to disk and using hierarchical partitioning for data storage. It achieves data compression, reduces write amplification, and improves disk usage efficiency by merging the latest SST (Sorted String Table) file with one or more old SSTs.

[0003] Currently, in the face of instantaneous writing of massive data, the storage delay between different layers of the traditional LSM disk may lead to uneven data processing, resulting in extreme situations of idle or full load between different layers of the disk. This not only reduces the overall utilization of the disk, but also slows down the data processing throughput rate.

[0004] Therefore, a new processing method is urgently needed to improve the utilization efficiency of idle disks, avoid congestion during data processing, and enhance the data processing throughput rate and the overall utilization rate of disks. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a disk buffer setting method based on LSM tree, so as to ensure that the data processing system can avoid or alleviate the congestion in the data processing process and enhance the data processing throughput rate by improving the utilization efficiency of idle disks when massive data is written instantaneously.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A disk buffer setting method based on LSM tree, in order to deal with the instantaneous influx of data that may occur during data storage, a buffer area is set up between different levels of disk storage. These buffers can effectively extract the surplus data of the previous layer, thereby alleviating data congestion, and perform numerical abstraction and preliminary sorting of the data so that the data can be written to the next layer of disk more quickly. This method not only prevents data congestion and loss, but also improves the speed of merging data at each layer in the case of data overload.

[0008] The method specifically comprises the following steps:

[0009] S1: Initialize the data writing module: establish a data channel, complete computing resource allocation, and perform software security checks;

[0010] S2: caches data into WAL logs to improve data security;

[0011] S3: Serialize the data in step S2 to meet the disk writing requirements;

[0012] S4: importing the data processed in step S3 into the input amount judgment module to implement real-time monitoring of the amount of written data;

[0013] S5: Obtain the evaluation result of step S4 to determine whether an overload warning is triggered;

[0014] S6: After confirming that the data is overloaded, the remaining space determination module is started to check the remaining space of the disk memory layer by layer. If the data is not overloaded, the flush data writing is completed normally;

[0015] S7: According to the checking result of step S6, matching one or more disk layers with sufficient remaining space;

[0016] S8: Start the space allocation module to allocate data space according to different data sizes, different data types and storage characteristics of the levels;

[0017] S9: Write the data into the storage space obtained in step S8 to complete data storage.

[0018] Further, in step S5, the specific process for determining whether to trigger an overload warning is as follows: when data is written from memory to disk, the input quantity judgment module is first entered to monitor the data in real time. If the data input quantity exceeds the set threshold, the remaining space judgment module is entered to dynamically adjust the storage space. If the threshold is not reached, the data is written normally.

[0019] Furthermore, in step S6, the specific process of checking the remaining space of the disk memory layer by layer is as follows: if the data input amount exceeds the set threshold, then the lowest layer disk space is judged to see whether its remaining space size has reached the set lower limit; if the remaining space size is greater than the set lower limit, the memory data is normally flushed to the disk; if it is lower than the lower limit, then the disk space sizes in the subsequent layers are judged in turn to see whether they have reached the set lower limit; the layers with remaining space greater than the set lower limit are allocated appropriate disk space as a buffer zone for the lower-level blocks, so that the data can be buffered and preliminarily sorted at the same time, thereby effectively improving the utilization rate of the free disk space and data throughput at the high level, and effectively preventing data congestion caused by excessive instantaneous data volume; if the remaining space on the highest layer disk is still less than the set lower limit, a notification of insufficient memory is issued.

[0020] The beneficial effects of the present invention are:

[0021] 1) The present invention can monitor the usage of memory and other storage resources in real time and dynamically allocate data storage according to the current resource availability. When it is detected that the memory space is close to the limit, the data flow direction can be automatically adjusted to improve the utilization rate of storage resources.

[0022] 2) The present invention ensures that no single storage component or layer is overloaded by load balancing data storage. This can be achieved through distributed storage solutions or evenly distributing data among multiple storage devices.

[0023] 3) The present invention intelligently migrates data from memory to disk or other storage levels based on the data access pattern and life cycle, which can not only reduce the pressure on memory, but also improve the efficiency and response speed of the entire storage system.

[0024] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0026] Figure 1 A data storage framework diagram of a disk buffer setting method based on an LSM tree of the present invention;

[0027] Figure 2 Detailed flow chart for data storage;

[0028] Figure 3 Sets the sample graph for the buffer. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0031] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0032] See also Figure 1 to Figure 3 The present invention proposes a disk buffer setting method based on LSM tree. In view of the data congestion and data loss caused by the instantaneous writing of massive data, a data storage buffer mechanism is proposed, a dynamic storage space is established, the load balancing of data storage is achieved, and the efficiency and response speed of the data storage system are effectively improved. The method specifically includes the following steps:

[0033] S1: Initialize the data writing module: establish data channels, complete computing resource allocation, and perform software security checks;

[0034] S2: caches data into the WAL log to ensure data security;

[0035] S3: Serialize the data in step S2 to meet the disk writing requirements;

[0036] S4: Import the data processed in step S3 into the input amount judgment module to implement real-time monitoring of the written data amount.

[0037] S5: The judgment process of the input volume module will judge in real time whether the input data is overloaded. Specifically, when the data is written from the memory to the disk, it first enters the input volume judgment module to monitor the data in real time. If the data input volume exceeds the set threshold, it enters the remaining space judgment module to implement dynamic adjustment of the storage space. If it does not reach the threshold, the data is written normally.

[0038] S6: According to the result of step S5, the remaining space judgment module is entered to judge the remaining storage space of the storage device. Specifically, if the amount of data input exceeds the set threshold, the lowest disk space is judged to see whether the remaining space size reaches the set lower limit. If the remaining space size is greater than the set lower limit, the memory data is normally flushed to the disk. If it is lower than the lower limit, the disk space size in the subsequent layers is judged in turn to see whether it reaches the set lower limit. The layers with remaining space greater than the set lower limit are allocated a certain amount of disk space as a buffer for the lower blocks, so that the data can be buffered and preliminarily sorted at the same time, effectively improving the utilization rate of the free disk space at the high level and the data throughput, and effectively preventing data congestion caused by excessive instantaneous data. If the remaining space of the highest disk is still less than the set lower limit, a notification of insufficient memory is issued.

[0039] S7: According to the checking result of step S6, matching one or more disk levels with sufficient remaining space;

[0040] S8: Start the space allocation module to allocate data space according to different data sizes, different data types and storage characteristics of the levels;

[0041] S9: Write the data into the storage space obtained in step S8 to complete data storage.

[0042] The specific example method of the dynamic buffer (i.e., the remaining space determination module and the space allocation module) designed by the present invention is as follows: Figure 3 As shown in the figure, it is assumed that the current data input volume exceeds the threshold within a certain period of time and the remaining space in the lower-level blocks of the disk cannot accommodate the data at this time. The data storage and buffer setting process is as follows: data is input from the log WAL in the disk to the memory MemTable. When the data in the MemTable reaches a certain threshold, it will be converted into an ImmutableMemTable that can only be read, and then written to the disk through the Flush process. After determining that only the highest-level disk has remaining space, part of the disk space at this level is extracted as a buffer and given to the lower level, so that overloaded data can be buffered through the buffer when the lower-level disk cannot accommodate it. At the same time, preliminary data sorting is performed in the buffer to facilitate data writing to the next level. When the input data volume returns to normal or the remaining space in the lower-level disk can accommodate the written data, the buffer will be used as a normal disk again.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A disk buffer setting method based on LSM tree, It is characterized in that The method specifically comprises the following steps: S1: Initialize the data writing module: establish a data channel, complete computing resource allocation, and perform software security checks; S2: caches data into WAL logs to improve data security; S3: Serialize the data in step S2 to meet the disk writing requirements; S4: importing the data processed in step S3 into the input amount judgment module to implement real-time monitoring of the amount of written data; S5: Obtain the evaluation result of step S4 to determine whether an overload warning is triggered; S6: After confirming that the data is overloaded, the remaining space determination module is started to check the remaining space of the disk memory layer by layer. If the data is not overloaded, the flush data writing is completed normally; S7: According to the checking result of step S6, matching one or more disk layers with sufficient remaining space; S8: Start the space allocation module to allocate data space according to different data sizes, different data types and storage characteristics of the levels; S9: Write the data into the storage space obtained in step S8 to complete data storage.

2. The disk buffer setting method based on LSM tree according to claim 1, It is characterized in that In step S5, the specific process for determining whether to trigger an overload warning is as follows: when data is written from memory to disk, the input quantity judgment module is first entered to monitor the data in real time. If the data input quantity exceeds the set threshold, the remaining space judgment module is entered to dynamically adjust the storage space. If it does not reach the threshold, the data is written normally.

3. The disk buffer setting method based on LSM tree according to claim 1, It is characterized in that In step S6, the specific process of checking the remaining space of the disk memory layer by layer is as follows: if the data input amount exceeds the set threshold, the lowest layer disk space is judged to see whether its remaining space size has reached the set lower limit; if the remaining space size is greater than the set lower limit, the memory data is normally flushed to the disk; if it is lower than the lower limit, the disk space sizes in the subsequent layers are judged in turn to see whether they have reached the set lower limit; the layers with remaining space greater than the set lower limit are allocated appropriate disk space as a buffer zone for the lower-level blocks, so that the data can be buffered and preliminarily sorted at the same time; if the remaining space on the highest layer disk is still less than the set lower limit, an insufficient memory notification is issued.