Data storage method and device and nonvolatile storage medium

By optimizing the selection and cutting boundaries of merged files in the LSM-tree storage system, and adopting reverse merging and corner alignment strategies, the write amplification problem of the LSM-tree storage system when merging data is solved, thereby improving the system's read and write performance and extending the lifespan of the storage media.

CN118981283APending Publication Date: 2024-11-19CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411048157.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

LSM-tree storage systems suffer from severe write amplification issues when merging data, leading to reduced system read and write performance.

Method used

By using reverse merging and corner alignment methods, the selection and cutting boundaries of merged files are optimized, and write amplification is reduced. Specifically, this involves obtaining the migration files of the first target layer, determining the files to be merged in the second target layer, merging them into a third migration file, and cutting them into multiple fourth migration files based on the file boundaries of the third target layer to migrate to the next layer.

Benefits of technology

It effectively reduces I/O consumption of merging operations, lowers write amplification, improves system read and write performance, and extends the lifespan of the underlying storage medium.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981283A_ABST
    Figure CN118981283A_ABST
Patent Text Reader

Abstract

The invention discloses a data storage method and device and a nonvolatile storage medium. The method comprises the following steps: acquiring a first migration file in a first target layer, and determining a to-be-merged file in a second target layer according to the first migration file; determining a second migration file according to the to-be-merged file, and merging the first migration file, the second migration file and the to-be-merged file into a third migration file; determining a cutting boundary of a third migration file according to the boundary of the file in the third target layer, cutting the third migration file into a plurality of fourth migration files according to the cutting boundary, and migrating the plurality of fourth migration files to the second target layer, the third target layer being the next layer of the second target layer. The boundaries are the minimum value and the maximum value of the data value range of the file. The technical problem that the read-write performance of the system is reduced due to serious write amplification during data merging of the storage system is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a data storage method, device and non-volatile storage medium. Background Art

[0002] Persistent kv (key-value) storage plays a key role in many data-intensive applications, including web indexing, e-commerce platforms, image storage, social platforms, etc. Kv storage provides basic support for these businesses such as efficient interpolation and random search. For write-intensive tasks, the kv storage system with LSM-tree (Log-Structured Merge-tree) as the underlying data structure has become the recognized optimal solution today. Distributed and local file storage systems built on LSM-tree are widely deployed in large-scale production environments, such as Google's BigTable and LevelDB, Facebook's HBase and RocksDB, etc.

[0003] The main advantage of LSM-tree over other index structures is that it can maintain a sequential access write mode for write operations, and can write kv pairs to disk sequentially. At the same time, in order to improve read performance, LSM-tree continuously compresses (compaction) in the background to maintain the order of data. However, this will cause the same data to be read and written many times, causing serious write amplification problems.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data storage method, device and non-volatile storage medium to at least solve the technical problem of reduced system read and write performance due to severe write amplification when a storage system merges data.

[0006] According to one aspect of an embodiment of the present application, a data storage method is provided, including: obtaining a first migration file in a first target layer, determining a file to be merged in a second target layer based on the first migration file, wherein the first target layer is a layer that needs to merge data into a next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; determining a second migration file based on the file to be merged, merging the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included in the data value range of the file to be merged; determining a cutting boundary of the third migration file based on the boundary of the file in the third target layer, cutting the third migration file into a plurality of fourth migration files based on the cutting boundary, and migrating the plurality of fourth migration files to the second target layer, wherein the third target layer is a layer below the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file.

[0007] Optionally, obtaining a first migration file in the first target layer, and determining the file to be merged in the second target layer based on the first migration file includes: determining the file to be merged based on a range of key-value pairs stored in the first migration file, wherein the range of key-value pairs stored in the file to be merged overlaps with the range of key-value pairs stored in the first migration file.

[0008] Optionally, determining the second migration file based on the file to be merged, and merging the first migration file, the second migration file and the file to be merged into the third migration file includes: obtaining the range of key-value pairs stored in the file to be merged, comparing other files in the first target layer based on the range of key-value pairs stored in the file to be merged, wherein the other files are files in the first target layer other than the first migration file; determining other files whose range of stored key-value pairs is included in the range of key-value pairs stored in the file to be merged as the second migration file; sorting the key-value pair data in the first migration file, the second migration file and the file to be merged, and merging them into the third migration file.

[0009] Optionally, based on the boundary of the file in the third target layer, a cutting boundary of the third migration file is determined, and the third migration file is cut into multiple fourth migration files based on the cutting boundary. Migrating the multiple fourth migration files to the second target layer includes: aligning the cutting boundary with the boundary of the file in the third target layer when the amount of data stored in the fourth migration file obtained by cutting based on the cutting boundary does not exceed the maximum allowed data amount of the file; cutting the third migration file into multiple fourth migration files based on the cutting boundary, and inserting the multiple fourth migration files into the second target layer in sequence.

[0010] Optionally, when the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowable data amount of the file, aligning the cutting boundary with the boundary of the file in the third target layer includes: when the boundary determined according to the maximum allowable data amount of the file falls within the range of key-value pairs stored in the file in the third target layer, determining the cutting boundary as the maximum value of the key-value pairs stored in the first adjacent file of the file in the third target layer, wherein the first adjacent file is a file among files adjacent to the file and the maximum value of the key-value pairs stored in the file is less than the minimum value of the key-value pairs stored in the file.

[0011] Optionally, when the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowed data amount of the file, after determining that the cutting boundary is aligned with the boundary of the files in the third target layer, the method also includes: when the amount of data stored in the second target layer reaches the maximum capacity allowed by this layer, migrating the files of the second target layer to the third target layer; when the boundaries of the files in the second target layer are aligned with the boundaries of the files of the third target layer, directly inserting the files that do not overlap with the data value range of the files of the third target layer into the third target layer.

[0012] Optionally, after migrating the plurality of fourth migration files to the second target layer, the method further includes: deleting the first migration file, the second migration file and the file to be merged.

[0013] According to another aspect of an embodiment of the present application, a data storage device is also provided, including: a first determination module, used to obtain a first migration file in a first target layer, and determine a file to be merged in a second target layer based on the first migration file, wherein the first target layer is a layer in the storage system that needs to merge data to the next layer when the amount of data stored reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; a second determination module, used to determine a second migration file based on the file to be merged, and merge the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included by the data value range of the file to be merged; a cutting migration module, used to determine a cutting boundary of the third migration file based on the boundary of the file in the third target layer, cut the third migration file into multiple fourth migration files based on the cutting boundary, and migrate the multiple fourth migration files to the second target layer, wherein the third target layer is a layer below the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute a data storage method.

[0015] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the data storage method is executed when the program is run.

[0016] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the data storage method is implemented.

[0017] In an embodiment of the present application, a first migration file in a first target layer is obtained, and a file to be merged in a second target layer is determined based on the first migration file, wherein the first target layer is a layer that needs to merge data to a next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; based on the file to be merged, a second migration file is determined, and the first migration file, the second migration file and the file to be merged are merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is overlapped with the data value range of the first migration file. The data value range of the file includes the files; according to the boundary of the file in the third target layer, the cutting boundary of the third migration file is determined, and the third migration file is cut into multiple fourth migration files according to the cutting boundary, and the multiple fourth migration files are migrated to the second target layer, wherein the third target layer is the next layer of the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file. By improving the selection rules of the merged file and the split boundary, the purpose of reducing write amplification is achieved, thereby achieving the technical effect of improving the read and write performance of the system, and then solving the technical problem of reduced system read and write performance caused by severe write amplification when the storage system merges data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a structural schematic diagram of a computer terminal (mobile device) provided according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of an LSM-tree architecture provided according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a specific process of data merging provided according to an embodiment of the present application;

[0022] Figure 4 It is a flowchart of a data storage method provided according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a reverse merger provided according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a traditional merge without a reverse merge operation provided according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a corner alignment process provided according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a specific process of merging without corner alignment provided according to an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of the structure of a data storage device provided according to an embodiment of the present invention; DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0031] Log-Structured Merge-tree (LSM-tree): A data structure used to implement efficient key-value storage. It is usually used in scenarios that require high-performance writing and reading, such as database systems, distributed storage systems, etc.

[0032] Memtable: It is a data structure used in databases and key-value storage systems. It is an important component in the LSM tree (Log-Structured Merge-Tree). It is a collection of key-value pairs stored in memory and used to temporarily store newly written data. During the writing process of the LSM tree, new data is first written to the MemTable. This allows for quick response to write operation requests because the read and write speed of memory is much faster than that of disk. As data is continuously written, when the MemTable reaches the preset size limit or other triggering conditions, it will be marked as immutable and trigger a process: the data in the MemTable will be refreshed to the disk to form a new SSTable (Sorted String Table). At this point, the data in the MemTable will no longer accept any modifications, and new write operations will point to a new, empty MemTable. This collaborative working mechanism between memory and disk enables the LSM tree to maintain high performance when processing large-scale data writes while ensuring persistent storage of data. The design of the MemTable is the key to optimizing write performance and balancing read performance in the LSM tree.

[0033] Immutable: A type of persistent data, which means that once the data is written, it can no longer be modified or deleted. This immutability is one of the core features of the LSM tree design, which allows the system to perform write operations by appending new data to the MemTable or creating a new SSTable without interfering with existing data. When the Immutable reaches a certain size or meets other conditions, it will be written to the disk to form a new immutable SSTable. This immutable feature simplifies data management because it avoids in-place updates in the data structure, thereby reducing the complexity of concurrency control and data consistency. At the same time, immutability also facilitates data persistence and recovery because the system can merge and compress data in the background without affecting ongoing read and write operations.

[0034] SSTable (Sorted String Table): A persistent data structure for storing structured data, which plays a key role in the Log Structure Merge Tree (LSM Tree). SSTable stores data in the form of key-value pairs, and all keys are sorted to ensure that data can be read and found efficiently. When the in-memory MemTable reaches a certain size or triggers specific conditions, it is written to disk and converted into an immutable SSTable file. SSTable optimizes read performance by reducing the number of disk I / O operations, while supporting periodic merge operations to reclaim disk space and reduce read amplification. This structure makes SSTable very suitable for write-intensive applications such as NoSQL databases and large-scale data storage systems.

[0035] Write amplification: refers to the phenomenon that when performing a write operation on a storage device, the amount of data actually written by the system is greater than the amount of data originally requested for the write. This is usually caused by several factors: first, many storage devices use blocks or pages as the basic write units of data. If the user's write request is smaller than these units, the storage device may need to erase the entire block or page before writing new data. This process is called a "write hole"; second, certain operations in the storage system, such as garbage collection and data merging, may cause data to move between different blocks or pages, resulting in additional write operations; finally, some storage technologies, such as flash memory, require data migration in different locations to maintain the life and performance of the device. Write amplification not only reduces the write performance of the storage device, but may also shorten its service life.

[0036] KV data: Key-Value Data is a simple and flexible data storage model in which data is stored in the form of unordered key-value pairs, where each key is unique and associated with a value. This model is well suited for storing various types of data because it does not require a fixed schema or structure, making the storage and retrieval of data very flexible. The key-value data model is very useful in many application scenarios, including caching systems, configuration storage, session management, distributed storage solutions, and NoSQL databases. In key-value storage, data can be quickly retrieved by its key, which makes it very valuable for applications that require high-speed access and processing of large amounts of data. In addition, the key-value data model supports horizontal expansion, allowing the system to increase storage capacity and processing power by adding more nodes, making it a popular choice for large-scale distributed systems.

[0037] I / O operations: (Input / Output Operations) refer to the process of reading and writing data in a computer system. It is the basic way to exchange information between devices and computer processors. I / O operations involve various types of devices, such as hard drives, solid-state drives, network interfaces, input devices (such as keyboards, mice), and output devices (such as monitors, printers). Read operations (input) refer to the process of obtaining data from external devices and transferring them to computer memory, while write operations (output) refer to the process of sending data from memory to external devices. The efficiency of I / O operations is crucial to system performance because they directly affect data access speed and application response time. In modern computer systems, in order to improve the performance of I / O operations, a variety of technologies are used, such as buffering, caching, direct memory access (DMA), and asynchronous I / O. In addition, I / O operations are also affected by factors such as operating system scheduling, device drivers, and hardware performance.

[0038] In the related art, the architecture of LSM-tree is as follows Figure 2 As shown, the system writes data sequentially to the memory. When the memtable in the memory reaches the limit, it becomes immutable and is written to the L0 layer as an SSTable. At this time, the SSTable of the L0 layer is out of order. When the L0 layer reaches its limit, it will be merged with the lower L1 layer. The data of the L0 layer will be sorted and then merged with the L1 layer. The other layers will also be merged with the lower layers after reaching the limit. Figure 3 The specific process of data merging is shown. Usually, when merging, the target layer (L i Layer) selects an SSTable, and then in the next layer of the target layer (L i+1 Layer) finds all SSTables whose key range overlaps with the SSTable, and finally parses the kv data in these SSTables to regenerate the SSTable, cuts it according to the target size limit of the SSTable of the next layer of the target layer, and writes the cut SSTable to the next layer of the target layer, and the old SSTable files will be deleted.

[0039] Among them, the significance of the merging process is to maintain the data in each layer in order according to the size of the key, so that each layer can be accessed at most once when querying the key, thereby ensuring the read performance. However, the merging operation will also introduce a lot of I / O operations, causing write amplification of the system. The original purpose of the system is only to write the data of the upper layer to the lower layer, but in order to keep the lower layer data arranged in key order, it is necessary to read the data overlapping with the upper layer key from the lower layer. This causes the upper layer to migrate an SSTable file to the lower layer, but actually writes multiple SSTable files to the lower layer, so there is a write amplification problem. When the system stores a lot of data, the write amplification of the system will become particularly serious. Severe write amplification will affect the write performance of the system. Since Immutable is converted into SSTable and written to L0 very quickly, if there are a lot of SSTable files in L0 and they are not merged to L1 in time, the write speed of the system will be affected. For example, under the default configuration of LevelDB, when the number of SSTable files in L0 exceeds 8, the write speed of the front-end data will be controlled, and when the number of SSTable files exceeds 12, the writing of the front-end data will be blocked. From this point, it can be seen that the front-end write requests and merge operations will compete for the system's I / O resources, so it is very necessary to reduce the I / O resources occupied by the merge operation, that is, to reduce the system's write amplification.

[0040] In order to solve this problem, a relevant solution is provided in the embodiments of the present application, which is described in detail below.

[0041] According to an embodiment of the present application, a method embodiment of a data storage method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing the data storage method is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data storage method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the vulnerability detection method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0047] In the above operating environment, the embodiment of the present application provides a data storage method, such as Figure 4 As shown, the method comprises the following steps:

[0048] Step S402, obtaining a first migration file in the first target layer, and determining a file to be merged in the second target layer according to the first migration file, wherein the first target layer is a layer that needs to merge data to the next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file;

[0049] In some embodiments of the present application, when the Li layer is merged into the Li+1 layer, after selecting an SSTable file from Li, all SSTables that overlap with the keys in the SSTable are selected from Li+1 as files to be merged.

[0050] Step S404, determining a second migration file according to the file to be merged, and merging the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included in the data value range of the file to be merged;

[0051] In some embodiments of the present application, after determining the files to be merged, we record the key value ranges of all the files to be merged at this time. If the key range of an SSTable file in the Li layer, that is, the first target layer, is completely included in this range, then this SSTable file is also selected to participate in the merge operation, which is a reverse merge of the files.

[0052] Specifically, Figure 5 A schematic diagram of a reverse merger is shown, such as Figure 5As shown in the figure, when selecting SST1 from the Li layer to merge with the lower layer, it is necessary to select SST3, SST4, and SST5 files from Li+1, because the keys in these SSTables overlap with the keys of the SST1 file. The key range in the SSTable file to be merged is [10,480], and the key range of the SST2 file in Li is [320,450], which is exactly included, so the SST2 page is included in this merge operation.

[0053] Through the reverse merge operation, the I / O consumption caused by the merge can be reduced. Specifically, Figure 6 FIG. 1 shows a schematic diagram of a traditional merge without a reverse merge operation. When selecting a merge file, the traditional merge without a reverse merge operation only merges SST1, and SST2 is not included in the merge operation. Then SST2 can only be selected in the subsequent merge from Li to Li+1. Figure 6 As shown in the figure, when only SST1 is selected for the merge operation, the files read are SST1, SST3, SST4 and SST5. After Li and Li+1 are merged, Li+1 generates new files: SST6, SST7, SST8 and SST9, and the previous SST1, SST3, SST4 and SST5 files are all deleted. At this time, the key ranges of SST7, SST8, SST9 in Li+1 overlap with SST2. If SST2 is selected in the next merge from Li to Li+1, SST7, SST8, SST9 files need to be selected in Li+1, then the files that need to be read for the merge are SST2, SST7, SST8 and SST9. In the actual system setting, the size of each SST file is similar. If the merge is divided into two times, that is, SST1 is merged first and then SST2 is merged, the total number of files that need to be read is 8 SST files. When there are fewer update and deletion operations, almost the same amount of writes will be generated. If the reverse merge strategy is adopted, SST1 and SST2 will be merged together, and only 5 SST files need to be read in total, which will reduce the reading and writing of 3 SST files. For actual systems, there are millions of SST files in the system, and reverse merging will undoubtedly effectively reduce the I / O consumption caused by merging.

[0054] Step S406, determining the cutting boundary of the third migration file based on the boundary of the file in the third target layer, cutting the third migration file into multiple fourth migration files based on the cutting boundary, and migrating the multiple fourth migration files to the second target layer, wherein the third target layer is the next layer of the second target layer, and the boundary is the minimum and maximum values ​​of the data value range of the file.

[0055] In some embodiments of the present application, in the merging operation, a corner alignment method is used to determine the cutting boundary of the third migration file. Figure 7 The specific corner alignment process is shown in FIG. Figure 7 As shown in the figure, when the SST1 file of L0 is merged downward to L1, the merged SStable is no longer cut according to the target size limit of the SSTable on L1, but is cut according to the boundary of the SSTable on the lower layer L2. When generating SST3, it is considered to align with the SSTable in L2 as much as possible, so that SST3 only overlaps with SST6 and SST7, and the data of D1 does not need to be merged twice. In addition, after the right boundary of SST4 is aligned with SST7, SST4 does not need to be sorted and compressed when it is merged downward, and can be moved directly from L1 to L2, which reduces write amplification.

[0056] Without the corner-aligned merge strategy, the merge operation will lead to an increase in write amplification. Specifically, Figure 8 The specific process of merging without corner alignment is shown in FIG. Figure 8 As shown, when the amount of L1 data reaches the upper limit, SSTable will be selected to merge to L2. In the case where SST3 is selected to merge downward, since the key value range of SST3 overlaps with multiple other SSTables, these SSTable files need to participate in the merge operation. Specifically, since the key value ranges of SST5 overlap with those of SST6 and SST7, these SST files will all participate in this merge, because SST2 overlaps with SST5, and SST4 overlaps with SST7. Then when SST2 and SST4 are compressed downward, the two segments of data D1 and D2 will be taken out and merged with SST2 or SST4, making the write amplification higher. In the absence of a corner-aligned merge strategy, the merge operation may lead to an increase in the write amplification rate. This is because multiple SSTable files need to be considered during the merge process, and there may be overlaps between these files, resulting in multiple write operations.

[0057] Through the above steps, the system's write amplification can be reduced and the system's read and write performance can be improved: the data writing during merging is reduced, effectively reducing write amplification without affecting read performance. The original structure of the LSM-tree is maintained, and the normal operation of other functions is not affected while reducing write amplification, solving the technical problem of reduced system read and write performance caused by severe write amplification when the storage system merges data. In addition, when the underlying storage medium is an SSD, the service life of the underlying storage medium can also be improved due to the reduction of write amplification.

[0058] The present application provides a data storage device. Fig. 9is a schematic diagram of the structure of the device, such as Fig. 9 As shown, the device includes: a first determination module 90, which is used to obtain a first migration file in a first target layer, and determine a file to be merged in a second target layer based on the first migration file, wherein the first target layer is a layer that needs to merge data to the next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; a second determination module 92, which is used to determine a second migration file based on the file to be merged, and merge the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included by the data value range of the file to be merged; a cutting migration module 94, which is used to determine a cutting boundary of the third migration file based on the boundary of the file in the third target layer, cut the third migration file into a plurality of fourth migration files based on the cutting boundary, and migrate the plurality of fourth migration files to the second target layer, wherein the third target layer is a layer below the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file.

[0059] In some embodiments of the present application, the first determination module 90 obtains the first migration file in the first target layer, and determines the file to be merged in the second target layer based on the first migration file, including: determining the file to be merged based on the range of key-value pairs stored in the first migration file, wherein the range of key-value pairs stored in the file to be merged overlaps with the range of key-value pairs stored in the first migration file.

[0060] In some embodiments of the present application, the second determination module 92 determines the second migration file based on the file to be merged, and merges the first migration file, the second migration file and the file to be merged into the third migration file, including: obtaining the range of key-value pairs stored in the file to be merged, and comparing other files in the first target layer according to the range of key-value pairs stored in the file to be merged, wherein the other files are files in the first target layer other than the first migration file; determining other files whose range of stored key-value pairs is included in the range of key-value pairs stored in the file to be merged as the second migration file; sorting the key-value pair data in the first migration file, the second migration file and the file to be merged, and merging them into the third migration file.

[0061] In some embodiments of the present application, the cutting migration module 94 determines the cutting boundary of the third migration file based on the boundary of the file in the third target layer, cuts the third migration file into multiple fourth migration files based on the cutting boundary, and migrates the multiple fourth migration files to the second target layer, including: aligning the cutting boundary with the boundary of the file in the third target layer when the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowed data amount of the file; cutting the third migration file into multiple fourth migration files based on the cutting boundary, and inserting the multiple fourth migration files into the second target layer in sequence.

[0062] In some embodiments of the present application, the cutting migration module 94 aligns the cutting boundary with the boundary of the file in the third target layer when the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowed data amount of the file, including: when the boundary determined according to the maximum allowed data amount of the file falls within the range of key-value pairs stored in the file in the third target layer, determining the cutting boundary as the maximum value of the key-value pairs stored in the first adjacent file of the file in the third target layer, wherein the first adjacent file is a file adjacent to the file and the maximum value of the key-value pairs stored in the file is less than the minimum value of the key-value pairs stored in the file.

[0063] In some embodiments of the present application, when the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowed data amount of the file, after determining that the cutting boundary is aligned with the boundary of the file in the third target layer, the method also includes: when the amount of data stored in the second target layer reaches the maximum capacity allowed by this layer, the files of the second target layer are migrated to the third target layer; when the boundaries of the files of the second target layer are aligned with the boundaries of the files of the third target layer, the files that do not overlap with the data value range of the files of the third target layer are directly inserted into the third target layer.

[0064] In some embodiments of the present application, after migrating the plurality of fourth migration files to the second target layer, the method further includes: deleting the first migration file, the second migration file, and the file to be merged.

[0065] It should be noted that the various modules in the above-mentioned data storage device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0066] The embodiment of the present application provides a non-volatile storage medium, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute the following data storage method: obtaining a first migration file in a first target layer, determining a file to be merged in a second target layer based on the first migration file, wherein the first target layer is a layer in which data needs to be merged to a next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; determining a second migration file based on the file to be merged, merging the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included in the data value range of the file to be merged; determining a cutting boundary of the third migration file based on the boundary of the file in the third target layer, cutting the third migration file into a plurality of fourth migration files based on the cutting boundary, and migrating the plurality of fourth migration files to the second target layer, wherein the third target layer is a layer below the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file.

[0067] An embodiment of the present application provides an electronic device, comprising: a memory and a processor, the processor being used to run a program stored in the memory, wherein the following data storage method is executed when the program is run: obtaining a first migration file in a first target layer, determining a file to be merged in a second target layer based on the first migration file, wherein the first target layer is a layer that needs to merge data to a next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; determining a second migration file based on the file to be merged, merging the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included in the data value range of the file to be merged; determining a cutting boundary of the third migration file based on the boundary of the file in the third target layer, cutting the third migration file into a plurality of fourth migration files based on the cutting boundary, and migrating the plurality of fourth migration files to the second target layer, wherein the third target layer is a layer below the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file.

[0068] The embodiment of the present application provides a computer program product, including a computer program, which implements the following data storage method when executed by a processor: obtaining a first migration file in a first target layer, determining a file to be merged in a second target layer based on the first migration file, wherein the first target layer is a layer in the storage system that needs to merge data to the next layer when the amount of data stored reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; determining a second migration file based on the file to be merged, merging the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included in the data value range of the file to be merged; determining a cutting boundary of the third migration file based on the boundary of the file in the third target layer, cutting the third migration file into a plurality of fourth migration files based on the cutting boundary, and migrating the plurality of fourth migration files to the second target layer, wherein the third target layer is a layer below the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file.

[0069] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0071] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0072] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0073] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0074] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data storage method, characterized in that: include: Obtain a first migration file in a first target layer, and determine a file to be merged in a second target layer according to the first migration file, wherein the first target layer is a layer that needs to merge data to a next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; Determine a second migration file according to the file to be merged, and merge the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included in the data value range of the file to be merged; Based on the boundary of the file in the third target layer, the cutting boundary of the third migration file is determined, and the third migration file is cut into multiple fourth migration files based on the cutting boundary, and the multiple fourth migration files are migrated to the second target layer, wherein the third target layer is the next layer of the second target layer, and the boundary is the minimum value and the maximum value of the data value range of the file.

2. The data storage method according to claim 1, characterized in that: Acquiring a first migration file in the first target layer, and determining the files to be merged in the second target layer according to the first migration file includes: The file to be merged is determined according to a range of key-value pairs stored in the first migration file, wherein the range of key-value pairs stored in the file to be merged overlaps with the range of key-value pairs stored in the first migration file.

3. The data storage method according to claim 1, characterized in that: Determining a second migration file according to the file to be merged, and merging the first migration file, the second migration file, and the file to be merged into a third migration file includes: Obtaining a range of key-value pairs stored in the file to be merged, and comparing other files in the first target layer according to the range of key-value pairs stored in the file to be merged, wherein the other files are files in the first target layer other than the first migration file; Determine other files whose stored key-value pairs are included in the range of the key-value pairs stored in the to-be-merged file as the second migration file; The key-value pair data in the first migration file, the second migration file and the file to be merged are sorted and merged into the third migration file.

4. The data storage method according to claim 1, characterized in that: Determining a cutting boundary of the third migration file according to a boundary of a file in the third target layer, cutting the third migration file into a plurality of fourth migration files according to the cutting boundary, and migrating the plurality of fourth migration files to the second target layer comprises: When the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowable amount of data of the file, aligning the cutting boundary with the boundary of the file in the third target layer; The third migration file is cut into a plurality of fourth migration files according to the cutting boundary, and the plurality of fourth migration files are inserted into the second target layer in sequence.

5. The data storage method according to claim 4, characterized in that: When the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowable amount of data of the file, aligning the cutting boundary with the boundary of the file in the third target layer includes: When the boundary determined based on the maximum allowable data size of the file falls within the range of key-value pairs stored in files in the third target layer, the cutting boundary is determined as the maximum value of the key-value pairs stored in the first adjacent file of the file in the third target layer, wherein the first adjacent file is a file adjacent to the file and in which the maximum value of the key-value pairs stored is less than the minimum value of the key-value pairs stored in the file.

6. The data storage method according to claim 4, characterized in that: When the amount of data stored in the fourth migration file obtained by cutting according to the cutting boundary does not exceed the maximum allowable amount of data of the file, after determining that the cutting boundary is aligned with the boundary of the file in the third target layer, the method further includes: When the amount of data stored in the second target layer reaches the maximum capacity allowed by this layer, the files in the second target layer are migrated to the third target layer; When the boundary of the file in the second target layer is aligned with the boundary of the file in the third target layer, the file whose data value range does not overlap with the file in the third target layer is directly inserted into the third target layer.

7. The data storage method according to claim 1, characterized in that: After migrating the plurality of fourth migration files to the second target layer, the method further includes: Delete the first migration file, the second migration file, and the file to be merged.

8. A data storage device, characterized in that: include: A first determination module is used to obtain a first migration file in a first target layer, and determine a file to be merged in a second target layer according to the first migration file, wherein the first target layer is a layer that needs to merge data to a next layer when the amount of data stored in the storage system reaches the maximum capacity allowed by the current layer, the second target layer is a layer below the first target layer, and the file to be merged is a file whose data value range overlaps with the data value range of the first migration file; A second determination module is configured to determine a second migration file based on the file to be merged, and merge the first migration file, the second migration file and the file to be merged into a third migration file, wherein the second migration file is a file whose data value range in the first target layer is included in the data value range of the file to be merged; A cutting migration module is used to determine the cutting boundary of the third migration file based on the boundary of the file in the third target layer, cut the third migration file into multiple fourth migration files based on the cutting boundary, and migrate the multiple fourth migration files to the second target layer, wherein the third target layer is the next layer of the second target layer, and the boundary is the minimum and maximum values ​​of the data value range of the file.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the data storage method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the data storage method according to any one of claims 1 to 7 is executed when the program is run.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 7 is implemented.