Data storage method, system and device based on log-structured merge tree

Abstract

The application discloses a data storage method, system and device based on a log-structured merge tree, obtains a target file and a target key value range to be stored, writes the target file into a memory storage space of the log-structured merge tree, detects whether a file storage quantity of the current memory storage space exceeds a first preset quantity threshold; if the file storage quantity of the memory storage space exceeds the first preset quantity threshold, a first segmented key value set corresponding to a first layer disk storage space in the log-structured merge tree is obtained; the files in the current memory storage space are merged, the merged files are segmented according to each first segmented key value pair, and at least one file to be stored is obtained; and each file to be stored is stored into a first storage sub-space according to a key value range corresponding to data in the file to be stored. The method can improve storage efficiency and utilization of storage space. The application can be widely applied in the technical field of data storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a data storage method, system and device based on a log-structured merge tree. BACKGROUND

[0002] In recent years, with the development of information technology, the amount of data storage, read and write is increasingly large, and the demand for managing distributed data is increasing. Therefore, the storage capacity of the database is also required to be higher. Among them, the excellent performance of the log-structured merge tree (LSM Tree) makes it become the underlying architecture of many popular key-value storage engines. The LSM Tree is used to optimize the write performance of the database. Compared with the traditional tree structure, it reduces the demand for random reading of the disk, thereby improving the write ability of the database to a certain extent.

[0003] In the related art, the LSM Tree is composed of two parts of memory and disk. The memory part is used to receive incremental updates, and the keys in the memory part are arranged in order. The disk part is used for persistent layered storage, and the data storage amount is larger as it goes to the lower layer. When storing data, the LSM Tree first writes all the key-value pairs (i.e., the corresponding relationship between the key and the data) into the memory. After the memory part is full, it is written into the SSTable file in the disk. The data in a SSTable file is arranged in order according to the key size, which facilitates fast query. When the storage space of a layer is exhausted, the disk part recycles redundant data through merging operation, and moves the data to the lower layer for storage. However, this merging method needs to process all the data of a layer before writing it into the next layer of storage space in the disk, and the processing efficiency is low. Moreover, since the data needs to be written into a layer after each merging, the data is repeatedly written, the write amplification (the amount of data written in the disk / the actual amount of data) is high, and the actual utilization rate of the storage space is reduced.

[0004] Therefore, the problems in the related art need to be solved. SUMMARY

[0005] The present application aims to at least partly solve one of the problems in the related art.

[0006] To this end, an object of the embodiments of the present application is to provide a data storage method based on a log-structured merge tree, which can improve the storage efficiency and the utilization rate of the storage space.

[0007] In order to achieve the above technical purpose, the technical solutions adopted by the embodiments of the present application include:

[0008] On the one hand, the embodiments of the present application provide a data storage method based on a log-structured merge tree, comprising:

[0009] obtaining a target file to be stored and a target key value range, the target file including a plurality of target data, the target key value range including a plurality of target key values, the target data and the target key values corresponding to each other in a one-to-one manner;

[0010] writing the target file into a memory storage space of a log-structured merge tree, and detecting whether a file storage quantity of the memory storage space exceeds a first preset quantity threshold;

[0011] if the file storage quantity of the memory storage space exceeds the first preset quantity threshold, obtaining a first segmented key value set corresponding to a first layer of disk storage space in the log-structured merge tree, the first segmented key value set including at least one first segmented key value, the first segmented key value being used to divide the first layer of disk storage space into a plurality of first storage subspaces, and each first storage subspace corresponding to storage of data of a first key value range;

[0012] merging files in the current memory storage space, segmenting the merged files according to each first segmented key value, and obtaining at least one to-be-stored file;

[0013] storing each to-be-stored file into the first storage subspace according to a key value range corresponding to data in the to-be-stored file.

[0014] In addition, the data storage method based on the log-structured merge tree according to the above embodiments of the present application can further have the following additional technical features:

[0015] Further, in an embodiment of the present application, the first segmented key value set is obtained by the following steps:

[0016] obtaining a total available storage key value list of the first layer of disk storage space;

[0017] randomly selecting a plurality of first segmented key values from the total available storage key value list to obtain the first segmented key value set.

[0018] Further, in an embodiment of the present application, the merging of the files in the current memory storage space includes:

[0019] detecting whether the data stored by each file has the same key value;

[0020] when the key values corresponding to the data stored by each file are all different, performing merging processing on the data stored by each file according to an arrangement order of the key values.

[0021] Further, in an embodiment of the present application, the merging of the files in the current memory storage space further includes:

[0022] When the data stored in various files contains the same key value, the same key value is identified as a merge key value.

[0023] Delete the data that corresponds to the merge key value and has an earlier storage time, and determine the data that corresponds to the merge key value and has a later storage time as the data currently corresponding to the merge key value;

[0024] The data stored in each file is merged according to the order of key values.

[0025] Furthermore, in one embodiment of this application, the method further includes:

[0026] Detect whether the number of files stored in the first-level disk storage space exceeds a second preset threshold.

[0027] If the number of files stored in the first-level disk storage space exceeds the second preset number threshold, obtain the second segment key value set corresponding to the second-level disk storage space in the log structure merging tree; the second segment key value set includes at least one second segment key value, which is used to divide the second-level disk storage space into multiple second storage subspaces, and each second storage subspace corresponds to storing data within the second key value range;

[0028] Merge the files in the current first-level disk storage space, and divide the merged files into segments according to the key values ​​of each second segment to obtain at least one file to be moved down;

[0029] Based on the key value range corresponding to the data in the files to be moved, each file to be moved is stored in the second storage subspace.

[0030] Furthermore, in one embodiment of this application, the first segmented key value set is a subset of the second segmented key value set.

[0031] Furthermore, in one embodiment of this application, the method further includes:

[0032] If the number of files stored in the memory storage space does not exceed the first preset number threshold, the target file is stored in the memory storage space.

[0033] On the other hand, embodiments of this application provide a data storage system based on a log structure merging tree, including:

[0034] The acquisition module is used to acquire the target file to be stored and the target key value range; the target file includes multiple target data, the target key value range includes multiple target key values, and the target data and the target key values ​​correspond one-to-one.

[0035] The detection module is used to write the target file into the memory storage space of the log structure merging tree and detect whether the number of files stored in the current memory storage space exceeds a first preset number threshold.

[0036] The judgment module is used to obtain the first segment key value set corresponding to the first layer disk storage space in the log structure merging tree if the number of files stored in the memory storage space exceeds the first preset number threshold. The first segment key value set includes at least one first segment key value, which is used to divide the first layer disk storage space into multiple first storage subspaces, and each first storage subspace corresponds to storing data within the first key value range.

[0037] The merge module is used to merge files in the current memory storage space, and to segment the merged files according to the key values ​​of each first segment to obtain at least one file to be stored.

[0038] The processing module is used to store each of the files to be stored into the first storage subspace according to the key value range corresponding to the data in the files to be stored.

[0039] On the other hand, embodiments of this application provide a computer device, including:

[0040] At least one processor;

[0041] At least one memory for storing at least one program;

[0042] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described data storage method based on a log structure merging tree.

[0043] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the above-described data storage method based on a log structure merging tree.

[0044] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:

[0045] The data storage method based on a log structure merging tree disclosed in this application involves obtaining a target file to be stored and a target key value range, writing the target file into the memory storage space of the log structure merging tree, detecting whether the number of files stored in the current memory storage space exceeds a first preset threshold; if the number of files stored in the memory storage space exceeds the first preset threshold, obtaining a first segment key value set corresponding to the first layer disk storage space in the log structure merging tree; the first segment key value set includes at least one first segment key value, which is used to divide the first layer disk storage space into multiple first storage subspaces, each of which stores data corresponding to a first key value range; merging the files in the current memory storage space, segmenting the merged files according to each first segment key value to obtain at least one file to be stored; and storing each file to be stored into the first storage subspace according to the key value range corresponding to the data in the file to be stored. This method implements the downward storage of data in the log structure merging tree by segmenting, which can greatly reduce write amplification and improve the actual utilization of storage space. During the downward movement of the file to be stored, the file only needs to be placed into the corresponding first storage subspace. The files in the first storage subspace do not need to be ordered. This way, there is no need to read the data in the file, but can be copied directly, thereby improving storage efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0047] Figure 1 This is a schematic diagram of a log structure merging tree provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the implementation environment for the data storage method based on log structure merging tree provided in the embodiments of this application;

[0049] Figure 3 This is a flowchart illustrating the data storage method based on a log structure merging tree provided in the embodiments of this application;

[0050] Figure 4 This is a schematic diagram of the stored data of a log structure merging tree provided in an embodiment of this application;

[0051] Figure 5This is a schematic diagram of newly added stored data in a log structure merging tree provided in this application embodiment;

[0052] Figure 6 This is a schematic diagram illustrating the merging of files in memory storage space according to an embodiment of this application;

[0053] Figure 7 This is a schematic diagram illustrating how to segment merged files according to an embodiment of this application;

[0054] Figure 8 This is a schematic diagram of the data storage system based on a log structure merging tree provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0056] The present application will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0057] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] A Log-Structured Merge Tree (LSM Tree) is a method of organizing ordered files or memory blocks using traditional indexes. Generally, an LSM Tree consists of two or more data storage structures. A basic LSM Tree comprises two parts: a memory portion, which can be any data structure that facilitates key-value lookups, and a disk portion, whose data structure is similar to a B-tree. Simply put, an LSM Tree is a forest spanning both memory and disk, containing multiple "subtrees." For example, see... Figure 1An LSM tree can be divided into multiple subtrees (N is a positive integer) of Level 0, Level 1, Level 2...Level n. Each subtree represents a storage level. Only Level 0 resides in memory, while Levels 1 through n are stored on disk. The Level 1 through Level n subtrees on disk are essentially files where data is sequentially written to disk in key-value order. Each subtree level has a file storage threshold; once the threshold is reached, a data merge operation is performed, and the merged result is written to the next level.

[0061] The data storage method, system, and device based on log structure merging tree provided in this application are specifically described through the following embodiments. First, the data storage method based on log structure merging tree in this application embodiment is described.

[0062] The data storage method provided in this application embodiment can be executed in applications involving shopping, games, social networking, navigation, and other fields. For example, in shopping applications, there may be a need to store user order information. In this case, the method provided in this application embodiment can be used to store relevant data involving user business information. Similarly, in social networking applications, there may be a need to store user conversation information. In this case, the method provided in this application embodiment can be used to store chat data between users.

[0063] Of course, it should be noted that the above application scenarios are merely illustrative and do not imply any limitation on the actual application of the methods in the embodiments of this application. Those skilled in the art will understand that the methods provided in the embodiments of this application can be used to perform specified tasks in different application scenarios.

[0064] In recent years, with the development of information technology, the amount of data stored and read / written has become increasingly massive, and the demand for managing distributed data has grown significantly. Consequently, higher demands have been placed on the storage capabilities of databases. Among these, the excellent performance of the Log Merge Tree (LSM Tree) has made it the underlying architecture of many popular key-value storage engines. LSM Tree emerged to optimize database write performance; compared to traditional tree structures, it reduces the need for random disk reads, thereby improving database write capabilities to some extent.

[0065] In related technologies, an LSM Tree consists of two parts: memory and disk. The memory part receives incremental updates, with keys arranged in an ordered manner. The disk part is used for persistent hierarchical storage, with the data volume increasing as you go down the hierarchy. When storing data, the LSM Tree first writes all key-value pairs (i.e., the correspondence between keys and data) into memory. Once the memory is full, it writes the data to an SSTable file on disk. The data in an SSTable file is arranged in order of key size for easy and fast lookup. When the storage space of one level is exhausted, the disk part reclaims redundant data through a merge operation and moves the data down the hierarchy. However, this merging method requires processing all the data in an entire level before writing it to the next level of storage space on disk, resulting in low processing efficiency. Moreover, since each merge requires rewriting the storage data of a level, a large amount of data is repeatedly written, leading to high write amplification (the amount of data written to disk / the actual amount of data), which reduces the actual utilization of storage space.

[0066] To address the problems existing in related technologies, embodiments of this application provide a data storage method, system, and device based on a log structure merging tree. The method involves obtaining a target file to be stored and a target key-value range; writing the target file into the memory storage space of the log structure merging tree; detecting whether the number of files stored in the current memory storage space exceeds a first preset threshold; if the number of files stored in the memory storage space exceeds the first preset threshold, obtaining a first segment key-value set corresponding to the first-level disk storage space in the log structure merging tree; the first segment key-value set includes at least one first segment key-value, which is used to divide the first-level disk storage space into multiple first storage subspaces, each of which stores data corresponding to a first key-value range; merging the files in the current memory storage space; segmenting the merged files according to each first segment key-value to obtain at least one file to be stored; and storing each file to be stored into the first storage subspace according to the key-value range corresponding to the data in the file to be stored. This method implements the downward storage of data in the log structure merging tree by segmenting, which can greatly reduce write amplification and improve the actual utilization of storage space. During the downward movement of the file to be stored, the file only needs to be placed into the corresponding first storage subspace. The files in the first storage subspace do not need to be ordered. This way, there is no need to read the data in the file, but can be copied directly, thereby improving storage efficiency.

[0067] Figure 2 This is a schematic diagram illustrating the implementation environment of the data storage method based on a log structure merging tree provided in this application embodiment. (Refer to...) Figure 2The main hardware and software components of this implementation environment include a user terminal 101 and a server 102, which are communicatively connected. The data storage method based on a log structure merging tree can be configured to be executed on the user terminal 101, on the server 102, or through interaction between the two. For example, when executed through interaction, the user terminal 101 can send the data to be stored to the server 102, and the server 102 can store the received data by executing the data storage method based on a log structure merging tree provided in this embodiment.

[0068] Specifically, the user terminal 101 in this application may include, but is not limited to, any one or more of the following: smartwatch, smartphone, computer, personal digital assistant (PDA), smart voice interaction device, smart home appliance, or in-vehicle terminal. The server 102 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The user terminal 101 and the server 102 can establish a communication connection via a wireless network or wired network. This wireless network or wired network uses standard communication technologies and / or protocols. The network can be the Internet or any other network, including but not limited to any combination of Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless network, private network, or virtual private network.

[0069] Of course, this is understandable. Figure 2 The implementation environment described in this application is merely one possible application scenario for the data storage method based on log structure merging tree provided in this embodiment; the actual application is not fixed. Figure 2 The software and hardware environment shown.

[0070] The following is combined Figure 2 The implementation environment shown will be used to provide a detailed description of the data storage method based on log structure merging tree provided in the embodiments of this application.

[0071] First, please refer to Figure 3 , Figure 3This is a flowchart illustrating the data storage method based on a log structure merging tree provided in this application embodiment. Figure 3 The data storage method based on the log structure merging tree shown is primarily executed on the server 102 side. This method can be applied to related computer devices on server 102, but is not limited to the above form. Figure 3 The methods in the process include, but are not limited to, steps 110 to 150.

[0072] Step 110: Obtain the target file to be stored and the target key value range; the target file includes multiple target data, the target key value range includes multiple target key values, and the target data and the target key values ​​correspond one-to-one;

[0073] In this step, when executing the data storage method in this embodiment, the data to be stored can first be obtained. For log structure merging tree, data storage generally requires converting the data into an SSTable (Sorted String Table) format file. An SSTable file is a format for storing key-value pairs, that is, processing the data into a string table where data and key values ​​correspond one-to-one, and then sorting the data according to the size of the key values, resulting in an ordered data storage structure that facilitates subsequent lookup and retrieval. In this embodiment, when storing data, a preprocessed SSTable file can be obtained and designated as the target file. Based on the foregoing description, it can be understood that the target file contains multiple data items, designated as target data, and the key values ​​corresponding to these target data items are designated as target key values. In this embodiment, the target key values ​​corresponding to these target data items can form a key value range. For example, when there are 20 data items, the key value corresponding to the first data item is 21, the key value corresponding to the second data item is 22, and so on, increasing to 40 for the 20th data item. The key value range corresponding to these data items is then 21 to 40. The range of target key values ​​can be denoted as the target key value range.

[0074] It should be noted that in this embodiment of the application, there are no restrictions on the size of the target file or the number of target data (i.e., the size of the target key value range), which can be flexibly adjusted according to specific needs.

[0075] Furthermore, it should be noted that in this embodiment of the application, there are no restrictions on the acquisition channels for the target file to be stored and the target key value range. They can be obtained directly by the user through human-computer interaction devices, or they can be obtained from other electronic devices and computer systems through data transmission interfaces or remote communication transmission.

[0076] It should be further noted that in the various specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user voice statements, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of this application obtained.

[0077] Step 120: Write the target file into the memory storage space of the log structure merging tree, and check whether the number of files stored in the current memory storage space exceeds the first preset number threshold.

[0078] In this step, after obtaining the target file to be stored, the target file can be written to the memory storage space of the log structure merging tree. In this embodiment, the memory storage space refers to the memory portion of the aforementioned log structure merging tree. The log structure merging tree uses both memory and disk storage to store data. When the memory storage space reaches its storage limit, a certain strategy can be used to sequentially flush the data from the memory storage space to the disk storage space, ensuring that the data is logically and physically stored in an ordered manner on the disk.

[0079] Specifically, since the storage capacity of the memory storage space itself is limited, when multiple files are written to the memory storage space, it may reach its storage limit and be unable to continue writing. Therefore, in this embodiment, a threshold number of files that the memory storage space can store can be preset, denoted as the first preset threshold number. Then, each time a new target file is written to the memory storage space of the log structure merging tree, the number of files stored in the current memory storage space can be detected. If the number of files stored in the current memory storage space does not exceed the first preset threshold number, it means that the storage capacity of the current memory storage space still has a lot of spare space and can continue to support file storage. At this time, the target file can be stored in the memory storage space. Conversely, if the number of files stored in the current memory storage space exceeds the first preset threshold number, it means that the storage capacity of the current memory storage space is already small and may not be able to continue to support file storage. At this time, the data in the memory storage space needs to be transferred to the disk storage space.

[0080] It should be noted that in the embodiments of this application, the specific size of the first preset quantity threshold is not limited. For example, in some embodiments, the size of the first preset quantity threshold can be 2 or 3. Of course, it can be flexibly adjusted according to actual needs.

[0081] Step 130: If the number of files stored exceeds the first preset number threshold, obtain the first segment key value set corresponding to the first layer disk storage space in the log structure merging tree; the first segment key value set includes at least one first segment key value, the first segment key value is used to divide the first layer disk storage space into multiple first storage subspaces, and each first storage subspace corresponds to storing data within the first key value range;

[0082] In this step, as mentioned earlier, if the number of files stored in the current memory storage space exceeds the first preset threshold, it indicates that the storage capacity of the current memory storage space is insufficient and may not be able to continue supporting file storage. In this case, it is necessary to transfer the data from the memory storage space to the disk storage space. Therefore, when it is determined that the number of files stored exceeds the first preset threshold, the corresponding transfer storage operation can be performed. Specifically, firstly, the first segment key-value set corresponding to the first level disk storage space in the log structure merging tree can be obtained. In this embodiment, it can be understood that the disk portion in the log structure merging tree can be divided into multiple levels of storage space, which can be sequentially denoted as the first level disk storage space, the second level disk storage space, the third level disk storage space... the nth level disk storage space (n is a positive integer) from top to bottom. For each level of disk storage space, it can be divided into multiple segments according to key values, and data can be stored in segmented form. For example, suppose the first level disk storage space can store data corresponding to 50 key values, where these 50 key values ​​are 1 to 50. The storage space can be segmented, for example, by pre-setting a first segment key value set, which includes at least one first segment key value. The first segment key value is the storage dividing point of the first-level disk storage space. In some embodiments, if the first segment key value set has only one first segment key value, and that first segment key value is 20, then the first-level disk storage space, which can store data with key values ​​from 1 to 50, can be divided into two segments, one for storing data with key values ​​from 1 to 19, and the other for storing data with key values ​​from 20 to 50. It should be noted that in this embodiment, the first segment key value can be assigned to either a segment with a larger key value or a segment with a smaller key value. For example, the first-level disk storage space, which can store data with key values ​​from 1 to 50, can be divided into two segments: one for storing data with key values ​​from 1 to 20, and the other for storing data with key values ​​from 21 to 50. Similarly, in some embodiments, if the first segment key value set has multiple first segment key values, the first-level disk storage space can be divided according to the above method. In this embodiment of the application, each segment of storage space obtained by dividing the first layer of disk storage space is called the first storage subspace. It can be understood that the number of first storage subspaces can be two or more, depending on the number of first segment key values. This application does not limit this.

[0083] In this embodiment, each of the divided first storage subspaces can be used to store data within a key value range. For example, if the first-level disk storage space, which can store data with key values ​​from 1 to 50, is divided into two segments—one for storing data with key values ​​from 1 to 20 and the other for storing data with key values ​​from 21 to 50—two first storage subspaces can be obtained. In this embodiment, the key value range corresponding to the data that can be stored in the first storage subspace is denoted as the first key value range. Thus, in the two first storage subspaces mentioned above, one corresponds to a first key value range of 1 to 20, and the other corresponds to a first key value range of 21 to 50.

[0084] Step 140: Merge the files in the current memory storage space, and divide the merged files into segments according to the key values ​​of each of the first segments to obtain at least one file to be stored;

[0085] In this step, after obtaining the first segment key value set, the files in the memory storage space can be merged. Then, the merged files can be segmented according to each first segment key value in the first segment key value set. In this way, multiple files to be stored corresponding to the first storage subspace can be obtained.

[0086] Specifically, in one possible implementation, merging files in the current memory storage space includes:

[0087] Check if the data stored in each file in the current memory storage space has the same key value;

[0088] When the key values ​​corresponding to the data stored in each file are different, the data stored in each file is merged according to the order of the key values.

[0089] In this embodiment, when merging files in the current memory storage space, it can be detected whether the data stored in each file in the current memory storage space has the same key value. If the key values ​​corresponding to the data stored in each file are all different, it means that the data stored in the current memory storage space has not been updated. For example, suppose there are two files in the current memory storage space, one file stores data with key values ​​ranging from 10 to 30, and the other file stores data with key values ​​ranging from 31 to 45. Then the data in these two files do not affect each other. In this case, they can be arranged according to the size of the key values, and the data stored in each file can be merged based on the arrangement order to obtain a complete merged file with key values ​​ranging from 10 to 45.

[0090] In another possible implementation, merging files in the current memory storage space also includes:

[0091] When the data stored in various files contains the same key value, the same key value is identified as a merge key value.

[0092] Delete the data that corresponds to the merge key value and has an earlier storage time, and determine the data that corresponds to the merge key value and has a later storage time as the data currently corresponding to the merge key value;

[0093] The data stored in each file is merged according to the order of key values.

[0094] In this embodiment, when detecting whether the data stored in each file in the current memory storage space has the same key value, if the key values ​​corresponding to the data stored in each file are the same, it indicates that the data stored in the current memory storage space needs to be updated. For example, suppose there are two files in the current memory storage space, one file stores data with key values ​​ranging from 10 to 35, and the other file stores data with key values ​​ranging from 25 to 45. The key value ranges of the data in these two files overlap. These identical key values ​​can be identified as mergeable key values, and then the data in these two files that corresponds to the mergeable key values ​​and has an earlier storage time can be deleted. For example, in the two files mentioned above, the key values ​​with a size between 25 and 35 are the merged key values. Assuming the file with key values ​​between 10 and 35 corresponding to the stored data was stored earlier, and the file with key values ​​between 25 and 45 corresponding to the stored data was stored later, it means that the file with key values ​​between 25 and 45 corresponding to the stored data updated the data corresponding to key values ​​between 25 and 35. Therefore, in this embodiment, the data corresponding to key values ​​between 25 and 35 in the file with key values ​​between 25 and 45 corresponding to the stored data can be deleted. In this way, the data corresponding to the merged key values ​​and stored later—that is, the data corresponding to key values ​​between 25 and 35 in the file with key values ​​between 25 and 45 corresponding to the stored data—can be identified as the data currently corresponding to the merged key values, thereby completing the data update iteration. After processing the data updates with merged key values, the data can be sorted according to the size of the key values. Based on the sorting order, the data stored in each file can be merged to obtain a complete merged file with key values ​​ranging from 10 to 45.

[0095] In this embodiment, after obtaining the merged file, it is further segmented using a first segment key value. Specifically, for example, if there is only one first segment key value in the set of first segment key values, and the size of the first segment key value is 20, then the merged file can be segmented using this first segment key value. It should be noted that the segmentation method of the merged file using the first segment key value is the same as the segmentation method for the first-level disk storage space; that is, the division method of the first segment key value is relatively consistent. For example, in the process of segmenting the first-level disk storage space, if the first segment key value is assigned to a segment with a larger key value, when segmenting the merged file with key values ​​ranging from 10 to 45 corresponding to the aforementioned stored data using this first segment key value, this data is divided into two parts: one part corresponds to key values ​​ranging from 10 to 19, and the other part corresponds to key values ​​ranging from 20 to 45. Similarly, in the process of segmenting the first-level disk storage space, where the first segment key value is divided into segments with smaller key values, when using this first segment key value to segment the merged files whose corresponding key values ​​range from 10 to 45, the data is divided into two parts: one part of the data corresponds to key values ​​ranging from 10 to 20, and the other part of the data corresponds to key values ​​ranging from 21 to 45. In this embodiment, the specific segmentation method used is not limited.

[0096] Of course, it is understandable that in some embodiments, the first segment key value may not exist in the key value range corresponding to the merged file storage data. In this case, it is not necessary to segment the merged file again, and it can be directly treated as an independent file to be stored.

[0097] Step 150: Store each of the files to be stored into the first storage subspace according to the key value range corresponding to the data in the files to be stored.

[0098] In this step, after the files to be stored are divided, they can be stored in the corresponding first storage subspace of the disk portion according to the key value range corresponding to the data in the files. Specifically, for example, a first-level disk storage space containing data with key values ​​from 1 to 50 is divided into two first storage subspaces. One first storage subspace, denoted as storage subspace A, can store data with key values ​​from 1 to 20; the other first storage subspace, denoted as storage subspace B, can store data with key values ​​from 21 to 50. Therefore, files containing data with key values ​​from 10 to 20 can be stored in storage subspace A, and files containing data with key values ​​from 21 to 45 can be stored in storage subspace B.

[0099] It is understood that in this embodiment, data migration and storage in the log structure merging tree is implemented through segmentation. When data in the inner storage space needs to be migrated to the first-level disk storage space, the entire inner storage space is no longer migrated together; instead, it is stored in units of segmented files to be stored. This way, segments in the first disk storage space that have essentially no data changes do not need to be processed, significantly reducing write amplification and improving the actual utilization of storage space. Furthermore, during the migration of files to be stored, only the file needs to be placed into the corresponding first storage subspace. The files in the first storage subspace do not need to be ordered, so there is no need to read the data from the files; they can be directly copied, thereby improving storage efficiency.

[0100] In some embodiments, the first segmented key-value set in this application is obtained through the following steps:

[0101] Obtain the total list of available storage key values ​​for the first-level disk storage space;

[0102] Randomly select several first segment key values ​​from the total available storage key value list to obtain the first segment key value set.

[0103] In this embodiment, when determining the first segment key value set, the total available storage key value list of the first-level disk storage space can be obtained, and then several first segment key values ​​can be randomly selected from it. For example, assuming that the first-level disk storage space can store data in the range of key values ​​1 to 100, then its total available storage key value list is the values ​​1 to 100. Then, three first segment key values ​​can be randomly selected, for example, 20, 45, and 86, to obtain the first segment key value set as {20, 45, 86}. Of course, in this embodiment, the specific number of first segment key values ​​is not limited.

[0104] In some embodiments, the method further includes:

[0105] Detect whether the number of files stored in the first-level disk storage space exceeds a second preset threshold.

[0106] If the number of files stored in the first-level disk storage space exceeds the second preset number threshold, obtain the second segment key value set corresponding to the second-level disk storage space in the log structure merging tree; the second segment key value set includes at least one second segment key value, which is used to divide the second-level disk storage space into multiple second storage subspaces, and each second storage subspace corresponds to storing data within the second key value range;

[0107] Merge the files in the current first-level disk storage space, and divide the merged files into segments according to the key values ​​of each second segment to obtain at least one file to be moved down;

[0108] Based on the key value range corresponding to the data in the files to be moved, each file to be moved is stored in the second storage subspace.

[0109] In this embodiment, there may be situations where the storage capacity of a certain storage space on the disk is insufficient, requiring further data movement downwards. Here, the data movement between the first and second disk storage spaces is illustrated as an example. Those skilled in the art will understand that the data movement between each (n-1)th disk storage space and the nth disk storage space can be implemented using the same process.

[0110] Specifically, in this embodiment, a storage limit for the number of files can first be set for the first-level disk storage space, denoted as the second preset number threshold. Then, the current number of files stored in the first-level disk storage space can be detected, and its value compared to the second preset number threshold can be determined. If it is less than or equal to the second preset number threshold, it indicates that the storage capacity of the first-level disk storage space is sufficient, and storage can continue; conversely, if the number of files stored in the first-level disk storage space exceeds the second preset number threshold, it indicates that the storage capacity of the first-level disk storage space is exhausted, and data needs to be moved down. At this time, the second segment key-value set corresponding to the second-level disk storage space in the log structure merging tree can be obtained. The second segment key-value set includes several second segment key-values, which are used to divide the second disk space into multiple second storage subspaces, and each second storage subspace corresponds to storing data within the second key-value range. Here, the setting method of the second segment key-values ​​is similar to that of the aforementioned first segment key-values, and will not be elaborated further. In particular, in order to improve the regularity of data storage, the first segment key value set can be set as a subset of the second segment key value set. That is, the second segment key value set can include all the first segment key values ​​and can cover additional second segment key values. This application does not limit this.

[0111] Next, the files in the current first-level disk storage space can be merged, and then the merged files can be segmented according to the key values ​​of each second segment to obtain at least one file to be moved down. Thus, based on the key value range corresponding to the data in the files to be moved down, each file to be moved down can be stored in the second storage subspace. The specific merging and segmentation processing methods are similar to those in the aforementioned embodiments and will not be described in detail here.

[0112] It is understood that, in this embodiment of the application, by executing the above-described data storage method, data compression can be performed periodically, that is, multiple files are selected and merged together, and duplicate updated content is deleted. This reduces data redundancy within the storage space and improves the utilization rate of the storage space.

[0113] The data storage method based on log structure merging tree proposed in this application will be described below with reference to specific embodiments.

[0114] Reference Figure 4 , Figure 4 The diagram illustrates a three-tiered storage structure, comprising Level 0 memory storage, Level 1 disk storage, and Level 2 disk storage. Figure 4 In the illustrated state, memory storage space Level 0 stores two files, with corresponding key value ranges of 2 to 37 and 23 to 48. For the first level of disk storage space Level 1, its corresponding first segment key value set includes two first segment key values: 15 and 70. For the second level of disk storage space Level 2, its corresponding second segment key value set includes four second segment key values: 15, 40, 70, and 95. (See reference...) Figure 5 At a certain moment, the storage unit receives a new file containing data with key values ​​ranging from 30 to 68. Assuming the first preset threshold for the number of elements in memory storage level 0 is 2, this exceeds the storage limit of memory storage level 0, thus requiring file merging. Specifically, refer to... Figure 6 When merging files, three files with key-value ranges of 2 to 37, 23 to 48, and 30 to 68 are merged. Overlapping key-value pairs are created using the newer data, while older data is deleted. This results in the merged file. In the merged file, the key-value range is 2 to 68. The file is segmented using the first segment key-value. Only one first segment key-value, 15, falls within this range, therefore... Figure 7 The file can be divided into two segments, one with key values ​​ranging from 2 to 14, and the other with key values ​​ranging from 15 to 68. The segmented file can then be placed into the corresponding segment in Level 1 of the first-level disk storage space. Similarly, Level 1 of the first-level disk storage space can be processed accordingly to further move data down the storage space; this will not be elaborated upon in this application.

[0115] The following describes a data storage system based on a log structure merging tree according to an embodiment of this application, with reference to the accompanying drawings.

[0116] Reference Figure 8The data storage system based on a log structure merging tree proposed in this application includes:

[0117] The acquisition module 810 is used to acquire the target file to be stored and the target key value range; the target file includes multiple target data, the target key value range includes multiple target key values, and the target data and the target key values ​​correspond one-to-one.

[0118] The detection module 820 is used to write the target file into the memory storage space of the log structure merging tree and detect whether the number of files stored in the current memory storage space exceeds a first preset number threshold.

[0119] The judgment module 830 is used to obtain the first segment key value set corresponding to the first layer disk storage space in the log structure merging tree if the number of files stored in the memory storage space exceeds the first preset number threshold; the first segment key value set includes at least one first segment key value, which is used to divide the first layer disk storage space into multiple first storage subspaces, and each first storage subspace corresponds to storing data within the first key value range.

[0120] The merging module 840 is used to merge files in the current memory storage space, and to segment the merged files according to the key values ​​of each first segment to obtain at least one file to be stored.

[0121] The processing module 850 is used to store each of the files to be stored into the first storage subspace according to the key value range corresponding to the data in the files to be stored.

[0122] It is understood that the content of the above identification method embodiments is applicable to this identification system embodiment. The specific functions implemented by this identification system embodiment are the same as those of the above identification method embodiments, and the beneficial effects achieved are also the same as those achieved by the above identification method embodiments.

[0123] Reference Figure 9 This application also discloses a computer device, including:

[0124] At least one processor 910;

[0125] At least one memory 920 is used to store at least one program;

[0126] When at least one program is executed by at least one processor 910, causing at least one processor 910 to achieve the following: Figure 3 The illustration shows an example of a data storage method based on a log structure merging tree.

[0127] It is understandable that, such as Figure 3The content of the data storage method embodiment based on log structure merging tree shown is applicable to the embodiment of this computer device. The specific functions implemented by the embodiment of this computer device are the same as those shown below. Figure 3 The data storage method based on log structure merging tree shown is the same as the embodiment, and the beneficial effects achieved are the same as those described above. Figure 3 The beneficial effects achieved by the data storage method embodiment based on log structure merging tree shown are also the same.

[0128] This application also discloses a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement, for example... Figure 3 The illustration shows an example of a data storage method based on a log structure merging tree.

[0129] It is understandable that, such as Figure 3 The content of the data storage method embodiment based on log structure merging tree shown is applicable to the embodiment of this computer-readable storage medium. The specific functions implemented by the embodiment of this computer-readable storage medium are the same as those shown below. Figure 3 The data storage method based on log structure merging tree shown is the same as the embodiment, and the beneficial effects achieved are the same as those described above. Figure 3 The beneficial effects achieved by the data storage method embodiment based on log structure merging tree shown are also the same.

[0130] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0131] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0132] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0134] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0135] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0136] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0137] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0138] The foregoing has provided a detailed description of the preferred embodiments of this application. However, this application is not limited to these embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

[0139] In the description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0140] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A data storage method based on a log structure merging tree, characterized in that, include: Get the target file to be stored and the target key-value range; The target file includes multiple target data, and the target key value range includes multiple target key values, with each target data and target key value corresponding one-to-one. Write the target file into the memory storage space of the log structure merge tree, and check whether the number of files stored in the current memory storage space exceeds the first preset number threshold. If the number of files stored in the memory storage space exceeds the first preset number threshold, obtain the first segment key value set corresponding to the first layer disk storage space in the log structure merging tree; the first segment key value set includes at least one first segment key value, the first segment key value is used to divide the first layer disk storage space into multiple first storage subspaces, and each first storage subspace corresponds to storing data within the first key value range; The files in the current memory storage space are merged, and the merged files are segmented according to the key values ​​of each of the first segments to obtain at least one file to be stored; Based on the key value range corresponding to the data in the files to be stored, each file to be stored is stored in the first storage subspace.

2. The data storage method based on a log structure merging tree according to claim 1, characterized in that, The first segmented key-value set is obtained through the following steps: Obtain the total list of available storage key values ​​for the first-level disk storage space; Randomly select several first segment key values ​​from the total available storage key value list to obtain the first segment key value set.

3. The data storage method based on a log structure merging tree according to claim 1, characterized in that, The merging of files in the current memory storage space includes: Check if the data stored in each file in the current memory storage space has the same key value; When the key values ​​corresponding to the data stored in each file are different, the data stored in each file is merged according to the order of the key values.

4. The data storage method based on a log structure merging tree according to claim 3, characterized in that, The merging of files in the current memory storage space also includes: When the data stored in various files contains the same key value, the same key value is identified as a merge key value. Delete the data that corresponds to the merge key value and has an earlier storage time, and determine the data that corresponds to the merge key value and has a later storage time as the data currently corresponding to the merge key value; The data stored in each file is merged according to the order of key values.

5. The data storage method based on a log structure merging tree according to claim 1, characterized in that, The method further includes: Detect whether the number of files stored in the first-level disk storage space exceeds a second preset threshold. If the number of files stored in the first-level disk storage space exceeds the second preset number threshold, obtain the second segment key value set corresponding to the second-level disk storage space in the log structure merging tree; the second segment key value set includes at least one second segment key value, which is used to divide the second-level disk storage space into multiple second storage subspaces, and each second storage subspace corresponds to storing data within the second key value range; Merge the files in the current first-level disk storage space, and divide the merged files into segments according to the key values ​​of each second segment to obtain at least one file to be moved down; Based on the key value range corresponding to the data in the files to be moved, each file to be moved is stored in the second storage subspace.

6. The data storage method based on a log structure merging tree according to claim 5, characterized in that, The first segmented key-value set is a subset of the second segmented key-value set.

7. The data storage method based on a log structure merging tree according to any one of claims 1-6, characterized in that, The method further includes: If the number of files stored in the memory storage space does not exceed the first preset number threshold, the target file is stored in the memory storage space.

8. A data storage system based on a log structure merging tree, characterized in that, include: The acquisition module is used to obtain the target file to be stored and the target key-value range; The target file includes multiple target data, and the target key value range includes multiple target key values, with each target data and target key value corresponding one-to-one. The detection module is used to write the target file into the memory storage space of the log structure merging tree and detect whether the number of files stored in the current memory storage space exceeds a first preset number threshold. The judgment module is used to obtain the first segment key value set corresponding to the first layer disk storage space in the log structure merging tree if the number of files stored in the memory storage space exceeds the first preset number threshold. The first segment key value set includes at least one first segment key value, which is used to divide the first layer disk storage space into multiple first storage subspaces, and each first storage subspace corresponds to storing data within the first key value range. The merge module is used to merge files in the current memory storage space, and to segment the merged files according to the key values ​​of each first segment to obtain at least one file to be stored. The processing module is used to store each of the files to be stored into the first storage subspace according to the key value range corresponding to the data in the files to be stored.

9. A computer device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the data storage method based on the log structure merging tree as described in any one of claims 1-7.

10. A computer-readable storage medium storing a processor-executable program, characterized in that: The processor-executable program, when executed by the processor, is used to implement the data storage method based on the log structure merging tree as described in any one of claims 1-7.

Images