Hot data storage method and device based on LSM hierarchical storage, equipment and medium

Abstract

The invention relates to the field of data processing and digital finance, and discloses a hot data storage method, device and equipment based on LSM hierarchical storage and a medium, the method comprises the following steps: determining a hot value threshold based on the hot value and SST capacity of each piece of data; and determining the data of which the heat value is greater than or equal to the heat value threshold as hot data, and writing the hot data into a high-speed medium. Through the above mode, the hot data and the cold data are intelligently identified and distinguished, so that storage resources are more reasonably distributed, the hot data are stored on a high-speed medium with higher cost and higher access speed, the time required for data retrieval is shortened, the query response speed of a database is remarkably improved, and the query efficiency of the database is improved. Along with the change of application requirements, the heat of the data may change, and the method allows the system to dynamically adjust the storage strategy of the data according to a real-time access mode so as to adapt to continuously changing requirements, and the storage efficiency of the hot data in the LSM database is improved.

Description

Technical Field

[0001] The present application relates to the fields of data processing and digital finance, and in particular to a hot data storage method, device, equipment and medium based on LSM hierarchical storage. Background Art

[0002] With the rapid development of the digital financial industry and information technology, database systems have become a key tool for storing and managing large amounts of data. Among many database systems, databases based on LSM (Log-Structured Merge-tree, a disk-based data structure) trees are widely used due to their efficient write performance. LSM trees achieve high-throughput write operations by storing data in layers on different media.

[0003] In the LSM tree model, data is first written to the MemTable (cache table) structure in memory. When the MemTable reaches a certain size, it is refreshed to disk to form an SST (Sorted String Table, immutable data storage). SST files are immutable and will generate multiple SST files over time. In order to optimize storage space and query performance, the system will periodically perform compaction operations to merge multiple small SST files into larger files and delete expired or duplicate data.

[0004] However, the existing technology has some shortcomings. Especially in the hierarchical storage system, although the first few levels (such as L0, L1) use high-speed media (such as NVMe) to improve write performance, the last level (such as L2) may use low-speed media (such as HDD), resulting in reduced data reading performance. In addition, since the LSM database does not divide the data into hot and cold, even if the data is frequently read, it may be continuously moved from the L0 layer to the upper layer due to the compaction service until it is stored on the low-speed medium, affecting the reading performance, and improving the hot data storage efficiency, thereby improving the hot data reading efficiency. Therefore, how to improve the storage efficiency of hot data in the LSM database has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a hot data storage method, device, equipment and medium based on LSM hierarchical storage to improve the storage efficiency of hot data in an LSM database.

[0006] In a first aspect, the present application provides a hot data storage method based on LSM hierarchical storage, the method comprising:

[0007] Determine the heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST;

[0008] The data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold is determined as hot data, and the hot data is written to a high-speed medium.

[0009] In a second aspect, the present application further provides a hot data storage device based on LSM hierarchical storage, the device comprising:

[0010] A heat value threshold determination module is used to determine the heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST;

[0011] The hot data storage module is used to determine the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and write the hot data into a high-speed medium.

[0012] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the above-mentioned hot data storage method based on LSM hierarchical storage when executing the computer program.

[0013] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the above-mentioned hot data storage method based on LSM hierarchical storage.

[0014] The present application discloses a method, device, equipment and medium for storing hot data based on LSM hierarchical storage. The method for storing hot data based on LSM hierarchical storage includes determining a heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST; determining the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and writing the hot data into a high-speed medium. In the above manner, the present application intelligently identifies and distinguishes hot data from cold data, so that storage resources are more reasonably allocated. Hot data is stored on high-speed media with higher costs but faster access speeds, which reduces the time required for data retrieval, thereby significantly improving the query response speed of the database. As application requirements change, the heat of data may change. This method allows the system to dynamically adjust the data storage strategy according to the real-time access mode to adapt to changing needs, thereby improving the storage efficiency of hot data in the LSM database. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 is a schematic flow chart of a hot data storage method based on LSM hierarchical storage provided by an embodiment of the present application;

[0017] Figure 2 A schematic diagram of heat query of a heat data storage method based on LSM hierarchical storage provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of data writing of a hot data storage method based on LSM hierarchical storage provided in an embodiment of the present application;

[0019] Figure 4 A schematic block diagram of a hot data storage device based on LSM hierarchical storage provided in an embodiment of the present application;

[0020] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0023] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0024] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0025] The embodiments of the present application provide a method, device, equipment and medium for hot data storage based on LSM hierarchical storage. Among them, the hot data storage method based on LSM hierarchical storage can be applied to the server, and the storage resources can be more reasonably allocated by intelligently identifying and distinguishing hot data and cold data. Hot data is stored on high-speed media with higher cost but faster access speed, which reduces the time required for data retrieval, thereby significantly improving the query response speed of the database. As the application requirements change, the popularity of the data may change. This method allows the system to dynamically adjust the data storage strategy according to the real-time access mode to adapt to the changing needs, thereby improving the storage efficiency of hot data in the LSM database. Among them, the server can be an independent server or a server cluster.

[0026] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] See also Figure 1 , Figure 1 This is a schematic flow chart of a hot data storage method based on LSM hierarchical storage provided by an embodiment of the present application. The hot data storage method based on LSM hierarchical storage can be applied to a server to intelligently identify and distinguish hot data from cold data so that storage resources are more reasonably allocated. Hot data is stored on high-speed media with higher costs but faster access speeds, which reduces the time required for data retrieval, thereby significantly improving the query response speed of the database. As application requirements change, the popularity of data may change. This method allows the system to dynamically adjust the data storage strategy according to the real-time access mode to adapt to changing needs, thereby improving the storage efficiency of hot data in the LSM database.

[0028] like Figure 1 As shown, the hot data storage method based on LSM hierarchical storage specifically includes steps S10 to S20.

[0029] Step S10: Determine a heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST;

[0030] In a specific embodiment, first, it is necessary to collect the heat value of each data item in the database, which is usually calculated based on the access frequency, analyze the collected heat values, understand the distribution of data heat, and determine a threshold based on the heat distribution of the data and the capacity of the immutable data storage SST (Sorted String Table). This threshold is used to distinguish hot data from cold data. Usually, the determination of the threshold takes into account the allocation of storage resources, access patterns, and performance requirements.

[0031] Step S20: Determine the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and write the hot data into a high-speed medium.

[0032] In a specific embodiment, according to the heat value threshold determined in step S10, the hot data in the database is identified, and a suitable high-speed storage medium, such as an SSD (solid state drive), is selected to store the hot data to improve the access speed, and the identified hot data is migrated to the selected high-speed medium. This may involve operations such as data duplication, compression and optimization.

[0033] It should be noted that the determination of the threshold needs to be adjusted according to actual business needs and system performance to balance storage costs and access efficiency. The identification and migration of hot data is a dynamic process and may need to be updated regularly or based on specific trigger conditions. The selection and use of high-speed media needs to consider factors such as cost, durability and compatibility.

[0034] In one embodiment, the first few levels of LSM hierarchical storage are high-speed media (such as NVME), and the last level L2 is low-speed media (such as HDD), which can reduce the cost of the system. However, when reading data, it is necessary to start searching from L0 to L2. If the data is in the L2 layer, the performance will be greatly reduced due to reading from the low-speed medium.

[0035] Because the LSM database that is transmitted does not divide the data into hot and cold data, when hot data is not written frequently, the compaction service will continuously move it from L0 to the upper layer until it reaches the low-speed layer L2, which will cause a significant drop in the hot data reading performance. Therefore, find the hot data according to the reading frequency and save the hot data on high-speed media to improve the reading performance.

[0036] When data is read, the heat table is first queried for this key. If this key does not exist, it is initialized to 1 and added to the heat table. If this key exists, its heat value is increased by 1 and its value in the heat table is updated. The keys in the heat table are arranged from high to low according to their heat, and this table is loaded into memory for query.

[0037] The present embodiment discloses a method, device, equipment and medium for hot data storage based on LSM hierarchical storage. The hot data storage method based on LSM hierarchical storage includes determining a heat value threshold based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity; determining the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and writing the hot data into a high-speed medium. Through the above-mentioned method, the present application makes storage resources more reasonably allocated by intelligently identifying and distinguishing hot data from cold data. Hot data is stored on high-speed media with higher cost but faster access speed, which reduces the time required for data retrieval, thereby significantly improving the query response speed of the database. As application requirements change, the heat of data may change. This method allows the system to dynamically adjust the data storage strategy according to the real-time access mode to adapt to the changing needs, thereby improving the storage efficiency of hot data in the LSM database.

[0038] based on Figure 1 In the embodiment shown, in this embodiment, before step S10, the following steps are included:

[0039] Pre-establish a data heat table, and record the reading frequency of each data in the LSM hierarchical database through the data heat table;

[0040] Based on the reading frequency, a heat value of each data in the LSM hierarchical database is determined.

[0041] In one embodiment, Figure 2 As shown, Figure 2 A schematic diagram of heat query for a heat data storage method based on LSM hierarchical storage provided in an embodiment of the present application. When data is read, the heat table is first queried for this key. If this key does not exist, it is initialized to 1 and added to the heat table. If this key exists, its heat value is +1 and its value in the heat table is updated. The keys in the heat table are arranged from high to low according to their heat, and this table is loaded into the memory for query.

[0042] When data is read, the heat table is first queried for this key. If this key does not exist, it is initialized to 1 and added to the heat table. If this key exists, its heat value is increased by 1 and its value in the heat table is updated. The keys in the heat table are arranged from high to low according to their heat, and this table is loaded into memory for query.

[0043] Specifically, start the database monitoring tool, record the read operations of all data items, ensure that the monitoring tool can accurately capture the detailed information of each data access, and create a data heat table, which should contain the following fields:

[0044] Data ID: uniquely identifies each piece of data.

[0045] Read frequency: records the number of times data is read.

[0046] Last access time: records the time when the data was last accessed.

[0047] Set periodic tasks, such as updating the reading frequency and last access time in the data heat table every day or every hour, and analyze the data access pattern based on the reading frequency data in the data heat table. Define the heat value calculation formula, for example:

[0048] Heat value = reading frequency × time decay factor;

[0049] Heat value = reading frequency × time decay factor;

[0050] Among them, the time decay factor is used to reduce the heat value of old data to reflect the timeliness of the data.

[0051] According to business needs and storage strategies, set a heat value threshold to distinguish hot data from cold data. According to the determined heat value threshold, identify hot data and migrate the identified hot data to high-speed storage media, such as SSD, to increase access speed. For cold data, consider migrating it to lower-cost storage media, such as HDD.

[0052] Regularly review the data heat table and heat value calculation method, and adjust it according to actual business needs.

[0053] The present embodiment discloses a hot data storage method, device, equipment and medium based on LSM hierarchical storage, wherein the hot data storage method based on LSM hierarchical storage includes pre-establishing a data heat table, and recording the reading frequency of each data in the LSM hierarchical database through the data heat table; determining the heat value of each data in the LSM hierarchical database based on the reading frequency; determining the heat value threshold based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity; determining the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and writing the hot data to a high-speed medium. In the above manner, the present application provides basic data for data heat evaluation by recording the reading frequency of the data. By using database monitoring tools or log systems, the number of times each piece of data is read is recorded in real time, and hot data and cold data are intelligently identified and distinguished, so that storage resources are allocated more reasonably. Hot data is stored on high-speed media with higher costs but faster access speeds, reducing the time required for data retrieval, thereby significantly improving the query response speed of the database. As application requirements change, the popularity of data may change. This method allows the system to dynamically adjust the data storage strategy based on real-time access patterns to adapt to changing needs, thereby improving the storage efficiency of hot data in the LSM database.

[0054] In a more preferred embodiment, based on the reading frequency, determining the heat value of each data in the LSM hierarchical database includes:

[0055] In response to the data query instruction, query the heat value of the selected data in the data heat table;

[0056] If the data to be selected and the heat value of the data to be selected exist in the data heat table, the heat value of the data to be selected is updated.

[0057] In one embodiment, when data is read, the heat table is first queried for this key. If this key does not exist, it is initialized to 1 and added to the heat table. If this key exists, its heat value is +1 and its value in the heat table is updated. The keys in the heat table are arranged from high to low according to their heat, and this table is loaded into memory for query.

[0058] The present embodiment discloses a hot data storage method, device, equipment and medium based on LSM hierarchical storage, wherein the hot data storage method based on LSM hierarchical storage includes pre-establishing a data heat table, and recording the reading frequency of each data in the LSM hierarchical database through the data heat table; in response to a data query instruction, querying the heat value of the selected data in the data heat table; if the selected data and the heat value of the selected data exist in the data heat table, updating the heat value of the selected data; determining a heat value threshold based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity; determining the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and writing the hot data to a high-speed medium. In the above manner, the present application provides basic data for data heat evaluation by recording the reading frequency of the data. By using database monitoring tools or log systems, the number of times each piece of data is read is recorded in real time, and hot data and cold data are intelligently identified and distinguished, so that storage resources are allocated more reasonably. Hot data is stored on high-speed media with higher costs but faster access speeds, reducing the time required for data retrieval, thereby significantly improving the query response speed of the database. As application requirements change, the popularity of data may change. This method allows the system to dynamically adjust the data storage strategy based on real-time access patterns to adapt to changing needs, thereby improving the storage efficiency of hot data in the LSM database.

[0059] In a specific embodiment, in response to the data query instruction, after querying the heat value of the selected data in the data heat table, the method includes:

[0060] If the candidate data does not exist in the data heat table, the heat value of the candidate data is set to an initial value.

[0061] Specifically, when data is read, the heat table first queries this key. If this key does not exist, it is initialized to 1.

[0062] The present embodiment discloses a method, device, equipment and medium for storing hot data based on LSM hierarchical storage. The method for storing hot data based on LSM hierarchical storage includes pre-establishing a data heat table, and recording the reading frequency of each data in the LSM hierarchical database through the data heat table; in response to a data query instruction, querying the heat value of the selected data in the data heat table; if the selected data does not exist in the data heat table, setting the heat value of the selected data to an initial value; determining a heat value threshold based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity; determining the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and writing the hot data to a high-speed medium. In the above manner, the present application provides basic data for data heat evaluation by recording the reading frequency of the data. By using database monitoring tools or log systems, the number of times each piece of data is read is recorded in real time, and hot data and cold data are intelligently identified and distinguished, so that storage resources are allocated more reasonably. Hot data is stored on high-speed media with higher costs but faster access speeds, reducing the time required for data retrieval, thereby significantly improving the query response speed of the database. As application requirements change, the popularity of data may change. This method allows the system to dynamically adjust the data storage strategy based on real-time access patterns to adapt to changing needs, thereby improving the storage efficiency of hot data in the LSM database.

[0063] based on Figure 1 In the embodiment shown, in this embodiment, step S10 includes:

[0064] Sorting the data in the data heat table according to a preset heat value sorting rule;

[0065] Determine a first data group and a second data group according to the heat values ​​of the sorted data;

[0066] The heat value threshold is determined based on the first data set, the second data set and the SST capacity.

[0067] Specifically, since the heat table is arranged from high to low according to the key heat, such as K1, K2...Kn, and hot(K1)>hot(k2)>...>hot(Kn), hot(Kn) is the heat of Kn. size(k1) is the size occupied by k1 and value1. size(SST) is the available space of SST.

[0068] Assume that size(K1)+size(K2)+...+size(Kn)<size(SST) and size(K1)+size(K2)+...+size(Kn)+size(Kn+1)>size(SST). Then hot(Kn) is the heat threshold value. That is, the Key with the smallest heat among all the hot data that can be accommodated in SST.

[0069] When the system initializes, the heat table is empty and the threshold value cannot be calculated. At this time, the compaction process still runs according to the previous process. When the system runs for a period of time and the sum of the sizes of all Keys in the heat table is greater than size(sst), the heat threshold value can be calculated.

[0070] This embodiment discloses a hot data storage method, device, equipment and medium based on LSM hierarchical storage. The hot data storage method based on LSM hierarchical storage includes sorting each data in the data heat table according to a preset heat value sorting rule; determining a first data group and a second data group according to the heat values of the sorted data; determining the heat value threshold based on the first data group, the second data group and the SST capacity; determining the data with a heat value greater than or equal to the heat value threshold in the LSM hierarchical database as hot data, and writing the hot data into a high-speed medium. By the above method, the present application intelligently identifies and distinguishes hot data and cold data, so that the storage resources are more reasonably allocated. The hot data is stored on a high-speed medium with higher cost but faster access speed, reducing the time required for data retrieval, thereby significantly improving the query response speed of the database. As the application requirements change, the heat of the data may change. This method allows the system to dynamically adjust the data storage strategy according to the real-time access pattern to adapt to the changing requirements and improve the storage efficiency of hot data in the LSM database.

[0071] In a more optimal embodiment, determining the heat value threshold based on the first data group, the second data group and the SST capacity includes:

[0072] Calculating the sum of the heat values of each data in the first data group as the first heat value sum;

[0073] Calculating the sum of the heat values of each data in the second data group as the second heat value sum;

[0074] When the SST capacity is greater than the first heat value sum and the SST capacity is less than the second heat value sum, determining the largest heat value in the first data group as the heat value threshold.

[0075] In a specific embodiment, for the data in the heat table, a heat threshold is set. Data above this threshold is considered hot data and is saved to high-speed media. Since the heat table is related to whether data is classified as hot data, the maintenance of the heat table and the setting of the threshold are also very important.

[0076] Setting of the threshold: Since the heat table is arranged in descending order of Key heat, such as K1, K2... Kn, and hot(K1)>hot(k2)>……>hot(Kn), where hot(Kn) is the heat of Kn. size(k1) is the size occupied by k1 and value1. size(SST) is the available space of SST.

[0077] Assume that size(K1)+size(K2)+...+size(Kn)<size(SST) and size(K1)+size(K2)+...+size(Kn)+size(Kn + 1)>size(SST). Then hot(Kn) is the heat threshold. That is, the Key with the lowest heat among all hot data that can be accommodated in SST.

[0078] When the system is initialized, the heat table is empty and the threshold cannot be calculated. At this time, the compaction process still runs according to the previous process. When the system runs for a period of time and the sum of the sizes of all Keys in the heat table is greater than size(sst), the heat threshold can be calculated.

[0079] Since the Keys in the heat table are constantly changing, the heat threshold also needs to be recalculated periodically according to the above formula.

[0080] When the SST capacity changes, recalculate the heat threshold according to the above formula.

[0081] Based on any of the above embodiments, in this embodiment, after step S20, it includes:

[0082] Determine the data with a heat value less than the heat value threshold in the LSM hierarchical database as cold data, and write the cold data to low-speed media.

[0083] In a specific embodiment, as Figure 3 shown, Figure 3 This is a schematic diagram of data writing for a heat data storage method based on LSM hierarchical storage provided by an embodiment of the present application. For hot data, special processing needs to be done for compaction to ensure that hot data is written to high-speed media. When doing compaction, compare the Key read with the threshold of the heat table. Keys and Values above this threshold will be saved in the high-frequency data SST, and Keys and Values below the threshold will be compacted to the next level.

[0084] See also Figure 4 , Figure 4 The embodiment of the present application provides a schematic block diagram of a hot data storage device based on LSM hierarchical storage, and the hot data storage device based on LSM hierarchical storage is used to execute the aforementioned hot data storage method based on LSM hierarchical storage. The hot data storage device based on LSM hierarchical storage can be configured on a server.

[0085] like Figure 4 As shown, the hot data storage device 400 based on LSM hierarchical storage includes:

[0086] A heat value threshold determination module 410 is used to determine a heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST;

[0087] The hot data storage module 420 is used to determine the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and write the hot data into a high-speed medium.

[0088] Furthermore, the LSM hierarchical storage-based hot data storage device 400 further includes:

[0089] A data heat table establishment module, used to establish a data heat table in advance, and record the reading frequency of each data in the LSM hierarchical database through the data heat table;

[0090] A heat value determination module is used to determine the heat value of each data in the LSM hierarchical database based on the reading frequency.

[0091] Furthermore, the heat value determination module includes:

[0092] A heat value query submodule, for querying the heat value of the selected data in the data heat table in response to a data query instruction;

[0093] The heat value updating submodule is used to update the heat value of the candidate data if the candidate data and the heat value of the candidate data exist in the data heat table.

[0094] Furthermore, the LSM hierarchical storage-based hot data storage device 400 further includes:

[0095] The heat value initialization module is used to set the heat value of the candidate data to an initial value if the candidate data does not exist in the data heat table.

[0096] Furthermore, the heat value threshold determination module 410 includes:

[0097] A heat value sorting submodule is used to sort the data in the data heat table according to a preset heat value sorting rule;

[0098] A data group determination submodule, used for determining a first data group and a second data group according to the heat values ​​of the sorted data;

[0099] The heat value threshold determination submodule is used to determine the heat value threshold based on the first data group, the second data group and the SST capacity.

[0100] Furthermore, the heat value threshold determination submodule includes:

[0101] A first heat value sum determination unit, used for calculating the sum of heat values ​​of each data in the first data group as the first heat value sum;

[0102] A second heat value sum determination unit, used for calculating the heat value sum of each data in the second data group as the second heat value sum;

[0103] A heat value threshold determination unit is used to determine the maximum heat value in the first data group as the heat value threshold when the SST capacity is greater than the first heat value sum and the SST capacity is less than the second heat value sum.

[0104] Furthermore, the LSM hierarchical storage-based hot data storage device 400 further includes:

[0105] The low-speed medium writing module is used to determine the data in the LSM hierarchical database whose heat value is less than the heat value threshold as cold data, and write the cold data into the low-speed medium.

[0106] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0107] The above-mentioned device can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.

[0108] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0109] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0110] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any hot data storage method based on LSM hierarchical storage.

[0111] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0112] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any hot data storage method based on LSM hierarchical storage.

[0113] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0115] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0116] Determine the heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST;

[0117] The data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold is determined as hot data, and the hot data is written to a high-speed medium.

[0118] In one embodiment, based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST, before determining the heat value threshold, it is used to implement:

[0119] Pre-establish a data heat table, and record the reading frequency of each data in the LSM hierarchical database through the data heat table;

[0120] Based on the reading frequency, a heat value of each data in the LSM hierarchical database is determined.

[0121] In one embodiment, based on the reading frequency, the heat value of each data in the LSM hierarchical database is determined to achieve:

[0122] In response to the data query instruction, query the heat value of the selected data in the data heat table;

[0123] If the data to be selected and the heat value of the data to be selected exist in the data heat table, the heat value of the data to be selected is updated.

[0124] In one embodiment, in response to the data query instruction, after querying the heat value of the selected data in the data heat table, it is used to implement:

[0125] If the candidate data does not exist in the data heat table, the heat value of the candidate data is set to an initial value.

[0126] In one embodiment, based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST, a heat value threshold is determined to achieve:

[0127] Sorting the data in the data heat table according to a preset heat value sorting rule;

[0128] Determine a first data group and a second data group according to the heat values ​​of the sorted data;

[0129] The heat value threshold is determined based on the first data set, the second data set and the SST capacity.

[0130] In one embodiment, the heat value threshold is determined based on the first data group, the second data group and the SST capacity to achieve:

[0131] Calculating the sum of the heat values ​​of each data in the first data group as the first heat value sum;

[0132] Calculating the sum of the heat values ​​of each data in the second data group as the second heat value sum;

[0133] When the SST capacity is greater than the first heat value sum and the SST capacity is less than the second heat value sum, the maximum heat value in the first data group is determined as the heat value threshold.

[0134] In one embodiment, based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity, after determining the heat value threshold, it is used to implement:

[0135] The data in the LSM hierarchical database whose heat value is less than the heat value threshold is determined as cold data, and the cold data is written to a low-speed medium.

[0136] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the thermal data storage methods based on LSM hierarchical storage provided in the embodiments of the present application.

[0137] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A hot data storage method based on LSM hierarchical storage, characterized in that: include: Determine the heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST; The data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold is determined as hot data, and the hot data is written to a high-speed medium.

2. The method for storing hot data based on LSM hierarchical storage according to claim 1, characterized in that: Before determining the heat value threshold based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity, the method includes: Pre-establish a data heat table, and record the reading frequency of each data in the LSM hierarchical database through the data heat table; Based on the reading frequency, a heat value of each data in the LSM hierarchical database is determined.

3. The hot data storage method based on LSM hierarchical storage according to claim 2 is characterized in that: The determining, based on the reading frequency, the heat value of each data in the LSM hierarchical database includes: In response to the data query instruction, query the heat value of the selected data in the data heat table; If the data to be selected and the heat value of the data to be selected exist in the data heat table, the heat value of the data to be selected is updated.

4. The method for storing hot data based on LSM hierarchical storage according to claim 3 is characterized in that: After querying the heat value of the selected data in the data heat table in response to the data query instruction, the method includes: If the candidate data does not exist in the data heat table, the heat value of the candidate data is set to an initial value.

5. The method for storing hot data based on LSM hierarchical storage according to claim 2, characterized in that: The step of determining the heat value threshold based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity includes: Sorting the data in the data heat table according to a preset heat value sorting rule; Determine a first data group and a second data group according to the heat values ​​of the sorted data; The heat value threshold is determined based on the first data set, the second data set and the SST capacity.

6. The method for storing hot data based on LSM hierarchical storage according to claim 5, characterized in that: The step of determining the heat value threshold based on the first data group, the second data group, and the SST capacity includes: Calculating the sum of the heat values ​​of each data in the first data group as the first heat value sum; Calculating the sum of the heat values ​​of each data in the second data group as the second heat value sum; When the SST capacity is greater than the first heat value sum and the SST capacity is less than the second heat value sum, the maximum heat value in the first data group is determined as the heat value threshold.

7. The method for storing hot data based on LSM hierarchical storage according to any one of claims 1 to 6, characterized in that: After determining the heat value threshold based on the heat value of each data in the LSM hierarchical database and the immutable data storage SST capacity, the method includes: The data in the LSM hierarchical database whose heat value is less than the heat value threshold is determined as cold data, and the cold data is written to a low-speed medium.

8. A hot data storage device based on LSM hierarchical storage, characterized in that: include: A heat value threshold determination module is used to determine the heat value threshold based on the heat value of each data in the LSM hierarchical database and the capacity of the immutable data storage SST; The hot data storage module is used to determine the data in the LSM hierarchical database whose heat value is greater than or equal to the heat value threshold as hot data, and write the hot data into a high-speed medium.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the hot data storage method based on LSM hierarchical storage as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the hot data storage method based on LSM hierarchical storage as described in any one of claims 1 to 7.

Images