A method and device for index persistence for KV storage system

Abstract

The present application provides an index persistence method and device for a KV storage system, wherein the method includes: querying the first preset layer of a log structure merge tree, and finding from the tail metadata area that the key value key range of the index is a first preset data range; querying the index file corresponding to the first preset range, finding a portion of the second preset data range from the first-level index area by a binary search method, and reading back the data results of the preset data; performing data query in the second preset layer of the log structure merge tree and the third preset layer of the log structure merge tree, and checking the key value key to be queried, and reading back the data results of the preset data; merging the data results read back in the first, second, and third preset layers and returning them. The disclosed index persistence method can reduce index file read amplification without increasing the metadata management pressure of the file system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for index persistence in a KV storage system. Background Art

[0002] Key-Value Store (KV) storage systems offer a flexible data model, simple access interfaces, and high scalability, providing an excellent solution for storing unstructured and semi-structured data. Write-intensive KV storage systems (such as LevelDB, RocksDB, and HBase) widely use the Log-Structured Merge Tree (LSM) tree as their storage engine. This achieves high write performance by converting random writes initiated by clients into sequential append-only writes. Each key value is assigned a globally unique, monotonically increasing version number upon write. Repeated writes of the same key value may result in multiple values ​​with different version numbers. However, due to this non-in-place value update method, LSM trees can contain garbage data and suffer from read amplification and space amplification. Read amplification refers to the amount of data actually read exceeding the user's desired amount, while space amplification refers to the additional storage space occupied by expired data that is not promptly cleared. To address these issues, LSM trees incorporate a merge mechanism that, under certain triggering conditions, merges data, retaining only the most recent value for the same key value to reclaim storage space. In general, the LSM tree log structure merge tree merges KV pairs and stores them in a complete SSTable sorted string table file. However, in some scenarios where the value is significantly larger than the key value, the industry generally adopts the method of storing KV separately in different files and merging them separately. This method has the following advantages: first, it delays the merging of values ​​to avoid frequent large merges that occupy too high IO bandwidth for a long time; second, it reduces the frequency of rewriting garbage values, which can help reduce write amplification. Especially for scenarios where the underlying storage medium is a solid-state drive, reducing write amplification helps extend the service life; finally, because the size of the key value is significantly smaller than the value, the key value can be quickly merged at a lower IO cost, shortening the IO path for read requests to query the key value and improving read performance.

[0003] The read request response process for a KV storage system based on an LSM tree log-structured merge-tree storage engine is as follows: Step 1: Query the currently writing index at level 0, which represents the latest data. If all data is found, a direct return is issued; if not, the query continues down the chain. Step 2: Query the data cache pool. If all data is found, a direct return is issued; if not, the query continues down the chain. Step 3: Query the immutable index tables from level 1 to level n, sorting by key values ​​and eliminating duplicates within each level. This step only queries the metadata of the immutable index, which consumes minimal memory and can be maintained in memory for a long time. This metadata includes key value range information and mappings to the SSTable sorted string table file. In a KV separation architecture, the SSTable sorted string table consists of an ordered index file and multiple data files on disk. Steps 4-5: The index that matches the index requires further reading of its entire content. The query begins with the index cache pool. If the cache matches, the query continues directly; if not, the index file is read from disk and stored in the index cache pool before querying. Step 6: After the index query is complete, the specific data is read based on the mapping relationship between the data files in the index record. After reading, it is added to the cache pool to speed up the next query. Step 7: After the read is completed, the data is returned.

[0004] Generally, in the mapping relationship between index files and data files in an SSTable sorted string table, an index file consists of several sequentially arranged index nodes. Each index node consists of a key value and metadata about the value. The metadata includes the name of the data file pointed to by the value, the starting offset of the value in the data file, and the length of the value in the data file. Key values ​​in the index file are ordered and unique. Therefore, during queries, the entire index file is typically read into memory and a binary search algorithm is used for fast queries. However, as the storage system continuously writes new data, the size of index files at deeper levels in the LSM tree log-structured merge tree can expand to hundreds of megabytes or even gigabytes. Splitting the index file into multiple smaller files by size increases the metadata management pressure on the underlying file system. Therefore, when the index file is large but the query scope is small, reading the entire index file into memory presents a significant read amplification problem, exacerbating response latency. Summary of the Invention

[0005] This application describes a method and apparatus for index persistence in a KV storage system.

[0006] In a first aspect, the present application provides an index persistence method for a KV storage system, the method comprising:

[0007] Step S110, querying a first preset layer of the log structure merge tree, wherein the first preset layer of the log structure merge tree includes a set of preset indexes, and a key value key range of the index found in the tail metadata area is a first preset data range;

[0008] Step S120: query the index file corresponding to the first preset range, read all secondary indexes from the index file, add them to the secondary index cache, search for the primary index area and the starting position of the filter that the query hits, read the filter and put it into the filter cache, if the filter hits, read the primary index area and put it into the cache, find the part of the second preset data range from the primary index area through a binary search method, and read back the data result of the preset data;

[0009] Step S130: Based on step S120, data query is performed in the second preset layer and the third preset layer of the log structure merge tree, and the key value to be queried is checked, and the data result of the preset data is read back;

[0010] Step S140 , merging the data results read back from the first preset layer, the second preset layer, and the third preset layer of the log structure merge tree and returning them.

[0011] In an exemplary embodiment of the present disclosure, the preset data in the method further includes:

[0012] The information value value file name, value value offset and length in the file, and the location of value value data.

[0013] In an exemplary embodiment of the present disclosure, step S130 of the method further includes:

[0014] Based on step S120, data query is performed in the second preset layer of the log structure merge tree, and the key value key to be queried is checked. The part of the third preset data range is found from the said first-level index area through the binary search method, and the data result of the preset data is read back.

[0015] In an exemplary embodiment of the present disclosure, step S130 of the method further includes:

[0016] Based on step S130, data query is performed in the third preset layer of the log structure merge tree, and the key value key to be queried is checked, and a part of the fourth preset data range is found from the said first-level index area through the binary search method, and the data result of the preset data is read back.

[0017] In an exemplary embodiment of the present disclosure, the method further includes:

[0018] In step S130 , if the key value to be queried is retrieved, step S140 is performed.

[0019] In an exemplary embodiment of the present disclosure, the method further includes:

[0020] The index file includes, from beginning to end, a header metadata area, a plurality of first-level index areas, a second-level index area, and a tail metadata area.

[0021] In an exemplary embodiment of the present disclosure, the method further includes:

[0022] By controlling the number of first-level index nodes in the first-level index area, the memory usage of the second-level index area, filter, and first-level index area in the index file can be adjusted.

[0023] In a second aspect, the present application provides an index persistence device for a KV storage system, the device comprising:

[0024] A query module is configured to query a first preset layer of a log structure merge tree, wherein the first preset layer of the log structure merge tree includes a set of preset indexes, and a key value key range of the index found in the tail metadata area is a first preset data range;

[0025] A first data reading module is configured to query the index file corresponding to the first preset range, read all secondary indexes from the index file, add them to the secondary index cache, search for the primary index area and the starting position of the filter that the query hits, read the filter and put it into the filter cache, and if the filter hits, read the primary index area and put it into the cache, find the part of the second preset data range from the primary index area through a binary search method, and read back the data result of the preset data;

[0026] A second data reading module is configured to perform data query in the second preset layer and the third preset layer of the log structure merge tree based on step S120, check the key value to be queried, and read back the data result of the preset data;

[0027] The data merging module is used to merge the data results read back from the first preset layer, the second preset layer, and the third preset layer of the log structure merge tree and return them.

[0028] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0029] In a fourth aspect, the present application shows a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method described in any of the above aspects.

[0030] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in any one of the above aspects.

[0031] The present application discloses an index persistence method for a KV storage system, wherein the method includes: querying the first preset layer of a log structure merge tree, and finding from the tail metadata area that the key value range of the index is a first preset data range; querying the index file corresponding to the first preset range, and finding the part of the second preset data range from the first-level index area by a binary search method, and reading back the data result of the preset data; performing data query in the second preset layer of the log structure merge tree and the third preset layer of the log structure merge tree, and checking the key value key to be queried, and reading back the data result of the preset data; merging the data results read back in the first, second, and third preset layers and returning them. The disclosed index persistence method can reduce the index file read amplification without increasing the metadata management pressure of the file system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flowchart of the steps of an index persistence method for a KV storage system of this application.

[0033] Figure 2 This is a data flow diagram of an index persistence method for a KV storage system of this application.

[0034] Figure 3 This is a structural block diagram of an index persistence device for a KV storage system of the present application.

[0035] Figure 4 This is a block diagram of an electronic device of the present application.

[0036] Figure 5 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0038] The present application discloses an index persistence method for a KV storage system, wherein the method includes: querying the first preset layer of a log structure merge tree, and finding from the tail metadata area that the key value range of the index is a first preset data range; querying the index file corresponding to the first preset range, and finding the part of the second preset data range from the first-level index area by a binary search method, and reading back the data result of the preset data; performing data query in the second preset layer of the log structure merge tree and the third preset layer of the log structure merge tree, and checking the key value key to be queried, and reading back the data result of the preset data; merging the data results read back in the first, second, and third preset layers and returning them. The disclosed index persistence method can reduce the index file read amplification without increasing the metadata management pressure of the file system.

[0039] Example 1:

[0040] In this example embodiment, in a KV (Key-Value) storage system:

[0041] LSM tree log-structured merge tree: LSM tree log-structured merge tree, a common storage engine for KV storage systems.

[0042] Compaction: A mechanism for releasing garbage data in the LSM tree log structure merge tree.

[0043] Memtable: An index table used in an LSM tree's log-structured merge tree to respond to write operations and supports random insertions and lookups. Immutable Memtable: An immutable index table, converted from an index table under certain conditions, does not support insertions and is generally used to respond to queries.

[0044] SSTable Sorted String Table: Sorted string table is a persistent expression of the immutable index table.

[0045] Reference Figure 1 , shows a flowchart of the steps of a method for index persistence in a KV storage system of the present application, which can be applied to electronic devices, wherein the method specifically may include the following steps:

[0046] In step S110, the first preset layer of the log structure merge tree can be queried. The first preset layer of the log structure merge tree includes a set of preset indexes, and the key value key range of the index found in the tail metadata area is the first preset data range.

[0047] In step S120, the index file corresponding to the first preset range can be queried, all secondary indexes can be read from the index file, added to the secondary index cache, and the first-level index area hit by the query and the starting position of the filter can be found, the filter can be read and put into the filter cache, if the filter hits, the first-level index area can be read and put into the cache, and the part of the second preset data range can be found from the first-level index area through the binary search method, and the data result of the preset data can be read back.

[0048] In the embodiment of this example, the preset data in the method further includes:

[0049] The information value value file name, value value offset and length in the file, and the location of value value data.

[0050] In step S130, data query can be performed in the second preset layer of the log structure merge tree and the third preset layer of the log structure merge tree based on step S120, and the key value key to be queried is checked, and the data result of the preset data is read back.

[0051] In the embodiment of this example, step S130 of the method further includes:

[0052] Based on step S120, data query is performed in the second preset layer of the log structure merge tree, and the key value key to be queried is checked. The part of the third preset data range is found from the said first-level index area through the binary search method, and the data result of the preset data is read back.

[0053] In the embodiment of this example, step S130 of the method further includes:

[0054] Based on step S130, data query is performed in the third preset layer of the log structure merge tree, and the key value key to be queried is checked, and a part of the fourth preset data range is found from the said first-level index area through the binary search method, and the data result of the preset data is read back.

[0055] In this exemplary embodiment, the method further includes:

[0056] In step S130 , if the key value to be queried is retrieved, step S140 is performed.

[0057] In step S150 , the data results read back from the first preset layer, the second preset layer, and the third preset layer of the log structure merge tree may be merged and returned.

[0058] In this exemplary embodiment, the method further includes:

[0059] The index file includes, from beginning to end, a header metadata area, a plurality of first-level index areas, a second-level index area, and a tail metadata area.

[0060] In this exemplary embodiment, the method further includes:

[0061] By controlling the number of first-level index nodes in the first-level index area, the memory usage of the second-level index area, filter, and first-level index area in the index file can be adjusted.

[0062] Example 2:

[0063] In the embodiment of this example, a method for persisting an index file that supports partial loading is proposed. The content of the index file includes a header metadata area, several first-level index areas, a second-level index area, and a tail metadata area from beginning to end.

[0064] Header metadata area: fixed length, contains file identification information, version information, etc.

[0065] The first-level index area consists of several first-level index nodes arranged in ascending order by key value. Each first-level index node contains the key value, the data file where the value resides, and the offset and length of the value within the data file. A Bloom filter is also included at the end of each first-level index area to quickly filter out non-existent requests during queries. The persistent format of the Bloom filter includes the number of hash functions, the filter length, and the binary contents of the filter. The specific construction of the filter is not the focus of this article and will not be discussed in detail.

[0066] The secondary index area is composed of several secondary index nodes of the same size. Each secondary index node uniquely points to a primary index area and contains the starting key, ending key, starting offset and length in the file, and the starting offset and length of the filter to which it belongs.

[0067] The tail metadata area has a fixed length and contains information that is determined only after all indexes in the index file are written. This includes the file's entire start key, end key, and the start offset and length of the secondary index.

[0068] In this exemplary embodiment, Figure 2 As shown in the figure, the query process using the persistence format is described by taking the range query key [50,70] as an example. Figure 3As shown. Step 1, query the level 1 index, which contains a set of unchanged indexes. The key range of the index is found in the tail metadata area, which is [1,100]. Step 2, first query the index file corresponding to [1,100]. First, read all the secondary indexes from the file and add them to the secondary index cache, and find the primary index area and the starting position of the filter that the query hits, read the filter and put it into the filter cache. If the filter also hits, continue to read the entire primary index area and put it into the cache. Use the binary search method to find the key [50,55] in the primary index area, and locate the specific location of the value data based on the information recorded therein, such as the value file name, the offset of the value in the file, and the length, and read back the data. Steps 3-4, repeat step 2 to continue querying the data in levels 2 and 3. After querying each index, check whether all the keys to be queried have been found. If not, continue to query backward. For level 2 and level 3, the index files corresponding to key [45, 80] and key [1, 150] are hit, respectively, and the required data is read back. Step 5 merges the read results and returns them. This process shows that for the hit index files, the content read from disk includes tail metadata, secondary index areas, a small number of primary index areas hit by the query, and filters. During implementation, the memory usage of the secondary index area, filters, and primary index area in the index file can be adjusted by controlling the number of primary index nodes in the primary index area. Because the secondary index area and filter area have a high cache hit rate and occupy less memory, the index cache allocation strategy should ensure that the secondary indexes and filters of all index files are cached as much as possible to avoid unnecessary file read overhead.

[0069] In the embodiment of this example, the method described in the present disclosure uses the above-mentioned index persistence method in the KV separation scenario, which can reduce the index file read amplification without increasing the file system metadata management pressure.

[0070] It should be noted that for the method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions involved are not necessarily required by this application.

[0071] Reference Figure 3 , shows a structural block diagram of an index persistence device for a KV storage system of the present application, the device comprising: a query module 210, a first data reading module 220, a second data reading module 230 and a data merging module 240:

[0072] A query module 210 is configured to query a first preset layer of a log-structured merge tree, wherein the first preset layer of the log-structured merge tree includes a set of preset indexes, and a key value key range of the index found in the tail metadata area is a first preset data range;

[0073] The first data reading module 220 is configured to query the index file corresponding to the first preset range, read all secondary indexes from the index file, add them to the secondary index cache, search for the primary index area and the starting position of the filter that the query hits, read the filter and store it in the filter cache, and if the filter hits, read the primary index area and store it in the cache, find the portion of the second preset data range from the primary index area using a binary search method, and read back the data result of the preset data;

[0074] The second data reading module 230 is used to perform data query in the second preset layer and the third preset layer of the log structure merge tree based on step S120, check the key value to be queried, and read back the data result of the preset data;

[0075] The data merging module 240 is used to merge the data results read back from the first preset layer, the second preset layer, and the third preset layer of the log structure merge tree and return them.

[0076] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0077] Optionally, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0078] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the various processes of the above-described method embodiments are implemented and the same technical effects are achieved. To avoid repetition, the details are not described here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0079] Figure 48 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0080] Reference Figure 4 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0081] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0082] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0083] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0084] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0085] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0086] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0087] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0088] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0089] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0090] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0091] Figure 5 19 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0092] Reference Figure 5 Computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as applications, that can be executed by processing component 1922. The application stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, processing component 1922 is configured to execute instructions to perform the above-described method.

[0093] The computer-readable storage medium 1900 may further include a power supply component 1926 configured to perform power management of the computer-readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer-readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer-readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0094] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0096] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0097] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0098] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0103] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An index persistence method for a KV storage system, characterized in that: The method comprises: Step S110, querying a first preset layer of the log structure merge tree, wherein the first preset layer of the log structure merge tree includes a set of preset indexes, and a key value key range of the index is found to be a first preset data range in the tail metadata area; Step S120, query the index file corresponding to the first preset range, read all secondary indexes from the index file, add them to the secondary index cache, and search for the primary index area and the starting position of the filter hit by the query, read the filter and put it into the filter cache, if the filter hits, read the primary index area and put it into the cache, find the part of the second preset data range from the primary index area through binary search method, and read back the data result of the preset data; Step S130, based on step S120, performing data query in the second preset layer of the log structure merge tree and the third preset layer of the log structure merge tree, and checking the key value key to be queried, and reading back the data result of the preset data; Step S140, merging the data results read back from the first preset layer, the second preset layer, and the third preset layer of the log structure merge tree respectively and returning them.

2. The method according to claim 1, characterized in that The preset data in the method further includes: The information value value file name, value value offset and length in the file, and the location of value value data.

3. The method according to claim 1, characterized in that Step S130 of the method further includes: Based on step S120, data query is performed in the second preset layer of the log structure merge tree, and the key value key to be queried is checked, and a part of the third preset data range is found from the first-level index area through a binary search method, and the data result of the preset data is read back.

4. The method according to claim 1, characterized in that Step S130 of the method further includes: Based on step S130, data query is performed in the third preset layer of the log structure merge tree, and the key value key to be queried is checked, and a part of the fourth preset data range is found from the first-level index area through a binary search method, and the data result of the preset data is read back.

5. The method according to claim 1, characterized in that The method further comprises: In step S130 , if the key value key to be queried is retrieved, step S140 is performed.

6. The method according to claim 1, characterized in that The method further comprises: The index file includes, from beginning to end, a header metadata area, a plurality of primary index areas, a secondary index area, and a tail metadata area.

7. The method according to claim 1, characterized in that The method further comprises: By controlling the number of primary index nodes in the primary index area, the memory ratio of the secondary index area, filter and primary index area in the index file can be adjusted.

8. An index persistence device for a KV storage system, characterized in that: The device comprises: A query module is used to query a first preset layer of a log structure merge tree, wherein the first preset layer of the log structure merge tree includes a set of preset indexes, and a key value key range of the index found in the tail metadata area is a first preset data range; A first data reading module is used to query the index file corresponding to the first preset range, read all secondary indexes from the index file, add the secondary index cache, and search for the primary index area hit by the query and the starting position of the filter, read the filter and put it into the filter cache, if the filter hits, read the primary index area and put it into the cache, find the part of the second preset data range from the primary index area through a binary search method, and read back the data result of the preset data; A second data reading module is used to perform data query in the second preset layer of the log structure merge tree and the third preset layer of the log structure merge tree based on step S120, and check the key value key to be queried, and read back the data result of the preset data; The data merging module is used to merge the data results read back from the first preset layer, the second preset layer, and the third preset layer of the log structure merge tree and return them.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images