A storage engine performance optimization method based on heat perception

Abstract

The application provides a heat-aware storage engine performance optimization method, which comprises the following steps: determining a target database, initializing a maximum heap in the memory of the target database, and maintaining the file heat of the system; reading file heat information from the log of the target database, and recovering the file heat identification mechanism based on the maximum heap; writing the file heat information into a Memtable of an LSM-tree memory component by a storage engine, and making the data of the Memtable fall to the disk and generate a CuckooTable file; establishing a Bloom filter for the CuckooTable file on an NVM; recording the corresponding relationship between the Bloom filter and the file in an index queue; and updating the index queue when the number of files in the LSM-tree reaches a preset threshold. The method provided by the application can be used to optimize the performance of a storage engine designed with a LSM-tree structure.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for optimizing storage engine performance based on heat perception. Background Technology

[0002] With the continuous development of the big data era, society has increasingly higher requirements for the read and write performance of key-value storage systems, and the demand for their application scope is also expanding. On the one hand, the popularization of interactive applications such as social networks has led to an increasing demand for low-latency data writing; on the other hand, big data analysis is gradually becoming a powerful tool for mining the value of big data, leading to an increasing demand for high-performance data querying. Broadly speaking, databases can be divided into non-relational databases (NoSQL) and relational databases (SQL). Among them, the mainstream non-relational databases on the market, such as column-oriented databases HBase and ClickHouse, and document databases MongoDB and Cassendra, have all chosen an LSM-tree-like structure to implement their main underlying storage engine; while the most widely used relational database, MySQL, also implemented an LSM-tree-based version, MyRocks, a few years ago. This is sufficient to prove the important position of LSM-tree-like storage engines in data storage.

[0003] During a query, in order to find the latest data, the storage engine must search each candidate file in the LSM-tree layer by layer from top to bottom. First, it accesses the Bloom filter to predict whether each candidate file contains the target data. Then, it reads files that might contain the target data and searches them. The false positive rate of the Bloom filter determines the number of disk I / O operations, and the complexity of the file data search determines the speed of reading key-value pairs; these are both key factors affecting database query performance.

[0004] Current technologies focus on optimizing query performance from the perspective of optimizing caching and designing indexes, but ignore the problems of high disk access and file data reading complexity caused by the false positive rate of Bloom filters: (1) As a predictive tool before file access, the false positive rate of Bloom filters has a significant impact on disk I / O. Since the space occupation of Bloom filters is inversely proportional to the false positive rate, limited memory resources cannot support the huge Bloom filter cache, resulting in a high average false positive rate of Bloom filters in the system, and also causing problems such as cache invalidation and disk I / O surge. (2) Existing space allocation strategies are optimized for Bloom filter cache queues, but in fact, the frequent compaction process in the background of LSM-tree will cause disk files to be continuously deleted and created, and Bloom filters in the cache queue will also frequently become invalid, swapped in and swapped out, increasing maintenance costs and causing performance jitter. (3) Existing data files contain a lot of content such as metadata, index data, and filters, requiring multiple steps of searching to locate the data block where the data is located, resulting in high query complexity and hindering fast data location. Summary of the Invention

[0005] This invention provides a heat-aware storage engine performance optimization method that can be used to optimize the performance of storage engines designed with an LSM-tree-like structure.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, embodiments of this application propose a storage engine performance optimization method based on popularity awareness, comprising: determining a target database; initializing a max heap in the memory of the target database to maintain file popularity in the system; reading file popularity information from the logs of the target database and restoring the file popularity identification mechanism based on the max heap; the storage engine writing the file popularity information into the memory component Memtable of the LSM-tree, and persisting the data of the Memtable to disk and generating a CuckooTable file; establishing a Bloom filter for the CuckooTable file on NVM; recording the correspondence between the Bloom filter and the files in an index queue; and updating the index queue when the number of files in the LSM-tree reaches a preset threshold.

[0008] In some implementations, the storage engine writes the file popularity information into the LSM-tree's memory component Memtable, and writes the data in the Memtable to disk and generates a CuckooTable file, including: if the Memtable is full, triggering a disk write operation.

[0009] In some implementations, the CuckooTable file is based on the Cuckoo Hash algorithm, which uses a hash function to insert and retrieve data.

[0010] In some implementations, recording the correspondence between the Bloom filter and the file in the index queue includes: setting the access frequency of the file to 0 and inserting it at the end of the max heap.

[0011] In some implementations, updating the index queue when the number of files in the LSM-tree reaches a preset threshold further includes: merging and sorting some of the files to reorganize them into a new CuckooTable file; synchronously deleting information related to the files in the max heap, and creating a new node at the end of the heap to store information related to the new files.

[0012] In some implementations, the method further includes: submitting a query request to a storage engine, which first searches the memory component Memtable of the LSM-tree; if no data is found, it searches the Bloom filter index queue in the NVM based on the data's key to find candidate files; sequentially accessing the Bloom filter of each candidate file; when the Bloom filter determines that data may exist in the current file, accessing the CuckooTable file on the disk; using the CuckooHash algorithm to search for data in the file and determine whether the data exists; if the data exists, using the CuckooHash algorithm to read the data.

[0013] In some embodiments, the method further includes:

[0014] If no data is found, the CuckooTable file on the disk is accessed again. In some implementations, before synchronously deleting information related to the file from the max-heap and creating a new node at the end of the heap to store information related to the new file, the method further includes: determining the node in the max-heap corresponding to the target file, incrementing its access popularity value by 1, and then adjusting the heap so that the access popularity of the file pointed to by the upper-level node is always greater than that of the file pointed to by the lower-level node. In some implementations, the method further includes: adjusting the size of the Bloom filter based on a file popularity identification mechanism in memory during off-peak periods.

[0015] In some implementations, the preset threshold is a threshold for the number of files in a single layer of the LSM-tree.

[0016] Secondly, this application proposes a heat-aware storage engine performance optimization device, comprising:

[0017] A database determination module is used to determine the target database, initialize a maximum heap in the memory of the target database, and maintain the file popularity of the system.

[0018] A recovery module is used to read file popularity information from the logs of the target database and restore the file popularity identification mechanism based on the max-heap.

[0019] The disk persistence module is used by the storage engine to write the file popularity information into the memory component Memtable of the LSM-tree, and to persist the data of the Memtable to disk and generate the CuckooTable file.

[0020] A new module is created to establish a Bloom filter for the CuckooTable file on NVM;

[0021] A recording module records the correspondence between the Bloom filter and the file into an index queue;

[0022] An update module is used to update the index queue when the number of files in the LSM-tree reaches a preset threshold.

[0023] The heat-aware storage engine performance optimization method and apparatus proposed in this application have the following beneficial effects:

[0024] 1. This invention designs a heat-aware Bloom filter space allocation strategy to ensure that the Bloom filter for hot files has a lower false positive rate, thereby reducing the system's average false positive rate. The false positive rate of a Bloom filter is inversely proportional to its storage space usage. To reduce the false positive rate, it is necessary to increase the number of bits allocated to each key in the Bloom filter. However, blindly increasing space to reduce the false positive rate is not advisable. Therefore, this invention designs a file heat recognition mechanism based on a max-heap in memory to assist the system in dynamically adjusting the size of the Bloom filter, ensuring that the Bloom filter for hot files has a lower false positive rate, thereby reducing the system's average false positive rate.

[0025] 2. By placing the Bloom filter in the faster persistent medium NVM, the problems of frequent filter cache invalidation and disk I / O surges caused by limited memory and cache contention are avoided. Due to limited memory resources and the high cost of physical memory expansion, the system cannot cache all Bloom filters, leading to frequent filter cache invalidation and disk I / O surges during queries. Therefore, this invention uses NVM, a persistent medium with read / write speeds close to memory and a price between memory and disk, to store the Bloom filter. This saves memory space for caching hot data and avoids disk I / O caused by accessing the filter.

[0026] 3. The storage engine's data storage format has been optimized, reducing the query complexity of data in files from a constant level of O(1), effectively improving the data query speed. Traditional storage engine file storage formats consist of four parts: metadata module, filter module, index module, and data module. Generally, finding a piece of data requires first reading the Bloom filter in the filter module to determine if the data might exist, then obtaining the index location from the metadata, and finally reading the corresponding data block according to the index, performing a binary search. However, the CuckooTable designed in this invention only needs to use the key to calculate the data location, requiring no metadata or index data assistance, and no binary search to quickly read the data, thus effectively improving search efficiency.

[0027] Thirdly, in a third aspect of the embodiments of this application, an electronic device is provided, the device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of the first aspect described above.

[0028] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, and a processor executes the computer program to implement the method of the first aspect above.

[0029] This invention provides a storage engine performance optimization method and apparatus based on popularity awareness. The method involves identifying a target database, initializing a max-heap in the target database's memory, and maintaining file popularity metrics. File popularity information is read from the target database's logs, restoring the max-heap-based file popularity identification mechanism. The storage engine writes the file popularity information into the Memtable, a memory component of the LSM-tree, and persists the Memtable data to disk, generating a CuckooTable file. A Bloom filter is created for the CuckooTable file on the NVM. The mapping between the Bloom filter and the files is recorded in an index queue. When the number of files in the LSM-tree reaches a preset threshold, the index queue is updated. The purpose of this invention is to provide a storage engine performance optimization method based on popularity awareness, which can be used to optimize the performance of storage engines designed with an LSM-tree-like structure. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating a heat-aware storage engine performance optimization method proposed in this application.

[0031] Figure 2 A system architecture diagram provided in this embodiment of the invention;

[0032] Figure 3A diagram of a heat-sensing Bloom filter allocation algorithm provided in an embodiment of the present invention;

[0033] Figure 4 This is a structural diagram of a file storage format, CuckooTable, provided in an embodiment of the present invention.

[0034] Figure 5 A data insertion flowchart is provided for an embodiment of the present invention;

[0035] Figure 6 A data query flowchart provided in an embodiment of the present invention;

[0036] Figure 7 A structural block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0037] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0038] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0039] The Log-Structured Merge Tree (LSM-tree) (O'NEIL P, CHENGE, GAWLICK D, et al. 1996. The log-structured merge-tree (LSM-tree). Acta Informatica [J], 33: 351-385.) employs an out-of-place update mechanism for data writing and updating. This means that data updates and deletions do not require locating the data or immediately replacing the old version. Instead, all operations are unified as write operations, with new data written directly to the system in the order it was written. The background compaction process then delays the sorting of data and the reclamation of old versions. While this out-of-place update mechanism transforms random disk writes into sequential writes, significantly reducing write latency, it also sacrifices some query performance due to the unordered and repetitive nature of the data.

[0040] This application proposes a heat-aware storage engine performance optimization method to improve the above-mentioned problems.

[0041] This application proposes a heat-aware storage engine performance optimization method, including steps S101 to S106.

[0042] S101: Determine the target database, initialize a max heap in the memory of the target database, and maintain the file hotness of the system.

[0043] Open the database, initialize a max-heap in memory, and maintain the system's file popularity index. Read the file popularity information from the logs and restore the max-heap-based file popularity identification mechanism.

[0044] S102: Read file popularity information from the logs of the target database and restore the file popularity identification mechanism based on the max-heap.

[0045] When the database receives a piece of data, the storage engine writes the data into the Memtable, the memory component of the LSM-tree. Once the Memtable is full, the disk write operation is triggered.

[0046] S103: The storage engine writes the file popularity information into the memory component Memtable of the LSM-tree, and writes the data of the Memtable to disk and generates the CuckooTable file.

[0047] The data from the Memtable is written to disk, forming a CuckooTable file. The CuckooTable file is implemented using the CuckooHash algorithm, employing a hash function for data insertion and retrieval. In addition to the data, the CuckooTable also stores metadata such as file size and data range.

[0048] S104: Create a Bloom filter for the CuckooTable file on NVM.

[0049] S105: Record the correspondence between the Bloom filter and the file into the index queue;

[0050] S106: When the number of files in the LSM-tree reaches a preset threshold, update the index queue.

[0051] When the number of files at a certain level of an LSM-tree reaches a threshold, and compaction is required for that level, some files are merged, sorted, and reorganized into a new CuckooTable file. Simultaneously, information related to these older files is deleted from the max-heap, and a new node is created at the end of the heap to store information about the new files. Next, the system deletes the Bloom filter corresponding to these older files on the NVM and creates a new Bloom filter, while also updating the index queue.

[0052] A Bloom filter is created for the CuckooTable file on NVM and added to a queue. Since the system cannot determine the appropriate size of the Bloom filter based on file access frequency when a new file is created, this invention utilizes the layer-by-layer search characteristic of LSM-trees to discover that file access frequency generally shows a decreasing trend layer by layer, and initializes the Bloom filter allocation space based on this pattern. After constructing the Bloom filter, the correspondence between the Bloom filter and the file is recorded in the index queue.

[0053] S201: In some embodiments, when querying data from a database, the method proposed in this application may further include the following steps:

[0054] The query request is submitted to the storage engine. The storage engine first searches the in-memory component Memtable of the LSM-tree. If no data is found, it searches the Bloom filter index queue in NVM based on the data's key to find candidate files.

[0055] S202: Access the Bloom filter of each candidate file in turn. Access the CuckooTable file on disk only if the Bloom filter determines that the data may exist in the current file.

[0056] Understandably, the CuckooHash algorithm is used to search for data in the file and determine if the data exists. If it does not exist, it means that the Bloom filter misjudged. This process is repeated until the target file is found, and then the CuckooHash algorithm is used to read the data.

[0057] Find the node in the max heap corresponding to the target file, increment its access popularity value by 1, and then adjust the heap so that the access popularity of the file pointed to by the upper node is always greater than that of the file pointed to by the lower node.

[0058] During off-peak hours, the system adjusts the size of the Bloom filter based on a file popularity mechanism in memory, ensuring that the Bloom filter for popular files has more space, while the Bloom filter for infrequently accessed files has less space. This effectively reduces the average false positive rate of the Bloom filter in the system, thereby reducing disk file access.

[0059] The system architecture design of this invention is attached. Figure 2As shown, the database is divided into three parts: a memory component, an NVM component, and a disk component. The memory component, in addition to the database's built-in data cache, introduces a file hotness identification mechanism based on a max-heap. The top of the heap represents hot files, and the bottom represents cold files. A node consists of an array that can point to multiple files. This mechanism is mainly used to record the access frequency of files during queries and distinguish between hot and cold files based on access frequency, facilitating the adjustment of the Bloom filter allocation space for the corresponding file. The NVM component includes Bloom filters for all files and their index queues. Before accessing a disk file, the corresponding Bloom filter for that file should be checked in the NVM; disk access is only performed if the Bloom filter determines that the data may exist. The disk component uses an LSM-tree data structure and is mainly used for physical data storage and file management. Each file is implemented based on Cuckoo Hash, leveraging the advantages of Cuckoo Hash such as low memory consumption, low query complexity, and low false positive rate to improve data query performance.

[0060] Suppose a key-value storage system has N files, N kv There are ___ key-value pairs, and the total space allocated to the Bloom filter is M, where the Bloom filter for the i-th file allocates a length of mi bits for each key, with an access frequency of p. i The average false positive rate of a Bloom filter in a key-value storage system is the sum of the products of the false positive rates of all Bloom filters and their access frequencies. Therefore, the formula for the false positive rate is... As can be seen, the average false positive rate of the system is shown in formula (4.1):

[0061]

[0062] Where N hash Indicates the use of N hash A hash function is used to insert a Bloom filter.

[0063] We aim to minimize the system's average false positive rate, which is to minimize This problem can be viewed as a combinatorial optimization problem:

[0064]

[0065] By solving the combinatorial optimization problem in (4.2), we can obtain the following conclusion: when the Bloom filter of the i-th file allocates m bits to each key... i satisfy:

[0066] m i =log p i -b, i = {1, ..., N} (4.3) The average false positive rate of the system query is the lowest. At this time, the average false positive rate is always less than the average false positive rate when allocating the same size (m) to each file.i Average false positive rate when the Bloom filter space is M / N

[0067]

[0068] in, A constant related to access frequency.

[0069] Therefore, the implementation scheme of the heat-sensing Bloom filter allocation algorithm of the present invention can be obtained, as shown in the appendix. Figure 3 As shown: During the initialization phase, this invention analyzes and obtains the access frequency P of each layer based on the law of decreasing file access popularity layer by layer. The number of Bloom filter bits m0 in layer L0 of the LSM-tree is specified by system parameters, and a constant b is calculated according to b = logP0 – m0. Next, different sizes of space are allocated to the Bloom filters of different layers according to mi = logPi – b. Subsequently, as business queries proceed, the file access frequency changes, and these changes are recorded in the file awareness mechanism. Each layer of the max-heap in the file awareness mechanism has a threshold Ki. When a file originally in layer i has an access frequency Pi that increases to greater than Ki, the file is moved up. The system periodically updates the Bloom filter space corresponding to these moved files, calculating the new space for the corresponding Bloom filter according to mi = logPi – b.

[0070] As a storage engine, this invention's basic operations are CRUD (Create, Read, Update, Delete). For an LSM-tree structure, to minimize latency during massive data modifications, it fully leverages the advantages of sequential disk writes, unifying modification and deletion into write operations. Therefore, for an LSM-tree-based storage engine, CRUD operations can be completed simply by inserting data. The following section will discuss this further. Figure 4 This section provides a detailed introduction to the data insertion operations of the storage engine.

[0071] When the database receives a piece of data, the storage engine inserts the data into the Memtable of the LSM-tree in an append-only manner.

[0072] Once the Memtable is full, the data is persisted as CuckooTable files to the L0 level of the LSM-tree, which is the first level on disk. CuckooTable files are a file storage format with lower lookup complexity and faster query speed, implemented based on the Cuckoo Hash algorithm. Its structure is shown in the attached figure. Figure 3As shown, it consists of two parts: a data block and an attribute block. The insertion process for the data block is as follows: The key is calculated using the hash function HashA to obtain position A, and then the key is calculated using the hash function HashB to obtain position B. If both positions are empty, the current data is inserted into either position; if one position is empty, the data is inserted into the empty position; if neither position is empty, the data in one position is removed, and the current data is inserted. This algorithm is repeated for the removed data until another empty position is found for insertion. In addition to the data, the attribute block in CuckooTable also stores some metadata, such as file size and data range.

[0073] A Bloom filter is built for the CuckooTable file on NVM and added to a queue. For newly created files, this invention analyzes the decreasing access frequency P of each layer based on the LSM-tree's pattern of decreasing file access popularity. Different sizes of space are initialized for the Bloom filters of different layers according to mi = logPi – b. Specifically, the construction process is as follows: Extract the database parameter bits-per-key to obtain the initial number of bits m0 for the L0 layer Bloom filter. Then, a hash function is used to map the key to m0 bits, setting the corresponding value to 1. Finally, the entry <Bloom filter number, file layer, file number, file data range> is inserted into the NVM Bloom filter index queue. This process is repeated for the key of each data entry during the construction of CuckooTable to complete the construction of the Bloom filter.

[0074] Add a node to the end of the max heap in memory to indicate that the access hotness of the new file is 0.

[0075] When the number of files in the Li layer of the LSM-tree reaches a threshold, the files in that layer need to be compacted. This involves merging and sorting a portion of the files in the Li and L(i+1) layers, reorganizing them into a new CuckooTable file, and placing it into L(i+1). Next, information related to these old files will be simultaneously deleted from the max-heap, and a new node will be created at the end of the heap to store information related to the new files.

[0076] Delete the Bloom filter corresponding to the old file on NVM, and create a new Bloom filter according to mi+1 = logPi+1 – b. Finally, update the index queue.

[0077] The database query process is as follows: Figure 6 As shown, it can be divided into the following steps:

[0078] S1: Submit the query request to the storage engine. The storage engine first performs a binary search on the LSM-tree's Memtable to check if the target data exists.

[0079] S2: If the data is not in the Memtable, then search the Bloom filter queue in NVM based on the data's key. The search process is as follows: First, filter candidate files that may contain data based on the file range recorded in the queue; then, sequentially access the Bloom filter corresponding to each candidate file to determine if the data exists. If the mi bit of the hash map is 0, then the data does not exist, and search for the next one; if all the mi bits of the hash map are 1, then access the CuckooTable file, use the CuckooHash algorithm to search for data in the file, and determine if the target data exists.

[0080] S3: If the target data does not exist, it means that the Bloom filter has misjudged. Repeat step 2 until the target file is found and the data is read.

[0081] S4: Adjust the node in the max heap corresponding to the target file in memory, and increment its access popularity by 1. If its access popularity already meets the upward movement threshold, then adjust the heap so that it always satisfies the condition that the access popularity of the file pointed to by the upper node is greater than that of the file pointed to by the lower node.

[0082] S5: Periodically, the system adjusts the size of the Bloom filter based on the file popularity recognition mechanism in memory. The adjustment rules are as follows: When a file changes level in the max heap, the Bloom filter corresponding to that file needs to be updated. Assuming its access frequency becomes Pi', the Bloom filter space is recalculated according to mi' = logPi' – b, and the original space is expanded or reduced accordingly. This ensures that the Bloom filter in the system always maintains a low average false positive rate, reducing disk file access.

[0083] The heat-aware storage engine performance optimization method proposed in this application has the following beneficial effects:

[0084] 1. This invention designs a heat-aware Bloom filter space allocation strategy to ensure that the Bloom filter for hot files has a lower false positive rate, thereby reducing the system's average false positive rate. The false positive rate of a Bloom filter is inversely proportional to its storage space usage. To reduce the false positive rate, it is necessary to increase the number of bits allocated to each key in the Bloom filter. However, blindly increasing space to reduce the false positive rate is not advisable. Therefore, this invention designs a file heat recognition mechanism based on a max-heap in memory to assist the system in dynamically adjusting the size of the Bloom filter, ensuring that the Bloom filter for hot files has a lower false positive rate, thereby reducing the system's average false positive rate.

[0085] 2. By placing the Bloom filter in the faster persistent medium NVM, the problems of frequent filter cache invalidation and disk I / O surges caused by limited memory and cache contention are avoided. Due to limited memory resources and the high cost of physical memory expansion, the system cannot cache all Bloom filters, leading to frequent filter cache invalidation and disk I / O surges during queries. Therefore, this invention uses NVM, a persistent medium with read / write speeds close to memory and a price between memory and disk, to store the Bloom filter. This saves memory space for caching hot data and avoids disk I / O caused by accessing the filter.

[0086] 3. The storage engine's data storage format has been optimized, reducing the query complexity of data in files from a constant level of O(1), effectively improving the data query speed. Traditional storage engine file storage formats consist of four parts: metadata module, filter module, index module, and data module. Generally, finding a piece of data requires first reading the Bloom filter in the filter module to determine if the data might exist, then obtaining the index location from the metadata, and finally reading the corresponding data block according to the index, performing a binary search. However, the CuckooTable designed in this invention only needs to use the key to calculate the data location, requiring no metadata or index data assistance, and no binary search to quickly read the data, thus effectively improving search efficiency.

[0087] Secondly, this application proposes a heat-aware storage engine performance optimization device, comprising:

[0088] A database determination module is used to determine the target database, initialize a maximum heap in the memory of the target database, and maintain the file popularity of the system.

[0089] A recovery module is used to read file popularity information from the logs of the target database and restore the file popularity identification mechanism based on the max-heap.

[0090] The disk persistence module is used by the storage engine to write the file popularity information into the memory component Memtable of the LSM-tree, and to persist the data of the Memtable to disk and generate the CuckooTable file.

[0091] A new module is created to establish a Bloom filter for the CuckooTable file on NVM;

[0092] A recording module records the correspondence between the Bloom filter and the file into an index queue;

[0093] An update module is used to update the index queue when the number of files in the LSM-tree reaches a preset threshold.

[0094] Optionally, embodiments of the present invention also provide an electronic device for performing the method provided in any embodiment of the present invention.

[0095] like Figure 7 As shown, electronic device 2000 may include processor 2001.

[0096] Optionally, the electronic device 2000 may also include a memory 2002.

[0097] The processor 2001 is coupled to the memory 2002, which can be connected via a communication bus.

[0098] The following is combined Figure 7 A detailed introduction to each component of the electronic device 2000:

[0099] The processor 2001 is the control center of the electronic device 2000. It can be a single processor or a collective term for multiple processing elements. For example, the processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0100] Optionally, the processor 2001 can perform various functions of the electronic device 2000 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0101] In a specific implementation, as one example, the processor 2001 may include one or more CPUs, for example... Figure 7 CPU0 and CPU1 are shown in the diagram.

[0102] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0103] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having an instruction or data structure form and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the processor 2001 or exist independently, and may be accessed through the interface circuit of the electronic device 2000. Figure 7 (Not shown in the figure) is coupled to processor 2001, and the embodiments of the present invention do not specifically limit this.

[0104] It should be noted that, Figure 7 The structure of the electronic device 2000 shown does not constitute a limitation on the electronic device. Actual electronic devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0105] Furthermore, the technical effects of the electronic device 2000 can be referred to the technical effects of the methods described in the above method embodiments, and will not be repeated here.

[0106] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0107] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0108] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0109] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0112] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A heat-aware storage engine performance optimization method, characterized in that, include: Identify the target database, initialize a max heap in the memory of the target database, and maintain the file popularity of the system; File popularity information is read from the logs of the target database to restore the file popularity identification mechanism based on max-heap, where the file popularity information is the access frequency of the file; The storage engine writes the file popularity information into the LSM-tree's memory component Memtable, and then persists the data in the Memtable to disk and generates a CuckooTable file. Create a Bloom filter for the CuckooTable file on NVM; The correspondence between the Bloom filter and the file is recorded in the index queue; When the number of files in the LSM-tree reaches a preset threshold, the index queue is updated.

2. The heat-aware storage engine performance optimization method according to claim 1, characterized in that, The storage engine writes the file popularity information into the LSM-tree's memory component, Memtable, and persists the data in the Memtable to disk, generating a CuckooTable file, including: If the Memtable is full, a disk write operation is triggered.

3. The heat-aware storage engine performance optimization method according to claim 2, characterized in that, The CuckooTable file is implemented based on the Cuckoo Hash algorithm, which uses a hash function to insert and retrieve data.

4. The heat-aware storage engine performance optimization method according to claim 3, characterized in that, The step of recording the correspondence between the Bloom filter and the file into the index queue includes: Set the access frequency of the file to 0 and insert it at the end of the max heap.

5. The heat-aware storage engine performance optimization method according to claim 4, characterized in that, The method further includes: The query request is submitted to the storage engine, which first searches the memory component Memtable of the LSM-tree. If no data is found, the storage engine searches the Bloom filter index queue in the NVM based on the data key to find candidate files. The Bloom filter of each candidate file is accessed sequentially. When the Bloom filter determines that the data may exist in the current file, the CuckooTable file on the disk is accessed. Use the CuckooHash algorithm to search for data in the file and determine if the data exists. If the data exists, the CuckooHash algorithm will be used to read it.

6. The heat-aware storage engine performance optimization method according to claim 5, characterized in that, The method further includes: If the data does not exist, the CuckooTable file on the disk is accessed again.

7. The method according to claim 6, characterized in that, The method further includes: During off-peak periods, the size of the Bloom filter is adjusted based on a file popularity recognition mechanism in memory.

8. The method according to claim 4, characterized in that, The preset threshold is the threshold for the number of files in a single layer of the LSM-tree.

Images