Data processing method and device, electronic equipment and readable medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a readable medium, and relates to the technical field of computers, in the method, for any specified hierarchy of a target database, the access frequency of specific key value pairs in the specified hierarchy is firstly obtained, and the specific key value pairs comprise key value pairs accessed within a preset duration; then, based on the access frequency of the specific key value pair in the specified hierarchy, the access heat of the SST in the specified hierarchy is determined, and the access heat of the SST is in positive correlation with the access frequency of the specific key value pair belonging to the SST; taking the SST of which the access popularity is greater than a popularity threshold value in the specified hierarchy as a target SST, and moving a specific key value pair in each target SST from the specified hierarchy to a target hierarchy; the target hierarchy is above the specified hierarchy. In this way, read amplification can be reduced to a certain extent.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data processing method, device, electronic device, and readable medium. Background Art

[0002] At present, key-value databases based on Log Structured Merge Tree (LSM-Tree) are increasingly widely used. Among them, LSM-Tree is a hierarchical, ordered, disk-oriented data structure, and its core idea is to make full use of the sequential writing of disk batches to ensure data writing performance.

[0003] In the key-value database of LSM-Tree, multiple levels are divided, and the data to be stored in the memory is stored in the form of sorted string tables (SST) in these multiple levels on the disk. Among them, a level can include multiple SSTs, and an SST can include multiple key-value pairs. In the prior art, when searching for data in a key-value database, it is necessary to search layer by layer from top to bottom, which will cause the problem of read amplification. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, device, electronic device and readable medium, which can reduce read amplification.

[0005] In order to solve the above problem, an embodiment of the present invention discloses a data processing method, which includes:

[0006] For any specified level of the target database, obtaining the access frequency of a specific key-value pair in the specified level; the specific key-value pair includes a key-value pair that has been accessed within a preset time period;

[0007] Determining the access popularity of the SST in the specified level based on the access frequency of the specific key-value pair in the specified level; the access popularity of the SST is positively correlated with the access frequency of the specific key-value pair belonging to the SST;

[0008] The SSTs in the designated level whose access heat is greater than a heat threshold are taken as target SSTs, and specific key-value pairs in each of the target SSTs are moved from the designated level to the target level; the target level is located above the designated level.

[0009] On the other hand, an embodiment of the present invention discloses a data processing device, the device comprising:

[0010] A first acquisition module is used to acquire, for any specified level of a target database, the access frequency of a specific key-value pair in the specified level; the specific key-value pair includes a key-value pair that has been accessed within a preset time period;

[0011] A determination module, configured to determine the access popularity of the SST in the specified level based on the access frequency of the specific key-value pair in the specified level; the access popularity of the SST is positively correlated with the access frequency of the specific key-value pair belonging to the SST;

[0012] The first moving module is used to take the SST with access heat greater than the heat threshold in the specified level as the target SST, and move the specific key-value pairs in each target SST from the specified level to the target level; the target level is located above the specified level.

[0013] On the other hand, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the aforementioned method.

[0014] The embodiment of the present invention further discloses a machine-readable medium on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described above.

[0015] The embodiment of the present invention includes the following advantages: In the data processing method provided by the embodiment of the present invention, for any specified level of the target database, the access frequency of the specific key-value pairs in the specified level is first obtained, and the specific key-value pairs include the key-value pairs that have been accessed within a preset time length. Then, based on the access frequency of the specific key-value pairs in the specified level, the access heat of the SST in the specified level is determined, and the access heat of the SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST. The SST in the specified level whose access heat is greater than the heat threshold is used as the target SST, and the specific key-value pairs in each target SST are moved from the specified level to the target level; the target level is located above the specified level. Since the access heat of the SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST, it can be considered that the specific key-value pairs in the target SST whose access heat is greater than the heat threshold are frequently accessed, and the specific key-value pairs in the target SST can represent hot data. Hot data is likely to be frequently accessed in the future. Accordingly, moving specific key-value pairs in the target SST to a higher target level can greatly reduce the number of levels required to query when accessing hot data in the future, thereby reducing read amplification to a certain extent.

[0016] At the same time, in an embodiment of the present invention, in the process of moving upward, only specific key-value pairs in the target SST are moved upward, which can avoid including unnecessary key-value pairs in the data moved upward to a certain extent, thereby avoiding unnecessary data reading and writing overhead, reducing the upward overhead, and avoiding write amplification problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 is a flow chart of steps of a data processing method provided by an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of upward movement provided by an embodiment of the present invention;

[0020] Figure 3 is another upward movement schematic diagram provided by an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a heat tracker provided by an embodiment of the present invention;

[0022] Figure 5 is another schematic diagram of upward movement provided by an embodiment of the present invention;

[0023] Figure 6 This is a flow chart of an upward movement process provided by an embodiment of the present invention;

[0024] Figure 7 is a block diagram of a data processing device provided by an embodiment of the present invention;

[0025] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] First, an application scenario involved in an embodiment of the present invention is described. With the vigorous development of data-intensive applications, key-value storage systems play an increasingly important role as key storage engines. Among them, the key-value storage system based on LSM-tree is widely used in large-scale data management scenarios due to its excellent performance. The data processing method provided in the embodiment of the present invention can be applied to the LSM-tree key-value storage system. The key-value storage system is a data storage and retrieval system. The key-value storage system can store and retrieve associated numerical data through a uniquely identified key, making data access and management more efficient. The key-value storage system can also be called a key-value database. The key-value database is one of the basic components of computer systems and software, and is the most basic underlying storage engine of the data center. Key-value databases are currently widely used in various data-intensive applications, such as search engine applications, distributed data system applications, OLTP / OLAP databases, etc. The LSM structure used by the key-value database is mainly composed of three parts: a memory table (Memory Table, Memtable), an ordered string table (Sorted String Table, SSTable) and a Log. Specifically, the Log can be a write-ahead log (WAL). Memtable is an ordered associative array in memory. Memtable can be used as a cache to store recently written records. When Memtable is full, it will be frozen as an immutable memtable and wait to be serialized into SSTable to be written to the storage device. Among them, the immutable memtable is used to record the data to be written to the storage device. The data in the immutable memtable is no longer modified. SSTable can include data blocks and index blocks. The data block stores key-value pair objects sorted by key.<key.value> , the index block stores the key range of all data blocks. The WAL stores the write records of data to ensure crash consistency.

[0028] The LSM-tree structure makes full use of the sequential write of the disk, maintains the Memtable in the memory, and writes the Memtable in batches to the disk after it is full, converting the random write of the data into sequential write. At the same time, it ensures the orderliness of each layer of data and provides better query performance. However, when querying a key, the structure will search layer by layer from top to bottom. In this way, there is a more serious read amplification problem, which is more serious during range query. Among them, read amplification means that in a reading process, the amount of data that needs to be read is less than the actual amount of data read. To this end, an embodiment of the present invention provides a data processing method, which is described in detail below.

[0029] Reference Figure 1, shows a flow chart of the steps of a data processing method provided by an embodiment of the present invention, which can be applied to electronic devices. Figure 1 As shown, the data processing method specifically includes the following steps:

[0030] Step 101: For any specified level of a target database, obtain the access frequency of a specific key-value pair in the specified level; the specific key-value pair includes a key-value pair that has been accessed within a preset time period.

[0031] Step 102: Determine the access popularity of the SST in the specified level based on the access frequency of the specific key-value pairs in the specified level; the access popularity of the SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST.

[0032] Step 103: Take the SSTs in the designated level whose access heat is greater than the heat threshold as target SSTs, and move specific key-value pairs in each target SST from the designated level to the target level; the target level is located above the designated level.

[0033] Among them, the target database can be a key-value database based on the LSM-tree structure. Accordingly, the target database can include multiple levels: Level0, Level1, ..., Leveln. Among them, the larger the level number, the lower the level is. For example, Leveln can be the lowest level, and Level0 can be the highest level. Each level can include multiple SSTs, and each SST can include key-value pairs. Among them, a key-value pair consists of a key (Key) and a corresponding value (Value). The specific data stored in the key-value pair can be set according to actual needs. For example, the object identifier can be used as the content of the key, and the object-related information corresponding to the object identifier can be used as the content of the value. The embodiment of the present invention does not limit this. Among them, the object can be an image, text, video, application, user account registered in the platform, etc.

[0034] The target level may include levels above the specified level. The specified level may be all levels in the target database or some levels in the target database. For example, the specified level may be the i-th to n-th levels. Where i may be not less than 2, and the specific value of i may be set according to the level to be tracked. For any specified level, the key-value pair that has been recently accessed in the specified level may be used as the specific key-value pair of the specified level. Specifically, if a key-value pair in the specified level has been accessed within a preset time length, the key-value pair may be determined as a specific key-value pair. Wherein, the preset time length may be set according to actual needs. For example, the preset time length may be 6 hours, that is, the key-value pair accessed 6 hours before the current moment may be used as a specific key-value pair. Alternatively, the preset time length may also be 3 days, and so on. The access frequency may represent the number of times a specific key-value pair is accessed within a specific time range, wherein when an operation is performed on the key-value pair, an access to the key-value pair is formed. The operation may include a read operation, a write operation, a modify operation, and the like. Accordingly, the specific time range may refer to the time period from the start of statistics to the execution of the acquisition operation in the above step 101. Specifically, the statistics may be started before the start of step 101.

[0035] Since the key-value pairs in the LSM-tree structure belong to SST, the access popularity of the SST in the specified level can be determined based on the access frequency of the specific key-value pairs. Among them, the access popularity can be used to characterize the access frequency of the SST. The positive correlation between the access popularity of the SST and the access frequency of the specific key-value pairs belonging to the SST means that: the higher the access frequency of the specific key-value pairs belonging to the SST, the higher the access popularity of the SST. Exemplarily, for any SST in the specified level, the specific key-value pairs belonging to the SST can be determined first, and then the average access frequency of the specific key-value pairs belonging to the SST is calculated as the access popularity of the SST. Specifically, the key of each specific key-value pair can be matched with the key of the key-value pair in the SST. If there is a matching key, it is determined that the specific key-value pair belongs to the SST. Alternatively, when the key-value pair is determined to be a specific key-value pair, a source tag can be set for the specific key-value pair, and the source tag is used to characterize the SST to which it belongs. Accordingly, the mean of the access frequency of a specific key-value pair whose source tag is consistent with the tag of the SST can be directly calculated to obtain the access heat of the SST, thereby ensuring the calculation efficiency. It is understandable that when the SST does not include a specific key-value pair, the access heat of the SST can be 0.

[0036] Since the access heat of SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST. Therefore, if the access heat of SST is higher, it means that the SST is accessed more frequently, and the possibility of the SST being frequently accessed in the future will be higher. Further, if the access heat of SST is greater than the heat threshold, it can be determined that the possibility of the SST being frequently accessed in the future reaches the pre-set standard. Therefore, the SST can be used as the target SST, and the specific key-value pairs in the target SST can be moved from the current designated level to the upper target level for storage. Specifically, the specific key-value pairs in the target SST can be written to the target level, and the specific key-value pairs in the target SST in the designated level can be deleted at the same time. Among them, the heat threshold can be pre-set, and for example, the average access frequency to the target database during the high-load period can be set as the heat threshold. Alternatively, the mean of the access frequency of the specific key-value pairs in the designated level can also be determined as the heat threshold. Accordingly, the target SST with access heat greater than the heat threshold can represent frequently accessed hot data. Specifically, since the access popularity of the target SST is determined based on the access frequency of the specific key-value pairs included therein, it can be determined that the hot data in the target SST is specifically the specific key-value pairs included therein, that is, the possibility of the specific key-value pairs in the target SST being frequently accessed in the future reaches a preset standard. Accordingly, when moving upward, only the specific key-value pairs in the target SST can be moved. In this way, it is possible to avoid including unnecessary key-value pairs in the data written to the target level, thereby avoiding the write amplification problem.

[0037] In summary, in the data processing method provided by the embodiment of the present invention, for any specified level of the target database, the access frequency of a specific key-value pair in the specified level is first obtained, and the specific key-value pairs include key-value pairs that have been accessed within a preset time length. Then, based on the access frequency of the specific key-value pairs in the specified level, the access heat of the SST in the specified level is determined, and the access heat of the SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST. The SST in the specified level whose access heat is greater than the heat threshold is taken as the target SST, and the specific key-value pairs in each target SST are moved from the specified level to the target level; the target level is located above the specified level. Since the access heat of the SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST, it can be considered that the specific key-value pairs in the target SST whose access heat is greater than the heat threshold are frequently accessed, and the specific key-value pairs in the target SST can represent hot data. Hot data is likely to be frequently accessed in the future. Accordingly, moving specific key-value pairs in the target SST to a higher target level can greatly reduce the number of levels required to query when accessing hot data in the future, thereby reducing read amplification to a certain extent.

[0038] At the same time, in an embodiment of the present invention, in the process of moving upward, only specific key-value pairs in the target SST are moved upward, which can avoid including unnecessary key-value pairs in the data moved upward to a certain extent, thereby avoiding unnecessary data reading and writing overhead, reducing the upward overhead, and avoiding write amplification problems.

[0039] Optionally, in an embodiment of the present invention, the levels in the target database are distributed in different types of storage media, and the faster the read and write speed of the storage medium, the higher the level distributed in the storage medium. Specifically, a plurality of different types of storage media can be used in the target database to form a heterogeneous storage environment. In an embodiment of the present invention, storage media such as solid state drives (SSDs), mechanical hard disks (HDDs), and memory (RAM) can be used to meet different data access modes and performance requirements. Among them, SSDs can be further divided into different types according to the protocols followed, for example, SSDs that follow the SATA protocol and SSDs that follow the NVMe protocol. The storage media used can be divided into fast media and slow media. Among them, fast media can be storage media with a read and write speed not less than a preset speed threshold, for example, SSDs and RAMs. Slow media can be storage media with a read and write speed less than a preset speed threshold, for example, HDDs. Alternatively, a storage medium with a read and write speed greater than a preset speed threshold can be used as a fast medium, and a storage medium with a read and write speed equal to the preset speed threshold can be used as an intermediate speed medium, and the embodiment of the present invention does not limit this.

[0040] The preset speed threshold can be set according to the actual situation. For example, the average read and write speeds of various types of storage media can be counted in advance, and then the median value of the average read and write speeds can be selected as the preset speed threshold. Alternatively, the preset speed threshold can also be set according to actual application requirements, and the embodiments of the present invention are not limited to this. Furthermore, multiple levels of data can be stored in the same type of storage medium, and the higher the read and write speed, the higher the level stored in the storage medium. For example, Level0-Leveli can be stored in a fast medium, and Leveli-Leveln can be stored in a slow medium.

[0041] At present, traditional storage systems usually use a single type of storage medium. However, with the increasing amount of data processed by network services and applications, and the different requirements of applications for storage performance and cost, single-media storage systems can no longer meet the diverse application scenarios well. Specifically, different applications have different requirements for the performance of the storage system. Some applications are more sensitive to read speed, while some applications have higher requirements for write latency. Different storage media also have different characteristics, with different read and write speeds, and different processing capabilities for random read and write and sequential read and write. For example, high-performance storage media, such as solid-state drives (SSDs), are usually more expensive and have smaller capacities. High-capacity and low-cost media, such as mechanical hard disks (HDDs), cannot provide sufficient read and write performance. In an embodiment of the present invention, by setting up a variety of storage media and integrating different storage media using a hybrid storage solution, different types of storage media can be more fully utilized, thereby meeting different requirements and achieving a better balance between performance and cost for the target database. At the same time, by adopting a hybrid storage method, the deployment cost of the target database can be reduced.

[0042] Because in the LSM-tree structure, as data is continuously written, a merge operation will be caused to move data downward. Correspondingly, in a hybrid storage environment, data will be migrated from fast media to slow media. In this way, when hot data is located in the lower layer, reading hot data in the lower layer from the slow medium will result in a large overhead. And because the access speed and delay of the fast media and slow media equipped in the target database are quite different, the read amplification caused by the query operation in the slow medium will become the performance bottleneck of the system. In the embodiment of the present invention, the locality of data access is utilized to first identify the hot data in the target database according to the access frequency, and the hot data is moved to the upper layer, so that the hot and cold data are separated, and the high-level is stored in a faster storage medium. Therefore, to a certain extent, the hot data in the lower layer can be promoted to the fast medium in the upper layer, so that the access to the hot data is performed as much as possible from the fast medium, thereby improving the data access efficiency, reducing the subsequent reading overhead and improving the performance of the target database, and effectively utilizing both fast and slow media.

[0043] Furthermore, in an embodiment of the present invention, when the reading and writing speed of the storage medium in the designated level is less than a preset speed threshold, the specific key-value pairs in each of the target SSTs may be moved from the designated level to the target level. That is, in an embodiment of the present invention, when the data in the designated level is stored in a slow medium, the specific key-value pairs in the target SST may be moved to the target level. In this way, since only the specific key values ​​in the SST whose heat is higher than the heat threshold are moved up, it is possible to avoid the problem of cold data (i.e., other key-value pairs except the specific key-value pairs) in the data moved up, causing unnecessary data reading and writing overhead, thereby reducing the processing pressure on the slow medium during movement, reducing the cost of movement, and avoiding the problem of excessive processing pressure in the slow medium affecting the performance of the target database.

[0044] In the embodiment of the present invention, the following can also be performed: Step S21, when the read and write speed of the storage medium in the designated level is not less than the preset speed threshold, directly move the target SST from the designated level to the target level. That is, when the data in the designated level is stored in a fast medium, directly move the target SST to the target level in units of SST. In this way, there is no need to extract specific key-value pairs in the target SST. By directly moving up, the efficiency of moving up can be improved while ensuring that hot data is moved up. Figure 2 is a schematic diagram of upward movement provided by an embodiment of the present invention, such as Figure 2 As shown, SST1 in the slow medium can be moved from Leveln to Level1 in the fast medium. Figure 3 is another upward movement schematic diagram provided by an embodiment of the present invention, such as Figure 3 As shown, SST1 can be moved directly from Leveln to Level0.

[0045] At the same time, since the upper space is limited, in the embodiment of the present invention, by moving the hot data upward, the cold data can be gradually pushed down to the lower layer as the hot data moves upward, thereby alleviating the problem of cold data occupying the high-level space to a certain extent.

[0046] Optionally, the embodiment of the present invention may further include the following steps:

[0047] Step S31, storing the key name of the specific key-value pair in the corresponding level for each sub-tracker of the heat tracker, and setting an access flag for each key name; the heat tracker includes sub-trackers corresponding to each level in the target database.

[0048] Step S32: when the key-value pair to which the key name belongs is accessed, increase the value of the access flag.

[0049] Correspondingly, the above-mentioned step of obtaining the access frequency of specific key-value pairs in the specified level may specifically include: step 1011, extracting the numerical value of each access flag bit from the sub-tracker corresponding to the specified level as the access frequency of each specific key-value pair in the specified level.

[0050] Among them, the heat tracker can be a pre-deployed application or script. The organizational form of the heat tracker can be consistent with the structure of the target database. Specifically, the heat tracker can be divided into multiple levels, and one level is a sub-tracker. The number of sub-trackers is consistent with the number of levels in the target database, and one sub-tracker corresponds to one level in the target database. For the n levels included in the target database, the heat tracker can include n sub-trackers. In this way, only one heat tracker is needed to perform statistics for multiple levels. Alternatively, in the specific implementation, an independent tracker can also be set for each layer, that is, n trackers are set, and the embodiment of the present invention is not limited to this.

[0051] For any sub-tracker, the sub-tracker can be used to count the access frequency of the key-value pairs that have been recently accessed in the corresponding level. Accordingly, the sub-tracker can store the key names of the key-value pairs that have been recently accessed in the corresponding level, that is, store the key names of specific key-value pairs in the corresponding level. Since in actual scenarios, the access frequency to high levels is often higher than that to low levels, and as time goes by, the key-value pairs that have been recently accessed will be continuously updated, the stored key names of each sub-tracker can be dynamically updated in an embodiment of the present invention. Accordingly, the access frequency of the key-value pairs corresponding to the key names stored in the corresponding high-level sub-tracker is often higher than the access frequency of the key-value pairs corresponding to the key names stored in the low-level sub-trackers.

[0052] Furthermore, the access flag bit can be a clock bit in the clock algorithm. For the key name stored in any sub-tracker, hash mapping can be performed concurrently with the key name as the index of the hash map, and then the key-value objects represented by each key name are mapped to the corresponding Clock bit. Among them, the key-value object refers to a key-value pair, and a key name in the sub-tracker corresponds to a Clock bit. A Clock bit may include multiple data bits, for example, it may include two data bits, and a data bit may include two values ​​0 and 1. Accordingly, a Clock bit can represent a value of 0-3. Figure 4 is a schematic diagram of a heat tracker provided by an embodiment of the present invention, such as Figure 4As shown, the heat tracker includes multiple sub-trackers. From top to bottom, the heat of the key names corresponding to these multiple sub-trackers is getting lower and lower. For example, key1~key5 represent the five key names included in sub-tracker 3, and Clock[n] represents the clock bit corresponding to key1 after hash mapping (i.e., Hash function in the figure).

[0053] Specifically, when the key name is first inserted into the sub-tracker, the value of the access flag bit can be set to 0, and with subsequent access operations, the value of the access flag bit can be increased accordingly. Specifically, when the key-value pair to which the key name belongs is accessed once, the value of the access flag bit can be increased by 1. Alternatively, the access flag bit can be maintained based on a preset Clock algorithm to count the access frequency and perform heat tracking on the key-value pair. Since the Clock algorithm is simple and easy to implement, the calculation and storage overhead are low. Therefore, in the embodiment of the present invention, the access frequency of a specific key-value pair is counted based on the Clock algorithm, which can reduce the calculation and storage overhead of hot and cold data perception. Specifically, within a time window, if the key-value pair to which the key name belongs has been accessed, at the end of the time window, the lowest bit in the access flag bit is set to 1 and all bits in the access flag bit are cyclically shifted to the left by one bit. Conversely, if the key-value pair to which the key name belongs has not been accessed, at the end of the time window, all bits in the access flag bit are cyclically shifted to the left by one bit and the lowest bit is cleared, that is, the access flag bit is decremented according to the Clock algorithm. Similarly, the access conditions of the key-value pairs to which the key name belongs in multiple time windows are counted. Furthermore, when the value of the access flag is 3, it can be determined that the access frequency of the key name has reached the maximum heat. Of course, more data bits can also be set for the data bit to expand the maximum heat that the access flag can represent.

[0054] Furthermore, when it is necessary to obtain the access frequency of a specific key-value pair in a specified level, the numerical value of each access flag bit can be directly extracted from the sub-tracker corresponding to the specified level, and the specific value represented by the numerical value is the access frequency of the specific key-value pair. Exemplarily, assuming that the specified level includes 3 specific key-value pairs, then the numerical values ​​of the access flag bits corresponding to the key names of the 3 specific key-value pairs can be extracted to obtain the access frequency of the 3 specific key-value pairs. It should be noted that in an embodiment of the present invention, the operation of obtaining the access frequency of a specific key-value pair in a specified level can be periodically performed according to a preset period, that is, hot data is periodically identified and moved upward. Alternatively, the operation of obtaining the access frequency of a specific key-value pair in a specified level can be performed when an acquisition instruction sent by a user is received, that is, hot data is identified and moved upward according to user needs.

[0055] In the embodiment of the present invention, a heat tracker is set to track the access frequency of specific key-value pairs in each level based on the heat tracker. Accordingly, when it is necessary to know the access frequency, the value of each access flag bit in the corresponding sub-tracker is directly read to determine the access frequency of each specific key-value pair in the level. In this way, the efficiency of determining the access frequency can be ensured to a certain extent.

[0056] Optionally, the embodiment of the present invention may further include the following steps:

[0057] Step S41: For any of the levels, in response to an access operation to a key-value pair in the level, if the access key name is not stored in the sub-tracker corresponding to the level, the access key name is stored in the sub-tracker corresponding to the level, and an access flag is set for the access key name; the access key name is the key name of the key-value pair accessed by the access operation.

[0058] Step S42: For any of the sub-trackers, when the key names stored in the sub-tracker reach a preset upper limit, the key names whose access flag values ​​in the sub-tracker are 0 are removed.

[0059] The execution order of the above-mentioned step S41 and step S42 may not be unique. For example, the two may be executed simultaneously, or step S42 may be executed first and then step S41. The embodiment of the present invention does not limit this.

[0060] Specifically, the access operation may include modification operations, write operations, read operations, and the like on the key-value pairs. The access key name may be the key name of the key-value pair accessed by the access operation. If the access key name is not stored in the sub-tracker corresponding to the level, it means that a new specific key-value pair has appeared. Therefore, the access key name may be inserted into the sub-tracker corresponding to the level, and an access flag may be set for the access key name. That is, in an embodiment of the present invention, each access operation of the upper-layer load to the target database may trigger an update operation on the corresponding Clock bit or an operation to insert a key name into the sub-tracker. As the upper-layer load changes dynamically, the more key-value pairs accessed by the upper-layer load are more likely to be identified as hot data, that is, data with high demand may be promoted to a higher level as hot data according to the needs of the upper-layer load.

[0061] Specifically, hash mapping can be performed with the key name as the index of the hash mapping, and the key-value object represented by the key name can be mapped to the corresponding Clock bit. In this way, it can be ensured that the key names stored in the sub-tracker can accurately cover the key-value pairs that have been accessed recently, thereby ensuring the comprehensiveness of the access frequency statistics. Furthermore, if the value of the access flag bit is 0, that is, the clock bit is 0, it can be determined that the key-value pair represented by the key name corresponding to the access flag bit has not been accessed for a period of time. Therefore, it can be considered that the key-value pair represented by the key name is not hot data. Accordingly, when the key names stored in the sub-tracker reach the preset upper limit value, that is, when the capacity upper limit is reached, the key names with the value of the access flag bit of 0 can be removed, that is, the objects with the clock bit of 0 are expelled. In this way, storage space can be saved, and at the same time, statistics on unnecessary key-value pairs can be avoided, thereby saving processing resources.

[0062] Specifically, the key names stored in the sub-tracker can be stored in the target database. Accordingly, in an embodiment of the present invention, the sub-tracker can save storage resources and reduce processing overhead by storing only the key names and clock bits of the key-value pairs that have been recently accessed in the database. It should be noted that the data volume of a typical key-value pair is often less than 1KB, while there are usually millions of key-value pairs in the target database. Therefore, in an embodiment of the present invention, the amount of metadata of the heat tracker can be limited to reduce the calculation and storage overhead of subsequent processing. Specifically, the storage size corresponding to the heat tracker can be set according to a preset ratio of the total number of key names in the target database. Among them, the preset ratio can be 10%-20%. The storage size corresponding to the heat tracker can be the total data size of the key names in the target database multiplied by the preset ratio. Among them, the storage size is the metadata amount, and the storage size can represent the size of the storage space available for the key name and its corresponding clock bit. In this way, the problem of low storage efficiency caused by too many tracked key names can be avoided to a certain extent, and at the same time, the accuracy of the heat tracker can be ensured to a certain extent.

[0063] In the embodiment of the present invention, the key-value pair is used as the tracking granularity, the access frequency of the recently accessed key-value pairs is counted, and the hot data in the target database is identified in a fine-grained manner, providing clear guidance for the subsequent upward movement operation. In this way, it is possible to avoid the problem of too large a granularity of statistics leading to too large a granularity of upward movement, which in turn leads to write amplification and excessive overhead.

[0064] Optionally, the embodiment of the present invention may further include the following steps:

[0065] Step S51, for any of the target SSTs, select the target level for the target SST from the other levels according to the average access popularity of other levels in the target database and the access popularity of the target SST; the other levels include levels higher than the specified level.

[0066] Among them, the average access popularity of a level can be the average of the access frequencies of specific key-value pairs in the level. In an embodiment of the present invention, by selecting a target level for the target SST from a level higher than the specified level, the selection range can be reduced, thereby improving the selection efficiency. Of course, in specific implementation, it is also possible to select from levels other than the specified level. If the selected target level is not higher than the specified level, it may not be moved, or it may be moved downward. This is not limited in the embodiment of the present invention.

[0067] Specifically, the target level can be determined by comparing the access frequency of the target SST with the average access popularity of each other level. Optionally, step S51 can specifically include:

[0068] Step S51a: For any of the other levels, calculate the numerical mean of the access flag bits in the sub-trackers corresponding to the other levels as the average access heat of the other levels.

[0069] Step S51b: Calculate the absolute value of the difference between the access heat of the target SST and the average access heat of the other levels.

[0070] Step S51c: determine the other level corresponding to the minimum absolute value as the target level.

[0071] Among them, the values ​​of all access flag bits can be extracted from the sub-tracker corresponding to the other level, and the average of the extracted values ​​can be calculated to obtain the average access heat of the other level. Then, the average access heat of the other level can be subtracted from the access heat of the target SST, and the absolute value of the difference can be determined. Accordingly, there are several other levels, and the corresponding number of absolute values ​​can be obtained. Finally, the other level corresponding to the absolute value with the smallest value among these absolute values ​​can be determined as the target level of the target SST. In this way, by selecting the other level corresponding to the minimum absolute value as the target level, it can be ensured that the access heat of the target SST is closest to the average access heat of the selected target level, thereby ensuring the adaptability of the target level and the target SST, and avoiding the problem of selecting too high a target level. At the same time, by selecting the target level for each target SST in a targeted manner, it can be avoided to a certain extent that all target SSTs are fixed to the same level, resulting in the problem of frequent triggering of the merge operation.

[0072] Furthermore, the step of moving the specific key-value pairs in each of the target SSTs from the specified level to the target level may specifically include:

[0073] Step 1031: Divide the target SSTs corresponding to the same target level into an SST group, and for any of the SST groups, extract specific key-value pairs included in each target SST in the SST group as target key-value pairs.

[0074] Step 1032: Encapsulate the target key value to obtain a combined SST.

[0075] Step 1033: Move the combined SST from the specified level to the target level.

[0076] Among them, the access popularity of different target SSTs in the same specified level may be different, and accordingly, the target levels selected for the target SST may be different. Therefore, in an embodiment of the present invention, the target SSTs corresponding to the same target level can be first divided into an SST group and processed in groups. The target level corresponding to the target SST is the target level selected for the target SST. Of course, when the target levels selected for each target SST in the specified level are the same, an SST group can be obtained.

[0077] For any SST group, the specific key-value pairs included in all target SSTs in the group can be used as the target key-value pairs corresponding to the group. Specifically, each key name in the sub-tracker corresponding to the specified level can be matched with the key name in the target SST in the group. If the match is successful, the key-value pair of the matched key name is determined as the target key-value pair corresponding to the group.

[0078] Then, M target key-value pairs can be extracted each time in units of M, and the extracted M target key-value pairs can be encapsulated into an SST until all target key-value pairs are encapsulated. Specifically, when extracting, the key names can be extracted in order of size, so that the key-value pairs stored in the SST finally encapsulated can be stored in order as much as possible, so as to facilitate subsequent queries. Among them, the specific value of M can be set according to actual needs, and M can control the maximum number of key-value pairs that can be stored in a combined SST. Specifically, M target key-value pairs can be extracted first, and then the extracted M target key-value pairs can be encapsulated into an SST. Then, when the number of remaining target key-value pairs is not greater than M, the remaining target key-value pairs are encapsulated into an SST. When encapsulating, the M target key-value pairs can be packaged into data blocks. Then, an index block is established for the data block, and the index block can include the key range composed of the key-value pairs in each data block. Finally, the data block and the index block are combined according to the organizational format of the SST to obtain a combined SST. For example, a combined SST is obtained by combining the data block, Bloom Filter, index block, and Footer in the format. Among them, Bloom Filter can be used to determine whether a key exists in the SST, thereby avoiding unnecessary disk read operations. Bloom Filter maps each data block to a bit array through multiple hash functions and sets the corresponding bits. The existence can be determined based on the value of the bit later. Footer refers to the tail of the SSTable file, which contains the metadata of the SSTable to help quickly locate data.

[0079] Furthermore, when the number of remaining target key-value pairs is greater than M, M target key-value pairs are extracted again, and then the extracted M target key-value pairs are encapsulated into one SST until all target key-value pairs are encapsulated. Alternatively, the target key-value pairs in N target SSTs may be encapsulated into one SST until all target key-value pairs are encapsulated, and this is not limited in the embodiment of the present invention. The specific value of N can be set according to actual needs, and N represents the maximum number of target SSTs covered in a combined SST.

[0080] Among them, the encapsulated combined SST can be recorded as a set of virtual HotSST. When moving up, a set of virtual HotSSTs can be maintained for each specified level, that is, for the target SST in each specified level, a set of virtual HotSSTs corresponding to the specified level can be encapsulated. Among them, the specific key-value pairs included in all target SSTs in the specified level can be used as the identified fine-grained heat data and stored in the virtual HotSSTTable corresponding to the specified level. That is, the virtual HotSSTTable corresponding to the specified level can be composed of the specific key-value pairs included in all target SSTs in the specified level. Accordingly, the encapsulated combined SST is moved from the specified level to the target level for storage, so that the moved-up SST only includes hot data. In this way, it is possible to avoid including unnecessary key-value pairs in the moved-up data, which leads to the problem of write amplification, thereby avoiding unnecessary movement overhead and reducing the moving-up overhead and cost. Accordingly, when the designated level belongs to a slow medium and the target level belongs to a fast medium, the encapsulated SST is moved up, which can reduce the mobile overhead and cost of cross-media migration. It should be noted that in the embodiment of the present invention, moving the target SST or the combined SST up to the target level can specifically be merging the target SST or the combined SST into the target level. Figure 5 is another schematic diagram of upward movement provided by an embodiment of the present invention, such as Figure 5 As shown, a set of HotSSTs maintained for Leveln can be floated up to the target level Leveln-1.

[0081] In the embodiment of the present invention, the target database can adopt a flexible and changeable LSM-tree structure. Among them, the traditional LSM-tree structure will set a rule that only allows data to move downward, that is, merge downward. In the embodiment of the present invention, the target level corresponding to the target SST is directly judged, and the hot data is directly promoted to the target level. That is, the floating mechanism adopted in the embodiment of the present invention relaxes the restrictions on merging, that is, no restriction rules are set. Hot data is allowed to float from the lower layer to the upper layer, and is no longer limited to two consecutive layers. When the target SST moves upward, it can be moved up across multiple layers in the target database. In this way, the query path can be shortened more effectively, the read amplification effect can be reduced, and the overall performance of the system can be effectively improved. And by identifying hot data and moving the hot data up, it is possible to separate hot and cold data along the hierarchical dimension, and vertically layer the hot and cold data, which can facilitate access to hot data.

[0082] In the embodiment of the present invention, the up-shift operation can be performed by creating a migration thread, so that the combined SST only moves up specific key-value pairs in the target SST, which can avoid creating a large number of migration threads, thereby reducing the up-shift cost and avoiding the problem of too much up-shift affecting the write performance. And compared with multiple upgrades layer by layer, in the embodiment of the present invention, when the access heat is greater than the preset access heat threshold, the operation of directly moving up to the target level will be triggered, so the difficulty of implementation can be reduced to a certain extent and the efficiency of implementation can be improved.

[0083] Furthermore, in an embodiment of the present invention, the access heat of the SST in a specified level can be determined based on the access frequency obtained by the aforementioned heat tracker. For a specified level, if the access heat of the SST in the specified level exceeds the heat threshold, then the SST can be marked as requiring an upward operation, that is, the SST can be used as a target SST to move upward in subsequent operations. Further, in an embodiment of the present invention, before extracting the operation of a specific key-value pair included in each target SST in the SST group, the upward cost can be determined first. Specifically, the embodiment of the present invention can also include the following steps:

[0084] Step S61: For any of the target SSTs, a first parameter value is generated based on the movement cost of the target SST, and a second parameter value is generated based on the movement benefit of the target SST.

[0085] Step S62: Based on the first parameter, the second parameter value and a preset cost function, obtain a corresponding movement cost value of the target SST.

[0086] Step S63: When the movement cost value indicates that the movement benefit of the target SST is not greater than the movement cost, the target SST is removed from the SST group.

[0087] Among them, the preset cost function can be pre-constructed. The preset cost function can be used to characterize the difference between the benefit that can be brought by moving the target SST to the target level and the cost selected to move to the target level, that is, the preset cost function can be defined as the difference between the second parameter value and the first parameter value. The moving cost may include the amount of data required to move the target SST (i.e., input / output (I / O) cost), the time required to consume (for example, the time temporarily unavailable due to data movement) and the amount of CPU resources required to move up (for example, CPU time, wherein CPU time may refer to the time required for the CPU to perform the move operation).

[0088] Correspondingly, the first parameter value can be: W = a1*I / O cost + a2*time required to consume + a3*amount of CPU resources required to consume. Wherein, a1, a2 and a3 represent pre-set weights. The I / O cost can be calculated based on the size of the target SST and the current I / O speed. For example, the ratio of the size of the target SST to the I / O speed is used as the I / O cost. The I / O cost can represent the IO duration required for the target SST. Furthermore, the CPU time consumed, the data unavailable time and the number of levels moved each time the SST moves can be recorded in advance. The number of levels moved indicates how many levels the SST has been moved, and then the ratio of the sum of the recorded number of levels to the sum of the recorded CPU time is calculated to obtain the CPU time required to move one level. And, the ratio of the sum of the recorded number of levels to the sum of the recorded data unavailable time is calculated to obtain the data unavailable time required to move one level. Accordingly, the number of levels between the specified level and the target level can be determined first, for example, the difference between the number of the specified level and the number of the target level can be used as the number of levels. Then, the product of the number of levels, the CPU time required to move one level, and the data unavailable time required to move one level are calculated respectively to obtain the time consumed by this upward movement and the amount of CPU resources consumed by this upward movement.

[0089] Furthermore, the access time required after each SST movement and the access time required before the movement can be recorded in advance, and then the time difference between the access time required before the movement and the access time required after the movement can be calculated. Also, the number of levels moved can be recorded. Finally, the ratio of the sum of the recorded number of levels to the sum of the recorded time differences is calculated to obtain the access time that may be reduced by moving one level. Accordingly, the difference between the number of the specified level and the number of the target level can be used as the number of levels. Then, the product of the number of levels and the access time that may be reduced by moving one level is calculated respectively to obtain the access time T that may be reduced by moving one level this time. The access time T can represent the cost of movement, and the access time T can be directly determined as the second parameter value.

[0090] The moving cost value can be the output value of the preset cost function. If the output value is not a positive value, it can be determined that the moving benefit of the target SST is not greater than the moving cost, which means that the read and write costs caused by the upward operation are not less than the benefits of the upward operation. Accordingly, the target SST can be removed from the SST group, that is, the target SST is not moved upward. If the output value is a positive value, it can be determined that the moving benefit of the target SST is greater than the moving cost. Accordingly, the target SST can be retained, that is, the target SST is moved upward. In this way, the problem of increased costs caused by the upward operation can be avoided, ensuring that the upward operation can bring moving benefits to the target database.

[0091] It should be noted that, in the embodiment of the present invention, when designing the cost function, the input can also be set to include the average access heat of other levels and the access heat of the target SST. Accordingly, when the cost function is processed, it can also select a target level for the target SST from the other levels according to the average access heat of other levels and the access heat of the target SST, and output the identifier of the target level selected for the target SST, that is, the above step S51 can be used as a function of the cost function and implemented by the cost function. Further, for the target SST retained in the SST group, the target key-value pairs in the combined SST that overlap with the data in the target level can be deleted before step 1033. Among them, there is data overlap: there is a key-value pair with the same key name as the target key-value pair in the target level. In this way, it can be ensured that there is no overlapping data in the SST in each level, and since the upper-level data is often updated, the old data can be deleted by deleting the target key-value pairs with the same key name in the combined SST, ensuring that the updated data can continue to be retained in the upper-level data. Figure 6 is a flow chart of an upward movement process provided by an embodiment of the present invention, such as Figure 6 As shown, you can first traverse the SST and determine the target SST. Determine the target level corresponding to the target SST. Determine whether the moving benefit of the target SST is not greater than the moving cost. If so, end the process directly, that is, do not move up. On the contrary, if not, delete the overlapping data and move up.

[0092] Optionally, the embodiment of the present invention may further include the following steps:

[0093] Step S71: For any type of storage medium, monitor the performance index corresponding to the type of storage medium.

[0094] Step S72: When the performance indicator meets the preset indicator condition, adjust the target parameter corresponding to the type of storage medium; wherein the performance indicators and target parameters corresponding to different types of storage media are different.

[0095] Specifically, the performance indicators and target parameters corresponding to different types of storage media are pre-set based on the performance characteristics of the storage media of this type. In the embodiment of the present invention, the performance indicators and target parameters corresponding to different types of storage media can be pre-determined based on empirical data. Specifically, the performance characteristics of various storage media can be analyzed in advance, and the performance characteristics can include indicators in many aspects such as read and write speed, capacity, and life. Exemplarily, SSD has the following characteristics: read and write speed is usually much higher than HDD, sequential read and write speed is fast, suitable for large file transfer and database operation, random read and write performance is relatively good, capacity is smaller than HDD capacity, and cost is higher than HDD, and flash memory unit has a limit on the number of erases and writes. HDD has the following characteristics: read and write speed is much lower than SSD, sequential read and write performance is good, but random read and write performance is poor, usually provides large storage capacity and low cost, and mechanical parts may wear out as the use time increases, thereby affecting the service life. RAM has the following characteristics: very fast read and write speed, usually smaller capacity than HDD and SSD, usually long life, but may fail due to long-term operation, overheating or power supply problems.

[0096] Based on the quantitative analysis of the performance characteristics, the performance indicators that need to be monitored and the parameters that need to be adjusted for various storage media are determined. In this way, the target database can be accurately adjusted in terms of parameter adjustment, so that the adjusted parameters are more adapted to the performance characteristics of various storage media. Exemplarily, a parameter adjuster can be pre-set, and the parameter adjuster can be an application or a script. Analysis is performed based on the parameter adjuster, and the above-mentioned monitoring and adjustment operations are performed. Specifically, the parameter adjuster can continuously monitor the performance indicators corresponding to various storage devices. Then, it is determined whether the real-time performance indicators monitored meet the preset indicator conditions. If they meet, the target parameters corresponding to the storage medium of this type can be adjusted. Otherwise, no adjustment is required. In this way, the key-value storage system can adjust the parameters in a timely manner, thereby ensuring to a certain extent that the key-value storage system can maintain efficient operation under different loads.

[0097] Exemplarily, for an SSD that complies with the SATA protocol, it also has performance characteristics: it performs well in the face of large-scale single-threaded write workloads, but there are serious IO blocking problems in multi-threaded merge operations. Accordingly, the performance indicators corresponding to the SSD that complies with the SATA protocol may include: I / O waiting time Tw, write delay Wl, and queue depth Qd. Among them, for Tw, if the waiting time of the I / O request is too long, it may indicate that there is a concurrent access conflict or the device processing capacity has reached its limit. For Wl, a long write delay may indicate that the controller or cache is saturated. The depth of Qd can indicate the current load level, and accordingly, monitoring Qd helps to determine the performance of the storage medium under high load. For example, if Qd is greater than a preset depth threshold, it can be determined that the storage medium is currently in a high load state. The target parameters corresponding to the SSD that complies with the SATA protocol may include: write merge strategy and cache strategy. Among them, the write merge strategy includes: using a single queue continuous write method, reducing the concurrency of write operations, and using larger write blocks to reduce disk addressing and management overhead. The cache strategy includes: optimizing cache utilization (e.g., increasing cache size) to buffer write operations and reduce pressure during peak hours. Specifically, the write merge strategy can be enabled when Tw is continuously higher than a preset waiting time threshold for a specified period of time. The cache strategy is enabled when Wl is higher than a preset delay threshold.

[0098] Exemplarily, for an SSD that complies with the NVMe protocol, it also has performance characteristics: it has a high data transmission bandwidth and is suitable for scenarios that process large amounts of data. When the batch size of the processing increases, the performance of the NVMe SSD may decline. Due to the large bandwidth, a write speed that is too fast may cause data to be deposited too quickly, thereby reducing performance. Accordingly, the performance indicators corresponding to the SSD that complies with the NVMe protocol may include: data throughput T, command latency Cl, and write amplification Wa. Among them, T is used to characterize the data processing capability of the storage medium to ensure that the bandwidth limit is not reached. Cl represents the response time of command processing, which can reflect the speed of processing requests. Wa is used to measure the ratio of the amount of written data to the amount of data actually written to the storage medium. High write amplification will lead to performance degradation and shortened life. The target parameters corresponding to the SSD that complies with the NVMe protocol may include: concurrency control, batch size, and write speed. Among them, concurrency control may include concurrency level, and by adjusting the concurrency level, the processing capability of the storage medium can be matched. Performance can be optimized and write amplification can be reduced by adjusting the batch size. Excessively fast data deposition can be avoided by controlling the write speed. Specifically, when the difference between the preset upper limit value and T is less than the preset difference threshold, that is, when the data throughput is close to the device limit, the batch size, write speed or concurrency level can be reduced to avoid performance bottlenecks. When Cl is greater than the preset delay threshold, the number of concurrent commands is reduced. Alternatively, the command queue management can be optimized (for example, the command queue depth is reduced), that is, the target parameter can also include the command queue depth. When Wa is greater than the preset write amplification threshold, the batch size is increased. Alternatively, a sequential write method is adopted, that is, the target parameter can also include a write method.

[0099] For HDD, performance indicators may include: disk temperature, disk response time. Corresponding target parameters may include: hard disk sleep time, disk cache settings. Specifically, when the disk temperature continues to be higher than the preset temperature threshold for a specified period of time, the hard disk sleep time is increased. When the disk response time is greater than the preset time threshold, the cache space in the disk cache setting is increased. For RAM, performance indicators may include: page swap rate, clock speed, and access delay. Corresponding target parameters may include: swap area size and memory timing. Specifically, when the page swap rate is greater than the preset swap rate, it can be determined that page swaps are currently occurring frequently. Accordingly, the swap area size can be increased. When the clock speed is less than the preset speed threshold and the access delay is less than the preset delay threshold, the memory timing is reduced.

[0100] Compared with adjusting the same target parameters for all types of storage media, which results in the inability to dynamically adjust, and the inability to make specific optimizations to the parameters according to the characteristics of each storage medium, the adaptability is poor, and thus the performance is poor. In the embodiment of the present invention, by predetermining the performance indicators and adjustment parameters required to be monitored for various storage media, and monitoring the performance indicators required to be monitored for various storage media in real time, and making targeted parameter adjustments, it is possible to achieve targeted dynamic specific adjustment of the parameters, thereby improving the adaptability of the target database, so that the performance characteristics of different devices can be better adapted in a hybrid storage environment, thereby ensuring the performance of the target database.

[0101] Reference Figure 7 , shows a block diagram of a data processing device provided by an embodiment of the present invention, such as Figure 7 As shown, the data processing device specifically includes:

[0102] The first acquisition module 201 is used to acquire the access frequency of a specific key-value pair in any specified level of the target database; the specific key-value pair includes a key-value pair that has been accessed within a preset time period;

[0103] A determination module 202, configured to determine the access popularity of the SST in the specified level based on the access frequency of the specific key-value pair in the specified level; the access popularity of the SST is positively correlated with the access frequency of the specific key-value pair belonging to the SST;

[0104] The first moving module 203 is used to take the SST in the specified level whose access heat is greater than the heat threshold as the target SST, and move the specific key-value pairs in each target SST from the specified level to the target level; the target level is located above the specified level.

[0105] Optionally, the levels in the target database are distributed in different types of storage media, and the faster the reading and writing speed of the storage medium is, the higher the level distributed in the storage medium is;

[0106] The first moving module 203 is specifically configured to: when the reading and writing speed of the storage medium at the designated level is less than a preset speed threshold, move the specific key-value pairs in each target SST from the designated level to the target level;

[0107] The device also includes: a second moving module, which is used to directly move the target SST from the designated level to the target level when the reading and writing speed of the storage medium at the designated level is not less than the preset speed threshold.

[0108] Optionally, the device further comprises:

[0109] A selection module, for selecting, for any of the target SSTs, the target level for the target SST from the other levels according to the average access heat of the other levels in the target database and the access heat of the target SST; the other levels include levels higher than the specified level;

[0110] The first moving module 203 is further specifically used for:

[0111] Divide the target SSTs corresponding to the same target level into an SST group, and for any of the SST groups, extract the specific key-value pairs included in each target SST in the SST group as the target key-value pairs;

[0112] Encapsulating the target key value to obtain a combined SST;

[0113] The combined SST is moved from the specified level to the target level.

[0114] Optionally, the device further comprises:

[0115] A generating module, for generating, for any of the target SSTs, a first parameter value based on a moving cost of the target SST, and a second parameter value based on a moving benefit of the target SST;

[0116] A second acquisition module, configured to acquire a corresponding movement cost value of the target SST based on the first parameter, the second parameter value and a preset cost function;

[0117] The elimination module is used to eliminate the target SST from the SST group when the movement cost value indicates that the movement benefit of the target SST is not greater than the movement cost.

[0118] Optionally, the device further comprises:

[0119] A first setting module is used to store the key name of the specific key-value pair in the corresponding level for each sub-tracker of the heat tracker, and set an access flag for each key name; the heat tracker includes sub-trackers corresponding to each level in the target database;

[0120] An adding module, used for increasing the value of the access flag bit when the key-value pair to which the key name belongs is accessed;

[0121] The first acquisition module is specifically used to extract the value of each access flag bit from the sub-tracker corresponding to the specified level as the access frequency of each specific key-value pair in the specified level.

[0122] Optionally, the device further comprises:

[0123] a second setting module, for any of the levels, in response to an access operation to a key-value pair in the level, if the access key name is not stored in the sub-tracker corresponding to the level, storing the access key name in the sub-tracker corresponding to the level, and setting an access flag for the access key name; the access key name is the key name of the key-value pair accessed by the access operation;

[0124] The removal module is used for removing the key names whose access flag values ​​in any of the sub-trackers are 0 when the key names stored in the sub-tracker reach a preset upper limit.

[0125] Optionally, the selection module 202 is specifically configured to:

[0126] For any of the other levels, calculate the numerical average of the access flag bits in the sub-trackers corresponding to the other levels as the average access heat of the other levels;

[0127] Calculate the absolute value of the difference between the access heat of the target SST and the average access heat of the other levels;

[0128] The other level corresponding to the minimum absolute value is determined as the target level.

[0129] Optionally, the device further comprises:

[0130] A monitoring module is used to monitor the performance indicators corresponding to any type of storage medium; an adjustment module is used to adjust the target parameters corresponding to the type of storage medium when the performance indicators meet the preset indicator conditions; wherein the performance indicators and target parameters corresponding to different types of storage media are different.

[0131] In summary, the data processing device provided by the embodiment of the present invention, for any specified level of the target database, first obtains the access frequency of a specific key-value pair in the specified level, and the specific key-value pair includes the key-value pairs that have been accessed within a preset time length. Then, based on the access frequency of the specific key-value pairs in the specified level, the access heat of the SST in the specified level is determined, and the access heat of the SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST. The SST in the specified level whose access heat is greater than the heat threshold is taken as the target SST, and the specific key-value pairs in each target SST are moved from the specified level to the target level; the target level is located above the specified level. Since the access heat of the SST is positively correlated with the access frequency of the specific key-value pairs belonging to the SST, it can be considered that the specific key-value pairs in the target SST whose access heat is greater than the heat threshold are frequently accessed, and the specific key-value pairs in the target SST can represent hot data. Hot data is likely to be frequently accessed in the future. Accordingly, moving specific key-value pairs in the target SST to a higher target level can greatly reduce the number of levels required to query when accessing hot data in the future, thereby reducing read amplification to a certain extent.

[0132] At the same time, in an embodiment of the present invention, in the process of moving upward, only specific key-value pairs in the target SST are moved upward, which can avoid including unnecessary key-value pairs in the data moved upward to a certain extent, thereby avoiding unnecessary data reading and writing overhead, reducing the upward overhead, and avoiding write amplification problems.

[0133] Reference Figure 8 , is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 8 As shown, the electronic device includes: a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the data processing method of the aforementioned embodiment.

[0134] An embodiment of the present invention provides a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the data processing method of the aforementioned embodiment. Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0135] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0136] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0137] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a predictable manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0141] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0142] The data processing method, the data processing device, the electronic device and one or more machine-readable media provided by the present invention are introduced in detail above. The principles and implementation methods of the present invention are explained in this article by using specific examples. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: For any specified level of the target database, obtaining the access frequency of a specific key-value pair in the specified level; the specific key-value pair includes a key-value pair that has been accessed within a preset time period; Determining the access popularity of the SST in the specified level based on the access frequency of the specific key-value pair in the specified level; the access popularity of the SST is positively correlated with the access frequency of the specific key-value pair belonging to the SST; The SSTs in the designated level whose access heat is greater than a heat threshold are taken as target SSTs, and specific key-value pairs in each of the target SSTs are moved from the designated level to the target level; the target level is located above the designated level.

2. The method according to claim 1, characterized in that The levels in the target database are distributed in different types of storage media, and the faster the reading and writing speed of the storage medium is, the higher the levels distributed in the storage medium are; The moving the specific key-value pairs in each of the target SSTs from the designated level to the target level includes: when the reading and writing speed of the storage medium at the designated level is less than a preset speed threshold, moving the specific key-value pairs in each of the target SSTs from the designated level to the target level; The method further includes: when the reading and writing speed of the storage medium at the designated level is not less than the preset speed threshold, directly moving the target SST from the designated level to the target level.

3. The method according to claim 2, characterized in that The method further comprises: For any of the target SSTs, according to the average access heat of other levels in the target database and the access heat of the target SST, the target level is selected for the target SST from the other levels; the other levels include levels higher than the specified level; The moving the specific key-value pairs in each of the target SSTs from the specified level to the target level includes: Divide the target SSTs corresponding to the same target level into an SST group, and for any of the SST groups, extract the specific key-value pairs included in each target SST in the SST group as the target key-value pairs; Encapsulating the target key value to obtain a combined SST; The combined SST is moved from the specified level to the target level.

4. The method according to claim 3, characterized in that The method further comprises: For any of the target SSTs, generating a first parameter value based on a movement cost of the target SST, and generating a second parameter value based on a movement benefit of the target SST; Based on the first parameter, the second parameter value and a preset cost function, obtaining a corresponding movement cost value of the target SST; When the movement cost value indicates that the movement benefit of the target SST is not greater than the movement cost, the target SST is removed from the SST group.

5. The method according to claim 3, characterized in that: The method further comprises: storing a key name of a specific key-value pair in a corresponding level for each sub-tracker of the heat tracker, and setting an access flag for each key name; the heat tracker includes a sub-tracker corresponding to each level in the target database; When the key-value pair to which the key name belongs is accessed, the value of the access flag is increased; The obtaining of the access frequency of the specific key-value pairs in the specified level includes: extracting the value of each access flag bit from the sub-tracker corresponding to the specified level as the access frequency of each specific key-value pair in the specified level.

6. The method according to claim 5, characterized in that The method further comprises: For any of the levels, in response to an access operation to a key-value pair in the level, if the access key name is not stored in the sub-tracker corresponding to the level, the access key name is stored in the sub-tracker corresponding to the level, and an access flag is set for the access key name; the access key name is the key name of the key-value pair accessed by the access operation; For any of the sub-trackers, when the key names stored in the sub-tracker reach a preset upper limit, the key names whose access flag values ​​in the sub-tracker are 0 are removed.

7. The method according to claim 5, characterized in that The step of selecting the target level for the target SST from the other levels according to the average access heat of the other levels in the target database and the access heat of the target SST includes: For any of the other levels, calculate the numerical average of the access flag bits in the sub-trackers corresponding to the other levels as the average access heat of the other levels; Calculate the absolute value of the difference between the access heat of the target SST and the average access heat of the other levels; The other level corresponding to the minimum absolute value is determined as the target level.

8. The method according to claim 2, characterized in that: The method further comprises: For any type of storage medium, monitoring the performance indicators corresponding to the type of storage medium; When the performance indicator meets the preset indicator condition, the target parameter corresponding to the type of storage medium is adjusted; wherein the performance indicators and target parameters corresponding to different types of storage media are different.

9. A data processing device, characterized in that: The device comprises: A first acquisition module is used to acquire, for any specified level of a target database, the access frequency of a specific key-value pair in the specified level; the specific key-value pair includes a key-value pair that has been accessed within a preset time period; A determination module, configured to determine the access popularity of the SST in the specified level based on the access frequency of the specific key-value pair in the specified level; the access popularity of the SST is positively correlated with the access frequency of the specific key-value pair belonging to the SST; The first moving module is used to take the SST with access heat greater than the heat threshold in the specified level as the target SST, and move the specific key-value pairs in each target SST from the specified level to the target level; the target level is located above the specified level.

10. An electronic device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store executable instructions, and the executable instructions enable the processor to execute the method according to any one of claims 1 to 8.

11. One or more machine-readable media, characterized in that Instructions are stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 8.

Images