A method, apparatus, and device for merging data in a database.

Abstract

This specification discloses a data merging method, apparatus, and device in a database. The database has multiple data storage layers, which are ordered temporally. The scheme includes: detecting whether the data stored in a specified memory storage area has reached a set threshold; if so, selecting data that meets the set conditions from the memory storage area and writing the data that meets the set conditions into the data storage layer with the latest temporal order among the multiple data storage layers, as a data subset; determining the total number and total storage space occupied by each of the existing data subsets in the data storage layer with the latest temporal order; and determining, based on the total number and the total storage space, whether to merge the data in the data storage layer with the latest temporal order into the data storage layer with the second newest temporal order among the multiple data storage layers.

Description

Technical Field

[0001] This specification relates to the field of database technology, and in particular to a data merging method, apparatus, and device in a database. Background Technology

[0002] With the advent of the big data era, the massive amounts of structured, semi-structured, and unstructured information generated daily in living and production environments require various stream processing systems to process and extract meaningful data. NoSQL databases can provide these stream processing systems with highly scalable, high-throughput, and highly available storage support. Although current NoSQL databases already possess powerful write capabilities, the ever-increasing volume of data urgently necessitates the emergence of databases with even higher throughput. Many NoSQL databases are built upon a Log Structured Merge (LSM) tree data structure; therefore, by optimizing the performance of LSM trees, this type of NoSQL database can achieve higher I / O capabilities and throughput, while reducing the latency required for its stored procedures.

[0003] The performance bottleneck of LSM trees is mainly due to write amplification, which leads to many invalid write operations, frequently triggering the LSM tree merging process, thereby reducing system throughput, causing severe performance fluctuations, and also adversely affecting the lifespan of non-volatile storage devices such as SSDs, increasing storage costs.

[0004] The merging strategy has a crucial impact on LSM tree performance. A good merging strategy, or appropriate parameter settings under a given workload, can significantly improve overall system performance. Conversely, a poor strategy can have a negative impact. This is because merging strategies often involve I / O resource allocation and issues related to system cache hits. Modifying the merging strategy aims not only to improve overall merging efficiency but also to ensure system stability.

[0005] Therefore, databases that use LSM trees require a more reliable and efficient data merging solution. Summary of the Invention

[0006] This specification provides one or more embodiments of a data merging method, apparatus, device, and storage medium in a database to solve the following technical problem: For databases using LSM trees, a more reliable and efficient data merging scheme is needed.

[0007] To solve the above-mentioned technical problems, one or more embodiments of this specification are implemented as follows:

[0008] This specification provides one or more embodiments of a data merging method in a database, wherein the database has multiple data storage layers, the multiple data storage layers have a temporal order, and the method includes:

[0009] Detect whether the data stored in the specified memory storage area has reached a set threshold;

[0010] If so, select data that meets the set conditions from the memory storage area, and write the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset;

[0011] In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy;

[0012] Based on the total quantity and the total storage space, determine whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0013] This specification provides a data merging apparatus for a database, wherein the database has multiple data storage layers and the multiple data storage layers have a temporal order, and the apparatus includes:

[0014] The memory data detection module detects whether the data stored in a specified memory storage area has reached a set threshold.

[0015] If so, the memory data transfer module selects data that meets the set conditions from the memory storage area and writes the data that meets the set conditions into the latest time-series data storage layer among the multiple data storage layers, as a data subset;

[0016] The merging factor determination module determines the total number and total storage space occupied by each existing data subset in the latest time-series data storage layer;

[0017] The inter-layer data merging module determines, based on the total quantity and the total storage space, whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0018] This specification provides a data merging device for a database according to one or more embodiments. The database has multiple data storage layers, and the multiple data storage layers have a temporal order. The device includes:

[0019] At least one processor; and,

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0022] Detect whether the data stored in the specified memory storage area has reached a set threshold;

[0023] If so, select data that meets the set conditions from the memory storage area, and write the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset;

[0024] In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy;

[0025] Based on the total quantity and the total storage space, determine whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0026] This specification provides one or more embodiments of a non-volatile computer storage medium, the database having multiple data storage layers having a temporal order, the medium storing computer-executable instructions configured as follows:

[0027] Detect whether the data stored in the specified memory storage area has reached a set threshold;

[0028] If so, select data that meets the set conditions from the memory storage area, and write the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset;

[0029] In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy;

[0030] Based on the total quantity and the total storage space, determine whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0031] The above-described at least one technical solution adopted in one or more embodiments of this specification can achieve the following beneficial effects: When the data stored in the memory storage area reaches a set threshold and triggers the transfer of data to the data storage layer, a dynamically sized subset of data can be obtained based on set conditions (for example, distinguishing between hot and cold data, and only taking cold data that meets the set conditions each time), instead of the traditional fixed-size subset of data, and written accordingly to the latest time-series data storage layer. At the same time, since the size of each subset of data is uncertain, when considering when to trigger the merging of data in the latest time-series data storage layer to the next newest time-series data storage layer, not only the total number of each subset of data in the latest time-series data storage layer is considered, but also the total space occupied by them, thereby helping to avoid frequent triggering of the merging process and enabling more efficient and reliable triggering and execution of the merging process. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating a data merging method in a database provided for one or more embodiments of this specification;

[0034] Figure 2 In the scenario of the LSM tree provided in one or more embodiments of this specification, Figure 1 A schematic diagram illustrating the implementation principle of the Chinese method;

[0035] Figure 3 This diagram illustrates the size of the data subset set for each data storage layer, provided for one or more embodiments of this specification.

[0036] Figure 4 A flowchart illustrating a data fusion and reverse merging scheme provided for one or more embodiments of this specification;

[0037] Figures 5(a) to (d) illustrate one or more embodiments provided in this specification. Figure 2 A schematic diagram illustrating the test results of the Chinese scheme;

[0038] Figure 6 A schematic diagram of the structure of a data merging device in a database provided for one or more embodiments of this specification;

[0039] Figure 7This is a schematic diagram of the structure of a data merging device in a database provided for one or more embodiments of this specification. Detailed Implementation

[0040] This specification provides a data merging method, apparatus, device, and storage medium in a database.

[0041] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0042] An LSM tree is a data structure that organizes data storage across memory and disk. Its core idea is incremental modification and off-site storage. It first caches all written data in memory, and then writes the data stored in memory to disk using sequential disk I / O.

[0043] In practical applications, LSM-based databases have multiple data storage layers, typically on disk, with a temporal order. The temporal order is usually represented by levels (the maximum depth of an LSM tree is set before creation; levels represent different layers within the tree), each level representing a data storage layer, such as level 0, level 1, ..., level n. Smaller numbers indicate newer data. New data is first cached in memory (e.g., a memTable, typically using R-trees, skip lists, or other data structures to organize ordered key-value pairs). When the data in the memtable reaches a set threshold (e.g., when full), the data is written to the most recently updated data storage layer. At appropriate times, the most recently updated data storage layer merges with the next most recently updated data storage layer, which is then cleared to await new data from memory. Similarly, the next most recently updated data storage layer merges its data with the next most recently updated data storage layer. This solution primarily focuses on when data in the latest time-series data storage layer should be merged into the next newest time-series data storage layer.

[0044] Currently, the data in the memory storage component is not differentiated. Each time a set threshold is triggered, it is written to the latest time-series data storage layer, forming a new fixed-size sorted string table (SSTable, generally used as the disk storage component of the LSM tree). However, in practical applications, if a large amount of hot data is updated, the data in the memory storage component will be frequently written to the latest time-series data storage layer, which will also frequently trigger the data merging process between data storage layers (referred to as merging).

[0045] Based on this, this solution separates hot and cold data in the memory storage component. Only cold data is written to the latest time-series data storage layer, while hot data mainly refers to data that has been updated multiple times. Hot data will gradually be converted into cold data. Based on this separation of hot and cold data, hot data will not be transferred to disk in a timely manner, especially its old values ​​will not be written to disk. However, new values ​​that have stabilized can be written to disk as they are converted from hot data to cold data. This greatly reduces the pressure on memory to disk writes and data storage layer merging caused by frequent updates of hot data.

[0046] Furthermore, if hot and cold data separation is adopted, since the cold data is not the full amount each time, its size may change dynamically. This can cause the SSTable to not be fully filled each time, leading to a faster increase in the number of SSTables and triggering a merge earlier, wasting storage space in the data storage layer. One approach considered triggering a merge based on the proportion of duplicate keys in the overlapping key ranges of adjacent data storage layers. For example, a merge could be triggered when the proportion of duplicate keys between level 0 and level 1 exceeds a set threshold. However, this strategy would also result in too many files in level 0, requiring the reading of a large number of files during queries or merges, causing a large number of I / O operations and reducing system throughput.

[0047] Based on this, the operation of separating hot and cold data is further improved by using SSTables whose size can dynamically change, and simultaneously considering the total number of SSTables in level 0 and the total storage space they occupy (two factors) to decide whether to trigger a merge. A larger total number or a larger total storage space increases the probability of triggering a merge. However, if the probability of triggering a merge for these two factors differs significantly, a more refined analysis and decision-making process is implemented. For example, when the total number is small, the decision to trigger a merge is primarily based on the total storage space. Conversely, when the total number is large, the decision to trigger a merge is primarily based on the total number of SSTables.

[0048] Furthermore, based on the above scheme, this application proposes a proactive reverse data merging scheme to improve query efficiency in LSM trees.

[0049] The following section will elaborate on the above ideas.

[0050] Figure 1 This is a flowchart illustrating a data merging method in a database provided by one or more embodiments of this specification. This process is executed, for example, on a database server or a business processing device connected to the database. Figure 1 In such scenarios, the database has multiple data storage layers with a temporal order. The LSM tree is a typical example of this type of structure. This solution is mainly applied to distributed databases, but it can also be applied to other databases.

[0051] Figure 1 The process includes the following steps:

[0052] S102: Detect whether the data stored in the specified memory storage area has reached the set threshold.

[0053] In one or more embodiments of this specification, the specified memory storage area includes the aforementioned MemTable, and it is detected whether the data in the MemTable has accumulated to a certain extent. A threshold is set, for example, to indicate that when the MemTable is full, the set threshold is considered to have been reached. One or more memory storage areas can be maintained in memory, where the stored data includes cold data and hot data. As explained above, with changes over time or data updates, cold and hot data may convert to each other. For example, data that has been modified multiple times recently or frequently accessed can be defined as hot data, and vice versa for cold data.

[0054] S104: If so, select data that meets the set conditions from the memory storage area and write the data that meets the set conditions into the latest time-series data storage layer (e.g., the level 0 layer mentioned above) among the multiple data storage layers as a data subset.

[0055] In one or more embodiments of this specification, based on triggering a set threshold and filtering with set conditions, only a portion of data is selected from the current memory storage area each time and written from memory to the data storage layer, thus obtaining a new data subset. This is different from the traditional method of writing all data. When using a hot / cold data separation scheme, the set condition includes "cold data," meaning only the cold data is selected for writing. Following this approach, there can be different or more settable conditions. For example, the set condition could also be based on a specific business type (e.g., a business type with a relatively large but less important volume of business data).

[0056] In one or more embodiments of this specification, based on the amount of data that meets set conditions, a corresponding sorted string table (SSTable) is generated in the latest-time data storage layer among multiple data storage layers. The data that meets the set conditions is written into the SSTable, and the written SSTable is used as a subset of data. It should be noted that both MemTable and SSTable are exemplary data structures used in memory or on disk, and obviously other forms of data structures can also be used to store data.

[0057] In traditional approaches, to facilitate data writing, the sizes of MemTable and SSTable might be set to the same and fixed. When one MemTable is full, the data is written to one SSTable on disk. However, in this approach, since only a portion of the data is written, a dynamically sized subset of data can be used. Therefore, in level 0, the sizes of each SSTable may differ and be smaller than the usual fixed size.

[0058] In one or more embodiments of this specification, when no additional data (such as optional index data, maintenance instruction data, or data from reverse merging mentioned later) needs to be written, a sorted string table of a size as consistent as possible with the size of the data that meets the set conditions can be generated in the latest time-series data storage layer. If there is additional data, the sorted string table can be expanded accordingly, so that the storage space in the sorted string table can be fully utilized.

[0059] S106: In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy.

[0060] In one or more embodiments of this specification, when the size of the data subset is fixed, the appropriate trigger for merging can be determined primarily based on the total number of data subsets. However, in this scheme, the size of the data subset may change each time. Therefore, the total number alone is insufficient to accurately reflect the current storage burden of the data storage layer. Based on this, the total storage space of each data subset is also considered. If the total number is too large, the IO resource usage pressure during merging will be high; if the total storage space is too large, the storage burden will be heavy and query efficiency will be poor. Of course, if both are too small, frequent merging will be triggered, which is also unsuitable. Therefore, this scheme will further provide a more sophisticated and detailed merging trigger control scheme based on these two factors later.

[0061] S108: Based on the total quantity and the total storage space, determine whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0062] In one or more embodiments of this specification, the above-described merging can also be referred to as compression. In an LSM tree, all data is divided into several storage components (e.g., the data storage layers mentioned above). Taking a strategy of triggering merging when full as an example, when the i-th storage component Ci is full, a merge will be triggered. Specifically, all leaf nodes in Ci will be merged with storage component Ci+1 to form a new storage component Ci+1, and then storage component Ci will be cleared.

[0063] In one or more embodiments of this specification, a similar approach can be used to determine whether to trigger the merging of data in the next newest data storage layer (level 1) into the next data storage layer (level 2), and so on, until merging is performed between adjacent data storage layers. The specific strategies used for merging between different layers may vary.

[0064] pass Figure 1 This method, when the data stored in the memory storage area reaches a set threshold, triggers the transfer of data to the data storage layer. Based on set conditions, it can obtain a dynamically sized subset of data (for example, distinguishing between hot and cold data, and only taking the cold data portion each time), instead of the traditional fixed-size subset of data, and write it to the latest time-series data storage layer accordingly. At the same time, since the size of each data subset is uncertain, when considering when to trigger the merging of data in the latest time-series data storage layer to the next newest time-series data storage layer, it will not only consider the total number of each data subset in the latest time-series data storage layer, but also the total space occupied by them. This helps to avoid frequent triggering of the merging process and can trigger and execute the merging process more efficiently and reliably.

[0065] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation schemes and extension schemes of this method, which will be further explained below.

[0066] Based on the foregoing description, it is intuitive that one or more embodiments of this specification also provide a scenario in which an LSM tree is used. Figure 1 The implementation principle diagram of the Chinese method and the diagram showing the data subset size set for each data storage layer are as follows: Figure 2 , Figure 3 As shown.

[0067] In a traditional LSM tree, once the size of the MemTable reaches a set threshold, a process of writing the MemTable from memory to disk is triggered, forming an SSTable. However, in this solution, as... Figure 3 As shown, MemTable implements hot and cold data differentiation, writing only cold data to disk. SSTables are generated in level 0, and then merged from level 0 to level 1, and then down to level n as needed. As explained earlier, this SSTable occupies less space and its size is uncertain compared to a normal SSTable. For ease of distinction, the SSTable generated in level 0 can be called a secondary SSTable. Other data storage layers (level 1, ..., level n) can use normal SSTables, or similarly, secondary SSTables. The following examples primarily illustrate the former approach.

[0068] exist Figure 2 In this example, a normal SSTable is assumed to be i MB, where i is a configurable value and MB is a unit of storage space. It can be seen that at level 0, the size of each secondary SSTable is smaller than that of a normal SSTable, for example, 0.5 times (0.5i MB), 0.75 times (0.75i MB), 0.6 times (0.6i MB), etc., while other data storage layers use normal SSTables.

[0069] In one or more embodiments of this specification, the presence of secondary SSTables makes the level 0 layer of this scheme more complex than the traditional approach. To avoid performance degradation caused by excessively frequent level 0 layer merging due to secondary SSTables, a new merge triggering control scheme is proposed. This new scheme determines the timing of the merge by comprehensively considering the total number of SSTables in the level 0 layer and the total storage space they occupy.

[0070] In summary, the larger the total storage space occupied by all SSTables in level 0, the more likely it is to trigger level 0 merging. The larger the total number of SSTables in level 0, the more likely it is to trigger level 0 merging. Generally speaking, the relationship between these two factors is roughly positively correlated and they grow in tandem. However, the introduction of dynamically sized secondary SSTables makes this positive correlation more volatile and no longer a simple linear relationship. Therefore, it is necessary to consider the different tendencies of the two factors, total storage space and total number, under different circumstances.

[0071] Based on this, we can determine whether to use the total number or the total storage space as the primary factor to decide whether to merge data from the latest time-series data storage layer into the next newest time-series data storage layer among multiple data storage layers. The primary factor can be determined, for example, based on the magnitude or growth rate of the total number. The process or result of determining the primary factor can then be used to decide whether to merge. Specifically, if the probabilities corresponding to these two factors differ significantly (for example, due to some secondary SSTables being much smaller than normal SSTables, resulting in a faster growth in the total number and a slower growth in the total storage space), when the total number is small, the total storage space can be used as the primary factor, while when the total number is large, the total number can be used as the primary factor.

[0072] Based on the above analysis, corresponding numerical values ​​can be calculated to measure the aforementioned probabilities. Generally, probability values ​​are normalized for easy comparison, so they are used for measurement. However, other forms of measurement are also acceptable. Based on the total storage space, the probability value for performing a merge is calculated as the first probability value. The total storage space is positively correlated with the first probability value. Based on the total quantity, the probability value for performing a merge is calculated as the second probability value. The total quantity is positively correlated with the second probability value. The first and second probability values ​​are then compared. If the first probability value is larger, the total storage space is used as the primary factor; if the second probability value is larger, the total quantity is used as the primary factor. Of course, in calculating the first and / or second probability values, the total quantity can be used as a parameter (at least when calculating the second probability value, the total quantity will be used, reflecting at least the positive correlation between the total quantity and the second probability value) to further reflect the aforementioned changes in tendency differences.

[0073] It should be noted that instead of directly comparing the size of the two probability values, the main factor can be determined based on the total number, and then the corresponding probability value can be used to determine whether to trigger merging.

[0074] In addition to determining whether to trigger a merge based on the aforementioned probability values, thresholds for total storage space and total quantity can be set to combine with the probability values ​​as conditions for decision-making. The relationship between these conditions—whether one or all must be satisfied—can be pre-defined according to actual needs. The following embodiments primarily focus on the relatively more complex condition of probability values.

[0075] In one or more embodiments of this specification, the calculation functions for the first probability value and the second probability value are designed to reflect the aforementioned changes in the tendency difference. Specifically, the rate at which the second probability value increases with the total quantity is higher than the rate at which the first probability value increases with the total storage space. For example, after the total quantity increases (e.g., after reaching a certain quantity), the rate at which the second probability value increases with the total quantity is higher than or tends to be higher than the rate at which the first probability value increases with the total storage space. The implementation methods are varied: for example, the calculation function for the first probability value uses a linear increasing function (e.g., a linear function that meets the conditions), and the calculation function for the second probability value uses a concave function with a higher growth rate (e.g., a quadratic function that meets the conditions); similarly, the calculation function for the second probability value can use an increasing function of a higher order; another example is that the calculation function for the first probability value uses a convex function, and the calculation function for the second probability value uses a concave function with a higher growth rate; yet another example is that a piecewise function can be designed based on the total quantity, such that when the total quantity is below a certain level, the growth rate corresponding to the calculation function for the first probability value is dominant, and when the total quantity is above a certain level, the growth rate corresponding to the calculation function for the second probability value is dominant; and so on.

[0076] In one or more embodiments of this specification, in addition to the factors mentioned above, normalization-related parameters are also introduced to calculate the probability value, such as the expected fixed size set for the data subset (which can be set to the normal size of the SSTable), the reference threshold for triggering the merging of the number of data subsets (which can be the threshold of the total number mentioned above; if the total number exceeds the threshold, merging can be considered), etc.

[0077] For example, based on the total storage space, the expected fixed size set for the data subset, and the set merge trigger reference threshold for the number of data subsets less than the total number, a probability value for performing a merge is calculated as the first probability value. In this case, the actual size of the data subset in the latest time-series data storage layer is less than the expected fixed size, and the expected fixed size and the merge trigger reference threshold are negatively correlated with the first probability value.

[0078] For example, based on the total number and the merge trigger reference threshold, the probability value of performing the merge is calculated as the second probability value, where the merge trigger reference threshold is negatively correlated with the second probability value.

[0079] Based on the parameters listed in the example above, an exemplary formula (Formula 1) is provided intuitively to calculate the probability value of merging at level 0:

[0080]

[0081] Where p represents the probability value, max represents taking the maximum value among the two functions within the curly braces, the result of the first function represents the first probability value, the result of the second function represents the second probability value, and level bytes This represents the total storage space mentioned above, also in MB. iMB represents the expected fixed size mentioned above, and k... l0 This represents the aforementioned merge trigger reference threshold, and n represents the total number mentioned above. In an example parameter value, for instance, if iMB is 2MB and the maximum number of data subsets set in level 0 is 8, then k l0 Take half of the maximum number, which is 4.

[0082] For level 0, the maximum value is calculated as the probability of triggering a merge each time a decision needs to be made. Formula 1 states that the larger the total storage space occupied by all SSTables in level 0, the more likely a merge will be triggered. Furthermore, the larger the total number of SSTables in level 0, the more likely a merge will be triggered. Moreover, the properties of the first term (a linear function) and the second term (a quadratic function) in the formula determine that when the total number of SSTables is small, the maximum value is mainly determined by the total storage space, while when the total number of SSTables is large, the maximum value is mainly determined by the total number.

[0083] Furthermore, for level 0 of the LSM tree, the more files there are, the greater the impact on system performance. For other data storage layers, the impact is relatively smaller, especially when other data storage layers use normal SSTables. Whether considering the total storage space or the total number of files, the effect is basically the same. In addition, since data storage layers with higher level numbers tend to accumulate more data, the storage space of subsequent data storage layers may be relatively larger. Therefore, the level number (or the corresponding time sequence age) can be used as a factor in calculating the probability value of whether to trigger a merge in other data storage layers.

[0084] Following the previous approach, based on the total storage space, the expected fixed size set for the data subset, and the temporal freshness of the specified data storage layer, the probability value of merging the data in the specified data storage layer into the next older data storage layer is calculated and used as the third probability value. Here, the specified data storage layer is the data storage layer other than the newest and oldest data storage layers among multiple data storage layers. The size of the data subset in the specified data storage layer is the expected fixed size. The total storage space is positively correlated with the third probability value, while the expected fixed size and the temporal freshness of the specified data storage layer are negatively correlated with the third probability value, or the temporal freshness of the specified data storage layer is positively correlated with the third probability value.

[0085] Intuitively, an exemplary formula (Formula 2) is provided to calculate the probability that data storage layers other than level 0 will perform a merge:

[0086]

[0087] Where p represents the probability value and level represents the level number, it is negatively correlated with the degree of time sequence oldness corresponding to the data storage layer and positively correlated with the degree of time sequence newness corresponding to the data storage layer. It can be seen that the probability value drops to one-tenth of the previous level for each lower level.

[0088] In summary, this solution combines the characteristics of SSTables generated under the hot and cold data separation optimization method and proposes an LSM tree data storage layer merging triggering strategy based on hot and cold data separation. Taking into account the characteristics of different data storage layers on the disk, and considering both the total number of stored files and the total storage space occupied, it makes it more suitable for LSM trees using hot and cold data separation optimization.

[0089] In one or more embodiments of this specification, at level 0, as described above, if the data subset is too small, it will lead to invariance and complicate the decision-making process triggered by merging. Therefore, this application further considers whether it is possible to actively expand the data subset, while simultaneously solving other problems, such as query efficiency issues, thus achieving two goals at once.

[0090] For databases, users frequently request to query target data rows based on some set filtering conditions. For most filtering conditions, there are often multiple or even a large number of data rows that meet the conditions. Among these filtering conditions, there are often overlapping parts. These overlapping parts may be the conditions that need to be frequently verified in daily filtering operations.

[0091] Based on this, we consider proactively merging data rows that at least meet these frequently validated conditions into the same data subset, rather than distributing them according to the order of fact insertion or modification. This makes the size of the secondary SSTable more controllable (and may even transform it into a normal SSTable), and also speeds up filtering by combining comparisons between filtering conditions, thereby improving query efficiency.

[0092] Furthermore, structures like LSM trees, when querying and scanning data, tend to scan from newer time-series data layers to older ones. Therefore, it's desirable to distribute the filtering results as much as possible within newer time-series data layers (e.g., level 0). This allows scanning one or a few relatively lightweight upper-level data layers to determine most of the filtering results. In practical applications, it may not be necessary to return all filtering results, so this majority of the filtering results may already meet the user's needs. Based on this, and combining the ideas from the previous three paragraphs, we consider proactively transferring data that easily meets the conditions from lower data layers to higher data layers (i.e., reverse merging, whereas traditionally forward merging is used), and merging them as much as possible into data subsets with the same filtering conditions, in order to efficiently obtain at least a sufficient number of usable filtering results.

[0093] Based on the ideas outlined above, one or more embodiments of this specification also provide a flowchart illustrating a data fusion and reverse merging scheme, such as... Figure 4 As shown.

[0094] Figure 4 The process may include the following steps:

[0095] S402: Based on the filtering conditions corresponding to multiple historical query requests to the database, determine the common sub-conditions contained in each filtering condition, and use them as the fusion filtering conditions.

[0096] The restriction level of the fusion filter condition is equal to or greater than that of the original filter condition. This allows for further tightening of the filter conditions, making it easier for the fusion filter condition to be universally included in subsequent filter conditions. If the fusion filter condition is satisfied, the original filter condition will necessarily be satisfied as well, but the reverse is not always true. For example, if the original filter condition is a data column C1>5, and C1>20 is used as the fusion filter condition, if a new filter condition C1>10 is subsequently introduced, the filter results that satisfy this fusion filter condition can be directly reused, and scanning level 0 may be sufficient.

[0097] The extent to which this restriction can be expanded depends on experience with historical data and the business meaning of the data in the data table. The business meaning will directly affect the range of filtering conditions that users may set.

[0098] S404: Determine that the data that meets the set conditions can meet the fusion filtering conditions.

[0099] S406: Obtain another portion of data that also meets the fusion filtering conditions from the next newest data storage layer in the multiple data storage layers.

[0100] S408: The data that meets the set conditions and the other part of the data are determined as data that meets the filter conditions.

[0101] S410: In the data storage layer with the latest time sequence among the multiple data storage layers, generate a sorted string table with the same data size as the filtering conditions.

[0102] S412: The data that meets the set conditions is written into the sorting string table, and the other part of the data is also written from the second newest time-series data storage layer into the sorting string table, so as to be merged back into the latest time-series data storage layer. In this way, it is easier to be hit during scanning more efficiently.

[0103] One or more embodiments provided in this specification Figure 2 Figure 5 shows some schematic diagrams illustrating the test results of the Chinese solution, specifically including... Figures 5(a) to 5(d) .

[0104] The YCSB testing tool was used for testing. The main comparison was the performance under YCSB load files A and D. During the dataset loading phase, throughput was primarily compared; during the testing phase, execution time was primarily compared. This solution was implemented in LevelDB and compared with the original LevelDB. The symbols related to LevelDB are LevelDB-loada, LevelDB-testa, LevelDB-loadd, and LevelDB-testd, corresponding to the results obtained by LevelDB after completing the loading and testing phases using load files A and D, respectively. The test results of this solution are labeled as tqLevelDB. The symbols related to tqLevelDB are n-tqLevelDB-a and n-tqLevelDB-d, representing the results of each phase after using load files A and D in tqLevelDB with a hot data area (used to store hot data) accounting for n% of the MemTable. "Unmodified" indicates that only the hot / cold data separation part was used, without using the merging strategy of this solution; "modified" indicates that the merging strategy of this solution was applied. One hyperparameter of this scheme is the ratio of the size of the hot data area to the size of the entire MemTable, which is divided into four cases: 0.2, 0.1, 0.05, and 0.025, as shown below. Figures 5(a) to 5(d)As shown.

[0105] The throughput during the loading phase reflects the write capability of the tested object. As shown in Figures 5(a) and 5(b), corresponding to load file A and load file D respectively, after applying this solution during the loading phase, the throughput of the system, which originally differed in the proportion of different hot data areas in MemTable, now tends to be close to the same value, demonstrating a significant performance improvement.

[0106] The execution time during the testing phase reflects the query performance of the object under test. As shown in Figures 5(c) and 5(d), corresponding to load file A and load file D respectively, it can be seen that the query performance implemented by this solution is not only not compromised, but is even improved in some cases. The positive effect of this solution on system stability can also be seen in the figures.

[0107] Based on the same idea, one or more embodiments of this specification also provide apparatus and devices corresponding to the above methods, such as... Figure 6 , Figure 7 As shown.

[0108] Figure 6 This specification provides a schematic diagram of the structure of a data merging device in a database, according to one or more embodiments. The database has multiple data storage layers, and the multiple data storage layers have a temporal order. The device includes:

[0109] The memory data detection module 602 detects whether the data stored in a specified memory storage area has reached a set threshold.

[0110] If so, the memory data transfer module 604 selects data that meets the set conditions from the memory storage area and writes the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset;

[0111] The merging factor determination module 606 determines the total number and total storage space occupied by each existing data subset in the latest time-series data storage layer;

[0112] The inter-layer data merging module 608 determines, based on the total quantity and the total storage space, whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0113] Optionally, the setting conditions include: belonging to cold data;

[0114] The memory data transfer module 604 performs hot and cold data differentiation processing on the data in the memory storage area and selects the cold data.

[0115] Optionally, the memory data transfer module 604 generates a corresponding sorted string table in the data storage layer with the latest time sequence among the multiple data storage layers, based on the data volume of the data that meets the set conditions.

[0116] The data that meets the set conditions is written into the sorted string table, and the sorted string table after writing is used as a data subset.

[0117] Optionally, the inter-layer data merging module 608 determines whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers, based on whether the total quantity or the total storage space is the primary factor.

[0118] Optionally, the inter-layer data merging module 608 calculates a probability value for performing the merging based on the total storage space, and uses it as a first probability value, wherein the total storage space is positively correlated with the first probability value;

[0119] Based on the total quantity, a probability value for performing the merger is calculated as a second probability value, wherein the total quantity is positively correlated with the second probability value;

[0120] Compare the first probability value with the second probability value. If the first probability value is larger, the total storage space is used as the primary factor. If the second probability value is larger, the total quantity is used as the primary factor.

[0121] Optionally, as the total quantity increases, the rate at which the second probability value increases with the increase of the total quantity tends to be higher than the rate at which the first probability value increases with the increase of the total storage space.

[0122] Optionally, the inter-layer data merging module 608 calculates a probability value for performing the merging based on the total storage space, the expected fixed size set for the data subset, and a merging trigger reference threshold set for the number of data subsets smaller than the total number, and uses this as a first probability value.

[0123] Wherein, the actual size of the data subset in the latest time-series data storage layer is less than the expected fixed size, the expected fixed size, the merge trigger reference threshold, and the first probability value are negatively correlated; and / or, the step of calculating the probability value of performing the merge based on the total quantity, as the second probability value, specifically includes:

[0124] The probability value for performing the merge is calculated based on the total number and the merge trigger reference threshold, and is used as the second probability value.

[0125] The merging trigger reference threshold is negatively correlated with the second probability value.

[0126] Optionally, the inter-layer data merging module 608 calculates a probability value for merging the data in the specified data storage layer into the next older data storage layer based on the total storage space, the expected fixed size set for the data subset, and the time sequence age of the specified data storage layer, and uses this as a third probability value.

[0127] Wherein, the designated data storage layer is the data storage layer other than the data storage layer with the latest time sequence and the data storage layer with the oldest time sequence among the multiple data storage layers, the size of the data subset in the designated data storage layer is the expected fixed size, the total storage space is positively correlated with the third probability value, and the expected fixed size and the degree of time sequence oldness corresponding to the designated data storage layer are negatively correlated with the third probability value.

[0128] Optionally, the data that meets the set conditions meets the set fusion filtering conditions;

[0129] The memory data transfer module 604 determines the fusion filtering conditions that the data that meets the set conditions can meet.

[0130] Another portion of data that also meets the fusion filtering conditions is obtained from the second newest data storage layer in the multiple data storage layers;

[0131] The data that meets the set conditions and the other part of the data are identified as data consistent with the filtering conditions.

[0132] In the data storage layer with the latest time sequence among the multiple data storage layers, a sorted string table with the same data size as the filtering conditions is generated, which is used to reverse merge the other part of the data into the data storage layer with the latest time sequence.

[0133] Optionally, the size of the sorted string table that matches the size of the data consistent with the filtering conditions is equal to the expected fixed size set for the data subset;

[0134] Before determining a portion of the data that meets the set conditions and meets the set fusion filtering conditions, the memory data transfer module 604 determines the common sub-conditions contained in each of the filtering conditions according to the filtering conditions corresponding to multiple historical query requests for the database, and uses them as the fusion filtering conditions.

[0135] The inter-layer data merging module 608 also performs:

[0136] The other part of the data is also written from the second newest time-series data storage layer into the sorted string table, so as to be merged back into the latest time-series data storage layer.

[0137] Figure 7 This specification provides a schematic diagram of the structure of a data merging device in a database, according to one or more embodiments. The database has multiple data storage layers, and the multiple data storage layers have a temporal order. The device includes:

[0138] At least one processor; and,

[0139] A memory communicatively connected to the at least one processor; wherein,

[0140] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0141] Detect whether the data stored in the specified memory storage area has reached a set threshold;

[0142] If so, select data that meets the set conditions from the memory storage area, and write the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset;

[0143] In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy;

[0144] Based on the total quantity and the total storage space, determine whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0145] Based on the same idea, one or more embodiments of this specification also provide corresponding... Figure 1 A non-volatile computer storage medium according to a method, the database having multiple data storage layers having a temporal order, the medium storing computer-executable instructions, the computer-executable instructions being configured as follows:

[0146] Detect whether the data stored in the specified memory storage area has reached a set threshold;

[0147] If so, select data that meets the set conditions from the memory storage area, and write the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset;

[0148] In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy;

[0149] Based on the total quantity and the total storage space, determine whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

[0150] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0151] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0152] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0153] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0154] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0160] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0162] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0163] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0164] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0165] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A data merging method in a database, the database having multiple data storage layers, the multiple data storage layers having a temporal order, the method comprising: Detect whether the data stored in the specified memory storage area has reached a set threshold; If so, select data that meets the set conditions from the memory storage area, and write the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset; In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy, with at least two data subsets having different sizes; Based on the total quantity and the total storage space, determine whether the total quantity or the total storage space is the primary factor to decide whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

2. The method as described in claim 1, wherein the setting conditions include: This is considered cold data; The step of selecting data that meets the set conditions from the memory storage area specifically includes: The data in the memory storage area is processed to distinguish between hot and cold data, and the cold data is selected.

3. The method as described in claim 1, wherein writing the data that meets the set conditions into the data storage layer with the latest time sequence among the plurality of data storage layers, as a subset of data, specifically includes: Based on the data volume of the data that meets the set conditions, a corresponding sorted string table is generated in the data storage layer with the latest time sequence among the multiple data storage layers; The data that meets the set conditions is written into the sorted string table, and the sorted string table after writing is used as a data subset.

4. The method as described in claim 1, wherein determining whether the total quantity or the total storage space is the primary factor specifically includes: Based on the total storage space, calculate the probability value of performing the merge, which is used as a first probability value. The total storage space is positively correlated with the first probability value. Based on the total quantity, a probability value for performing the merger is calculated as a second probability value, wherein the total quantity is positively correlated with the second probability value; Compare the first probability value with the second probability value. If the first probability value is larger, the total storage space is used as the primary factor. If the second probability value is larger, the total quantity is used as the primary factor.

5. The method of claim 4, wherein after the total quantity increases, the rate at which the second probability value increases with the increase of the total quantity tends to be higher than the rate at which the first probability value increases with the increase of the total storage space.

6. The method of claim 4, wherein calculating the probability value of performing the merge based on the total storage space, as a first probability value, specifically includes: Based on the total storage space, the expected fixed size set for the data subset, and the set merging trigger reference threshold for the number of data subsets smaller than the total number, calculate the probability value of performing the merge, which is used as the first probability value. Wherein, the actual size of the data subset in the latest time-series data storage layer is less than the expected fixed size, the expected fixed size, the merge trigger reference threshold, and the first probability value are negatively correlated; and / or, The step of calculating the probability value of performing the merger based on the total quantity, as the second probability value, specifically includes: Based on the total quantity and the merge trigger reference threshold, calculate the probability value of performing the merge, which serves as the second probability value; The merging trigger reference threshold is negatively correlated with the second probability value.

7. The method of claim 1, further comprising: Based on the total storage space, the expected fixed size set for the data subset, and the time sequence of the specified data storage layer, calculate the probability value of merging the data in the specified data storage layer into the next older data storage layer, and use this as the third probability value. Wherein, the designated data storage layer is the data storage layer other than the data storage layer with the latest time sequence and the data storage layer with the oldest time sequence among the multiple data storage layers, the size of the data subset in the designated data storage layer is the expected fixed size, the total storage space is positively correlated with the third probability value, and the expected fixed size and the degree of time sequence oldness corresponding to the designated data storage layer are negatively correlated with the third probability value.

8. The method as described in claim 3, wherein generating a corresponding sorted string table in the data storage layer with the latest time sequence among the plurality of data storage layers based on the data volume of the data meeting the set conditions specifically includes: Determine the fusion filtering conditions that the data meeting the set conditions can meet; Another portion of data that also meets the fusion filtering conditions is obtained from the second newest data storage layer in the multiple data storage layers; The data that meets the set conditions and the other part of the data are identified as data consistent with the filtering conditions. In the data storage layer with the latest time sequence among the multiple data storage layers, a sorted string table with the same data size as the filtering conditions is generated, which is used to reverse merge the other part of the data into the data storage layer with the latest time sequence.

9. The method of claim 8, wherein the size of the sorted string table that is consistent with the size of the data consistent with the filtering conditions is equal to the expected fixed size set for the data subset; Before determining that the data meeting the set conditions can meet the set fusion filtering conditions, the method further includes: Based on the filtering conditions corresponding to multiple historical query requests to the database, the common sub-conditions contained in each filtering condition are determined and used as the fusion filtering conditions. The step of writing the data that meets the set conditions into the sorted string table further includes: The other part of the data is also written from the second newest time-series data storage layer into the sorted string table, so as to be merged back into the latest time-series data storage layer.

10. A data merging apparatus for a database, the database having multiple data storage layers having a temporal order, the apparatus comprising: The memory data detection module detects whether the data stored in a specified memory storage area has reached a set threshold. If so, the memory data transfer module selects data that meets the set conditions from the memory storage area and writes the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset; The merging factor determination module determines the total number and total storage space occupied by each existing data subset in the latest time-series data storage layer, wherein at least two of the data subsets are of different sizes; The inter-layer data merging module determines, based on the total quantity and the total storage space, whether to use the total quantity or the total storage space as the primary factor to decide whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

11. The apparatus of claim 10, wherein the setting conditions include: This is considered cold data; The memory data transfer module performs hot and cold data differentiation processing on the data in the memory storage area and selects the cold data.

12. The apparatus of claim 10, wherein the memory data transfer module generates a corresponding sorted string table in the data storage layer with the latest time sequence among the plurality of data storage layers, based on the data volume of the data that meets the set conditions; The data that meets the set conditions is written into the sorted string table, and the sorted string table after writing is used as a data subset.

13. The apparatus of claim 10, wherein the inter-layer data merging module calculates a probability value for performing the merging based on the total storage space, as a first probability value, wherein the total storage space is positively correlated with the first probability value; Based on the total quantity, a probability value for performing the merger is calculated as a second probability value, wherein the total quantity is positively correlated with the second probability value; Compare the first probability value with the second probability value. If the first probability value is larger, the total storage space is used as the primary factor. If the second probability value is larger, the total quantity is used as the primary factor.

14. The apparatus of claim 13, wherein as the total quantity increases, the rate at which the second probability value increases with the increase of the total quantity tends to be higher than the rate at which the first probability value increases with the increase of the total storage space.

15. The apparatus of claim 13, wherein the inter-layer data merging module calculates a probability value for performing the merging based on the total storage space, the expected fixed size set for the data subset, and a merging trigger reference threshold set for the number of data subsets less than the total number, as a first probability value; in, The actual size of the data subset in the latest time-series data storage layer is less than the expected fixed size, and the expected fixed size and the merge trigger reference threshold are negatively correlated with the first probability value. And / or, The step of calculating the probability value of performing the merger based on the total quantity, as the second probability value, specifically includes: The probability value for performing the merge is calculated based on the total number and the merge trigger reference threshold, and is used as the second probability value. The merging trigger reference threshold is negatively correlated with the second probability value.

16. The apparatus of claim 10, wherein the inter-layer data merging module calculates a probability value, as a third probability value, based on the total storage space, the expected fixed size set for the data subset, and the temporal age of the specified data storage layer; in, The designated data storage layer is the data storage layer other than the latest and oldest data storage layers among the multiple data storage layers. The size of the data subset in the designated data storage layer is the expected fixed size. The total storage space is positively correlated with the third probability value. The expected fixed size and the degree of time aging corresponding to the designated data storage layer are negatively correlated with the third probability value.

17. The apparatus of claim 12, wherein the memory data transfer module determines the fusion filtering conditions that the data meeting the set conditions can meet; Another portion of data that also meets the fusion filtering conditions is obtained from the second newest data storage layer in the multiple data storage layers; The data that meets the set conditions and the other part of the data are identified as data consistent with the filtering conditions. In the data storage layer with the latest time sequence among the multiple data storage layers, a sorted string table with the same data size as the filtering conditions is generated, which is used to reverse merge the other part of the data into the data storage layer with the latest time sequence.

18. The apparatus of claim 17, wherein the size of the sorted string table that is consistent with the size of the data consistent with the filtering conditions is equal to the expected fixed size set for the data subset; Before determining that the data that meets the set conditions can meet the set fusion filtering conditions, the memory data transfer module determines the common sub-conditions contained in each of the filtering conditions according to the filtering conditions corresponding to multiple historical query requests for the database, and uses them as the fusion filtering conditions. The inter-layer data merging module also performs: The other part of the data is also written from the second newest time-series data storage layer into the sorted string table, so as to be merged back into the latest time-series data storage layer.

19. A data merging device for a database, the database having multiple data storage layers having a temporal order, the device comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Detect whether the data stored in the specified memory storage area has reached a set threshold; If so, select data that meets the set conditions from the memory storage area, and write the data that meets the set conditions into the data storage layer with the latest time sequence among the multiple data storage layers, as a data subset; In the latest time-series data storage layer, determine the total number of existing data subsets and the total storage space they occupy, with at least two data subsets having different sizes; Based on the total quantity and the total storage space, determine whether the total quantity or the total storage space is the primary factor to decide whether to merge the data in the latest time-series data storage layer into the next newest time-series data storage layer among the multiple data storage layers.

Images