Data merging method and electronic device

Abstract

This application discloses a data merging method and electronic device, relating to the field of artificial intelligence technology. The method includes: writing data files into the initial sub-layer of the write layer to retain the high-speed writing advantage of the log structure merging tree architecture; when any sub-layer meets preset merging conditions, only multiple data files within that sub-layer are merged to generate at least one new data file, breaking down the originally uniform coarse-grained merging across the entire layer into fine-grained operations at the sub-layer level; if any sub-layer is not the bottom sub-layer of the write layer, the newly generated data file is written into its adjacent next-order sub-layer, driving the data to be progressively organized layer by layer within the write layer, gradually increasing the file size and alleviating file fragmentation; if any sub-layer is the bottom sub-layer, the new data file is retained in that layer. This progressive merging architecture between sub-layers improves data reading performance.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data merging method and electronic device. Background Technology

[0002] In data storage systems, a tiered storage architecture is typically used to improve data writing efficiency. Data is first written to the top storage tier. As write operations continue, a large number of data files accumulate in the top storage tier. When the number of files is too large, reading data requires searching multiple files, leading to increased read latency. Furthermore, significant overhead occurs during subsequent merging processes. Severe file fragmentation within the top storage tier negatively impacts overall system performance.

[0003] To address the aforementioned issues, relevant technologies propose merging multiple data files within the first storage level when the number of files reaches a certain threshold, generating a new data file, and writing it to the next level. This method reduces the number of files within the first storage level, thereby alleviating the file fragmentation problem.

[0004] However, in the above scheme, the first storage level is managed as a whole, and the granularity of merging is relatively coarse. It is impossible to perform fine-grained layering of files within the first storage level, which leads to a decrease in data read performance. Summary of the Invention

[0005] This application provides a data merging method and electronic device to at least solve the problem in related technologies that the inability to perform fine-grained layering of files within the top storage level leads to reduced data reading performance.

[0006] This application provides a data merging method applied to a hierarchical storage system with multiple storage levels. The hierarchical storage system includes at least a write layer, which is divided into multiple sub-layers, each of which includes at least an initial sub-layer and a bottom sub-layer. The method includes: writing data files to the initial sub-layer; if any sub-layer among the multiple sub-layers meets a preset merging condition, merging the multiple data files within that sub-layer to generate at least one new data file, wherein the preset merging condition is determined based on the number of data files and / or the total size of the data files within any sub-layer; if any sub-layer is not the bottom sub-layer, writing at least one new data file to the adjacent sub-layer of that sub-layer; and if any sub-layer is the bottom sub-layer, retaining at least one new data file in the bottom sub-layer.

[0007] This application also provides a data merging apparatus applied to a hierarchical storage system with multiple storage levels. The hierarchical storage system includes at least a write layer, which is divided into multiple sub-layers, each sub-layer including at least an initial sub-layer and a bottom sub-layer. The apparatus includes: a first write module for writing data files to the initial sub-layer; a merging module for merging multiple data files within any sub-layer to generate at least one new data file if any sub-layer meets a preset merging condition, wherein the preset merging condition is determined based on the number of data files and / or the total size of the data files within any sub-layer; and a second write module for writing at least one new data file to an adjacent sub-layer of any sub-layer if any sub-layer is not the bottom sub-layer, and retaining at least one new data file in the bottom sub-layer if any sub-layer is the bottom sub-layer.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of the first data merging method described above when executing the computer program.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the first data merging method described above.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the first data merging method described above.

[0011] The data merging method and electronic device provided in this application employ a progressive merging approach within the write layer. Specifically, data files containing multiple key-value pairs arranged in a key-key sequence are first written to the initial sub-layer of the write layer, preserving the high-speed writing advantage of the log structure's merging tree architecture. When any sub-layer within the write layer meets preset merging conditions, merging is performed only on the multiple data files within that sub-layer, generating at least one new data file. This breaks down the originally uniform coarse-grained merging across the entire layer into fine-grained sub-layer operations, avoiding sudden IO overhead peaks caused by merging across the entire layer. If any sub-layer is not the lowest sub-layer in the write layer, the newly generated data file is written to its adjacent next sub-layer, driving the data to be progressively organized layer by layer within the write layer, gradually increasing the file size and alleviating file fragmentation. If any sub-layer is the lowest sub-layer, the new data file is retained in that layer. This progressive merging architecture between sub-layers can smooth out merging overhead while allowing for larger single-file sizes and fewer total files within sub-layers with higher IDs. Consequently, the number of files that need to be traversed during a query is reduced, effectively improving data reading performance. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A hierarchical storage system provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 3 A flowchart illustrating a data merging method provided in this application embodiment. Figure 1 ; Figure 4 A flowchart illustrating a data merging method provided in this application embodiment. Figure 2 ; Figure 5 A flowchart illustrating a data merging method provided in this application embodiment. Figure 3 ; Figure 6 This is a schematic diagram of a data merging device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] In a hierarchical storage system based on a log-structured merging tree, data is first written to a memory table. Once the table is full, it is frozen and flushed to disk, generating an ordered string table and storing it in the top-level storage layer. This transforms random writes into sequential writes, improving write efficiency. However, as write operations continue, the top-level layer accumulates a large number of data files with overlapping key-value ranges. Querying requires traversing multiple files, increasing read latency. Simultaneously, the computational and I / O (Input / Output) overhead of merging increases, exacerbating file fragmentation. Existing technologies typically merge the entire top-level file layer and move it to the next level when the number of files reaches a threshold. This approach has a coarse-grained merging management, failing to perform fine-grained hierarchical processing of data files within the top-level layer. It is difficult to optimize the read path while controlling merging overhead, thus limiting the overall read / write performance of the system. Therefore, this application proposes a data merging scheme.

[0017] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] The specific application environment architecture or specific hardware architecture on which the execution of the data merging method depends is described here.

[0019] Figure 1 This is a schematic diagram of a hierarchical storage system provided in an embodiment of this application. Figure 1 As shown, the tiered storage system 100 includes multiple storage tiers.

[0020] The storage layers are arranged sequentially along the data persistence path, with the complete data flow as follows: memory table → immutable memory table → write layer (L0) → layer 1 (L1) → layer 2 and deeper layers. The write layer L0, the first storage layer on the disk side, receives all data files flushed from memory to disk; therefore, this layer has the most up-to-date data version and the highest write frequency. Unlike the layers below, the data files in write layer L0 are generated directly from the memory table during flushing to disk, allowing overlapping key-value ranges between files without requiring globally ordered data.

[0021] To address the issues of excessively coarse granularity and sudden bursts of merging overhead in existing write layer merging techniques, this application further subdivides the write layer L0 into multiple sequentially numbered sublayers. Smaller sublayer numbers indicate higher positions and newer data versions. Each sublayer stores several ordered string table (SSTable) data files, with key-value pairs arranged in ascending order within each file. Key-value ranges are allowed to overlap between multiple files within the same sublayer.

[0022] Let the base size of the initial sub-layer (L0-0) be S_base, and the size of the target file corresponding to the i-th sub-layer satisfies the following formula (1): S_level_i = S_base × Formula (1) The larger the sub-layer number, the larger the target file size grows exponentially, while the number of data files within the sub-layer decreases accordingly. When querying and traversing this sub-layer, the number of files that need to be checked decreases simultaneously, thus improving read efficiency. By dividing the write layer into sub-layers, the coarse-grained merging that was originally executed uniformly across the entire layer can be broken down into progressive fine-grained merging between sub-layers, reducing the amount of data and IO overhead in a single merging and preventing merging operations from blocking business read and write operations.

[0023] In all sub-layers of write layer L0, the smallest numbered sub-layer, L0-0, is the initial sub-layer. When the memory table is full and reaches the threshold, it will be frozen into an immutable memory table. A background thread will then perform a disk flushing operation to completely export all ordered key-value pairs in the immutable memory table into a single data file, which will be directly stored in the initial sub-layer L0-0. Continuous writing and disk flushing will accumulate multiple small data files in this sub-layer.

[0024] The largest numbered L0-n is the bottom sub-layer of the write layer. Data starts from L0-0, and after multiple merging and reorganizations between sub-layers, the file size gradually increases and the number of files gradually decreases, progressing layer by layer towards the sub-layers with larger numbers, eventually entering the bottom sub-layer L0-n. When the files in the bottom sub-layer meet the merging conditions, they can be further merged with the files in the overlapping area of ​​the first layer, and then sink down to the first layer. n is a natural number.

[0025] This progressive merging architecture, from the initial sub-layer to the underlying sub-layer, can gradually alleviate the file fragmentation problem in the write layer, while avoiding the sudden high overhead caused by merging the entire layer, thus smoothing out the merging overhead and ensuring the stability of the system's read and write performance.

[0026] The number of sub-layers can be dynamically adjusted based on the system memory size and expected write load, with a value range of 2-8; here, it is set to 4. Specific parameters are as follows: L0-0 sub-layer: target file size 64MB, number threshold 4; L0-1 sub-layer: target file size 128MB, number threshold 6; L0-2 sub-layer: target file size 256MB, number threshold 8; L0-3 sub-layer: target file size 512MB, number threshold 10.

[0027] It should be noted that this system is built upon the hierarchical storage concept of Log Merge Tree (LSM-Tree), containing multiple levels of disk storage. Each level is arranged sequentially according to the data flow from newest to oldest and from top to bottom. Data writes are initially completed in memory. Once the memory table is full, data is batch-fed to the top level of disk, and then gradually pushed down to lower levels through merge and compression operations. This transforms random writes into sequential writes, reducing disk random I / O overhead and improving write throughput. The deeper the level, the older the data version, the larger the total data volume, and the larger the file granularity, the fewer files need to be accessed during queries.

[0028] The hierarchical storage system provided in this application is built on a log-structured merge tree architecture, containing multiple storage levels arranged sequentially according to the data persistence flow. The disk level at the top is the write layer (level 0), which is further divided into multiple sequentially arranged sub-layers, each storing ordered key-value data files. The sub-layer with the smallest number is the initial sub-layer, receiving the latest data generated by memory flushing to disk, while the sub-layer with the largest number is the bottom sub-layer. Data is progressively merged and organized layer by layer before sinking to deeper levels. This fine-grained division of the write layer's internal sub-layers reduces the number of files traversed during querying while controlling merging overhead, thus balancing write performance and read efficiency.

[0029] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 2 As shown, the electronic device 200 may include a processor 210 and a memory 220.

[0030] The memory 220 stores machine-executable instructions that can be executed by the processor 210. When the electronic device 200 is running, these machine-executable instructions are executed. The processor 210 and the memory 220 communicate via a bus. The processor 210 can execute these machine-executable instructions to implement the data merging method.

[0031] The memory 220, processor 210, and bus components are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected via one or more communication buses or signal lines. The memory 220 includes at least one software functional module, which is stored or embedded in the operating system (OS) of the electronic device 200 in the form of software or firmware. This software functional module includes at least one executable module. The processor 210 is used to execute the executable modules stored in the memory 220, such as the software functional modules and computer programs included in the data merging method.

[0032] The memory 220 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0033] The electronic device 200 can be selected according to the actual situation; for example, the electronic device 200 can be selected as a computer device. Furthermore, the electronic device 200 has software, an application program (APP), or a hierarchical storage system capable of executing data merging methods, and the hierarchical storage system can be used to execute data merging methods, etc.

[0034] The data merging method provided in this application embodiment can be executed by a processor in the electronic device 200. The data merging method provided in this application embodiment will be further explained below. Figure 3 A flowchart illustrating a data merging method provided in this application embodiment. Figure 1 .like Figure 3 As shown, the method may include: S310. Write the data file to the initial sublayer.

[0035] The data file stores key-value pairs arranged in a sequential order of multiple keys.

[0036] In one possible implementation, since the sub-layers of the write layer L0 are arranged in ascending order of their numbers, corresponding to the temporal relationship of data versions from newest to oldest, the initial sub-layer L0-0, which carries the most recently generated data file, ensures the monotonically increasing version consistency between sub-layers. This allows query operations to retrieve data sequentially by sub-layer number from smallest to largest, and once the target key is matched, the result can be returned directly without traversing all sub-layers. This ensures both the correctness of the data version and improves read efficiency. Simultaneously, the initial sub-layer, as the disk entry point for the write layer, directly receives incremental data from memory flushing to disk without needing to merge it with already processed files in lower layers. This minimizes flushing latency, and subsequent gradual merging between sub-layers can progressively process data down to lower layers, evenly distributing the merging overhead and avoiding sudden I / O spikes caused by a unified merging of the entire layer.

[0037] Based on this, data write requests first enter a memory table in memory. The memory table uses a skip list structure to store key-value pairs, ensuring that the key-value pairs are always arranged in ascending order by key, supporting efficient random insertion, update, and deletion. Deletion operations are implemented by writing tombstone markers. When the memory table's occupied capacity reaches a preset threshold, the memory table is frozen as an immutable memory table, ceasing to receive new write requests. Simultaneously, the system creates a new empty memory table to continue handling business writes. The background disk flushing thread traverses the immutable memory table, serially writing all key-value pairs to disk according to the ascending order of the internal keys, generating a single ordered string table data file. This file stores multiple key-value pairs arranged in ascending order by key, and the size of a single file is close to the preset base size S_base. Finally, this data file is stored in the initial sub-layer at the beginning of the write layer. Each disk flushing operation generates one data file, and multiple flushes will accumulate multiple data files with overlapping key-value ranges in the initial sub-layer.

[0038] S320. If any sub-layer among multiple sub-layers meets the preset merging conditions, then the multiple data files in any sub-layer are merged to generate at least one new data file.

[0039] The preset merging condition is a preset judgment rule used to trigger the merging operation of data files within a sub-layer. Its function is to control the fragmentation degree of data files in each sub-layer of the write layer and achieve a dynamic balance between read performance and merging overhead. The preset merging condition can be determined based on the number of data files and / or the total size of data files in any sub-layer. The preset merging condition may include at least one of the following: a threshold for the number of data files in a sub-layer, a threshold for the cumulative data volume of a sub-layer, and a threshold for the overlap of key intervals. Different threshold parameters are configured for sub-layers with different numbers. The target file size corresponding to the i-th sub-layer satisfies the above formula (1). The larger the sub-layer number, the lower the corresponding file number threshold and the higher the target file size threshold. By setting the preset merging condition for the layers, the excessive accumulation of files in the sub-layer can be avoided, which can lead to a surge in read traversal overhead. It can also avoid the additional input and output overhead caused by too frequent merging operations.

[0040] In one possible implementation, because the key-value ranges of data files within the write layer overlap, the more files there are, the more files need to be traversed when querying that sublayer, resulting in higher read latency. Here, "any sublayer" refers to any sublayer within the write layer, and each sublayer can trigger merging independently without waiting for a unified trigger across the entire layer. When the fragmentation index of any sublayer reaches a preset threshold, the files within that sublayer are merged promptly. This multi-way merging eliminates duplicate key-value data and merges redundant information, directly reducing the total number of files in the sublayer and lowering the traversal overhead for subsequent queries. Simultaneously, triggering fine-grained merging on a per-sublayer basis, instead of the traditional unified merging across the entire layer, evenly distributes the merging input and output overhead, avoiding performance fluctuations caused by large-scale merging. Furthermore, the independent triggering method for each sublayer can adapt to the data growth rate of different sublayers, improving the flexibility and adaptability of merging scheduling.

[0041] Based on this, firstly, all data files within any sub-layer participating in this merge are selected according to preset merging conditions. A locking operation is performed on the selected data files. During the locking period, the files can be read and accessed, but are prohibited from participating in other merge tasks and from being deleted by garbage collection. Then, an iterator for ascending key traversal is created for each data file participating in the merge. Based on the multi-way merge algorithm, the key-value pairs of all files are traversed synchronously, and the key-value sequence is output in ascending order. During the merge process, conflict handling is performed based on the version number or timestamp attached to each key-value pair. The latest valid data is retained, while expired data and deleted data with tombstone markers are discarded. If the new file generated by the merge is still circulating within the sub-layer of the write layer, a single new data file is output, with a file size close to the target file size of the corresponding sub-layer. If the merge result sinks to the first layer, the sorted data is divided into one or more new data files with continuous and non-overlapping key ranges according to the single file size threshold of the first layer. Finally, the new data files are registered and written to the target layer, the original data files are unlocked and marked as recyclable, and the background garbage collection thread performs disk file deletion when the file reference count is zero.

[0042] S330. If any sublayer is not the bottom sublayer, then write at least one new data file to the adjacent sublayer of any sublayer.

[0043] In a possible implementation, if any sub-layer performing the merging operation is not the bottommost sub-layer arranged last in the write layer, the new data file generated by merging is written to its adjacent next-numbered sub-layer, with the purpose of maintaining the architectural logic of progressive layered sorting inside the write layer, and ensuring the file size gradient and data version time sequence of each sub-layer. The target file size of each sub-layer in the write layer increases exponentially as the number increases, and the design thereof is to disperse the merging overhead in a manner of layer-by-layer merging and stepwise amplification, so as to avoid the burst input / output peak caused by traditional full-layer merging. If the merged file still remains in this sub-layer, the stepwise increase of file granularity cannot be realized, and the sorting significance of sub-layer division is lost; if the file is directly written across multiple sub-layers to a deeper layer, the file size distribution and version time sequence of each sub-layer will be destroyed, resulting in aggravated fragmentation of some sub-layers and disordered query paths. Writing to the adjacent sub-layer can not only ensure that the merged file matches the target size gradient of the next layer, but also maintain the monotonous order of data versions from new to old between sub-layers, so as to ensure the correctness and efficiency of the query operation that retrieves sequentially from the smallest to the largest sub-layer number.

[0044] Based on this, any sub-layer currently performing merging is denoted as the L0-i-th sub-layer, where i is the sub-layer number and satisfies 0 ≤ i ≤ n, and n is the number of the bottommost sub-layer of the write layer; first, a plurality of data files with overlapping key-value intervals are screened out from the L0-i-th sub-layer and upper sub-layers according to a preset merging condition, a locking operation is performed on the selected files, and during the locking period, the files support read access and are prohibited from participating in other merging tasks and garbage collection. Then, the ordered key-value pairs inside each file are read through a multi-way merging algorithm, conflict processing is performed according to the version number or timestamp attached to the key-value pairs, the valid data with the latest version is retained, and the expired data and the data corresponding to the deletion tombstone mark are discarded; since the merging occurs inside the write layer, the constraint that the key intervals of files in the same layer do not overlap with each other does not need to be satisfied, so the output result is not split, and a single new data file is generated after the multi-way merging is completed. The size of the new file matches the target file size of the adjacent next-numbered sub-layer of any sub-layer, that is, the L0-(i+1)-th sub-layer, and the target file size of the L0-(i+1)-th sub-layer satisfies the above formula (1); finally, the new data file is registered and written into the metadata of the L0-(i+1)-th sub-layer, and the original merged old data files are synchronously marked as recyclable, and the disk deletion is asynchronously performed by the background garbage collection thread according to the file reference count. After the new file is written into the adjacent sub-layer, it can continue to participate in the subsequent progressive merging at deeper layers, and gradually complete the improvement of file granularity and data sorting.

[0045] S340. If any sub-layer is the bottommost sub-layer, retaining at least one new data file in the bottommost sub-layer.

[0046] In one possible implementation, to maintain a gradual merging rhythm within the write layer and avoid frequent cross-layer merging that could cause write amplification and a surge in I / O overhead, the sub-layers within the write layer follow a progressive merging logic from smallest to largest. Data files gradually move to sub-layers with larger numbers as they are being processed. The bottom sub-layer is the deepest level within the write layer and also serves as a transitional level between the write layer and the next larger level. If the newly generated files are directly moved down to the next larger level at this point, it would frequently trigger cross-layer merging, significantly increasing disk I / O and computational overhead. However, keeping the new files in the bottom sub-layer allows for sufficient gradual processing within the write layer, gradually bringing the file size closer to the target threshold. Once the cross-layer merging conditions are met, the files are moved down uniformly, smoothing out merging overhead, reducing write amplification, and maintaining a stable number of files within the write layer, while avoiding unnecessary extension of the read path to deeper levels.

[0047] Based on this, firstly, according to the preset merge triggering conditions, multiple data files participating in this merge are selected within the write layer, and the lowest sub-layer covered by this merge is determined, i.e., any sub-layer. When it is determined that any sub-layer is the bottom sub-layer of the write layer, multi-way merge sort is performed on the key-value pairs in all selected data files. During the merge process, conflict handling is performed based on the timestamp or version number attached to the key-value pairs. The latest valid key-value pairs are retained, and expired data and tombstone data marked for deletion are discarded to obtain a complete and ordered key-value sequence. Since this merge is a merge operation within the write layer, the merge result generates a single new data file, whose file size corresponds to the target file size formula (1) of the last sub-layer. Then, it is determined whether the new file meets the cross-layer merge triggering conditions for sinking to the first layer. If the triggering conditions are not met, the new data file is retained and registered in the metadata of the bottom sub-layer of the write layer, and can participate in the next round of merge of the sub-layer, or the sinking operation is performed when the cross-layer conditions are met.

[0048] The data merging method provided in this application first writes the data files to the initial sub-layer of the write layer to retain the high-speed write advantage of the log structure merging tree architecture. When a sub-layer within the write layer meets a preset merging condition, only multiple data files within that sub-layer are merged to generate at least one new data file. This breaks down the originally uniform coarse-grained merging across the entire layer into fine-grained operations at the sub-layer level, avoiding sudden IO overhead peaks caused by merging across the entire layer. The preset merging condition can be determined based on the number of data files and / or the total size of the data files in any sub-layer. If any sub-layer is not the bottom sub-layer in the write layer, the newly generated data file is written to its adjacent next sub-layer, pushing the data to be progressively organized layer by layer within the write layer, gradually increasing the file size and alleviating the file fragmentation problem. If any sub-layer is the bottom sub-layer, the new data file is retained in that layer. This progressive merging architecture between sub-layers can smooth out merging overhead while allowing for larger single-file sizes and fewer total files within sub-layers with higher IDs. Consequently, the number of files that need to be traversed during a query is reduced, effectively improving data reading performance.

[0049] Optionally, the above method further includes reserving at least one new data file after the underlying sub-layer. In response to the triggering of a preset cross-layer merging condition, the data files in the lower-level sub-layer are merged with the data files in the adjacent lower layer of the write layer.

[0050] Among them, the preset cross-layer merging conditions can be at least one of the following: the total data capacity threshold of the last sub-layer of the writing layer, the number of files threshold, or periodic timed triggering. When the conditions are met, the cross-layer merging operation is triggered.

[0051] In one possible implementation, the minimum and maximum keys of all data files to be merged within the lowest sub-layer are first obtained to determine the key-value range covered by this merge. Then, based on the key-value ranges recorded in the metadata of adjacent lower-layer data files, all data files overlapping with this merge range are selected, and these, along with the files to be merged in the lowest sub-layer, form the input set for this merge. Subsequently, a multi-way merge sort is performed on all data files in the input set. During the merge, the version number or timestamp attached to the key-value pairs is used as the basis to resolve conflicts between multiple versions of the same key, retaining the latest valid key-value pairs and discarding expired data and tombstone data marked for deletion, resulting in a globally ordered key-value sequence.

[0052] Because adjacent lower layers require that the key ranges of data files within the same layer do not overlap, during the merge output stage, the ordered key-value sequence is split according to the preset maximum single-file capacity threshold for that layer, generating one or more new data files with continuous and non-overlapping key ranges. Finally, all new data files are registered and written to the adjacent lower layer, completing the data sinking from the write layer to deeper layers. By progressively reorganizing and reducing the amount of data merged across layers within the preceding write layer, the write amplification effect and disk I / O overhead caused by cross-layer merging can be effectively reduced, while maintaining the ordered structure of deeper layers and improving read efficiency.

[0053] The data merging method provided in this application, in response to the triggering of a preset cross-layer merging condition, performs multi-way merge merging on data files in the lower-level sub-layer of the write layer and data files with overlapping key intervals in the next adjacent level of the write layer. By merging across layers, the ordered data of the write layer, which has been progressively sorted, is pushed down to a deeper level. This ensures the globally non-overlapping ordered structure of the key intervals of the next level files and optimizes read performance, while avoiding write amplification and disk I / O overhead caused by frequent cross-layer merging, thus achieving balanced optimization of read and write performance.

[0054] Figure 4 A flowchart illustrating a data merging method provided in this application embodiment. Figure 2 .like Figure 4 As shown, in the above method, if any sub-layer among multiple sub-layers meets the preset merging condition, then the multiple data files within that sub-layer are merged, including: S410: Count the number of data files and the total size of data files in any sub-layer.

[0055] In one possible implementation, an independent file index is maintained for each sub-layer, recording metadata such as file identifier, individual file size, and key value range for all data files within that sub-layer. Whenever a new file is written to the sub-layer, or an old file is marked as recyclable and removed, the total number of files and total capacity of the sub-layer are updated synchronously, achieving real-time updating of statistical information. This statistics are presented from two dimensions: the number of data files directly determines the number of files that need to be traversed when querying the sub-layer, and is directly related to read latency; the total size of the data files reflects the storage load level of the sub-layer, determining the data processing volume and disk I / O overhead of subsequent merge operations. Through lightweight statistics at the metadata level, the operating status of the sub-layer can be grasped with extremely low operating overhead, providing a reliable basis for merge scheduling.

[0056] S420. If the number of data files is greater than or equal to a preset quantity threshold, and / or the total size of the data files is greater than or equal to a preset size threshold, then multiple data files within any sub-layer are merged.

[0057] The preset size threshold is the maximum total size of data files that any sub-layer can accommodate, which is used to limit the maximum amount of data that the sub-layer can carry and avoid the sub-layer storage overload leading to a sudden increase in IO overhead for subsequent merge operations; the preset quantity threshold is the maximum number of data files that any sub-layer can accommodate, which is used to limit the maximum number of files that the sub-layer is allowed to exist and prevent excessive file fragmentation from causing excessively long read paths.

[0058] In one possible implementation, a merge operation is initiated if the number of data files is greater than or equal to a preset threshold, and / or the total size of the data files is greater than or equal to a preset size threshold. The use of a "greater than or equal to" threshold ensures that the reorganization is performed immediately when the sub-layer's state just reaches the threshold, preventing further exceedances in file number or size. After triggering the merge, the system selects the target data file set within any sub-layer, performs multi-way merge sorting, retains the latest valid data using version numbers or timestamps, removes expired and deleted data, and generates a new, integrated data file. This reduces the number of files within the sub-layer, compresses data redundancy, and brings the sub-layer back to a reasonable operating state.

[0059] For example, the merging and promotion methods of sub-layers within the write layer can adopt a hybrid strategy based on file size, file quantity, or a combination of both: The strategy based on the size of the ordered string table uses the single file size of the initial sub-layer as the base size S_base, and the target file size corresponding to the i-th sub-layer satisfies the above formula (1). When the size of the new file generated by merging within the sub-layer reaches the corresponding target size (i.e., the preset size threshold), it is promoted to the next sub-layer with a larger number. The strategy based on the number of ordered string tables presets a file quantity threshold (i.e., a preset quantity threshold) for each sub-layer. When the number of files within the sub-layer reaches the corresponding preset file quantity threshold, the merging operation of that sub-layer is triggered. The preset quantity threshold of the initial sub-layer is relatively small, and the preset quantity threshold gradually increases with the sub-layer number to balance the merging execution frequency and the file organization effect. The hybrid strategy combines the two dimensions of file size and file quantity. When either condition is met, the merging operation can be triggered, taking into account both the degree of file fragmentation and the storage scale.

[0060] The data merging method provided in this application counts the number of data files and the total size of data files within any sub-layer in real time. When the number of data files is greater than or equal to a preset threshold, or the total size of data files is greater than or equal to a preset size threshold, a merging operation is triggered on multiple data files within any sub-layer. The preset size threshold is the maximum total data capacity that the sub-layer can accommodate, and the preset number threshold is the maximum number of files that the sub-layer can accommodate. This dual-dimensional threshold triggering method can simultaneously control the file fragmentation level and the total storage scale of the sub-layer, avoiding both excessive file counts leading to increased read traversal overhead and excessively large merge I / O peaks caused by exceeding the total data limit of the sub-layer, thus achieving a dynamic balance between read performance and merging overhead.

[0061] Optionally, the above method for merging multiple data files within any sub-layer includes: Lock multiple data files involved in merging within any sub-layer.

[0062] One possible implementation involves locking multiple data files participating in the merging process within any sub-layer. This is achieved at the individual data file level through file status markers in the sub-layer's metadata, rather than applying a global lock to the entire sub-layer. The system sets a status marker for each selected data file participating in the merging process, applying shared read and exclusive write lock controls: locked files can still respond to data query requests normally, ensuring that upper-layer read operations are not blocked; however, they are prohibited from being repeatedly selected by other merging tasks, from being accidentally deleted by the garbage collection module, and from any modification of their content. Other files in the same layer that are not selected are unaffected by the lock, and the system can still append newly generated data files to this sub-layer. These new files are unlocked by default and will not be included in the input range for this merging. This fine-grained locking method ensures the stability and consistency of the merging input data while maximizing the preservation of the sub-layer's concurrent read and write capabilities, avoiding any impact on business traffic caused by the merging operation.

[0063] Sort multiple key-value pairs from multiple locked data files to obtain sorted data.

[0064] In one possible implementation, multiple key-value pairs from multiple locked data files are sorted to obtain sorted data. A multi-way merge sort algorithm is used, which aligns with the inherent ordered key property of an ordered string table. The system creates an independent sequential iterator for each locked data file. Each iterator reads key-value pairs in ascending order from the beginning of the corresponding file. The multiple iterators proceed in parallel, selecting the entry with the smallest key value from all iterators at each step and outputting it to the result sequence. Version conflict handling and data cleaning are performed simultaneously during the merge sort process: for multiple records with the same key, the timestamp or version number of each key-value pair is compared, retaining the latest valid record and discarding expired redundant data; simultaneously, tombstone entries marked for deletion are filtered out to prevent invalid data from occupying storage space. Finally, a complete key-value sequence is output, globally sorted in ascending order, with no duplicate valid keys and no invalid deletion markers, providing an ordered data source for subsequent file generation.

[0065] The sorted data is split according to a preset size threshold to generate at least one new data file.

[0066] The segmentation process employs sequential writing and threshold-based segmentation.

[0067] In one possible implementation, starting from the beginning of the ordered key-value sequence, key-value pairs are sequentially written to the current new file, while the amount of data written to the current file is counted in real time. When the amount of data written is greater than or equal to the target file size corresponding to the sub-layer (i.e., the preset size threshold), the current file is closed, a new blank data file is created, and the remaining ordered data is written until all key-value pairs are written. If the total size of the sorted data does not reach the target size of a single file (i.e., the preset size threshold), only a new data file is generated. After the partitioning is completed, the key-value pairs within each new file are continuous and ordered, the key-value ranges between adjacent files do not overlap, and the file size matches the sub-layer specifications. This facilitates subsequent promotion management between sub-layers and also helps optimize the disk I / O efficiency of subsequent queries.

[0068] For example, taking the L0-1 sub-layer of the write layer as any sub-layer, there are 3 ordered string table data files that meet the merging conditions within this sub-layer. The system first applies fine-grained locking to each of the 3 data files participating in the merging, marking them as being merged. This allows normal response to read requests but prohibits them from being selected by other merging tasks or being garbage collected. Subsequently, the system starts an independent key iterator for each locked data file, performs multi-way merge sorting, retains the latest valid data with the same key based on the version number attached to the key-value pair, and filters and deletes tombstone markers to obtain complete ordered data arranged in ascending order of keys. Finally, the sorted data is sequentially written and split according to the single file target size preset by the L0-1 sub-layer. The total data volume after merging does not exceed the single file target size, and finally, a new integrated ordered string table data file is generated.

[0069] The data merging method provided in this application performs fine-grained locking on multiple target data files participating in the merge within any sub-layer, ensuring concurrent reading while avoiding merged data conflicts. Subsequently, it reads key-value pairs from all locked files and performs multi-way merge sorting, combining version numbers or timestamps to perform redundancy removal and invalid data filtering, resulting in a globally ordered key-value sequence. Finally, the sorted data is split according to the target file size corresponding to that sub-layer, generating at least one new data file. Fine-grained locking balances merge consistency with business concurrency capabilities, multi-way merging achieves the integration and version unification of multi-file data, and splitting by target size ensures that the granularity of the output file matches the sub-layer specifications, providing support for subsequent gradual upgrades and read efficiency optimization.

[0070] Optionally, the above method sorts multiple key-value pairs in multiple locked data files to obtain sorted data, including: For multiple key-value pairs in multiple locked data files, the keys are compared sequentially in the order they are pressed.

[0071] In one possible implementation, multiple key-value pairs in multiple locked data files are compared sequentially by key. This is achieved using the inherent ascending key order of the ordered string table file itself, employing a multi-way merge iterator approach, eliminating the need to load all data into memory and perform a full sort. The system allocates an independent sequential iterator to each locked data file, with each iterator initially pointing to the first key-value pair in its corresponding file. During the merge process, all iterators advance synchronously in ascending key order. In each round, only the key value currently pointed to by each iterator is compared, selecting the record with the smallest key as the current output item, and shifting the corresponding iterator one position to the right. This sequential comparison method allows for the merging of multiple file data in linear time complexity, with memory overhead proportional only to the number of files involved in the merge, significantly reducing memory usage and computational overhead, thus meeting the low-overhead requirements of high-concurrency storage systems.

[0072] If multiple key-value pairs with the same key value exist, the key-value pair with the latest timestamp or the largest version number is selected as the valid data based on the timestamp or version number carried by each key-value pair.

[0073] In one possible implementation, if multiple key-value pairs with the same key exist, valid data is filtered based on timestamps or version numbers. This addresses the key duplication and multiple versioning issues caused by append-only writes in a log structure merging tree architecture. In LSM tree writes, data updates and deletions are implemented by appending new records, without directly modifying historical files. Therefore, the same key may have multiple historical version records in different data files. Each key-value pair is accompanied by a unique timestamp or a monotonically increasing version number to identify the order in which data was written. When multiple records with the same key are found during merge comparison, the system compares the timestamps or version numbers, retaining only the record with the latest timestamp and the largest version number as valid data, discarding the remaining historical versions. Simultaneously, tombstone records marked for deletion can be filtered, thus achieving data deduplication and redundancy cleanup, ensuring data consistency and accuracy, and reducing storage space occupied by invalid data.

[0074] The valid data and key-value pairs without duplicate keys are arranged by key order to generate sorted data.

[0075] In one possible implementation, sorting the valid data and key-value pairs (those without duplicates) into sorted data is the process of integrating the scattered records from multiple merge outputs into a continuous, ordered, and complete dataset. After the aforementioned key comparison and version filtering, all unique key-value pairs and the filtered valid data are output sequentially in ascending key order, forming a globally unique, continuously ordered key-value sequence. This sequence contains no duplicate keys, and each record represents the latest valid version. This sorted data, as the output of the merge stage, can be directly used to subsequently split the data into new files according to the target size, ensuring that the generated new files have ordered key values ​​and no redundancy.

[0076] The data merging method provided in this application performs multi-way merge comparisons on key-value pairs in multiple locked data files, comparing the key-value pairs at their current positions in each data file in ascending order of keys. When multiple key-value pairs with the same key exist, the method filters based on the timestamp or version number carried by each key-value pair, selecting the record with the latest timestamp or the largest version number as valid data, thus completing the deduplication of redundant data and resolving version conflicts. Finally, the filtered valid data and the unique key-value pairs are arranged in the same key order to generate globally ordered sorted data. This process relies on the inherent key-order characteristic of the data files to achieve linear time merging, while ensuring data consistency through versioning, avoiding duplicate key-value pairs from occupying storage space, and providing an ordered data source for the subsequent generation of standard-granularity data files.

[0077] Optionally, the sorted data is split according to a preset size threshold to generate at least one new data file, including: Write the sorted data into the keys in the correct order, and then calculate the size of the data already written.

[0078] In one possible implementation, the sorted data is written sequentially by key, and the size of the written data is counted. This pure sequential write operation leverages the global ascending key order of the sorted data. Starting from the beginning of the sorted key-value sequence, the system writes each key-value pair, along with its associated timestamp, version number, and other metadata, to a newly created file on disk. The write process is a continuous sequential I / O, requiring no random addressing and resulting in extremely low disk I / O overhead. During the write process, the cumulative byte size of the written data is simultaneously and in real-time calculated by summing the key length, value length, and metadata length of each key-value pair. This eliminates the need for additional file traversal, making the statistical overhead negligible.

[0079] If the size of the written data is greater than or equal to the preset size threshold, or if all the sorted data has been written, then the current write operation ends and a new data file is generated from the written data.

[0080] In one possible implementation, if the size of the written data is greater than or equal to a preset size threshold, or if all sorted data has been written, the current write operation ends and a new data file is generated. The preset size threshold is the target size of a single file in the corresponding sub-layer, which satisfies the above formula (1). When the amount of written data reaches or exceeds the target file size, splitting is actively triggered to avoid exceeding the size limit of a single file; or when all sorted data has been written, the write operation ends naturally. The judgment boundary uses "greater than or equal to" to ensure that the file size does not exceed the target specification. After the file is generated, the system will supplement its file metadata, including key value range, file size, version information, etc., and register it in the file index of the corresponding sub-layer.

[0081] If not all the sorted data is written, the remaining unwritten data will be treated as new data to be split, and the operation of writing and counting the size of the written data will continue until all the sorted data is written, resulting in at least one new data file.

[0082] In one possible implementation, if not all sorted data is written, the remaining unwritten data is treated as new data to be split, and writing and statistical operations continue until all data is processed, resulting in at least one new data file. This process handles large-volume merge results through iterative splitting. After a single split and closing of the current file, if there is still unwritten data remaining in the sorted sequence, the remaining ordered data is used as the new processing object. A new blank file is created, and sequential writing, size statistics, and threshold judgment are performed. Each iteration generates a new file that meets the target size until all key-value data is written. All newly generated files have key-value ranges arranged continuously in ascending order and do not overlap, and all except the last file meet the target file size specifications. When the merged data volume is small, only a single file is generated; when the data volume is large, it is automatically split into multiple standard-granularity files, flexibly adapting to different scales of merge results while maintaining consistency in file granularity, facilitating subsequent hierarchical management and query optimization.

[0083] The data merging method provided in this application writes the globally ordered key-value data obtained after sorting into new files in ascending order of keys, and calculates the cumulative size of the written data in real time. When the amount of written data is greater than or equal to the preset sub-layer target file size, or when all sorted data has been written, the current writing process ends, and the written content is packaged into a complete new data file. If there is still unwritten remaining data, the remaining data is treated as a new sequence to be segmented, and the sequential writing and size calculation operations are repeated until all data has been written, ultimately generating at least one new data file that meets the target granularity. Thus, this application ensures that the output file size matches the sub-layer specifications while ensuring that the key values ​​within each new file are continuous and ordered, and that the key ranges of adjacent files do not overlap. This not only adapts to the progressive promotion management between sub-layers but also effectively reduces the disk I / O overhead of subsequent queries.

[0084] Figure 5 A flowchart illustrating a data merging method provided in this application embodiment. Figure 3 .like Figure 5 As shown, the above method also includes: S510, Mark the multiple data files that have participated in the merging with a recycling identifier.

[0085] In one possible implementation, multiple data files that have participated in the merging process are marked with a recycling flag. This process is executed after the merging operation is complete, the newly generated data file has been successfully written to the target level, and metadata registration is finished. The system sets a recycling flag field in the metadata entry of the corresponding data file, marking the file as awaiting recycling. Data files marked with a recycling flag will no longer participate in file filtering for any subsequent merging tasks, nor will they receive new write operations. However, their data content remains intact, and they can still respond to initiated read requests normally. This two-step operation of marking before recycling ensures that the data link is not interrupted during the merging and replacement process, avoids data query gaps caused by directly deleting old files before the new files take effect, and guarantees the continuity of the system's read and write services.

[0086] S520. When the reference count of the data file marked for recycling is zero, delete the data file marked for recycling.

[0087] The reference count is used to record the number of read operations currently accessing the data file marked with a recycling identifier.

[0088] In one possible implementation, deletion is performed when the reference count of a data file marked for reclamation reaches zero. The reference count, maintained by the system's file access management module, records the number of read operations currently accessing the file. Each time a read request initiates access to the data file, the reference count is incremented; when a read operation ends and file access is terminated, the reference count is decremented. A reference count of zero indicates that no ongoing read operations are accessing the file, and deletion at this point will not trigger access exceptions or data errors. The system will then remove the file's metadata index and release the corresponding disk storage space. This reference counting method achieves safe delayed deletion of old files, ensuring the normal execution of read operations while promptly releasing storage resources, thus balancing system availability and space reclamation efficiency.

[0089] The data merging method provided in this application, after the merging is completed and the new data file is registered and effective, marks all old data files that participated in the merging with a recycling flag, causing them to exit the subsequent merging scheduling and write chain. The system records the number of read operations currently in progress for each data file to be recycled in real time through reference counting. Only when the reference count of a data file drops to zero will it be physically deleted and its space reclaimed. Thus, this application can avoid access anomalies caused by deleting files during read operations, ensuring the continuity and stability of data reading, while releasing disk storage space in a timely manner without affecting business operations, achieving smooth recycling of storage resources.

[0090] Optionally, the above method further includes: The reference count is incremented when a read operation begins accessing a data file marked as reclaimable.

[0091] In one possible implementation, file metadata is used as the carrier, and the reference count is updated through atomic operations. The reference count is a unique field in the metadata of each data file, corresponding one-to-one with the file, used to record the number of read operations currently actively accessing that file. When the system receives a read request and locates the target data file through a key range, if the file is detected to have a recycling flag, the system will atomically increment the reference count of that file before opening the file handle and formally starting the read process. Atomic operations avoid counting races and statistical errors in multi-threaded concurrent read scenarios, ensuring the accuracy of the count. This step declares to the garbage collection module that the file is currently being actively read, prohibiting physical deletion during the read process, avoiding read failures and IO errors caused by the file being removed midway, and ensuring the stability and data integrity of read operations.

[0092] The reference count is decreased when a read operation is completed and the data file marked as reclaimed is no longer referenced.

[0093] In one possible implementation, decrementing the reference count when a read operation is complete and the data file marked for reclamation is no longer referenced is a count rollback step strictly paired with the access increment operation, covering all read termination scenarios. When a read operation is completed, including scenarios where the data file is no longer needed (such as single-point query result completion, normal closure of the range scan iterator, and abnormal read termination), the system atomically decrements the reference count of the corresponding file while releasing the file handle and cleaning up the read context. The increment and decrement of the reference count follow the "one access, one increment / decrement" pairing principle: each read initiation corresponds to one increment, and each read completion corresponds to one decrement, ensuring the count reflects the real-time active access status of the file. Timely decrementing after a read operation prevents old files from remaining unreclaimed due to persistently high reference counts, ensuring timely release of storage space and improving the recycling rate of storage resources.

[0094] When a data file marked for recycling is no longer accessed by read operations, its reference count is zero.

[0095] In one possible implementation, when a data file marked for recycling is no longer accessed by read operations, its reference count becomes zero, indicating that the file has no active references from the business side and has reached a state where it can be safely physically recycled. A reference count of zero is one of the prerequisites for physical deletion during garbage collection: only when a data file simultaneously meets both the conditions of being marked for recycling and having a reference count of zero will the system's garbage collection module physically remove the file from the disk and simultaneously clean up the corresponding file entry in the metadata index. This determination forms a safe recycling process of "marking, counting verification, and physical deletion," ensuring that data files currently being accessed by the business are not accidentally deleted, and that old files no longer used after merging are promptly cleaned up. This achieves efficient reuse of storage space without affecting normal read and write operations.

[0096] The data merging method provided in this application increments the reference count of the corresponding data file when a read operation begins accessing it; decrements the reference count when the read operation is completed and the data file is no longer referenced; and resets the reference count to zero when a data file marked for recycling is no longer accessed by any read operation. Thus, this application, through fine-grained reference counting tracking, ensures that files marked for recycling are safely released when there is no active access, while maintaining data consistency during concurrent reads. This avoids IO errors caused by premature file deletion during the read process and prevents invalid old files from occupying storage space for extended periods, achieving a balance between read stability and storage resource utilization.

[0097] Optionally, the above method merges the data files in the lower-level sub-layer with the data files in the adjacent lower layer of the write layer, including: Retrieve all data files in the underlying sub-layer and record the maximum and minimum keys of key-value pairs in all data files to obtain the first key-value range.

[0098] In one possible implementation, all data files in the bottom sub-layer are retrieved, and key-value boundaries are extracted to obtain the first key-value range. This is achieved through lightweight computation using file metadata, without needing to read the actual data content of the files. The bottom sub-layer is the lowest level of the progressive merging process within the write layer, with the highest degree of data integration, serving as a transition node for the write layer to sink to deeper levels. The system traverses the metadata index of this sub-layer, reads the pre-recorded minimum and maximum key fields in the metadata of each data file, takes the minimum value among all file minimum keys as the left boundary of the range, and takes the maximum value among all file maximum keys as the right boundary of the range, ultimately obtaining the first key-value range that completely covers all data in this sub-layer. This step uses the complete key range of the last sub-layer as the filtering criterion for cross-layer merging, ensuring that the data at the lowest level of the write layer is completely included in the merging process, avoiding data omissions. At the same time, the range calculation based on metadata consumes very few computing resources, requiring no additional disk I / O overhead.

[0099] From each sub-layer before the bottom sub-layer, select data files whose key value range is within the first key value range to obtain the first set of data files to be merged.

[0100] In one possible implementation, to fully collect all new version data related to the merge interval within the write layer, the data versions of sub-layers before the lower-level sub-layers are updated, the file granularity is smaller, and the key interval overlap is higher. The system sequentially traverses the file metadata of these sub-layers, comparing the key value range of each file with the overlap of the first key value range. If there is an intersection between the two key intervals, the corresponding file is included in the first set of data files to be merged. By filtering by key range, only files with overlapping keys are selected for merging, rather than including all files from the upper layer. This significantly reduces the size of the merge input, reduces the reading and writing of invalid data, lowers the IO overhead and write amplification effect of the merge operation, and ensures the integrity of the data versions within the merge range, preventing old data from overwriting new data.

[0101] From the adjacent lower layers of the write layer, select data files whose key value range is within the first key value range and determine them as the second set of data files to be merged.

[0102] In one possible implementation, to match the existing ordered data in the lower layer and ensure global consistency of the merged lower layer data, the adjacent lower layer of the write layer, namely the L1 layer, has non-overlapping key ranges and is globally ordered. The system traverses all data files in this layer through metadata indexes, filtering out files whose key value ranges overlap with the first key value range, forming a second set of data files to be merged. Since the key ranges of the L1 layer files are naturally non-overlapping, the number of filtered files is controllable, and these files happen to be those with key conflicts with the sinking data in the write layer. Including these files in the merge allows the new merged output files to directly replace the original overlapping range files in the lower layer, maintaining the non-overlapping key range characteristics of the L1 layer files, avoiding key range intersections and data redundancy, and ensuring the efficiency of single-point queries in the lower layer.

[0103] After locking the first set of data files to be merged and the second set of data files to be merged, sorting is performed to generate at least one merged data file.

[0104] In one possible implementation, fine-grained locking is first performed on all selected sets of first and second data files to be merged. A merging status flag is set for each data file, allowing normal response to read requests but preventing repeated selection by other merging tasks and accidental deletion by garbage collection, ensuring the stability of the merged input data. Then, a multi-way merge sort algorithm is used, creating an independent key iterator for each locked file. The current key value of each iterator is compared sequentially in ascending order. For multiple records with the same key, the latest valid data is retained based on the attached timestamp or version number, while tombstone entries marked for deletion are filtered out, resulting in a globally ordered key-value sequence. Finally, the ordered sequence is split according to the target size of the adjacent lower-level single files, generating one or more merged data files with non-overlapping key intervals. The granularity of the output file directly matches the storage specification of the lower level, requiring no secondary processing.

[0105] Write at least one merged data file to the adjacent lower layer of the write layer.

[0106] In one possible implementation, the merged data file is written to the adjacent lower layer of the write layer, completing the orderly sinking and persistent organization of data from the write layer to deeper levels. The system sequentially writes one or more generated merged data files to the storage area of ​​the adjacent lower layer, synchronously updating the metadata index of that layer and registering information such as the file identifier, key-value range, and file size of the new file. After the new file is written and the metadata takes effect, the locking status of all old files that participated in the merge is released, and the old files are marked with a recycling flag, waiting for their reference count to reach zero before being physically deleted by the garbage collection module. Through this step, the data in the write layer, after gradual organization, is orderly settled to deeper levels, gradually realizing the hierarchical solidification of data, while maintaining low fragmentation and global order of lower-level files, ensuring the overall read performance and storage resource utilization efficiency of the system.

[0107] The data merging method provided in this application first obtains all data files in the bottom sub-layer of the write layer, extracts the maximum and minimum keys of each file's key-value pairs, and integrates them to obtain a first key-value range covering all data in that sub-layer. Then, in each sub-layer before the last sub-layer within the write layer, data files whose key-value ranges overlap with the first key-value range are selected to form a first set of data files to be merged. Simultaneously, in the adjacent lower layers of the write layer, data files whose key-value ranges fall within the first key-value range are selected to form a second set of data files to be merged. After performing fine-grained locking on all data files in the first and second sets of data files to be merged, multi-way merge sorting and redundancy cleanup are performed to generate at least one merged data file. Finally, at least one merged data file is written to the adjacent lower layer of the write layer. Therefore, this application uses the key range of the last sub-layer as a basis for filtering and merging inputs, merging only files with overlapping key ranges, avoiding the high IO overhead and write amplification caused by full-level merging, while ensuring the orderly sinking of the data after the write layer to the lower layers, guaranteeing the global orderliness of the lower-level files.

[0108] To clearly understand the solution of this application, this application also provides a specific embodiment of the data merging method, as described below: Example 1: Merging sub-layers within the write layer: When the number of files in the L0-i sub-layer reaches the corresponding quantity threshold, or the total file size reaches the corresponding capacity threshold S_level_i, the internal merge operation of that sub-layer is triggered. The process is as follows: First, fine-grained locking is performed on all ordered string table files participating in this merge within that sub-layer. Under the locking state, files can be read and accessed but are prohibited from being written or modified, and are prohibited from being repeatedly selected by other merge tasks. During the merge, the sub-layer can still append new ordered string table files, and the newly written files are not included in the scope of this lock. The process begins by merging the input; then, key-value pairs from all locked files are read and loaded into memory, and multi-way merge sort is performed. During the merge, conflicts are handled based on the timestamp or version number carried by each key-value pair, retaining the latest valid data and discarding expired redundant data and tombstone data marked for deletion. Next, the globally ordered key-value sequence obtained by merging is split according to the target file size, generating one or more new ordered string table files with continuous and non-overlapping key ranges. The generated new files are then written to the L0-(i+1) sub-level. If the current i is the last sub-level number of the writing layer (e.g., i = K-1), the new file is kept in the original last sub-level, waiting for subsequent cross-level merges. Simultaneously, all existing ordered string table files participating in this merge are marked as recyclable, and the active access status is determined based on the reference counters corresponding to each file. Physical deletion is performed after all ongoing query operations are completed and the reference count reaches zero. Finally, the locking status of all files participating in the merge within this sub-level is released, and the merge operation within this sub-level is completed.

[0109] For example, the write layer L0 is divided into three sub-layers: L0-0, L0-1, and L0-2. The initial base file size of the sub-layer L0-0 is preset to S_base = 16MB. The target file sizes of each sub-layer satisfy the above formula (1), that is, the target size of L0-0 is 16MB, the target size of L0-1 is 32MB, and the target size of L0-2 is 64MB. At the same time, the file number trigger threshold for each sub-layer is set to 4. During continuous data writing, the memory table is flushed to disk after being filled one by one. In the L0-0 sub-layer, four ordered string table files of approximately 16MB each are generated successively: S1, S2, S3, and S4. When the number of files reaches the preset threshold, the internal merging operation of the L0-0 sub-layer is triggered. The merge process is as follows: First, the system performs fine-grained locking on the four files (S1, S2, S3, and S4) involved in the merge. While locked, these files are accessible for reading but cannot be selected by other merge tasks, nor can they be modified or deleted. During the merge, the L0-0 sublayer can still receive the newly generated ordered string table S5, but S5 is not included in this locking and merge process. Then, the system reads all key-value pairs from the four locked files and loads them into memory, performing a multi-way merge sort. During the merge, conflicts are handled based on the version number carried by each key-value pair, retaining the latest valid data and discarding expired redundant data. Based on the tombstone data marked for deletion; after merging, a single integrated new ordered string table file is generated, with a file size of approximately 32MB, reaching the target file size of the L0-1 sublayer. The system adds this new file to the L0-1 sublayer; simultaneously, S1, S2, S3, and S4 are marked as recyclable, and the active read status is monitored based on the reference counters corresponding to each file. After all ongoing query operations are completed and the reference counts return to zero, the physical deletion of the old files is gradually completed; finally, the locking status of S1, S2, S3, and S4 is released, and this merging operation between sublayers within the L0 layer is completed.

[0110] Example 2: Cross-layer merging from write layer L0 to storage layer L1 is triggered by two conditions, and the merging operation is initiated when either condition is met: First, the number of files in the highest sub-layer L0-(K-1) of write layer L0 reaches the corresponding threshold T_(K-1) or the total size reaches the corresponding capacity threshold S_level(K-1), indicating that the incremental reorganization capacity of write layer L0 has reached its limit and the data needs to be merged down to L1 layer; Second, the total size of files in all sub-layers of write layer L0 exceeds the preset total capacity threshold S_total to avoid excessive data accumulation in write layer L0 and increase query overhead.

[0111] For example, taking the scenario where the number of sub-layers K=3 in the write layer as an example, the write layers are L0-0, L0-1, and L0-2 in sequence, where L0-2 is the highest sub-layer. The preset threshold for the number of files in L0-2 is 4, the threshold for the total size is 64MB, and the threshold for the total capacity of the entire L0 layer is 128MB. When the number of files in the L0-2 sub-layer reaches 4, the cross-layer merging from the L0 layer to the L1 layer is triggered. During the merging process, the system first extracts the key-value boundaries of all ordered string tables in the L0-2 sublayer to obtain the overall key range [b, g]. Then, it selects ordered string tables from the L0-0 and L0-1 sublayers whose key-value range falls within the [b, g] interval, and combines them with the files in L0-2 to form the L0-side set to be merged. At the same time, based on this key range, it selects ordered string tables SST-P[f, h] in the L1 layer whose key range overlaps with this set, forming the L1-side candidate set. Subsequently, it performs fine-grained locking on all selected files on both sides, reads all key-value pairs to perform multi-way merge sort, retains the latest valid data based on timestamps, filters out deleted data, generates a merged ordered string table SST_NEW with a key range of [a, h], and writes it to the L1 layer. After the new file is successfully written and visible to external services, it marks all original files participating in this merge as recyclable. It monitors the active read status based on the reference counters corresponding to each file, and gradually completes the physical deletion of old files and releases storage space after all read operations are completed and the reference count is zeroed.

[0112] Example 3: Data query processing follows a hierarchical priority order from memory to disk and from newest to oldest: The query request first accesses the memory table and immutable memory table in memory. If the target key-value pair is found in memory, the result is returned directly. If not found in memory, the query proceeds by traversing sub-layers L0-0 to L0-(K-1) of the write layer L0 in ascending order of sub-layer number. Since the key-value ranges of data files within the same sub-layer overlap, a query requires traversing all relevant files within that sub-layer. However, thanks to the progressive merging and reorganizing mechanism within the write layer, the larger the sub-layer number, the higher the degree of file integration, the larger the single file size, and the fewer the total number of files within the sub-layer. This reduces the number of files that need to be traversed for the query, effectively improving read efficiency. If the target data is not found in the entire write layer L0, the query continues to the first layer and deeper layers.

[0113] Example 4: The core parameters involved in this invention, such as the number of sub-layers in the write layer and the threshold for the number of files in each sub-layer, can be flexibly adjusted according to the actual workload of the system. For write-intensive workload scenarios, the number of sub-layers K in the write layer and the threshold for the number of files in each sub-layer can be appropriately increased to extend the time that data stays in the write layer, fully utilize the write buffering function of the write layer, and handle high-throughput write requests. For read-intensive workload scenarios, the number of sub-layers in the write layer can be appropriately reduced to accelerate the sinking speed of data files to deeper levels, reduce the read amplification effect caused by file overlap in the write layer, and improve read efficiency. For read-write mixed workload scenarios, a dynamic optimization strategy can be adopted. Based on real-time monitoring of system read-write ratio, file growth rate, and other operating indicators, the number of sub-layers and trigger thresholds can be dynamically adjusted online to achieve a dynamic balance between system write performance and read performance.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0115] Based on the same inventive concept, this application also provides a data merging device. Since the principle of the device in this application is similar to the data merging method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0116] Figure 6 This is a schematic diagram of a data merging device provided in an embodiment of this application. Figure 6 As shown, this is applied to a tiered storage system with multiple storage levels. The tiered storage system includes at least: a write layer, which is divided into multiple sub-layers, the multiple sub-layers including at least: an initial sub-layer and a bottom sub-layer; the data merging device 600 includes: The first writing module 601 is used to write the data file to the initial sublayer; The merging module 602 is used to merge multiple data files in any sub-layer if any sub-layer meets the preset merging conditions, so as to generate at least one new data file. The preset merging conditions are determined based on the number of data files in any sub-layer and / or the total size of the data files. The second writing module 603, if any sub-layer is not the bottom sub-layer, writes at least one new data file to the adjacent sub-layer of any sub-layer; if any sub-layer is the bottom sub-layer, retains at least one new data file in the bottom sub-layer.

[0117] In an optional implementation, the second writing module 603 is further configured to: merge the data files in the lower sub-layer with the data files in the adjacent lower layer of the writing layer in response to the triggering of a preset cross-layer merging condition.

[0118] In one optional implementation, the merging module 602 is specifically used to: count the number of data files and the total size of data files within any sub-layer; if the number of data files is greater than or equal to a preset number threshold, and / or the total size of data files is greater than or equal to a preset size threshold, then merge the multiple data files within any sub-layer, where the preset size threshold is the maximum total size of data files that any sub-layer can accommodate; and the preset number threshold is the maximum number of data files that any sub-layer can accommodate.

[0119] In one optional implementation, the merging module 602 is specifically used to: lock multiple data files participating in the merging within any sub-layer; sort multiple key-value pairs in the locked multiple data files to obtain sorted data; and divide the sorted data according to a preset size threshold to generate at least one new data file.

[0120] In one optional implementation, the merging module 602 is specifically used to: compare the key values ​​of multiple key-value pairs in multiple locked data files in the order of key presses; if multiple key-value pairs with the same key value exist, select the key-value pair with the latest timestamp or the largest version number as valid data based on the timestamp or version number carried by each key-value pair; and arrange the valid data and the key-value pairs without duplicate key values ​​in the order of key presses to generate sorted data.

[0121] In one optional implementation, the merging module 602 is specifically used to: write the sorted data sequentially according to the key order, and count the size of the written data; if the size of the written data is greater than or equal to a preset size threshold, or if all the sorted data has been written, then the current writing operation ends, and a new data file is generated from the written data; if not all the sorted data has been written, then the remaining unwritten data is used as new data to be split, and the operation of writing sequentially and counting the size of the written data continues until all the sorted data has been written, resulting in at least one new data file.

[0122] In an optional implementation, the data merging device 600 is further configured to: mark the multiple data files that have participated in the merging with a recycling identifier; and delete the data file marked with a recycling identifier when the reference count of the data file marked with a recycling identifier is zero, wherein the reference count is used to record the number of read operations currently accessing the data file marked with a recycling identifier.

[0123] In an optional implementation, the data merging device 600 is further configured to: increment the reference count when a read operation begins accessing a data file marked with a recycling identifier; decrement the reference count when the read operation is completed and the data file marked with a recycling identifier is no longer referenced; and set the reference count to zero when the data file marked with a recycling identifier is no longer accessed by read operations.

[0124] In one optional implementation, the merging module 602 is specifically configured to: obtain all data files in the underlying sub-layer and record the maximum and minimum keys of key-value pairs in all data files to obtain a first key-value range; select data files whose key-value range falls within the first key-value range from each sub-layer arranged before the last sub-layer in the writing layer to obtain a first set of data files to be merged; select data files whose key-value range falls within the first key-value range from the adjacent lower layer of the writing layer to determine a second set of data files to be merged; lock the first set of data files to be merged and the second set of data files to be merged and perform sorting processing to generate at least one merged data file; and write the at least one merged data file to the adjacent lower layer of the writing layer.

[0125] It should be noted that the description of the features in the embodiment corresponding to the data merging device can be found in the relevant description of the embodiment corresponding to the data merging method, and will not be repeated here.

[0126] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0127] Optionally, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the first data merging method embodiment described above when running.

[0128] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0129] Optionally, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the first data merging method embodiment described above.

[0130] Optionally, embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the first data merging method embodiment.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The data merging method and electronic device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and its concept. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data merging method, characterized in that, A method applicable to a tiered storage system with multiple storage levels, the tiered storage system comprising at least: a write layer, the write layer being divided into multiple sub-layers, the multiple sub-layers comprising at least: an initial sub-layer and a bottom sub-layer; the method comprising: Write the data file into the initial sublayer; If any of the multiple sub-layers meets the preset merging condition, then the multiple data files in the sub-layer are merged to generate at least one new data file. The preset merging condition is determined based on the number of data files in the sub-layer and / or the total size of the data files. If any of the sub-layers is not the bottom sub-layer, then the at least one new data file is written to the adjacent sub-layer of any of the sub-layers; If any of the sub-layers is the bottom sub-layer, then the at least one new data file is retained in the bottom sub-layer.

2. The method according to claim 1, characterized in that, The method further includes, after retaining the at least one new data file in the underlying sub-layer: In response to the triggering of a preset cross-layer merging condition, the data files in the lower-level sub-layer are merged with the data files in the adjacent lower layer of the writing layer.

3. The method according to claim 1, characterized in that, If any one of the multiple sub-layers meets the preset merging condition, then the multiple data files within that sub-layer are merged, including: Count the number of data files and the total size of the data files within any sub-layer; If the number of data files is greater than or equal to a preset number threshold, and / or the total size of the data files is greater than or equal to a preset size threshold, then multiple data files within any sub-layer are merged. The preset size threshold is the maximum total size of data files that any sub-layer can accommodate; the preset number threshold is the maximum number of data files that any sub-layer can accommodate.

4. The method according to claim 3, characterized in that, The merging of multiple data files within any sub-layer includes: Lock multiple data files involved in the merging process within any of the sub-layers; Sort multiple key-value pairs from multiple locked data files to obtain sorted data; The sorted data is divided according to the preset size threshold to generate at least one new data file.

5. The method according to claim 4, characterized in that, The process of sorting multiple key-value pairs in multiple locked data files to obtain sorted data includes: For multiple key-value pairs in the locked data files, the key values ​​of the key-value pairs in different data files are compared sequentially according to the order of the keys pressed. If there are multiple key-value pairs with the same key value, the key-value pair with the latest timestamp or the largest version number is selected as the valid data based on the timestamp or version number carried by each key-value pair. The valid data and key-value pairs without duplicate keys are arranged in the order of the keys to generate the sorted data.

6. The method according to claim 4, characterized in that, The step of dividing the sorted data according to the preset size threshold to generate at least one new data file includes: Write the sorted data into the keys in the correct order, and count the size of the data already written. If the size of the written data is greater than or equal to the preset size threshold, or if all the sorted data has been written, then the current writing operation ends and a new data file is generated from the written data. If not all of the sorted data is written, the remaining unwritten data is treated as new data to be split, and the operation of writing and counting the size of the written data is continued until all of the sorted data is written, resulting in at least one new data file.

7. The method according to claim 1, characterized in that, The method further includes: Mark multiple data files that have already participated in the merging process with a recycling flag; When the reference count of a data file marked for recycling reaches zero, the data file marked for recycling is deleted. The reference count is used to record the number of read operations currently accessing the data file marked for recycling.

8. The method according to claim 7, characterized in that, The method further includes: When a read operation begins accessing the data file marked with the recycling identifier, the reference count is incremented; When the read operation is completed and the data file marked with the recycling identifier is no longer referenced, the reference count is decreased; The reference count is zero when the data file marked for recycling is no longer accessed by the read operation.

9. The method according to claim 2, characterized in that, The step of merging the data files in the lower-level sub-layer with the data files in the adjacent lower layer of the write layer includes: Obtain all data files in the underlying sub-layer, and record the maximum and minimum keys of key-value pairs in all data files to obtain the first key-value range; From each sub-layer preceding the bottom sub-layer, select data files whose key value range is within the first key value range to obtain the first set of data files to be merged; From the adjacent lower layer of the write layer, select data files whose key value range is within the first key value range and determine them as the second set of data files to be merged; After locking the first set of data files to be merged and the second set of data files to be merged, sorting is performed to generate at least one merged data file; Write the at least one merged data file to the adjacent lower layer of the write layer.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as claimed in any one of claims 1 to 9.

Images