An optimization method and terminal for file merging

Abstract

This invention provides an optimized method and terminal for file merging. It includes: acquiring small files to be merged, where the file size is below a first preset threshold; sorting the small files based on their data writing order to obtain an ordered file sequence; allocating the small files in the ordered file sequence to multiple merging groups according to a second preset threshold; and for each target small file in a merging group, reading and merging the target small files based on their arrangement order in the ordered file sequence to generate a merged file. This invention achieves sequential reading and zero-fragment merging by sorting according to the writing order and flexibly grouping the entire file, significantly improving reading efficiency, reducing system overhead, and solving the problems of low efficiency and excessive fragmentation in traditional methods.

Description

Technical Field

[0001] This invention relates to the field of Internet products, and in particular to an optimized method and terminal for file merging. Background Technology

[0002] In today's massive big data scenarios, as data volume continues to grow, a large number of small files are constantly generated in the file system. Each small file corresponds to an independent metadata record and occupies corresponding storage resources. When the number of small files reaches a certain scale, the metadata management overhead increases dramatically, severely consuming the underlying storage and computing resources of the database and affecting the overall system performance. To alleviate the above problems, related technologies usually adopt a file merging mechanism, which merges a certain number of small files into a larger file to reduce the total number of files and reduce the metadata load. However, this method has obvious shortcomings in practical applications. For example, the merging process requires cross-range data reading and splitting operations on small files. When the file size distribution is uneven or the slice boundaries are not set properly, new residual small files are easily generated, resulting in an ineffective control of the number of files after merging, and even triggering a secondary merging requirement, which affects the overall merging efficiency. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an optimized method and terminal for file merging, so as to further improve the overall efficiency of file merging.

[0004] An optimization method for file merging, the method comprising: Obtain the small files to be merged, wherein the small files are those whose file size is lower than a first preset threshold; Based on the data writing order of the small files, the small files are sorted to obtain an ordered file sequence; According to the second preset threshold, the small files in the ordered file sequence are sequentially assigned to multiple merge groups; Based on the target small files in each merge group, and based on the arrangement order of the target small files in the ordered file sequence, the target small files are read and merged to generate a merged file.

[0005] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the small files to be merged, wherein the small files are those whose file size is lower than a first preset threshold; Based on the data writing order of the small files, the small files are sorted to obtain an ordered file sequence; According to the second preset threshold, the small files in the ordered file sequence are sequentially assigned to multiple merge groups; Based on the target small files in each merge group, and based on the arrangement order of the target small files in the ordered file sequence, the target small files are read and merged to generate a merged file.

[0006] The beneficial effects of this invention are as follows: By organically integrating a sorting mechanism based on data writing order with a whole-file sequential allocation strategy, it effectively solves the problems of low reading efficiency and file fragmentation caused by fixed threshold cutting in traditional file merging methods. By acquiring small files to be merged and defining files with sizes below a first preset threshold as small files, it accurately identifies storage objects requiring optimization from the merging source. Unlike traditional indiscriminate merging or methods that rely solely on modification time for filtering, this solution uses file size as an objective criterion, avoiding the inclusion of large files in the merging process and avoiding unnecessary I / O overhead, ensuring that computing resources are focused on the small file scenarios that truly need optimization. On this basis, by sorting the data writing order based on small files, an ordered file sequence is obtained, achieving a high degree of consistency between the data layout of the merging task and the physical writing order of the underlying storage medium, effectively solving performance bottlenecks such as high read latency and low throughput caused by frequent head seeks. By allocating small files in the ordered file sequence to multiple merging groups according to a second preset threshold, a flexible grouping mechanism with whole files as the smallest allocation unit is constructed, effectively solving the core problems of low merging efficiency and high system load in scenarios with massive small files. Attached Figure Description

[0007] Figure 1 A flowchart illustrating the steps of an optimized file merging method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an optimized terminal for file merging provided in an embodiment of the present invention; Label Explanation: 1. An optimized terminal for file merging; 2. A processor; 3. A memory. Detailed Implementation

[0008] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0009] Please refer to Figure 1 An optimized method for merging files includes steps 110 to 140.

[0010] Step 110: Obtain the small files to be merged. Small files are those whose size is below a first preset threshold. For example, in the storage layer of a database or file system, scan all data files generated within a preset time window, filter out all files whose size is below the first preset threshold (e.g., 100MB), and use these as the set of small files to be merged. This ensures that the merge task focuses on the small file scenarios that truly need optimization, and avoids including large files in the merge process, which would cause unnecessary I / O overhead.

[0011] Step 120: Sort the small files based on their data writing order to obtain an ordered file sequence. For example, arrange all the small files according to the order in which they were written to obtain an ordered file sequence. This ensures that the data access pattern in the subsequent merging process is highly consistent with the physical writing order of the underlying storage medium, laying the foundation for sequential reading.

[0012] Step 130: Based on the second preset threshold, distribute the small files in the ordered file sequence to multiple merge groups in sequence. For example, based on the set second preset threshold, distribute the small files in the ordered file sequence to multiple merge groups in the order they are arranged. In this way, merge groups are constructed with the entire file as the smallest allocation unit, completely eliminating file splitting behavior at the allocation strategy level.

[0013] Step 140: For each target small file in the merge group, based on the order of the target small files in the ordered file sequence, read and merge the target small files to generate a merged file. For example, for each target small file in the merge group, according to the order of the target small files in the ordered file sequence, read all the data of each target small file consecutively, write the read data sequentially into the target merged file and concatenate the data to generate a merged file.

[0014] As described above, the beneficial effects of this invention are as follows: By using a sorting mechanism based on the data writing order, the data layout of the merging task and the physical writing order of the storage medium are highly unified, transforming random access into sequential access and improving read efficiency and throughput. Simultaneously, a flexible grouping mechanism is constructed using the entire file as the smallest allocation unit, completely eliminating the secondary cutting behavior and residual small file generation problems caused by strict threshold cutting in traditional methods at the allocation strategy level, fundamentally suppressing file fragmentation. Based on this, through end-to-end sequential processing from sorting and grouping to merging, the entire merging process can be completed with only one sequential traversal, significantly reducing CPU computational overhead and metadata memory pressure on files of hundreds of billions in scale.

[0015] Further, step 120 includes steps 121 to 123.

[0016] Step 121: Extract the metadata of each small file. For example, in a file system, each file is associated with a corresponding metadata record, which is stored in the file system's metadata area or a database directory table. By calling the underlying file system's interface or scanning the metadata storage table, the metadata of each file in the set of small files to be merged is obtained one by one.

[0017] Step 122: Determine the write order of small files based on the file write times contained in the metadata. For example, extract the file write time from the metadata of each small file as the basis for determining the write order of the small files.

[0018] Step 123: Sort the small files based on the write order to obtain an ordered file sequence. For example, based on the determined write time, arrange all small files in ascending order of timestamp. For cases with the same timestamp, a secondary sorting rule can be used based on the file inode number or the lexicographical order of the file path name to ensure the uniqueness and determinism of the sorting result. After sorting, an ordered file sequence that strictly follows the order in which data was written is generated, laying the foundation for subsequent sequential allocation and sequential reading.

[0019] As described above, by extracting metadata from small files and analyzing their write times, the accurate restoration of the file's historical write order is achieved. Unlike traditional methods that process data by size, filename, or random order, this solution uses the data write order as the sorting criterion. This ensures that the subsequent merging process fully follows the physical write trajectory of the data on the storage medium, creating the preconditions for maximizing the use of disk prefetching mechanisms and cache locality for sequential reading. This lays the foundation for improving merging efficiency from a data organization perspective.

[0020] Further, step 130 includes steps 131 to 135.

[0021] Step 131: Sequentially traverse the ordered file sequence. For example, starting from the beginning of the ordered file sequence, visit each small file in the sorted order. The traversal process points to the file objects in the sequence one by one, ensuring that each file is processed only once, providing an ordered input stream for subsequent allocation operations.

[0022] Step 132: Calculate the current total size of the current merge group after assigning the first small file encountered to it. For example, initialize the first merge group by making the first small file encountered the first member of the group. Record the size of the file and set it as the cumulative total size of the current merge group. At the same time, write the file's identification information into the member list of the merge group to establish the file's affiliation with the group.

[0023] Step 133: Determine if the current total size exceeds the second preset threshold. For example, obtain the size of the next file to be allocated, i.e., the second smallest file, and calculate the sum of the current total size of the merge group and the size of that file. Compare this sum with the second preset threshold (e.g., 1GB) to determine if it exceeds the threshold range.

[0024] Step 134: If the limit is not exceeded, continue iterating. For example, if the calculated sum is less than or equal to the second preset threshold, it indicates that the current merge group still has remaining capacity, and the system adds the second smallest file to the current merge group. The size of this file is added to the total cumulative size of the current merge group, and the file identifier is added to the member list.

[0025] Step 135: If the limit is exceeded, a new merge group is created as the current merge group, and the next small file is treated as the first file in the new merge group to continue the traversal. For example, if the calculated sum is greater than the second preset threshold, it indicates that the current merge group cannot accommodate the next file. The system immediately terminates the file addition operation in the current merge group and marks it as allocated. Subsequently, the next small file is treated as the first member of the new merge group, that is, it is regarded as the new first small file, a new merge group is created, and the process restarts from step 132 to continue the allocation process of subsequent files.

[0026] As described above, a flexible grouping allocation with the entire file as the smallest unit is achieved by combining sequential traversal with dynamic threshold judgment. During the allocation process, only one sequential scan is needed to complete the grouping of all files, making the algorithm simple and efficient. Simultaneously, this mechanism ensures that each small file completely belongs to a single merge group, fundamentally avoiding the cross-group segmentation and residual file generation problems caused by file splitting in traditional methods, thus laying a solid data foundation for subsequent sequential reading and merging.

[0027] Furthermore, step 133 is followed by steps 136 to 138.

[0028] Step 136: If the threshold is exceeded, further determine whether the first merge group was initially empty before adding the second small file. For example, when step 133 determines that adding the second small file will exceed the second preset threshold, the system first backtracks to check the state of the current merge group: whether the group already contained any files before attempting to add the second small file. Specifically, check whether the member list of the current merge group is empty, or whether the cumulative total size is zero. This determination is used to distinguish between two different boundary scenarios: one is the normal exceedance situation where the current group already contains several files, and the other is the special case where the current group has not yet contained any files, i.e., the first file itself has already exceeded the threshold.

[0029] Step 137: If the first merge group is not initially empty, the allocation of the first merge group ends, and a second merge group is created using the second small file as the starting file. For example, if the current first merge group already contains files (i.e., it is not empty), and adding the next small file would exceed the threshold, it indicates that the current group has reached its capacity limit. At this time, the system marks the current merge group as allocated, closes the member addition operation for this group. Subsequently, the second small file is used as the first member of the new merge group, and a new merge group is initialized using it as the starting file, and the subsequent traversal and allocation process continues.

[0030] Step 138: If the first merge group is initially empty, the second small file is assigned to the first merge group, and the second preset threshold is used as the upper limit of the first merge group's capacity. For example, if the current first merge group is empty (i.e., no files have been assigned yet), and the size of the second small file to be assigned already exceeds the second preset threshold, the usual "create a new group if it exceeds the limit" logic cannot be used, because the newly created combination will still face the same over-limit file. For this boundary case, the system uses special handling: it forcibly assigns the over-limit file to the current empty group, making it a separate merge group, and uses the second preset threshold as the upper limit identifier for this combination's capacity. This means that although the combination contains only one file, its size exceeds the regular threshold, and the system treats it as a special case, no longer attempting to merge it with other files.

[0031] As described above, by introducing a mechanism to determine the initial empty group, the boundary condition handling in the file allocation process is improved. Based on the conventional "create a new group if the limit is exceeded" logic, a compatible handling mechanism is added for the special scenario where the first file exceeds the limit, giving the allocation strategy complete exception coverage capabilities. This mechanism ensures that regardless of the file size distribution, each file can be reasonably allocated to a merge group, guaranteeing both the robustness of the allocation process and maintaining the integrity principle of the entire file allocation, providing a complete data foundation for subsequent merge operations.

[0032] Furthermore, step 130 includes step 139.

[0033] Step 139: Distribute the small files in the ordered file sequence into multiple merge groups in sequence. The total data size of each merge group at the end of the distribution falls within a preset fluctuation range based on a second preset threshold. For example, after allocating all small files, the system calculates the total data size of each generated merge group. The final size of these merge groups is not strictly equal to the second preset threshold, but rather fluctuates naturally within a certain range around that threshold. Specifically, a valid grouping result is accepted when the cumulative total size of a group of files reaches or slightly exceeds the threshold after adding the last file, or is slightly lower than the threshold due to file size distribution. The preset fluctuation range can be 10% or 20% above or below the threshold, with the specific value pre-configured based on the actual application scenario and file size distribution characteristics. For example, if the second preset threshold is 1GB, the preset fluctuation range can be set to 900MB to 1100MB, allowing the merge group size to fluctuate within this range without file splitting or forced adjustment.

[0034] As described above, by introducing the concept of a preset floating range, the traditional strictly fixed-size merging is transformed into flexible range merging. This mechanism fully embodies the core idea that the final generated files do not necessarily have to be the same size, and can have some size fluctuations or range adjustments. Compared to the rigid constraint of traditional methods requiring merged files to be strictly equal to the target size, this scheme allows the merged group size to float naturally within a certain range, thus eliminating the need for file splitting during the allocation process and fundamentally avoiding the data segmentation and residual small file problems caused by forcibly aligning thresholds. At the same time, this mechanism maintains the overall controllability of the merged file size, preventing excessive fluctuations that could lead to files that are too large or too small, while providing the necessary flexibility for the allocation process, improving processing efficiency while ensuring merging effectiveness.

[0035] Further, step 140 includes steps 141 and 142.

[0036] Step 141: Create an independent processing thread for each merge group. For example, after completing the grouping and allocation of all small files, the system obtains multiple independent merge groups. To fully utilize the parallel computing capabilities of multi-core processors, the system allocates an independent processing thread to each merge group. Thread pool technology can be used to manage and schedule these threads, avoiding the overhead of frequently creating and destroying threads. Each processing thread is independently responsible for all data reading and merging operations of a merge group. The threads do not interfere with each other, and there is no need for lock synchronization or resource contention, thereby achieving parallel processing of the merging task.

[0037] Step 142: Within each processing thread, the contents of the small files in the merge group are read sequentially according to their order in the ordered file sequence, and the data is concatenated to merge the small file data. For example, within each independent processing thread, a list of all small files in the merge group is first obtained, which strictly follows their original order in the ordered file sequence. Subsequently, the thread opens each small file in this order, reading its contents completely into the memory buffer using a sequential reading method. After reading each file, its data is immediately appended to the end of the target merged file, achieving continuous data concatenation. The entire process maintains a linear flow of file opening, reading, writing, and closing, ensuring that the data writing order on the disk is completely consistent with the reading order, maximizing the performance advantages of sequential I / O. After reading is complete, all original small files are closed, and a complete merged file containing all the data in the merge group is finally generated.

[0038] As described above, a significant improvement in merging efficiency is achieved through a dual mechanism of multi-threaded parallel processing and sequential reading within each merge group. On one hand, parallel execution between different merge groups fully utilizes multi-core computing resources, drastically shortening the overall merging task completion time. On the other hand, sequential reading within each merge group ensures a high degree of compatibility between the data access pattern and the physical characteristics of the storage medium, fully leveraging the effectiveness of disk prefetching and cache locality. Through the organic combination of thread-level parallelism and I / O-level sequential processing, this solution maximizes merging throughput while ensuring data integrity, effectively solving the merging performance bottleneck problem in scenarios with massive amounts of small files.

[0039] Further, step 110 includes step 111.

[0040] Step 111: In the storage layer of the database or file system, scan all small files generated within a preset time window whose file size is below a first preset threshold, and compile all small files into a set to be merged. For example, the system initiates the small file scanning process based on preset merge trigger conditions, such as scheduled tasks, file quantity threshold triggers, or periods of low system load. The scan scope is limited to specific directories in the file system, data tablespaces, or storage partitions in the database. During the scan, the system first filters files generated within a preset time window, such as the last 24 hours or the last 7 days, based on their creation time or last modification time. Subsequently, the files within the time window are filtered by size, retaining only files whose size is below the first preset threshold (e.g., 100MB). Through dual filtering by time and size, the system finally obtains a set of small files focused on recent generation to be merged. Each file in this set meets the definition of a small file and has a similar generation time, facilitating subsequent sorting and merging processes. After the scan is completed, the system persists the file list or passes it to the next processing stage as input data for subsequent steps.

[0041] As described above, the system achieves accurate identification and efficient collection of small files to be merged through a dual mechanism of time window filtering and size threshold determination. On one hand, the introduction of a preset time window ensures that the merging task focuses on recently generated files, avoiding the additional overhead of frequently scanning the entire historical data set, while also adhering to the principles of data localization and timeliness. On the other hand, the determination of the first preset threshold filters out large files that do not need to be merged from the source, concentrating computing resources on small file scenarios that truly require optimization. Through this dual filtering strategy, the system can construct a high-quality, high-merging-value set of files to be processed at minimal cost, laying a solid data foundation for subsequent sorting, allocation, and merging, significantly improving the overall return on investment of the merging process.

[0042] Furthermore, it also includes 143.

[0043] Step 143: The processing threads corresponding to different merge groups execute the operations of reading file content and concatenating data in parallel. For example, after creating an independent processing thread for each merge group in step 141, the system starts all processing threads simultaneously through a thread scheduling mechanism. Each thread is independent and does not block the others, executing concurrently on its assigned computing core. While thread A is reading the first small file in its assigned merge group, thread B may be writing data to its target merge file, and thread C is performing data concatenation. The system achieves true parallel execution of multiple merge group processing tasks through the scheduling strategy of the operating system or thread pool, fully utilizing the parallel computing capabilities of multi-core CPUs. During this process, each thread only accesses file resources within its own merge group, eliminating cross-thread resource contention and lock waiting overhead, ensuring maximum efficiency of parallel execution. Once all processing threads have completed the reading, concatenation, and writing operations for their respective merge groups, the system reclaims thread resources, completing the entire merge process.

[0044] As described above, by implementing parallel execution of processing threads for different merge groups, the originally serial merge task is transformed into a parallel processing mode, improving the overall merge efficiency. In a multi-core processor architecture, the parallel mechanism can linearly scale processing capacity: as the number of merge groups increases, the system can maintain processing timeliness by increasing the number of concurrent threads, avoiding task backlog and processing delays caused by a single thread processing all combinations sequentially. Simultaneously, since each merge group is inherently independent and has no data dependency, parallel execution does not require complex synchronization mechanisms, ensuring data processing correctness while avoiding performance losses caused by lock contention. Through the organic combination of thread-level parallelism and I / O-level sequential processing, this solution maximizes merge throughput while ensuring data integrity, effectively solving the merge performance bottleneck problem in scenarios with massive amounts of small files. The completion time of the merge task is approximately inversely proportional to the number of merge groups, improving the system's horizontal scalability and resource utilization.

[0045] Further, after step 140 reads and merges the small files to generate the merged file, step 1401 is included.

[0046] Step 1401: When generating the merged file, create a single metadata entry for the merged file. For example, after the processing thread completes the data reading and concatenation of all small files within a merge group and writes the merged data completely to the target merged file, the system immediately triggers the metadata creation process. Specifically, before or simultaneously with the file closing operation, the file system or database engine assigns a unique inode node or file identifier to the newly generated merged file and creates a corresponding metadata record. This metadata record contains basic attribute information of the merged file, such as filename, file size, creation time, storage location, and permission information. Unlike the traditional method of retaining independent metadata for each small file, this step integrates the metadata information originally scattered across multiple small files into a single metadata entry, recording only the attributes of the merged file itself, and no longer retaining the independent metadata of the original small files. In a distributed file system or database, this metadata entry will be written to a metadata server, directory table, or data dictionary, and a mapping relationship with the storage area will be established. For systems that support transactions, the creation of metadata and the writing of file data are usually completed in the same transaction context, ensuring the consistency of data and metadata.

[0047] As described above, by creating a single metadata entry when generating the merged file, significant compression of metadata scale is achieved. The original small file set may contain tens of thousands of files, corresponding to the same number of metadata records, consuming a large amount of memory resources; after merging, only one metadata record is needed to manage all data, greatly reducing the memory pressure and indexing overhead of the metadata server. The single metadata entry simplifies the file access path; when reading the merged file subsequently, only one metadata query is needed to locate all data, avoiding the additional latency caused by multiple metadata accesses. By combining data merging with metadata integration, this solution achieves end-to-end optimization from data storage to metadata management, fundamentally solving the metadata bloat problem in small file scenarios.

[0048] The file merging optimization method and terminal described above are applicable to various storage system scenarios that require processing massive amounts of small files, and have significant advantages, especially in application environments such as non-relational databases, distributed file systems, and big data storage engines for merging and managing small files. The following detailed implementation methods illustrate these advantages.

[0049] The following details the application embodiments of this application. This application can apply the above solution to data governance scenarios in large-scale distributed storage systems, especially in the field of big data platforms that require frequent processing of small file merging to optimize storage performance and metadata management. Taking the big data storage system of an internet company as an example, it generates massive amounts of log files daily, many of which are small files less than 100MB in size, leading to excessive NameNode memory pressure and low data reading efficiency. The specific implementation includes the following steps: 1. In the small file collection phase, the system scans all data files generated within the previous 24 hours in the file system's storage layer according to a preset scheduled task. It filters out all small files smaller than 100MB by file size and records their path, size, creation time, and other information as a list of files to be merged. This is equivalent to steps 110 and 111 above.

[0050] 2. During the file sorting stage, the system extracts the metadata of each small file and obtains its file creation timestamp as the basis for the write sequence. Based on the ascending order of the timestamps, all small files to be merged are globally sorted, generating an ordered file sequence that strictly follows the order in which data was written. This sorting result ensures that subsequent read operations can fully adapt to the sequential access characteristics of the disk. This is equivalent to steps 120 and 121 to 123 above.

[0051] 3. During the group allocation phase, the system uses 1GB as the second preset threshold and sequentially traverses the ordered file sequence. The current file is allocated to the current merge group, and its size is accumulated. It is then determined whether adding the next smaller file will exceed 1GB. If not, the file continues to be added; if it exceeds 1GB and the current group is not empty, the current group is closed, and a new group is started with the next smaller file; if the current group is empty and the first file exceeds 1GB, that file is grouped separately, with 1GB as its capacity limit. During the allocation process, the final size of each group is allowed to naturally converge within a preset fluctuation range of 10% above and below 1GB. This is equivalent to steps 130 and 131 to 139 above.

[0052] 4. During the parallel merging phase, the system creates an independent processing thread for each merging group, using a thread pool to schedule multiple groups for parallel processing. Within each thread, files are opened sequentially according to their order in the ordered sequence, all data is read sequentially, and the data is continuously written to the target merged file. Processing threads for different merging groups do not interfere with each other and concurrently execute read and concatenation operations. After merging, the system creates a single metadata entry for each newly generated merged file and marks the original small files as awaiting recycling. Based on the awaiting recycling status marked in the metadata, and after confirming that the merged file has been completely generated and the metadata has been updated, the system starts a background cleanup thread to delete the original small files and their corresponding metadata records, releasing storage space. After cleanup, system monitoring metrics are updated, recording statistics such as the total number of files processed, the number of merged files, and the number of saved metadata entries, for future optimization reference. This is equivalent to steps 140, 141 to 143, and 1401 above.

[0053] Through the above application examples, this solution implements a complete, efficient, and automated file merging and governance system, achieving closed-loop management from small file collection, sorting and grouping, parallel merging to metadata integration and resource reclamation. In actual deployment environments, this solution improves file merging efficiency, reduces metadata memory usage, and decreases disk I / O wait time, effectively solving typical problems such as low read efficiency, severe file fragmentation, and excessive system load caused by fixed threshold segmentation in scenarios with massive amounts of small files. This solution has significant engineering value in improving storage system performance, optimizing resource utilization, and reducing operational costs.

[0054] Please refer to Figure 2 An optimized terminal 1 for file merging includes a memory 3, a processor 2, and a computer program stored on the memory 3 and running on the processor 2. When the processor 2 executes the computer program, it implements the various steps of the optimized file merging method described above.

[0055] In summary, this invention provides an optimized method and terminal for file merging. By constructing a sorting mechanism based on data writing order and a flexible grouping strategy for the entire file, efficient merging processing is achieved in scenarios involving massive amounts of small files.

[0056] By accurately extracting file metadata and deeply analyzing write times, the system reconstructs the physical write trajectory of files on the storage medium, generating an ordered file sequence that strictly follows the write time order. This sorting mechanism allows subsequent read operations to fully adapt to disk prefetching and the principle of cache locality, transforming random access mode into sequential access mode, fundamentally improving data read efficiency and throughput.

[0057] During file allocation, the system employs a flexible grouping mechanism combining sequential traversal and dynamic threshold judgment. By using the entire file as the smallest allocation unit and introducing the concept of a preset floating range, the size of the merge group is allowed to fluctuate naturally within a second preset threshold, avoiding the secondary cutting behavior and residual small file generation problems caused by the strict threshold cutting of traditional methods. Simultaneously, for boundary scenarios where the first file exceeds the limit, the system designs an initial empty group judgment mechanism to ensure that each file can be reasonably allocated to the merge group, achieving complete coverage and robustness of the allocation strategy.

[0058] During the merge execution phase, the system creates an independent processing thread for each merge group, enabling parallel processing between different merge groups. Within each thread, data is read sequentially according to the order in which the files are ordered, and then continuously concatenated and written sequentially to the target merge file. Through the organic combination of thread-level parallelism and I / O-level sequential processing, the system maximizes merge throughput while ensuring data integrity, significantly reducing CPU computational overhead at the scale of hundreds of billions of files.

[0059] Furthermore, during the generation of merged files, the system synchronously creates a corresponding single metadata entry, integrating the metadata information originally scattered across multiple small files into a single record. This mechanism compresses the metadata size, effectively alleviating the memory pressure and indexing overhead of the metadata server, and fundamentally solving the metadata bloat problem in small file scenarios. After the merge is complete, the original small files are marked as awaiting recycling, forming a complete closed loop from data merging to resource reclamation.

[0060] This invention is applicable to various scenarios such as non-relational databases, distributed file systems, and big data storage engines. By constructing a low-overhead, high-throughput, and zero-fragmentation file merging optimization scheme, it effectively solves the core problems of low reading efficiency, severe file fragmentation, and excessive system load caused by fixed threshold cutting in traditional methods, thereby improving the merging efficiency and overall performance of the storage system in scenarios with massive small files.

[0061] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An optimized method for file merging, characterized in that, include: Obtain the small files to be merged, wherein the small files are those whose file size is lower than a first preset threshold; Based on the data writing order of the small files, the small files are sorted to obtain an ordered file sequence; According to the second preset threshold, the small files in the ordered file sequence are sequentially assigned to multiple merge groups; For each target small file in the merge group, based on the arrangement order of the target small files in the ordered file sequence, the target small files are read and merged to generate a merged file.

2. The optimized method for file merging according to claim 1, characterized in that, The step of sorting the small files based on their data writing order to obtain an ordered file sequence includes: Extract the metadata of each of the small files; The writing order of the small files is determined based on the file write time contained in the metadata; Based on the writing order, the small files are sorted to obtain an ordered file sequence.

3. The optimized method for file merging according to claim 1, characterized in that, The step of allocating the small files in the ordered file sequence to multiple merge groups in order according to a second preset threshold includes: The ordered file sequence is traversed sequentially. Calculate the current total size of the current merge group after allocating the first small file encountered in the current traversal to the current merge group; Determine whether the current total size exceeds the second preset threshold; If the number of iterations has not exceeded the limit, continue iterating. If the number of files exceeds the limit, a new merge group is created as the current merge group, and the next small file is used as the first file in the new merge group to continue traversing.

4. The optimized method for file merging according to claim 3, characterized in that, Also includes: If the current total size exceeds the second preset threshold, then determine whether the current merge group was an initial empty group before the first small file was added; If the current merge group is not an initial empty group, then a new merge group is created as the first merge group, the first small file is taken as the first file of the new merge group, and the traversal continues from the first small file. If the current merge group is an initial empty group, then after the first small file is assigned to the second merge group, the traversal continues from the second small file after the first small file.

5. The optimized method for file merging according to claim 1, characterized in that, The step of allocating the small files in the ordered file sequence to multiple merge groups in order according to a second preset threshold includes: The small files in the ordered file sequence are sequentially allocated to multiple merge groups, wherein the total data volume of each merge group at the end of the allocation is within a preset floating range based on the second preset threshold.

6. The optimized method for file merging according to claim 1, characterized in that, The step of reading and merging the target small files in each merge group based on the order of the target small files in the ordered file sequence includes: Create a separate processing thread for each of the merged groups; In each processing thread, the file contents of the target small files in the merge group are read sequentially according to their order in the ordered file sequence, and the data is concatenated to merge the data of the target small files.

7. The optimized method for file merging according to claim 1, characterized in that, The process of obtaining the small files to be merged includes: In the storage layer of the database or file system, scan all small files generated within a preset time window whose file size is lower than the first preset threshold, and use all the small files as a set of small files to be merged.

8. The optimized method for file merging according to claim 6, characterized in that, Also includes: The processing threads corresponding to different merging groups execute the operations of reading file content and concatenating data in parallel.

9. The optimized method for file merging according to claim 1, characterized in that, After reading and merging the small files to generate a merged file, the process includes: When generating the merged file, a single metadata entry is created for the merged file.

10. An optimized terminal for file merging, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the file merging optimization method according to any one of claims 1 to 9.

Images