Small file processing method, device and computer storage medium

By grouping and judging the conditions of small file groups, rewriting tasks are generated and automatically merged, the problem of small files in the data processing engine affecting query performance is solved, and the rewriting efficiency and data processing efficiency are improved.

CN118568061BActive Publication Date: 2024-11-29ZHEJIANG DAHUA TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411049488.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-29
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

When the data processing engine processes data, small files generated by long-term operation of tasks affect query performance. The existing technology is difficult to ensure the speed of small files merging and poor rewriting efficiency.

Method used

By obtaining the file partition of the small file to be rewritten, grouping and determining whether the file group meets the rewrite conditions. If it is satisfied, a rewrite task will be generated and rewrite it. Use multiple operators (IcebergRewriteTaskEmitter, IcebergStreamRewriter, IcebergRewriteFilesCommitter) to automatically merge small files. document.

Benefits of technology

It improves the efficiency of small file rewriting, reduces the impact on query performance, realizes real-time merging of small files, and improves data processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118568061B_ABST
    Figure CN118568061B_ABST
Patent Text Reader

Abstract

The present application proposes a small file processing method, device and computer storage medium, wherein the small file rewriting method comprises: obtaining file partitions of several small files to be rewritten; grouping the small files to be rewritten according to the file partitions to obtain several file groups; judging whether the file group meets the rewriting conditions according to the number of files and / or the file size in each file group; if so, generating a rewriting task for the target file group; and rewriting the target file group based on the rewriting task. In the above manner, each file group is automatically rewritten according to the file situation of the file group, thereby improving the rewriting efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a small file processing method, device and computer storage medium. Background Art

[0002] When the data processing engine processes data, the task usually runs for a long time in the cluster and generates a certain number of small files, which will affect query performance.

[0003] The current method for processing small files is to merge small files regularly through external triggers or scheduled tasks. It is difficult to ensure that the merging speed can meet the speed of small file generation, and the rewriting efficiency is poor. Summary of the invention

[0004] The present application provides a small file processing method, device and computer storage medium.

[0005] In order to solve the above technical problems, the present application proposes a small file processing method, which obtains file partitions of several small files to be rewritten; groups the small files to be rewritten according to the file partitions to obtain several file groups; determines whether the file group meets the rewrite conditions based on the number of files and / or file size in each file group; if so, generates a rewrite task for the target file group; and rewrites the file group that meets the target based on the rewrite task.

[0006] Among them, before obtaining the file partitions of several small files to be rewritten, the small file processing method also includes: obtaining the current write file and its current write ID; judging whether the last write ID of the last write file is the parent of the current write ID; if not, obtaining the deleted files between the current write ID and the last write ID; wherein the small files to be rewritten include the current write file, the last write file, and the deleted files.

[0007] Wherein, judging whether the file group meets the rewrite condition based on the number of files and / or the file size in each file group includes: when the number of files in the file group is greater than or equal to a first preset number, and the total size of the files in the file group is greater than or equal to a preset threshold, determining that the file group meets the rewrite condition; wherein, the preset threshold is determined by the maximum read value of a single task read by the data processing engine.

[0008] Wherein, judging whether the file group satisfies the rewrite condition based on the number of files and / or the file size in each file group includes: when the total size of the files in the file group is less than the preset threshold, and the number of files in the file group is greater than or equal to a second preset number, determining that the file group satisfies the rewrite condition; wherein, the second preset number is greater than the first preset number.

[0009] Wherein, judging whether the file group satisfies the rewrite condition according to the number of files and / or the file size in each file group includes: obtaining the minimum cost value of the data processing engine reading a single file; obtaining the maximum read value of the data processing engine reading a single task; and determining the second preset number based on the ratio of the maximum read value to the minimum cost value.

[0010] Among them, judging whether the file group meets the rewrite condition based on the number of files and / or file size in each file group includes: obtaining the number of waiting submissions for the file group; when the number of waiting submissions is greater than the maximum number of waiting submissions, determining that the file group meets the rewrite condition.

[0011] Among them, the obtaining of file partitions of several small files to be rewritten includes: obtaining a filtering range based on the target file size; filtering target files smaller than the minimum value of the filtering range and target files larger than the maximum value of the filtering range to obtain file partitions of several small files to be rewritten.

[0012] To solve the above technical problems, the present application proposes a small file processing method, which is applied to a small file processing system, wherein the small file processing system includes an original operator, and a first operator and a second operator sequentially connected after the original operator, and the small file processing method includes: the original operator sends a number of small files to be rewritten to the first operator; the first operator obtains file partitions of a number of small files to be rewritten; the first operator groups the small files to be rewritten according to the file partitions to obtain a number of file groups; the first operator determines whether the file group meets the rewrite condition based on the number of files and / or file size in each file group; if so, the first operator generates a rewrite task for the target file group and sends it to the second operator; the second operator rewrites the file group that meets the target based on the rewrite task.

[0013] To solve the above technical problems, the present application proposes a small file processing device, which includes a processor and a memory; a computer program is stored in the memory, and the processor is used to execute the computer program to implement the steps of the above small file processing method.

[0014] In order to solve the above technical problem, the present application proposes a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed, the steps of the above small file processing method are implemented.

[0015] Different from the prior art, the beneficial effects of the present application are: obtaining file partitions of several small files to be rewritten; grouping the small files to be rewritten according to the file partitions to obtain several file groups; judging whether the file group meets the rewriting conditions according to the number of files and / or the file size in each file group; if so, generating a rewriting task for the target file group; and rewriting the target file group based on the rewriting task. In the above manner, each file group is automatically rewritten according to the file situation of the file group, thereby improving the rewriting efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a flowchart of the first embodiment of the small file processing method provided by the present application;

[0018] Figure 2 It is a flow chart of generating and rewriting tasks provided by this application;

[0019] Figure 3 It is a real-time merge schematic diagram provided by this application;

[0020] Figure 4 It is a flowchart of a second embodiment of the small file processing method provided by the present application;

[0021] Figure 5 It is a structural schematic diagram of an embodiment of a small file processing device provided by the present application;

[0022] Figure 6 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The small file processing method of the present application is applied to a small file processing device, wherein the small file processing device of the present application can be a server, or a system composed of a server and a local terminal cooperating with each other. Accordingly, the various parts of the small file processing device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or can be set in the server and the local terminal respectively.

[0025] Further, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the small file processing method of the embodiment of the present application can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0026] The embodiment of the present application takes the application of Flink (distributed data processing engine) and Iceberg (data storage format) real-time data warehouse as an example to automatically merge small files. It should be noted that this method is applicable to other data processing and distributed data processing processes.

[0027] Apache Flink (distributed data processing engine) is an open source, distributed stream processing and batch processing framework designed to handle large-scale and high-speed data streams. It provides an efficient, reliable and scalable way to process and analyze real-time and historical data. Supports multiple data sources and data receivers: Flink can read data from various data sources and write processing results to various data receivers (such as databases, file systems, message queues, etc.).

[0028] Apache Iceberg (distributed data processing engine) is a high-performance format for large analytical tables. Iceberg (data storage format) brings the reliability and simplicity of SQL (database) tables to big data, while enabling multiple data processing engines to safely work with the same table at the same time.

[0029] Real-time tasks usually run for a long time in the cluster. Set the minute level to ensure the timeliness of the data, and generate a certain number of small files for each submission. The shorter the submission interval, the more snapshots are generated. A large number of small files are generated when the stream task is running. This application improves data processing efficiency by providing an automatic merging of small files.

[0030] This application automatically compresses small files while Flink streams into the Iceberg table, and the small file processing device adds three operators after the original operator submits the file operator.

[0031] The first operator is the IcebergRewriteTaskEmitter operator, the second operator is the IcebergStreamRewriter operator, and the third operator is the IcebergRewriteFilesCommitter. For the convenience of expression, the first operator, the second operator, and the third operator are used in the embodiments of the present application. For the execution function of each operator, please refer to the specific embodiments of the present application.

[0032] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of the first embodiment of the small file processing method provided by the present application, Figure 2 This is a flowchart of generating a rewriting task provided by this application.

[0033] like Figure 1 As shown, the specific steps are as follows:

[0034] Step S11: Obtain file partitions of several small files to be rewritten.

[0035] Specifically, the small file processing device adds an automatic rewrite parameter in the table attributes, which is represented by true or false, indicating whether automatic rewriting of small files is enabled. The default value is true, indicating that automatic rewriting is not enabled by default. If enabled, three automatic rewrite-related operators are added to the original operator submitting the file operator in the original operator chain, namely the first operator, the second operator and the third operator.

[0036] In other embodiments of the present application, if automatic rewriting is not enabled, the small file processing device does not merge small files, which is consistent with the process of writing tables by the native data processing engine and will not be repeated here.

[0037] The original operator submits the small file written in this checkpoint when the checkpoint occurs. After the submission is successful, the data is visible. After the submission is successful, the original operator sends the submitted small file and the generated snapshot ID to the first operator.

[0038] The first operator obtains several small files to be rewritten and snapshot IDs sent by the original operator, and obtains storage partition information for each small file.

[0039] In one embodiment of the present application, in order to prevent file loss, the small file rewriting device obtains the current write file and its current write ID; determines whether the previous write ID of the previous write file is the parent of the current write ID; if not, obtains the deleted file between the current write ID and the previous write ID.

[0040] The small files to be rewritten include the currently written file, the last written file, and the deleted file.

[0041] Specifically, the first operator receives the file and snapshot ID sent from the upstream. The received snapshot ID of the submitted result is compared with the snapshot ID received last time. If the snapshot ID received last time is not the parent of the current snapshot ID, it means that a concurrent writer has submitted other snapshots to the table. At this time, all deleted files between the last snapshot received and the current snapshot need to be collected to avoid losing any deleted files during rewriting.

[0042] The first operator recovers the deleted files written by the task from the checkpoint, and collects all deleted files written by other writers between the restored snapshot ID and the latest snapshot ID to avoid losing any deleted files during rewriting.

[0043] Step S12: Group the small files to be rewritten according to the file partitions to obtain a number of file groups.

[0044] Specifically, the small file rewriting device groups the small files to be rewritten according to the file partitions, groups the small files from the same partition into a file group, and obtains a plurality of file groups.

[0045] In one embodiment of the present application, it is necessary to filter the small files to be rewritten, and the small file rewriting device obtains a filtering range based on the target file size; the target files smaller than the minimum value of the filtering range and the target files larger than the maximum value of the filtering range are filtered to obtain file partitions of several small files to be rewritten.

[0046] In a specific embodiment of the present application, the files that are too large or too small and need to be rewritten are filtered out according to the table attributes, and the default values ​​are 5% of the target file size and 180% of the target file size of the table native rewrite parameter default values, respectively.

[0047] Step S13: judging whether the file group meets the rewriting condition according to the number of files and / or the file size in each file group.

[0048] Specifically, the small file rewriting device determines whether the small files in each file group meet the rewriting conditions according to the small file conditions of the file group.

[0049] This application proposes four embodiments for determining whether a small file in a file group meets the rewriting condition, as follows:

[0050] In one embodiment of the present application, when the number of files in a file group is greater than or equal to a first preset number, and the total size of the files in the file group is greater than or equal to a preset threshold, it is determined that the file group meets the rewrite condition; wherein the preset threshold is determined by the maximum read value of a single task read by the data processing engine.

[0051] The first preset number is any real number greater than or equal to 2 and less than the second preset number, and the default value is 2, that is, the file group can be overwritten only when there are at least 2 small files in the file group.

[0052] Furthermore, when the number of files in a file group is greater than or equal to a first preset number, the total file size of each file group is obtained, that is, the file size obtained by adding up all small files. When the total file size meets a preset threshold, the file group meets the rewrite condition.

[0053] The preset threshold is determined by the parameter. The parameter indicates that when the total size of files in the file group reaches this value, a rewrite task will be generated and sent to the downstream operator. Because the data processing engine Flink reads a task in the table and reads files of a maximum total size of a parameter, the default value is 128M, so this parameter is set to a default value of 128M.

[0054] In one embodiment of the present application, when the total size of files in the file group is less than a preset threshold and the number of files in the file group is greater than or equal to a second preset number, it is determined that the file group meets the overwrite condition; wherein the second preset number is greater than the first preset number.

[0055] In one embodiment of the present application, a small file rewriting device obtains the minimum cost value of a data processing engine reading a single file; obtains the maximum read value of a data processing engine reading a single task; and determines the second preset number based on the ratio of the maximum read value to the minimum cost value.

[0056] Since the default value of the minimum cost of a file in a data processing engine reading task is 4M, the default value of this parameter is equal to the default value 128M / default value 4M=32, that is, when the number of files in a file group is 32, the file group will be rewritten.

[0057] In one embodiment of the present application, the small file rewriting device obtains the number of waiting submissions of the file group; when the number of waiting submissions is greater than the maximum number of waiting submissions, it is determined that the file group meets the rewriting condition.

[0058] If after waiting for the number of commits, the file group will be rewritten regardless of whether the number of files in the file group exceeds 2 or the size exceeds 128M.

[0059] If any of the above conditions is met, the small file processing device continues to execute step S14.

[0060] Step S14: Generate a rewriting task for the target file group.

[0061] Specifically, the first operator generates a rewrite task for the file group that meets the condition, and sends the rewrite task to the second operator.

[0062] In the embodiment of the present application, the second operator is an operator that performs a rewriting task, that is, an operator that merges small files in a file group into a large file. In order to improve processing efficiency, multiple second operators can be set for parallel processing.

[0063] Step S15: rewrite the target file group based on the rewrite task.

[0064] Specifically, the second operator rewrites the file group that meets the rewrite condition according to the rewrite task issued by the first operator, that is, merges the small files in the file group that meets the rewrite condition into a large file.

[0065] In one embodiment of the present application, merging is performed at a checkpoint, and when data reaches the second operator, a checkpoint is triggered, and merging is performed at the checkpoint, which does not affect streaming data transmission and does not block the execution of the main link data.

[0066] In the embodiment of the present application, file data is submitted in two stages in the checkpoint. The first stage submits the data file to make the data visible, and the second stage submits the merged file, which does not affect the data visibility and performs an imperceptible merge process on the user side.

[0067] Furthermore, the second operator sends the rewriting result to the third operator, which receives the rewriting result sent by the second operator and saves it in memory. When the next checkpoint comes, the table native API is called to submit the rewriting result. After the submission is completed, the latest file is the merged file. The generated small file address is directly cached in memory, and no additional task is required to scan small files separately.

[0068] This application can merge small files after data is written, and the merging time is high, which can improve the reading performance in the first place and solve the problem of read amplification. The addresses of small files generated in real time are directly cached into the memory, and no additional tasks are required to scan small files separately. The configuration parameters of this proposal are simple. In most scenarios, you only need to enable automatic rewriting. Other parameter values ​​are configured with good default values ​​based on Flink's Iceberg writing principle.

[0069] For details, please see Figure 3-Figure 4 , Figure 3 It is a real-time merge schematic diagram provided by this application; Figure 4 It is a flowchart of the second embodiment of the small file processing method provided by the present application.

[0070] The small file processing method is applied to a small file processing system, the small file processing system includes an original operator, and a first operator and a second operator sequentially connected after the original operator. After inputting data, a task is constructed according to table information and sent to the original operator. The small file processing method includes:

[0071] Step S21: the original operator sends a number of small files to be rewritten to the first operator.

[0072] Step S22: The first operator obtains file partitions of several small files to be rewritten.

[0073] Step S23: The first operator groups the small files to be rewritten according to the file partitions to obtain a number of file groups.

[0074] Step S24: the first operator determines whether the file group meets the rewriting condition according to the number of files and / or the file size in each file group.

[0075] If yes, execute step S25.

[0076] Step S25: The first operator generates a rewriting task for the target file group and sends it to the second operator.

[0077] Step S26: The second operator rewrites the file group that meets the target based on the rewriting task.

[0078] Among them, step S21 to step S26 are the same as step S11 to step S15, and will not be repeated here.

[0079] In order to implement the small file processing method of the above embodiment, the present application also provides a small file processing device. Figure 5 , Figure 5 It is a structural diagram of an embodiment of a small file processing device provided by the present application.

[0080] like Figure 5 As shown, the small file processing device 600 of this embodiment includes a processor 61, a memory 62, an input and output device 63 and a bus 64.

[0081] The processor 61 , the memory 62 , and the input / output device 63 are respectively connected to the bus 64 . The memory 62 stores a computer program, and the processor 61 is used to execute the computer program to implement the small file processing method of the above embodiment.

[0082] In this embodiment, the processor 61 may also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The processor 61 may also be a GPU (Graphics Processing Unit), also known as a display core, a visual processor, or a display chip, which is a microprocessor that is specifically used for image computing on computers, workstations, game consoles, and some mobile devices (such as tablet computers, smart phones, etc.). The purpose of the GPU is to convert and drive the display information required by the computer system, and to provide a line scan signal to the display to control the correct display of the display. It is an important component that connects the display and the computer motherboard. As an important component of the computer host, the graphics card is responsible for outputting display graphics. The general-purpose processor may be a microprocessor or the processor 61 may also be any conventional processor, etc.

[0083] The present application also provides a computer storage medium, such as Figure 6 As shown, the computer storage medium 700 is used to store a computer program 71. When the computer program 71 is executed by a processor, it is used to implement the method described in the embodiment of the small file processing method of the present application.

[0084] The method involved in the embodiment of the small file processing method of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0085] The above description is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A small file processing method, It is characterized in that The small file processing method comprises: Get the file partitions of several small files to be rewritten; Grouping the small files to be rewritten according to the file partitions to obtain a plurality of file groups; Determining whether the file group meets the rewrite condition according to the number of files and / or the file size in each file group; If so, generate a rewrite task for the target file group; Rewriting the target file group based on the rewriting task; The step of judging whether the file group meets the rewriting condition according to the number of files and / or the file size in each file group includes: When the number of files in the file group is greater than or equal to a first preset number, and the total size of the files in the file group is greater than or equal to a preset threshold, it is determined that the file group meets the rewrite condition; wherein the preset threshold is determined by the maximum read value of the data processing engine reading a single task; When the total size of the files in the file group is less than the preset threshold, and the number of files in the file group is greater than or equal to a second preset number, determining that the file group meets the rewrite condition; wherein the second preset number is greater than the first preset number; Get the minimum cost of a data processing engine reading a single file; Obtaining the maximum read value of a single task read by the data processing engine; The second preset number is determined based on a ratio of the maximum read value to the minimum cost value.

2. The small file processing method according to claim 1, It is characterized in that Before obtaining the file partitions of the plurality of small files to be rewritten, the small file processing method further includes: Get the current write file and its current write ID; Determine whether the last written ID of the last written file is the parent of the current written ID; If not, obtain the deleted files between the current write ID and the last write ID; The small files to be rewritten include the currently written file, the last written file, and the deleted file.

3. The small file processing method according to claim 1, It is characterized in that The step of judging whether the file group meets the rewriting condition according to the number of files and / or the file size in each file group includes: Get the number of submissions waiting for the file group; When the number of waiting submissions is greater than the maximum number of waiting submissions, it is determined that the file group meets the rewrite condition.

4. The small file processing method according to claim 1, It is characterized in that The step of obtaining the file partitions of the plurality of small files to be rewritten comprises: Get the filtering range based on the target file size; Filter the target files that are smaller than the minimum value of the filtering range and the target files that are larger than the maximum value of the filtering range to obtain file partitions of several small files to be rewritten.

5. A small file processing method, It is characterized in that The small file processing method is applied to a small file processing system, the small file processing system includes an original operator, and a first operator and a second operator sequentially connected after the original operator, and the small file processing method includes: The original operator sends a number of small files to be rewritten to the first operator; The first operator obtains file partitions of several small files to be rewritten; The first operator groups the small files to be rewritten according to the file partitions to obtain a plurality of file groups; The first operator determines whether the file group meets the rewrite condition according to the number of files and / or the file size in each file group; If yes, the first operator generates a rewrite task for the target file group and sends it to the second operator; The second operator rewrites the target file group based on the rewriting task; When the number of files in the file group is greater than or equal to a first preset number, and the total size of the files in the file group is greater than or equal to a preset threshold, the first operator determines that the file group meets the rewrite condition; wherein the preset threshold is determined by the maximum read value of the data processing engine reading a single task; When the total size of files in the file group is less than the preset threshold, and the number of files in the file group is greater than or equal to a second preset number, the first operator determines that the file group meets the rewrite condition; wherein the second preset number is greater than the first preset number; The first operator obtains the minimum cost of the data processing engine reading a single file; The first operator obtains the maximum read value of the data processing engine reading a single task; The first operator determines the second preset number based on a ratio of the maximum read value to the minimum cost value.

6. A small file processing device, It is characterized in that The small file processing device includes a processor and a memory; the memory stores a computer program, and the processor is used to execute the computer program to implement the steps of the small file processing method as described in any one of claims 1 to 4.

7. A computer storage medium, It is characterized in that The computer storage medium stores a computer program, and when the computer program is executed, the steps of the small file processing method as described in any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Small file merging method and device, electronic equipment and storage medium

    CN116795790A