File merging method and device

Abstract

Disclosed in the present invention are a file merging method and device. The method comprises: acquiring initial data stored in a plurality of data lake files to be merged, wherein the initial data is data having undergone one or more of the following processings: serialization, compression, and coding; directly using the initial data as storage data in a target data lake file, and on the basis of the storage position of the storage data in the target data lake file, obtaining target metadata of the target data lake file; and writing the storage data and the target metadata into the target data lake file to obtain a target data lake file obtained by merging the plurality of data lake files to be merged.

Description

File merging method and device

[0001] This application claims priority to Chinese Patent Application No. 202310919449.7 filed on July 25, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to a file merging method and device. Background Art

[0003] Lake-warehouse integration is a new data management model that merges the differences between data warehouses and data lakes, building the data warehouse on a data lake to form a lake-warehouse system. This system effectively simplifies data infrastructure, improving data storage elasticity and quality while reducing costs and data redundancy. The underlying data in a lake-warehouse system is typically stored in the Hadoop Distributed File System (HDFS) cluster in Parquet format.

[0004] As data volumes and application scenarios continue to expand, HDFS clusters can encounter issues. One of the most common problems is small files. Small files are files significantly smaller than a single data block. Storing large numbers of small files can lead to a variety of problems, including increased metadata, wasted storage space, inefficient data replication, low access efficiency, and increased system management complexity.

[0005] In order to avoid the above problems caused by storing a large number of small files, the small files can be merged. Currently, the efficiency of merging small files is low. Therefore, a solution is urgently needed to solve the above problems.

[0006] Summary of the Invention

[0007] In order to solve or partially solve the above technical problems, the embodiments of the present disclosure provide a file merging and placing device.

[0008] The present disclosure provides a method for merging files, the method comprising:

[0009] Obtaining initial data stored in multiple data lake files to be merged, where the initial data is data that has undergone one or more of serialization, compression, and encoding;

[0010] Directly use the initial data as storage data in the target data lake file;

[0011] Obtaining target metadata of the target data lake file according to a storage location of the stored data in the target data lake file;

[0012] The stored data and the target metadata are written into the target data lake file.

[0013] Optionally, writing the stored data and the target metadata into the target data lake file includes:

[0014] First, the initial data is directly written into the target data lake file, and then the target metadata is written into the target data lake file.

[0015] Optionally, obtaining target metadata of the target data lake file according to a storage location of the stored data in the target data lake file includes:

[0016] After the initial data is directly written into the target data lake file, target metadata of the target data lake file is obtained according to a storage location of the stored data in the target data lake file.

[0017] Optionally, before obtaining the initial data stored in the plurality of data lake files to be merged, the method further includes:

[0018] receiving a target structured query language statement, where the target structured query language statement is used to specify a data lake file to be merged;

[0019] The multiple data lake files to be merged are determined according to the target structured query language statement.

[0020] Optionally, the target structured query language statement further includes a data lake file merging statement, where the data lake file merging statement is used to instruct to merge the multiple data lake files to be merged.

[0021] Optionally, the target structured query language statement further includes a file size setting statement, where the file size setting statement is used to indicate that the size of the target data lake file is a target size, and the size of the generated target data lake file is the target size.

[0022] Optionally, obtaining the initial data stored in the multiple data lake files to be merged includes:

[0023] Traverse the multiple data lake files to be merged, and perform the following operations for each data lake file to be merged:

[0024] Determining, based on the metadata of the data lake file to be merged, the offset and size of the row group in the data lake file to be merged in the data lake file to be merged;

[0025] Determining, based on the metadata of the row group, the offset and size of the column block in the to-be-merged data lake file;

[0026] The data in the column block is obtained as the initial data.

[0027] The present disclosure also provides a file merging device, the device comprising:

[0028] An acquisition unit, configured to acquire initial data stored in multiple data lake files to be merged, wherein the initial data is data that has undergone one or more of serialization, compression, and encoding;

[0029] A first determining unit is configured to directly use the initial data as storage data in a target data lake file;

[0030] a second determining unit, configured to obtain target metadata of the target data lake file according to a storage location of the stored data in the target data lake file;

[0031] A writing unit is configured to write the stored data and the target metadata into the target data lake file.

[0032] Optionally, the writing unit is used to:

[0033] First, the initial data is directly written into the target data lake file, and then the target metadata is written into the target data lake file.

[0034] Optionally, the second determining unit is configured to:

[0035] After the initial data is directly written into the target data lake file, target metadata of the target data lake file is obtained according to a storage location of the stored data in the target data lake file.

[0036] Optionally, the device further includes:

[0037] a receiving unit, configured to receive a target structured query language statement before obtaining the initial data stored in the plurality of data lake files to be merged, wherein the target structured query language statement is used to specify the data lake files to be merged;

[0038] A third determining unit is configured to determine the multiple data lake files to be merged according to the target structured query language statement.

[0039] Optionally, the target structured query language statement further includes a data lake file merging statement, where the data lake file merging statement is used to instruct to merge the multiple data lake files to be merged.

[0040] Optionally, the target structured query language statement further includes a file size setting statement, where the file size setting statement is used to indicate that the size of the target data lake file is a target size, and the size of the generated target data lake file is the target size.

[0041] Optionally, the acquiring unit is configured to:

[0042] Traverse the multiple data lake files to be merged, and perform the following operations for each data lake file to be merged:

[0043] Determining, based on the metadata of the data lake file to be merged, the offset and size of the row group in the data lake file to be merged in the data lake file to be merged;

[0044] Determining, based on the metadata of the row group, the offset and size of the column block in the to-be-merged data lake file;

[0045] The data in the column block is obtained as the initial data.

[0046] The embodiment of the present disclosure further provides a file merging device, the device comprising a processor and a memory;

[0047] The processor is configured to execute instructions stored in the memory, so that the device performs the method as described in any one of the first aspects above.

[0048] An embodiment of the present disclosure further provides a computer-readable storage medium, comprising instructions, wherein the instructions instruct a device to execute the method as described in any one of the above first aspects.

[0049] The embodiment of the present disclosure further provides a computer program product, which, when executed on a computer, enables the computer to execute any one of the methods described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] FIG1 is a flow chart of a method for executing file merging in a conventional technology;

[0052] FIG2 is a schematic diagram of the structure of content stored in a data lake file provided by an embodiment of the present disclosure;

[0053] FIG3 is a schematic diagram of storing an original data table as a data lake file according to an embodiment of the present disclosure;

[0054] FIG4 is a flow chart of a file merging method provided by an embodiment of the present disclosure;

[0055] FIG5 is a schematic diagram of a scenario of a file merging method provided by an embodiment of the present disclosure; and

[0056] FIG6 is a schematic structural diagram of a file merging device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] To help those skilled in the art better understand the present disclosure, the following will provide a clear and complete description of the technical solutions in the embodiments of the present disclosure, in conjunction with the accompanying drawings. It should be understood that the described embodiments are only a portion of the embodiments of the present disclosure, not all of them. All other embodiments derived by persons skilled in the art based on the embodiments of the present disclosure without inventive effort are intended to fall within the scope of protection of the present disclosure.

[0058] The inventors of this disclosure have discovered through research that conventional technologies for merging data lake files are inefficient. Specifically, reference may be made to FIG1 , which illustrates a flowchart of a conventional method for merging files.

[0059] As shown in Figure 1, when merging multiple data lake files to be merged (such as Parquet small files), you can first de-encode, de-compress, and deserialize the data in the multiple data lake files to be merged to obtain the data to be merged. Then, when merging the data to be merged, you need to serialize, compress, and encode the data to be merged to obtain the merged data lake file.

[0060] The aforementioned process of de-encoding, decompressing, deserializing, serializing, compressing, and encoding data is relatively cumbersome, resulting in low efficiency in file merging. Therefore, there is an urgent need for a solution that can solve the above problems and improve the efficiency of file merging.

[0061] In view of this, embodiments of the present disclosure provide a file merging method and device.

[0062] Before introducing the file merging method and device provided by the embodiments of the present disclosure, the data lake file is first introduced.

[0063] See Figure 2, which is a schematic diagram of the structure of the content stored in a data lake file provided by an embodiment of the present disclosure. As shown in Figure 2:

[0064] A data lake file consists of three parts: header, data, and footer.

[0065] The header contains very little content, only 4 bytes, which is a fixed magic number used to indicate that this is a Parquet file.

[0066] Data is used to store data. When the original data is stored in a Parquet file, it is first divided into multiple row groups (RowGroup) according to the row. For example, in Figure 2, three RowGroup0s may store the data from row 1 to row 100 in the original data table.

[0067] Within each RowGroup, it is divided into multiple column chunks (ColumnChunks) according to the columns, corresponding to the columns in the original data table. For example, in Figure 2, ColumnChunk0 may store the data of the first column in RowGroup1.

[0068] ColumnChunk can be further divided into multiple pages (Page), each page includes a page header (Page Header) and page data (Page Data), and the page header is used to describe the encoding information of the page.

[0069] The footer is mainly used to record metadata, including the storage location of each RowGroup and each ColumnChunk in the data lake file. The storage location can be reflected through information such as offset and size. The footer can be used to quickly locate the location of data in the file and read it on demand.

[0070] It should be noted that Figure 2 is only shown for the purpose of understanding the content structure of the data lake file, and does not constitute a limitation on the embodiments of the present disclosure. A data lake file is not limited to including 3 RowGroups, a RowGroup is not limited to being able to be split into 3 ColumnChunks, and the number of Pages included in a ColumnChunk is not limited to the 3 shown in Figure 2.

[0071] See Figure 3, which illustrates a method for storing a raw data table as a data lake file, according to an embodiment of the present disclosure. Figure 3 shows a raw data table with two rows and three columns, and the data lake file generated from it. For details about the contents of this data lake file, refer to the above description and are not repeated here.

[0072] The embodiment of the present disclosure provides a file merging method, which includes: obtaining initial data stored in multiple data lake files to be merged, wherein the initial data is data that has been processed by one or more of serialization, compression and encoding. After obtaining the initial data, the initial data can be directly used as the storage data in the target data lake file, and the target metadata of the target data lake file can be obtained based on the storage position of the storage data in the target data lake file. After obtaining the storage data and the target metadata, the storage data and the target metadata can be written into the target data lake file, thereby obtaining a target data lake file obtained by merging the multiple data lake files to be merged. It can be seen that, using the solution of the embodiment of the present disclosure, when merging multiple data lake files to be merged into a target data lake file, there is no need to first process the data stored in the data lake files to be merged, but the original data can be directly used as the storage data in the target data lake file, thereby improving the efficiency of merging multiple data lake files to be merged into a target data lake file.

[0073] Various non-limiting embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0074] Referring to FIG4 , which is a flowchart of a file merging method provided by an embodiment of the present disclosure, in this embodiment, the method may be executed by a server, for example, and may include the following steps: S101-S104.

[0075] S101: Acquire initial data stored in multiple data lake files to be merged, where the initial data is data that has undergone one or more of serialization, compression, and encoding.

[0076] The initial data mentioned here refers to the data in the data lake file to be merged. In the embodiment of the present disclosure, the initial data stored in the multiple data lake files to be merged can be directly read from the multiple data lake files to be merged.

[0077] In one example, before executing S101, the multiple data lake files to be merged can also be determined. In a specific example, the multiple data lake files to be merged are pre-specified. In another specific example, the multiple data lake files to be merged are determined by the server based on a target structured query language (SQL) statement. Specifically, the server can receive a target SQL statement for specifying the data lake files to be merged. The target SQL statement, for example, includes the conditions that the multiple data lake files to be merged need to meet. After the server receives the target SQL statement, it can execute a query operation according to the target SQL statement to obtain the multiple data lake files to be merged. The embodiment of the present disclosure does not specifically limit the conditions that the data lake files to be merged need to meet. The condition can be, for example, the date on which the data lake file is generated, or the size of the data lake file, and so on.

[0078] For example, the target SQL statement may be: ALTER TABLE$tableName PARTITION(date='***'). The target SQL statement specifies the dates of the multiple data lake files to be merged. In this case, the server may determine the data lake file with the date "***" as the data lake file to be merged.

[0079] In one example, the target SQL statement may also include a data lake file merge statement, which is used to instruct the merging of the multiple data lake files to be merged. In this case, the target SQL statement can be used not only to specify the data lake files to be merged, but also to specify the merge operation to be performed on the data lake files to be merged. In this case, after receiving the target SQL statement, the server can, on the one hand, determine the multiple data lake files to be merged, and on the other hand, execute S101-S104 to merge the multiple data lake files to be merged.

[0080] For example, the data lake file merge statement may be: COMPACT 'FastCompactionStrategy', where the fast merge strategy (FastCompactionStrategy) indicates that the multiple data lake files to be merged are merged using the file merging method provided by the embodiment of the present disclosure.

[0081] In another example, the target SQL statement may also include a file size setting statement, which is used to set the size of the target data lake file generated after the merge to a target size. In this case, when the server merges the multiple data lake files to be merged, the size of the target data lake file generated is the target size. In this case, the target SQL statement can be used not only to specify the data lake files to be merged, but also to specify the size of the target data lake file generated by merging the data lake files to be merged.

[0082] In one example, the server can determine the storage locations of the data in the multiple data lake files to be merged based on the metadata of the multiple data lake files to be merged, and further read the initial data from the corresponding storage locations. In a specific example, when S101 is implemented, each of the multiple data lake files to be merged can be traversed and the following steps A1-A3 can be performed on each data lake file to be merged.

[0083] A1: Determine the offset and size of the row group in the data lake file to be merged based on the metadata of the data lake file to be merged.

[0084] In one example, metadata of the data lake file to be merged may be read to determine the offset and size of each row group in the data lake file to be merged.

[0085] A2: Determine the offset and size of the column block in the to-be-merged data lake file according to the metadata of the row group.

[0086] In one example, for each row group, the offset and size of the column blocks included in the row group in the data lake file to be merged can be determined based on the metadata of the row group, thereby obtaining the offset and size of each column block in the data lake file to be merged in the data lake file to be merged.

[0087] A3: Acquire the data in the column block as the initial data.

[0088] After obtaining the offset and size of the column block in the data lake file to be merged, the storage location of the column block in the data lake file to be merged can be determined based on the offset and size. Accordingly, the data in the column block can be read from the determined storage location as the initial data. The data in the column block can be binary fragments.

[0089] In one example, the server may first create a target data lake file and write a header (ie, a magic number) into the target data lake file. Furthermore, S102 and subsequent steps are executed.

[0090] S102: directly use the initial data as storage data in the target data lake file.

[0091] S103: Obtain target metadata of the target data lake file according to a storage location of the stored data in the target data lake file.

[0092] S104: Writing the stored data and the target metadata into the target data lake file.

[0093] In order to improve the efficiency of merging data lake files in the embodiment of the present disclosure, after obtaining the initial data, the initial data is no longer subjected to one or more of the processes of de-encoding, de-compression, deserialization, serialization, compression, and encoding. Instead, the initial data is directly used as the storage data in the target data lake file, thereby reducing data processing and improving file merging efficiency.

[0094] As can be seen from the previous description of the data lake file, the data lake file includes a header, data, and a footer, and the footer is used to store metadata. Therefore, in the embodiment of the present disclosure, the target metadata of the target data lake file can also be obtained based on the storage position of the stored data in the target data lake file. In a specific example, the position of the stored data in the target data lake file may include the offset and size of each row group in the aforementioned initial data in the target data lake file. After determining the stored data and the target metadata, the stored data and the target metadata can be written into the target data lake file, thereby obtaining a target data lake file obtained by merging multiple data lake files to be merged.

[0095] In the embodiment of the present disclosure, S103 can be implemented in multiple ways.

[0096] In one example, the storage location of the stored data in the target data lake file can be pre-planned based on its size, and then the target metadata can be obtained based on the planned storage location. In this case, S104 can be implemented by, for example, synchronously writing the stored data and the target metadata to the target data lake file.

[0097] In another example, after executing S102, the initial data can first be written directly into the target data lake file, and then, based on the storage location of the initial data in the target data lake file, the target metadata of the target data lake file can be obtained. In this case, after the initial data is written into the target data lake file, the storage location of the initial data in the target data lake file can be obtained, without pre-planning the storage location of the initial data in the target data lake file. Accordingly, in this case, after obtaining the target metadata of the target data lake file based on the storage location of the initial data in the target data lake file, the target metadata can be further written into the target data lake file, thereby obtaining a target data lake file obtained by merging multiple data lake files to be merged.

[0098] From the above description, it can be seen that, using the solution of the embodiments of the present disclosure, when merging multiple data lake files to be merged into a target data lake file, there is no need to perform one or more of the following processes on the data stored in the data lake files to be merged, such as de-encoding, decompression, deserialization, serialization, compression, and encoding. Instead, the original data can be directly used as the storage data in the target data lake file, thereby improving the efficiency of merging multiple data lake files to be merged into the target data lake file.

[0099] Next, the file merging method provided by the embodiment of the present disclosure is introduced with reference to specific examples.

[0100] See Figure 5, which is a scenario diagram of a file merging method provided by an embodiment of the present disclosure. As shown in Figure 5, Parquet file 1 to be merged and Parquet file 2 to be merged are merged into a target Parquet file. Data 510 and 520 in Parquet file 1 to be merged are directly copied to the target Parquet file as storage data for the target Parquet file. Data 530 in Parquet file 2 to be merged is directly copied to the target Parquet file as storage data for the target Parquet file.

[0101] Based on the methods provided in the above embodiments, an embodiment of the present disclosure further provides a device, which is described below in conjunction with the accompanying drawings.

[0102] 6 , which is a schematic diagram of the structure of a file merging device provided by an embodiment of the present disclosure, wherein the device 600 may specifically include: an acquiring unit 601 , a first determining unit 602 , a second determining unit 603 , and a writing unit 604 .

[0103] An acquisition unit 601 is configured to acquire initial data stored in multiple data lake files to be merged, where the initial data is data that has undergone one or more of serialization, compression, and encoding.

[0104] A first determining unit 602 is configured to directly use the initial data as storage data in a target data lake file;

[0105] A second determining unit 603 is configured to obtain target metadata of the target data lake file according to a storage location of the stored data in the target data lake file;

[0106] The writing unit 604 is configured to write the stored data and the target metadata into the target data lake file.

[0107] Optionally, the writing unit 604 is configured to:

[0108] First, the initial data is directly written into the target data lake file, and then the target metadata is written into the target data lake file.

[0109] Optionally, the second determining unit 603 is configured to:

[0110] After the initial data is directly written into the target data lake file, target metadata of the target data lake file is obtained according to a storage location of the stored data in the target data lake file.

[0111] Optionally, the device further includes:

[0112] a receiving unit, configured to receive a target structured query language statement before obtaining the initial data stored in the plurality of data lake files to be merged, wherein the target structured query language statement is used to specify the data lake files to be merged;

[0113] A third determining unit is configured to determine the multiple data lake files to be merged according to the target structured query language statement.

[0114] Optionally, the target structured query language statement further includes a data lake file merging statement, where the data lake file merging statement is used to instruct to merge the multiple data lake files to be merged.

[0115] Optionally, the target structured query language statement further includes a file size setting statement, where the file size setting statement is used to indicate that the size of the target data lake file is a target size, and the size of the generated target data lake file is the target size.

[0116] Optionally, the acquiring unit 601 is configured to:

[0117] Traverse the multiple data lake files to be merged, and perform the following operations for each data lake file to be merged:

[0118] Determining, based on the metadata of the data lake file to be merged, the offset and size of the row group in the data lake file to be merged in the data lake file to be merged;

[0119] Determining, based on the metadata of the row group, the offset and size of the column block in the to-be-merged data lake file;

[0120] The data in the column block is obtained as the initial data.

[0121] Since the device 600 is a device corresponding to the method provided in the above method embodiment, the specific implementation of each unit of the device 600 is based on the same concept as the above method embodiment. Therefore, regarding the specific implementation of each unit of the device 600, please refer to the description part of the above method embodiment, which will not be repeated here.

[0122] The embodiment of the present disclosure further provides a file merging device, the device comprising a processor and a memory;

[0123] The processor is configured to execute instructions stored in the memory, so that the device executes the file merging method provided in the above method embodiment.

[0124] An embodiment of the present disclosure provides a computer-readable storage medium, including instructions, wherein the instructions instruct a device to execute the file merging method provided by the above method embodiment.

[0125] The embodiments of the present disclosure further provide a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the file merging method provided by the above method embodiments.

[0126] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0127] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0128] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A file merging method, comprising: Obtaining initial data stored in multiple data lake files to be merged, where the initial data is data that has undergone one or more of serialization, compression, and encoding; Directly use the initial data as storage data in the target data lake file; Obtaining target metadata of the target data lake file according to a storage location of the stored data in the target data lake file; as well as The stored data and the target metadata are written into the target data lake file.

2. The method according to claim 1, wherein: The step of writing the stored data and the target metadata into the target data lake file includes: First, the initial data is directly written into the target data lake file, and then the target metadata is written into the target data lake file.

3. The method according to claim 2, wherein: The obtaining target metadata of the target data lake file according to the storage position of the stored data in the target data lake file includes: After the initial data is directly written into the target data lake file, target metadata of the target data lake file is obtained according to a storage position of the stored data in the target data lake file.

4. The method according to claim 1, wherein: Before obtaining the initial data stored in the plurality of data lake files to be merged, the method further includes: receiving a target structured query language statement, where the target structured query language statement is used to specify the data lake file to be merged; and According to the target structured query language statement, the multiple data lake files to be merged are determined.

5. The method according to claim 4, wherein: The target structured query language statement includes a data lake file merging statement, and the data lake file merging statement is used to instruct to merge the multiple data lake files to be merged.

6. The method according to claim 4 or 5, wherein: The target structured query language statement also includes a file size setting statement, where the file size setting statement is used to indicate that the size of the target data lake file is a target size, and the size of the generated target data lake file is the target size.

7. The method according to any one of claims 1 to 6, wherein: The obtaining of initial data stored in multiple data lake files to be merged includes: Traverse the multiple data lake files to be merged, and perform the following operations for each data lake file to be merged: Determine, according to the metadata of the to-be-merged data lake file, the offset and size of the row group in the to-be-merged data lake file in the to-be-merged data lake file; Determining, according to the metadata of the row group, an offset and a size of a column block in the to-be-merged data lake file in the to-be-merged data lake file; and The data in the column block is obtained as the initial data.

8. A file merging device, wherein: The device comprises: An acquisition unit is configured to acquire initial data stored in a plurality of data lake files to be merged, wherein the initial data is data that has undergone one or more of serialization, compression, and encoding; A first determining unit is configured to directly use the initial data as storage data in a target data lake file; A second determining unit is configured to obtain target metadata of the target data lake file according to a storage location of the stored data in the target data lake file; and A writing unit is configured to write the stored data and the target metadata into the target data lake file.

9. A file merging device, wherein: The file merging device includes at least one processor and a memory; The at least one processor is configured to execute instructions stored in the memory, so that the file merging device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: The method comprises instructions for instructing a device to execute the method according to any one of claims 1 to 7.

Images