A file merging method, system, storage medium and electronic device

Abstract

Embodiments of the present application provide a file merging method, system, storage medium and electronic device. The method comprises: in the case that data of a target data table changes, obtaining the latest change time of a first metadata file; obtaining the latest change time of each metadata file of the target data table; from the metadata files of the target data table, screening metadata files with the latest change time not earlier than the latest change time, to obtain at least one second metadata file; from the second metadata files, screening at least metadata files with the data operation time recorded in the second metadata file later than the first data operation time, to obtain at least one third metadata file; and merging data files corresponding to the third metadata files respectively, to obtain a merged data file. The present application reduces the number of data files and shortens the query time by means of file merging.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a file merging method, system, storage medium, and electronic device. Background Technology

[0002] When data is written to or modified in a data table, multiple data files are created, each storing a portion of the table's data. When querying the table, because the data matching the query criteria is scattered across multiple data files, each data file needs to be opened individually to retrieve the data, resulting in a lengthy query time. Summary of the Invention

[0003] The purpose of this invention is to provide a file merging method, system, storage medium, and electronic device that reduces the number of data files and shortens query time through file merging. The specific technical solution is as follows:

[0004] This invention provides a file merging method, comprising:

[0005] If the data in the target data table changes, the latest change time of the first metadata file is obtained; the first metadata file is the metadata file corresponding to the data file of the target data table that participated in the previous merge operation.

[0006] Obtain the latest change time of each metadata file of the target data table;

[0007] From the metadata files of the target data table, filter the metadata files whose latest change time is no earlier than the latest change time to obtain at least one second metadata file;

[0008] From each of the second metadata files, at least one third metadata file is obtained by filtering out metadata files in the second metadata files whose data operation time is later than the first data operation time; the first data operation time is the data operation time recorded in the first metadata file corresponding to the latest change time.

[0009] The data files corresponding to each of the aforementioned third metadata files are merged to obtain the merged data file.

[0010] Optionally, before obtaining at least one third metadata file by filtering at least one of the second metadata files from each of the second metadata files whose data operation time is later than the first data operation time, the method further includes:

[0011] The data operation type recorded in the second metadata file is compared with the first data operation type; wherein, the first data operation type is the data operation type recorded in the metadata file corresponding to the latest change time;

[0012] If the data operation types are consistent, then proceed to the following step: from each of the second metadata files, at least filter the metadata files whose data operation time is later than the first data operation time to obtain at least one third metadata file.

[0013] Optionally, from the second metadata file, at least one metadata file is filtered to obtain at least one third metadata file whose data operation time is later than the first data operation time, including:

[0014] From the second metadata file, filter out metadata files whose data operation time is later than the first data operation time to obtain the third metadata file;

[0015] or,

[0016] From the second metadata file, filter out metadata files whose data operation time is later than the first data operation time and whose data operation version number is greater than the first data operation version number to obtain the third metadata file; wherein, the first data operation version number is the data operation version number recorded in the metadata file corresponding to the latest change time.

[0017] Optionally, the process of determining that the data in the target data table has changed includes:

[0018] Obtain the latest operation record version number of the target data table;

[0019] The latest operation record version number is compared with the first operation record version number; the first operation record version number is the operation record version number of the target data table generated after the last data file of the target data table was merged.

[0020] If the latest operation record version number is greater than the first operation record version number, then it is determined that the data in the target data table has changed.

[0021] Optionally, after obtaining the merged data file, the method further includes:

[0022] The metadata file corresponding to the latest change time of the third metadata file is used as the target metadata file;

[0023] The target metadata file is stored to record the change time, as well as the data operation time, data operation type, and data operation version number recorded in the target metadata file.

[0024] The operation record version number of the target data table generated after merging the data file corresponding to the third metadata file is stored.

[0025] Optionally, after obtaining the merged data file, the method further includes:

[0026] The merged data file is stored in the storage location where the data file corresponding to the third metadata file is located;

[0027] Delete the data file corresponding to the third metadata file.

[0028] Optionally, the step of merging the data files corresponding to each of the third metadata files to obtain a merged data file includes:

[0029] The target data indicated by the metadata in each of the third metadata files is merged to obtain a merged data file, provided that the number of data entries in the merged data file does not exceed a preset number of data entries, or that the amount of data in the merged data file does not exceed a preset amount of data; wherein, the target data is the data in the data file corresponding to the third metadata file.

[0030] The present invention also provides a file merging system, comprising:

[0031] The first-time acquisition module is used to obtain the latest change time of the first metadata file when the data of the target data table changes; the first metadata file is the metadata file corresponding to the data file of the target data table that participated in the previous merge operation;

[0032] The second time acquisition module is used to obtain the latest change time of each metadata file of the target data table;

[0033] The first filtering module is used to filter metadata files from each metadata file of the target data table whose latest change time is not earlier than the latest change time, to obtain at least one second metadata file;

[0034] The second filtering module is used to filter from each of the second metadata files at least the metadata files in the second metadata files whose data operation time is later than the first data operation time, to obtain at least one third metadata file; the first data operation time is the data operation time recorded in the first metadata file corresponding to the latest change time.

[0035] The file merging module is used to merge the data files corresponding to each of the aforementioned third metadata files to obtain the merged data file.

[0036] The present invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described file merging method.

[0037] The present invention also provides an electronic device, comprising:

[0038] At least one processor, and at least one memory and bus connected to the processor;

[0039] The processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the above-described file merging method.

[0040] The present invention provides a file merging method, system, storage medium, and electronic device. By comparing the latest change time of the recorded metadata file with the latest change time of the metadata file, a second metadata file in which the metadata has changed can be found. Then, metadata files with data operation times later than the first data operation time are filtered from the second metadata file to find newly generated data files. The found data files are then merged, which can reduce the number of data files and shorten the query time.

[0041] Of course, any product or method implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart of a file merging method provided in an embodiment of the present invention;

[0044] Figure 2 A file merging system structure diagram provided for embodiments of the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] When data is written to a table stored in HDFS (Hadoop Distributed File System) using the Spark data processing engine, the table is split into multiple data files on HDFS, with each file storing a portion of the table's data. When querying this table, because the data matching the query criteria is scattered across multiple data files, each file needs to be opened individually to retrieve the data, resulting in a significant query time.

[0048] To address the aforementioned problems, this invention provides a file merging method that reduces the number of data files and shortens query time. For example... Figure 1 As shown, the method includes:

[0049] Step 101: If the data in the target data table changes, obtain the latest change time of the first metadata file; the first metadata file is the metadata file corresponding to the data file of the target data table that participated in the previous merge operation.

[0050] Performing a transaction on a data table will change the table's version number. This transaction includes, but is not limited to, adding, modifying, and deleting data. The data in a data table is stored in folders. Files storing the data are called data files, and files storing metadata are called metadata files. Metadata describes the data, such as the operation time, operation type, and operation version. One metadata file can correspond to one data file.

[0051] Optionally, the target data table can be a delta table. When data is written to the delta table stored in HDFS by a data processing engine such as Spark or Flink, the table will form many delta files on HDFS, and each file stores a part of the delta table's data.

[0052] Determining whether data in a data table has changed can be done by comparing the table's version number. As an optional implementation, the process of determining if data in a target data table has changed includes: obtaining the latest operation record version number of the target data table; comparing the latest operation record version number with a first operation record version number; the first operation record version number is the version number of the operation record generated after the last merging of the target data table's data files; if the latest operation record version number is greater than the first operation record version number, then it is determined that the data in the target data table has changed.

[0053] Suppose the latest operation record version number of the target data table is 'a', and the version number of the target data table after the last file merge is 'b' (i.e., the first operation record version number is 'b'). If a > b, it means that at least one transaction operation was performed on the target data table after the file merge, causing the version number to increase. When a > b, it indicates that the data in the target data table has changed.

[0054] If the data in the target table changes, to find files that haven't been merged, you can filter by the change time of the metadata file. In practice, you can compare the latest change time of the metadata file recorded after the last file merge with the latest change time of the metadata file to find the metadata file corresponding to the unmerged data file.

[0055] During the last file merge, the data files of the target data table were merged, and the metadata files corresponding to the data files participating in the merge were used as the first metadata files. If the data files change, the corresponding metadata files will also change. When the metadata files change, the change time of the metadata files will be recorded. During a file merge process, if the change time of the metadata file is the latest, it means that the data file corresponding to this metadata file may be the data file that changed the latest. Finding the first metadata file can facilitate the search for unmerged files in the next step. For example, in the last merge operation of the target data table, there are 3 data files, namely data file 1, data file 2, and data file 3. The first metadata files are metadata file 1, metadata file 2, and metadata file 3 respectively, and the corresponding data files are data file 1, data file 2, and data file 3 respectively. The change time of metadata file 1 is t1, the change time of metadata file 2 is t2, and the change time of metadata file 3 is t3. If t1 < t2 < t3, then the latest change time of the first metadata file is t3. If the data files 1, 2, and 3 are merged to generate a new data file, data file 4, and the corresponding metadata file is metadata file 4, then the change time of this metadata file 4 is greater than t1, t2, and t3. Therefore, by recording the latest change time of the first metadata file, when performing file merge next time, the data file 4 can be found through the change time of the metadata file 4 as the newly generated data file, so that the merge operation can be performed on the data file 4.

[0056] Step 102: Obtain the latest change time of each metadata file of the target data table.

[0057] When the data in the target data table changes, the version number of the target data table is 'a', meaning the latest operation record version number of the target data table is 'a'. In this case, we obtain the latest change time of each metadata file of the target data table at version number 'a'. For example, when the version number of the target data table is 'a', there are 6 metadata files: metadata file 1 (latest change time t1), metadata file 2 (latest change time t2), metadata file 3 (latest change time t3), metadata file 4 (latest change time t4), metadata file 5 (latest change time t5), and metadata file 6 (latest change time t6). When the version number of the target data table is 'b', meaning the first operation record version number is 'b', there are 3 metadata files: metadata file 1 (latest change time t1), metadata file 2 (latest change time t2), and metadata file 3 (latest change time t3). In other words, after the last file merge operation, the newly added metadata files are metadata file 4, metadata file 5, and metadata file 6. These metadata files describe data files that were newly generated after the last merge operation, and these unmerged data files need to be found through subsequent steps.

[0058] Step 103: From the metadata files of the target data table, filter the metadata files whose latest change time is no earlier than the latest change time to obtain at least one second metadata file.

[0059] To locate unmerged data files, we can first obtain a second metadata file using the change times of the metadata files. If a data file has been added or modified, the change times of the metadata files corresponding to the changed data files will change. Assuming the latest change time is t3, if the latest change time of a metadata file is later than t3, it means the data file corresponding to that metadata file changed after the last merge operation. In this case, we can filter out that metadata file to facilitate finding the corresponding data file and then performing a file merge operation on that data file. Of course, we cannot rule out cases where the granularity of the change time is large. If the latest change time of a metadata file is equal to t3, the corresponding data file may or may not have changed. To improve the accuracy of finding unmerged data files, we can filter out metadata files whose latest change time is equal to the latest change time. For example, metadata file 4 has a latest change time of t4, metadata file 5 has a latest change time of t5, and metadata file 6 has a latest change time of t6. Since t4, t5, and t6 are all greater than the latest change time t3, the filtered second metadata files are metadata files 4, 5, and 6.

[0060] Step 104: From each second metadata file, at least filter the metadata files in the second metadata files whose data operation time is later than the first data operation time to obtain at least one third metadata file; the first data operation time is the data operation time recorded in the first metadata file corresponding to the latest change time.

[0061] As an optional implementation, before step 104, the method provided by the present invention further includes: comparing the data operation type recorded in the second metadata file with the first data operation type; wherein, the first data operation type is the data operation type recorded in the metadata file corresponding to the latest change time; if the data operation types are consistent, then step 104 is executed.

[0062] Since changes to the target data table can occur through data addition, modification, or deletion, new data files are generated during data addition and modification, but not during data deletion. Therefore, this invention can determine the data operation type recorded in the second metadata file based on the data operation type, specifically by comparing it with the first data operation type. Since the first data type is the data operation type recorded in the metadata file corresponding to the latest change time, and this data operation type is either data addition or data modification, metadata files with consistent data types—that is, metadata files whose second metadata files record operation types as data addition or data modification—can be selected before proceeding to step 104.

[0063] If the data operation type recorded in the second metadata file is data addition or data modification, at least one third metadata file is obtained by filtering the metadata files in the second metadata file whose data operation time is later than the first data operation time.

[0064] As an optional implementation, from the second metadata file, the metadata files whose data operation time is later than the first data operation time are selected to obtain the third metadata file.

[0065] In this implementation, the metadata file describes the data file and records the data operation time. If the first metadata file corresponding to the latest change time after the last merge operation is metadata file 3, and the data operation time recorded in metadata file 3 is s1 (i.e., the first data operation time is s1), then the data operation time recorded in the second metadata file is searched. If the second metadata files are metadata files 4, 5, and 6, where the data operation time recorded in metadata file 4 is s2, the data operation time recorded in metadata file 5 is s3, and the data operation time recorded in metadata file 6 is s0, and s2 and s3 are greater than s1, and s0 is less than s1, then the third metadata files are metadata files 4 and 5. Since the data operation time recorded in metadata file 6 is earlier than the first data operation time, it means that the data file corresponding to the metadata file has already participated in the file merge operation, and the file merge operation will not be performed again on the data file corresponding to metadata file 6.

[0066] As another optional implementation, from the second metadata file, the metadata files whose data operation time is later than the first data operation time and whose data operation version number is greater than the first data operation version number are selected to obtain the third metadata file; wherein, the first data operation version number is the data operation version number recorded in the metadata file corresponding to the latest change time.

[0067] In this implementation, in addition to filtering metadata files whose data operation time is later than the first data operation time, it is also necessary to ensure that the data operation version number recorded in the second metadata file is greater than the first data operation version number. After the data in the data file is operated on, the data operation version number will change. If the data changes twice, the data operation version number of the previous change will be less than the data operation version number of the next change. If the first metadata file corresponding to the latest change time after the last merge operation is metadata file 3, and the data operation version number recorded in metadata file 3 is v1 (i.e., the first data operation version number is v1), the metadata files whose data operation time is later than the first data operation time are metadata file 4 and metadata file 5. If the data operation version number recorded in metadata file 4 is v2, and the data operation version number recorded in metadata file 5 is v3, and both v2 and v3 are greater than v1, then the third metadata files are metadata file 4 and metadata file 5.

[0068] Step 105: Merge the data files corresponding to each third-party metadata file to obtain the merged data file.

[0069] As an optional implementation, the data files corresponding to each third metadata file are merged to obtain a merged data file, including: merging the target data indicated by the metadata in each third metadata file according to the number of data entries in the merged data file not exceeding a preset number of data entries, or according to the amount of data in the merged data file not exceeding a preset amount of data, to obtain a merged data file; wherein, the target data is the data in the data file corresponding to the third metadata file.

[0070] In practical applications, to ensure that the number of data entries in the merged data file does not exceed a preset number of data entries, the target data indicated by the metadata in each third-party metadata file is merged. The ratio of the number of data entries in the merged data file to the preset number of data entries can be rounded up to obtain the number of merged files. Alternatively, to ensure that the data volume of the merged data file does not exceed a preset data volume, the target data indicated by the metadata in each third-party metadata file is merged. Again, the ratio of the data volume of the merged data file to the preset data volume can be rounded up to obtain the number of merged files. Then, based on the obtained number of merged files, the target data indicated by the metadata in each third-party metadata file is merged to obtain the merged data file.

[0071] As an optional implementation, after obtaining the merged data file, the method of the present invention further includes: using the metadata file corresponding to the latest change time of the third metadata file as the target metadata file; storing the change time of the target metadata file, and storing the data operation time, data operation type and data operation version number recorded in the target metadata file; and storing the operation record version number of the target data table generated after merging the data file corresponding to the third metadata file.

[0072] After merging the data files, the latest modification time of the third metadata file can be stored. It can also store the data operation time, data operation type, and data operation version number recorded in the metadata file corresponding to the latest modification time of the third metadata file. Furthermore, it stores the operation record version number of the target data table generated after the merge, to facilitate finding the changed data file during the next merge operation. If the third metadata files are metadata file 4 and metadata file 5, with modification time t4 for metadata file 4 and modification time t5 for metadata file 5, and if t4 > t5, then the latest modification time of the third metadata file is t4, the target metadata file is metadata file 4, and the data operation time s2, data operation type (add), and data operation version number v2 recorded in metadata file 4 are stored. If the operation record version number of the target data table after merging the data files corresponding to the third metadata file is c, then operation record version number c is stored. Optionally, the latest modification time t4, operation record version number c, data operation time s2, data operation type (add), and data operation version number v2 can be stored in ZooKeeper.

[0073] As an optional implementation, after obtaining the merged data file, the method of the present invention further includes: storing the merged data file in the storage location where the data file corresponding to the third metadata file is located; and deleting the data file corresponding to the third metadata file.

[0074] After locating the data file corresponding to the third-party metadata file, the name of the found data file can be obtained. Using the data file name and the path information of the target data table, the storage location of the merged data file can be determined. Specifically, if the target data table is a delta table, the path information of the delta table can be located on HDFS based on the configuration information of Hadoop, Hive (a data warehouse tool based on Hadoop), and Spark.

[0075] Optionally, deleting the data file corresponding to the third-party metadata file can be done by marking the data file as deleted, making it unreadable, and allowing for file recovery. If the deleted data file is expired, it should be permanently deleted.

[0076] The file merging method provided by this invention can perform file merging at regular intervals. This timed operation can be achieved by setting a scheduled task in the xxl-job scheduler, so that after performing a file merging operation, the process waits for the next scheduling cycle of xxl-job before executing the file merging process described in steps 101-105.

[0077] This invention also provides a file merging system, such as... Figure 2 As shown, the system includes:

[0078] The first-time acquisition module 201 is used to obtain the latest change time of the first metadata file when the data of the target data table changes; the first metadata file is the metadata file corresponding to the data file of the target data table that participated in the previous merge operation.

[0079] Optionally, the first-time acquisition module 201 is specifically used to determine the process of data changes in the target data table, including: obtaining the latest operation record version number of the target data table; comparing the latest operation record version number with the first operation record version number; the first operation record version number is the operation record version number of the target data table generated after the last data file of the target data table was merged; if the latest operation record version number is greater than the first operation record version number, it is determined that the data of the target data table has changed.

[0080] The second time acquisition module 202 is used to obtain the latest change time of each metadata file of the target data table.

[0081] The first filtering module 203 is used to filter metadata files from each metadata file of the target data table whose latest change time is no earlier than the latest change time, so as to obtain at least one second metadata file.

[0082] The second filtering module 204 is used to filter at least one third metadata file from each second metadata file whose data operation time is later than the first data operation time; the first data operation time is the data operation time recorded in the first metadata file corresponding to the latest change time.

[0083] Optionally, the second filtering module 204 is specifically used for:

[0084] From the second metadata file, filter out metadata files whose data operation time is later than the first data operation time to obtain the third metadata file;

[0085] or,

[0086] From the second metadata file, filter out the metadata files whose data operation time is later than the first data operation time and whose data operation version number is greater than the first data operation version number to obtain the third metadata file; wherein, the first data operation version number is the data operation version number recorded in the metadata file corresponding to the latest change time.

[0087] The file merging module 205 is used to merge the data files corresponding to each third-party metadata file to obtain the merged data file.

[0088] The file merging module 205 is specifically used to: merge the target data indicated by the metadata in each third metadata file according to the number of data entries in the merged data file not exceeding a preset number of data entries, or according to the amount of data in the merged data file not exceeding a preset amount of data, to obtain a merged data file; wherein, the target data is the data in the data file corresponding to the third metadata file.

[0089] As an optional implementation, the system of the present invention further includes:

[0090] The data operation type comparison module is used to compare the data operation type recorded in the second metadata file with the first data operation type; wherein, the first data operation type is the data operation type recorded in the metadata file corresponding to the latest change time; if the data operation types are consistent, then the second filtering module 204 is executed.

[0091] The storage module is used to take the metadata file corresponding to the latest change time of the third metadata file as the target metadata file; store the change time of the target metadata file, as well as the data operation time, data operation type and data operation version number recorded in the target metadata file; and store the operation record version number of the target data table generated after merging the data files corresponding to the third metadata file.

[0092] The deletion module is used to store the merged data file in the storage location where the data file corresponding to the third-party metadata file is located; and to delete the data file corresponding to the third-party metadata file.

[0093] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described file merging method.

[0094] This invention provides an electronic device, such as... Figure 3 As shown, the electronic device 30 includes at least one processor 301, at least one memory 302 connected to the processor 301, and a bus 303; wherein the processor 301 and the memory 302 communicate with each other through the bus 303; the processor 301 is used to call program instructions in the memory 302 to execute the above-described file merging method. The electronic device in this document may be a server, PC, PAD, mobile phone, etc.

[0095] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform a program that initializes the steps included in the above-described file merging method.

[0096] This application is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] In a typical configuration, the device includes one or more processors (CPUs), memory, and a bus. The device may also include input / output interfaces, network interfaces, etc.

[0098] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0099] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0102] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0103] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A file merging method characterized by, The method comprises the following steps: In the case that the data of the target data table changes, obtaining the latest change time of the first metadata file; The first metadata file is a metadata file corresponding to the data file of the target data table participating in the last merge operation; Obtaining the latest change time of each metadata file of the target data table; From each metadata file of the target data table, filtering the metadata file whose latest change time is not earlier than the latest change time to obtain at least one second metadata file; From each second metadata file, at least filtering the metadata file whose data operation time recorded in the second metadata file is later than the first data operation time to obtain at least one third metadata file; The first data operation time is the data operation time recorded in the first metadata file corresponding to the latest change time; Merging the data files corresponding to each third metadata file respectively to obtain the merged data file.

2. The file merging method of claim 1, wherein, Before the step of filtering the metadata file whose data operation time recorded in the second metadata file is later than the first data operation time from each second metadata file to obtain at least one third metadata file, the method further comprises: Comparing the data operation type recorded in the second metadata file with the first data operation type; wherein the first data operation type is the data operation type recorded in the metadata file corresponding to the latest change time; If the data operation types are consistent, then performing the step of filtering the metadata file whose data operation time recorded in the second metadata file is later than the first data operation time from each second metadata file to obtain at least one third metadata file.

3. The file merging method according to claim 1 or 2, characterized by, The step of filtering the metadata file whose data operation time recorded in the second metadata file is later than the first data operation time from each second metadata file to obtain at least one third metadata file comprises: Filtering the metadata file whose data operation time recorded in the second metadata file is later than the first data operation time from the second metadata file to obtain the third metadata file; Or, Filtering the metadata file whose data operation time recorded in the second metadata file is later than the first data operation time and whose data operation version number recorded in the second metadata file is greater than the first data operation version number from the second metadata file to obtain the third metadata file; wherein the first data operation version number is the data operation version number recorded in the metadata file corresponding to the latest change time.

4. The file merging method of claim 1, wherein, The process of determining that the data of the target data table changes comprises: Obtaining the latest operation record version number of the target data table; Comparing the latest operation record version number with the first operation record version number; the first operation record version number is the operation record version number of the target data table generated after the last merge operation on the data file of the target data table; If the latest operation record version number is greater than the first operation record version number, then determining that the data of the target data table changes.

5. The file merging method of claim 1, wherein, After the merged data file is obtained, the method further comprises: taking a metadata file corresponding to the latest change time of the third metadata file as a target metadata file; storing the change time of the target metadata file, and storing the data operation time, data operation type and data operation version number recorded in the target metadata file; storing the operation record version number of the target data table generated after merging the data file corresponding to the third metadata file.

6. The file merging method of claim 1, wherein, After the merged data file is obtained, the method further comprises: storing the merged data file into a storage location where the data file corresponding to the third metadata file is located; deleting the data file corresponding to the third metadata file.

7. The file merging method of claim 1, wherein, The merging of the data files corresponding to the third metadata files respectively to obtain the merged data file comprises: merging target data indicated by the metadata in each of the third metadata files to obtain the merged data file according to that the number of data in the merged data file does not exceed a preset number of data, or according to that the data amount in the merged data file does not exceed a preset data amount; wherein the target data is data in the data file corresponding to the third metadata file.

8. A file merging system characterized by, comprise: a first time obtaining module configured to obtain the latest change time of a first metadata file in a case where data of a target data table changes; the first metadata file is a metadata file corresponding to a data file of the target data table that participated in a last merging operation; a second time obtaining module configured to obtain the latest change time of each metadata file of the target data table; a first screening module configured to screen, from each metadata file of the target data table, a metadata file whose latest change time is not earlier than the latest change time to obtain at least one second metadata file; a second screening module configured to screen, from each of the second metadata files, a metadata file whose data operation time recorded in the second metadata file is later than a first data operation time to obtain at least one third metadata file; the first data operation time is a data operation time recorded in the first metadata file corresponding to the latest change time; a file merging module configured to merge data files corresponding to each of the third metadata files respectively to obtain a merged data file.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium has a program stored thereon, and the program is executed by a processor to implement the file merging method in any one of claims 1-7.

10. An electronic device, comprising: comprise: at least one processor, and at least one memory connected to the processor through a bus; the processor, the memory and the bus complete communication between each other; the processor is configured to invoke program instructions in the memory to execute the file merging method in any one of claims 1-7.

Images