File merging method and device, electronic device, and storage medium

Abstract

The present application discloses a file merging method and device, electronic device, and storage medium. The file merging method includes: obtaining the storage path and data storage type of a target data table; obtaining all subdirectories under the storage path of the target data table; filtering out target subdirectories from the subdirectories; for each target subdirectory, calling a computing engine to read all files under the target subdirectory, and synthesizing all files under the target subdirectory into a merged file of the data storage type; and replacing all files under the target subdirectory with the merged file. Thus, files of Hive data tables generated in various ways can be merged, reducing the number of files under the Hive data table.

Description

Technical Field

[0001] The present application relates to the field of file processing technology, and in particular to a file merging method and device, electronic equipment, and storage medium. Background Art

[0002] Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. Existing big data platforms commonly use Hive as a data warehouse. However, a Hive table often contains a large number of small files, resulting in a large ratio between the number of files and partitions in the table. This results in a significant consumption of computing resources such as memory and CPU when using applications to process data.

[0003] To reduce the number of files in a Hive table, a major existing method is to set execution parameters when generating a new Hive table using Hive-SQL schema data, thereby effectively reducing the number of files in the generated Hive table.

[0004] However, in a big data environment, in addition to being generated through Hive-SQL, many Hive tables are generated through other methods, such as mapping existing HDFS files to obtain Hive tables. In this case, the existing method cannot be used to reduce the number of files in the generated Hive tables. Moreover, the existing method is only for the process of generating Hive tables and cannot be applied to the generated Hive. Therefore, it can be seen that the scope of application of the existing method is very limited. Summary of the Invention

[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a file merging method and device, electronic device, and storage medium to solve the problem that the number of files cannot be effectively reduced within the scope of use of the prior art.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The first aspect of the present application provides a method for merging files, comprising:

[0008] Get the storage path and data storage type of the target data table;

[0009] Obtain all subdirectories under the storage path of the target data table;

[0010] Filter out the target subdirectory from the subdirectories;

[0011] For each target subdirectory, calling a computing engine to read all files under the target subdirectory, and synthesizing all files under the target subdirectory into a merged file belonging to the data storage type; wherein the computing engine has a data merging function;

[0012] Replace all files in the target subdirectory with the merged file.

[0013] Optionally, in the file merging method provided above, filtering out a target subdirectory from all subdirectories includes:

[0014] Delete the empty directories and multi-level directories in the subdirectory; wherein the empty directory is the subdirectory that does not contain any files; the multi-level directory is the subdirectory that contains the next-level directory;

[0015] Each of the subdirectories that has not been deleted is determined as a target subdirectory.

[0016] Optionally, the file merging method provided above further includes:

[0017] A temporary directory is created under the target subdirectory; wherein the temporary directory is used to store the merged file synthesized from the files under the target subdirectory.

[0018] Optionally, the file merging method provided above further includes:

[0019] Calling the computing engine to store the merged file in the temporary directory;

[0020] Verifying the validity of the merged file in the temporary directory; if the merged file in the temporary directory passes the validity check, replacing all files in the target subdirectory with the merged file;

[0021] If the merged file in the temporary directory fails the validity check, the temporary directory and the merged file under the temporary directory are deleted, and an error message is fed back.

[0022] Optionally, the file merging method provided above further includes:

[0023] Clean up the temporary directory under the target subdirectory.

[0024] Optionally, in the file merging method provided above, for each target subdirectory, calling a computing engine to read all files under the target subdirectory, and synthesizing all files under the target subdirectory into a merged file belonging to the data storage type includes:

[0025] For each target subdirectory, calling a computing engine to read and input the data storage type, and reading all files under the target subdirectory through the computing engine;

[0026] The calculation engine is called to read and input a set number, and all files under the target subdirectory are synthesized by the calculation engine into the set number of merged files of the data storage type.

[0027] Optionally, in the file merging method provided above, for each target subdirectory, calling a computing engine to read all files under the target subdirectory, and synthesizing all files under the target subdirectory into a merged file belonging to the data storage type includes:

[0028] Traversing the target subdirectories to be processed in the subdirectory list; wherein the subdirectory list includes each of the filtered target subdirectories;

[0029] Calling the computing engine to read all files under the target subdirectory currently traversed, and synthesizing all files under the target subdirectory into a merged file belonging to the data storage type;

[0030] After the processing of the target subdirectory currently traversed is completed, determining whether the target subdirectory to be processed still exists in the subdirectory list;

[0031] If it is determined that the target subdirectory to be processed still exists in the subdirectory list, the process returns to execute the traversal of the target subdirectory to be processed in the subdirectory list.

[0032] A second aspect of the present application provides a file merging device, comprising:

[0033] A first acquiring unit is used to acquire the storage path and data storage type of the target data table;

[0034] A second acquiring unit is configured to acquire all subdirectories under the storage path of the target data table;

[0035] A screening unit, configured to screen out a target subdirectory from the subdirectories;

[0036] a merging unit, configured to call a computing engine to read all files under each target subdirectory and synthesize all files under the target subdirectory into a merged file of the data storage type; wherein the computing engine has a data merging function;

[0037] The replacement unit is used to replace all files in the target subdirectory with the merged file.

[0038] Optionally, in the file merging device provided above, the screening unit includes:

[0039] The screening sub-unit is used to delete empty directories and multi-level directories in the sub-directories; wherein the empty directories are sub-directories that do not contain any files; and the multi-level directories are sub-directories that contain a directory at a lower level;

[0040] The determining unit is configured to determine each of the subdirectories that has not been deleted as a target subdirectory.

[0041] Optionally, the file merging device provided above further includes:

[0042] A creating unit is used to create a temporary directory under the target subdirectory; wherein the temporary directory is used to store the merged file synthesized from the files under the target subdirectory.

[0043] Optionally, the file merging device provided above further includes:

[0044] A storage unit, configured to call the computing engine to store the merged file in the temporary directory;

[0045] a verification unit, configured to verify the validity of the merged file in the temporary directory; wherein, if the merged file in the temporary directory passes the validity verification, the replacement unit executes the replacement of all files in the target subdirectory with the merged file;

[0046] A feedback unit is used to delete the temporary directory and the merged file under the temporary directory and feedback error information when the merged file in the temporary directory fails to pass the validity check.

[0047] Optionally, the file merging device provided above further includes:

[0048] A cleaning unit is configured to clean up the temporary directory under the target subdirectory after the replacement unit replaces all files under the target subdirectory with the merged file.

[0049] Optionally, in the file merging device provided above, the merging unit includes:

[0050] a reading unit, configured to call a computing engine to read and input the data storage type for each target subdirectory, and read all files under the target subdirectory through the computing engine;

[0051] The first merging sub-unit is used to call a calculation engine to read and input a set number, and to synthesize all files under the target sub-directory into the set number of merged files of the data storage type through the calculation engine.

[0052] Optionally, in the file merging device provided above, the merging unit includes:

[0053] a traversal unit, configured to traverse the target subdirectories to be processed in the subdirectory list; wherein the subdirectory list includes the screened target subdirectories;

[0054] A second merging sub-unit is configured to call the computing engine to read all files under the target sub-directory currently traversed, and synthesize all the files under the target sub-directory into a merged file belonging to the data storage type;

[0055] A judging unit, configured to judge whether there is any target subdirectory to be processed in the subdirectory list after the processing of the target subdirectory currently traversed is completed;

[0056] The returning unit is configured to return to the traversing unit to execute the traversal of the target subdirectory to be processed in the subdirectory list if it is determined that the target subdirectory to be processed still exists in the subdirectory list.

[0057] A third aspect of the present application provides an electronic device, characterized by comprising:

[0058] memory and processor;

[0059] Wherein, the memory is used to store programs;

[0060] The processor is used to execute the program, and when the program is executed, it is specifically used to implement the file merging method as described in any one of the above items.

[0061] A fourth aspect of the present application provides a computer storage medium for storing a computer program, wherein when the computer program is executed, the computer storage medium is used to implement the file merging method as described in any one of the above items.

[0062] The present application provides a file merging method, which obtains the storage path and data storage type of a target data table, then obtains all subdirectories under the storage path of the target data table, and filters out target subdirectories containing only files from all subdirectories, thereby obtaining subdirectories that can be merged. Then, for each target subdirectory, a computing engine is called to read all files under the target subdirectory, and all files under the target subdirectory are synthesized into a merged file belonging to the data storage type. Finally, the merged file replaces all files under the target subdirectory, thereby merging multiple files in a generated data table without changing the file type, effectively reducing the number of files in the generated data table. Therefore, for data tables generated in various ways, the number of files therein can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0064] Figure 1 A flowchart of a method for merging files provided in an embodiment of the present application;

[0065] Figure 2 A flowchart of a method for screening target subdirectories provided in an embodiment of the present application;

[0066] Figure 3 A flowchart of a method for calling a computing engine to merge files provided in an embodiment of the present application;

[0067] Figure 4 A flowchart of another method for calling a computing engine to merge files provided in another embodiment of the present application;

[0068] Figure 5 A flowchart of another method for merging files provided in another embodiment of the present application;

[0069] Figure 6 A schematic structural diagram of a file merging device provided in another embodiment of the present application;

[0070] Figure 7 A schematic structural diagram of a screening unit provided in another embodiment of the present application;

[0071] Figure 8 A schematic structural diagram of a merging unit provided in another embodiment of the present application;

[0072] Figure 9 A schematic structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0074] In this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0075] The present application embodiment provides a method for merging files, such as Figure 1 As shown, the following steps are included:

[0076] S101: Obtain the storage path and data storage type of the target data table.

[0077] The target data table may be any hive data table in the data warehouse tool hive that needs to be merged.

[0078] Therefore, you can call the API interface of the data warehouse tool Hive to obtain the storage path of the target data table, so that you can determine the files of the target data table based on the storage path, and obtain the data storage type of the target data table, such as Parquet, Orc, etc., so that you can merge data according to the data storage type to ensure that the storage types of the files before and after the merge are the same.

[0079] S102: Acquire all subdirectories under the storage path of the target data table.

[0080] It should be noted that the files under a hive data table are usually stored in different subdirectories, that is, a hive data table usually consists of multiple subdirectories. Therefore, to obtain the files of the target data table, you need to first obtain all subdirectories under the target data table according to the storage path of the target data table obtained in step S101.

[0081] Alternatively, you can call the HDFS API to retrieve all subdirectories from the storage path of the target data table.

[0082] S103: Filter out target subdirectories from the subdirectories.

[0083] The target subdirectory is a subdirectory that is required and can be merged. Since a subdirectory may contain files and therefore do not require merging, or contain other data and cannot be used for merging, such as a subdirectory at the next level, in this embodiment of the application, it is necessary to filter out the subdirectories containing files, and only the subdirectories containing files are used as target subdirectories.

[0084] Therefore, if Figure 2 As shown, a specific implementation of step S103 includes:

[0085] S201. Delete empty directories and multi-level directories in subdirectories.

[0086] An empty directory is a subdirectory that contains no files. A multi-level directory is a subdirectory that contains a directory at the next level.

[0087] It should be noted that filtering out subdirectories of empty directories and subdirectories that contain the next-level directory refers to filtering out the read subdirectories. This is to filter out the target subdirectories and only process them, rather than deleting these subdirectories from the Hive data table.

[0088] Since empty directories do not contain files, they do not need to be merged, so they need to be filtered out. As for subdirectories that contain subdirectories, they contain directories rather than files and cannot be merged, so they also need to be filtered out.

[0089] S202: Determine each subdirectory that has not been deleted as a target subdirectory.

[0090] S104: For each target subdirectory, call a computing engine to read all files under the target subdirectory, and synthesize all files under the target subdirectory into a merged file belonging to the data storage type.

[0091] The computing engine has a data merging function, and specifically can be a computing engine such as Spark. Specifically, the API interface of the computing engine Spark can be called to read all files in the target subdirectory, and then the API interface of the computing engine Spark can be called again to synthesize all the files in the read target subdirectory into a merged file of the data storage type.

[0092] It should be noted that after the target subdirectories are screened, steps S104 and S105 are executed for each target subdirectory. Alternatively, each target subdirectory may be traversed sequentially until all target subdirectories are processed, or multiple target subdirectories may be processed simultaneously through multiple threads.

[0093] In an embodiment of the present application, all files under the target subdirectory are read by calling the calculation engine, and all files are merged.

[0094] Specifically, the interface of the computing engine is called to read all files in the target subdirectory, and merge files in the target subdirectory that belong to the acquired data storage type.

[0095] Optionally, all files under a target subdirectory can be combined into one file or multiple files, and can also be set according to requirements.

[0096] So specifically, Figure 3 As shown, a specific implementation of step S104 may include:

[0097] S301 : For each target subdirectory, call a computing engine to read and pass in a data storage type, and read all files under the target subdirectory through the computing engine.

[0098] It should be noted that the data storage type input is the data storage type of the target data table obtained in step S101.

[0099] Optionally, in the embodiment of the present application, the computing engine may adopt the computing engine Spark, and of course other computing engines with data merging functions may also be adopted.

[0100] Specifically, the API interface of the computing engine Spark is called first to pass in the data storage type, so that the computing engine Spark can read all files in the target subdirectory in a corresponding manner according to the passed-in data storage type, and can also merge files according to the data storage type when merging.

[0101] S302: Call the calculation engine to read and input the set number, and use the calculation engine to combine all files in the target subdirectory into a set number of merged files of the data storage type.

[0102] Optionally, the API interface of the computing engine Spark can be called again, specifically the "coalesce()" interface of the computing engine Spark, through which a set number of files are input, and the interface synthesizes all files under the target subdirectory into a set number of merged files of the data storage type.

[0103] The set number is the number of files that need to be generated, so the set number is an integer greater than zero and less than the total number of all files in the target subdirectory.

[0104] Alternatively, as Figure 4 As shown, another specific implementation of step S104 may include:

[0105] S401: traverse the target subdirectories to be processed in the subdirectory list.

[0106] The subdirectory list includes each filtered target subdirectory.

[0107] It should be noted that, in the embodiment of the present application, each target subdirectory is processed in sequence, so the target subdirectories to be processed in the subdirectory list are traversed in sequence to perform subsequent steps.

[0108] The target subdirectory to be processed is a target subdirectory that has not been subjected to the file merging process, that is, a target subdirectory for which step S402 has been executed.

[0109] S402: Calling a computing engine to read all files in the currently traversed target subdirectory, and synthesizing all files in the target subdirectory into a merged file of a data storage type.

[0110] Optionally, the specific implementation of step S402 can be implemented in the manner of the above-mentioned steps S301 and S302, so reference can be made to the above-mentioned corresponding steps and no further details will be given here.

[0111] Optionally, before executing step S402, a temporary directory may be created under the currently traversed target subdirectory to store the merged file obtained by merging the files under the target subdirectory, so as to facilitate subsequent verification, rollback, and replacement of the files under the target subdirectory with the merged file.

[0112] S403: After the processing of the currently traversed target subdirectory is completed, it is determined whether there is any target subdirectory to be processed in the subdirectory list.

[0113] It should be noted that the processing of the target subdirectory is completed, specifically after merging it without replacing the files under the target subdirectory. Of course, it can also be after determining that the file merging of the files under the target subdirectory has failed. At this time, the next target subdirectory can be processed, so it is determined at this time whether there are any target subdirectories to be processed in the subdirectory list.

[0114] If it is determined that there is still a target subdirectory to be processed in the subdirectory list, the process returns to step S401. If it is determined that there is still a target subdirectory to be processed in the subdirectory list, the process proceeds to step S404.

[0115] S404: End the traversal of the subdirectory list.

[0116] S105. Replace all files in the target subdirectory with the merged file.

[0117] Specifically, after all files under a target subdirectory are merged into a merged file, all files under the target subdirectory are deleted, and the merged file is stored in the target subdirectory, thereby achieving file replacement.

[0118] An embodiment of the present application provides a file merging method, which obtains the storage path and data storage type of a target data table, then obtains all subdirectories under the storage path of the target data table, and filters out target subdirectories containing only files from all subdirectories, thereby obtaining subdirectories that can be merged. Then, for each target subdirectory, a computing engine is called to read all files under the target subdirectory, and all files under the target subdirectory are synthesized into a merged file belonging to the data storage type. Finally, the merged file replaces all files under the target subdirectory, thereby merging multiple files in a generated hive data table without changing the file type, effectively reducing the number of files in the generated hive data table. Therefore, for hive data tables generated by various methods, the number of files therein can be effectively reduced.

[0119] Another embodiment of the present application provides another method for merging files, such as Figure 5 Shown, including:

[0120] S501: Obtain the storage path and data storage type of the target data table.

[0121] It should be noted that the specific implementation of step S501 may refer to step S101 in the above method embodiment, and will not be repeated here.

[0122] S502: Acquire all subdirectories under the storage path of the target data table.

[0123] It should be noted that the specific implementation of step S502 may refer to step S102 in the above method embodiment, and will not be repeated here.

[0124] S503: Filter out the target subdirectory from all subdirectories.

[0125] The target subdirectory is a subdirectory that contains only files.

[0126] It should be noted that the specific implementation of step S503 may refer to step S103 in the above method embodiment, and will not be repeated here.

[0127] S504: Create a temporary directory under each target subdirectory.

[0128] It should be noted that in the embodiment of the present application, after the target subdirectories are screened out, steps S504 to S511 are executed for each target subdirectory. Alternatively, the subdirectory list including each target subdirectory may be traversed, and steps S504 to S511 may be executed for each traversed target subdirectory, thereby looping through each target subdirectory in turn.

[0129] It should also be noted that since problems may arise during the merging process, in order to facilitate subsequent verification and rollback, in an embodiment of the present application, a temporary directory is first created under the target subdirectory. The temporary directory is used to store the merged files synthesized from all the files under the target subdirectory.

[0130] S505 : Calling a computing engine to read all files in the target subdirectory, and synthesizing all files in the target subdirectory into a merged file belonging to the data storage type.

[0131] It should be noted that the specific implementation of step S505 may refer to step S104 in the above method embodiment, and will not be repeated here.

[0132] S506: Call the calculation engine to store the merged file in a temporary directory.

[0133] In order to facilitate subsequent verification and rollback, in an embodiment of the present application, after the merged file is generated, the merged file is not directly used to replace the file in the target subdirectory, but is first stored in a temporary directory.

[0134] S507: Verify the validity of the merged file in the temporary directory.

[0135] Since there may be omissions or missing files during the merging process, it is necessary to first perform a validity check on the merged file in the embodiment of the present application. Therefore, it is optional to check whether the format of the merged file and the number of records are consistent with those before the merge.

[0136] For example, suppose the files in the target subdirectory contain 100 records, each containing 10 fields. After obtaining the merged file, it is necessary to verify that the number of records in the merged file is 100 and that each record contains 10 fields. If both conditions are met, the merged file passes the validity check. If either condition is not met, the merged file fails the validity check.

[0137] S508: Determine whether the merged file in the temporary directory passes the validity check.

[0138] If the merged file in the temporary directory fails the validity check, step S509 is executed. If the merged file in the temporary directory passes the validity check, it can be replaced, so step S510 is executed at this time.

[0139] S509: Delete the temporary directory and the merged file in the temporary directory, and feedback an error message.

[0140] Since the merged file in the temporary directory did not pass the validity check, the merge of the files in the target subdirectory failed. Therefore, both the temporary directory and the merged file in the temporary directory are no longer needed, so they need to be deleted. This also facilitates the subsequent merge of the target subdirectory.

[0141] When deleting the temporary directory and the merged files in the temporary directory, an error message will be fed back to the user to let the user know the merge failure in time.

[0142] S510: Replace all files in the target subdirectory with the merged file.

[0143] It should be noted that the specific implementation of step S510 may refer to step S105 in the above method embodiment, and will not be repeated here.

[0144] S511. Clean up the temporary directory under the target subdirectory.

[0145] It should be noted that after the merged file replaces all files in the target subdirectory, the temporary directory and the verified information in the temporary directory are no longer needed, so it is necessary to clean it up, that is, delete the temporary directory and all files in it.

[0146] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] Although the operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0148] Another embodiment of the present application provides a file merging device, such as Figure 6 Shown, including:

[0149] The first acquiring unit 601 is configured to acquire the storage path and data storage type of the target data table.

[0150] The second acquiring unit 602 is configured to acquire all subdirectories under the storage path of the target data table.

[0151] The screening unit 603 is used to screen out a target subdirectory from all subdirectories.

[0152] The merging unit 604 is configured to call a computing engine to read all files in each target subdirectory and synthesize all files in the target subdirectory into a merged file of a data storage type.

[0153] Among them, the calculation engine has the data merging function.

[0154] The replacement unit 605 is configured to replace all files in the target subdirectory with the merged file.

[0155] Optionally, in a file merging device provided in another embodiment of the present application, the screening unit, such as Figure 7 Shown, including:

[0156] The screening sub-unit 701 is used to delete empty directories and multi-level directories in the sub-directories.

[0157] Among them, an empty directory is a subdirectory that does not contain any files, and a multi-level directory is a subdirectory that contains the next level of directory.

[0158] The determining unit 702 is configured to determine each subdirectory that has not been deleted as a target subdirectory.

[0159] Optionally, in another embodiment of the present application, a file merging device further includes:

[0160] The creation unit is used to create a temporary directory under the target subdirectory. The temporary directory is used to store the merged file synthesized from the files under the target subdirectory.

[0161] Optionally, in another embodiment of the present application, a file merging device further includes:

[0162] The storage unit is used to call the calculation engine to store the merged file in a temporary directory.

[0163] Verification unit, used to verify the validity of the merged file in the temporary directory.

[0164] If the merged file in the temporary directory passes the validity check, the replacement unit executes to replace all files in the target subdirectory with the merged file.

[0165] The feedback unit is used to delete the temporary directory and the merged file in the temporary directory and feedback error information when the merged file in the temporary directory fails to pass the validity check.

[0166] Optionally, in another embodiment of the present application, a file merging device further includes:

[0167] The cleaning unit is used to clean up the temporary directory under the target subdirectory after the replacement unit replaces all files under the target subdirectory with the merged file.

[0168] Optionally, in a file merging device provided in another embodiment of the present application, the merging unit, such as Figure 8 Shown, including:

[0169] The reading unit 801 is used to call the calculation engine to read and input the data storage type for each target subdirectory, and read all files under the target subdirectory through the calculation engine.

[0170] The first merging sub-unit 802 is used to call the calculation engine to read and input a set number, and to use the calculation engine to combine all files in the target sub-directory into a set number of merged files of the data storage type.

[0171] Optionally, a merging unit in a file merging device provided in another embodiment of the present application includes:

[0172] The traversal unit is used to traverse the target subdirectory to be processed in the subdirectory list.

[0173] The subdirectory list includes each filtered target subdirectory.

[0174] The second merging sub-unit is used to call the calculation engine to read all files in the target sub-directory currently traversed, and synthesize all the files in the target sub-directory into a merged file of the data storage type.

[0175] The judging unit is used to judge whether there is any target subdirectory to be processed in the subdirectory list after the processing of the currently traversed target subdirectory is completed.

[0176] The return unit is used to return to the traversal unit to traverse the target subdirectory to be processed in the subdirectory list if it is determined whether there is a target subdirectory to be processed in the subdirectory list.

[0177] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the corresponding steps in the above method embodiments, and will not be repeated here.

[0178] Another embodiment of the present application provides an electronic device, such as Figure 9 Shown, including:

[0179] Memory 901 and processor 902 .

[0180] The memory 901 is used to store programs.

[0181] The processor 902 is used to execute the program stored in the memory 901, and when the program is executed, it is specifically used to implement the file merging method provided in any one of the above embodiments.

[0182] Another embodiment of the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is used to implement the file merging method provided in any of the above embodiments.

[0183] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0184] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0185] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for merging files, characterized in that: include: Get the storage path and data storage type of the target data table; Obtain all subdirectories under the storage path of the target data table; Filtering out a target subdirectory from the subdirectory includes: deleting empty directories and multi-level directories in the subdirectory; wherein the empty directory is the subdirectory that does not contain any files; the multi-level directory is the subdirectory that contains a lower-level directory; and determining each of the subdirectories that has not been deleted as a target subdirectory; For each target subdirectory, calling a computing engine to read all files under the target subdirectory, and synthesizing all files under the target subdirectory into a merged file belonging to the data storage type, including: respectively calling the computing engine to read and input the data storage type for each target subdirectory, and reading all files under the target subdirectory through the computing engine; calling the computing engine to read and input a set number, and synthesizing all files under the target subdirectory into the set number of merged files of the data storage type through the computing engine; wherein the computing engine has a data merging function; Replace all files in the target subdirectory with the merged file.

2. The file merging method according to claim 1, characterized in that: Also includes: A temporary directory is created under the target subdirectory; wherein the temporary directory is used to store the merged file synthesized from the files under the target subdirectory.

3. The method for merging files according to claim 2, wherein: Also includes: Calling the computing engine to store the merged file in the temporary directory; Verifying the validity of the merged file in the temporary directory; if the merged file in the temporary directory passes the validity check, replacing all files in the target subdirectory with the merged file; If the merged file in the temporary directory fails the validity check, the temporary directory and the merged file under the temporary directory are deleted, and an error message is fed back.

4. The method for merging files according to claim 3, wherein: Also includes: Clean up the temporary directory under the target subdirectory.

5. The file merging method according to claim 1, characterized in that: For each target subdirectory, calling a computing engine to read all files under the target subdirectory and synthesizing all files under the target subdirectory into a merged file belonging to the data storage type includes: Traversing the target subdirectories to be processed in the subdirectory list; wherein the subdirectory list includes each of the filtered target subdirectories; Calling the computing engine to read all files under the target subdirectory currently traversed, and synthesizing all files under the target subdirectory into a merged file belonging to the data storage type; After the processing of the target subdirectory currently traversed is completed, determining whether the target subdirectory to be processed still exists in the subdirectory list; If it is determined that the target subdirectory to be processed still exists in the subdirectory list, the process returns to execute the traversal of the target subdirectory to be processed in the subdirectory list.

6. A file merging device, characterized in that: include: A first acquiring unit is used to acquire the storage path and data storage type of the target data table; A second acquiring unit is configured to acquire all subdirectories under the storage path of the target data table; A screening unit, configured to screen out a target subdirectory from the subdirectories; The screening unit includes: a screening subunit and a determination unit; The screening sub-unit is used to delete empty directories and multi-level directories in the sub-directories; wherein the empty directories are sub-directories that do not contain any files; and the multi-level directories are sub-directories that contain a directory at the next level; The determining unit is configured to determine each of the subdirectories that has not been deleted as a target subdirectory; a merging unit, configured to call a computing engine to read all files under each target subdirectory and synthesize all files under the target subdirectory into a merged file of the data storage type; wherein the computing engine has a data merging function; The merging unit includes: a reading unit and a first merging sub-unit; The reading unit is configured to call the computing engine to read and input the data storage type for each target subdirectory, and read all files under the target subdirectory through the computing engine; The first merging sub-unit is configured to call a computing engine to read and input a set number, and to use the computing engine to combine all files in the target sub-directory into the set number of merged files of the data storage type; The replacement unit is used to replace all files in the target subdirectory with the merged file.

7. An electronic device, characterized in that: include: memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the file merging method according to any one of claims 1 to 5.

8. A computer storage medium, characterized in that Used to store a computer program, which, when executed, is used to implement the file merging method according to any one of claims 1 to 5.

Images