A file storage method, file storage device and computer storage medium

Abstract

The present application proposes a file storage method, a file storage device, and a computer storage medium. The file storage method includes: in response to a file merge instruction, obtaining a submitted first temporary directory; creating a first data set based on the first temporary directory; traversing the first data set, creating a second data set according to small files whose file data volume is less than a preset data volume threshold; aggregating small files in the second data set based on the partition value to create a third data set; merging all small files of each first small file set into a large file according to the third data set and storing it in a second temporary directory; in response to the completion of the merger of all first small file sets, submitting the files of the second temporary directory to the final table directory. Through the above-mentioned file storage method, the technology of the temporary directory is used to ensure data consistency and avoid dirty data problems. The user does not need to configure any parameters, and the small file merger is realized senselessly, and the files are merged to the optimal size during the task execution.

Description

Technical Field

[0001] The present application relates to the technical field of big data analysis, and in particular to a file storage method, a file storage device, and a computer storage medium. Background Art

[0002] Since SparkSQL is based on in-memory computing, full-stage code generation, and efficient Row format, its computing efficiency in offline analysis is significantly higher than other computing engines. More and more businesses choose to use SparkSQL to complete big data analysis tasks.

[0003] SparkSQL completes the calculation of user SQL (Structured Query Language) in a distributed manner. If the total amount of data output is relatively small, a large number of small files will be generated. The existence of a large number of small files in a distributed cluster will increase the pressure on the storage system and reduce storage performance. On the other hand, when downstream tasks process a large number of small files, they need to spend more time to obtain file metadata information, resulting in a decrease in the throughput of computing tasks. Summary of the invention

[0004] In order to solve the above technical problems, the present application proposes a file storage method, a file storage device and a computer storage medium.

[0005] In order to solve the above technical problems, the present application proposes a camera-based file storage method, which includes:

[0006] In response to the file merge instruction, obtaining the submitted first temporary directory;

[0007] Creating a first data set based on the first temporary directory, wherein the first data set stores data file paths and file data volumes of all small files;

[0008] Traversing the first data set, creating a second data set according to the small files whose file data volume is less than a preset data volume threshold, wherein the second data set includes a partition value of each small file;

[0009] Aggregating the small files in the second data set based on the partition value to create a third data set, wherein the third data set includes a set of first small files corresponding to each partition value;

[0010] According to the third data set, all small files of each first small file set are merged into a large file and stored in the second temporary directory;

[0011] In response to completion of merging all first small file sets, the files in the second temporary directory are submitted to the final table directory.

[0012] In order to solve the above technical problems, the present application also proposes another file storage method, which is applied to a file storage system, wherein the file storage system includes a driver node and an executor node; the file storage method includes:

[0013] In response to the file merge instruction, the driver node obtains the submitted first temporary directory;

[0014] The driver node creates a first data set based on the first temporary directory, wherein the first data set stores data file paths and file data volumes of all small files;

[0015] The executor node traverses the first data set, and creates a second data set according to the small files whose file data volume is less than a preset data volume threshold, wherein the second data set includes a partition value of each small file;

[0016] The executor node aggregates the small files in the second data set based on the partition value to create a third data set, wherein the third data set includes a set of first small files corresponding to each partition value;

[0017] The executor node merges all small files of each first small file set into one large file according to the third data set and stores the large file in the second temporary directory;

[0018] In response to completion of merging all first small file sets, the executor node submits the files in the second temporary directory to the final table directory.

[0019] To solve the above technical problems, the present application also proposes a file storage device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the file storage method as described above.

[0020] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above file storage method.

[0021] Compared with the prior art, the beneficial effects of the present application are as follows: the file storage device obtains the submitted first temporary directory in response to the file merge instruction; creates a first data set based on the first temporary directory, wherein the first data set stores the data file paths and file data volumes of all small files; traverses the first data set, and creates a second data set according to the small files whose file data volumes are less than the preset data volume threshold, wherein the second data set includes the partition values ​​of each small file; aggregates the small files in the second data set based on the partition values ​​to create a third data set, wherein the third data set includes the first small file sets corresponding to each partition value; merges all the small files of each first small file set into a large file according to the third data set and stores it in the second temporary directory; in response to the completion of the merger of all first small file sets, submits the files of the second temporary directory to the final table directory. Through the above-mentioned file storage method, the technology of the temporary directory is used to ensure data consistency and avoid dirty data problems, and the user does not need to configure any parameters, and the small file merger is realized senselessly, and the files are merged to the optimal size during the task execution process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0023] Figure 1 It is a flowchart of an embodiment of a file storage method provided by the present application;

[0024] Figure 2 It is a schematic diagram of the overall process of the file storage method provided by this application;

[0025] Figure 3 This is a flowchart of the total number of small files output provided by this application;

[0026] Figure 4 It is a flowchart of the preparation work for merging small files provided by this application;

[0027] Figure 5 This is a flowchart of the small file merging work provided by this application;

[0028] Figure 6 yes Figure 1 A schematic diagram of a specific process of step S15 in the file storage method shown;

[0029] Figure 7 This is a flowchart of the small file merging result submission work and the job-level submission work provided by this application;

[0030] Figure 8 It is a flowchart of another embodiment of the file storage method provided by the present application;

[0031] Fig. 9 It is a structural schematic diagram of an embodiment of a file storage device provided by the present application;

[0032] Fig.10 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] In the SparkSQL technical solution, the SQL submitted by the user will be divided into several minimum execution units (Tasks) and distributed to different computing nodes for execution. If the user submits an INSERT OVERWRITE task, each Task will process a portion of the data in the entire task and generate several files. For example, if the task needs to insert data into a partitioned table (assuming the number of partitions is N), then the small portion of data processed by each Task may contain data from each partition, so each Task may create at least the number of files (N files) of the table partitions.

[0036] Assuming that the total number of tasks in the SQL data writing phase is M, the SQL will eventually generate M×N files. If M or N is large, a large number of files will be generated. Therefore, if the total amount of data output is relatively small, a large number of small files will be generated.

[0037] Generally speaking, the driver node of a distributed computing task needs to obtain the path information of all files in the table in order to divide the SQL into the smallest execution units. If the number of files is large, it will cause huge pressure on the driver node memory, resulting in driver node memory overflow and task failure.

[0038] For SparkSQL, there is currently no small file merging method that does not invade user code, is not noticeable to users, and can ensure data consistency. Therefore, in order to solve this problem, this application provides a SparkSQL adaptive small file merging method, which can automatically and adaptively merge the small files generated by the task into the optimal file size of the distributed file system according to the relevant configuration of the file system during the execution of the SparkSQL task without intruding user SQL and code, and without the user configuring any parameters. The scheme has a high degree of data consistency, and when an exception occurs during the small file merging process, no dirty data will be generated.

[0039] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of an embodiment of a file storage method provided by the present application. Figure 2 It is a schematic diagram of the overall process of the file storage method provided by this application.

[0040] The file storage method of the present application is applied to a file storage device, wherein the file storage device of the present application can be a server, a terminal device, or a system composed of a server and a terminal device. Accordingly, the various parts of the file storage device, such as various units, subunits, modules, and submodules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.

[0041] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.

[0042] like Figure 2As shown, the file storage method provided by this application is mainly divided into four steps, namely preparation, small file merging process, small file merging result submission, and job-level submission. The logic contained in the four steps will be introduced below:

[0043] The preparations for merging small files are as follows:

[0044] Distributed file systems, including but not limited to Hadoop Distrust File System (HDFS), Lustre (a parallel distributed file system), etc., all have the concepts of file blocks and data blocks. Generally speaking, the closer the size of a small file is to the size of a block, the higher the file storage efficiency.

[0045] Therefore, the driver can obtain the block size configuration of the file system according to the type of distributed file system used by the task, which is used as the target size targetSize for merging small files, that is, the preset data volume threshold described later. At the same time, the optimal file size value of the actual distributed file system can also be obtained through performance testing and theoretical derivation, and the method is not limited.

[0046] Specifically, the targetSize set in the present application may be determined by the system performance of the distributed file system, that is, the distributed file system stores the storage solution with the best storage performance according to its storage capacity.

[0047] SparkSQL tasks submitted by users, including but not limited to DataFrame tasks, DataSet tasks, SQL tasks, etc., will create a driver node (Driver) and several executor nodes (Executor) in the cluster. The Driver will parse the user's task logic and divide it into several minimum execution units (Tasks), and then distribute these Tasks to multiple Executors for execution.

[0048] For SparkSQL data writing tasks, each Task will write several files to the distributed file system. Each Task will record the size of the written files, and after execution, report the number of files smaller than the targetSize to the Driver.

[0049] It should be noted that the data written by each Task will first be stored in the Task-level temporary directory. After the Task is successfully executed, the temporary data written by the Task will be moved to the Job-level temporary directory. This process is called Task-level submission. After all Tasks are successfully executed, the files are moved from the Job-level temporary directory to the final data directory. This step is called Job-level submission.

[0050] This application inserts the relevant adaptive small file merging logic after all tasks are successfully executed, after all tasks complete task-level submission, and before job-level submission.

[0051] like Figure 1 As shown, the specific steps are as follows:

[0052] Step S11: In response to the file merge instruction, obtain the submitted first temporary directory.

[0053] In the embodiment of the present application, after the file storage device completes the preparation work for merging small files, if the conditions for file merging are met, a file merging instruction is generated to start the small file merging process.

[0054] The generation logic of the file merging instruction is as follows:

[0055] In response to a data writing task, the file storage device obtains a number of small files written to the distributed file system; counts the number of small files whose file data volume is less than a preset data volume threshold among the several small files; and generates a file merge instruction in response to the number of small files being greater than a preset volume threshold.

[0056] Specifically, the file storage device needs to first count the total number of small files. For details, please refer to Figure 3 , Figure 3 It is a flowchart of counting the total number of small files output provided by this application.

[0057] like Figure 3 As shown in the figure, the Driver side counts the total number of files smaller than the targetSize output by each Task to determine whether the number of small files exceeds the set threshold. The threshold for the number of small files can be obtained through performance testing; it can also be estimated based on parameters such as cluster size and disk media type; or an empirical value can be set, such as 100,000. If the number of small files exceeds the set threshold, the output file format, targetSize and other information are further stored in the Driver context.

[0058] Please continue to combine Figure 3 See also Figure 4 , Figure 4 It is a flowchart of the preparation work for merging small files provided by this application.

[0059] like Figure 4As shown, the file storage device obtains the Block setting size from the distributed file system as the target size targetSize for merging small files. When the Task is executed, the number of files generated that are smaller than the targetSize is returned to the Driver, and the Driver counts the total number of small files smaller than the targetSize generated by all Tasks. If the total number of small files exceeds the set threshold, the file format and targetSize are stored in the small file merging context. If the total number of small files does not exceed the set threshold, the subsequent small file merging process will not be executed.

[0060] It should be noted that when the file storage device determines that the small file merging process needs to be executed, it needs to obtain the format of the output file. This format is mainly used to create a merger for merging small files, that is, each merger merges several identical and corresponding small files.

[0061] Step S12: creating a first data set based on the first temporary directory, wherein the first data set stores the data file paths and file data volumes of all small files.

[0062] In the embodiments of the present application, Figure 2 The small file merging process shown in the figure: Before the job-level submission, the adaptive small file merging process is executed. For details, please refer to Figure 5 , Figure 5 It is a flowchart of the small file merging work provided by this application.

[0063] Specifically, since the files output by each Task will not be written directly to the table directory, a multi-level temporary directory is created, including a Job-level temporary directory and a Task-level temporary directory. The files output by each Task are first written to the Task temporary directory. After the Task is successfully completed, each Task executed in the Executor moves the files to the Job-level temporary directory. After all Tasks are successfully executed, the Driver moves all files in the Job-level temporary directory to the final Hive table directory. Therefore, the file storage method of the present application can insert a merge small file operation after the Task completes the Task-level submission and before the Job-level submission. After the small files output by each Task are merged, the merged files are submitted to the final table directory.

[0064] First, the file storage device obtains the submission directory list of all Tasks on the Driver side, that is, the first temporary directory. Then, the submission directory list of all Tasks is created as the first dataset, that is, a Spark distributed elastic dataset (RDD, Resilient Distributed Datasets) - RDD1[String], where each line records the submission directory of the Task. After the memory data is constructed as a distributed elastic dataset, the distributed computing capability of Spark can be used to merge small files in a distributed parallel manner, which greatly improves the efficiency of small file merging.

[0065] It should be noted that the distributed parallel approach is to assign small files of different sets to several tasks to execute simultaneously.

[0066] Step S13: traverse the first data set, and create a second data set according to the small files whose file data volume is less than a preset data volume threshold, wherein the second data set includes the partition value of each small file.

[0067] In an embodiment of the present application, the file storage device traverses the leaf files in the submission directory of each Task in RDD1 in a distributed manner, filters files smaller than targetSize, and obtains the paths and file sizes of all filtered data files - RDD2[(String, Long)], where each line records a tuple consisting of a data file path and a file size.

[0068] Then, the file storage device extracts the Hive table partition Key (partition value) and bucket ID (bucket number) from the file path in RDD2 as the grouping key of RDD3, and obtains the second data set RDD3[(String, (String, Long))], where each row of data is a tuple, and the first element of the tuple is the grouping key; the second element is a tuple, where the first element is the data file path, and the second element is the file size.

[0069] Step S14: Aggregate the small files in the second data set based on the partition value to create a third data set, wherein the third data set includes a set of first small files corresponding to each partition value.

[0070] In an embodiment of the present application, the file storage device aggregates the data in RDD3 according to the grouping key to obtain a combination of all files under each grouping key. These small files are aggregatable files, and a third data set RDD4[(String,Array((String, Long)))] is obtained, in which each row of data has the first element as the grouping key and the second element as an array, representing the set of all mergeable files under the grouping key.

[0071] Step S15: According to the third data set, all small files in each first small file set are merged into one large file and stored in the second temporary directory.

[0072] In an embodiment of the present application, the file storage device traverses each group in RDD4, and divides the mergeable file set into several to-be-merged file sets according to targetSize using algorithms including but not limited to greedy algorithms.

[0073] The requirement for division is to make the sum of the sizes of small files in each set of files to be merged as close to the targetSize as possible, and obtain the fourth data set RDD5[String, Array(Array(String, Long))], in which the first element of each row of data is the grouping key, the second element is an array, and each element in the array is also an array, which represents a set of files to be merged. All small files in the set can be merged into a new file.

[0074] The file storage device traverses each row of records in RDD5, creates a corresponding file merger according to the file format in the Driver context, and merges the file collections to be merged into the Task-level temporary directory one by one. One file collection to be merged generates one merged file.

[0075] Furthermore, when creating a file merger, the file storage device creates a corresponding merger according to the file format of each file set to be merged, and then divides the file merging task of the file set to be merged into several minimum execution units, namely Tasks, and each Task executes the file merging task of at least one first small file set.

[0076] Furthermore, after the small file merging process is completed, the merged file is generated in the Task level temporary directory and is not submitted to the Job level temporary directory. Among them, the first temporary directory of this application is the Task level temporary directory, and the second temporary directory is the Job level temporary directory.

[0077] The following is an introduction to the submission of small file merging results:

[0078] See also Figure 6 and Figure 7 , Figure 6 yes Figure 1 The specific flow chart of step S15 in the file storage method shown in FIG. Figure 7 This is a flowchart of the small file merging result submission work and the Job-level submission work provided by this application.

[0079] like Figure 6 As shown, the specific steps are as follows:

[0080] Step S151: according to the third data set, all small files in each first small file set are merged into one large file and stored in the first temporary directory.

[0081] In the embodiment of the present application, after the file storage device completes merging a set of small files, it can store the merged large file in a Task-level temporary directory.

[0082] Step S152: Determine whether all first small file sets are merged successfully.

[0083] In the embodiment of the present application, the file storage device determines whether the overall process of merging small files is executed successfully. If it fails, the small file merging process is abandoned, and the Job level submission is directly executed to submit all small files from the Job level temporary directory to the final table directory, and clean up the merged temporary files in the Task level temporary directory, that is, enter step S154. If successful, the merged files are submitted from the Task level temporary directory to the Job level temporary directory, that is, enter step S153.

[0084] Step S153: Submit the content of the first temporary directory to the second temporary directory.

[0085] In the embodiment of the present application, the file storage device determines whether the merged file is successfully submitted from the Task-level temporary directory to the Job-level temporary directory. If it fails, an exception is thrown, causing the task to fail. If successful, try to clean up all small files in RDD2. After the execution is completed, it is determined whether the cleanup operation is successful. If successful, the Job-level submission process continues; if it fails, an exception is thrown, causing the task to fail.

[0086] The file storage device determines whether all small files recorded in RDD2 are successfully deleted. If successfully deleted, the original SparkSQL job-level submission process is continued to submit all data in the job-level temporary directory to the final file directory. If failed, the corresponding exception is thrown to make the task fail, and all files in the job-level temporary directory are cleaned up.

[0087] Step S154: Submit the files in the current second temporary directory to the final table directory.

[0088] In an embodiment of the present application, since the data file after the small files are merged is temporarily stored in the Task-level temporary directory, the Job-level submission will not move the data in the Task-level temporary directory to the final data directory. Therefore, even if there are some merged small files in the Task temporary directory, they will not be moved to the final data directory, so there will be no dirty data problem.

[0089] Step S16: In response to the completion of merging all first small file sets, submit the files in the second temporary directory to the final table directory.

[0090] In the embodiments of this application, please continue to refer to Figure 7 The job-level submission process is a SparkSQL native process that is responsible for moving data from the job-level temporary directory to the final directory, making the data files visible, and cleaning up files in all temporary directories.

[0091] This process is completed by the Driver. Since the small files have been merged into merged files that meet the targetSize, the number of files moved will be greatly reduced. This application can significantly improve the speed of job-level submission.

[0092] In the present application, the file storage device responds to the file merge instruction, obtains the submitted first temporary directory; creates a first data set based on the first temporary directory, wherein the first data set stores the data file path and file data volume of all small files; traverses the first data set, and creates a second data set according to the small files whose file data volume is less than the preset data volume threshold, wherein the second data set includes the partition value of each small file; aggregates the small files in the second data set based on the partition value to create a third data set, wherein the third data set includes the first small file set corresponding to each partition value; merges all the small files of each first small file set into a large file according to the third data set and stores it in the second temporary directory; in response to the completion of the merger of all first small file sets, submits the files of the second temporary directory to the final table directory. Through the above-mentioned file storage method, the technology of the temporary directory is used to ensure data consistency and avoid dirty data problems. The user does not need to configure any parameters, and the small file merger is realized senselessly, and the files are merged to the optimal size during the task execution.

[0093] Please continue reading Figure 8 , Figure 8 It is a flowchart of another embodiment of the file storage method provided by the present application.

[0094] like Figure 8 As shown, the specific steps are as follows:

[0095] Step S31: In response to the file merge instruction, the driver node obtains the submitted first temporary directory.

[0096] Step S32: The driver node creates a first data set based on the first temporary directory, wherein the first data set stores the data file paths and file data volumes of all small files.

[0097] Step S33: the executor node traverses the first data set, and creates a second data set according to the small files whose file data volume is less than a preset data volume threshold, wherein the second data set includes the partition value of each small file.

[0098] Step S34: the executor node aggregates the small files in the second data set based on the partition value to create a third data set, wherein the third data set includes a set of first small files corresponding to each partition value.

[0099] Step S35: the executor node merges all small files of each first small file set into one large file according to the third data set and stores it in the second temporary directory.

[0100] Step S36: In response to completion of merging all first small file sets, the executor node submits the files in the second temporary directory to the final table directory.

[0101] It should be noted that the driver node and executor node in the embodiment of the present application are both processing nodes of the file storage device in the above embodiment. The file storage device executes different file storage processes by the driver node and executor node respectively, thereby improving the efficiency of file storage.

[0102] The file storage method of the present application has a high level of automation. It does not require user participation or any parameter configuration. After configuring the relevant global parameters in the background, all SparkSQL tasks submitted by the user can adaptively merge small files, thereby improving the efficiency of small file merging.

[0103] The file storage method of the present application has high real-time performance for small file merging. The small file merging process is completed before the data file is submitted to the final data directory. When the data is submitted to the data directory, there are no small files. Other downstream tasks can completely avoid the small file problem and do not need to wait for the small file merging process.

[0104] The file storage method of the present application has high data consistency. Considering the possibility that dirty data may be generated due to abnormal failure during the small file merging process, other solutions do not consider this scenario, which may cause dirty data to be generated in the final data directory, resulting in low data consistency.

[0105] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0106] In order to implement the above-mentioned file storage method, the present application also proposes a file storage device, for details, please refer to Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of a file storage device provided by the present application.

[0107] The file storage device 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .

[0108] The processor 41 , the memory 42 , and the input / output device 43 are respectively connected to the bus 44 . The memory 42 stores program data. The processor 41 is used to execute the program data to implement the file storage method described in the above embodiment.

[0109] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.

[0110] This application also provides a computer storage medium, please continue to refer to Fig.10 , Fig.10 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the file storage method of the above embodiment.

[0111] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0112] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A file storage method, characterized in that: The file storage method comprises: In response to the file merge instruction, obtaining the submitted first temporary directory; Creating a first data set based on the first temporary directory, wherein the first data set stores data file paths and file data volumes of all small files; Traversing the first data set, creating a second data set according to the small files whose file data volume is less than a preset data volume threshold, wherein the second data set includes a partition value of each small file; Aggregating the small files in the second data set based on the partition value to create a third data set, wherein the third data set includes a set of first small files corresponding to each partition value; According to the third data set, all small files of each first small file set are merged into a large file and stored in the second temporary directory; In response to completion of merging all first small file sets, the files in the second temporary directory are submitted to the final table directory.

2. The file storage method according to claim 1, characterized in that: After aggregating the small files in the second data set based on the partition value to create the third data set, the file storage method further includes: Dividing each first small file set in the third data set into a plurality of second small file sets according to the preset data volume threshold, to create a fourth data set; The step of merging all small files of each first small file set into a large file according to the third data set and storing the large file in the second temporary directory includes: According to the fourth data set, all small files of each second small file set are merged into one large file and stored in the second temporary directory.

3. The file storage method according to claim 1 or 2, characterized in that: The step of merging all small files of each first small file set into a large file according to the third data set and storing the large file in the second temporary directory includes: Obtaining the file format of the small files in the third data set; Creating a merger based on the file format; The merger is used to merge all the small files in each first small file set into a large file, and the large file is stored in a second temporary directory.

4. The file storage method according to claim 3, characterized in that: The step of combining all the small files in each first small file set into a large file by using the merger, and storing the large file in a second temporary directory includes: Dividing the file merging task of the third data set into a plurality of minimum execution units, wherein the minimum execution unit executes the file merging task of at least one first small file set; Each minimum execution unit performs the following file merging task through an executor node: using the merger to merge all small files of the first small file set into a large file, and storing the large file in a second temporary directory.

5. The file storage method according to claim 1, characterized in that: The step of merging all small files of each first small file set into a large file according to the third data set and storing the large file in the second temporary directory includes: According to the third data set, all small files in each first small file set are merged into one large file and stored in the first temporary directory; Determine whether all first small file sets are merged successfully; If so, submit the contents of the first temporary directory to the second temporary directory; If not, submit the files in the current second temporary directory to the final table directory.

6. The file storage method according to claim 1, characterized in that: The file storage method further includes: In response to the data writing task, a number of small files written to the distributed file system are obtained; Counting the number of small files whose file data volume is less than a preset data volume threshold among the plurality of small files; In response to the number of small files being greater than a preset number threshold, the file merging instruction is generated.

7. The file storage method according to claim 1 or 6, characterized in that: The file storage method further includes: The preset data volume threshold is set based on the system performance of the distributed file system.

8. A file storage method, characterized in that: The file storage method is applied to a file storage system, wherein the file storage system includes a driver node and an executor node; the file storage method includes: In response to the file merge instruction, the driver node obtains the submitted first temporary directory; The driver node creates a first data set based on the first temporary directory, wherein the first data set stores data file paths and file data volumes of all small files; The executor node traverses the first data set, and creates a second data set according to the small files whose file data volume is less than a preset data volume threshold, wherein the second data set includes a partition value of each small file; The executor node aggregates the small files in the second data set based on the partition value to create a third data set, wherein the third data set includes a set of first small files corresponding to each partition value; The executor node merges all small files of each first small file set into one large file according to the third data set and stores the large file in the second temporary directory; In response to completion of merging all first small file sets, the executor node submits the files in the second temporary directory to the final table directory.

9. A file storage device, characterized in that: The file storage device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the file storage method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the file storage method according to any one of claims 1 to 8.

Images