Method, device and equipment for merging files in distributed file system

Abstract

The invention discloses a method, device and equipment for merging files in a distributed file system, and the method comprises the steps: responding to a file merging instruction, and checking whether an execution plan file is stored in the distributed file system or not; if the execution plan file does not exist, creating an execution plan file for merging at least one source file into the target file through metadata service, and persistently storing the execution plan file into the distributed file system; if the execution plan file exists or storage of the execution plan file is completed, obtaining the stored execution plan file from the distributed file system through metadata service; and analyzing and executing the execution plan file by the metadata service, creating a hidden directory under a parent directory of a target file, creating a subdirectory named by a target file identifier under the hidden directory, and moving at least one source file to the subdirectory. Therefore, the problem of low merging efficiency of mechanically reading the data of the source file into the memory and then writing the data into the target file in the prior art is solved.

Description

Background Art

[0002] In a distributed storage system, it is inevitable to face the scenario of generating a large number of small files. As the number of files increases, the file metadata that the storage system needs to maintain will also increase, which will undoubtedly put pressure on the storage system. Therefore, for HDFS (Hadoop Distributed File System), merging files can reduce the pressure of metadata services, improve the performance of processing data read and write requests, and optimize storage space.

[0003] Open source HDFS divides data into fixed-size blocks for maintenance. The concat operation only needs to integrate the metadata of all blocks corresponding to the target file and the source file, and it is based on memory operation, so the speed is very fast. However, it is also a trend to replace the storage base of HDFS with distributed file storage products from mature storage vendors on the market. This can make it unnecessary to ensure data consistency and high availability at the HDFS level, simplify management and reduce costs. In addition, most distributed file storage products on the market support large-scale parallel I / O operations, which can meet the high throughput requirements of HDFS.

[0004] However, after HDFS is lightweight, HDFS does not maintain block information, which means it faces the problem of efficiently merging files. Mechanically reading the data of the source file into the memory and then writing it to the target file is obviously not feasible. Summary of the invention

[0005] The purpose of the present application is to provide a method, device and equipment for merging files in a distributed file system, which is used to solve the problem of low efficiency of the existing method of mechanically reading data of source files into memory and then writing them into target files.

[0006] In a first aspect, an embodiment of the present application provides a method for merging files in a distributed file system, which is applied to a node in the distributed file system, and the method includes:

[0007] In response to the file merge instruction, checking whether an execution plan file is stored in the distributed file system through a metadata service;

[0008] If there is no execution plan file, create an execution plan file that merges at least one source file into the target file through the metadata service and store it persistently in the distributed file system;

[0009] If there is an execution plan file or the storage of the execution plan file is completed, the stored execution plan file is obtained from the distributed file system through the metadata service;

[0010] The metadata service parses and executes the execution plan file, creates a hidden directory under the parent directory of the target file, creates a subdirectory named with the target file identifier under the hidden directory, and moves at least one source file to the subdirectory.

[0011] In the embodiment of the present application, an execution plan file is created through the metadata service in the distributed file system, and different nodes can choose to use the execution plan file, and file merging is performed by creating a hidden directory to move the source file. There is no need to read the data of the source file, and the file movement and merging can be achieved by simply modifying the file index position of the component file. On the premise of using the mature distributed file storage system of the storage manufacturer as the storage base, the file merging operation is cleverly converted into a file moving operation, and efficient file merging is achieved, so that the HDFS file merging operation based on the distributed file storage system has the same performance as the file merging operation of the native HDFS, and the effect is significant for scenes with a large number of file merging operations. This solves the problem of low efficiency of the existing mechanical reading of the source file data into the memory and then writing it into the target file.

[0012] In some possible embodiments, the step of creating an execution plan file for merging at least one source file into a target file through a metadata service includes:

[0013] Determine a target file and at least one source file to be merged;

[0014] Splitting the merging task of merging the at least one source file into the target file into a plurality of subtasks, and generating an execution plan file including the plurality of subtasks;

[0015] Among them, the multiple subtasks include a subtask of creating a hidden directory under the parent directory of the target file, a subtask of creating a subdirectory named with the target file identifier under the hidden directory, and a subtask of merging source files into the target files one by one in a merging order.

[0016] In the embodiment of the present application, when multiple source files are merged into one target file, the movement of one source file corresponds to one subtask, and the execution order of the subtasks is arranged according to the merging order of the source files. The merging order of the source files is determined according to user instructions.

[0017] In some possible embodiments, at least one of the following steps is further included:

[0018] After the execution of the subtask of merging any source file into the target file is interrupted, re-obtaining the execution plan file after the interrupted subtask from the execution plan file and executing it;

[0019] When creating an execution plan file that merges at least one source file into a target file through the metadata service, add write lock lease information for each source file and target file;

[0020] In the process of parsing and executing the execution plan file through the metadata service, applying for renewal according to the write lock lease information, and clearing the renewal after the execution of the execution plan file is completed;

[0021] After the execution of the execution plan file through the metadata service is completed, clean up the obtained execution plan file.

[0022] The embodiment of the present application supports the processing after the interruption of subtasks in the execution plan file, and limits the access rights of different threads to each source file and target file by writing lock lease information, thereby limiting the calling thread in the metadata service to only execute the execution plan when it has access rights; and the execution plan file is cleaned up after the execution is completed to prevent repeated execution.

[0023] In some possible embodiments, moving at least one source file to the subdirectory includes:

[0024] Determine a merge order of at least one source file, and when moving the current source file to the subdirectory according to the merge order, calculate the file size of the target file before merging the current source file;

[0025] Move the current source file to the subdirectory and name it according to the file size.

[0026] In the embodiment of the present application, the target file is named according to the size of the source file before the merged file is moved. The merge order of the files can be determined by the file naming. When reading the merged file, the file is read in the merge order according to the file naming.

[0027] In some possible embodiments, creating a hidden directory under the parent directory of the target file, creating a subdirectory named with the target file identifier under the hidden directory, and moving at least one source file to the subdirectory includes:

[0028] When it is determined that there is no hidden directory under the parent directory of the target file, create a hidden directory under the parent directory of the target file, and create a subdirectory named with the target file identifier under the created hidden directory;

[0029] When it is determined that a hidden directory exists under the parent directory of the target file, a subdirectory named after the target file identifier is created under the hidden directory;

[0030] When it is determined that there is a hidden directory under the same-level directory of any source file, the source file and the files under the subdirectory named with the source file identifier under the hidden directory are read as the files to be merged;

[0031] After completing the movement of any source file, if there is a hidden directory of the same level before the source file is moved, clean up the subdirectory named with the source file identifier under the hidden directory of the same level. If there is no subdirectory under the hidden directory after cleaning, clean up the hidden directory.

[0032] The solution provided by the embodiment of the present application supports file merging in different situations: the target file to be merged belongs to an unmerged file; the target file to be merged belongs to a merged file; the source file to be merged is a merged file; after the source file to be merged is moved, the hidden directory at the same level is cleaned up when the conditions are met.

[0033] In some possible embodiments, the method further includes:

[0034] In response to the file reading instruction, reading a corresponding file according to the file name of the read file;

[0035] When it is determined that there is a hidden directory at the same level as the read file, all files in the subdirectory named with the read file identifier under the hidden directory are obtained.

[0036] The embodiment of the present application is based on the above-mentioned file merging method. Since a hidden directory is created, the user cannot see the merged files under the hidden directory but can only see the target files. In other words, the user is unaware of it. However, during the file reading process, the merged files can be read through the subdirectories under the hidden directory, thereby realizing the reading of the merged files.

[0037] In some possible embodiments, at least one of the following steps is further included:

[0038] The metadata service periodically checks whether the execution plan file is blocked during execution at any node. If there is a blockage and it is not executed at this node, the execution plan file is obtained and re-executed;

[0039] The metadata service polls the execution plan file currently being executed and applies for renewal for the execution plan file being executed.

[0040] The embodiment of the present application determines whether a node is blocked by periodic inspection. If blocked, the planned execution file is obtained and re-executed; at the same time, the executed file is renewed by polling to ensure normal execution.

[0041] In a second aspect, an embodiment of the present application provides a device for merging files in a distributed file system, the device comprising:

[0042] A file checking module, configured to respond to a file merging instruction and check whether an execution plan file is stored in the distributed file system through a metadata service;

[0043] A persistent storage module, used for creating an execution plan file that merges at least one source file into a target file through a metadata service if there is no execution plan file, and storing the execution plan file persistently in a distributed file system;

[0044] A file acquisition module is used to acquire the stored execution plan file from the distributed file system through the metadata service if there is an execution plan file or after the storage of the execution plan file is completed;

[0045] The file merging module is used for the metadata service to parse and execute the execution plan file, create a hidden directory under the parent directory of the target file, create a subdirectory named with the target file identifier under the hidden directory, and move at least one source file to the subdirectory.

[0046] In the third aspect, another embodiment of the present application also provides a device for merging files in a distributed file system, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method for merging files in a distributed file system provided in an embodiment of the present application.

[0047] In a fourth aspect, another embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method for merging files in any distributed file system provided in the embodiments of the present application.

[0048] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A flow chart of a method for merging files in a distributed file system provided according to an embodiment of the present application;

[0051] Figure 2 Schematic diagrams of two situations of generating execution plans for nodes according to embodiments of the present application;

[0052] Figure 3 A schematic diagram of a metadata service maintenance execution plan lifecycle according to an embodiment of the present application;

[0053] Figure 4 A schematic diagram of merging a source file into a target file according to an example of an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a file merging device in a distributed file system provided according to an embodiment of the present application;

[0055] Figure 6 A structural diagram of a file merging device in a distributed file system provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the control device is executed.

[0057] At present, the file merging technology is mainly divided into logical file merging at the metadata level, or relying on concurrent processing platforms such as MapReduce and Spark to merge multiple small files into larger input fragments, which involves reading and rewriting file data. MapReduce is a programming model for parallel computing of large-scale data sets (greater than 1TB), and Spark is a fast and general computing engine designed for large-scale data processing.

[0058] The existing technologies have their own application scenarios, but the above solutions are not suitable for scenarios where the distributed file storage products of storage vendors are used as the storage base of HDFS and the metadata management function of HDFS is weakened. The main problems are as follows:

[0059] 1) Reading the target file and then writing it requires an additional parallel computing platform to allocate the merge task in order to increase the merge rate to a certain extent, which is unreasonable. In addition, although the products launched by storage vendors have good read and write performance, there is still an order of magnitude difference compared to the merge rate of native HDFS based on metadata operations in memory.

[0060] 2) Based on metadata operations, it is necessary to add the weakened metadata management function back to the entire HDFS storage system after the improved storage base, which goes against the original intention of using the storage products of the storage manufacturer.

[0061] In view of the above-mentioned problems existing in the related art, the present application proposes a method, device and equipment for merging files in a distributed file system.

[0062] The relevant technical terms involved in the method for merging files in a distributed file system provided in the embodiment of the present application are explained as follows:

[0063] 1) POSIX protocol: A standard protocol developed by IEEE. POSIX defines a set of interfaces, commands, and tools that enable applications to be easily ported between operating systems that conform to the POSIX standard.

[0064] 2) HDFS: A distributed file system developed by the Apache Hadoop project, designed for large-scale data processing;

[0065] 3) Concat: A command that merges multiple small files in HDFS into a larger file;

[0066] 4) Transaction: A mechanism used to ensure that multiple distributed nodes or components maintain data consistency and integrity when performing a series of operations;

[0067] 5) NameNode: The metadata service of HDFS, responsible for managing metadata.

[0068] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0069] The following is a detailed description of the method for merging files in a distributed file system in an embodiment of the present application in conjunction with the accompanying drawings.

[0070] The present application provides a method for merging files in a distributed file system, which is applied to nodes in a distributed file system, such as Figure 1 As shown, the method includes:

[0071] Step 101, in response to a file merge instruction, checking whether an execution plan file is stored in a distributed file system through a metadata service;

[0072] The metadata service of each node in the distributed file system can interact with the distributed file system, so as to sense whether there is an execution plan file.

[0073] The above file merge instruction is triggered by the user. Through the file merge instruction, at least one source file to be merged and the target file to be merged can be determined, so that it can interact with the distributed file system to determine whether an execution plan file that can complete the merge is currently stored.

[0074] Step 102: If there is no execution plan file, create an execution plan file that merges at least one source file into the target file through the metadata service, and store it persistently in the distributed file system;

[0075] The metadata service of each node has the function of creating an execution plan file. The created execution plan is instantiated by the execution plan file to realize file merging, and the created execution plan file is stored in the distributed file storage system in a persistent storage manner.

[0076] Step 103: if there is an execution plan file or after the storage of the execution plan file is completed, obtain the stored execution plan file from the distributed file system through the metadata service;

[0077] The execution plan file is stored in the distributed file system. When a node determines to merge the execution files, it obtains the execution plan file from the distributed system and executes the execution plan file through the metadata service.

[0078] Step 104, the metadata service parses and executes the execution plan file, creates a hidden directory under the parent directory of the target file, creates a subdirectory named after the target file identifier under the hidden directory, and moves at least one source file to the subdirectory.

[0079] In the embodiment of the present application, an execution plan file is created through the metadata service in the distributed file system, and different nodes can choose to use the execution plan file, and file merging is performed by creating a hidden directory to move the source file. There is no need to read the data of the source file, and the file movement and merging can be achieved by simply modifying the file index position of the component file. On the premise of using the mature distributed file storage system of the storage manufacturer as the storage base, the file merging operation is cleverly converted into a file moving operation, and efficient file merging is achieved, so that the HDFS file merging operation based on the distributed file storage system has the same performance as the file merging operation of the native HDFS, and the effect is significant for scenes with a large number of file merging operations. This solves the problem of low efficiency of the existing mechanical reading of the source file data into the memory and then writing it into the target file.

[0080] In the embodiment of the present application, the above-mentioned moving of at least one source file to the sub-directory is specifically moving the metadata of at least one source file to the sub-directory. The metadata of the source file can index the content of the source file, so the present application does not involve the operation of reading and then writing the content of the source file.

[0081] Specifically, the optimized concat in the embodiment of the present application is a file movement operation based on the POSIX protocol. Because for the file system, the file movement (mv) operation through the metadata service only processes the metadata and does not involve the data movement of the file.

[0082] The distributed file system in the embodiment of the present application can be, but is not limited to, HDFS. The execution plan implemented by the execution plan file complies with the transaction mechanism, ensures the atomicity of the Concat operation, and avoids the generation of data in an intermediate state, which leads to unrecoverable data. The so-called execution plan is to split a complete task into several subtasks, each of which is either successful or failed, and records the completion status of each subtask, and ensures that the entire task can be completed in the end through the fault recovery mechanism.

[0083] In some possible embodiments, the step of creating an execution plan file for merging at least one source file into a target file through a metadata service includes:

[0084] Determine a target file and at least one source file to be merged;

[0085] Splitting the merging task of merging the at least one source file into the target file into a plurality of subtasks, and generating an execution plan file including the plurality of subtasks;

[0086] Among them, the multiple subtasks include a subtask of creating a hidden directory under the parent directory of the target file, a subtask of creating a subdirectory named with the target file identifier under the hidden directory, and a subtask of merging source files into the target files one by one in a merging order.

[0087] In the embodiment of the present application, when multiple source files are merged into one target file, the movement of one source file corresponds to one subtask, and the execution order of the subtasks is arranged according to the merging order of the source files. The merging order of the source files is determined according to user instructions.

[0088] When merging files, the embodiment of the present application realizes file merging by modifying directory information, specifically, creating a hidden directory named .concat under the parent directory of the target file in the distributed file system. The hidden directory is invisible to the user, but visible to the distributed file system. A subdirectory named after the inode number of the target file is created under the .concat directory, the source file is moved to the subdirectory, and the source file is renamed, thereby merging the source file into the target file. Taking the merging of file 1 into merging 2 as an example, the inode number corresponding to file 2 is 2, and a hidden directory named .concat will be created under the parent directory of the file, a subdirectory named 2 will be created under concat, the metadata of file 1 will be moved to the subdirectory, and file 1 will be renamed.

[0089] In some possible embodiments, moving at least one source file to the subdirectory includes: determining a merge order of at least one source file, and when moving the current source file to the subdirectory according to the merge order, calculating the file size of the target file before merging the current source file; and moving the current source file to the subdirectory and naming it with the file size. This ensures that the merged file can be read according to the merge order when it is read.

[0090] In some possible embodiments, the method further includes: in response to the file reading instruction, reading the corresponding file according to the file name of the read file; when it is determined that there is a hidden directory at the same level as the read file, obtaining all files under the subdirectory named with the read file identifier under the hidden directory. Further, when reading all files under the subdirectory, corresponding to the file name named with the file size, the corresponding files are read one by one in the order of file size from small to large.

[0091] Take the example of merging file 1 and file 2 into merging 3. The inode number corresponding to file 3 is 3. A hidden directory named .concat will be created under the parent directory of file 3. A subdirectory named 3 will be created under concat. The metadata of file 1 will be moved to the subdirectory and renamed to the file size of file 3. The metadata of file 2 will be moved to the subdirectory and renamed to the file size corresponding to file 1 + file 3. In this way, when receiving the user's instruction to read file 3, file 3 is read first to determine whether there is a hidden directory named .concat in the same directory as file 3. All files under the subdirectory named 3 under concat are read. The file size can be determined according to the names of all files. According to the order of file size from small to large, the order of reading file 1 first and then reading file 2 can be read, thereby ensuring that the order of data reading is consistent with the order of file merging.

[0092] In some possible embodiments, the metadata service in the embodiments of the present application may be interrupted during the execution of the execution plan file. In this case, after the metadata service is interrupted in executing the subtask of merging any source file into the target file, the execution plan file after the interrupted subtask is re-obtained from the execution plan file and executed, thereby deserializing and generating an execution plan.

[0093] In some possible embodiments, when creating an execution plan file that merges at least one source file into a target file through a metadata service, write lock lease information for each source file and target file is added; the write lock lease information limits the access rights of different threads to each source file and target file, thereby limiting the execution plan to be executed only when the thread in the metadata service is called and has access rights. If a thread in the metadata service does not have access rights to the source file or the target file, a lease conflict is determined and the program exits abnormally. The specific process is as follows: Figure 2 Two methods of generating an execution plan in an embodiment of the present application are shown, mainly including: initially checking whether an execution plan file exists in the distributed file system; if the metadata service is interrupted when executing a subtask of merging any source file into a target file, re-obtaining the execution plan file after the interrupted subtask from the execution plan file and executing it, thereby deserializing and generating an execution plan; if the execution plan file does not exist in the distributed file system, obtaining at least one source file to be merged; adding write lock lease information for each source file and target file; creating an execution plan for merging at least one source file into the target file in the above manner, generating an execution plan file and storing it in the distributed file system; when other nodes access and merge the source files and target files, they can detect a lease conflict based on the write lock lease information and execute an abnormal exit.

[0094] In some possible embodiments, during the process of parsing and executing the execution plan file through the metadata service, renewal is applied for according to the write lock lease information, and the renewal is cleared after the execution of the execution plan file is completed; as mentioned above, each source file and target file has corresponding write lock lease information, so when a thread calls the metadata service to execute the execution plan file, it will apply for renewal according to the write lock lease information, and if it meets the write lock information, it has access rights and successfully achieves renewal. The renewed source files and target files are locked and can no longer be accessed by other threads.

[0095] In some possible embodiments, the metadata service periodically checks whether the execution plan file is blocked during execution at any node. If there is a blockage and it is not executed at this node, the execution plan file is obtained and re-executed. In the embodiment of the present application, HDFS provides many open source services, and the metadata service is one of them. The metadata service includes many threads, including a periodic inspection thread. The periodic inspection threads running on each node are used to identify whether the execution plan of the current distributed system is blocked during execution. If there is a blockage, it is checked whether the execution plan is registered with the NameNode of the local node, that is, whether it has been applied for execution. If not, it triggers the re-acquisition of the execution plan file and execution.

[0096] In some possible embodiments, the metadata service polls the execution plan file currently being executed, and applies for renewal of the execution plan file being executed. In the embodiment of the present application, the metadata service includes a polling thread, which continuously polls the execution plan file currently being executed, and then applies for renewal of the newly operated file, so as to ensure that the access weights of the source file and the target file can be continuously obtained during the process of merging files.

[0097] In some possible embodiments, after the execution of the execution plan file is completed through the metadata service, the acquired execution plan file is cleaned up.

[0098] The life cycle of the execution plan in the embodiment of the present application is maintained by the metadata service (NameNode) of HDFS, such as Figure 3 The figure shows a process diagram of the metadata service maintaining the life cycle of the execution plan, which mainly includes: according to the user's file merge instruction, when it is determined that the corresponding execution plan file does not exist in the distributed file system, the merged file is instantiated according to the above-mentioned method of generating the execution plan file, and the generated execution plan file is persistently stored in the distributed file system; there are two ways to execute the subsequent merge task, one is to obtain the stored execution plan file from the distributed file system to execute the merge task, and in the process of executing the merge task, it is executed in the order of execution of the subtasks and applies for renewal of the new operation file through polling; the other way is to detect whether it is blocked through regular inspection. If there is a blockage in the execution process of the execution plan file on other nodes, the corresponding execution plan file is obtained to execute the merge task; after the task execution is completed, the renewal application for the source file and the target file is cleaned up, and the obtained execution plan file is cleaned up.

[0099] In some possible embodiments, creating a hidden directory under the parent directory of the target file, creating a subdirectory named with the target file identifier under the hidden directory, and moving at least one source file to the subdirectory includes:

[0100] When it is determined that no hidden directory exists under the parent directory of the target file, a hidden directory is created under the parent directory of the target file, and a subdirectory named after the target file identifier is created under the created hidden directory; that is, if there is no .concat hidden directory at the same level as the target file, a .concat hidden directory and a concat directory named after the inodeId are created;

[0101] When it is determined that a hidden directory exists under the parent directory of the target file, a subdirectory named with the target file identifier is created under the hidden directory. If a .concat hidden directory already exists, there is no need to create the .concat hidden directory. For example, if there are files 2 and 3 under directory A, and a .concat hidden directory is created under directory A when file 1 is merged into file 2, there is no need to create the .concat hidden directory when file 4 is merged into file 3.

[0102] When it is determined that there is a hidden directory under the same-level directory of any source file, the source file and the files under the subdirectory named with the source file identifier under the hidden directory are read as files to be merged; the source file to be merged in the embodiment of the present application may be an unmerged file or a merged file. If it is a merged file, then according to the file reading method provided by the present application, after reading the source file, check whether there is a .concat hidden directory at the same level as the source file. If there is a .concat hidden directory, all files under the subdirectory named with the source file identifier under the .concat hidden directory need to be read, that is, the files merged by Concat need to be corrected to the main file (the file carried in the read command) + the subfile in the corresponding subdirectory under the concat hidden directory;

[0103] After completing the move of any source file, if there is a hidden directory at the same level before the source file is moved, clean up the subdirectory named with the source file identifier under the hidden directory at the same level. If there is no subdirectory under the hidden directory after cleaning, clean up the hidden directory. Figure 4As shown, if there are files 7 and 10 in directory A, file 7 merges files 8 and 9, and file 10 merges files 11 and 12, then directory A has files 7, 10, and a hidden directory named .concat, and the hidden directory named .concat has subdirectories named 7 and 10, and the subdirectory named 7 has renamed files 8 and 9, and the subdirectory named 10 has renamed files 11 and 12. There are files 3 and 6 in directory B, file 3 merges files 1 and 2, and file 6 merges files 4 and 5, then directory B has files 3, 6, and a hidden directory named .concat, and the hidden directory named .concat has subdirectories named 3 and 6, and the subdirectory named 3 has renamed files 1 and 2, and the subdirectory named 6 has renamed files 4 and 5. If file 3 needs to be merged into file 7, when reading file 3, if it is found that there is a .concat hidden directory, file 1 and file 2 in the subdirectory named 3 under the .concat hidden directory are read out, so that file 1, file 2 and file 3 are used as files to be merged, and it is determined that there is a .concat hidden directory at the same level as file 7, and there is a subdirectory named 7 under the .concat hidden directory, and file 3, file 1 and file 2 are moved to the subdirectory in order. The method of renaming the files after moving refers to the description of the above embodiment and will not be repeated here.

[0104] The method for merging files in a distributed file system provided in the embodiment of the present application merges a source file into a target file. The source file may be an unmerged file or a merged file. Specifically, it can be determined by determining whether there is a .concat hidden directory. If the source file has not been merged, it is directly read out. If the source file is a merged file, the way to read the file and merge it is described in the above embodiment. The target file may be an unmerged file or a merged file. Specifically, it can be determined by determining whether there is a .concat hidden directory. If it is an unmerged file, it is necessary to create a .concat hidden directory. If it is a merged file, it is not necessary to create a .concat hidden directory. That is, there are the following merging scenarios:

[0105] 1) Merge two unmerged files;

[0106] 2) Merge the unmerged source files into a merged target file;

[0107] 3) Merge the merged source files into an unmerged target file;

[0108] 4) Merge the merged source files into a merged target file.

[0109] For the target file, if it has not been merged, the .concat hidden directory may have been created in the directory at the same level or the .concat hidden directory may not have been created. For the specific processing method, please refer to the description of the above embodiment, which will not be repeated here.

[0110] Based on the same inventive concept, the present application also provides a file merging device in a distributed file system, such as Figure 5 As shown, the device comprises:

[0111] A file checking module 501 is used to check whether an execution plan file is stored in the distributed file system through a metadata service in response to a file merge instruction;

[0112] The persistent storage module 502 is used to create an execution plan file that merges at least one source file into a target file through a metadata service if there is no execution plan file, and store it persistently in a distributed file system;

[0113] The file acquisition module 503 is used to acquire the stored execution plan file from the distributed file system through the metadata service if there is an execution plan file or after the storage of the execution plan file is completed;

[0114] The file merging module 504 is used for the metadata service to parse and execute the execution plan file, create a hidden directory under the parent directory of the target file, create a subdirectory named with the target file identifier under the hidden directory, and move at least one source file to the subdirectory.

[0115] In some possible embodiments, the persistent storage module creates an execution plan file for merging at least one source file into a target file through a metadata service, including:

[0116] Determine a target file and at least one source file to be merged;

[0117] Splitting the merging task of merging the at least one source file into the target file into a plurality of subtasks, and generating an execution plan file including the plurality of subtasks;

[0118] Among them, the multiple subtasks include a subtask of creating a hidden directory under the parent directory of the target file, a subtask of creating a subdirectory named with the target file identifier under the hidden directory, and a subtask of merging source files into the target files one by one in a merging order.

[0119] In some possible embodiments, at least one of the following steps is further included:

[0120] A de-sequence module, for re-obtaining and executing an execution plan file starting from the interrupted subtask from the execution plan file after the subtask of merging any source file into the target file is interrupted;

[0121] A write lock lease module, used to add write lock lease information for each source file and target file when creating an execution plan file for merging at least one source file into a target file through a metadata service;

[0122] A renewal processing module is used to apply for renewal according to the write lock lease information during the process of parsing and executing the execution plan file through the metadata service, and clean up the renewal after the execution plan file is executed;

[0123] The file cleaning module is used to clean up the execution plan file obtained after the execution of the execution plan file through the metadata service is completed.

[0124] In some possible embodiments, the file merging module moves at least one source file to the subdirectory, including:

[0125] Determine a merge order of at least one source file, and when moving the current source file to the subdirectory according to the merge order, calculate the file size of the target file before merging the current source file;

[0126] Move the current source file to the subdirectory and name it according to the file size.

[0127] In some possible embodiments, the file merging module creates a hidden directory under the parent directory of the target file, creates a subdirectory named with the target file identifier under the hidden directory, and moves at least one source file to the subdirectory, including:

[0128] When it is determined that there is no hidden directory under the parent directory of the target file, create a hidden directory under the parent directory of the target file, and create a subdirectory named with the target file identifier under the created hidden directory;

[0129] When it is determined that a hidden directory exists under the parent directory of the target file, a subdirectory named after the target file identifier is created under the hidden directory;

[0130] When it is determined that there is a hidden directory under the same-level directory of any source file, the source file and the files under the subdirectory named with the source file identifier under the hidden directory are read as the files to be merged;

[0131] After completing the movement of any source file, if there is a hidden directory of the same level before the source file is moved, clean up the subdirectory named with the source file identifier under the hidden directory of the same level. If there is no subdirectory under the hidden directory after cleaning, clean up the hidden directory.

[0132] In some possible embodiments, it further includes:

[0133] The file reading module is used to respond to the file reading instruction and read the corresponding file according to the file name of the read file; when it is determined that there is a hidden directory at the same level as the read file, all files in the subdirectory named with the read file identifier under the hidden directory are obtained.

[0134] In some possible embodiments, at least one of the following steps is further included:

[0135] A timed inspection module is used for the metadata service to regularly inspect whether the execution plan file is blocked during execution at any node. If there is a blockage and it is not executed at this node, the execution plan file is obtained and re-executed;

[0136] The timing polling module is used for polling the execution plan file currently being executed by the metadata service, and applying for renewal for the execution plan file being executed.

[0137] After introducing the method and apparatus for merging files in a distributed file system according to an exemplary embodiment of the present application, next, a device for merging files in a distributed file system according to another exemplary embodiment of the present application is introduced.

[0138] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0139] In some possible implementations, the file merging device in the distributed file system according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the file merging method in the distributed file system according to various exemplary implementations of the present application described above in this specification.

[0140] Refer to the following Figure 6 The file merging device 160 in the distributed file system according to this embodiment of the present application is described. Figure 6 The file merging device 160 in the distributed file system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 6As shown, the file merging device 160 in the distributed file system is presented in the form of a general electronic device. The components of the file merging device 160 in the distributed file system may include but are not limited to: the at least one processor 161, the at least one memory 162, and a bus 163 connecting different system components (including the memory 162 and the processor 161).

[0142] Bus 163 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0143] The memory 162 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1621 and / or a cache memory 1622 , and may further include a read-only memory (ROM) 1623 .

[0144] Memory 162 may also include a program / utility 1625 having a set (at least one) of program modules 1624, such program modules 1624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0145] The merging device 160 for files in a distributed file system may also communicate with one or more external devices 164 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the merging device 160 for files in a distributed file system, and / or may communicate with any device that enables the merging device 160 for files in a distributed file system to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 165. Furthermore, the merging device 160 for files in a distributed file system may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 166. As shown, the network adapter 166 communicates with other modules of the merging device 160 for files in a distributed file system via a bus 163. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the merging device 160 for files in a distributed file system, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] In some possible implementations, various aspects of a method for merging files in a distributed file system provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a method for merging files in a distributed file system according to various exemplary implementations of the present application described above in this specification.

[0147] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0148] The program product for merging files in a distributed file system of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0149] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0151] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0152] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0153] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0154] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0155] The present application is described with reference to the flowcharts and block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and block diagram, as well as the combination of the processes and boxes in the flowchart and block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and block diagram. Figure 1 Process or multiple processes and boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 Process or multiple processes and boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0159] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for merging files in a distributed file system, applied to a node in the distributed file system, characterized in that: The method comprises: In response to the file merge instruction, checking whether an execution plan file is stored in the distributed file system through a metadata service; If there is no execution plan file, create an execution plan file that merges at least one source file into the target file through the metadata service and store it persistently in the distributed file system; If there is an execution plan file or the execution plan file is stored, the stored execution plan file is obtained from the distributed file system through the metadata service; The metadata service parses and executes the execution plan file, creates a hidden directory under the parent directory of the target file, creates a subdirectory named with the target file identifier under the hidden directory, and moves at least one source file to the subdirectory.

2. The method according to claim 1, characterized in that The step of creating an execution plan file for merging at least one source file into a target file through a metadata service includes: Determine a target file and at least one source file to be merged; Splitting the merging task of merging the at least one source file into the target file into a plurality of subtasks, and generating an execution plan file including the plurality of subtasks; Among them, the multiple subtasks include a subtask of creating a hidden directory under the parent directory of the target file, a subtask of creating a subdirectory named with the target file identifier under the hidden directory, and a subtask of merging source files into the target files one by one in a merging order.

3. The method according to claim 2, characterized in that It also includes at least one of the following steps: After the execution of the subtask of merging any source file into the target file is interrupted, re-obtaining the execution plan file after the interrupted subtask from the execution plan file and executing it; When creating an execution plan file that merges at least one source file into a target file through the metadata service, add write lock lease information for each source file and target file; In the process of parsing and executing the execution plan file through the metadata service, applying for renewal according to the write lock lease information, and clearing the renewal after the execution of the execution plan file is completed; After the execution of the execution plan file through the metadata service is completed, clean up the obtained execution plan file.

4. The method according to any one of claims 1 to 3, characterized in that: Move at least one source file to the subdirectory, including: Determine a merge order of at least one source file, and when moving the current source file to the subdirectory according to the merge order, calculate the file size of the target file before merging the current source file; Move the current source file to the subdirectory and name it according to the file size.

5. The method according to any one of claims 1 to 3, characterized in that: Creating a hidden directory under the parent directory of the target file, creating a subdirectory named with the target file identifier under the hidden directory, and moving at least one source file to the subdirectory, including: When it is determined that there is no hidden directory under the parent directory of the target file, create a hidden directory under the parent directory of the target file, and create a subdirectory named with the target file identifier under the created hidden directory; When it is determined that a hidden directory exists under the parent directory of the target file, a subdirectory named after the target file identifier is created under the hidden directory; When it is determined that there is a hidden directory under the same-level directory of any source file, the source file and the files under the subdirectory named with the source file identifier under the hidden directory are read as the files to be merged; After completing the movement of any source file, if there is a hidden directory of the same level before the source file is moved, clean up the subdirectory named with the source file identifier under the hidden directory of the same level. If there is no subdirectory under the hidden directory after cleaning, clean up the hidden directory.

6. The method according to claim 1, characterized in that Also includes: In response to the file reading instruction, reading a corresponding file according to the file name of the read file; When it is determined that there is a hidden directory at the same level as the read file, all files in the subdirectory named with the read file identifier under the hidden directory are obtained.

7. The method according to claim 1, characterized in that It also includes at least one of the following steps: The metadata service periodically checks whether the execution plan file is blocked during execution at any node. If there is a blockage and it is not executed at this node, the execution plan file is obtained and re-executed; The metadata service polls the execution plan file currently being executed and applies for renewal for the execution plan file being executed.

8. A device for merging files in a distributed file system, characterized in that: The device comprises: A file checking module, configured to respond to a file merging instruction and check whether an execution plan file is stored in the distributed file system through a metadata service; A persistent storage module, used for creating an execution plan file that merges at least one source file into a target file through a metadata service if there is no execution plan file, and storing the execution plan file persistently in a distributed file system; A file acquisition module is used to acquire the stored execution plan file from the distributed file system through the metadata service if there is an execution plan file or after the storage of the execution plan file is completed; The file merging module is used for the metadata service to parse and execute the execution plan file, create a hidden directory under the parent directory of the target file, create a subdirectory named with the target file identifier under the hidden directory, and move at least one source file to the subdirectory.

9. A file merging device in a distributed file system, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 7.

Images