File cleaning method and device for distributed file system

Abstract

The invention provides a file cleaning method and device for a distributed file system, and the method comprises the steps: obtaining logs generated when all nodes in the distributed file system operate files in a volume in different time periods, and generating a summary file according to the obtained logs, generating time information of the summary file; determining a time condition for cleaning the file; according to the time information of each summary file, determining the summary file meeting a time condition; and determining a to-be-cleaned file according to the target log in the summary file meeting the time condition, and deleting the to-be-cleaned file. According to the method and the device, dispersed logs are integrated according to the time dimension, so that the to-be-cleaned file is determined and deleted according to the time condition, the to-be-cleaned file meeting the time condition can be quickly determined according to the generation time of the summary file, fragmentation polling and recursion operation on a distributed file system cluster are not needed, and the efficiency is improved. And the execution efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a file cleaning method and device for a distributed file system. Background Art

[0002] Distributed file systems have been widely used. However, with the continuous operation and growth of business, a large amount of cold data will remain in the distributed file system, such as reconciliation files. In order to release effective space, cold data needs to be cleaned up regularly. There are two implementation ideas for traditional cleaning solutions: (1) The client mounts the distributed volume and recursively cleans the files. (2) Circularly access the recursive directories of each shard on the server for cleaning. For the above solution (1), when mounting the distributed volume for cleaning, a large number of ls operations need to be performed on the client. When the file directory hierarchy is too deep or the file list is large, the cleaning takes too long. For the above solution (2), when cyclically accessing the recursive directories of each shard on the server for cleaning, the hard links in GlusterFS (hereinafter referred to as GFS) need to be deleted at the same time, otherwise the space cannot be released. It can be seen that the above two methods are complicated to clean up files, and it is impossible to perform refined cleaning according to the file change time. Summary of the invention

[0003] Therefore, the technical problem to be solved by the present invention is to overcome the defect in the prior art that files cannot be processed in a refined manner according to the file change time, thereby providing a file cleaning method and device for a distributed file system.

[0004] A first aspect of the present invention provides a file cleaning method for a distributed file system, comprising:

[0005] The method comprises the following steps: obtaining logs generated by each node in a distributed file system when operating files in a volume in different time periods, and generating a summary file based on the obtained logs. The logs generated by each node in a volume file operation in one period generate a summary file, which includes the logs generated in one period and the time information of generating the summary file; determining the time condition for cleaning files; determining the summary files that meet the time condition based on the time information of each summary file; determining the files to be cleaned in the distributed file system based on the target logs in the summary files that meet the time condition, and deleting the files to be cleaned.

[0006] Optionally, in the file cleaning method of the distributed file system provided by the present invention, a volume contains multiple directories, each directory corresponds to multiple files, and the logs generated by each node in the distributed file system when operating the files in the volume in different time periods are obtained, and a summary file is generated based on the obtained logs, including: obtaining the logs generated by each node in the distributed file system when operating each file in different directories in the volume in different time periods; and generating summary files corresponding to each directory based on the logs generated by each node when operating the files in each directory.

[0007] Optionally, in the file cleaning method of the distributed file system provided by the present invention, the step of determining the time condition for cleaning files includes: determining the time condition for cleaning files corresponding to each directory respectively according to the first cleaning strategy corresponding to each directory respectively.

[0008] Optionally, in the file cleaning method of the distributed file system provided by the present invention, the first cleaning strategy includes a file expiration date, and the time condition for cleaning the file is determined according to the file expiration date.

[0009] Optionally, in the file cleaning method for a distributed file system provided by the present invention, the step of determining the files to be cleaned in the distributed file system according to the target log in the summary file that meets the time condition includes: determining a file name filtering condition according to a second cleaning strategy; if the file name expression of the file recorded in the target log meets the file name filtering condition, determining the file as a file to be cleaned.

[0010] Optionally, in the file cleaning method of the distributed file system provided by the present invention, the target log includes the file path of the file, and the step of deleting the file to be cleaned includes: deleting the file to be cleaned according to the file path of the file to be cleaned recorded in the target log.

[0011] Optionally, in the file cleaning method for a distributed file system provided by the present invention, the target log is generated when the file is created.

[0012] The second aspect of the present invention provides a file cleaning device for a distributed file system, which is applied to a file cleaning server. The device includes: a log summary module, which is used to obtain logs generated by each node in the distributed file system when operating files in a volume in different time periods, and generate a summary file based on the obtained logs. The logs generated when the nodes operate files in the volume in one period generate a summary file, and the summary file includes the logs generated in one period and the time information of generating the summary file; a cleaning condition determination module, which is used to determine the time condition for cleaning files; a summary file determination module, which is used to determine the summary files that meet the time condition based on the time information of each summary file; a file deletion module, which is used to determine the files to be cleaned in the distributed file system based on the target logs in the summary files that meet the time condition, and delete the files to be cleaned.

[0013] The third aspect of the present invention provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, thereby executing the file cleaning method of the distributed file system provided in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the file cleaning method of the distributed file system provided by the first aspect of the present invention.

[0015] The technical solution of the present invention has the following advantages:

[0016] 1. The file cleaning method of the distributed file system provided by the present invention obtains the logs generated when each node in the distributed file system operates the files in the volume in different periods, and generates summary files corresponding to different time periods respectively, which can integrate the relatively scattered logs according to the time dimension, so that after determining the time condition for cleaning the files, the files to be cleaned can be determined and deleted according to the time condition, and when determining the files to be cleaned, the files to be cleaned that meet the time condition can be quickly determined according to the generation time of the summary file, without polling and recursive operation of the distributed file system cluster shards, thereby improving the execution efficiency. Since the logs for generating the summary files are generated when operating the files, the log generation time can be determined according to the generation time of the summary files, thereby determining the file update time. Therefore, after determining the time condition, the logs in the summary files that meet the time condition are determined to be cleaned and deleted, realizing refined cleaning based on the file update time.

[0017] 2. In the file cleaning method for the distributed file system provided by the present invention, the file path is recorded in the log. When deleting a file, the file corresponding to the file path can be directly deleted without the need for a recursive file list, thereby improving execution efficiency.

[0018] 3. The file cleaning method for a distributed file system provided by the present invention determines the cleaning conditions through a preset volume cleaning policy, and different cleaning conditions can be configured in the preset volume cleaning policy, thereby achieving differentiated cleaning of different directories under the same volume.

[0019] 4. The file cleaning device of the distributed file system provided by the present invention obtains the logs generated when each node in the distributed file system operates on the file in the volume in different periods, and generates summary files corresponding to different time periods, which can integrate the relatively scattered logs according to the time dimension, so that after determining the time condition for cleaning the file, the file to be cleaned can be determined and deleted according to the time condition, and when determining the file to be cleaned, the file to be cleaned that meets the time condition can be quickly determined according to the generation time of the summary file, without polling and recursive operation of the distributed file system cluster shards, thereby improving the execution efficiency. Since the log for generating the summary file is generated when operating the file, the log generation time can be determined according to the generation time of the summary file, thereby determining the file update time. Therefore, after determining the time condition, the log in the summary file that meets the time condition is determined to be cleaned and deleted, realizing refined cleaning based on the file update time. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of a specific example of a method for cleaning files in a distributed file system according to an embodiment of the present invention;

[0022] Figure 2 A flowchart of another specific example of a method for cleaning files in a distributed file system according to an embodiment of the present invention;

[0023] Figure 3 It is a principle block diagram of a specific example of a file cleaning device for a distributed file system in an embodiment of the present invention;

[0024] Figure 4 The present invention is a block diagram of a specific example of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In the description of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0027] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] Distributed file systems have been widely used. With the continuous operation and growth of business, a large amount of useless cold data will remain in the distributed file system and need to be cleaned up. The existing file cleaning method is complicated to operate and cannot achieve refined cleaning based on file change time. In order to solve this problem, an embodiment of the present invention provides a file cleaning method for a distributed file system, which can be applied to the GFS distributed file system.

[0029] like Figure 1 As shown, the file cleaning method of the distributed file system provided by the embodiment of the present invention includes:

[0030] Step S11: Obtain the logs generated by each node in the distributed file system when operating files in the volume in different time periods, and generate a summary file based on the obtained logs. The logs generated by the node when operating files in the volume in one period generate a summary file, and the summary file includes the logs generated in one period and the time information of generating the summary file. In a specific embodiment, the log collection function can be enabled in the GFS master node, and the logs generated by each node can be regularly collected to the GFS master node, and synchronized to the file unified cleaning server.

[0031] Normally, GFS logs are continuously generated and are relatively scattered. They are not generated according to periodic date dimensions, and the log content is not readable by users. Therefore, it is difficult to use file update time as a deletion condition.

[0032] In order to solve this problem, in the method provided in the embodiment of the present invention, the logs generated when operating the file in different time periods are collected, and a summary file is generated, and time information is added to the summary file. The time information of the summary file can indicate the generation time of the summary file. The log generation time can be roughly determined through the time information of the summary file, so as to roughly determine the file update time. Specifically, the update time of the file is between the generation time of the summary file recording the file and the generation time of the summary file of the previous cycle of the summary file recording the file. The time period corresponding to a summary file can be adjusted according to actual needs. If it is necessary to determine the update time of the file more accurately, a shorter time period can be set. If the accuracy requirement for determining the file update time is low, the time period can be appropriately extended. For example, in the file cleaning method of the distributed file system provided in the embodiment of the present invention, the time period is determined as one day, and one summary file corresponds to the logs generated in one day, so that the file can be cleaned up with one day as the time unit.

[0033] Step S12: Determine the time condition for cleaning files.

[0034] In an optional embodiment, the time condition for cleaning files may be obtained in different ways. For example, the time condition may be determined by a cleaning instruction with a time condition input by a user, or the time condition for cleaning files may be determined according to a preset configuration.

[0035] Step S13: Determine the summary file that meets the time condition according to the time information of each summary file.

[0036] The time condition may be a time node. When the time condition is a time node, the summary files satisfying the time condition are summary files generated before the time node, and the files recorded in the summary files generated before the time node need to be deleted.

[0037] The time condition can also be the file validity period. When the time condition is the file validity period, the summary file that meets the time condition is the summary file whose duration between the generation time and the current time is greater than the file validity period. The files recorded in this part of the summary file need to be deleted.

[0038] Step S14: Determine the files to be cleaned up in the distributed file system according to the target logs in the summary file that meet the time condition, and delete the files to be cleaned up.

[0039] The log in the summary file is generated when the file is operated. Therefore, the log contains basic information and operation information of the file. The basic information includes the file name, file path, file type, etc. The operation information includes the operation type, such as creation, deletion, modification, etc. When determining the files to be cleaned up according to the log in the summary file, it can be determined according to the basic information of the files recorded in the log. Specifically, in order to avoid the occurrence of multiple files when determining the files to be cleaned up according to the basic information, the basic information used when determining the files to be cleaned up is information that can uniquely identify the files, such as the file path of the files, etc.

[0040] In an optional embodiment, if the log contains a file path, after determining the file to be cleaned, the file under the file path is directly deleted according to the file path of the file to be cleaned, without the need to recursively query the file list to be cleaned, thereby improving execution efficiency.

[0041] In an optional embodiment, when performing operations such as creating, deleting, and modifying files, logs are generated. When clearing files, if the log generated when the file is created is determined as the target log, the file can be cleared according to the time when the file was generated.

[0042] The file cleaning method of the distributed file system provided by the embodiment of the present invention obtains the logs generated by each node in the distributed file system when operating the file in different time periods, and generates a summary file, which can integrate the relatively scattered logs according to the time dimension, so that after determining the time condition for cleaning the file, the file to be cleaned can be determined and deleted according to the time condition, and when determining the file to be cleaned, the file to be cleaned that meets the time condition can be quickly determined according to the generation time of the summary file, without polling and recursive operation of the distributed file system cluster shards, thereby improving the execution efficiency. Since the log generating the summary file is generated when the file is operated, the log generation time can be determined according to the generation time of the summary file, thereby determining the file update time, therefore, after determining the time condition, the log in the summary file that meets the time condition is determined to be cleaned and deleted, and refined cleaning based on the file update time is realized.

[0043] In an optional embodiment, a volume includes multiple directories, each directory corresponds to multiple files, and the above step S11 specifically includes:

[0044] First, logs generated when each node in the distributed file system operates on each file in different directories in the volume in different time periods are obtained.

[0045] Then, according to the logs generated when each node operates the files in each directory, a summary file corresponding to each directory is generated respectively.

[0046] That is, in the embodiment of the present invention, the log generated when operating the files in a directory generates a summary file corresponding to the directory. By generating summary files corresponding to different directories, different directories can be processed in different ways according to the particularity of each directory, thereby realizing refined processing of files.

[0047] In an optional embodiment, in the above step S12, when determining the time condition for cleaning files, the time condition for cleaning files corresponding to each directory is determined according to the first cleaning strategy corresponding to each directory.

[0048] In the embodiment of the present invention, different directories correspond to different first cleaning strategies. Therefore, when processing files in different directories, the obtained time conditions may not be the same.

[0049] In an optional embodiment, the first cleaning strategy includes a file expiration date, and the time condition for cleaning the file is determined according to the file expiration date.

[0050] The first cleanup policy is pre-configured. In a specific embodiment, in addition to the time condition, a variety of different conditions can be configured in the volume cleanup policy, and each single condition can be combined according to actual needs to form a new condition.

[0051] In an optional embodiment, when cleaning files, in addition to determining the time condition for cleaning files according to the first cleaning strategy, the process of determining the files to be cleaned in the above step S14 includes:

[0052] The file name screening condition is determined according to the second cleaning strategy; if the file name expression of the file recorded in the target log meets the file name screening condition, the file is determined as a file to be cleaned.

[0053] Among them, the file name screening condition can be a file name matching rule or a file name exclusion rule. After configuring the preset file name in the second cleaning strategy, when the file name screening condition is the file name matching rule, it means that the files containing the preset file name in the file name will be deleted, and when the file name screening condition is the file name exclusion rule, it means that the files that do not contain the preset file name in the file will be deleted. Exemplarily, there are two files / data / aaa.txt and / test / bbb.txt in the log. If the preset file name "^ / data / " is set and it is a file name matching rule, only / data / aaa.txt will be cleaned up. If the preset file name "^ / data / " is set and it is a file name exclusion rule, only / test / bbb.txt will be cleaned up, thereby achieving more refined processing.

[0054] In a specific embodiment, the process of cleaning files in the distributed file system is as follows: Figure 2 As shown, the following steps are included:

[0055] Step S21: Obtain logs generated when each node in the distributed file system operates on each file in different directories in the volume in different time periods.

[0056] Step S22: Generate summary files corresponding to each directory based on the logs generated when each node operates on the files in each directory.

[0057] Step S23: Determine the time conditions for cleaning files corresponding to each directory according to the first cleaning strategy corresponding to each directory, wherein the first cleaning strategy includes a file expiration date, and the time conditions for cleaning files are determined according to the file expiration date.

[0058] Step S24: Determine the summary file that meets the time condition according to the time information of each summary file.

[0059] Step S25: determining the file name screening condition according to the second cleaning strategy; determining the files whose file name expressions recorded in the target log meet the file name screening condition as files to be cleaned.

[0060] Step S26: Delete the files to be cleaned according to the file paths of the files to be cleaned recorded in the target log.

[0061] The details of step S21 to step S26 refer to the description in the above embodiment and will not be repeated here.

[0062] The embodiment of the present invention also provides a file cleaning device for a distributed file system, which is applied to a file cleaning server, such as Figure 3 As shown, the device comprises:

[0063] The log aggregation module 31 is used to obtain the logs generated by each node in the distributed file system when operating files in the volume in different time periods, and generate a summary file based on the obtained logs. The logs generated when the nodes operate files in the volume in one period generate a summary file, which includes the logs generated in one period and the time information of generating the summary file. For details, please refer to the description in the above embodiment, which will not be repeated here.

[0064] The cleaning condition determination module 32 is used to determine the time condition for cleaning files. For details, please refer to the description in the above embodiment and will not be repeated here.

[0065] The summary file determination module 33 is used to determine the summary file that meets the time condition according to the time information of each summary file. For details, please refer to the description in the above embodiment and will not be repeated here.

[0066] The file deletion module 34 is used to determine the files to be cleaned up in the distributed file system according to the target logs in the summary file that meet the time condition, and delete the files to be cleaned up. For details, please refer to the description in the above embodiment, which will not be repeated here.

[0067] The file cleaning device of the distributed file system provided by the embodiment of the present invention obtains the logs generated when each node in the distributed file system operates on the file in the volume in different periods, and generates summary files corresponding to different time periods respectively, which can integrate the relatively scattered logs according to the time dimension, so that after determining the time condition for cleaning the file, the file to be cleaned can be determined and deleted according to the time condition, and when determining the file to be cleaned, the file to be cleaned that meets the time condition can be quickly determined according to the generation time of the summary file, without polling and recursive operation of the distributed file system cluster shards, thereby improving the execution efficiency. Since the log generating the summary file is generated when the file is operated, the log generation time can be determined according to the generation time of the summary file, thereby determining the file update time, therefore, after determining the time condition, the log in the summary file that meets the time condition is determined to be cleaned and deleted, and refined cleaning based on the file update time is realized.

[0068] The embodiment of the present invention also provides a computer device, such as Figure 4 As shown, the computer device mainly includes one or more processors 51 and a memory 52. Figure 4 A processor 51 is taken as an example.

[0069] The computer device may further include: an input device 53 and an output device 54 .

[0070] The processor 51, the memory 52, the input device 53 and the output device 54 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0071] The processor 51 may be a central processing unit (CPU). The processor 51 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or a combination of the above-mentioned chips. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of a file cleaning device of a distributed file system, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the file cleaning device of the distributed file system through a network. The input device 53 can receive the calculation request (or other digital or character information) input by the user, and generate key signal input related to the file cleaning device of the distributed file system. The output device 54 can include a display device such as a display screen to output the calculation result.

[0072] The embodiment of the present invention also provides a computer-readable storage medium, which stores computer instructions, and the computer storage medium stores computer executable instructions, which can execute the file cleaning method of the distributed file system in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0073] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A file cleaning method for a distributed file system, characterized in that: The method is executed by a file cleaning server and includes: Obtaining logs generated by each node in the distributed file system when operating files in the volume in different time periods, and generating a summary file based on the obtained logs, wherein the logs generated by the node when operating files in the volume in one period generate a summary file, and the summary file includes the logs generated in one period and the time information of generating the summary file; Determine the time conditions for cleaning files; Determining a summary file that meets the time condition according to the time information of each summary file; According to the target log in the summary file that meets the time condition, the files to be cleaned up in the distributed file system are determined, and the files to be cleaned up are deleted.

2. The file cleaning method of a distributed file system according to claim 1, characterized in that: A volume contains multiple directories, each directory corresponds to multiple files. The logs generated when each node in the distributed file system operates on the files in the volume in different time periods are obtained, and a summary file is generated according to the obtained logs, including: Obtain the logs generated by each node in the distributed file system when operating each file in different directories in the volume in different time periods; According to the logs generated when each node operates on the files in each directory, a summary file corresponding to each directory is generated respectively.

3. The file cleaning method of a distributed file system according to claim 2, characterized in that: The steps to determine the time conditions for cleaning files include: According to the first cleaning strategies respectively corresponding to the directories, the time conditions for cleaning files respectively corresponding to the directories are determined.

4. The file cleaning method of a distributed file system according to claim 3, characterized in that: The first cleaning strategy includes a file expiration date, and a time condition for cleaning the file is determined according to the file expiration date.

5. The file cleaning method of a distributed file system according to claim 1, characterized in that: The step of determining the files to be cleaned up in the distributed file system according to the target log in the summary file that meets the time condition comprises: Determine a file name screening condition according to a second cleaning strategy; If the file name expression of the file recorded in the target log meets the file name screening condition, the file is determined as a file to be cleaned.

6. The file cleaning method of a distributed file system according to claim 1, characterized in that: The target log includes a file path of a file, and the step of deleting the file to be cleaned includes: Delete the files to be cleaned according to the file paths of the files to be cleaned recorded in the target log.

7. The file cleaning method of a distributed file system according to any one of claims 1 to 6, characterized in that: The target log is generated when a file is created.

8. A file cleaning device for a distributed file system, characterized in that: Applied to a file cleaning server, the device comprises: A log aggregation module is used to obtain logs generated by each node in the distributed file system when operating files in the volume in different time periods, and generate a summary file based on the obtained logs. The logs generated by the node when operating files in the volume in one period generate a summary file, and the summary file includes the logs generated in one period and the time information of generating the summary file; A cleaning condition determination module is used to determine the time condition for cleaning files; A summary file determination module, used to determine a summary file that meets the time condition according to the time information of each summary file; The file deletion module is used to determine the files to be cleaned up in the distributed file system according to the target logs in the summary file that meet the time condition, and delete the files to be cleaned up.

9. A computer device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to execute the file cleaning method of the distributed file system as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the file cleaning method of the distributed file system according to any one of claims 1 to 7.

Images