Method, device, electronic device and storage medium for managing massive small files
By obtaining system disk configuration information and metadata, guessing directories based on file creation rules, calculating and storing directory sizes, the problem of massive small file management complexity and relying on third-party software in the existing technology is solved, and the massive small file management in lightweight is achieved File management, supports management of multiple directories and different structures.
Patent Information
- Application Number
- CN202410383080.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-01
AI Technical Summary
The existing technology is difficult to effectively manage massive small files without changing the original file structure and inode index nodes, and often requires relying on third-party software or libraries, and the deployment is complex and cannot be independently adapted to the computer system.
By obtaining system disk configuration information and metadata, guessing directories based on file creation rules, verifying and storing directory lists, calculating directory sizes, automatically cleaning directories that exceed thresholds, and using KV database to store directory paths and sizes, realizing lightweight massive small files Management, supports multiple directories and different structures.
It realizes the accurate calculation of disk occupancy of massive small files without relying on third-party software or libraries, automatically controls disk occupancy, supports multi-directory management, and simplifies the deployment and use process.
Smart Images

Figure CN118093531B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer file management and storage, and in particular to a method, device, electronic equipment and storage medium for managing massive small files. Background Art
[0002] The existing technology for managing the massive small files in the computer system is to merge the small files into a large file and store the meta information of the file separately. This solution is relatively complex to deploy and cannot well support the situation where the existing system already has a large number of small files. Therefore, it cannot be used directly and often needs to be redeployed or imported. For some users who do not expect to couple the file system, the solution of merging files is not particularly friendly. Some solutions for managing massive small files require the installation of other third-party software or third-party libraries, and cannot run independently to adapt to the computer system. Some solutions for managing massive small files require changing the original file structure, directory, and inode index node to effectively manage massive small files.
[0003] Therefore, it is necessary to invent a method and device that can be easily deployed and effectively manage a large number of small files without changing the original file structure and inode index node. Summary of the invention
[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present invention provides a method, device, electronic device and storage medium for managing massive small files, which does not rely on third-party software or third-party libraries, is lightweight and easy to deploy and use, and can accurately calculate the disk occupancy of massive small files, automatically control the disk occupancy rate, and support small file management of multiple directories or multiple directories with different structures.
[0005] One aspect of the present invention provides a method for managing a large number of small files, comprising:
[0006] Obtaining the configuration information of the system disk and the metadata of the disk file, reading the data block size of the disk file system from the configuration information, and obtaining the number of data blocks of the disk occupied by each file from the metadata;
[0007] Based on the file creation rules, guess the system file directory, verify the guessed directory, form a directory list of the verified existing directories, and store it in the KV database;
[0008] If the number of files in the lowest directory in the directory list is greater than or equal to the preset number, the size of the disk occupied by the lowest directory is calculated according to the number of files in the lowest directory, the number of disk data blocks occupied by each file, and the data block size of the disk file system; the sizes of the disk occupied by all the lowest directories are summed to obtain the size of the disk occupied by the parent directory of the lowest directory; the directory list is traversed to calculate the size of the disk occupied by each directory in the above manner; if the distance between the last calculation time of the preset directory and the current time is greater than the preset value, the size of the disk occupied by the preset directory is recalculated;
[0009] The calculation results of each directory in the directory list are stored in the KV database, and the KV database includes a Key value representing the directory path and a value representing the directory size; if the size of the disk occupied by the directory in the directory list exceeds a preset threshold, the subdirectory with the earliest creation time under the directory is deleted, and the Key value in the KV database corresponding to the subdirectory is deleted.
[0010] Furthermore, the method further includes: before storing the directory list in the KV database, the method further includes: clearing empty directories in the KV database.
[0011] Furthermore, the method further includes: summing up the size of the disk occupied by each first-level directory to calculate the total size occupied by the disk.
[0012] Furthermore, it also includes: if the number of files under any directory in the directory list is less than a preset number, traversing the files under all directories in the directory list, and calculating the size of the disk occupied by each file based on the number of data blocks of the disk occupied by each file and the data block size of the disk file system.
[0013] Another aspect of the present invention further provides a management device for a large number of small files, comprising:
[0014] An acquisition module is configured to acquire configuration information of a system disk and metadata of a disk file, read a data block size of the disk file system from the configuration information, and acquire the number of data blocks of the disk occupied by each file from the metadata;
[0015] The directory list guessing module is configured to guess the system file directory based on the file creation rules, verify the guessed directory, form a directory list of the verified existing directories, and store it in the KV database;
[0016] The directory size calculation module is configured to calculate the size of the disk occupied by the lowest directory according to the number of files in the lowest directory, the number of disk data blocks occupied by each file and the data block size of the disk file system if the number of files in the lowest directory in the directory list is greater than or equal to the preset number; sum the sizes of the disk occupied by all the lowest directories to obtain the size of the disk occupied by the parent directory of the lowest directory; traverse the directory list and calculate the size of the disk occupied by each directory in the above manner; recalculate the size of the disk occupied by the preset directory if the distance between the last calculation time of the preset directory and the current time is greater than the preset value;
[0017] The directory cleaning module is configured to store the calculation results of each directory in the directory list into a KV database, wherein the KV database includes a Key value representing the directory path and a value representing the directory size; if the size of the disk occupied by a directory in the directory list exceeds a preset threshold, the subdirectory with the earliest creation time under the directory is deleted, and the Key value in the KV database corresponding to the subdirectory is deleted.
[0018] Furthermore, the directory cleaning module is further configured to: clear empty directories in the KV database before storing the directory list in the KV database.
[0019] Furthermore, it also includes: a disk capacity calculation module, which is configured to sum the size of the disk occupied by each first-level directory to calculate the total size occupied by the disk.
[0020] Furthermore, it also includes:
[0021] The file size calculation module is configured to traverse the files in all directories in the directory list if the number of files in any directory in the directory list is less than a preset number, and calculate the size of the disk occupied by each file based on the number of data blocks of the disk occupied by each file and the data block size of the disk file system.
[0022] Another aspect of the present invention provides an electronic device, comprising:
[0023] one or more processors;
[0024] A storage device for storing one or more programs;
[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for managing massive small files.
[0026] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned method for managing a large number of small files when executed by a processor.
[0027] The method, device, electronic device and storage medium for managing massive small files provided by the present invention do not need to rely on third-party software or third-party libraries, are lightweight and easy to deploy and use, can accurately calculate the disk occupancy of massive small files, and can automatically control the disk occupancy rate, and support small file management in multiple directories or multiple directories with different structures. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0029] Figure 1 This is a flow chart of a method for managing massive small files provided by an embodiment of the present application;
[0030] Figure 2 This is a logical flow chart of directory cleaning provided by an embodiment of the present application;
[0031] Figure 3 It is a structural diagram of a device for managing massive small files provided by an embodiment of the present application;
[0032] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0035] It should be understood that although the terms first, second, third, etc. may be used to describe the acquisition modules in the embodiments of the present invention, the acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.
[0036] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0037] It should be noted that the directional words such as "upper", "lower", "left", and "right" described in the embodiments of the present invention are described at the angles shown in the drawings and should not be understood as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is formed "on" or "under" another element, it can not only be formed directly "on" or "under" another element, but also be formed "on" or "under" another element indirectly through an intermediate element.
[0038] See also Figure 1 An embodiment of the present application provides a method for managing a large number of small files, comprising the following steps:
[0039] Step S101, obtaining configuration information of the system disk and metadata of the disk file, reading the data block size of the disk file system from the configuration information, and obtaining the number of data blocks of the disk occupied by each file from the metadata.
[0040] The following takes the log file system EXT4 under the Linux system as an example.
[0041] First, you need to read the relevant configuration information of the system disk. Specifically, you can use the computer system read command to read the data block size and sector size of the disk file system. Usually, when a computer file system reads a file, it will read multiple sectors at a time. Multiple sectors are called a data block. The most common data block size is 4096 bytes, and the sector size is 512 bytes. Eight consecutive sectors constitute a data block.
[0042] In Linux systems, files are composed of metadata and data blocks. Metadata is used to record the creator, creation date, size, etc. of a file. The area that stores file metadata information is called an inode index node. The number of disk data blocks occupied by each file is read from the file metadata. The size of each file on disk is the number of data blocks occupied by the file multiplied by the size of each data block.
[0043] Step S102, based on the file creation rule, guess the system file directory, and verify the guessed directory, form a directory list of the verified existing directories, and store it in the KV database.
[0044] In general, the creation of directories has certain rules, for example: 2021-01-02 / 12 / 88 / abc.jpg, 2021-01-02 / 12 / sw.jpg, a / 12 / sw.jpg, from which it can be seen that directories are mostly named after dates, hours, and numbers in a fixed range, which can be abstracted as: YYYY-MM-DD / [00-23] / [00-99] / **, YYYY-MM-DD / [00-23] / **, etc. According to the above file creation rules, the present invention has a built-in file directory guessing algorithm for guessing the file directory of the system.
[0045] See also Figure 2 Before guessing the system's file directory each time, clear the empty directories in the KV database, that is, delete the parent directory with empty subdirectories in the KV database. An empty subdirectory means there is no file in the subdirectory. The purpose of this is to regularly clean up the data in the KV database, otherwise too many directories will cause each loop to calculate the data in the KV database to take too long.
[0046] The following is an example of directory guessing, taking a directory of the type YYYY-MM-DD / [00-99] / ** as an example:
[0047] The first step is to traverse the directory list of the first-level directory and guess the possible directory results are:
[0048] 2024-01-09 /
[0049] 2024-01-10 /
[0050] 2024-01-11 /
[0051] 2024-01-12 /
[0052] 2024-01-13 /
[0053] 2024-01-14 /
[0054] 2024-01-15 /
[0055] 2024-01-16 /
[0056] 2024-01-25 /
[0057] 2024-01-27 /
[0058] 2024-01-31 /
[0059] 2024-02-01 /
[0060] 2024-02-07 /
[0061] 2024-02-26 /
[0062] 2024-02-28 /
[0063] 2024-02-29 /
[0064] 2024-03-01 /
[0065] The second step is to guess the possible subdirectories based on the pre-configuration
[0066] For example, we guess that there may be a 00 directory under the 2024-01-09 / directory, and then use the stat command to verify whether the 00 directory really exists. If so, record it. If not, continue to determine whether there is a 01 directory, and so on, until all the subdirectories of the first-level directory are traversed and guessed.
[0067] According to the above method, all possible directories are guessed, and the directories that are verified to exist are formed into a directory list and stored in the KV database. The directory list includes the first-level directory and all sub-directories.
[0068] Step S103, if the number of files in the lowest-level directory in the directory list is greater than or equal to the preset number, then calculate the size of the disk occupied by the lowest-level directory based on the number of files in the lowest-level directory, the number of disk data blocks occupied by each file, and the data block size of the disk file system; sum up the sizes of the disk occupied by all lowest-level directories to obtain the size of the disk occupied by the parent directory of the lowest-level directory; traverse the directory list and calculate the size of the disk occupied by each level of directory in the above manner; if the distance between the last calculation time of the preset directory and the current time is greater than the preset value, recalculate the size of the disk occupied by the preset directory.
[0069] Specifically, in order to quickly view the files in any directory of the system, it is necessary to limit the number of files in each directory. If there are too many files in a directory for file-related operations, the system response will become very slow, causing considerable difficulties for operation and maintenance. In order to limit the number of files in a directory, it is necessary to count the size of the disk occupied by any directory, and delete the corresponding files for directories that exceed the preset threshold.
[0070] For further information, see Figure 2, there may be a certain lag in writing files in the directory. For example, a directory is 2024-03-08 / 10 / , where 10 represents 10 am. However, files may be written to directory 10 at 12:00 on March 8. At this time, it is necessary to re-count the files in directory 10. Summarizing the above situation, it can be found that since the data in the last N hours may be written late, it is necessary to consider which directories need to be recalculated. Therefore, it is very necessary to add a flag to the directory guessing algorithm that requires mandatory detection of the directory size in the last N hours, so as to regularly recalculate the size of such directories.
[0071] After the directory guessing is completed, in order to speed up the calculation of the directory size, for directories with relatively small numbers of files or below the preset threshold (such directories are determined according to the configuration), the disk usage of each file in all directories in the directory list is traversed and calculated, and the size of the disk occupied by each file is obtained based on the product of the number of disk data blocks occupied by each file and the data block size of the disk file system.
[0072] In scenarios where there are a large number of small files in a directory, the efficiency of nested traversal becomes low and it takes a long time. For example, it may take several hours to count millions of files. In this case, the size of the disk occupied by the directory can be calculated based on the number of files in the directory, the number of disk data blocks occupied by each file, and the data block size of the disk file system.
[0073] Step S104, storing the calculation results of each directory in the directory list into a KV database, wherein the KV database includes a Key value representing the directory path and a Value value representing the directory size; if the size of the disk occupied by a directory in the directory list exceeds a preset threshold, the subdirectory with the earliest creation time under the directory is deleted, and the Key value in the KV database corresponding to the subdirectory is deleted.
[0074] Specifically, the calculated size of each directory is recorded in the KV database. The KV database adopts the form of KV storage. The KV storage format includes the key value representing the directory path and the value representing the directory size. KV storage is a single file storage based on the golang language and can be adapted to all operating systems.
[0075] For example:
[0076] / data / 2023-11-11 / 06 / 10:905216
[0077] / data / 2023-11-11 / 06 / 11:1003520
[0078] / data / 2023-11-11 / 06 / 12:974848
[0079] / data / 2023-11-11 / 06 / 13:819200
[0080] / data / 2023-11-11 / 06 / 14:1200128
[0081] / data / 2023-11-11 / 06 / 15:995328
[0082] Among them, the keys are: / data / 2023-11-11 / 06 / 10, / data / 2023-11-11 / 06 / 11, / data / 2023-11-11 / 06 / 12, / data / 2023-11-11 / 06 / 13, / data / 2023-11-11 / 06 / 14, / data / 2023-11-11 / 06 / 15. The values are: 905216, 1003520, 974848, 819200, 1200128, 995328.
[0083] When it is necessary to perform statistics on the 2023-11-11 / 06 directory, the above KV storage is traversed, and the value under each key value is summed to obtain the size of the disk occupied by the 2023-11-11 / 06 directory. It should be pointed out that although the size of the 2023-11-11 / 06 directory can also be directly calculated by the calculation method in step S103, if the size of 2023-11-11 / 06 as the parent directory is directly calculated, when the size of the directory is re-counted, the size of all directories must be calculated again, and the calculation time will be very long, and it is impossible to incrementally count the size of the newly added directory.
[0084] See also Figure 2 , limit the number of files in each directory. Preferably, limit the size of the disk occupied by each directory. If the size of the disk occupied by a directory in the directory list exceeds a preset threshold, delete the subdirectory with the earliest creation time under the directory, and delete the Key value in the KV database corresponding to the subdirectory.
[0085] For example, the maximum disk usage of the aa directory is configured, for example: aa / 2024-01-09 / , aa / 2024-01-10 / , aa / 2024-01-11 / , with a maximum of 1G. The maximum disk usage of the aa directory needs to be limited. At this time, the size of the aa directory is calculated based on the statistical size of each subdirectory under the aa directory. If it exceeds 1G, the subdirectories of the aa directory are deleted.
[0086] The specific deletion logic and deletion content include:
[0087] (1) Since the latest data needs to be retained, the oldest data is deleted first, that is, the directory with the earliest creation time is deleted.
[0088] (2) After deleting the directory with the earliest creation time, in order to ensure the accuracy of statistics, it is also necessary to delete the key value in the KV database corresponding to the directory that has been deleted.
[0089] See also Figure 3 Another embodiment of the present invention further provides a management device 200 for massive small files, including an acquisition module 201, a directory list guessing module 202, a directory size calculation module 203, and a directory cleaning module 204. The management device 200 can execute the management method for massive small files in the method embodiment.
[0090] Specifically, the management device 200 for a large number of small files includes:
[0091] The acquisition module 201 is configured to acquire the configuration information of the system disk and the metadata of the disk file, read the data block size of the disk file system from the configuration information, and acquire the number of data blocks of the disk occupied by each file from the metadata;
[0092] The directory list guessing module 202 is configured to guess the system file directory based on the file creation rule, verify the guessed directory, form a directory list with the verified existing directory, and store it in the KV database;
[0093] The directory size calculation module 203 is configured to calculate the size of the disk occupied by the lowest directory according to the number of files in the lowest directory, the number of disk data blocks occupied by each file, and the data block size of the disk file system if the number of files in the lowest directory in the directory list is greater than or equal to the preset number; sum the sizes of the disk occupied by all the lowest directories to obtain the size of the disk occupied by the parent directory of the lowest directory; traverse the directory list and calculate the size of the disk occupied by each directory in the above manner; recalculate the size of the disk occupied by the preset directory if the distance between the last calculation time of the preset directory and the current time is greater than the preset value;
[0094] The directory cleaning module 204 is configured to store the calculation results of each directory in the directory list into a KV database, wherein the KV database includes a Key value representing the directory path and a Value value representing the directory size; if the size of the disk occupied by a directory in the directory list exceeds a preset threshold, the subdirectory with the earliest creation time under the directory is deleted, and the Key value in the KV database corresponding to the subdirectory is deleted.
[0095] Furthermore, the directory list guessing module 202 is further configured to: before storing the directory list in the KV database, clear the empty directories in the KV database.
[0096] Furthermore, it also includes a disk capacity calculation module 205, which is configured to sum the size of the disk occupied by each first-level directory to calculate the total size occupied by the disk.
[0097] Furthermore, it also includes a file size calculation module 206, which is configured to traverse the files in all directories in the directory list if the number of files in any directory in the directory list is less than a preset number, and calculate the size of the disk occupied by each file based on the number of data blocks of the disk occupied by each file and the data block size of the disk file system.
[0098] It should be noted that the device 200 for managing massive small files provided in this embodiment corresponds to a technical solution that can be used to execute various method embodiments, and its implementation principle and technical effects are similar to those of the method, which will not be repeated here.
[0099] Figure 4 A schematic diagram of the structure of an electronic device provided in another embodiment of the present invention, the electronic device is used to implement the method for managing a large number of small files in the method embodiment. The electronic device 300 in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, laptop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0100] like Figure 4 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes to implement the method of the embodiment of the present invention according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage device 308 to the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0101] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0102] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention also provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby realizing the management method of a large number of small files as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present invention are executed.
[0103] Another embodiment of the present invention further provides a computer readable medium, which may be included in the above electronic device or may exist independently without being installed in the electronic device. When the computer program is executed by a processor, the above method for managing a large number of small files can be implemented.
[0104] The above description is only a preferred embodiment of the present invention. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other to form a technical solution.
Claims
1. A method for managing massive small files, characterized in that: include: Obtaining system disk configuration information and disk file metadata, reading the data block size of the disk file system from the configuration information, and obtaining the number of disk data blocks occupied by each file from the metadata; Based on the file creation rules, guess the system file directory and verify the guessed directory. The directory list of the verified existing directories is stored in the KV database. If the number of files in the lowest directory in the directory list is greater than or equal to a preset number, the size of the disk occupied by the lowest directory is calculated based on the number of files in the lowest directory, the number of disk data blocks occupied by each file, and the data block size of the disk file system; Sum the disk sizes occupied by all the lowest-level directories to get the disk sizes occupied by the parent directories of the lowest-level directories. Traverse the directory list and calculate the size of the disk occupied by each level of directory; If the time between the last calculation of the preset directory and the current time is greater than the preset value, the disk size occupied by the preset directory will be recalculated; The calculation result of each directory in the directory list is stored in a KV database, wherein the KV database includes a key value representing the directory path and a value representing the directory size; If the size of the disk occupied by a directory in the directory list exceeds a preset threshold, the subdirectory with the earliest creation time under the directory is deleted, and the key value in the KV database corresponding to the subdirectory is deleted.
2. A method for managing massive small files according to claim 1, characterized in that: Before storing the directory list in the KV database, the process further includes: clearing empty directories in the KV database.
3. A method for managing massive small files according to claim 1, characterized in that: Also includes: Sum the disk size occupied by each first-level directory to calculate the total disk size.
4. A method for managing massive small files according to claim 1, characterized in that: Also includes: If the number of files in any directory in the directory list is less than the preset number, the files in all directories in the directory list are traversed, and the size of the disk occupied by each file is calculated based on the number of data blocks of the disk occupied by each file and the data block size of the disk file system.
5. A device for managing massive amounts of small files, characterized in that: include: an acquisition module configured to acquire configuration information of the system disk and metadata of the disk file, read the data block size of the disk file system from the configuration information, and acquire the number of data blocks of the disk occupied by each file from the metadata; The directory list guessing module is configured to guess the system file directory based on the file creation rules, verify the guessed directory, form a directory list of the verified existing directories, and store it in the KV database; a directory size calculation module configured to calculate the disk size occupied by the lowest directory in the directory list based on the number of files in the lowest directory, the number of disk data blocks occupied by each file, and the data block size of the disk file system if the number of files in the lowest directory in the directory list is greater than or equal to a preset number; Sum the disk sizes occupied by all the lowest-level directories to get the disk sizes occupied by the parent directories of the lowest-level directories. Traverse the directory list and calculate the size of the disk occupied by each level of directory; If the time between the last calculation of the preset directory and the current time is greater than the preset value, the disk size occupied by the preset directory will be recalculated; A directory cleaning module is configured to store the calculation result of each directory in the directory list into a KV database, wherein the KV database includes a key value representing the directory path and a value representing the directory size; If the size of the disk occupied by a directory in the directory list exceeds a preset threshold, the subdirectory with the earliest creation time under the directory is deleted, and the key value in the KV database corresponding to the subdirectory is deleted.
6. A device for managing massive small files according to claim 5, characterized in that: The directory list guessing module is further configured to: clear empty directories in the KV database before storing the directory list in the KV database.
7. A device for managing massive small files according to claim 5, characterized in that: Also includes: The disk capacity calculation module is configured to sum the size of the disk occupied by each first-level directory to calculate the total size occupied by the disk.
8. A device for managing massive small files according to claim 5, characterized in that: Also includes: The file size calculation module is configured to traverse the files in all directories in the directory list if the number of files in any directory in the directory list is less than a preset number, and calculate the size of the disk occupied by each file based on the number of data blocks of the disk occupied by each file and the data block size of the disk file system.
9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for managing massive small files as described in any one of claims 1 to 4.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for managing massive small files as described in any one of claims 1 to 4 is implemented.
