Cluster file statistical method and device, equipment and storage medium

Abstract

The invention discloses a cluster file statistical method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: for file directories in a cluster, generating a file directory information table corresponding to each file directory according to a first target data structure, and creating a storage information table for recording table names and storage paths of the file directory information tables; monitoring the file through a cluster platform file management entry, and generating a corresponding file management operation data table according to a second target data structure for the monitored file management operation; regularly screening out the changed target file directory information table, and updating the target file directory information table; and querying one or more of the storage information table, the file directory information table and the file management operation data table according to service requirements. The file data retrieval efficiency can be improved, cluster resources are saved, and the cluster performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a cluster file statistics method, device, equipment and storage medium. Background Art

[0002] In order to improve the availability of services and system performance, clusters came into being, and a massive amount of file data is stored in the cluster. In related technologies, the management of massive files usually involves directly traversing all stored files to obtain information such as the size of each file directory. However, traversing each file in turn will increase the input and output (IO) resources of the storage, and will also continuously consume the computing and memory resources of the business service. In addition, the accuracy of the statistical results is not high, and there is a delay in the statistics. Therefore, how to optimize cluster file statistics and reduce resource consumption is an urgent problem to be solved. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a cluster file statistics method, device, equipment and storage medium, which can improve the efficiency of file data retrieval, save cluster resources and improve cluster performance. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a cluster file statistics method, comprising:

[0005] For the file directories in the cluster, generate a file directory information table corresponding to each file directory according to the first target data structure, and create a storage information table for recording the table name and storage path of the file directory information table;

[0006] Monitor the files through the cluster platform file management portal, and generate a corresponding file management operation data table according to the second target data structure for the monitored file management operations;

[0007] Regularly screening out the target file directory information table that has changed, and updating the target file directory information table;

[0008] One or more of the storage information table, the file directory information table, and the file management operation data table are queried according to business requirements.

[0009] Optionally, the target file directory information table that has changed is filtered out periodically, including:

[0010] Filter out a file directory information table that has changed again from the historical target file directory information table, and use it as the target file directory information table; the historical target file directory information table is a file directory information table that has changed and is filtered out in the historical screening operation within a recent preset time period;

[0011] Optionally, the target file directory information table that has changed is filtered out periodically, including:

[0012] The file directory information table with the most recent internal changes within a preset time period is regularly screened out as the target file directory information table.

[0013] Optionally, the cluster file statistics method further includes:

[0014] Periodically reading the creation time of the file management operation data table, and if the time interval between the creation time of the file management operation data table and the current time is greater than a preset time length, deleting the file management operation data table;

[0015] Or, regularly count the storage space occupied by all stored file management operation data tables; if the storage space is larger than a preset space size, determine the size of the data to be deleted based on the difference between the preset space size and the storage space; and select file management operation data tables of the same size as the data to be deleted in order of time from far to near as the data tables to be deleted, and delete the data tables to be deleted.

[0016] Optionally, before generating a corresponding file management operation data table according to the second target data structure, the method further includes:

[0017] Determine the file operation type corresponding to the monitored file management operation according to the correspondence between the file management operation and the file operation type;

[0018] The file management operation data table is generated based on the file operation type.

[0019] Optionally, the cluster file statistics method further includes:

[0020] The storage information table, the file directory information table and the file management operation data table are stored in a lightweight database; the lightweight database is independent of the cluster service database.

[0021] Optionally, the step of regularly screening out the target file directory information table that has changed and updating the target file directory information table includes:

[0022] Filtering the target file directory information table according to a target parameter in the file directory information table; the target parameter is a latest modification time, and the latest modification time is a modification time synchronously obtained when any file in the file directory corresponding to the file directory information table is modified;

[0023] Determine the changed target directory according to the target file directory information table, determine the current directory size of the target directory, and update the corresponding target file directory information table according to the current directory size;

[0024] Determine the levels involved in all target directories that have changed according to the target file directory information table, and obtain the target level;

[0025] According to the file directory information table, the sizes of all directories at the same level corresponding to each target level are counted respectively to obtain the total size of the directories at the same level;

[0026] Determine the previous level of the current target level, and update the file directory information table corresponding to the previous level according to the sum of the sizes of the directories in the same level corresponding to the current target level;

[0027] Until the corresponding file directory information table at the top level is updated.

[0028] In a second aspect, the present application discloses a cluster file statistics device, comprising:

[0029] A file directory information table generation module, used for generating a file directory information table corresponding to each file directory in the cluster according to the first target data structure, and creating a storage information table for recording the table name and storage path of the file directory information table;

[0030] A file management operation data table generation module, used to monitor files through the cluster platform file management portal, and generate a corresponding file management operation data table according to the second target data structure for the monitored file management operations;

[0031] A file directory information table updating module, used to regularly filter out the target file directory information table that has changed, and update the target file directory information table;

[0032] The table query module is used to query one or more of the storage information table, the file directory information table and the file management operation data table according to business requirements.

[0033] In a third aspect, the present application discloses an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] The processor is used to execute the computer program to implement the aforementioned cluster file statistics method.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the aforementioned cluster file statistics method when executed by a processor.

[0037] In the present application, for the file directories in the cluster, a file directory information table corresponding to each file directory is generated according to the first target data structure, and a storage information table for recording the table name and storage path of the file directory information table is created; the files are monitored through the cluster platform file management portal, and the corresponding file management operation data table is generated according to the second target data structure for the monitored file management operation; the target file directory information table that has changed is regularly screened out, and the target file directory information table is updated; one or more of the storage information table, the file directory information table, and the file management operation data table are queried according to business needs. As can be seen from the above, the relevant information of the file directory is stored through the preset data structure, the file management operation is monitored in real time, and the relevant information of the file management operation is stored according to the preset data structure, so that when there is a business demand, the required data can be quickly located by looking up the table, thereby improving the efficiency of file data retrieval; and by only updating the changed target file directory information table, cluster resources can be saved compared to the global traversal method, and the cluster business execution performance can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] FIG1 is a flow chart of a cluster file statistics method provided by the present application;

[0040] FIG. 2 is a specific file management operation statistics flow chart provided by this application

[0041] FIG3 is a schematic diagram of a specific cluster file statistics system architecture provided by the present application;

[0042] FIG4 is a schematic diagram of the structure of a cluster file statistics device provided by the present application;

[0043] FIG5 is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the related art, the management of massive files usually directly traverses all stored files to obtain information such as the size of each file directory. However, traversing each file in turn will increase the storage input and output (IO) resources, and will also continuously consume computing and memory resources of business services. In addition, the accuracy of the statistical results is not high, and there is a delay in the statistics. Therefore, how to optimize cluster file statistics while reducing resource consumption is a problem that needs to be solved urgently. In order to overcome the above technical problems, this application proposes a cluster file statistics method that can improve the efficiency of file data retrieval, save cluster resources and improve cluster performance.

[0046] The present application embodiment discloses a cluster file statistics method, as shown in FIG1 , the method may include the following steps:

[0047] Step S11: for the file directories in the cluster, generate a file directory information table corresponding to each file directory according to the first target data structure, and create a storage information table for recording the table name and storage path of the file directory information table.

[0048] The above cluster may be an AI (artificial intelligence) cluster. The AI ​​cluster is a cluster platform for managing and scheduling computing resources (GPU (Graphics Processing Unit), CPU (Central Processing Unit) and large-scale storage resources. The cluster can support business scenarios such as AI model training on a large scale. With the vigorous development of AI-related industries, more and more research has higher and higher requirements for computing power. The construction of AI cluster platform effectively solves the requirements for computing power. A core function of AI cluster is to manage and use storage with massive data. The storage of AI cluster includes the home directory of AI platform users, public directory, training data set directory and training model directory, etc. Users can operate files through file management tools. With the large number of files, the storage resources of AI cluster are consumed very much. AI platform storage always generates IO operations frequently. Therefore, how to quickly update information on massive files in large-scale cluster storage, shorten the time consumed by AI platform business functions, and not cause the AI ​​platform business functions to affect storage performance has become an urgent problem to be solved in large-scale storage of AI cluster, which is related to model training and production efficiency of AI platform users. In the related technology, the storage statistics of the cluster adopt the quota function provided by the storage itself, such as Nfs, Gpfs, Beegfs and other file storage systems. The advantages of quota are very obvious, with fast statistics and accurate statistical results, but it is limited to storage user size statistics. Its disadvantages are: it cannot count the size of the directory level; the storage environment is simple and unique, and it does not support mounting multiple storages for AI scenarios; it cannot count the size of public directories and data set directories for the size of each business in the AI ​​scenario; it will cause storage performance to decrease, and the statistical management of massive file information in private storage in AI scenarios consumes a lot of storage resources such as network, disk IO, CPU and memory, affecting the efficiency of model training in AI clusters. It can be seen that this solution is not suitable for cluster scenarios.

[0049] In this embodiment, for the file directories in the cluster, a file directory information table corresponding to each file directory will be generated according to the first target data structure, and a storage information table for recording the table name and storage path of the file directory information table will be created, that is, one file directory information table corresponds to one storage information table.

[0050] Specifically, the first target data structure includes information of the directory number information, directory index information, directory last modification time, directory size, last statistical directory size, directory owner, directory level, parent directory, directory information table update time, etc.; wherein each type of information includes an information name field, a parameter field, a field for representing the parameter type, and a field for representing the meaning and purpose of the information. For example, as shown in Table 1 below:

[0051] Table 1: File directory information storage data structure

[0052]

[0053] Among them, for different types of information, each includes the name, meaning and purpose, instance parameter, and parameter type of the instance parameter corresponding to the type of information. Taking the directory number information as an example, the name is id, the meaning and purpose is the directory id number, the instance parameter is 1, and the parameter type is INTEGER (integer). TEXT represents the large object data type.

[0054] The storage information table corresponding to the file directory information table is shown in Table 2 below:

[0055] Table 2: Data structure design of storage information table

[0056]

[0057] The storage information table is used to store the table name and storage path of the corresponding file directory information table.

[0058] It can be seen that according to the database table structure designed in the above manner, firstly, the file directory information will be stored in different tables according to different paths and different storages, which effectively alleviates and reduces the pressure of storing massive data in a single table; secondly, the table name where the file directory information is stored can be obtained by dynamic loading; the storage information table can automatically add and delete tables according to different paths and different storages, preventing table redundancy and reducing the table data occupying the underlying storage disk space.

[0059] Step S12: Monitor the files through the cluster platform file management portal, and generate a corresponding file management operation data table according to the second target data structure for the monitored file management operations.

[0060] In this embodiment, the files in the cluster are monitored in real time through the cluster platform file management portal, and the monitored file management operations are statistically analyzed, and the corresponding file management operation data table is generated according to the second target data structure. It can be understood that the platform file management portal will monitor all file operations and split and refine the file operation steps for storage.

[0061] Specifically, the second target data structure includes: file number information, file index node information, file operation type, file last modification time, parent directory where the file is located, and other types of information; wherein each type of information includes an information name field, a parameter field, a field for characterizing the parameter type, and a field for characterizing the meaning and purpose of the information. Before generating the corresponding file management operation data table according to the second target data structure, it is necessary to first determine the file operation type corresponding to the file management operation. The specific operation is: according to the correspondence between the file management operation and the file operation type, determine the file operation type corresponding to the monitored file management operation; the file management operation includes file upload, file copy, file deletion, file editing, file renaming, file decompression, and file cutting; the file operation type includes creation, deletion, and modification.

[0062] That is, the file management operations in the cluster include: file (folder) upload, file (folder) copy, file (folder) deletion, file editing, file (folder) renaming, decompression, cutting and other operations. The file management of the AI ​​cluster can manage all operation types of different file storages. The file management operations on storage can be divided into three types: creation, deletion and modification. Among them, uploading corresponds to storage creation, copying corresponds to storage deletion and creation, and editing corresponds to storage modification operations. These three operations correspond to all stored file modifications. The correspondence between specific file management operations and file operation types is shown in Table 3 below:

[0063] Table 3: Correspondence between file management operation types and storage operation types

[0064]

[0065] The file management operation data table is shown in Table 4 below:

[0066] Table 4: File management operation data table storage data structure

[0067]

[0068] According to the above database table structure, when the user performs file management operations on the platform, the underlying layer will record the file operation information in the table.

[0069] Taking the AI ​​cluster as an example, all file management operation types are entered through the file management interface of the AI ​​cluster management platform. Some time-consuming file operations will be entered into the progress list. The AI ​​platform file management operation types are followed by corresponding log records, which record the creation, deletion, and modification operations of files on the storage to manage all storage files. This is used for operations such as storage directory size statistics and storage file search to support related business operations of the AI ​​platform, such as disk quotas and file search.

[0070] Specifically, the flow chart of file management operation monitoring in the AI ​​platform is shown in Figure 2. If file operation monitoring is turned on, the storage file operation in the AI ​​cluster is monitored, and the monitored data information is recorded in the database. In the AI ​​platform, file management performs file operations, and the cluster platform calls the file operation API (Application Programming Interface). For example, the file management copy operation enters the file management progress list. After the file copy is completed, the API interface will record two log messages of file deletion and creation in the database.

[0071] In some embodiments, the storage information table, the file directory information table and the file management operation data table are specifically stored in a lightweight database; the lightweight database is independent of the cluster business database. For example, the above lightweight database can be an SQLite3 database. SQLite3 is a lightweight database that implements "directory data storage". It is a relational database management system that complies with atomicity, consistency, independence and persistence, and implements a self-sufficient, serverless, zero-configuration, transactional SQL database engine. That is, all file directory information under AI scene storage is stored in the SQLite3 database. It only needs to store all directory information and perform directory information management. It is not necessary to store file information. The data directory information is used to store and record storage directory structure information. For storage directory management and application, by obtaining the directory information in the SQLite3 database table, since SQLite3 is a file type file library, files can be migrated at any time without configuring the database and starting components. All operation information about file management is stored in the QLite3 database. The QLite3 database is a file type database that can be quickly backed up and migrated. It is independent of the AI ​​platform business database and does not affect the read and write performance of the AI ​​platform business database.

[0072] In some embodiments, in order to prevent the data table information content from increasing, automatic cleaning is adopted, for example, only one day (24 hours) of data table information content is retained to prevent data table redundancy, reduce the table data occupying the underlying storage disk space, and improve the table data read and write performance. Specifically, through time judgment, if the time interval between the creation time of the file management operation data table and the current time is greater than the preset time length, the file management operation data table is deleted.

[0073] In some embodiments, data cleaning can also be performed through the total space occupied by all file management operation data tables. Specifically: the storage space occupied by all stored file management operation data tables is counted. If the storage space is larger than the preset space size, the size of the data to be deleted is determined based on the difference between the preset space size and the storage space; in order from far to near in time, the file management operation data tables of the same size as the size of the data to be deleted are screened out as the data tables to be deleted; and the data tables to be deleted are deleted. That is, by regularly counting the size of all file management operation data tables, when the preset space size is exceeded, the size of the excess part is determined, and some files are selected from the oldest files in chronological order for deletion, so that the total space of all file management operation data tables after deletion is smaller than the preset space size. This can avoid data redundancy occupying the underlying storage disk space, while improving the table data reading and writing performance.

[0074] Step S13: regularly screening out the target file directory information table that has changed, and updating the target file directory information table.

[0075] Since the file directory information table changes with the changes of files, it is necessary to update the file directory information table regularly. The specific update cycle can be customized according to the situation. The higher the update frequency, the more accurate the information obtained by querying the file directory information table; the lower the update frequency, the less computing resources are consumed.

[0076] In some embodiments, the periodic screening of the target file directory information table that has changed and updating the target file directory information table may specifically include the following steps:

[0077] S201: Filter out the target file directory information table according to the target parameter in the file directory information table; the target parameter is the latest modification time, and the latest modification time is the modification time synchronized when any file in the file directory corresponding to the file directory information table is modified the last time.

[0078] It is understandable that after a file is updated by a management operation, the last modification time in the file directory information table of the directory where the file is directly located will be automatically updated. Therefore, by querying the file directory information table, the target directory that has changed can be quickly located, and then the directory information can be updated.

[0079] S202: Determine the changed target directory according to the target file directory information table, determine the current directory size of the target directory, and update the corresponding target file directory information table according to the current directory size.

[0080] After determining the changed target file directory information table, the current directory size of the target directory corresponding to the target file directory information table is read, so as to update the target file directory information table according to the current directory size.

[0081] S203: Determine the levels involved in all target directories that have changed according to the target file directory information table, and obtain the target level.

[0082] S204: According to the file directory information table, the sizes of all directories at the same level corresponding to each target level are counted respectively to obtain the total size of the directories at the same level.

[0083] S205: Determine the upper level of the current target level, and update the file directory information table corresponding to the upper level according to the sum of the sizes of the directories in the same level corresponding to the current target level.

[0084] S206: until the corresponding file directory information table at the top level is updated.

[0085] Update the changed directories. According to the characteristics of the file operating system, if the subdirectories and subfiles of a directory change, the last modification time of the directory will change. A bottom-up method is used for storage traversal. This traversal method needs to be implemented in combination with stack and queue data structures. To determine whether the last modification time of a directory has changed, all directories in the storage need to be stored in the database.

[0086] Count the file sizes of the current layer (i.e. a certain layer) of all changed directories, and sum them with the sizes of the sub-directories at the same layer, sort all the obtained directories by the number of layers, and put the directories that are parent and child directories together, for example, / a has 1 layer, / a / b has 2 layers, / a / b / c has 3 layers, and sort them from large to small. In this way, there is no need to find the smallest sub-directory, just go up from the smallest sub-directory to the largest sub-directory, and finally update to the public root directory. Finally, calculate the latest directory information, and directly update the changed directory related information in the storage directory table, without traversing all the storage directory information, to achieve the purpose of quickly updating and maintaining the storage directory information.

[0087] That is to say, when a directory changes, the parent directory of the directory also needs to be updated. This application implements the update of each layer of directories by updating upward. The directory is divided into levels. First, the sum of the directory sizes of all directories contained in the current target level is counted, and then the size of the previous directory level is updated accordingly. For example, under the a1 directory is the b level, and the b level contains n directories, b1...bn. According to the sum of the sizes of each directory of these n directories, the a1 directory is updated. And so on until the corresponding file directory information table at the top level is updated.

[0088] In addition, if it is determined by reading the directory that there is no sub-directory currently in the directory, it is necessary to query whether there is sub-directory information in the file directory information table, and if so, delete it to update the sub-directory related information.

[0089] In some embodiments, the periodic screening of the target file directory information table that has changed may include: screening out the file directory information table that has changed again from the historical target file directory information table as the target file directory information table; the historical target file directory information table is the file directory information table that has changed that has been screened out in the historical screening operation within the most recent preset time period. It is understandable that since there are fewer directories that have changed, many directories will basically not change, and some directories will change frequently. By marking the file directory information table of the directories that have changed that have been screened out historically, and then screening them out regularly to see if these directories that have changed historically have changed again, it can be seen that repeated traversal of the storage directory is reduced, all directory information is quickly synchronized to the database, the update delay problem is reduced, and the storage performance of the platform is quickly improved.

[0090] In some embodiments, the periodic screening of the target file directory information table that has changed includes: periodically screening the target file directory information table that has undergone endogenous changes within the recent preset time period. It is understandable that since the full traversal storage is very time-consuming, it is necessary to traverse the changed directories separately and update the directory information. In order to reduce the storage of all directory traversals, another scheduled task is started to perform directory traversal updates within a recent period of time, which is independent of the current storage incremental statistics scheduled task. For example, the acquisition duration is configured in days, the default is 1 day (i.e., push forward 24 hours), and all changed directories are obtained. Since there are fewer changed directories, many directories will basically not change, and some directories will change frequently. By only screening the directories that have undergone endogenous changes within the recent preset time period, it can be seen that repeated traversals of storage directories are reduced, all directory information is quickly synchronized to the database, and update delays are reduced.

[0091] Step S14: query one or more of the storage information table, the file directory information table and the file management operation data table according to business requirements.

[0092] For example, as shown in Figure 3, the cluster storage management service directly queries the storage information table, the file directory information table or the file management operation data table from the database according to the business needs, and directly obtains the query data from the table. It can efficiently and quickly support the business functions related to the AI ​​platform. For the storage statistics of the AI ​​platform, the disk directory size and the user disk usage size can be obtained by directly looking up the table. The directory information is updated only for the first time, and the subsequent directory updates are partial and small updates, which reduces the large amount of time consumed by storage traversal; at the same time, the AI ​​platform file management can also quickly locate the directory location according to the directory information in the storage table, and apply it to the storage directory retrieval function; this storage directory design method based on file management can also be applied to many AI platform business-related scenarios, such as monitoring whether the storage directory information has changed, supporting the precise management of business module files, algorithm files, and data set files, making the use of AI platform business functions more efficient and intelligent.

[0093] Through the AI ​​platform file management monitoring operation files and directories, the SQLite3 database is used to set the log file table data structure, and the file management operation directory is traversed through the scheduled task, the information of each directory is quickly traversed and stored, and the SQLite3 database table data structure is used to update the information of each directory; the AI ​​cluster resources are reasonably used to improve the storage update directory speed, and finally, after the storage directory is updated, it can be applied to business areas such as disk directory statistics, user disk quota verification, model directory file monitoring, and data set directory file monitoring. Rationally and fully use cluster storage resources to reduce cluster storage load pressure, reduce the long-term occupation and consumption of business module resources, and enhance the stability of AI cluster platform storage. It is applicable to a variety of storage directory management scenarios, can greatly improve the efficiency of file management operations, improve the management efficiency of directories in AI cluster storage, improve the stability of efficient operation of AI cluster storage, and improve the work efficiency of algorithm personnel using AI cluster platform, which is beneficial to expand cluster scale and reduce production costs. By saving directory statistics resources, the time for algorithm personnel to train models can be effectively shortened, the storage performance of the AI ​​platform can be improved, and the problem of monitoring, management and statistics of massive storage file directories can be solved. At the same time, the performance pain points of frequent interactive statistics of the network and IO of the AI ​​cluster are solved, the storage file operation and management performance is improved, and the overall utilization rate of resources occupied by AI cluster services is reduced, making the use of AI cluster storage smoother.

[0094] As can be seen from the above, in this embodiment, for the file directories in the cluster, a file directory information table corresponding to each file directory is generated according to the first target data structure, and a storage information table for recording the table name and storage path of the file directory information table is created; the files are monitored through the cluster platform file management portal, and the corresponding file management operation data table is generated according to the second target data structure for the monitored file management operation; the target file directory information table that has changed is regularly screened out, and the target file directory information table is updated; one or more of the storage information table, the file directory information table, and the file management operation data table are queried according to business needs. As can be seen from the above, the relevant information of the file directory is stored through the preset data structure, the file management operation is monitored in real time, and the relevant information of the file management operation is stored according to the preset data structure, so that when there is a business demand, the required data can be quickly located by looking up the table, thereby improving the efficiency of data retrieval; and by only updating the changed target file directory information table, cluster resources can be saved compared to the global traversal method, and the cluster business execution performance can be improved.

[0095] Correspondingly, the embodiment of the present application further discloses a cluster file statistics device, as shown in FIG4 , the device includes:

[0096] A file directory information table generating module 11 is used to generate a file directory information table corresponding to each file directory in the cluster according to the first target data structure, and to create a storage information table for recording the table name and storage path of the file directory information table;

[0097] A file management operation data table generating module 12, which is used to monitor files through the cluster platform file management portal, and generate a corresponding file management operation data table according to the second target data structure for the monitored file management operations;

[0098] A file directory information table updating module 13 is used to regularly filter out the target file directory information table that has changed, and update the target file directory information table;

[0099] The table query module 14 is used to query one or more of the storage information table, the file directory information table and the file management operation data table according to business requirements.

[0100] As can be seen from the above, in this embodiment, for the file directories in the cluster, a file directory information table corresponding to each file directory is generated according to the first target data structure, and a storage information table for recording the table name and storage path of the file directory information table is created; the files are monitored through the cluster platform file management portal, and the corresponding file management operation data table is generated according to the second target data structure for the monitored file management operation; the target file directory information table that has changed is regularly screened out, and the target file directory information table is updated; one or more of the storage information table, the file directory information table, and the file management operation data table are queried according to business needs. As can be seen from the above, the relevant information of the file directory is stored through the preset data structure, the file management operation is monitored in real time, and the relevant information of the file management operation is stored according to the preset data structure, so that when there is a business demand, the required data can be quickly located by looking up the table, thereby improving the efficiency of data retrieval; and by only updating the changed target file directory information table, cluster resources can be saved compared to the global traversal method, and the cluster business execution performance can be improved.

[0101] In some specific embodiments, the file directory information table updating module 13 may specifically include:

[0102] The file directory information table updating unit is used to filter out the file directory information table that has changed again from the historical target file directory information table as the target file directory information table; the historical target file directory information table is the file directory information table that has changed and is filtered out in the historical screening operation within the most recent preset time period.

[0103] In some specific embodiments, the file directory information table updating module 13 may specifically include:

[0104] The file directory information table updating unit is used to regularly filter out the file directory information table with the most recent internal changes within a preset period of time as the target file directory information table.

[0105] In some specific embodiments, the cluster file statistics device may specifically include:

[0106] The file operation type determination unit is used to determine the file operation type corresponding to the monitored file management operation according to the correspondence between the file management operation and the file operation type before generating the corresponding file management operation data table according to the second target data structure.

[0107] In some specific embodiments, the cluster file statistics device may further include:

[0108] A first deleting unit, configured to periodically read the creation time of the file management operation data table, and if the time interval between the creation time of the file management operation data table and the current time is greater than a preset time length, delete the file management operation data table;

[0109] The second deletion unit is used to periodically count the storage space occupied by all stored file management operation data tables. If the storage space is larger than a preset space size, the size of the data to be deleted is determined according to the difference between the preset space size and the storage space. The file management operation data tables with the same size as the size of the data to be deleted are screened out in order from far to near in time as the data tables to be deleted, and the data tables to be deleted are deleted.

[0110] In some specific embodiments, the cluster file statistics device may further include:

[0111] A storage unit is used to store the storage information table, the file directory information table and the file management operation data table in a lightweight database; the lightweight database is independent of the cluster business database.

[0112] In some specific embodiments, the file directory information table updating module 13 may specifically include:

[0113] a target file directory information table determination unit, configured to filter out the target file directory information table according to a target parameter in the file directory information table; the target parameter is a last modification time, and the last modification time is a modification time synchronously obtained when any file under the file directory corresponding to the file directory information table is modified;

[0114] A file directory information table updating unit, configured to determine a changed target directory according to the target file directory information table, determine a current directory size of the target directory, and update the corresponding target file directory information table according to the current directory size;

[0115] A target level determination unit, configured to determine the levels involved in all target directories that have changed according to the target file directory information table, and obtain a target level;

[0116] a same-layer directory size sum determination unit, configured to count the sizes of all the same-layer directories corresponding to each of the target levels according to the file directory information table, and obtain the sum of the sizes of the same-layer directories;

[0117] The file directory information table updating unit is used to determine the previous level of the current target level, and update the file directory information table corresponding to the previous level according to the sum of the sizes of the same-level directories corresponding to the current target level, until the file directory information table corresponding to the top level is updated.

[0118] Furthermore, an embodiment of the present application also discloses an electronic device, as shown in FIG5 . The content in the figure cannot be regarded as any limitation on the scope of use of the present application.

[0119] FIG5 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the cluster file statistics method disclosed in any of the aforementioned embodiments.

[0120] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0121] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including a file directory information table, etc. The storage method can be temporary storage or permanent storage.

[0122] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the cluster file statistics method executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0123] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the cluster file statistics method disclosed in any of the aforementioned embodiments.

[0124] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the cluster file statistics method steps disclosed in any of the aforementioned embodiments are implemented.

[0125] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0126] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0127] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0128] The above is a detailed introduction to a cluster file statistics method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A cluster file statistics method, characterized in that: include: For the file directories in the cluster, generate a file directory information table corresponding to each file directory according to the first target data structure, and create a storage information table for recording the table name and storage path of the file directory information table; Monitor the files through the cluster platform file management portal, and generate a corresponding file management operation data table according to the second target data structure for the monitored file management operations; Regularly screening out the target file directory information table that has changed, and updating the target file directory information table; One or more of the storage information table, the file directory information table, and the file management operation data table are queried according to business requirements.

2. The cluster file statistics method according to claim 1, characterized in that: Regularly filter out the target file directory information table that has changed, including: A file directory information table that has changed again is screened out from the historical target file directory information table as the target file directory information table; the historical target file directory information table is a file directory information table that has changed and is screened out in a historical screening operation within a recent preset time period.

3. The cluster file statistics method according to claim 1, characterized in that: Regularly filter out the target file directory information table that has changed, including: The file directory information table with the most recent internal changes within a preset time period is regularly screened out as the target file directory information table.

4. The cluster file statistics method according to claim 1, characterized in that: Also includes: Periodically reading the creation time of the file management operation data table, and if the time interval between the creation time of the file management operation data table and the current time is greater than a preset time length, deleting the file management operation data table; Or, regularly count the storage space occupied by all stored file management operation data tables; if the storage space is larger than a preset space size, determine the size of the data to be deleted based on the difference between the preset space size and the storage space; and select file management operation data tables of the same size as the data to be deleted as the data tables to be deleted in order of time from far to near, and delete the data tables to be deleted.

5. The cluster file statistics method according to claim 1, characterized in that: Before generating the corresponding file management operation data table according to the second target data structure, the method further includes: Determine the file operation type corresponding to the monitored file management operation according to the correspondence between the file management operation and the file operation type; The file management operation data table is generated based on the file operation type.

6. The cluster file statistics method according to claim 1, characterized in that: Also includes: Storing the storage information table, the file directory information table and the file management operation data table in a lightweight database; The lightweight database and the cluster service database are independent of each other.

7. The cluster file statistics method according to any one of claims 1 to 6, characterized in that: The step of regularly screening out the target file directory information table that has changed and updating the target file directory information table includes: Filtering the target file directory information table according to a target parameter in the file directory information table; the target parameter is a latest modification time, and the latest modification time is a modification time synchronously obtained when any file in the file directory corresponding to the file directory information table is modified; Determine the changed target directory according to the target file directory information table, determine the current directory size of the target directory, and update the corresponding target file directory information table according to the current directory size; Determine the levels involved in all target directories that have changed according to the target file directory information table, and obtain the target level; According to the file directory information table, the sizes of all directories at the same level corresponding to each target level are counted respectively to obtain the total size of the directories at the same level; Determine the previous level of the current target level, and update the file directory information table corresponding to the previous level according to the sum of the sizes of the directories in the same level corresponding to the current target level; Until the corresponding file directory information table at the top level is updated.

8. A cluster file statistics device, characterized in that: include: A file directory information table generation module, used for generating a file directory information table corresponding to each file directory in the cluster according to the first target data structure, and creating a storage information table for recording the table name and storage path of the file directory information table; A file management operation data table generation module, used to monitor files through the cluster platform file management portal, and generate a corresponding file management operation data table according to the second target data structure for the monitored file management operations; A file directory information table updating module, used to regularly filter out the target file directory information table that has changed, and update the target file directory information table; The table query module is used to query one or more of the storage information table, the file directory information table and the file management operation data table according to business requirements.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the cluster file statistics method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by a processor, the cluster file statistics method as described in any one of claims 1 to 7 is implemented.

Images