Distributed file system and management method thereof

Abstract

The application provides a distributed file system and a management method thereof, comprising: a primary directory network attached storage, a backup directory network attached storage and a plurality of data network attached storages; the primary directory network attached storage and the backup directory network attached storage are distributed in different storage servers and have the same directory structure; the directory structure is a tree structure, all nodes are directories except that the leaf nodes are links pointing to the data network attached storages, and the root / root group has the permission of 755; each data network attached storage is an independent data network attached storage, and the outer layer directories are the root / root group and have the permission of 755 except that the bottom layer directory is open to the specific user account and group account and has the directory read-write-execution permission. Due to the two-layer directory design, the distributed file system can be efficiently managed by the operation and maintenance personnel through the directory network attached storage access.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a distributed file system and a management method thereof. BACKGROUND

[0002] At present, the data warehouse and data lake architecture in the banking industry generally uses unified access, unified cleaning and unified standardization to use data. Different business systems in the bank provide data to the data warehouse and data lake through data files, complete data loading, and then access and use by various business systems in the bank. Since data file sharing needs to be implemented, data is generally first stored in a network attached storage (NAS).

[0003] However, with the continuous growth of banking business, especially the development of digital business, the quantity and capacity requirements of network attached storage are increasing. Generally, there are hundreds of business systems sharing and using dozens of network attached storages for data source storage. Maintenance personnel need to maintain multiple network attached storages and massive data files, and there are a large number of data unloading, transmission, deletion and compression every day, which brings a high challenge to the data warehouse file management of bank maintenance personnel. SUMMARY

[0004] Therefore, the present application provides a distributed file system and a management method thereof, which can enable maintenance personnel to efficiently manage the distributed file system.

[0005] The first aspect of the present application provides a distributed file system, comprising:

[0006] a master directory network attached storage, a backup directory network attached storage and a plurality of data network attached storages;

[0007] The master directory network attached storage and the backup directory network attached storage are distributed in different storage servers and have the same directory structure. The directory structure is a tree structure. Except that the leaf nodes are links pointing to the data network attached storages, the other nodes are directories, and the group permissions of all nodes are root / root, and the permissions are 755. Each data network attached storage is an independent data network attached storage. Except that the bottom directory is open to use specific user accounts and group accounts to open directory read-write-execution permissions, the outer directory is root / root group, and the permission is 755.

[0008] Optionally, the distributed file system further comprises:

[0009] An early warning module is configured to check consistency of the primary directory network-attached storage and the backup directory network-attached storage.

[0010] Optionally, the early warning module comprises:

[0011] a directory structure checking module, a leaf node checking module, a permission checking module, a data network-attached storage permission checking module, and a checking report generation module.

[0012] The directory structure checking module is configured to compare directory structures by using a python difference comparison module to obtain a first checking result.

[0013] The leaf node checking module is configured to perform a depth-first traversal of a tree-shaped directory structure to determine whether a leaf node attribute is a link and whether a permission is a root 777 permission to obtain a second checking result.

[0014] The permission checking module is configured to check whether a target object in all directory structures is a root 755 permission to obtain a third checking result, wherein the target object is a non-link file.

[0015] The data network-attached storage permission checking module is configured to check whether a directory permission of a target directory is a root 755 permission and whether only standard date directories are present in the target directory to obtain a fourth checking result, wherein the target directory is a directory other than the bottommost standard ddl, ctrl, and data directories of the data network-attached storage.

[0016] The checking report generation module is configured to generate a checking report according to the first checking result, the second checking result, the third checking result, and the fourth checking result.

[0017] Optionally, the distributed file system further comprises:

[0018] A management module is configured to manage the primary directory network-attached storage, the backup directory network-attached storage, and each data network-attached storage.

[0019] Optionally, the management module comprises:

[0020] an adding module, a deleting module, a migrating module, and a statistical module.

[0021] The adding module is configured to, when receiving an adding storage directory request, create a new directory on a target data network-attached storage in the adding storage directory request, create a link pointing to the data network-attached storage in a corresponding directory of a leaf node in the primary directory network-attached storage and the backup directory network-attached storage, and write clean-up day configuration and compression day configuration of the new directory into a database.

[0022] The deleting module is configured to delete a target directory in the deleting storage directory request and delete a link file and a data directory when receiving the deleting storage directory request, and clean up configuration information related to the target directory in the database;

[0023] The migrating module is configured to establish a same-name data directory and a subdirectory on a new data network attached storage when receiving a migrating directory request, add a next-day date directory, establish a link of the next-day date directory pointing to a date directory under an original directory, and finally modify a link pointing to the original data directory to a new data directory.

[0024] The statistical module is configured to obtain a target deadline configuration from the database, compress and clean up the target deadline configuration, and perform size statistics of a data directory and information collection of the data network attached storage after the compression and cleanup of the target deadline configuration are completed.

[0025] Optionally, the distributed file system further comprises:

[0026] The analysis module is configured to analyze a directory size growth trend of a data file on the master directory network attached storage, the backup directory network attached storage, and each data network attached storage, and analyze a size and a growth trend of a single data file.

[0027] Optionally, the analysis module comprises:

[0028] The recording module is configured to, for each data file, count a generation time of the data file and record an arrival time of the data file.

[0029] The content analysis module is configured to perform content analysis on the same-name data files to obtain an analysis result, wherein the analysis result indicates whether the same-name data files are the same file.

[0030] The trend analysis module is configured to draw a growth trend graph of a directory and a file.

[0031] The second aspect of the present application provides a management method of a distributed file system, comprising:

[0032] A directory structure comparison is performed by using a difference comparison module of python to obtain a first checking result.

[0033] A tree-shaped directory structure is traversed in depth to determine whether a leaf node attribute is a link and whether a permission is root 777 permission to obtain a second checking result.

[0034] A target object in all directory structures is checked to determine whether the target object is root 755 permission to obtain a third checking result, wherein the target object is a non-link file.

[0035] The fourth check result is obtained by checking whether the directory permissions of the target directory are root 755 and whether the target directory can only contain standard date directories; wherein, the target directory is the directory other than the lowest level standard ddl, ctrl, and data directories in the data network auxiliary storage.

[0036] An inspection report is generated based on the first inspection result, the second inspection result, the third inspection result, and the fourth inspection result.

[0037] Optionally, the management method of the distributed file system further includes:

[0038] Receive requests to add new storage directories;

[0039] Create a new directory on the target data network-attached storage in the new storage directory request;

[0040] Create new links within the directories of the corresponding leaf nodes on the main directory network auxiliary storage and the backup directory network auxiliary storage, pointing to the data network auxiliary storage;

[0041] Write the cleanup day configuration and compression day configuration of the newly created directory into the database.

[0042] Optionally, the management method of the distributed file system further includes:

[0043] Receive a request to delete the storage directory;

[0044] Delete the target directory in the request to delete the storage directory;

[0045] Delete linked files and data directories;

[0046] Clean up the configuration information related to the target directory in the database.

[0047] Optionally, the management method of the distributed file system further includes:

[0048] Receive migration directory request;

[0049] Create a data directory and subdirectories with the same name on the new data network's attached storage;

[0050] Add a new directory for the next day's date, and create a link to the date directory under the existing directory;

[0051] Change the link that points to the original data directory to point to the new data directory.

[0052] Optionally, the management method of the distributed file system further includes:

[0053] Obtaining a target deadline configuration from a database;

[0054] Compressing and cleaning the target deadline configuration;

[0055] After the daily compression and cleaning is completed, the size of the data directory is counted and information of the data network attached storage is collected.

[0056] Optionally, the management method of the distributed file system further comprises:

[0057] For each data file, the generation time of the data file is counted and the arrival time of the data file is recorded;

[0058] The same-named data files are analyzed in content to obtain an analysis result;

[0059] The analysis result indicates whether the same-named data files are the same file;

[0060] A growth trend graph of the directory and the file is drawn.

[0061] As can be seen from the above solutions, the present application provides a distributed file system and a management method thereof, the distributed file system comprising: a main directory network attached storage, a backup directory network attached storage, and a plurality of data network attached storages; wherein the main directory network attached storage and the backup directory network attached storage are distributed in different storage servers and have the same directory structure; the directory structure is a tree structure, except that the leaf nodes are links pointing to the data network attached storages, other nodes are directories, and the group permissions of all nodes are root / root, and the permissions are 755; each data network attached storage is an independent data network attached storage, except that the bottom directory is open to use a specific user account and group account to open the directory read-write-execution permissions, the outer directory is root / root group, and the permission is 755. The two-layer structure directory design in the present application shields the upper layer application, and the application does not need to know the real storage location of the data, but only needs to access through the directory network attached storage, so that the operation and maintenance personnel can efficiently manage the distributed file system. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0063] Figure 1 A schematic diagram of a distributed file system provided by the embodiments of the present application;

[0064] Figure 2 An architecture diagram of a distributed file system provided for another embodiment of the present application is shown in FIG. 3.

[0065] Figure 3 An architecture diagram of a pre-warning module provided for another embodiment of the present application is shown in FIG. 4.

[0066] Figure 4 An architecture diagram of a management module provided for another embodiment of the present application is shown in FIG. 5.

[0067] Figure 5 An architecture diagram of an analysis module provided for another embodiment of the present application is shown in FIG. 6.

[0068] Figures 6-11 A flow chart of a management method of a distributed file system provided for another embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0070] It should be noted that the terms "first", "second", and the like in the present application are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units. The terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the phrase "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0071] The embodiments of the present application provide a distributed file system, as shown in FIG. 1, which specifically includes: Figure 1

[0072] A primary directory network attached storage 100, a backup directory network attached storage 200, and a plurality of data network attached storages 300.

[0073] ​The main directory network attached storage (NAS) 100 and the backup directory network attached storage 200 are distributed in different storage servers and have the same directory structure; the directory structure is a tree structure, and all nodes except the leaf nodes are directories, and the all-node permission group is root / root, and the permission is 755, which can limit application users from destroying the directory structure; when a new system accesses data, a new leaf node can be allocated as needed; and each data network attached storage 300 is an independent data network attached storage, and except that the bottom directory is open to use a specific user account and group account to open the directory read-write-execution permission, the outer directory is the root / root group with the permission 755.

[0074] As shown in Figure 2 The architecture schematic diagram of the distributed file system provided by the embodiment of the application is shown. The directory NAS_A is the main directory NAS, the directory NAS_B is the backup directory NAS, and the plurality of data NASs are provided.

[0075] The NAS directory structure (the main directory NAS and the backup directory NAS) and the data NAS are stored separately by the operating system soft connection, the directory structure is maintained by a privileged user, the privileged user is used only when the directory structure is added or deleted, and the privileged user is not needed for daily maintenance. The architecture design is not sensed by all shared data storage systems, and the influence of NAS expansion and migration on the application is eliminated.

[0076] Optionally, in another embodiment of the application, an embodiment of the distributed file system further includes:

[0077] The warning module 400 is configured to check the consistency of the main directory network attached storage 100 and the backup directory network attached storage 200.

[0078] Optionally, in another embodiment of the application, an embodiment of the warning module 400 includes: Figure 3

[0079] The directory structure checking module 401, the leaf node checking module 402, the permission checking module 403, the data network attached storage permission checking module 404, and the checking report generation module 405.

[0080] The directory structure checking module 401 is configured to compare the directory structure by using the difference comparison module of python to obtain a first checking result.

[0081] ​The leaf node checking module 402 is configured to traverse the tree directory structure in depth, determine whether the leaf node attribute is a link and whether the permission is root 777 permission, and obtain a second checking result.

[0082] The permission checking module 403 is configured to check whether the target object in all directory structures is root 755 permission, and obtain a third checking result.

[0083] The target object is a non-link file.

[0084] The data network attached storage permission checking module 404 is configured to check whether the directory permission of the target directory is root 755, and whether the target directory can only have standard date directories, and obtain a fourth checking result.

[0085] The target directory is a directory in the data network attached storage except the standard ddl, ctrl and data directories in the bottom layer.

[0086] The checking report generation module 405 is configured to generate a checking report according to the first checking result, the second checking result, the third checking result and the fourth checking result.

[0087] As shown in Table 1, a script list used in the early warning module provided by the embodiment of the application is shown.

[0088] Check script name Function Comment dirnas_diff.py Directory structure consistency comparison Directory structure consistency comparison dirnas_author.py Directory NAS directory permission and link check Directory NAS directory permission and link check dirnas_link.py Directory NAS link check Directory NAS leaf node link check datanas_author.py Data NAS permission check Data NAS directory permission check datanas_content.py Data NAS content check Data NAS underlying directory content check

[0089] Table 1

[0090] Optionally, in another embodiment of the application, an implementation of the distributed file system further includes:

[0091] The management module 500 is configured to manage the master directory network attached storage 100, the backup directory network attached storage 200 and each data network attached storage 300.

[0092] Optionally, in another embodiment of the application, an implementation of the management module 500 includes, as shown in Figure 4

[0093] The adding module 501, the deleting module 502, the migrating module 503 and the statistical module 504.

[0094] The adding module 501 is configured to, when receiving an adding storage directory request, create a directory on the target data network attached storage in the adding storage directory request, and simultaneously create a link pointing to the data network attached storage in the directory to which the leaf node belongs in the master directory network attached storage and the backup directory network attached storage, and write the clean-up day configuration and the compression day configuration of the created directory into a database.

[0095] ​The deleting module 502 is configured to delete the target directory in the deleting storage directory request and delete the link file and the data directory when receiving the deleting storage directory request.

[0096] The migrating module 503 is configured to establish the same-named data directory and subdirectory on the new data network attached storage, add a next day date directory, establish a link of the next day date directory pointing to the date directory under the original data directory, and finally modify the link pointing to the original data directory to the new data directory when receiving the migrating directory request.

[0097] The statistical module 504 is configured to obtain the target period configuration from the database, compress and clean up the target period configuration, and perform size statistics of the data directory and information collection of the data network attached storage after the daily compression and cleanup are completed.

[0098] Specifically, the saving period configuration and the special file processing configuration of different directories are set to provide customized cleanup and storage of data, and the adaptive algorithm is provided to expand the capacity to avoid the influence of data migration and storage capacity reaching the limit on a large number of applications.

[0099] As shown in Table 2, a list of scripts used in the management module provided in the embodiment of the application is provided.

[0100] Check script name Function Comment datanas_mgdir.py Directory structure management Add, delete, and store directories datanas_mv.py Directory migration Directory migration datanas_clear.py File cleaning Provide customized cleaning datanas_compress.py File compression Provide multi-threaded compression and customized compression datanas_compute.py File index statistics Collect NAS file-related information and store in database

[0101] Table 2

[0102] The directories in the data NAS are customized and managed according to different needs, different data are retained for different time periods in different business fields, and the corresponding parameters are persisted in the database.

[0103] Optionally, in another embodiment of the application, an embodiment of the distributed file system further includes:

[0104] The analysis module 600 is configured to analyze the directory size growth trend of the data files on the main directory network attached storage, the backup directory network attached storage, and each data network attached storage, and analyze the size and growth trend of a single data file.

[0105] Optionally, in another embodiment of the application, an embodiment of the analysis module 600 includes: Figure 5

[0106] The recording module 601 is configured to, for each data file, count the generation time of the data file and record the arrival time of the data file.

[0107] The content analysis module 602 is configured to perform content analysis on the same-named data files to obtain an analysis result.​

[0108] The analysis result indicates whether the data files with the same name are the same file.

[0109] Specifically, if the ddl files are consistent, the files are determined to be similar (the same), otherwise, the files are determined to be irrelevant.

[0110] The trend analysis module 603 is configured to draw a growth trend graph of the catalog and the file.

[0111] As shown in Table 3, a list of scripts used in the analysis module provided by the embodiment of the application is provided.

[0112] Check script name Function Comment filesize_analysis.py File size analysis Analyze the growth trend of a single file filetime_analysis.py File time analysis Analyze the arrival time and other trends of a file filename_analysis.py File name analysis Analyze files with the same name dirsize_analysis.py Directory size trend analysis Analyze the growth trend of a directory

[0113] Table 3

[0114] On the basis of the file system monitoring, the daily trend analysis of the catalog level and the file level is added, which is helpful for the capacity analysis and the capacity management of the related personnel.

[0115] According to the above scheme, the application provides a distributed file system, which comprises a master catalog network attached storage 100, a backup catalog network attached storage 200 and a plurality of data network attached storages 300; wherein the master catalog network attached storage 100 and the backup catalog network attached storage 200 are distributed on different storage servers and have the same catalog structure; the catalog structure is a tree structure, and except that the leaf node is a link pointing to the data network attached storage 300, other nodes are all catalogs, and the group of all node permissions is root / root, and the permission is 755; each data network attached storage 300 is an independent data network attached storage, and except that the bottom catalog is opened to use a specific user account and a group account to open the catalog read-write-execution permission, the outer catalog is all root / root group, and the permission is 755. The two-layer catalog structure design in the application shields the upper application, and the application does not need to know the real storage location of the data, but only needs to access through the catalog network attached storage, so that the operation and maintenance personnel can efficiently manage the distributed file system.

[0116] Another embodiment of the application provides a management method of a distributed file system, as shown in Figure 6 The method comprises the following steps.

[0117] S601, a directory structure comparison is performed by using a difference comparison module of python to obtain a first checking result.

[0118] S602, a tree-shaped catalog structure is traversed in depth to determine whether the leaf node attribute is a link and whether the permission is root777 permission, and a second checking result is obtained.

[0119] S603, check whether the target object in all directory structures is root 755 permission, and obtain a third check result.

[0120] The target object is a non-link file.

[0121] S604, check whether the directory permission of the target directory is root 755, and the target directory can only have standard date directories, and obtain a fourth check result.

[0122] The target directory is a directory other than the bottom layer standard ddl, ctrl, and data directories of the data network attached storage.

[0123] S605, generate a check report according to the first check result, the second check result, the third check result, and the fourth check result.

[0124] Thus, the purpose of real-time monitoring of the distributed file system is achieved, so that when an exception occurs, the exception can be processed in time.

[0125] Optionally, in another embodiment of the application, an embodiment of the management method of the distributed file system, as shown in Figure 7 , further includes:

[0126] S701, receive a request for adding a storage directory.

[0127] S702, create a directory on the target data network attached storage in the request for adding a storage directory.

[0128] S703, create a link pointing to the data network attached storage in the directory to which the corresponding leaf node belongs on the master directory network attached storage and the backup directory network attached storage.

[0129] S704, write the cleaning day configuration and compression day configuration of the newly created directory into a database.

[0130] Optionally, in another embodiment of the application, an embodiment of the management method of the distributed file system, as shown in Figure 8 , further includes:

[0131] S801, receive a request for deleting a storage directory.

[0132] S802, delete the target directory in the request for deleting a storage directory.

[0133] S803, delete the link file and the data directory.

[0134] S804, clean up the target directory related configuration information in the database.

[0135] Optionally, in another embodiment of the present application, an embodiment of the management method of the distributed file system, as shown in Figure 9 includes the following steps:

[0136] S901, receiving a migration directory request.

[0137] S902, establishing a same-name data directory and subdirectory on a new data network attached storage.

[0138] S903, adding a next day date directory, and establishing a link of the next day date directory pointing to a date directory under the original directory.

[0139] S904, modifying the link pointing to the original data directory to a new data directory.

[0140] Optionally, in another embodiment of the present application, an embodiment of the management method of the distributed file system, as shown in Figure 10 includes the following steps:

[0141] S1001, obtaining the configuration of the target deadline from the database.

[0142] S1002, compressing and cleaning the configuration of the target deadline.

[0143] S1003, after the daily compression and cleaning is completed, performing size statistics of the data directory and information collection of the data network attached storage.

[0144] Specifically, by setting the retention period configuration and special file processing configuration of different directories, customized cleaning and storage of data are provided. An adaptive algorithm is provided to expand the capacity, so as to avoid the influence of data migration and storage capacity reaching the limit on a large number of applications. Different time periods are retained for different business fields, and the corresponding parameters are persisted in the database.

[0145] Optionally, in another embodiment of the present application, an embodiment of the management method of the distributed file system, as shown in Figure 11 includes the following steps:

[0146] S1101, for each data file, statistics the generation time of the data file and records the arrival time of the data file.

[0147] S1102, performing content analysis on the same-name data files to obtain an analysis result.

[0148] The analysis result indicates whether the same-name data files are the same file.

[0149] Specifically, if the ddl files are consistent, it is judged as similar (the same) files, otherwise it is judged as irrelevant files.

[0150] S1103, draw a growth trend chart of the catalog and the file.

[0151] The daily trend analysis of the catalog level and the file level is added on the basis of the file system monitoring, which is helpful for the capacity analysis and the capacity management of the related personnel.

[0152] In the above-described embodiments disclosed in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus and method embodiments described above are only illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0153] In addition, the function modules in the various embodiments of the present disclosure can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. When the functions are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a live device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0154] Those skilled in the art could make or employ the present application without undue experimentation having regard to the detailed description herein. Various modifications can be made to the embodiments described herein, and the generic principles illustrated herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed file system, characterized in that, include: Main directory network auxiliary storage, backup directory network auxiliary storage, and multiple data network auxiliary storage; The primary directory network auxiliary storage and the backup directory network auxiliary storage are distributed on different storage servers and have the same directory structure. The directory structure is a tree structure, where all nodes except the leaf nodes are links to the data network auxiliary storage and all other nodes are directories. All nodes have permissions belonging to the root / root group and permissions of 755. Each data network auxiliary storage is an independent data network auxiliary storage. Except for the bottom directory which allows specific user accounts and group accounts to grant directory read, write, and execute permissions, the outer directory belongs to the root / root group and has permissions of 755. The distributed file system also includes: The early warning module is used to check the consistency between the network auxiliary storage of the main directory and the network auxiliary storage of the backup directory; The early warning module includes: The system includes a directory structure check module, a leaf node check module, a permission check module, a data network auxiliary storage permission check module, and a check report generation module. The directory structure checking module is used to compare the directory structure using Python's difference comparison module to obtain the first check result. The leaf node inspection module is used to perform a depth-first traversal of the tree-like directory structure to determine whether the leaf node attribute is a link and whether the permission is root 777, and to obtain the second inspection result. The permission checking module is used to check whether the target object in all directory structures has root 755 permissions and obtain a third check result; wherein, the target object is a non-linked file; The data network auxiliary storage permission checking module is used to check whether the directory permission of the target directory is root755, and the target directory can only contain standard date directories, thus obtaining the fourth check result; wherein, the target directory is the data network auxiliary storage except for the lowest level standard ddl, ctrl, and data directories. The inspection report generation module is used to generate an inspection report based on the first inspection result, the second inspection result, the third inspection result, and the fourth inspection result.

2. The distributed file system according to claim 1, characterized in that, Also includes: The management module is used to manage the main directory network auxiliary storage, the backup directory network auxiliary storage, and each of the data network auxiliary storages.

3. The distributed file system according to claim 2, characterized in that, The management module includes: Add a module, delete a module, migrate a module, and add a statistics module; The newly added module is used to create a new directory on the target data network auxiliary storage in the new storage directory request when receiving a new storage directory request, and at the same time create a new link in the directory of the corresponding leaf node on the main directory network auxiliary storage and the backup directory network auxiliary storage pointing to the data network auxiliary storage, and write the cleanup day configuration and compression day configuration of the newly created directory into the database. The deletion module is used to delete the target directory in the deletion storage directory request when receiving a deletion storage directory request, and to delete the linked files and data directories, and clean up the database related configuration information of the target directory; The migration module is used to, when receiving a migration directory request, create a data directory and subdirectories with the same name on the new data network auxiliary storage, add a directory for the next day's date, create a link for the directory for the next day's date pointing to the date directory under the original directory, and finally modify the link pointing to the original data directory to point to the new data directory. The statistics module is used to retrieve the configuration of the target period from the database, compress and clean the configuration of the target period, and perform statistics on the size of the data directory and collect information from the data network auxiliary storage after the compression and cleaning is completed on the same day.

4. The distributed file system according to claim 1, characterized in that, Also includes: The analysis module is used to analyze the growth trend of the directory size of the data files on the main directory network auxiliary storage, the backup directory network auxiliary storage, and each of the data network auxiliary storages, as well as the size and growth trend of individual data files.

5. The distributed file system according to claim 4, characterized in that, The analysis module includes: The recording module is used to calculate the generation time and the arrival time of each data file. The content analysis module is used to perform content analysis on data files with the same name and obtain analysis results; wherein, the analysis results indicate whether the data files with the same name are the same file; The trend analysis module is used to draw growth trend charts for directories and files.

6. A method for managing a distributed file system, characterized in that, The distributed file system includes: a primary directory network auxiliary storage, a backup directory network auxiliary storage, and multiple data network auxiliary storages; wherein, the primary directory network auxiliary storage and the backup directory network auxiliary storages are distributed on different storage servers and have the same directory structure; the directory structure is a tree structure, except that the leaf nodes are links to the data network auxiliary storages, all other nodes are directories, and all nodes have permissions belonging to the root / root group and permissions of 755; each data network auxiliary storage is an independent data network auxiliary storage, except that the lower-level directory has access to read, write, and execute permissions for specific user accounts and group accounts, the outer directory is all owned by the root / root group and has permissions of 755; The management method includes: The directory structure was compared using Python's difference comparison module to obtain the first inspection result; Perform a depth-first traversal of the tree-like directory structure, determine whether the leaf node attribute is a link, and whether the permissions are root 777, to obtain the second check result; Check whether the target object in all directory structures has root 755 permissions to obtain the third check result; wherein, the target object is a non-linked file; The fourth check result is obtained by checking whether the directory permissions of the target directory are root 755 and whether the target directory can only contain standard date directories; wherein, the target directory is the directory other than the lowest level standard ddl, ctrl, and data directories in the data network auxiliary storage. An inspection report is generated based on the first inspection result, the second inspection result, the third inspection result, and the fourth inspection result.

7. The management method according to claim 6, characterized in that, Also includes: Receive requests to add new storage directories; Create a new directory on the target data network-attached storage in the new storage directory request; Create new links within the directories of the corresponding leaf nodes on the main directory network auxiliary storage and the backup directory network auxiliary storage, pointing to the data network auxiliary storage; Write the cleanup day configuration and compression day configuration of the newly created directory into the database.

8. The management method according to claim 7, characterized in that, Also includes: Receive a request to delete the storage directory; Delete the target directory in the request to delete the storage directory; Delete linked files and data directories; Clean up the configuration information related to the target directory in the database.

Images