A distributed file system mass file automatic deletion optimization method and system

Abstract

The application discloses a kind of distributed file system mass file automatic deletion optimization method and system, method includes: screening out to be deleted directory, obtains the first current modification time of to be deleted directory;To be deleted directory is executed multi-level access constraint condition verification, when meeting verification condition, execute next step;The state of to be deleted directory is switched to lock state;When locking succeeds, the second current modification time of to be deleted directory is reacquired;Second current modification time is compared with the first modification time, if consistent, enter next step;The state of to be deleted directory is switched to start emptying state, triggers out-of-band deletion engine module to delete to be deleted directory;Receive emptying completion result returned by triggering out-of-band deletion engine module, and the state of to be deleted directory is updated to emptying completion state;Empty directory, trigger global quota statistics module to update directory level quota data.By the method of the application, delete efficiency is improved and MDS load is reduced.

Description

Technical Field

[0001] This application belongs to the field of file processing technology, specifically relating to a method and system for automatically deleting and optimizing massive files in a distributed file system. Background Technology

[0002] With the rapid development of the digital economy and the deep integration of emerging technologies such as cloud computing, big data, and artificial intelligence, enterprise data volumes are growing exponentially. Distributed file systems, as the core underlying storage infrastructure, bear the responsibility for storing, managing, and scheduling massive amounts of unstructured data. In recent years, with the continuous expansion of business scenarios, the number of files carried by distributed file systems has rapidly increased from tens of millions and hundreds of millions to tens of billions.

[0003] In scenarios involving billions of files, distributed file systems face unprecedented technical challenges, with automated lifecycle management of massive files becoming a key bottleneck restricting system performance and stability. Data lifecycle management (DLM), as a core function of storage systems, is responsible for automatically cleaning up expired, invalid, and infrequently accessed files, freeing up storage resources, reducing operating costs, and improving storage utilization.

[0004] The distributed file system adopts an architecture that separates metadata from data. Its core components include a client, a metadata service (MDS), an object data storage service (OSD), a metadata pool, and a data pool. The MDS, as the core management node, is responsible for managing metadata such as directory structure, file attributes, permission information, and file data distribution mapping relationships. It also processes client metadata requests, ensuring data consistency and security. The data pool stores the actual file data content and consists of multiple OSD nodes, supporting distributed data storage, replica redundancy, and fault self-healing. The client, as the user access point, is responsible for forwarding user requests, caching frequently used data, and handling data read / write interactions.

[0005] The distributed file system adopts an architecture design that separates metadata from data. Its core components include a client, a metadata service (MDS), an object data storage service (OSD), a metadata pool, and a data pool. The MDS, as the core management node, is responsible for managing metadata such as directory structure, file attributes, permission information, and file data distribution mapping relationships. It also processes client metadata requests and ensures data consistency and security. The data pool stores the actual file data content and consists of multiple OSD nodes, supporting distributed data storage, replica redundancy, and fault self-healing. The client, as the user access point, is responsible for forwarding user requests, caching frequently used data, and handling data read / write interactions.

[0006] To achieve automated lifecycle management for scenarios with billions of files, major distributed storage vendors in the industry have launched independent automatic deletion services. These services periodically scan system directories, filter expired and invalid files, and perform batch deletion operations, replacing manual deletion and improving cleanup efficiency. The core logic of existing automatic deletion services is "traversal decoupling and deletion coupling," meaning that only the directory traversal process is optimized, while the deletion process still uses traditional mechanisms and is highly dependent on MDS processing.

[0007] To achieve automated lifecycle management for scenarios with billions of files, major distributed storage vendors in the industry have launched independent automatic deletion services for periodically scanning system directories, filtering expired and invalid files, and performing batch deletion operations. Existing automatic file deletion solutions mention one approach, as follows: Traditional automatic file deletion services traverse directories through client interfaces, requiring frequent interaction with the MDS (Multi-Demand Storage System), consuming significant MDS resources. Existing solutions directly access the metadata pool through underlying interfaces, or synchronize metadata to an independent database for direct database access, bypassing the client and MDS to directly read directory metadata, completing directory traversal and file filtering. This effectively reduces the resource consumption of the MDS during the traversal process, decoupling the traversal process from the MDS. The automatic deletion service filters expired and invalid files that meet deletion criteria based on preset lifecycle strategies (such as file storage duration, file size, etc.), generating a list of files to be deleted. The automatic deletion service iterates through the list of files to be deleted and sends a single file deletion request to MDS one by one. After receiving the request, MDS completes the entire process of deleting file metadata, deleting data objects, updating directory structure, and updating quota statistics in sequence. After all files are deleted, the automatic deletion service cleans up the empty directory and completes the batch deletion task.

[0008] Existing technical solutions have alleviated the pressure on MDS resources to some extent by optimizing the directory traversal process, but the core deletion process still adopts a coupled mode of single file request and centralized processing by MDS, which seriously affects system performance and stability in scenarios with hundreds of billions of files.

[0009] Existing automatic file deletion technologies suffer from four major flaws in scenarios involving billions of files, severely limiting system performance, stability, and user experience:

[0010] (1) Severe contention for MDS metadata service resources leads to significant conflicts between front-end and back-end business operations. As the core management node of the distributed file system, MDS handles core requests such as directory queries, file read / write operations, and permission verification for front-end user businesses, and is crucial for ensuring the normal operation of front-end businesses. In the existing deletion scheme, when deleting billions of files in batches, the automatic deletion service needs to send massive file deletion requests to MDS, continuously consuming core resources such as CPU, memory, and network I / O of MDS. The ability to process front-end business requests is greatly reduced, resulting in increased front-end business response latency and decreased throughput, seriously affecting the user business experience and failing to meet the business requirements of high concurrency and low latency.

[0011] (2) The high frequency of metadata operations puts enormous pressure on the metadata pool, easily causing system lag. The existing file deletion mode adopts the design logic of "single file, single metadata operation". When deleting a directory containing 1 million files, 1 million metadata deletion requests need to be sent to MDS. Each request needs to go through the entire process of request encapsulation, transmission, verification, processing and response, generating a large number of fragmented metadata read and write operations. In the scenario of batch deletion of tens of billions of files, this leads to IO congestion in the metadata pool, accumulation of request queues, and a significant reduction in the processing efficiency of the metadata pool.

[0012] (3) The efficiency of deleting massive amounts of files is extremely low, the batch task execution cycle is too long, and the release of storage resources is slow. In the single file request mode, each deletion request needs to go through multiple stages such as request encapsulation, network transmission, permission verification, metadata processing, data deletion, and response return. The interaction overhead is extremely high. Although the time taken to delete a single file is short, the cumulative effect is significant. In the scenario of hundreds of billions of files, the existing solution may take several days to complete a batch deletion task. The deletion efficiency is low and cannot meet the business needs of rapid cleaning of massive data and rapid release of storage resources. A large number of invalid files occupy storage resources for a long time, resulting in low storage utilization and waste of resources. Summary of the Invention

[0013] This application provides a method and system for optimizing the automatic deletion of massive files in a distributed file system, aiming to solve problems such as significant conflicts between front-end and back-end business processes, huge load pressure on the metadata pool, and extremely low efficiency in deleting massive files in existing technologies.

[0014] Firstly, a method for automatically deleting massive files in a distributed file system, applied to lifecycle management services, includes:

[0015] Step S1: Filter out directories to be deleted according to preset conditions, traverse the directories to be deleted, obtain the first current modification time of the directories to be deleted and store it;

[0016] Step S2: Perform multi-level admission constraint verification on the directory to be deleted. If the verification conditions are met, proceed to the next step.

[0017] Step S3: Switch the status of the directory to be deleted to locked;

[0018] Step S4: After successful locking, re-obtain the second current modification time of the directory to be deleted; compare the second current modification time with the first modification time shown. If they match, proceed to the next step.

[0019] Step S5: Switch the state of the directory to be deleted to the start-clear state, trigger the out-of-band deletion engine module to delete the directory to be deleted, so that the out-of-band deletion engine module bypasses MDS and establishes a communication connection with the data pool, and uses the batch data access interface to delete the data objects corresponding to all files in the directory to be deleted;

[0020] Step S6: Receive the clearing completion result returned by the out-of-band deletion engine module, update the status of the directory to be deleted to the clearing completion status, and mark that all file data in the directory to be deleted has been cleared and the directory is empty;

[0021] Step S7: Call the rmdir directory deletion interface to delete empty directories and trigger the global quota statistics module to update the directory-level quota data.

[0022] Optionally, in the above scheme, the multi-level access constraint verification includes:

[0023] Determine the file type of the directory to be deleted; all files in the directory to be deleted are regular data files, and the directory to be deleted does not contain subdirectories, symbolic link files, hard link files, or device files;

[0024] Determine the deletion range of the directory to be deleted, wherein the deletion range is that all files in the directory to be deleted meet the preset deletion conditions;

[0025] Determine the file reference constraints within the directory to be deleted; all files in the directory to be deleted have no valid references, and the valid references include metadata service module memory cache references, metadata service module log persistence references, client mount references, process open references, and snapshot references;

[0026] The total number of files in the directory to be deleted is greater than the preset configuration threshold.

[0027] In the above scheme, optionally, after switching the state of the directory to be deleted to the locked state in step S3, the following constraint rules are applied to the directory's running state:

[0028] The directory to be deleted is allowed to perform read-only operations such as attribute query, node retrieval, permission query, and quota query.

[0029] The directory to be deleted is allowed to be traversed and return an empty result, but returning a list of files in the directory is prohibited;

[0030] The directory to be deleted is prohibited from performing modification operations, which include modifying directory attributes, creating files, moving files, renaming files, and modifying permissions.

[0031] The directory to be deleted is allowed to be deleted only when the directory to be deleted is in a cleared state;

[0032] The quota statistics remain unchanged during the out-of-band deletion process.

[0033] Optionally, in the above scheme, the preset condition is that the file modification time exceeds a set value.

[0034] In the above scheme, optionally, the status of the directory to be deleted can be switched to locked, clearing start, or clearing complete by calling the custom dir.clear extended property interface of the metadata service module.

[0035] Optionally, in the above scheme, the lifecycle management service is deployed independently of the metadata service module, adopts a distributed architecture design, and supports multi-node cluster deployment.

[0036] Optionally, in the above scheme, if the multi-level access constraint condition verification is not met for the directory to be deleted, or the second current modification time is inconsistent with the first modification time shown, or the locking fails when the state of the directory to be deleted is switched to the locked state, the state of the directory to be deleted is rolled back to the initial state, the directory lock is released, and the deletion task is rolled back to the standard file deletion interface for execution.

[0037] Secondly, a distributed file system for automatic deletion and optimization of massive files, the system comprising:

[0038] Filtering module: Used to filter directories to be deleted based on preset conditions, traverse the directories to be deleted, obtain the first current modification time of the directories to be deleted and store it;

[0039] Multi-level admission constraint verification module: used to perform multi-level admission constraint verification on the directory to be deleted, and execute the locking module when the verification conditions are met;

[0040] Locking module: Used to switch the status of the directory to be deleted to a locked state;

[0041] Time comparison module: After successful locking, it retrieves the second current modification time of the directory to be deleted; compares the second current modification time with the first modification time shown; if they match, it proceeds to the deletion module.

[0042] The deletion module is used to switch the state of the directory to be deleted to the start of clearing state, trigger the out-of-band deletion engine module to delete the directory to be deleted, so that the out-of-band deletion engine module can bypass MDS and establish a communication connection with the data pool, and use the batch data access interface to delete the data objects corresponding to all files in the directory to be deleted.

[0043] Clear Complete Status Flag Module: Used to receive the clear complete result returned by the out-of-band deletion engine module, update the status of the directory to be deleted to clear complete status, and mark that all file data in the directory to be deleted has been cleared and the directory is empty;

[0044] Directory update module: Used to call the rmdir directory deletion interface to delete empty directories and trigger the global quota statistics module to update the directory-level quota data.

[0045] Thirdly, a computer device includes a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the distributed file system automatic deletion optimization method for massive files described in the first aspect.

[0046] Fourthly, a computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the distributed file system's method for automatically deleting and optimizing massive files as described in the first aspect.

[0047] Compared with the prior art, this application has at least the following beneficial effects:

[0048] This invention effectively avoids the risk of accidentally deleting valid business files and damaging the system's directory structure by using a lifecycle management service to filter directories to be deleted and performing multi-level access constraint checks, triggering out-of-band deletion only for directories that meet security conditions. It also ensures that no concurrent changes occur to the directory during the deletion preparation period by calling the custom extended attribute interface of the metadata service module and performing a secondary consistency check based on the directory modification time, guaranteeing data consistency and state stability during the deletion process. Finally, it bypasses the metadata service module by triggering the out-of-band deletion engine module to directly establish a communication connection with the data pool and uses a batch data access interface to delete all files within the directory. The data objects corresponding to the files reduce the overhead of metadata deletion request processing, which is positively correlated with the number of files and is traditionally borne by the metadata service module, to only requiring the metadata service module to participate in a limited number of state switching and directory deletion operations. This significantly reduces the load pressure on the metadata service module, eliminates resource conflicts between large-scale deletion tasks in the background and front-end business, and ensures low latency and stable operation of the front-end business. At the same time, by leveraging the distributed parallel processing capabilities of the out-of-band deletion engine module, the batch deletion efficiency of massive files is greatly improved, and the storage resource release cycle is effectively shortened. Thus, efficient, low-interference, and highly reliable automatic deletion of massive files in scenarios with hundreds of billions of files is achieved. Attached Figure Description

[0049] Figure 1 This application provides an architecture diagram of a module in a distributed file system's automatic deletion optimization method for massive files, as an embodiment of the present application.

[0050] Figure 2 A flowchart illustrating an optimization method for automatic deletion of massive files in a distributed file system, provided as an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of state transition in a distributed file system's automatic deletion optimization method for massive files, as described in one embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0053] This invention aims to solve the performance bottlenecks and efficiency problems faced by existing distributed file systems when handling the automatic deletion of massive amounts of files. Specifically, the objectives are:

[0054] (1) Reduce the load on the metadata server: The "N" file deletion requests that originally needed to be processed by MDS were "integrated" into 3 extended attribute setting requests that only need to be processed. A directory deletion request.

[0055] (2) Isolate the impact of front-end and back-end business: ensure that large-scale deletion tasks in the back-end do not occupy the metadata service capabilities of MDS, thus ensuring the stability and low latency of front-end business.

[0056] (3) Improve deletion efficiency: Use non-standard interfaces to directly delete underlying objects in batches, avoid frequent competition for metadata locks, and significantly improve the cleaning speed of massive small files.

[0057] (4) Ensure data consistency: By extending attributes as "lightweight signaling" and combining them with the directory locking mechanism of MDS, other operations are avoided during the process of clearing directory data.

[0058] The purpose of this application is to overcome the technical shortcomings of existing automatic file deletion technologies for massive files in scenarios with hundreds of billions of files, such as excessive MDS load, conflicts between front-end and back-end business resources, low deletion efficiency, high pressure on metadata pool, and difficulty in ensuring data consistency. This application aims to provide an optimized method for automatic file deletion in distributed file systems for scenarios with hundreds of billions of files.

[0059] 1. System Overall Architecture Design:

[0060] To optimize automatic file deletion, this invention employs an out-of-band deletion strategy. The system comprises five core modules working collaboratively, as shown in Figure 1. The specific functions, composition, and collaborative logic of each core module are as follows:

[0061] Client Module: The client module serves as the entry point for user interaction with the system, supporting all user operations such as file reading and writing, directory operations, permission management, and lifecycle policy configuration. This invention extends the client module by adding a read / write interface for the `dir.clear` extended attribute, supporting the setting and querying of directory deletion status outside the specified area, providing a standardized status management entry point for upper-layer services; it is compatible with traditional file operation interfaces and does not affect normal business access for front-end users.

[0062] Metadata Service Module (MDS): As the core management module of the system, MDS is responsible for managing metadata such as directory structure, file attributes, permission information, quota data, and data distribution mapping relationships. It is crucial for ensuring data consistency and security. This invention reconstructs and expands the functionality of the MDS module, adding features such as out-of-band deletion status verification and directory lock verification management.

[0063] Lifecycle Management Service (LMS): As the core for initiating and scheduling out-of-band deletion tasks, LMS is deployed independently of MDS, adopts a distributed architecture design, and supports multi-node cluster deployment to improve task processing capacity and reliability. LMS core functions include: system-wide directory traversal, lifecycle policy parsing, filtering directories to be deleted, multi-level access condition verification, deletion task generation, task scheduling and distribution, status synchronization updates, result reception and processing, and log aggregation. It is responsible for the entire lifecycle management of automatic deletion tasks, coordinating the collaboration of Client, MDS, and ODE, and acts as a bridge connecting the metadata layer and the data layer, ensuring the orderly and efficient execution of the out-of-band deletion process.

[0064] Out-of-band Deletion Engine Module (ODE): As the main entity for data deletion execution, ODE is deployed completely independently of MDS, adopts a distributed parallel architecture design, and supports multi-node load balancing. ODE's core functions include: direct access to the data pool, batch data object deletion, real-time result reporting, and exception handling retries; it bypasses MDS to communicate directly with the data pool, using a batch data access interface to achieve out-of-band data deletion execution.

[0065] Metadata Pool and Data Pool: The metadata pool and data pool serve as the system's underlying storage carriers, employing a distributed, highly available architecture that supports replica redundancy, fault self-healing, and horizontal scaling. The metadata pool stores all system metadata, including directory structure, file attributes, permission information, quota data, extended attributes, status information, and log records, and is directly managed by the MDS. The data pool stores the actual file data content, consisting of multiple data nodes, and supports distributed data storage, load balancing, and failover. Traditionally, file deletion operations involve the MDS accessing the data pool to clear data. This invention adds a feature where the ODE directly accesses the data pool to perform data deletion operations.

[0066] The five core modules communicate and interact through standardized interfaces, resulting in low coupling, strong scalability, and high reliability. Together, they form an efficient, stable, and reliable out-of-band deletion system, adaptable to the needs of automated file deletion in scenarios with tens of billions of files.

[0067] In one embodiment, such as Figure 2 As shown, a method for automatically deleting massive files in a distributed file system is provided, applied to lifecycle management services. The method includes:

[0068] Step S1: Filter out directories to be deleted according to preset conditions, traverse the directories to be deleted, obtain the first current modification time of the directories to be deleted and store it.

[0069] In step S1, the preset condition is that the file modification time exceeds a set value. After LMS traverses and filters out the directories to be deleted, it first obtains the current modification time (mtime) of the directories to be processed through the stat command, records it as an initial verification snapshot, and persists it.

[0070] Design purpose: To be used in subsequent secondary verification steps to determine whether the directory has undergone concurrent changes (such as file additions or deletions) during the verification period, and to avoid the risk of inconsistent verification conditions caused by file changes in the directory during the verification period.

[0071] Step S2: Perform multi-level access constraint verification on the directory to be deleted. If the verification conditions are met, proceed to the next step.

[0072] In step S2, LMS performs multi-level admission constraint checks on the directory, checking file type (only ordinary files), deletion range (full deletion), and number of files (≥ preset threshold).

[0073] Verification result determination: If all conditions are met, proceed to the next step; if any condition is not met, directly downgrade to the traditional file interface deletion mode to ensure that the deletion task is not interrupted.

[0074] Step S3: Switch the status of the directory to be deleted to locked.

[0075] In step S3, after the admission verification is passed, LMS calls the custom dir.clear extended attribute interface provided by MDS to attempt to switch the status of the directory to be deleted from the initial state None (initial state) to the locked state Prepare (locked state), thus completing the global locking of the directory.

[0076] Verification check: Verify the file reference status (no valid references).

[0077] Locking successful: The directory status is updated to Prepare (locked), indicating that the directory lock is effective, there are no concurrent conflicts, and it is safe to perform subsequent operations. Proceed to the next step.

[0078] Locking failed: This indicates that the directory is being operated on concurrently, is already locked, or is in an abnormal state. The system will automatically downgrade to the traditional deletion mode to avoid conflicts.

[0079] Step S4: After successful locking, retrieve the second current modification time of the directory to be deleted; compare the second current modification time with the first modification time shown. If they match, proceed to the next step.

[0080] In step S4, after the directory is successfully locked, LMS re-acquires the current modification time (mtime) of the directory and compares it with the mtime recorded in the initial directory state snapshot. If mtime has not changed, it means that no concurrent modifications have occurred in the directory during the verification period, the directory state is stable, and the data is consistent, proceeding to the next step. If mtime has changed, it means that files in the directory have been updated during the verification period (such as adding or deleting files), posing a risk of invalidating the verification result. The directory state is immediately rolled back to the initial state None, the directory lock is released, the out-of-band deletion process is terminated, and the process is downgraded to the traditional deletion mode to ensure data security.

[0081] Step S5: Switch the state of the directory to be deleted to the start of clearing state, trigger the out-of-band deletion engine module to delete the directory to be deleted, so that the out-of-band deletion engine module bypasses MDS and establishes a communication connection with the data pool, and uses the batch data access interface to delete the data objects corresponding to all files in the directory to be deleted.

[0082] In step S5, after the secondary verification passes, LMS calls the extended attribute interface again to switch the directory status from the locked state (Prepare) to the start-up-clearing state (Start), officially triggering ODE to start the background batch data clearing task. Successful status update: ODE receives the clearing command, starts the data clearing process, and proceeds to the next step; Failed status update: The system automatically retryes the status update operation, logging multiple retries to avoid single-directory anomalies blocking batch tasks.

[0083] After receiving the clear command, ODE bypasses MDS and directly establishes a communication connection with the data pool. It uses the batch data access interface to delete the data objects corresponding to all files in the directory, without requiring MDS to participate in the data deletion logic.

[0084] Execution logic: ODE parses the directory metadata, obtains the data object identifiers of all files in the directory, sends deletion commands to the data pool in batches, and the data pool deletes the data objects in batches to release storage space;

[0085] Advantages: Batch processing significantly reduces the number of network interactions, lowers transmission overhead, and improves deletion efficiency; no MDS forwarding is required, thus stripping away the MDS load and eliminating resource contention conflicts.

[0086] Step S6: Receive the clearing completion result returned by the out-of-band deletion engine module, update the status of the directory to be deleted to the clearing completion status, and mark that all file data in the directory to be deleted has been cleared and the directory is empty.

[0087] In step S6, after ODE completes the deletion of all file data objects in the directory, it returns the result of clearing to LMS. After receiving the result, LMS calls the extended attribute interface to update the directory status from the Start state to the Commit state, marking that all file data in the directory has been cleared and the directory is empty.

[0088] Status update successful: Confirmed directory clearing complete, proceed to the next step;

[0089] Status update failed: Automatically retry the status update operation to ensure that the status is consistent with the actual operation and avoid subsequent process abnormalities caused by inconsistent status.

[0090] Step S7: Call the rmdir directory deletion interface to delete empty directories and trigger the global quota statistics module to update the directory-level quota data.

[0091] In step S7, after LMS confirms that the directory status is "Commit cleared" and then performs cleanup operations:

[0092] The `rmdir` directory deletion interface is called to delete empty directories. The file metadata contained in the directory is saved to the key-value data table of the directory object. LMS avoids multiple key-value operations on the object by directly deleting the object, which greatly reduces the number of operations on the metadata pool.

[0093] The system's global quota statistics module is triggered in one go, updating the directory-level quota data and releasing the quota usage simultaneously;

[0094] Execution complete: The out-of-band deletion process is now closed, the directory and corresponding data and quotas have been completely cleaned up, and storage resources have been successfully released.

[0095] In this embodiment, the above core steps achieve standardized, closed-loop, secure, and efficient execution of out-of-band deletion of massive files. The entire process requires only 4 interactions from the MDS (3 state transitions + 1 directory deletion), reducing the frequency of metadata operations by more than 99%, significantly improving deletion efficiency, reducing system load, and ensuring data security.

[0096] To achieve refined management of the entire out-of-band deletion process and ensure standardized state transitions, identifiable anomalies, and traceable operations, this invention innovatively designs a four-state directory state machine, coupled with a custom `dir.clear` extended attribute interface and standardized state transition rules. The following are the definitions of the four directory states:

[0097] This invention divides the runtime state of a directory in the out-of-band deletion process into four mutually exclusive states:

[0098] None (Initial State): The system default state. No out-of-band deletion process has been initiated in the directory. All directory operations, including normal reading, writing, modification, deletion, and querying, are supported without any locking or constraints.

[0099] Prepare (Locked State): The directory has passed most of the access control checks, completed global locking, prohibits all modification operations, and is waiting for the background data clearing task to start; it supports read-only query operations to ensure the normal operation of read-only front-end services.

[0100] Start (Start of clearing status): The system has started a background file data clearing task. ODE is deleting file data objects in the directory in batches; directory traversal and all modification operations are prohibited, and only basic attribute queries are supported.

[0101] Commit (cleared status complete): All file data objects in the directory have been deleted, the directory is empty, and it is waiting for the directory deletion operation to be executed; all modification operations are prohibited, and read-only queries are supported.

[0102] The following are the rules for legal state transitions: To ensure process standardization, data security, and state consistency, this invention strictly limits state transition paths, prohibits any illegal state jumps, and all state transitions must meet preset conditions, such as... Figure 3 As shown, the legal migration paths are as follows:

[0103] 1. None → Prepare: The trigger condition is that the directory passes all access condition checks and the directory is successfully locked, and then enters the preparation locking state;

[0104] 2. Prepare→None: The triggering conditions are directory locking failure, access verification failure, concurrent changes or abnormal rollbacks occurring within the verification period. Release the directory lock and restore the initial state.

[0105] 3. Prepare→Start: The triggering condition is that the directory's second snapshot verification passes and the ODE starts the data clearing task, entering the clearing state;

[0106] 4. Start→Commit: The trigger condition is that the batch deletion of all file data objects in the ODE directory is completed and the data is cleared, and the process enters the clearing completion state.

[0107] In addition to the four legal migration paths mentioned above, all other state migration combinations are defined as illegal operations. The system refuses to execute them and returns a unique error code to ensure that state transitions are standardized and orderly.

[0108] To enable the setting, querying, and management of directory status, this invention adds a custom extended attribute interface called dir.clear to MDS. This interface is lightweight and highly reliable, supports atomic status operations, and ensures consistency between status settings and queries.

[0109] Interface functionality: Supports setting, querying, modifying, and deleting directory states (None / Prepare / Start / Commit); supports atomic state switching to avoid exceptions caused by concurrent state modifications; supports permission verification, ensuring that only LMS can call the interface to modify the state, thus guaranteeing the security of state management.

[0110] In the aforementioned optimized method for automatic deletion of massive files in a distributed file system, this invention effectively avoids the risks of accidentally deleting valid business files and damaging the system's directory structure by using a lifecycle management service to filter directories to be deleted and performing multi-level access constraint checks, triggering out-of-band deletion only for directories that meet security conditions; by calling the custom extended attribute interface of the metadata service module to switch the directory status to a locked state and performing a secondary consistency check based on the directory modification time, it ensures that no concurrent changes occur to the directory during the deletion preparation period, guaranteeing data consistency and state stability during the deletion process; and by triggering the out-of-band deletion engine module to bypass the metadata service module and directly establish a communication connection with the data pool, and by using batch data... The data objects corresponding to all files in the directory are deleted via the access interface. The overhead of metadata deletion request processing, which is positively correlated with the number of files in the traditional mode and is borne by the metadata service module, is reduced to only requiring the metadata service module to participate in a limited number of state switching and directory deletion operations. This significantly reduces the load pressure on the metadata service module, eliminates resource conflicts between large-scale deletion tasks in the background and front-end business, and ensures low latency and stable operation of the front-end business. At the same time, with the help of the distributed parallel processing capability of the out-of-band deletion engine module, the batch deletion efficiency of massive files is greatly improved, and the storage resource release cycle is effectively shortened. Thus, efficient, low-interference, and highly reliable automatic deletion of massive files in scenarios with tens of billions of files is achieved.

[0111] In one embodiment, multi-level admission constraint verification includes:

[0112] Determine the file type of the directory to be deleted; all files in the directory to be deleted are regular data files, and the directory to be deleted does not contain subdirectories, symbolic link files, hard link files, or device files;

[0113] Determine the deletion scope of the directory to be deleted. The deletion scope is when all files in the directory to be deleted meet the preset deletion conditions.

[0114] Determine file reference constraints within the directory to be deleted; all files in the directory to be deleted have no valid references. Valid references include references to the metadata service module's memory cache, references to the metadata service module's persistent logs, references to the client's mount, references to the process's open function, and references to snapshots.

[0115] The total number of files in the directory to be deleted exceeds the preset threshold.

[0116] In this embodiment, to ensure the security, accuracy, and consistency of out-of-band deletion operations and to avoid accidental deletion of valid business files, disruption of the system directory structure, and data inconsistency, this invention designs multi-level, strict out-of-band deletion access constraints. All constraints are divided into two categories: LMS program self-verification and MDS interface verification. Only directories that simultaneously meet all constraints can perform out-of-band deletion operations; if any condition is not met, the system automatically downgrades to the traditional file interface deletion mode, ensuring the safe and reliable execution of deletion tasks.

[0117] Constraint 1: File type constraint (LMS self-validation): The directory to be processed is only allowed to contain ordinary files, and does not contain subdirectories, soft link files, hard link files, device files and other special files.

[0118] Design purpose: Special files have special storage structures, access methods and lifecycle management logic. Directly clearing them will lead to problems such as file residue and metadata abnormalities. Such directories are prohibited from being deleted outside the directory and will be automatically downgraded to the traditional mode.

[0119] Verification method: LMS traverses all files in the directory and verifies the file type one by one. If a special file is found, it is determined that the condition is not met.

[0120] Constraint 2: Deletion Scope Constraint (LMS Self-Verification): Out-of-band deletion only supports full directory clearing and does not support selective deletion of some files within a directory.

[0121] Design purpose: Selective deletion of some files requires updating directory metadata instead of directly deleting the directory. Metadata management is complex and consistency is difficult to guarantee. Out-of-band deletion focuses on batch full cleanup scenarios to ensure directory data consistency and metadata integrity. Some deletion requirements uniformly adopt traditional deletion solutions.

[0122] Verification method: LMS verifies the deletion task type; only tasks that are completely cleared can proceed to the next step.

[0123] Constraint 3: File Reference Constraint (MDS Interface Validation): No valid references may exist in any file within the directory, including MDS memory cache references, MDS log persistence references, client mount references, process open references, snapshot references, etc.

[0124] Design purpose: Files with valid references are active business files that are being accessed or used by front-end businesses. Directly deleting them by bypassing MDS would lead to serious consequences such as business interruption, data loss, and file corruption. This design aims to avoid the risk of accidentally deleting active business data.

[0125] Verification method: LMS calls the MDS-specific reference verification interface to query the file reference status in the directory. If a valid reference exists, it is determined that the condition is not met.

[0126] Constraint 4: File Quantity Constraint (LMS Self-Verification): The total number of files in the directory to be processed must be greater than or equal to a preset configurable threshold. The threshold is set to 100 by default and can be flexibly adjusted by the user according to the business scenario.

[0127] Design purpose: Out-of-band deletion improves efficiency through batch processing. In scenarios with small directories (fewer than the threshold), the traditional deletion process has lower interaction overhead and higher efficiency. Out-of-band deletion has no performance advantage. Threshold filtering avoids inefficient deletion scenarios in small directories and ensures optimal deletion performance in all scenarios.

[0128] Verification method: LMS counts the total number of files in the directory and compares it with a preset threshold. If the count is lower than the threshold, the condition is not met.

[0129] In one embodiment, after switching the state of the directory to be deleted to the locked state in step S3, the following constraint rules are applied to the directory's running state:

[0130] The directory to be deleted is allowed to perform read-only operations such as attribute query, node retrieval, permission query, and quota query.

[0131] The directory to be deleted is allowed to be traversed and return an empty result, but returning a list of files in the directory is prohibited;

[0132] Modification operations are prohibited on the directory to be deleted. Modification operations include changing directory attributes, creating files, moving files, renaming files, and changing permissions.

[0133] The directory to be deleted is allowed to be deleted only when the directory is in the "cleared and completed" state.

[0134] The quota statistics remain unchanged during the out-of-band deletion process.

[0135] Specifically, during out-of-band deletion, the directory is in a special intermediate state, and concurrent operations can easily lead to data corruption and metadata anomalies. To ensure directory stability, avoid concurrent conflicts, and maintain data consistency, this invention designs strict directory operation constraint rules covering the entire deletion process (preparation locking, start clearing, clearing completion), providing fine-grained control over directory operations and ensuring the safe and orderly execution of the deletion process.

[0136] Constraint Rule 1: Query operation compatibility ensures normal operation of front-end read-only services: During out-of-band deletion, all read-only operations such as directory attribute query, node retrieval, permission query, and quota query are supported without affecting the normal access of front-end services to basic directory information.

[0137] Specifically, the client can execute read-only interfaces such as lookup and getattr normally to obtain attribute information such as directory modification time, creation time, and permissions, ensuring that the front-end read-only business is unaware of and uninterrupted.

[0138] Constraint Rule 2: Directory traversal is blocked to avoid concurrent conflicts between traversal and clearing: During the deletion process, the readdir directory traversal operation will return an empty result, blocking the display of the file list in the directory.

[0139] Design purpose: Traversal operations read the list of files in the directory, and clear operations delete files in batches. Concurrent execution can easily lead to problems such as missing file counts, duplicate processing, and inconsistent traversal results. Blocking traversal operations can avoid such concurrent conflicts.

[0140] Rule 3: Disable modification operations to prevent data corruption caused by concurrent modifications: After the out-of-band deletion process is started, all directory modification operations are prohibited, including directory attribute modification, file creation, file movement, file renaming, permission modification, etc.

[0141] Design purpose: Modification operations can change directory structure, file status or data content. Executing them concurrently with deletion operations can lead to serious problems such as metadata chaos, data inconsistency, and file corruption. Disabling modification operations can ensure directory stability and data consistency.

[0142] Constraint Rule 4: Directory Deletion Timing Constraint to Ensure Process Closed Loop: Directory deletion can only be performed after the directory has been completely cleared of all files and its status has been updated to "Commit Clear Complete". Deleting directories in intermediate states such as "Prepare" or "Start" is prohibited.

[0143] Design purpose: There may be files that have not been deleted in the intermediate state directory. Directly deleting the directory will lead to problems such as file residue, metadata silos, and data loss. Strict timing constraints ensure a closed-loop deletion process and data integrity.

[0144] Constraint Rule 5: Quota Statistics Steady-State Constraint to Ensure Quota Data Accuracy: The system quota statistics remain unchanged during the directory deletion process, and the directory-level and user-level quota data are only updated once after the directory is completely deleted.

[0145] Design purpose: Frequent updates to quota data will generate a large number of metadata operations, consume MDS resources, and cause quota data fluctuations. One-time updates can reduce metadata operations, ensure the accuracy of quota statistics, and avoid frequent fluctuations affecting business billing and resource management.

[0146] The directory operation status constraint rules construct a full-process control system from five dimensions: business compatibility, conflict avoidance, data security, process standardization, and statistical accuracy. This effectively avoids the risks of concurrent operations, ensures the safe, stable, and orderly execution of out-of-band deletion processes, and maintains the consistency and integrity of system data.

[0147] In one embodiment, a distributed file system for automatically deleting and optimizing massive files is provided, comprising:

[0148] Filtering module: Used to filter directories to be deleted based on preset conditions, traverse the directories to be deleted, obtain the first current modification time of the directories to be deleted and store it;

[0149] Multi-level admission constraint verification module: used to perform multi-level admission constraint verification on the directory to be deleted, and execute the locking module when the verification conditions are met;

[0150] Locking module: Used to switch the status of the directory to be deleted to a locked state;

[0151] Time comparison module: After successful locking, it retrieves the second current modification time of the directory to be deleted; compares the second current modification time with the first modification time shown; if they match, it proceeds to the deletion module.

[0152] The deletion module is used to switch the state of the directory to be deleted to the start of clearing state, trigger the out-of-band deletion engine module to delete the directory to be deleted, so that the out-of-band deletion engine module can bypass MDS and establish a communication connection with the data pool, and use the batch data access interface to delete the data objects corresponding to all files in the directory to be deleted.

[0153] Clear Complete Status Flag Module: Used to receive the clear complete result returned by the out-of-band deletion engine module, update the status of the directory to be deleted to clear complete status, and mark that all file data in the directory to be deleted has been cleared and the directory is empty;

[0154] Directory update module: Used to call the rmdir directory deletion interface to delete empty directories and trigger the global quota statistics module to update the directory-level quota data.

[0155] Compared to existing traditional deletion technologies, this invention, through architectural restructuring, decoupling optimization, and refined state machine management, achieves substantial progress and application value in multiple dimensions, including MDS resource optimization, front-end and back-end business isolation, efficiency of massive file deletion, and data consistency. Specific beneficial effects are as follows:

[0156] First, it reduces the load on the MDS, minimizes resource conflicts between the front-end and back-end, and ensures the stable operation of core businesses. Traditional solutions require tens of thousands or even millions of metadata requests to the MDS to delete massive amounts of files. This invention innovatively integrates directory-level metadata operations, simplifying the MDS interaction requests for deleting massive files in a single directory to four lightweight state operations, significantly reducing the number of metadata requests. At the same time, it completely decouples data object deletion to an independent out-of-band engine, with the MDS retaining only lightweight security control capabilities. This releases the MDS's computing power and IO resources, decoupling the back-end batch deletion tasks from the front-end core businesses, reducing the impact on front-end business latency and throughput, and ensuring the stable operation of production-grade businesses.

[0157] Secondly, the efficiency of deleting massive amounts of files is improved. This invention introduces an out-of-band deletion engine, which, through distributed deployment and batch file data deletion, can quickly reclaim storage resources in the data pool. The use of directory-level metadata deletion operations greatly reduces the number of requests to the metadata pool, significantly improving metadata deletion efficiency. Simultaneously, it intelligently filters inefficient scenarios involving small directories using file quantity thresholds, achieving optimal performance across all scenarios.

[0158] Third, ensuring data and directory structure consistency. This invention guarantees data and directory structure consistency in concurrent scenarios through a security system that includes four-state machine full-process control, multi-level access constraints, global directory locking, secondary verification, and illegal operation interception.

[0159] The main technological innovations of this application are as follows:

[0160] 1. Integrated Directory and Metadata Operations: This invention innovatively integrates directory-level metadata operations, simplifying the MDS interaction requests for deleting massive files in a single directory to four lightweight state operations, significantly reducing the MDS load during directory file deletion. Furthermore, the use of directory-level metadata pool requests greatly reduces the load on the metadata pool, improving metadata deletion efficiency.

[0161] 2. Decoupling of architecture: Through an independent out-of-band deletion engine, the deletion of massive file data is completely decoupled from the MDS metadata service. MDS only retains lightweight state management and removes the overhead of deleting data objects.

[0162] 3. Refined state machine management: A four-state (None / Prepare / Start / Commit) catalog full lifecycle management state machine, with supporting legal state transition rules, to achieve standardization, traceability, controllability, and rollback capability for the entire out-of-band deletion process.

[0163] 4. Multi-level security access and concurrency protection: Construct four-dimensional access constraints based on file type, deletion range, file reference, and file quantity, combined with a full-process concurrency protection mechanism including global directory locking, secondary verification, traversal operation blocking, and write operation disabling.

[0164] 5. Steady-state quota update: The quota statistics steady-state constraint mechanism keeps the quota data stable throughout the deletion process, and updates it in batches at once after the process is closed, eliminating quota disorder and frequent fluctuations and ensuring the accuracy of data statistics.

[0165] In one embodiment, a computer device is also provided, wherein the processor of the computer device is used to provide computing and control capabilities, and the computer device loads and runs a computer program to implement the above-described method for automatically deleting and optimizing massive files in a distributed file system.

[0166] In one embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the methods described in the embodiments above.

[0167] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for automatically deleting massive amounts of files in a distributed file system, characterized in that, Applied to lifecycle management services, the method includes: Step S1: Filter out directories to be deleted according to preset conditions, traverse the directories to be deleted, obtain the first current modification time of the directories to be deleted and store it; Step S2: Perform multi-level admission constraint verification on the directory to be deleted. If the verification conditions are met, proceed to the next step. Step S3: Switch the status of the directory to be deleted to locked; Step S4: After successful locking, re-obtain the second current modification time of the directory to be deleted; compare the second current modification time with the first modification time shown. If they match, proceed to the next step. Step S5: Switch the state of the directory to be deleted to the start-clear state, trigger the out-of-band deletion engine module to delete the directory to be deleted, so that the out-of-band deletion engine module bypasses MDS and establishes a communication connection with the data pool, and uses the batch data access interface to delete the data objects corresponding to all files in the directory to be deleted; Step S6: Receive the clearing completion result returned by the out-of-band deletion engine module, update the status of the directory to be deleted to the clearing completion status, and mark that all file data in the directory to be deleted has been cleared and the directory is empty; Step S7: Call the rmdir directory deletion interface to delete empty directories and trigger the global quota statistics module to update the directory-level quota data.

2. The method for automatically deleting massive files in a distributed file system according to claim 1, characterized in that, Multi-level admission constraint verification includes: Determine the file type of the directory to be deleted; all files in the directory to be deleted are regular data files, and the directory to be deleted does not contain subdirectories, symbolic link files, hard link files, or device files; Determine the deletion range of the directory to be deleted, wherein the deletion range is that all files in the directory to be deleted meet the preset deletion conditions; Determine the file reference constraints within the directory to be deleted; all files in the directory to be deleted have no valid references, and the valid references include metadata service module memory cache references, metadata service module log persistence references, client mount references, process open references, and snapshot references; The total number of files in the directory to be deleted is greater than the preset configuration threshold.

3. The method for automatically deleting massive files in a distributed file system according to claim 1, characterized in that, After switching the status of the directory to be deleted to the locked state in step S3, the following constraint rules are applied to the directory's running state: The directory to be deleted is allowed to perform read-only operations such as attribute query, node retrieval, permission query, and quota query. The directory to be deleted is allowed to be traversed and return an empty result, but returning a list of files in the directory is prohibited; The directory to be deleted is prohibited from performing modification operations, which include modifying directory attributes, creating files, moving files, renaming files, and modifying permissions. The directory to be deleted is allowed to be deleted only when the directory to be deleted is in a cleared state; The quota statistics remain unchanged during the out-of-band deletion process.

4. The method for automatically deleting massive files in a distributed file system according to claim 1, characterized in that, The preset condition is that the file modification time exceeds a set value.

5. The method for automatically deleting massive files in a distributed file system according to claim 1, characterized in that, Switching the status of the directory to be deleted to locked, started clearing, or clearing complete is achieved by calling the custom dir.clear extended property interface of the metadata service module.

6. The method for automatically deleting massive files in a distributed file system according to claim 1, characterized in that, The lifecycle management service is deployed independently of the metadata service module, adopts a distributed architecture design, and supports multi-node cluster deployment.

7. The method for automatically deleting massive files in a distributed file system according to claim 1, characterized in that, When the multi-level access constraint conditions of the directory to be deleted are not met, or the second current modification time is inconsistent with the first modification time shown, or the locking fails when the state of the directory to be deleted is switched to the locked state, the state of the directory to be deleted is rolled back to the initial state, the directory lock is released, and the deletion task is rolled back to the standard file deletion interface for execution.

8. A distributed file system for automatically deleting and optimizing massive files, characterized in that, The system includes: Filtering module: Used to filter directories to be deleted based on preset conditions, traverse the directories to be deleted, obtain the first current modification time of the directories to be deleted and store it; Multi-level admission constraint verification module: used to perform multi-level admission constraint verification on the directory to be deleted, and execute the locking module when the verification conditions are met; Locking module: Used to switch the status of the directory to be deleted to a locked state; Time comparison module: After successful locking, it retrieves the second current modification time of the directory to be deleted; compares the second current modification time with the first modification time shown; if they match, it proceeds to the deletion module. The deletion module is used to switch the state of the directory to be deleted to the start of clearing state, trigger the out-of-band deletion engine module to delete the directory to be deleted, so that the out-of-band deletion engine module can bypass MDS and establish a communication connection with the data pool, and use the batch data access interface to delete the data objects corresponding to all files in the directory to be deleted. Clear Complete Status Flag Module: Used to receive the clear complete result returned by the out-of-band deletion engine module, update the status of the directory to be deleted to clear complete status, and mark that all file data in the directory to be deleted has been cleared and the directory is empty; Directory update module: Used to call the rmdir directory deletion interface to delete empty directories and trigger the global quota statistics module to update the directory-level quota data.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in claims 1-7.

Images