Design method and system for HDFS recycle bin with intelligent isolation and asynchronous cleaning

Abstract

The invention discloses an intelligent isolation and asynchronous cleaning HDFS recycle bin design method and system, and the method comprises the steps: creating an independent recycle bin directory for each user; one metadata file is maintained under each recycle bin directory; setting access permissions of a specific user and a system administrator for each recycle bin directory; metadata in a recycle bin of files is stored in a RocksDB, and two tables are designed in the RocksDB: the table 1 is used for clearing out expired files, and the table 2 is used for recovering and displaying the files; when the file is deleted, determining a recycle bin ID of a user according to a path before the file is deleted, and inserting recycle bin metadata of the file into the RocksDB; the specified file is found and recovered by querying metadata in the table 2, and the RocksDB is updated; and cleaning the expired file regularly or as required by utilizing the self-defined expiration time in the table 1, and updating the RocksDB. According to the method and the system, the problems of performance bottleneck and management complexity of the Hadoop recycle bin in a high-concurrency and large-data-volume scene are solved.

Description

Technical Field

[0001] The present invention relates to the field of Hadoop recycle bin design, and in particular to a design method and system for an HDFS recycle bin with intelligent isolation and asynchronous cleaning. Background Art

[0002] Although the Hadoop recycle bin design is simple and effective, it also has some drawbacks in practical applications, especially in scenarios with high concurrency and large amounts of data, the following problems may occur:

[0003] 1. In a high-concurrency and large-data-volume environment, multiple users may delete a large number of files at the same time. Each deletion operation will update the metadata in the Recycle Bin directory.

[0004] (1) Frequent updates of the recycle bin metadata result in a large number of write operations, which occupy a large amount of NameNode resources.

[0005] (2) When multiple users delete files at the same time, it will cause metadata lock competition, resulting in increased system lock waiting time, further reducing the concurrency and response speed of operations.

[0006] (3) Since NameNode needs to process a large number of metadata update requests, the response time of file deletion operations will increase significantly.

[0007] 2. In the default design of Hadoop, all users' deleted files are stored in the same / Trash directory. This design will cause the following problems in scenarios with high concurrency and large data volumes:

[0008] (1) When a large number of users delete files, the / Trash directory will expand rapidly, containing a large number of files and subdirectories, which increases the complexity of directory management. The entire Recycle Bin needs to be scanned during the recovery operation, which prolongs the recovery time.

[0009] (2) When multiple users operate the / Trash directory at the same time, it is easy to cause file name conflicts and operation conflicts, increasing the complexity of the system.

[0010] (3) When all users share the same / Trash directory, the design and implementation of management tools and monitoring systems becomes more difficult.

[0011] (4) Since all users share the same recycle bin directory, it is difficult to set different recycling policies according to the needs of different users.

[0012] 3. Although it is simple to set the retention time of the recycle bin by configuring parameters for regular cleaning, it has limitations and disadvantages in practical applications:

[0013] (1) The current recycle bin retention time is a global configuration and cannot be personalized for different users or file types, resulting in poor flexibility.

[0014] (2) Because the cleanup operation requires traversing and processing a large amount of metadata, if the cleanup operation is triggered during the system peak period, it will cause a sudden increase in system load and affect the performance of normal business operations.

[0015] (3) The one-time periodic cleanup operation is complex and time-consuming, and is prone to failure due to various reasons (such as system errors, insufficient resources, etc.), resulting in the inability to clean up expired files in a timely manner. Summary of the invention

[0016] In order to solve the performance bottleneck and management complexity problems of Hadoop recycle bin in high concurrency and large data volume scenarios, the present invention provides a design method and system for HDFS recycle bin with intelligent isolation and asynchronous cleaning, which solves the above problems by setting independent recycle bin directories for different users, putting the file complete deletion operation into the background task queue, asynchronously cleaning data, and introducing RocksDB to store metadata in the recycle bin.

[0017] To achieve the above purpose, the present invention adopts the following technical solutions:

[0018] In one embodiment of the present invention, a design method for an HDFS recycle bin with intelligent isolation and asynchronous cleanup is proposed, the method comprising:

[0019] Create an independent recycle bin directory for each user; maintain a metadata file under each recycle bin directory to record relevant information about deleted files; set access permissions for specific users and system administrators for each recycle bin directory;

[0020] The metadata of the file's recycle bin is stored in RocksDB. Two tables are designed in RocksDB: Table 1 stores the combination of the user's recycle bin ID, the file's deletion time, the custom expiration time, and the file ID, which is used to clean up expired files; Table 2 stores the combination of the user's recycle bin ID, the file's path before being deleted, and the file ID, which is used for file recovery and display;

[0021] When a file is deleted, the user's recycle bin ID is determined based on the file's path before deletion, and the metadata in the file's recycle bin is inserted into RocksDB;

[0022] By querying the metadata in Table 2, find the specified file and restore the file from the recycle bin to the original location, and update RocksDB to delete the related records;

[0023] Use the custom expiration time in Table 1 to clean up expired files in the recycle bin regularly or on demand, and update RocksDB to delete related records.

[0024] Furthermore, the records in Table 1 and Table 2 in RocksDB all use composite keys. The composite key of Table 1 is: user's Recycle Bin ID: file deletion time, and the composite key of Table 2 is: user's Recycle Bin ID: file path before deletion. The key-value pair of Table 1 is: key is user's Recycle Bin ID: file deletion time, and value is file ID, and the key-value pair of Table 2 is: key is user's Recycle Bin ID: file path before deletion, and value is file ID.

[0025] Furthermore, the expired file information is added to the background GC task queue, and the GC thread asynchronously performs the cleanup operation.

[0026] In one embodiment of the present invention, a design system for an HDFS recycle bin with intelligent isolation and asynchronous cleanup is also proposed, and the system includes:

[0027] The recycle bin directory design module is used to create an independent recycle bin directory for each user; a metadata file is maintained under each recycle bin directory to record relevant information of deleted files; and access rights for specific users and system administrators are set for each recycle bin directory;

[0028] The metadata storage design module is used to store the metadata in the file's recycle bin in RocksDB. Two tables are designed in RocksDB: Table 1 stores the combination of the user's recycle bin ID, the file's deletion time, the custom expiration time, and the file ID, which is used to clean up expired files; Table 2 stores the combination of the user's recycle bin ID, the file's path before being deleted, and the file ID, which is used for file recovery and display;

[0029] The file operation module is used to determine the user's recycle bin ID according to the path of the file before being deleted when the file is deleted, and insert the metadata in the recycle bin of the file into RocksDB; by querying the metadata in Table 2, find the specified file and restore the file from the recycle bin to the original location, and update RocksDB to delete the relevant records;

[0030] The file maintenance module is used to use the custom expiration time in Table 1 to clean up expired files in the recycle bin regularly or on demand, and update RocksDB to delete related records.

[0031] Furthermore, the records in Table 1 and Table 2 in RocksDB all use composite keys. The composite key of Table 1 is: user's Recycle Bin ID: file deletion time, and the composite key of Table 2 is: user's Recycle Bin ID: file path before deletion. The key-value pair of Table 1 is: key is user's Recycle Bin ID: file deletion time, and value is file ID, and the key-value pair of Table 2 is: key is user's Recycle Bin ID: file path before deletion, and value is file ID.

[0032] Furthermore, the expired file information is added to the background GC task queue, and the GC thread asynchronously performs the cleanup operation.

[0033] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a design method for the aforementioned intelligent isolation and asynchronous cleanup HDFS recycle bin is designed.

[0034] In one embodiment of the present invention, a computer-readable storage medium is also proposed, which stores a computer program for a design method of an HDFS recycle bin that performs intelligent isolation and asynchronous cleanup.

[0035] Beneficial effects:

[0036] 1. The independent recycle bin directory design of the present invention ensures the isolation of recycle bins between different users, and effectively solves the management complexity problem caused by multiple users sharing one directory in the traditional Hadoop recycle bin design.

[0037] 2. The present invention uses RocksDB's efficient sequential read and write performance to store metadata in the recycle bin in RocksDB. The use of RocksDB significantly improves the metadata read and write performance and concurrent processing capabilities, including efficient file recovery operations, etc. This is a major improvement compared to the traditional Hadoop recycle bin metadata management method.

[0038] 3. The present invention introduces the concept of custom expiration time, which can set different retention times for different users or file types. The combination of regular scanning and on-demand cleaning ensures that the system can efficiently clean up expired files under different load conditions, free up storage space, and avoid negative impact on system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic flow chart of a design method of an HDFS recycle bin for intelligent isolation and asynchronous cleanup according to an embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of the design system structure of an HDFS recycle bin for intelligent isolation and asynchronous cleanup according to an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0042] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a system, system, device, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0044] According to the implementation mode of the present invention, a design method for an HDFS recycle bin with intelligent isolation and asynchronous cleaning is proposed. By setting independent recycle bin directories for different users, the file complete deletion operation is placed in the background task queue, and data is cleaned up asynchronously. At the same time, RocksDB is introduced to store metadata in the recycle bin, so as to solve the performance bottleneck and management complexity problems of the Hadoop recycle bin in high concurrency and large data volume scenarios.

[0045] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0046] Figure 1 1 is a flow chart of a design method for an HDFS recycle bin with intelligent isolation and asynchronous cleanup according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0047] 1. Independent Recycle Bin directory design: Create an independent Recycle Bin directory for each user. For example, the Recycle Bin directory for user A may be / Trash / userA, while the Recycle Bin directory for user B is / Trash / userB. The Recycle Bin directory structure includes information such as deletion time and file ID to support efficient file recovery and cleanup operations. When a user deletes a file, the file is moved to its corresponding Recycle Bin directory instead of a shared Recycle Bin directory.

[0048] 2. Store the file's recycle bin metadata in RocksDB, which is a lightweight database that uses key-value data. It uses its efficient sequential read and write performance to improve metadata read and write performance and concurrent processing capabilities. Two tables are designed in RocksDB for different purposes:

[0049] Table 1: A combination of the user's recycle bin ID, file deletion time, custom expiration time, and file ID, which is used to clean up expired files.

[0050] Table 2: A combination of the user's recycle bin ID, the file's path before being deleted, and the file ID, for file recovery and display.

[0051] 3. File deletion operation

[0052] Metadata update: When a file is deleted, the user's recycle bin ID is determined based on the file's path before it is deleted, and the file's metadata is inserted into RocksDB. Tables 1 and 2 update the file's deletion time and the file's path information before it is deleted, respectively.

[0053] Asynchronous cleanup: The file's metadata will be placed in the background GC task queue before being completely deleted, and the GC thread will handle the deletion operation asynchronously, reducing the pressure on the NameNode.

[0054] 4. File recovery operation

[0055] By querying the metadata in Table 2, you can quickly find and restore the specified file. The restore operation restores the file from the recycle bin to the original location and updates RocksDB to remove the related records.

[0056] 5. Expired file cleanup

[0057] Use the custom expiration time information in Table 1 to clean up expired files regularly or on demand. Delete expired files from the recycle bin and update RocksDB based on the comparison between the current time and the custom expiration time.

[0058] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0059] In order to explain more clearly the design method of the HDFS recycle bin with intelligent isolation and asynchronous cleaning, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0060] Example:

[0061] 1. Independent Recycle Bin Directory Design

[0062] (1) Create an independent directory: Create an independent recycle bin directory for each user.

[0063] For example: User A's Recycle Bin directory is / Trash / userA. User B's Recycle Bin directory is / Trash / userB. These directories are used to store files deleted by users to avoid confusion with other users' data.

[0064] (2) Metadata file structure: A metadata file is maintained in each recycle bin directory to record relevant information of deleted files, such as the deletion time of the file, file ID, the path of the file before deletion, etc. The metadata file structure can be in JSON or XML format.

[0065] Example data:

[0066] {

[0067] "deleted_files":[

[0068] {

[0069] "file_id":"12345",

[0070] "original_path":" / user / data / file1.txt",

[0071] "delete_time":"2024-07-2710:00:00",

[0072] "custom_expiration_time":"2024-09-2710:00:00",

[0073] "size":"15MB"

[0074] },

[0075] {

[0076] "file_id":"12346",

[0077] "original_path":" / user / data / file2.txt",

[0078] "delete_time":"2024-07-2811:00:00",

[0079] "custom_expiration_time":"2024-08-2811:00:00",

[0080] "size":"20MB"

[0081] }

[0082] / / Other files ]

[0084] }

[0085] (3) Permissions and security. By setting directory permissions, ensure that only the user and the system administrator can access their respective recycle bin directories. Use Unix permissions or ACL (access control list) to implement secure access control of directories.

[0086] For each user, set the owner of the Recycle Bin directory to the user himself and set the permissions to allow only the owner to read and write.

[0087] hdfs dfs-mkdir-p / Trash / userA

[0088] hdfs dfs-chown userA:userA / Trash / userA

[0089] hdfs dfs-chmod 700 / Trash / userA

[0090] HDFS ACL provides more fine-grained permission control and can set access permissions for specific users and system administrators for each recycle bin directory.

[0091] hdfs dfs-mkdir-p / Trash / userA

[0092] hdfs dfs-setfacl-m user:userA:rwx / Trash / userA

[0093] hdfs dfs-setfacl-m user:admin:rwx / Trash / userA

[0094] 2. Use RocksDB to store metadata in the recycle bin

[0095] (1) Table structure design.

[0096] RocksDB has two tables designed for different purposes.

[0097] Table 1: Expired file management table

[0098] Fields:

[0099] TrashID: The user's Trash ID, such as userA or userB.

[0100] DeleteTime: The deletion time of the file, used to determine whether the file is expired.

[0101] CustomExpirationTime: Custom expiration time allows you to set different retention times for different files. This time is set by the user when deleting the file, or automatically assigned by the system based on the file type or policy.

[0102] FileID: A unique identifier for a file, used to uniquely identify the file in RocksDB.

[0103] Example data:

[0104] TrashID DeleteTime CustomExpirationTime FileID

[0105] userA 2024-07-2710:00:00 2024-09-2710:00:00 12345

[0106] userB 2024-07-2711:00:00 2024-09-2711:00:00 12346

[0107] Table 2: File path index table

[0108] Fields:

[0109] TrashID: The user's Trash ID, such as userA or userB.

[0110] FilePath: The path of the file before it was deleted.

[0111] FileID: A unique identifier for the file.

[0112] Example data:

[0113] TrashID FilePath FileID

[0114] userA / user / data / file1.txt 12345

[0115] userB / user / data / file2.txt 12346

[0116] Using TrashID can isolate metadata of different users and avoid data confusion.

[0117] TrashID’s role:

[0118] It is the user's Recycle Bin identifier, which serves to isolate and quickly locate metadata during file deletion, recovery, and cleanup.

[0119] By assigning a unique TrashID to each user, it is possible to ensure that the recycle bin files and metadata of different users are not mixed up, thus achieving resource isolation.

[0120] (2) Index rules.

[0121] Composite key-based indexes enable efficient data storage and retrieval.

[0122] Composite key design: To optimize search and cleanup efficiency, composite keys are used for records in Table 1 and Table 2.

[0123] Table 1 composite key: TrashID:DeleteTime.

[0124] Table 2 Key combination: TrashID:FilePath.

[0125] The key-value pairs in Table 1 are: the key is TrashID:DeleteTime and the value is FileID.

[0126] The key-value pairs in Table 2 are: the key is TrashID:FilePath and the value is FileID.

[0127] Example data:

[0128] User userA deleted the file / user / data / file1.txt at 10:00 on July 27, 2024, and set the custom expiration time to 2024-09-27 10:00:00.

[0129] The generated composite key and key-value pair are:

[0130] Table 1: userA:2024-07-27 10:00:00->(12345,2024-09-2710:00:00)

[0131] Table 2: userA: / user / data / file1.txt->12345

[0132] 3. File deletion operation

[0133] When a user deletes a file, the file is not physically moved to the user's Recycle Bin directory. The system marks the file as deleted by modifying the metadata and records the relevant metadata in the Recycle Bin directory.

[0134] (1) Insert metadata.

[0135] The process of inserting metadata into table 1:

[0136] Capture file deletion events. When a user deletes a file, the system captures this event and collects related metadata, including user ID (TrashID), file path (FilePath), file ID (FileID), deletion time (DeleteTime), and expiration time (CustomExpirationTime).

[0137] Generate a composite key: TrashID and DeleteTime.

[0138] Set the key-value pair: the corresponding values ​​include FileID and CustomExpirationTime, which are used to identify the file and the custom expiration time.

[0139] Insert metadata: Insert the above key-value pairs into table 1 of RocksDB.

[0140] Example data:

[0141] User: userA

[0142] File path: / user / data / file1.txt

[0143] FileID:12345

[0144] Deletion time: 2024-07-27 10:00:00

[0145] Custom expiration time: 2024-09-27 10:00:00

[0146] Table 1 inserts data: userA:2024-07-27 10:00:00->(12345,2024-09-2710:00:00)

[0147] The process of inserting metadata into table 2:

[0148] Generate a composite key: TrashID and FilePath.

[0149] Set a key-value pair: the value is the FileID of the file.

[0150] Insert metadata: Insert the above key-value pairs into table 2 of RocksDB.

[0151] Example data:

[0152] User: userA

[0153] File path: / user / data / file1.txt

[0154] FileID:12345

[0155] Deletion time: 2024-07-27 10:00:00

[0156] Custom expiration time: 2024-09-27 10:00:00

[0157] Table 2 inserts data: userA: / user / data / file1.txt->12345

[0158] (2) Update metadata.

[0159] Updating metadata mainly occurs in the following situations: when a file is restored from the Recycle Bin, or when the file's metadata (such as a custom expiration time) changes and needs to be updated.

[0160] Trigger conditions: This process is not periodic, but is triggered by specific operations, such as file recovery, user request to change metadata, or system cleanup of expired files.

[0161] For details on file recovery scenarios, see step 4.

[0162] Modifying the custom expiration time of deleted files results in updating metadata in the following scenarios:

[0163] Find records:

[0164] Table 1: Use the key combination (TrashID:FileID) to find the record to be modified and obtain the current metadata of the file.

[0165] Since the user, file name and file path have not changed, the data in Table 2 does not need to be modified.

[0166] Delete old data:

[0167] Table 1: Delete old records, such as userA:12345:2024-07-2710:00:00.

[0168] Insert new data:

[0169] Table 1: Insert updated records with custom expiration time.

[0170] Key combination: userA:12345:2024-07-2710:00:00

[0171] Value: 12345,2024-10-0110:00:00

[0172] Example data:

[0173] Newly inserted record in table 1: userA:12345:2024-07-2710:00:00->12345,2024-10-0110:00:00

[0174] 4. File recovery operation

[0175] For file restore operations, the system only needs to update the metadata to restore the file, making it available again to the user.

[0176] The following takes the restoration of the / Trash / userA / 12345 file of user A as an example.

[0177] (1) Find existing metadata.

[0178] Table 1:

[0179] Goal: Find records containing information about deleted files.

[0180] Composite key: Use the composite key (TrashID:FileID) to find the corresponding metadata record in Table 1.

[0181] Example data:

[0182] Key combination: userA:12345:2024-07-2710:00:00.

[0183] Record: 12345, 2024-09-27 10:00:00.

[0184] Table 2:

[0185] Goal: Find the record of file path information.

[0186] Composite key: Use the composite key (TrashID:FilePath) to find the record

[0187] Example data:

[0188] Key combination: userA: / Trash / userA / 12345.

[0189] Record: 12345 (including FileID).

[0190] This process is to accurately locate the records that need to be deleted to ensure that the metadata of other files is not deleted by mistake.

[0191] (2) Restore files.

[0192] Modify metadata: Update the file's metadata status to indicate that the file has been restored from the Trash. This typically involves removing the marker from the Trash path (such as / Trash / userA / ) and restoring the file status to a "normal" state.

[0193] Confirm Restore: Ensures that the file's metadata is updated and notifies the user that the file has been successfully restored.

[0194] (3) Delete metadata.

[0195] Delete old metadata: Delete old metadata records related to the restored files from Table 1 and Table 2 of RocksDB.

[0196] Example data:

[0197] Delete the metadata record of the restored file from Table 1.

[0198] Delete record: userA:12345:2024-07-2710:00:00.

[0199] Delete the file path record from Table 2.

[0200] Delete record: userA: / Trash / userA / 12345.

[0201] 5. Expired file cleanup

[0202] Expired file cleanup is a critical maintenance task. It ensures that the system regularly cleans up files that are no longer needed to free up storage space and maintain system performance. While supporting the ability to configure different expiration times for different users or different files of the same user, the system needs to flexibly manage and execute cleanup tasks.

[0203] (1) Regular scanning tasks.

[0204] The system triggers a scan task periodically (such as daily or weekly) to check the file deletion records in RocksDB's Table 1. The system traverses all records in Table 1, compares the current time with the CustomExpirationTime of each file, and identifies expired files.

[0205] (2) Calculation of retention time threshold

[0206] Get the current system time and compare the current time with CustomExpirationTime. If the current time exceeds CustomExpirationTime, the file is considered expired.

[0207] (3) Record expired files

[0208] All identified expired file information (including TrashID, FileID, DeleteTime and CustomExpirationTime) is recorded in a list to be cleaned up. These expired file information are added to the background cleanup task queue so that the cleanup operation can be performed asynchronously.

[0209] (4) Cleaning operation

[0210] A dedicated GC (garbage collection) thread extracts expired file information from the task queue and performs cleanup tasks. The system deletes file data from physical storage based on the records. For Hadoop systems, this usually involves deleting file blocks from HDFS. All metadata records related to these files are deleted from Tables 1 and 2 of RocksDB.

[0211] (5) Record the cleanup log

[0212] Log information of the cleanup operation is recorded, including information of deleted files, cleanup time and results, etc. If an error occurs during the cleanup process, the system should record the error information and try to clean up again or notify the administrator.

[0213] Example data:

[0214] Regular scanning and identification of expired files

[0215] Current time: 2024-09-28

[0216] Table 1 Records

[0217] userA:2024-07-2710:00:00->(12345,2024-09-2710:00:00)

[0218] userB:2024-07-2711:00:00->(12346,2024-08-2711:00:00)

[0219] Expiration judgment: The CustomExpirationTime of file ID 12345 is 2024-09-27 10:00:00, which has expired; the CustomExpirationTime of file ID 12346 is 2024-08-2711:00:00, which has also expired

[0220] Record expired files:

[0221] List to be cleaned up

[0222] userA:12345

[0223] userB:12346

[0224] Task queue: Add the above file information to the background GC task queue.

[0225] Cleanup operations:

[0226] Physical deletion path

[0227] Delete file data from / Trash / userA / 12345

[0228] Delete file data from / Trash / userB / 12346

[0229] Metadata removal:

[0230] The data deleted from Table 1 are as follows:

[0231] userA:2024-07-2710:00:00->(12345,2024-09-2710:00:00)

[0232] userB:2024-07-2711:00:00->(12346,2024-08-2711:00:00)

[0233] The data deleted from Table 2 are as follows:

[0234] userA: / user / data / file1.txt->12345

[0235] userB: / user / data / file2.txt->12346

[0236] Based on the same inventive concept, the present invention also proposes a design system for an HDFS recycle bin with intelligent isolation and asynchronous cleanup. The implementation of the system can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware designed for a predetermined function. Although the system described in the following embodiments is preferably designed with software, the design of hardware, or a combination of software and hardware is also possible and conceivable.

[0237] Figure 2 1 is a schematic diagram of the design system structure of the HDFS recycle bin for intelligent isolation and asynchronous cleaning according to an embodiment of the present invention. Figure 2 As shown, the system includes:

[0238] The recycle bin directory design module 101 is used to create an independent recycle bin directory for each user; maintain a metadata file under each recycle bin directory to record relevant information of deleted files; and set access rights for specific users and system administrators for each recycle bin directory.

[0239] The metadata storage design module 102 is used to store the metadata in the file's recycle bin in RocksDB. Two tables are designed in RocksDB: Table 1 stores the combination of the user's recycle bin ID, the file's deletion time, the custom expiration time, and the file ID, which is used to clean up expired files; Table 2 stores the combination of the user's recycle bin ID, the file's path before being deleted, and the file ID, which is used for file recovery and display;

[0240] The records in Table 1 and Table 2 in RocksDB both use composite keys. The composite key of Table 1 is: user's Recycle Bin ID: file deletion time, and the composite key of Table 2 is: user's Recycle Bin ID: file path before deletion. The key-value pair of Table 1 is: key is user's Recycle Bin ID: file deletion time, and value is file ID, and the key-value pair of Table 2 is: key is user's Recycle Bin ID: file path before deletion, and value is file ID.

[0241] The file operation module 103 is used to determine the user's recycle bin ID according to the path of the file before being deleted when the file is deleted, and insert the metadata in the recycle bin of the file into RocksDB; by querying the metadata in Table 2, find the specified file and restore the file from the recycle bin to the original location, and update RocksDB to delete the relevant records.

[0242] The file maintenance module 104 is used to use the self-defined expiration time in Table 1 to clean up expired files in the recycle bin regularly or on demand, and update RocksDB to delete related records; the expired file information is added to the background GC task queue, and the GC thread asynchronously performs the cleaning operation.

[0243] It should be noted that although several modules of the design system of the HDFS recycle bin with intelligent isolation and asynchronous cleanup are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0244] Based on the above invention concept, Figure 3As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220, and a design method for the aforementioned intelligent isolation and asynchronous cleanup HDFS recycle bin is designed when the processor 220 executes the computer program 230.

[0245] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium, which stores a computer program for the design method of the HDFS recycle bin that executes the aforementioned intelligent isolation and asynchronous cleanup.

[0246] The design method and system of the HDFS recycle bin with intelligent isolation and asynchronous cleaning proposed in the present invention have the following highlights:

[0247] 1. The independent recycle bin directory design ensures the isolation of recycle bins between different users, effectively solving the management complexity problem caused by multiple users sharing one directory in the traditional Hadoop recycle bin design.

[0248] 2. Use RocksDB's efficient sequential read and write performance to store the recycle bin metadata in RocksDB. The use of RocksDB significantly improves the metadata read and write performance and concurrent processing capabilities, including efficient file recovery operations, which is a major improvement compared to the traditional Hadoop recycle bin metadata management method.

[0249] 3. The concept of custom expiration time is introduced, and different retention times can be set for different users or file types. The combination of regular scanning and on-demand cleaning ensures that the system can efficiently clean up expired files under different load conditions, free up storage space, and avoid negative impact on system performance.

[0250] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0251] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A design method for an HDFS recycle bin with intelligent isolation and asynchronous cleanup, characterized in that: The method includes: Create an independent recycle bin directory for each user; maintain a metadata file under each recycle bin directory to record relevant information about deleted files; set access permissions for specific users and system administrators for each recycle bin directory; The metadata of the file's recycle bin is stored in RocksDB. Two tables are designed in RocksDB: Table 1 stores the combination of the user's recycle bin ID, the file's deletion time, the custom expiration time, and the file ID, which is used to clean up expired files; Table 2 stores the combination of the user's recycle bin ID, the file's path before being deleted, and the file ID, which is used for file recovery and display; When a file is deleted, the user's recycle bin ID is determined based on the file's path before deletion, and the metadata in the file's recycle bin is inserted into RocksDB; By querying the metadata in Table 2, find the specified file and restore the file from the recycle bin to the original location, and update RocksDB to delete the related records; Use the custom expiration time in Table 1 to clean up expired files in the recycle bin regularly or on demand, and update RocksDB to delete related records.

2. The design method of HDFS recycle bin with intelligent isolation and asynchronous cleaning according to claim 1 is characterized in that: The records in Table 1 and Table 2 in the RocksDB use composite keys. The composite key of Table 1 is: user's Recycle Bin ID: file deletion time, and the composite key of Table 2 is: user's Recycle Bin ID: file path before deletion. The key-value pair of Table 1 is: key is user's Recycle Bin ID: file deletion time, and value is file ID, and the key-value pair of Table 2 is: key is user's Recycle Bin ID: file path before deletion, and value is file ID.

3. The design method of HDFS recycle bin with intelligent isolation and asynchronous cleaning according to claim 1 is characterized in that: Add the expired file information to the background GC task queue, and the GC thread will perform the cleanup operation asynchronously.

4. A design system for HDFS recycle bin with intelligent isolation and asynchronous cleanup, characterized in that: The system includes: The recycle bin directory design module is used to create an independent recycle bin directory for each user; a metadata file is maintained under each recycle bin directory to record relevant information of deleted files; and access rights for specific users and system administrators are set for each recycle bin directory; The metadata storage design module is used to store the metadata in the file's recycle bin in RocksDB. Two tables are designed in RocksDB: Table 1 stores the combination of the user's recycle bin ID, the file's deletion time, the custom expiration time, and the file ID, which is used to clean up expired files; Table 2 stores the combination of the user's recycle bin ID, the file's path before being deleted, and the file ID, which is used for file recovery and display; The file operation module is used to determine the user's recycle bin ID according to the path of the file before being deleted when the file is deleted, and insert the metadata in the recycle bin of the file into RocksDB; by querying the metadata in Table 2, find the specified file and restore the file from the recycle bin to the original location, and update RocksDB to delete the relevant records; The file maintenance module is used to use the custom expiration time in Table 1 to clean up expired files in the recycle bin regularly or on demand, and update RocksDB to delete related records.

5. The design system of HDFS recycle bin with intelligent isolation and asynchronous cleaning according to claim 4 is characterized in that: The records in Table 1 and Table 2 in the RocksDB use composite keys. The composite key of Table 1 is: user's Recycle Bin ID: file deletion time, and the composite key of Table 2 is: user's Recycle Bin ID: file path before deletion. The key-value pair of Table 1 is: key is user's Recycle Bin ID: file deletion time, and value is file ID, and the key-value pair of Table 2 is: key is user's Recycle Bin ID: file path before deletion, and value is file ID.

6. The design system of HDFS recycle bin with intelligent isolation and asynchronous cleaning according to claim 4, characterized in that: Add the expired file information to the background GC task queue, and the GC thread will perform the cleanup operation asynchronously.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 3.

Images