Storage object deleting method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a storage object deleting method and device, a storage medium and electronic equipment, and relates to the field of computers.The method comprises the steps that a version information set corresponding to a free directory in a metadata server is loaded, the version information set comprises multiple pieces of version information corresponding to multiple storage objects, and the version information is stored in the metadata server; the free directory is used for caching a plurality of storage objects; traversing the multiple pieces of version information to obtain at least one piece of version information meeting an object deletion condition; and deleting at least one storage object corresponding to the at least one piece of version information in the free directory.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for deleting stored objects, a storage medium, and an electronic device. Background Technology

[0002] In a distributed file system, when a user requests the deletion of a storage object (such as a file or directory), the system typically marks the object as "logically deleted" and moves its information to a detached directory. A background service then performs the physical deletion asynchronously. However, when the number of storage objects in the detached directory is too large, especially in large cluster environments or high-concurrency scenarios, fully loading all items in the detached directory for deletion significantly increases the startup time of the metadata server (MDS) and the business response time, resulting in low deletion efficiency for objects stored in the detached directory. Summary of the Invention

[0003] This application provides a method and apparatus for deleting stored objects, a storage medium, and an electronic device, to at least solve the technical problem of low deletion efficiency of stored objects in free directories in related technologies.

[0004] According to one embodiment of this application, a method for deleting a storage object is provided, comprising: loading a set of version information corresponding to a detached directory in a metadata server, wherein the set of version information includes multiple version information corresponding to multiple storage objects, and the detached directory is used to cache multiple storage objects; traversing the multiple version information to obtain at least one version information that meets the object deletion conditions; and deleting at least one storage object corresponding to at least one version information in the detached directory.

[0005] According to another embodiment of this application, a storage object deletion device is provided, comprising: a loading unit for loading a set of version information corresponding to a detached directory in a metadata server, wherein the set of version information includes multiple version information corresponding to multiple storage objects, and the detached directory is used to cache multiple storage objects; a traversal unit for traversing the multiple version information and obtaining at least one version information that meets the object deletion conditions; and a deletion unit for deleting at least one storage object corresponding to at least one version information in the detached directory.

[0006] According to yet another embodiment of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0007] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0008] Through the embodiments provided in this application, by loading only the version information set of the detached directory, MDS can complete the startup process in a very short time and quickly restore the service status, avoiding startup delays caused by loading a large number of detached directory entries. This ensures a shorter system fault recovery time and maintains a fast response speed even under high system load, improving system availability and user experience. When traversing the version information set of the detached directory, the system can quickly obtain version information that meets the object deletion conditions without loading and processing all detached directory entries. In this way, the system can selectively process the storage objects that need to be deleted, freeing up valuable storage space, reducing memory consumption, avoiding excessive resource occupation, and improving resource utilization efficiency. This achieves the technical effect of improving the deletion efficiency of stored objects in the detached directory, solving the technical problem of low deletion efficiency of stored objects in the detached directory in related technologies. Attached Figure Description

[0009] Figure 1 This is a hardware structure block diagram of a method for deleting a storage object according to an embodiment of this application;

[0010] Figure 2 This is a flowchart of a method for deleting a storage object according to an embodiment of this application;

[0011] Figure 3 This is a flowchart illustrating an implementation method for deleting a storage object according to an embodiment of this application;

[0012] Figure 4 This is a structural block diagram of a storage object deletion device according to an embodiment of this application. Detailed Implementation

[0013] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of deleting a stored object according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0016] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the mapping relationship determination method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0017] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0018] As an alternative, the method for storing object deletion, such as Figure 2 As shown, the specific steps include:

[0019] S202, Load the version information set corresponding to the detached directory in the metadata server. The version information set includes multiple version information corresponding to multiple storage objects. The detached directory is used to cache multiple storage objects.

[0020] S204: Traverse multiple version information and obtain at least one version information that meets the object deletion condition;

[0021] S206, In the detached directory, delete at least one storage object corresponding to at least one version information.

[0022] Optionally, in this embodiment, the storage object refers to any file or directory in the distributed file system that can be accessed and managed by the user, including but not limited to text files, databases, images, videos, or any type of data collection.

[0023] Optionally, in this embodiment, the metadata server (MDS) is a component in the distributed file system responsible for managing file and directory metadata, providing maintenance and query services for file attributes, directory structure, permission information, etc.

[0024] Optionally, in this embodiment, in order to reduce the immediate impact of deletion operations on system performance in the distributed file system, files and directories are not immediately physically deleted after being logically deleted. Instead, they are first moved to a detached directory to await background processing. The detached directory records information about all storage objects that have been marked for deletion but have not yet been physically deleted.

[0025] Optionally, in this embodiment, each storage object has corresponding version information in the distributed file system to track changes in the state of the storage object, including but not limited to timestamps and version numbers of creation, modification and deletion operations, so as to ensure data consistency and security.

[0026] Optionally, in this embodiment, when the MDS starts up, or when it needs to process storage objects in the detached directory during business operation, the system first loads the version information set of the detached directory, instead of the full set of file or directory entries. The version information set contains key version information of all storage objects in the detached directory, such as the object's version number and the time it was marked for deletion, which is used to quickly determine the status of the storage objects without loading complete metadata, thereby significantly reducing loading time and system resource consumption.

[0027] The system iterates through the loaded version information set and determines which storage objects meet the criteria for physical deletion by checking version information (such as whether they are protected by snapshots or whether snapshots have expired). This step is based on a snapshot-aware management mechanism, ensuring that protected storage objects are not accidentally deleted, while also identifying and handling detached objects that no longer need to be stored in the system.

[0028] For storage objects determined to be safe for deletion during the traversal process, the system will physically delete them in the detached directory using batch processing and flow control mechanisms. By loading, processing, and using appropriate sleep intervals, the method of this invention ensures that the deletion operation does not significantly impact system performance. Even if the number of stored objects in the detached directory is very large, it can be processed smoothly, maintaining system stability and business continuity.

[0029] Understandably, loading a lightweight set of version information instead of the full set of detached directory data enables MDS to start up and provide services quickly, avoiding long startup delays. During business operations, the system can determine the deletion status of stored objects based on version information. For objects that meet the deletion criteria, a batch loading-processing-sleep flow control approach is adopted to ensure that the deletion operation has minimal impact on business operations while avoiding excessive resource consumption. In addition, snapshot-protected storage objects are specifically considered. Through a snapshot-aware mechanism, it can effectively distinguish which objects can be safely deleted and which need to be retained, thus achieving a balance between ensuring data integrity and system performance. This approach not only improves the system's startup and response speed but also optimizes resource usage. It is particularly suitable for large-scale storage clusters and high-concurrency access scenarios, significantly improving the stability and efficiency of the distributed file system while maintaining data security and compliance.

[0030] Through the embodiments provided in this application, by loading only the version information set of the detached directory, MDS can complete the startup process in a very short time and quickly restore the service status, avoiding startup delays caused by loading a large number of detached directory entries. This ensures a shorter system fault recovery time and maintains a fast response speed even under high system load, improving system availability and user experience. When traversing the version information set of the detached directory, the system can quickly obtain version information that meets the object deletion conditions without loading and processing all detached directory entries. In this way, the system can selectively process the storage objects that need to be deleted, freeing up valuable storage space, reducing memory consumption, avoiding excessive resource occupation, and improving resource utilization efficiency, thereby achieving the technical effect of improving the deletion efficiency of storage objects in the detached directory.

[0031] As an optional approach, load the version information set corresponding to the detached directory in the metadata server, including:

[0032] The N version information subsets corresponding to the detached directory are loaded sequentially. The loading process of the i-th version information subset is performed after the storage objects corresponding to the version information that meet the object deletion conditions in the (i-1)-th version information subset are deleted from the detached directory. N is a positive integer greater than 1, and i is a positive integer greater than 1 and not greater than N.

[0033] Optionally, in this embodiment, the version information subset is a subset of the version information set, which includes version information of a portion of the stored objects in the detached directory, and is used for loading and processing in batches to reduce the occupation of system resources and the impact on business operations.

[0034] Optionally, in this embodiment, the N version information subsets refer to dividing the entire version information set into N subsets for processing. N is a positive integer greater than 1, representing the total number of batches processed in batches, where i is a positive integer ranging from 1 to N, indicating that the currently processed subset is the i-th batch in the entire batch processing process.

[0035] Optionally, in this embodiment, the first subset of version information in the detached directory is loaded; thereafter, each time a subset of version information is loaded, it is done after the storage objects that meet the deletion conditions in the previous subset have been physically deleted.

[0036] For each subset of version information, the system iterates through the version information, finds objects that meet the deletion criteria, and physically deletes them. After the deletion operation is complete, the system briefly pauses (e.g., sleeps) to avoid resource consumption fluctuations caused by continuous operations before loading and processing the next subset of version information. This flow control mechanism ensures the smooth execution of deletion operations and reduces interference with business operations.

[0037] By decomposing the version information set into N subsets for processing, the system can dynamically adjust the value of N and the size of each subset based on the current system resource status and business load, thereby optimizing resource utilization. For example, when system resources are sufficient, the size of each subset can be increased to reduce processing batches and improve deletion efficiency; while under high load, the size of the subsets can be decreased to extend the sleep interval and ensure that business is not affected.

[0038] Understandably, by processing the version information set in batches, efficient management and deletion of storage objects to be deleted in the detached directory are achieved. This method first divides the version information set in the detached directory into N version information subsets according to a process flow, and loads and processes them sequentially starting from the first subset. When processing the i-th subset, the system first checks and deletes storage objects that meet the deletion conditions in the previous subset (i-1-th subset), ensuring that the next batch is processed only after the previous batch is completed. This avoids concurrency issues and resource contention.

[0039] Through the embodiments provided in this application, by dynamically loading and processing version information in batches, the system can adjust resource consumption according to actual operating conditions, improving the efficiency of deletion operations while ensuring high availability and high performance. When faced with millions or even hundreds of millions of detached directory entries, this method can significantly reduce deletion latency, avoid excessive resource consumption, and thus improve the overall operating efficiency and user experience of the distributed file system.

[0040] As an optional approach, load the N version information subsets corresponding to the detached directory sequentially, including:

[0041] Load the first version information subset corresponding to the detached directory;

[0042] If the storage object corresponding to the version information that meets the object deletion conditions in the first version sub-information set is deleted from the detached directory, it will be put into hibernation for a preset period of time.

[0043] After a preset hibernation period, the second version information subset corresponding to the detached directory in the metadata server is loaded. The second version information subset is the next version information subset after the first version information subset among N version information subsets.

[0044] Optionally, in this embodiment, when processing the set of version information in the detached directory in batches, the system first loads and processes a subset of version information. This subset contains key version information of some stored objects in the detached directory, which is used to initially screen objects that meet the deletion criteria.

[0045] Optionally, in this embodiment, after the system finishes processing a subset of version information, it will automatically enter a sleep period to avoid performance impact caused by continuous operations, so as to release system resources and allow business operations to take priority. The length of the preset time period is usually fixed, such as 2 seconds, and can be adjusted according to system performance and business needs.

[0046] Optionally, in this embodiment, after processing the first subset of version information and sleeping for a preset time period, the system will load and process the next set of version information, which is the next set in the sequence of N subsets of version information.

[0047] Understandably, by subdividing the version information set into N subsets and loading and processing only one subset at a time, efficient management of the detached directory is achieved. Loading the first subset of version information is the starting point of the entire process; the system quickly determines which objects can be safely deleted based on the version information. After completing the first batch of deletion operations, the system automatically enters a dormant state. During this period, business operations receive higher priority in system resource allocation, ensuring the normal operation of the business. After the dormant state ends, the system seamlessly transitions to loading and processing the second subset of version information. This process continues to cycle until all stored objects in all subsets have been evaluated and appropriately deleted.

[0048] The embodiments provided in this application effectively distribute the pressure of detached directory processing through batch processing and flow control mechanisms, reducing the instantaneous demand on system resources and avoiding delays or interruptions in business operations. Simultaneously, the preset time period of hibernation provides the system with an opportunity for self-recovery, ensuring continuous system stability and performance output during large-scale detached directory cleanup. This is an efficient solution for distributed file systems that need to handle the deletion of a large number of storage objects.

[0049] As an optional approach, load the version information set corresponding to the detached directory in the metadata server, including:

[0050] Obtain version number information and total number of entries for multiple storage objects. The version information for each storage object includes the version number information and the total number of entries for each storage object.

[0051] Load the version information for each stored object.

[0052] Optionally, in this embodiment, in the distributed file system, each storage object (file or directory) is assigned a version number when it is created, modified, or deleted. This version number is used to track the object's lifecycle and ensure data consistency and integrity. Higher version numbers typically indicate a newer state.

[0053] Optionally, in this embodiment, the total number of entries refers to the total number of information entries recorded by the storage object in the detached directory, including the information entries of the file itself and the metadata information entries related to it, such as permission information, link count, etc.

[0054] Optionally, in this embodiment, the version number information, the total number of entries information, and other possible key metadata (such as snapshot information, permission settings, etc.) are combined to form a set of version information of the stored object in the detached directory.

[0055] Optionally, in this embodiment, when the MDS starts or performs detached directory processing, the system first obtains the version number information and total number of entries for all stored objects in the detached directory, which is the basic data for the version information set. This process involves reading metadata records stored on the disk and extracting specific fields, such as version number and entry count.

[0056] After obtaining the version number and total number of entries, the system loads the complete version information for each storage object. This means that the system will process each record containing the version number, total number of entries, and other possible metadata information to determine which storage objects can be safely removed from the detached directory and which still need to be retained.

[0057] This section details how to load the version information set corresponding to the detached directory, focusing particularly on the acquisition and loading process of version number and total number of entries. In the context of a distributed file system, the detached directory serves as a temporary storage location for metadata of deleted objects, and its size and status directly impact system performance and service quality. Therefore, when loading the version information set, the system prioritizes acquiring the version number and total number of entries for the stored objects. These two types of information are crucial for determining whether a stored object can be safely physically deleted.

[0058] The embodiments provided in this application not only accelerate the startup speed of the MDS and ensure rapid service recovery, but also optimize resource utilization, guarantee the stability and responsiveness of the system under high load, and significantly improve the overall performance and user experience of the distributed file system. In practical applications, this fine-grained version information loading strategy can significantly reduce performance problems caused by the backlog of detached directories, and its effect is particularly evident in scenarios with high concurrency access and large data volume storage.

[0059] As an optional approach, iterate through multiple version information to obtain at least one version that meets the object deletion criteria, including:

[0060] Iterate through multiple version information, including multiple version number information and multiple total number of entries information, to obtain reference version number information that meets the version number update condition and reference total number of entries that meets the total number of entries update condition.

[0061] The version information corresponding to the reference version number information and the version information corresponding to the total number of reference entries information are determined to be at least one version information.

[0062] Optionally, in this embodiment, each storage object undergoes multiple state changes during its lifecycle, and these changes are recorded as a version number. In the detached directory, the version number information of each of the multiple storage objects represents the state of its last state change, used to determine whether the object currently meets the deletion conditions.

[0063] Optionally, in this embodiment, the total number of entries information indicates the number of metadata entries for each storage object recorded in the detached directory, reflecting the size of the metadata space occupied by the object in the system.

[0064] Optionally, in this embodiment, the version number update condition is a set rule or standard used to determine whether the version number of a stored object has been updated to a state that allows it to be deleted from the detached directory. For example, if an object is no longer protected by snapshots, its version number status may be marked as deletable.

[0065] Optionally, in this embodiment, the condition for updating the total number of entries is similar to the condition for updating the version number, but the focus is on the total number of entries for the stored object. When the state of an object changes to the point where metadata records no longer need to be retained, its total number of entries may meet a preset deletion condition.

[0066] Optionally, in this embodiment, the system traverses the version number and total number of entries information of each stored object in the detached directory, and checks whether this information meets the preset update conditions. This process involves carefully reading the metadata record of each object and extracting the version number and total number of entries fields.

[0067] During the traversal, the system identifies and marks storage objects whose version number or total entry count information has been updated to a state that allows them to be deleted from the detached directory. The version number and total entry count information of these marked objects will be used as reference information.

[0068] Ultimately, storage objects whose version number and total number of entries are both marked as meeting the deletion criteria will have their version information determined to be at least one version. This means that these objects can be physically deleted and removed from the detached directory.

[0069] The embodiments provided in this application enable a more precise selection of which storage objects can be safely removed from the detached directory and which should be retained. This method improves the accuracy of deletion operations, avoids unnecessary deletions, and reduces the impact on current business operations because deletion only targets storage objects that are truly no longer needed. For large-scale distributed file systems, this method can significantly improve resource utilization, optimize system performance, and ensure data security and consistency.

[0070] In addition, this approach provides flexibility for handling detached directories, allowing the system to dynamically adjust version number update conditions and total number of entries update conditions based on current resource status and business needs, ensuring that deletion operations are both efficient and secure.

[0071] As an optional approach, before loading the version information set corresponding to the detached directory in the metadata server, the method also includes:

[0072] The storage objects that meet the first expected condition are determined from all storage objects included in the free directory, and the storage objects that meet the second expected condition are determined from all storage objects. The first expected condition is used to indicate that the storage object has no snapshot relationship in the free directory, and the second expected condition is used to indicate that the storage object has a snapshot relationship in the free directory and the establishment time corresponding to the snapshot relationship is greater than a preset time threshold.

[0073] The storage objects that meet the first expected condition and the storage objects that meet the second expected condition are identified as multiple storage objects; the multiple version information corresponding to the multiple storage objects is obtained.

[0074] Optionally, in this embodiment, the first expected condition refers to the fact that in the detached directory, the current state of the stored object is not associated with any snapshot protection, that is, the object is not locked or protected by any snapshot, which means that the object can be physically deleted immediately without affecting the consistency and integrity of the snapshot data.

[0075] Optionally, in this embodiment, the second expected condition is the opposite of the first expected condition, which means that the stored object has a snapshot relationship in the detached directory, but the snapshot was created after a preset duration threshold has been exceeded, the snapshot relationship is no longer valid or the protection period has expired, and the object can be physically deleted at this time.

[0076] Optionally, in this embodiment, the preset duration threshold is a set time value used to determine whether the snapshot relationship has expired. If the duration of the snapshot exceeds this threshold, the system considers the snapshot relationship to be invalid, and the stored object can be physically deleted.

[0077] Optionally, in this embodiment, before loading the version information set, the system first filters out objects that meet the first and second expected conditions from all stored objects in the detached directory. For objects that meet the first expected condition, it means that they are not protected by any snapshot and can be physically deleted immediately; while for objects that meet the second expected condition, although there is a snapshot relationship, since the snapshot creation time has exceeded the preset duration threshold, these objects can also be safely removed from the detached directory.

[0078] The system identifies the storage objects that meet the first and second expected conditions as the multiple storage objects whose version information sets will be loaded next. This means that only these storage objects that are confirmed to be deletable will have their version information loaded and processed.

[0079] Finally, the system loads the version information set from these identified multiple storage objects, including version number, total number of entries, etc., to prepare for subsequent deletion operations. In this way, the system can effectively distinguish which objects can be safely deleted and which objects need to be retained, thereby avoiding accidental deletion of critical data and improving the efficiency of deletion operations.

[0080] The embodiments provided in this application not only optimize the use of storage resources but also improve the intelligence and flexibility of the system's snapshot relationship management. In actual operation, this strategy can effectively avoid performance bottlenecks caused by improper handling of detached directories, while ensuring the normal operation of the snapshot protection mechanism, which is of great value in improving the stability and efficiency of the distributed file system. Especially in large-scale data management scenarios, this method can significantly reduce the resource waste caused by improper snapshot protection and provide the system with a more efficient and secure means of managing the deletion of storage objects.

[0081] As an optional approach, load the version information set corresponding to the detached directory in the metadata server, including the following:

[0082] In response to the first deletion request, load the version information set corresponding to the detached directory, wherein the first deletion request is triggered by executing a preset deletion operation;

[0083] In response to the second deletion request, load the version information set corresponding to the detached directory, where the second deletion request is triggered automatically periodically.

[0084] Optionally, in this embodiment, the first deletion request is a deletion request manually triggered by a user or system administrator. It triggers the loading of the set of object version information stored in the detached directory by performing a preset deletion operation. This type of request is typically used during system maintenance, exception handling, or emergencies to ensure immediate response and processing.

[0085] Optionally, in this embodiment, the second deletion request differs from the first deletion request. The second deletion request is automatically triggered periodically by the system without manual intervention. Through a preset periodic mechanism, the system periodically checks the storage objects in the roaming directory, loads and processes version information sets to clean up unnecessary storage objects and release system resources. This mechanism aims to achieve automated management, reduce the need for manual operation, and improve the system's self-maintenance capabilities.

[0086] Optionally, in this embodiment, the preset deletion operation refers to a deletion operation performed by the system or user under specific conditions (such as after system anomaly repair or when business requirements change), which is used to trigger the first deletion request, thereby starting the loading and processing of the set of object version information stored in the detached directory.

[0087] Optionally, in this embodiment, periodic automatic triggering refers to the system automatically executing the second deletion request at preset time intervals, periodically checking the status of the detached directory, loading the version information set, and cleaning up the stored objects. This period can be a fixed time point daily, weekly, or monthly, or it can be a dynamic period that is automatically adjusted according to the system load.

[0088] Optionally, in this embodiment, when the system receives the first deletion request, it responds immediately and begins loading the version information set corresponding to the detached directory. This process may be manually triggered by the user, or it may be a preset deletion operation automatically executed when the system detects an abnormal situation requiring urgent handling. Loading the version information set allows the system to check the status of the stored objects and determine whether they can be safely deleted.

[0089] Unlike the first deletion request, the second deletion request is automatically triggered by the system according to a preset periodicity. At each periodic point, the system automatically checks the detached directory, loads the version information set, and checks whether the stored object meets the deletion conditions. Compared with the first deletion request, the second deletion request focuses more on long-term, routine storage resource management, requiring no manual intervention and improving the efficiency of system self-maintenance.

[0090] The embodiments provided in this application, through the response mechanisms for these two deletion requests, achieve comprehensive coverage of detached directory management, enabling both urgent resource release needs and long-term resource optimization and management. In large-scale distributed file systems, this flexible version information set loading strategy provides the system with an efficient and secure means of managing storage object deletion, greatly improving system stability and response speed, while ensuring the normal operation of the snapshot protection mechanism, thus significantly enhancing the overall system performance.

[0091] As an alternative, the above-mentioned method for deleting storage objects can be applied to scenarios where large detached directories in a distributed file system are overloaded. In this scenario, file and directory deletion operations adopt a two-stage model of "logical deletion + asynchronous physical deletion":

[0092] Logical deletion phase: After the client initiates a deletion request, the metadata server (MDS) immediately updates the metadata, marking the target file / directory as "deleted" to quickly respond to the client's operation. The deleted file / directory is first temporarily stored in an "orphaned directory".

[0093] Physical deletion phase: The actual deletion of stored data is performed asynchronously by a background task, which completely removes the data from the underlying storage (such as RADOS object storage).

[0094] Optionally, in this embodiment, only version information is loaded during the metadata service startup process, and the full list of entries to be deleted is not loaded. The detached directory is loaded only once after the metadata service starts, and will not be loaded again during subsequent business operations. If the detached directory has a large backlog after the metadata service starts (e.g., nearly 100 million entries in a project site are not deleted), loading 100 million entries would take hours. The distributed file system would block metadata operations such as deletion if the detached directory is not fully loaded, which is unacceptable. Loading the detached directory after the metadata service starts is actually to determine if there are any entries in the detached directory that meet the deletion criteria and has no other function. Therefore, only necessary version information is loaded after the metadata service starts, and the actual detached directory entries are not loaded. This will not block business operations, and business can be quickly restored after a switch.

[0095] Optionally, in this embodiment, detached directory scanning is triggered manually and periodically. Since the metadata service does not load detached directories after startup, and a large number of detached directories cannot all be stored in the cache (otherwise, it would consume hundreds of gigabytes of memory), manual triggering of detached directory loading is required after service recovery. To avoid the need for manual intervention, timed triggering of detached directory loading is also required; that is, by configuring a timed strategy, the detached directory is loaded periodically, and items that truly meet the deletion criteria are deleted.

[0096] Optionally, in this embodiment, a batch loading-batch deletion-sleep serial flow control mode is adopted. Since a large number of entries may accumulate in the detached directory, loading them all at once would be insufficient; therefore, batch loading is necessary. However, since batch loading occurs during business operations, system resources such as CPU, memory, and threads are limited and must be allocated to normal business requests. Furthermore, the autonomous loading, scanning, and deletion of entries in the detached directory are background tasks and cannot excessively occupy system resources, otherwise it would affect user services, such as increasing latency or consuming excessive memory. Therefore, batch loading is necessary. The actual design involves loading 20,000 entries at a time, processing 20,000 entries, sleeping for 2 seconds, then loading the next 20,000 entries, processing another 20,000 entries, and sleeping for another 2 seconds, repeating this cycle. In practice, this method has almost no performance impact on the business.

[0097] To further illustrate, the specific implementation process is as follows: Figure 3 As shown, this includes: After MDS starts, it only loads information such as the version of the detached directory, without loading all detached directory entries, to accelerate business recovery. During business operation, detached directory scanning is triggered manually or periodically. A batch of detached directory entries is loaded, for example, 20,000. A batch of detached directory entries is processed. The process pauses for 2 seconds. It checks whether the detached directory entries are fully loaded. If fully loaded, the process ends. If not fully loaded, it checks whether to interrupt the current loading; if yes, the process ends; otherwise, it returns and reloads a new batch of detached directory entries.

[0098] File and directory deletions first enter the detached directory. Once in the detached directory, user operations are quickly responded to, and the actual deletion is performed in the background. After the metadata service starts, the full set of detached directory entries is no longer loaded; only necessary version and other consistency information is loaded to ensure the detached directory's availability. After the metadata service starts, the distributed file system can provide normal business services. At this point, detached directory loading and scanning can be manually triggered at a later date using the provided tools. If manual intervention is not desired, a timed strategy can be configured, such as specifying the time of day and interval, to automatically trigger the detached directory scan upon expiration. Manual or timed detached directory scanning employs a batch loading-batch processing-2-second sleep method. Loading and processing are performed serially in batches, with a 2-second sleep used for flow control after each batch is completed to mitigate the performance impact on business operations until all loading and processing are finished. During the detached directory loading process, corresponding tools are provided, allowing for system maintenance interruptions at any time.

[0099] In this embodiment, significant benefits are brought about in the management of detached directories in the distributed file system, specifically in the following four aspects:

[0100] 1. A qualitative leap in metadata service startup performance

[0101] In traditional solutions, the Metadata Service (MDS) requires a full load of all metadata from the detached directory upon startup. When the directory size reaches hundreds of millions, this loading process can take several hours. This invention employs a minimal loading strategy that loads only version information, making the startup process almost instantaneous. This improvement significantly reduces system failure recovery time, which is particularly important in large-scale cluster environments. For example, in a storage cluster with thousands of nodes, rapid MDS recovery means clients can reconnect faster, significantly reducing business interruption time.

[0102] 2. Optimization and balancing of system resource usage

[0103] The batch loading mechanism combined with dynamic hibernation strategy has almost no impact on peak memory usage. This improvement in resource utilization efficiency brings three direct benefits: First, it reduces the system's hardware requirements, saving hardware costs; second, it reduces the risk of OutOfMemoryError (OOM) due to insufficient memory; and finally, it enables the system to handle more concurrent requests simultaneously, improving overall throughput. Especially in cloud environments, this resource optimization can translate into significant cost savings.

[0104] 2. Ensuring the integrity of the snapshot function

[0105] The snapshot-aware management mechanism ensures that snapshot-protected directories are not accidentally deleted, while also effectively cleaning up directories that do not depend on snapshots. This balance is particularly crucial for scenarios requiring long-term data protection (such as the financial and healthcare industries), satisfying compliance requirements for long-term data retention while preventing the unlimited expansion of storage space.

[0106] 4. Significant improvement in business continuity

[0107] The dynamic flow control mechanism breaks down high-load operations into smaller batches and inserts adjustable sleep intervals between batches, ensuring that deletion operations have virtually no impact on normal business operations. This smooth performance is crucial for online businesses that require 24 / 7 uninterrupted service, preventing a decline in user experience caused by background maintenance operations.

[0108] It's important to note that traditional methods require loading all metadata from the detached directory during MDS startup, leading to significant startup delays. This embodiment creatively proposes a "version information priority" loading strategy, decomposing the startup process into two phases: the first phase loads only lightweight header information containing key metadata such as version number and total number of entries; the second phase asynchronously loads the complete directory entries in the background. This two-phase design offers three advantages: first, it enables MDS to reach a basically usable state within seconds; second, it allows the system to continue processing client requests while loading in the background; and finally, it lays the foundation for subsequent optimizations (such as dynamic batch processing). This design approach is universally applicable in the distributed system domain and can be extended to other types of metadata management scenarios.

[0109] To address the performance issues caused by one-time loading, this embodiment proposes an intelligent batch processing framework. This framework comprises three innovative components: a dynamic batch size adjuster, a processing-sleep scheduler, and a resource monitoring module. The dynamic batch size adjuster automatically calculates the optimal batch size (e.g., 20,000-50,000 records / batch) based on the current system load; the processing-sleep scheduler ensures a controllable sleep interval (default 2 seconds) after each batch processing; and the resource monitoring module collects CPU, memory, and I / O metrics in real time for decision-making. This adaptive mechanism improves efficiency by more than 30% compared to fixed batch size methods, and performs particularly well in production environments with fluctuating loads.

[0110] To address the backlog problem of snapshot-protected directories, this embodiment breaks away from the traditional "all or nothing" deletion strategy and proposes a refined snapshot dependency management scheme. This scheme includes three key technical points: First, it uses an efficient directory-snapshot mapping table (based on a hash index) to achieve dependency queries with O(1) time complexity; second, it introduces the concept of "protection levels" to implement differentiated strategies for different types of snapshot-related directories; and finally, it provides a manual intervention interface to handle special cases. This art of balance has high practical value in the field of data management.

[0111] To address the need for cleaning up large-scale detached directories, this embodiment designs a complete scanning framework comprising four components: a task scheduler, a resource controller, a progress tracker, and an exception handler. The task scheduler supports two triggering modes (manual / scheduled) and flexible timing strategy configuration.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0113] This embodiment also provides a device for deleting stored objects, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0114] Figure 4 This is a structural block diagram of a storage object deletion device according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0115] Loading unit 402 is used to load the version information set corresponding to the detached directory in the metadata server. The version information set includes multiple version information corresponding to multiple storage objects. The detached directory is used to cache multiple storage objects.

[0116] Traversal unit 404 is used to traverse multiple version information and obtain at least one version information that meets the object deletion conditions; deletion unit 406 is used to delete at least one storage object corresponding to at least one version information in the detached directory.

[0117] As an optional solution, loading unit 402 includes:

[0118] The first loading module is used to load the N version information subsets corresponding to the detached directory in sequence. The loading process of the i-th version information subset is carried out after the storage objects corresponding to the version information that meet the object deletion conditions in the (i-1)-th version information subset are deleted from the detached directory. N is a positive integer greater than 1, and i is a positive integer greater than 1 and not greater than N.

[0119] As an optional approach, the first loading module includes:

[0120] The first loading submodule is used to load the first version information subset corresponding to the detached directory;

[0121] The hibernation submodule is used to hibernate the storage objects corresponding to version information that meet the object deletion conditions in the first version sub-information set for a preset time period when they are deleted from the detached directory.

[0122] The second loading submodule is used to load the second version information subset corresponding to the detached directory in the metadata server after a preset hibernation period. The second version information subset is the next version information subset after the first version information subset among N version information subsets.

[0123] As an optional solution, loading unit 402 includes:

[0124] The first acquisition module is used to acquire version number information of multiple storage objects and total number of entries of multiple storage objects. The version information of each storage object includes version number information of each storage object and total number of entries of each storage object.

[0125] The second loading module is used to load the version information of each stored object.

[0126] As an optional solution, deletion unit 406 includes:

[0127] The traversal module is used to traverse multiple version information, including multiple version number information and multiple total number of entries information, and obtain reference version number information that meets the version number update condition and reference total number of entries that meets the total number of entries update condition.

[0128] The first determining module is used to determine the version information corresponding to the reference version number information and the version information corresponding to the total number of reference entries as at least one version information.

[0129] As an optional solution, the device also includes:

[0130] The second determining module is used to determine, before loading the version information set corresponding to the detached directory in the metadata server, the storage objects that meet the first expected condition from all the storage objects included in the detached directory, and the storage objects that meet the second expected condition from all the storage objects. The first expected condition is used to indicate that the storage object has no snapshot relationship in the detached directory, and the second expected condition is used to indicate that the storage object has a snapshot relationship in the detached directory and the establishment time corresponding to the snapshot relationship is greater than a preset time threshold.

[0131] The third determining module is used to determine the storage objects that meet the first expected condition and the storage objects that meet the second expected condition as multiple storage objects before loading the version information set corresponding to the detached directory in the metadata server.

[0132] The second acquisition module is used to acquire multiple version information corresponding to multiple storage objects before loading the version information set corresponding to the free directory in the metadata server.

[0133] As an optional solution, loading unit 402 includes one of the following:

[0134] The third loading module is used to load the version information set corresponding to the detached directory in response to the first deletion request, wherein the first deletion request is triggered by executing a preset deletion operation;

[0135] The fourth loading module is used to load the version information set corresponding to the detached directory in response to the second deletion request, wherein the second deletion request is triggered automatically periodically.

[0136] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0138] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0139] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0140] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0141] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0142] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0143] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing the computer program product, wherein the computer program, when executed by a processor, implements the steps of the methods in various embodiments of this application.

[0144] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0145] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0146] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method of deleting a storage object, characterized by, The method comprises: loading a version information set corresponding to a free directory in a metadata server, wherein the version information set comprises a plurality of version information corresponding to a plurality of storage objects, and the free directory is used to cache the plurality of storage objects; traversing the plurality of version information to obtain at least one version information meeting an object deletion condition; deleting at least one storage object corresponding to the at least one version information in the free directory.

2. The method of claim 1, wherein, The loading of the version information set corresponding to the free directory in the metadata server comprises: loading N version information sub-sets corresponding to the free directory in sequence, wherein the loading process of an i-th version information sub-set is performed after deleting a storage object corresponding to a version information meeting the object deletion condition in an (i-1)-th version information sub-set from the free directory, N is a positive integer greater than 1, and i is a positive integer greater than 1 and not greater than N.

3. The method of claim 2, wherein, The loading of the N version information sub-sets corresponding to the free directory in sequence comprises: loading a first version information sub-set corresponding to the free directory; sleeping for a preset time period in a case where a storage object corresponding to a version information meeting the object deletion condition in the first version information sub-set is deleted from the free directory; loading a second version information sub-set corresponding to the free directory in the metadata server after sleeping for the preset time period, wherein the second version information sub-set is a next version information sub-set after the first version information sub-set in the N version information sub-sets.

4. The method of claim 1, wherein, The loading of the version information set corresponding to the free directory in the metadata server comprises: obtaining version number information of the plurality of storage objects and total entry number information of the plurality of storage objects, wherein the version information of each storage object in the plurality of storage objects comprises version number information of the each storage object and total entry number information of the each storage object; loading the version information of the each storage object.

5. The method of claim 4, wherein, The traversing of the plurality of version information to obtain at least one version information meeting an object deletion condition comprises: traversing a plurality of version number information and a plurality of total entry number information included in the plurality of version information to obtain reference version number information meeting a version number update condition and reference total entry number information meeting a total entry number update condition; determining, as the at least one version information, version information corresponding to the reference version number information and version information corresponding to the reference total entry number information.

6. The method according to any one of claims 1 to 5, characterized in that, Before the loading of the version information set corresponding to the free directory in the metadata server, the method further comprises: determining, from all storage objects included in the free directory, a storage object meeting a first expected condition and a storage object meeting a second expected condition, wherein the first expected condition is used to indicate that a storage object has no snapshot relationship in the free directory, and the second expected condition is used to indicate that a storage object has the snapshot relationship in the free directory and an establishment time length corresponding to the snapshot relationship is greater than a preset time length threshold. determine the storage object meeting the first expected condition and the storage object meeting the second expected condition as the plurality of storage objects; obtain the plurality of version information corresponding to the plurality of storage objects.

7. The method according to any one of claims 1 to 5, characterized in that, The version information set corresponding to the free directory in the loading metadata server includes one of the following: In response to a first deletion request, load the version information set corresponding to the free directory, wherein the first deletion request is triggered by performing a preset deletion operation; In response to a second deletion request, load the version information set corresponding to the free directory, wherein the second deletion request is periodically automatically triggered.

8. A deletion apparatus of a storage object, characterized by, It includes: a loading unit for loading a version information set corresponding to a free directory in a metadata server, wherein the version information set includes a plurality of version information corresponding to a plurality of storage objects, and the free directory is used to cache the plurality of storage objects; a traversal unit for traversing the plurality of version information and obtaining at least one version information meeting an object deletion condition; a deletion unit for deleting at least one storage object corresponding to at least one version information in the free directory.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

Images