File deletion method and electronic device

Abstract

This application discloses a file deletion method and electronic device, relating to the field of distributed file system technology. The method categorizes files to be deleted into large and small files based on a preset first volume threshold. Different deletion processes are applied to files of different sizes, allowing for simultaneous deletion of large and small files, thus improving file deletion efficiency. Specifically, for small files, data block information of at least two small files is recorded in a storage object, and the physical storage space corresponding to the data block information of at least two small files in the storage object is released in batches. This avoids frequent interactions with the storage device, improving the efficiency of physical storage space release, and further enhancing the efficiency of small file deletion.

Description

Technical Field

[0001] This application relates to the field of distributed file system technology, and in particular to a file deletion method and electronic device. Background Technology

[0002] In distributed file systems, deleting small files is one of the core tasks in system operation and resource management. With the popularization of cloud computing, big data analytics, and artificial intelligence training, distributed file systems need to handle the storage and management of massive amounts of small files.

[0003] In related technologies, when deleting files, the files to be deleted are first moved to a temporary storage directory and then added to a cleanup queue, where tasks are processed serially by a cleanup thread. However, in the above methods, especially in scenarios where a large number of small files need to be deleted, frequent interaction with the storage device is required when releasing the physical storage space of the files to be deleted. Furthermore, the single-threaded processing mechanism limits the deletion throughput, resulting in low file deletion efficiency. Summary of the Invention

[0004] This application provides a file deletion method and an electronic device for improving file deletion efficiency.

[0005] This application provides a file deletion method, including:

[0006] Get the files to be deleted, and divide them into large files and small files according to a preset first volume threshold. The large files have a file size greater than the first volume threshold, and the small files have a file size less than the first volume threshold.

[0007] For large files, add them to the cleanup queue, and release the metadata information and physical storage space of the large files in the cleanup queue serially through the cleanup thread;

[0008] For small files, the data block information of at least two small files is recorded in a storage object, and the physical storage space of at least two small files corresponding to the data block information in the storage object is released in batches. The data block information is used to indicate the identification information of the physical storage space corresponding to the small file, and the storage object is used to record the data block information of at least two small files.

[0009] This application also provides a file deletion device, comprising:

[0010] The acquisition module is used to acquire files to be deleted and divide the files to be deleted into large files and small files according to a preset first volume threshold. Among them, the file size of large files is greater than the first volume threshold, and the file size of small files is less than the first volume threshold.

[0011] The first processing module is used to add large files to the cleanup queue and release the metadata information and physical storage space of large files in the cleanup queue in a serial manner through the cleanup thread.

[0012] The second processing module is used to record the data block information of at least two small files into a storage object, and to release the physical storage space of at least two small files corresponding to the data block information in the storage object in batches. The data block information is used to indicate the identification information of the physical storage space corresponding to the small file, and the storage object is used to record the data block information of at least two small files.

[0013] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described file deletion methods.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described file deletion methods.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described file deletion methods.

[0016] This application categorizes files to be deleted into large and small files based on a preset first volume threshold. By applying different deletion processes to files of different sizes, the efficiency of file deletion can be improved. Specifically, for small files, the data block information of at least two small files is recorded in a storage object, and the physical storage space of at least two small files corresponding to the data block information in the storage object is released in batches. This avoids frequent interactions with the storage device, improves the efficiency of physical storage space release, and further improves the efficiency of deleting small files. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of a distributed file system architecture provided in an embodiment of this application;

[0019] Figure 2 Flowchart of the file deletion method provided in this application embodiment Figure 1 ;

[0020] Figure 3Flowchart of the file deletion method provided in this application embodiment Figure 2 ;

[0021] Figure 4 for Figure 3 A flowchart illustrating the specific implementation of step S207 in the illustrated embodiment;

[0022] Figure 5 This is a schematic diagram of the structure of the file deletion device provided in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0025] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0026] refer to Figure 1 , Figure 1 This is a schematic diagram of a distributed file system architecture provided in an embodiment of this application. Taking CephFS distributed file system as an example, it includes: client 10, metadata server (MDS) 11 and object storage device (OSD) 12. Client 10 communicates with MDS 11 and OSD 12, and MDS 11 communicates with OSD 12.

[0027] Client 10 is a terminal device used to receive file operation requests; for example, a user can initiate a file deletion command through client 10. MDS 11 is the core service or daemon in the distributed file system used to manage file system metadata. The metadata managed by MDS 11 includes file information, directory structure, data location mapping, etc. OSD 12 is the physical or logical storage unit responsible for actually storing data objects in the distributed storage cluster; each OSD typically corresponds to one physical disk or storage device.

[0028] In related technologies, when deleting files, the client 10 first receives the file deletion command initiated by the user and sends a deletion request to the MDS 11. The deletion request includes the path information of the small file to be deleted. After receiving the deletion request, the MDS 11 deletes the small file from the metadata of its parent directory, moves the inode (index node, i.e., metadata information) of the small file to a dedicated stray directory, adds the small file deletion task to the purge queue in sequence, and returns a successful deletion response to the client 10. After the purge thread of the MDS 11 sequentially retrieves the files to be deleted from the purge queue, it obtains the distribution information of the file data blocks in the OSD cluster, and sends space release requests to each OSD 12 that stores the file data. The space release request is such as "OP_OSD_UNLINK". After receiving the space release request, the OSD 12 marks the corresponding physical storage space as reusable. After all data blocks are released, the corresponding metadata information in the stray directory is deleted.

[0029] However, the above methods have the following problems: (1) Serial processing limitation: The purge queue is processed by a single thread, which causes the deletion speed to be limited by the thread throughput; (2) Multi-level delay superposition: The conversion of files from the stray directory to the purge queue, queue waiting, OSD response and other stages all generate delays, especially for small files, the proportion of metadata operations is high, and the delay is further amplified; (3) Low efficiency of small files: The data space reclamation of small files requires frequent interaction with OSD, and single-thread processing cannot be parallelized, resulting in low deletion throughput; (4) Resource competition problem: A large number of small files accumulated in the stray directory are not cleaned up in time, occupying metadata server resources and affecting system performance.

[0030] Based on this, this application provides a file deletion method. The files to be deleted are divided into large files and small files according to a preset first volume threshold. Different deletion processes are applied to files of different sizes. Large file deletion and small file deletion can be performed simultaneously, which can improve the efficiency of file deletion. Specifically, for small files, the data block information of at least two small files is recorded in a storage object, and the physical storage space of at least two small files corresponding to the data block information in the storage object is released in batches. This can avoid frequent interaction with the storage device, improve the efficiency of physical storage space release, and further improve the efficiency of small file deletion.

[0031] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Figure 2 Flowchart of the file deletion method provided in this application embodiment Figure 1 The execution entity of this method is the metadata server. For example... Figure 2 As shown, an embodiment of this application provides a file deletion method, which is described in detail below:

[0033] S101. Obtain the files to be deleted and divide them into large files and small files according to the preset first volume threshold.

[0034] For example, after receiving a file deletion request submitted by a user, the metadata server can determine and obtain the file to be deleted as indicated in the file deletion request, and classify the file to be deleted into large files and small files according to a preset first volume threshold. This first volume threshold can be pre-set or dynamically calculated in real time, and is used to classify the files to be deleted, for example, into large files and small files. Large files are files with a volume greater than or equal to the first volume threshold, and small files are files with a volume less than the first volume threshold. In this embodiment, different deletion processes are performed for large files and small files. Step S102 is the deletion process for large files, and step S103 is the deletion process for small files. For example, the first volume threshold is 20M, meaning large files are files with a volume greater than or equal to 20M, and small files are files with a volume less than 20M.

[0035] Optionally, after obtaining the file to be deleted, the method may also include: moving the file to be deleted to the persistent directory.

[0036] The stray directory is used to temporarily store files that need to be deleted. The file to be deleted is the file specified in the file deletion request sent by the user through the client. After receiving the file to be deleted, MDS first checks if there are any unreleased resources for that file, such as the client still holding capabilities or the file still being opened. If the file has no references, MDS marks it as safe to delete. Then, it checks if the file to be deleted is greater than or equal to a first volume threshold. If it is, the file is classified as a large file; if it is smaller, it is classified as a small file. For large files, step S102 releases metadata information and physical storage space; for small files, step S103 releases physical storage space and metadata information.

[0037] Optionally, the size of the first volume threshold can be dynamically adjusted according to the system load.

[0038] The first size threshold for distinguishing between small and large files is dynamically adjusted based on the current system load status (such as MDS resource utilization and OSD load balancing status). For example, in high-load scenarios, the first size threshold is adjusted from a fixed value (e.g., 20M) to a lower value (e.g., 10M) to expand the batch processing range for small files; in low-load scenarios, the first size threshold is increased to reduce the frequency of metadata operations. The threshold can be dynamically adjusted by real-time monitoring of system resource status combined with load prediction algorithms (such as sliding window average calculation).

[0039] Through a dynamic threshold mechanism, the system can adapt to resource demands under different business scenarios. For example, in scenarios involving intensive small file deletion, more small files can be processed first, reducing the backlog in the purge queue; while when the system is idle, the batch processing range of small files is reduced to avoid excessive consumption of storage resources. This strategy significantly improves the utilization rate of system resources while reducing performance fluctuations caused by fixed thresholds.

[0040] S102. For large files, add them to the cleanup queue and release the metadata information and physical storage space of the large files in the cleanup queue serially through the cleanup thread.

[0041] For large files, they are added to a cleanup queue, and the cleanup thread releases the metadata and physical storage space of the large files in the cleanup queue. The processing is as follows: Figure 1 The methods for deleting files shown in the related technologies are the same and will not be repeated here. The purge thread is the same as the cleanup thread, and the purge queue is the same as the cleanup queue.

[0042] S103. For small files, record the data block information of at least two small files into a storage object, and release the physical storage space of at least two small files corresponding to the data block information in the storage object in batches.

[0043] Among them, the data block information is used to indicate the identification information of the physical storage space corresponding to the small file, and the storage object is used to record the data block information of at least two small files.

[0044] Data block information records the index information of small files' distribution locations within storage nodes. This information includes, for example, the OSD ID and offset. A storage object is a container used to store the data block information of small files; it can be a file or object with a specific naming convention. Multiple data block information entries can be recorded within a storage object.

[0045] In one possible implementation, the data block information of at least two small files can be recorded by pre-setting the capacity of the storage object. For example, when the capacity of the storage object is 100M, it means that a storage object can store the data block information of up to 5 small files with a size of 20M. When the storage capacity of the current storage object has not reached the maximum capacity, the data block information of the small files can be recorded in the current storage object. If the storage capacity of the current storage object has reached the maximum capacity, a new storage object is created and the data block information of the small files is recorded in the new storage object.

[0046] In one possible implementation, the data block information recorded in a storage object can be determined by time intervals. For example, a storage object is created every hour, and each storage object records the data block information of small files within that time period.

[0047] Optionally, stored objects can be named using a specific naming format to facilitate filtering by keywords. An example naming format is: release_log_timestamp_random number.

[0048] When releasing physical storage space, the MDS can control a background thread to perform the release operation. Specifically, based on the data block information, the OSD storing the small file is determined, a release request is sent to the OSD, causing the OSD to release the corresponding physical storage space and return the released information to the MDS, thus completing the release of the physical storage space.

[0049] Optionally, before releasing the physical storage space of the small file corresponding to the data block information in the storage object, the method may also include deleting the metadata information of the small file.

[0050] Metadata information describes the attributes and storage location of small files. This information may include filename, size, directory structure, and data block distribution. Physical storage space refers to the location of the small file within the OSD.

[0051] After recording the data block information of the small file into the storage object, the metadata information of the target object is immediately deleted, thereby completely deleting it from the stray directory. Only the physical storage space of the small file remains to be released. Then, the physical storage space of the small file corresponding to the data block information in the storage object is released through the background processing line.

[0052] MDS can remove the metadata information of small files from the metadata of their parent directory based on the metadata information.

[0053] By recording small files into a storage object, immediately deleting the metadata information of the small files, and then releasing the physical storage object of the small files, the number of files in the stray directory can be reduced. The deletion of metadata information does not need to wait for the physical storage space to be released, thus avoiding the delays caused by queue waiting and OSD response waiting stages in related technologies.

[0054] The file deletion method provided in this application divides the files to be deleted into large files and small files according to a preset first volume threshold. Different deletion processes are applied to files of different sizes, which can improve the efficiency of file deletion. Specifically, for small files, the data block information of at least two small files is recorded in a storage object, and the physical storage space of at least two small files corresponding to the data block information in the storage object is released in batches. This can avoid frequent interaction with the storage device, improve the efficiency of physical storage space release, and further improve the efficiency of file deletion of small files.

[0055] Figure 3 Flowchart of the file deletion method provided in this application embodiment Figure 2 .like Figure 3 As shown, an embodiment of this application provides a file deletion method, which is described in detail below:

[0056] S201. Obtain the files to be deleted and divide them into large files and small files according to the preset first volume threshold.

[0057] For large files, proceed to step S208; for small files, proceed to steps S202 to S207.

[0058] S202. Determine the storage status of the storage object based on the data block information and the preset second volume threshold.

[0059] Storage states include active state and expired state. In the active state, the size of the data block information in the storage object is less than the second volume threshold, while in the expired state, the size of the data block information in the storage object is greater than or equal to the second volume threshold.

[0060] The second volume threshold limits the size of the small files corresponding to all data block information that a storage object can record. For example, a second volume threshold of 100MB means that the storage object is in an active state when the size of the small files corresponding to the data block information recorded in the storage object is less than 100MB; and in an expired state when the size of the small files corresponding to the data block information recorded in the storage object is greater than or equal to 100MB. An active state indicates that new small file data block information can continue to be appended to the current storage object; an expired state indicates that the current storage object can no longer record data block information.

[0061] Optionally, the storage status of a storage object can be determined based on a preset time threshold. For example, a time threshold of 1 hour: when the storage object has recorded data block information for 1 hour, the storage object is in an expired state; when the storage object has recorded data block information for less than 1 hour, the storage object is in an active state.

[0062] S203. Update the first attribute information of the storage object according to the storage status. The first attribute information is used to indicate the storage status of the storage object.

[0063] The first attribute information is used to record the storage status of the storage object. The first attribute information is established based on the storage object's extended (xattr) attributes, which allow users to associate additional attributes with the file system object. The first attribute information is such as `expire = true` or `expire = false`, where `expire = true` indicates the storage object is in an expired state, and `expire = false` indicates the storage object is in an active state. Optionally, if the storage object does not have first attribute information or the first attribute information is empty, it indicates that the storage object is in an active state.

[0064] S204. Based on the first attribute information, record the data block information of the small file into the storage object.

[0065] Specifically, when the first attribute information indicates that the storage object is active, the data block information of the small file is recorded in the active storage object. When the first attribute information indicates that the storage object is expired, a new storage object is created according to a preset naming rule, and the data block information of the small file is recorded in the new storage object. The preset naming rule is, for example: release_log_timestamp_random number.

[0066] S205. Delete the metadata information of small files.

[0067] S206. Periodically check the first attribute information of the storage objects, and identify the storage objects whose first attribute information indicates an expired state as archived objects.

[0068] Multiple storage objects may exist simultaneously in MDS. The background thread reads all storage objects in a prefix format according to a preset time period, then iterates through each storage object and checks the first attribute information. If the first attribute information indicates that the storage object is in an expired state, i.e., expire = true, then the storage object is determined as an archive object. Step S207 is executed on the archive object to release the physical storage space of the small files of the data block information objects in the archive object.

[0069] If the first attribute information indicates that the storage object is in an active state, i.e., expire = false, then the data block information of the new small file can continue to be appended to the storage object until the storage state of the storage object becomes expired, and then step S207 is executed.

[0070] S207. Release the physical storage space of the small files corresponding to the data block information in the archived object.

[0071] The background thread determines the OSD corresponding to the physical storage space, along with the OSD's address and offset, based on the data block information recorded in the archive file. It then interacts with the OSD to release the physical storage space, ultimately releasing both the target's metadata and the physical storage space, completing the deletion of the small file. After deleting the small file corresponding to the current storage space, it continues to periodically operate on other storage objects.

[0072] In one possible implementation, releasing the physical storage space of small files corresponding to data block information in the archived object includes: determining the identification information of the physical storage space based on the data block information in the archived object; determining the storage device storing the small files based on the identification information of the physical storage space; and sending a space release request to the storage device so that the storage device releases the physical storage space of the small files corresponding to the data block information in the archived object.

[0073] The identification information is used to indicate the OSD ID and the specific location of small files in the OSD. The corresponding storage device is determined based on the OSD ID, and then a space release request is sent to the storage device, so that the storage device releases the corresponding physical storage space and returns the released information to the MDS, thus completing the release of the physical storage space.

[0074] By establishing a precise mapping and release mechanism from data block information to physical storage space, efficient recycling of storage resources is achieved. By accurately locating target storage devices through identification information and directly sending release commands to them, a complete and automated space release closed loop is formed, which effectively improves the accuracy and automation level of storage space management. At the same time, it clarifies the separation of responsibilities between control flow and data flow in the system, providing a clear and reliable solution for resource management in a distributed storage environment.

[0075] Optional, such as Figure 4 As shown, releasing the physical storage space of the small files corresponding to the data block information in the archived object also includes:

[0076] S2071. Based on the preset flow control threshold, determine the size of the physical storage space that can be released at one time, and obtain the maximum release amount.

[0077] Background threads cannot continuously release physical storage space. Releasing physical storage space requires constant interaction with the OSD, affecting normal I / O (Input / Output) requests. Secondly, after reading all data block information in the archive object, there might be an exception such as a power outage after releasing some physical storage space. Upon recovery, it would be impossible to detect which physical storage spaces had been released, potentially leading to a double release issue. Therefore, flow control design is required.

[0078] The flow control threshold is predetermined and is used to limit how much physical storage space can be released at one time, i.e., the maximum release amount. The flow control threshold is, for example, 50M or 50 data blocks.

[0079] Optionally, the flow control threshold can be dynamically adjusted based on the size of the data block information in the archived object; and / or, the flow control threshold can be dynamically adjusted based on the system load status.

[0080] For smaller archive objects (e.g., less than 50M), the physical storage addresses of small files corresponding to all data block information can be released in a single instance, i.e., the flow control threshold is the size of all data block information in the archive object; for larger archive objects (e.g., greater than 500M), the flow control threshold can be set according to a fixed ratio, such as releasing 20% ​​of the physical storage space each time.

[0081] System load parameters, such as MDS resource utilization and OSD load balancing status, can be dynamically adjusted based on system load conditions. For example, the flow control threshold can be lowered under high load scenarios and raised under low load scenarios.

[0082] An adaptive flow control strategy balances deletion efficiency with resource stability. When handling large archive objects or under high system load, phased release prevents OSD node overload caused by releasing too much physical storage at once. Conversely, when handling small archive objects or under low system load, quickly releasing all physical storage reduces background thread waiting time. This mechanism significantly improves the system's adaptability to archive objects of varying sizes.

[0083] S2072. Based on the maximum release amount, release the corresponding size of physical storage space and record the release progress.

[0084] If the data block information in an archived object has not reached the maximum release amount, the next storage object will continue to be released; if the data block information in the current archived object exceeds the maximum release amount, the release progress of the current storage object will be recorded. The release progress can be recorded by recording the offset of the last released physical storage space.

[0085] S2073. Update the second attribute information of the archived object according to the release progress.

[0086] The second attribute information indicates the offset of the physical storage space that has already been released, so that the next release will begin from the offset indicated by the second attribute information. For example, the second attribute information is `progress=offset`, where `offset` represents the offset of the currently released physical storage space. When releasing again, the `progress` of the archive object is read, and the release continues from that progress; previously released physical storage space is not released again. If the second attribute information does not exist, it is assumed that the archive object has not yet started releasing, and the release will begin from the beginning.

[0087] Optionally, when restoring the metadata service after a power outage, the first attribute information of storage objects that were active before the power outage can be modified to an expired state.

[0088] Active storage objects are only stored in memory. When a system power failure occurs, the original active storage objects may not have reached 100MB or met the 1-hour time limit. That is, the first attribute information of the original active storage objects is active, so they are no longer stored in memory. During MDS recovery, all storage objects are scanned according to the prefix format, all storage objects are retrieved, and the first attribute information of the active storage objects is modified to expired. In other words, after the fault is recovered, all storage objects are recorded as archived objects. When files are deleted subsequently, new active storage objects are created to ensure the recovery status.

[0089] Optionally, when modifying the system time, the first attribute information of the active storage object is changed to expired. Since the storage object names include timestamps, creating new storage objects when modifying the system time may result in mismatched names, potentially causing ambiguity. Therefore, when modifying the system time, the first attribute information of the active storage object needs to be changed to expired, resulting in an archived object. New small files added after the system time modification are recorded in a newly created storage object named with the modified timestamp.

[0090] Through mandatory archiving and recovery mechanisms, the system can ensure the integrity of the deletion process during fault recovery. By marking active storage objects as expired when the system time is modified, the system can prevent confusion in the lifecycle management of storage objects caused by time rollback or jumps, thereby improving the reliability and robustness of file deletion.

[0091] By using the flow control threshold mechanism, fine-grained control and fault safety assurance can be achieved in the physical storage space release process. This avoids system input / output overload caused by large-scale centralized release. Furthermore, by accurately recording the release breakpoint through the second attribute information, it ensures that the release operation can be accurately resumed from the breakpoint after system interruption or restart. Thus, while improving the efficiency of storage space reclamation, it also ensures the stability of system resources and the recoverability of operations.

[0092] S208. Add large files to the cleanup queue, and release the metadata information and physical storage space of large files in the cleanup queue serially through the cleanup thread.

[0093] The file deletion method provided in this application achieves efficient decoupling and seamless connection between the two stages of data block information writing and physical storage space release through state division based on a second volume threshold and periodic state checks. It can ensure the reliable recording and orderly organization of data block information after the rapid deletion of metadata during the deletion of small files. Furthermore, through a state-driven release triggering mechanism, it ensures the timeliness and systematic nature of storage space reclamation, thereby improving deletion performance while maintaining the stable load and resource management efficiency of the storage system.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0095] Figure 5 This is a schematic diagram of the file deletion device provided in an embodiment of this application. Figure 5 As shown, embodiments of this application also provide a file deletion device 20, comprising:

[0096] The acquisition module 21 is used to acquire files to be deleted and divide the files to be deleted into large files and small files according to a preset first volume threshold. The large files have a file size greater than the first volume threshold, and the small files have a file size less than the first volume threshold.

[0097] The first processing module 22 is used to add large files to the cleanup queue and release the metadata information and physical storage space of large files in the cleanup queue serially through the cleanup thread.

[0098] The second processing module 23 is used to record the data block information of at least two small files into a storage object and to release the physical storage space of at least two small files corresponding to the data block information in the storage object in batches. The data block information is used to indicate the identification information of the physical storage space corresponding to the small file, and the storage object is used to record the data block information of at least two small files.

[0099] In one possible implementation, the second processing module 23 is specifically used for:

[0100] Based on the data block information and the preset second volume threshold, the storage status of the storage object is determined. The storage status includes active status and expired status. In the active status, the size of the data block information in the storage object is less than the second volume threshold, and in the expired status, the size of the data block information in the storage object is greater than or equal to the second volume threshold.

[0101] Based on the storage status, the first attribute information of the storage object is updated. The first attribute information is used to indicate the storage status of the storage object.

[0102] Based on the first attribute information, the data block information of the small file is recorded in the storage object. Specifically, when the first attribute information indicates that the storage object is in an active state, the data block information of the small file is recorded in the active storage object. When the first attribute information indicates that the storage object is in an expired state, a new storage object is created according to the preset naming rules, and the data block information of the small file is recorded in the new storage object.

[0103] In one possible implementation, the second processing module 23 is specifically used for:

[0104] Periodically check the first attribute information of the stored objects and identify the stored objects that are in an expired state as archived objects;

[0105] Release the physical storage space of the small files corresponding to the data block information in the archived object.

[0106] In one possible implementation, the second processing module 23 is further configured to:

[0107] Based on the data block information in the archived object, determine the identification information of the physical storage space;

[0108] The storage device for storing small files is determined based on the identification information of the physical storage space;

[0109] Send a space release request to the storage device to cause the storage device to release the physical storage space of the small files corresponding to the data block information in the storage object.

[0110] In one possible implementation, the second processing module 23 is further configured to:

[0111] Based on the preset flow control threshold, determine the size of the physical storage space that can be released at one time, and obtain the maximum release amount;

[0112] Based on the maximum release amount, release the corresponding size of physical storage space and record the release progress;

[0113] Based on the release progress, update the second attribute information of the archived object. The second attribute information is used to indicate the offset of the physical storage space that has been released, so that the physical storage space will be released from the offset indicated by the second attribute information in the next release.

[0114] In one possible implementation, the file deletion device 20 is also used for:

[0115] When restoring the metadata service after a power outage, the first attribute information of storage objects that were active before the power outage will be modified to an expired state.

[0116] When modifying the system time, the first attribute information of the active storage object is changed to expired.

[0117] In one possible implementation, the file deletion device 20 is also used for:

[0118] Retrieve the files to be deleted and move them to the directory where they remain.

[0119] The files to be deleted are divided into small files and large files according to a preset first volume threshold. The file size of small files is less than the first volume threshold, and the file size of large files is greater than or equal to the first volume threshold.

[0120] Large files are added to the cleanup queue, and the cleanup thread releases the metadata information and physical storage space of the large files in the cleanup queue in sequence.

[0121] In one possible implementation, the file deletion device 20 is also used to move the file to be deleted to the persistent directory.

[0122] In one possible implementation, the file deletion device 20 is also used for:

[0123] Dynamically adjust flow control thresholds based on the size of data blocks in the archived objects; and / or,

[0124] The flow control threshold is dynamically adjusted based on the system load status.

[0125] For a description of the features in the embodiment corresponding to the file deletion device, please refer to the relevant description in the embodiment corresponding to the file deletion method, which will not be repeated here.

[0126] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 30 provided in this embodiment includes at least one processor 301 and a memory 302. Optionally, the electronic device 30 further includes a communication component 303. The processor 301, memory 302, and communication component 303 are connected via a bus.

[0127] In a specific implementation, at least one processor 301 executes computer execution instructions stored in memory 302, causing at least one processor 301 to execute the above-described file deletion method embodiment.

[0128] The specific implementation process of processor 301 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0129] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0130] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0131] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0132] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described file deletion method embodiments when it is run.

[0133] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0134] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described file deletion method embodiments.

[0135] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described file deletion method embodiments.

[0136] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] The foregoing has provided a detailed description of a file deletion method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for deleting files, characterized in that, include: Obtain the files to be deleted, and divide the files to be deleted into large files and small files according to a preset first volume threshold, wherein the file size of the large files is greater than the first volume threshold, and the file size of the small files is less than the first volume threshold; For the large file, the large file is added to the cleanup queue, and the metadata information and physical storage space of the large file in the cleanup queue are released serially by the cleanup thread; For the small files, the data block information of at least two of the small files is recorded in a storage object, and the physical storage space of at least two of the small files corresponding to the data block information in the storage object is released in batches. The data block information is used to indicate the identification information of the physical storage space corresponding to the small files, and the storage object is used to record the data block information of at least two small files. The step of recording the data block information of at least two of the small files into the storage object includes: Based on the data block information and the preset second volume threshold, the storage state of the storage object is determined. The storage state includes an active state and an expired state. In the active state, the size of the data block information in the storage object is less than the second volume threshold, and in the expired state, the size of the data block information in the storage object is greater than or equal to the second volume threshold. Based on the storage state, the first attribute information of the storage object is updated, and the first attribute information is used to indicate the storage state of the storage object; Based on the first attribute information, the data block information of the small file is recorded in the storage object. When the first attribute information indicates that the storage object is in an active state, the data block information of the small file is recorded in the active storage object. When the first attribute information indicates that the storage object is in an expired state, a new storage object is created according to a preset naming rule, and the data block information of the small file is recorded in the new storage object.

2. The file deletion method according to claim 1, characterized in that, The batch release of the physical storage space of at least two small files corresponding to the data block information in the storage object includes: The first attribute information of the storage object is periodically checked, and the storage object that is indicated as expired by the first attribute information is identified as an archived object; Release the physical storage space of the small files corresponding to the data block information in the archived object.

3. The file deletion method according to claim 2, characterized in that, Releasing the physical storage space of the small files corresponding to the data block information in the archived object includes: Based on the data block information in the archived object, determine the identification information of the physical storage space; The storage device for storing the small file is determined based on the identification information of the physical storage space; Send a space release request to the storage device so that the storage device releases the physical storage space of the small files corresponding to the data block information in the storage object.

4. The file deletion method according to claim 2, characterized in that, Releasing the physical storage space of the small files corresponding to the data block information in the archived object includes: Based on the preset flow control threshold, determine the size of the physical storage space that can be released at one time, and obtain the maximum release amount; Based on the maximum release amount, release the corresponding size of physical storage space and record the release progress; Based on the release progress, update the second attribute information of the archived object. The second attribute information is used to indicate the offset of the physical storage space that has been released, so that the physical storage space will be released from the offset indicated by the second attribute information in the next release.

5. The file deletion method according to any one of claims 1 to 4, characterized in that, The method further includes: When restoring the metadata service after a power outage, the first attribute information of storage objects that were active before the power outage will be modified to an expired state. When modifying the system time, the first attribute information of the active storage object is changed to expired.

6. The file deletion method according to claim 1, characterized in that, After obtaining the file to be deleted, the method further includes: moving the file to be deleted to the persistent directory.

7. The file deletion method according to claim 1, characterized in that, Before releasing the physical storage space of the small file corresponding to the data block information in the storage object, the method further includes: deleting the metadata information of the small file.

8. The file deletion method according to claim 4, characterized in that, The method further includes: The flow control threshold is dynamically adjusted based on the size of the data block information in the archived object; and / or, The flow control threshold is dynamically adjusted based on the system load status.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the file deletion method as described in any one of claims 1 to 8 when executing the computer program.

Images