File processing method and device, and electronic device

Abstract

The application discloses a file processing method and device and electronic equipment. The method comprises the following steps: receiving a file processing request, wherein the file processing request at least contains cursor information, and the cursor information is used for identifying a first to-be-deleted file in a database cluster; according to a preset deletion sequence, the cursor information is used to delete part of file data of at least one to-be-deleted file in the database cluster in a parallel deletion mode; after each deletion operation is completed, the total amount of deleted data is counted, and when the total amount of deleted data is greater than a preset threshold, metadata information is updated, wherein the metadata information at least comprises the latest target cursor information and the remaining target data amount of each to-be-deleted file, and the target cursor information is used to represent the first to-be-deleted file when the next deletion operation is performed. The application solves the technical problem of low garbage file deletion efficiency in the prior art.

Description

Technical Field

[0001] This application relates to the field of document processing, and more specifically, to a document processing method, apparatus, and electronic device. Background Technology

[0002] With the advent of the big data era, database clusters are being widely used in the implementation of distributed file systems in order to process data more efficiently.

[0003] Existing distributed file systems, when deleting multiple junk files from a database cluster, employ a sequential, file-by-file deletion approach, always starting from the beginning of each junk file and deleting data from front to back. This method is inefficient in two ways: firstly, it cannot handle multiple junk files simultaneously, resulting in low file deletion efficiency; secondly, if the distributed file system restarts during the head-to-tail deletion process, it needs to re-detect each junk file, requiring some already deleted files to be retrieved again, further exacerbating the low file deletion efficiency problem.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a file processing method, apparatus, and electronic device to at least solve the technical problem of low efficiency in deleting junk files in the prior art.

[0006] According to one aspect of the embodiments of this application, a file processing method is provided, comprising: receiving a file processing request, wherein the file processing request includes at least cursor information, the cursor information being used to identify the first file to be deleted in a database cluster; deleting a portion of the file data of at least one file to be deleted in the database cluster according to a preset deletion order and using a parallel deletion method based on the cursor information, wherein the amount of the partial file data is less than or equal to a preset data amount, and when deleting each file to be deleted, the file data is deleted from the end of each file to be deleted backwards; after each deletion operation is completed, counting the total amount of deleted data, and updating metadata information when the total amount of deleted data is greater than or equal to a preset threshold, wherein the metadata information includes at least the latest target cursor information and the remaining target data amount of each file to be deleted, the target cursor information being used to characterize the first file to be deleted when the next deletion operation is performed.

[0007] Furthermore, the file processing method also includes: determining a preset number of files to be deleted from the database cluster according to a preset deletion order; deleting the preset number of files to be deleted using a parallel deletion method, wherein the deletion process includes at least one deletion operation, and the maximum amount of file data deleted in each deletion operation is a preset amount of data.

[0008] Furthermore, the file processing method also includes: detecting whether there is any abnormality in the file to be deleted during each deletion operation; determining that the file to be deleted is an abnormal file when there is an abnormality in the deletion process; and moving the garbage file record corresponding to the abnormal file to the first file queue.

[0009] Furthermore, the file processing method also includes: each time a file to be deleted is deleted, checking whether the file data of the file to be deleted has been completely deleted; if the file data of the file to be deleted has not been completely deleted, moving the garbage file record corresponding to the file to be deleted to the second file queue; if the file data of the file to be deleted has been completely deleted, moving the garbage file record corresponding to the file to be deleted to the third file queue.

[0010] Furthermore, the file processing method also includes: each time a file to be deleted is performed, determining the amount of file data that has not yet been deleted as the amount of data to be deleted; when the amount of data to be deleted is greater than or equal to a fixed amount of data, deleting the fixed amount of file data from the end of the file to be deleted backwards, wherein the fixed amount of data is the maximum amount of data that can be deleted from a file to be deleted in each deletion operation; when the amount of data to be deleted is less than the fixed amount of data, deleting the entire file to be deleted.

[0011] Furthermore, the file processing method also includes: determining the target file based on the garbage file records in the second file queue, wherein the target file is the file with the smallest file sequence number among all files to be deleted in the second file queue, and the file sequence number is used to represent the deletion order of the files to be deleted; updating the cursor information based on the target file to obtain target cursor information, wherein the target cursor information indicates that the target file is the first file to be deleted when the next deletion operation is performed; updating the data volume of the file corresponding to each garbage file record in the second file queue to obtain the target data volume; updating the data volume of the file corresponding to each garbage file record in the third file queue to 0; and updating the metadata information based on the target data volume and the target cursor information.

[0012] Furthermore, the file processing method also includes: when updating the cursor information according to the target file to obtain the target cursor information, determining the junk file records to be deleted based on the target cursor information and the cursor information; and deleting the junk file records to be deleted.

[0013] Furthermore, the file processing method also includes: after updating the metadata information, clearing the total amount of deleted data, and recalculating the total amount of deleted data after the next deletion operation is completed; clearing the junk file records in the first file queue and the third file queue.

[0014] Furthermore, the file processing method also includes: after updating the metadata information, detecting whether the distributed file system used to manage the database cluster has malfunctioned and restarted; after the distributed file system restarts successfully, deleting the remaining files to be deleted in the database cluster according to the target cursor information.

[0015] Furthermore, the file processing method also includes: receiving a file processing termination request, wherein the file processing termination request is used to request the termination of the currently executing file processing request within a preset duration; determining a maximum waiting duration based on the preset duration, wherein the maximum waiting duration is less than the preset duration; detecting a first file to be deleted that has completed the deletion operation within the maximum waiting duration; obtaining the latest data volume of the first file to be deleted; updating the metadata information based on the latest data volume; and prohibiting the execution of new file processing requests.

[0016] Furthermore, the file processing method also includes: after determining the maximum waiting time according to the preset duration, detecting whether the update operation of the metadata information has been completed within the maximum waiting time; if the update operation of the metadata information has been completed within the maximum waiting time, generating a first response message, wherein the first response message indicates that the file processing request has been completed.

[0017] According to another aspect of the embodiments of this application, a file processing apparatus is also provided, comprising: a receiving module, configured to receive a file processing request, wherein the file processing request includes at least cursor information, the cursor information being used to identify the first file to be deleted in a database cluster; a deletion module, configured to delete partial file data of at least one file to be deleted in the database cluster according to a preset deletion order and using a parallel deletion method based on the cursor information, wherein the amount of partial file data is less than or equal to a preset data amount, and when deleting each file to be deleted, the file data is deleted from the end of each file to be deleted backwards; and an updating module, configured to count the total amount of deleted data after each deletion operation is completed, and update metadata information when the total amount of deleted data is greater than or equal to a preset threshold, wherein the metadata information includes at least the latest target cursor information and the remaining target data amount of each file to be deleted, the target cursor information being used to characterize the first file to be deleted when the next deletion operation is performed.

[0018] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the above-described file processing method when it is run.

[0019] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to implement the file processing method described above.

[0020] In this application, a parallel deletion method is adopted for files to be deleted in a database cluster. First, a file processing request is received, which includes at least cursor information used to identify the first file to be deleted in the database cluster. Then, according to a preset deletion order, partial file data of at least one file to be deleted in the database cluster is deleted using a parallel deletion method based on the cursor information. After each deletion operation, the total amount of deleted data is counted, and the metadata information is updated when the total amount of deleted data is greater than or equal to a preset threshold. The amount of partial file data is less than or equal to a preset data amount. When deleting each file to be deleted, the file data is deleted from the end of each file backwards. The metadata information includes at least the latest target cursor information and the remaining target data amount of each file to be deleted. The target cursor information is used to identify the first file to be deleted when the next deletion operation is performed.

[0021] As described above, this application changes the sequential file deletion method to a parallel deletion method. During each deletion operation, at least one file to be deleted is deleted simultaneously, thus solving the problem of low file deletion efficiency in existing technologies due to the inability to delete multiple files simultaneously. Furthermore, this application changes the data deletion method for each file, deleting data from the end of each file to be deleted backwards. When the total amount of deleted data is greater than or equal to a preset threshold, the cursor information and the remaining data amount of each file to be deleted are updated. In other words, this application records which file is currently to be deleted. Even after the distributed file system restarts, it can continue deleting the remaining files based on the latest target cursor information without re-searching for already deleted files, thereby further improving file deletion efficiency.

[0022] Therefore, the technical solution of this application achieves the goal of avoiding the need to re-search for deleted files, thereby improving the efficiency of file deletion and solving the technical problem of low efficiency in deleting junk files in the prior art. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a flowchart of an optional file processing method according to an embodiment of this application;

[0025] Figure 2 This is a schematic diagram illustrating the deletion of junk files according to an embodiment of this application;

[0026] Figure 3 This is an optional flowchart for deleting junk files according to an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of an optional document processing apparatus according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] According to an embodiment of this application, a document processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the following explains some terms used in the embodiments of this application:

[0033] Distributed file system: A storage system that provides file interface semantics to the outside world, used to distribute data across multiple storage servers and combine these distributed storage resources into a virtual storage device;

[0034] MDS: Meta Data System, a daemon process used to manage, store, and distribute metadata information in a distributed file system;

[0035] Junk files: These are files whose metadata has been deleted by the distributed file system, but whose data is still stored in the database cluster. Junk files are files that cannot be recovered.

[0036] Garbage collection space: storage space used for persistent garbage file records, where each garbage file record contains file information of a garbage file;

[0037] Garbage collection: When garbage files are deleted, the metadata space has been released, but the data space has not yet been released. Garbage collection is a deletion operation that releases the space occupied by data and deletes the corresponding garbage file records in the garbage collection space.

[0038] Figure 1 This is a flowchart of an optional file processing method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0039] Step S101: Receive a file processing request.

[0040] In step S101, the file processing request includes at least cursor information, which is used to identify the first file to be deleted in the database cluster.

[0041] Specifically, a database cluster managed by a distributed file system may contain multiple junk files. The metadata of these junk files has been deleted by the distributed file system, but the file data of the junk files still remains in the database cluster.

[0042] It's important to note that files in a distributed file system consist of two parts: metadata and data. Data can be understood as the file's physical data, while metadata describes the file's data, such as modification time, size, and permissions. To improve deletion efficiency, when a user deletes a file through the distributed file system, it first removes the file's metadata so that, from the user's perspective, the file appears deleted. Since the file data may be large, the distributed file system subsequently removes the file data from the database cluster. To record these garbage files with their metadata removed, the distributed file system has a garbage collection space to store garbage file records. These records contain information about each garbage file, such as its location within the database cluster and its size.

[0043] In addition, before deleting junk files, the distributed file system first receives a file processing request. This request can be a scheduled task of the distributed file system, which automatically generates a file processing request at preset time intervals, or it can be a file processing request manually created by the user in the distributed file system, for example, through the command line. The file processing request contains at least cursor information to identify the first file to be deleted in this junk file deletion process. For example, if the database cluster contains 1000 junk files from 000 to 999, and the cursor information points to junk file 000, then junk file 000 is the first file to be deleted in this process.

[0044] Step S102: According to the preset deletion order, delete part of the file data of at least one file to be deleted in the database cluster using a parallel deletion method based on the cursor information.

[0045] In step S102, if the amount of data in some files is less than or equal to a preset data amount, when deleting each file to be deleted, the file data is deleted from the end of each file to be deleted, proceeding backwards. The preset data amount is the product of a preset quantity and a fixed data amount. The preset quantity is the maximum number of files that can be deleted in each deletion operation, and the fixed data amount is the maximum amount of data that can be deleted from a single file to be deleted in each deletion operation.

[0046] Specifically, the preset deletion order is from front to back. This deletion order can be understood as the order in which files are deleted. For example, sorting by the sequence number of the junk files, if the first file to be deleted is junk file with sequence number 000, the next files to be deleted will be junk files with sequence number 001, 002, 003, and so on. Furthermore, in this application, the distributed file system uses a parallel deletion method to delete junk files. In other words, each deletion operation can delete multiple files simultaneously; for example, 10 files can be selected and deleted at the same time.

[0047] Furthermore, it should be noted that, in order to improve deletion efficiency, in this application, the maximum amount of data to be deleted from a single junk file is less than or equal to a fixed amount of data. Also, unlike the prior art which deletes file data from the beginning of the junk file, this application deletes the file data of each junk file from the end of the junk file.

[0048] Optionally, each time a file to be deleted is to be deleted, the distributed file system first determines the amount of file data that has not yet been deleted as the amount of data to be deleted. If the amount of data to be deleted is greater than or equal to a fixed amount of data, the fixed amount of file data is deleted from the end of the file to be deleted, starting from the end and working backwards. The fixed amount of data is the maximum amount of data that can be deleted from a file to be deleted in each deletion operation. If the amount of data to be deleted is less than the fixed amount of data, the entire file to be deleted is deleted.

[0049] in, Figure 2 This illustration shows a schematic diagram of junk file deletion according to an embodiment of this application, such as... Figure 2 As shown, assuming a garbage collection (corresponding to one file processing request) reads 100 garbage file records from the garbage collection space, the maximum number of concurrent garbage files processed is set to 4, meaning a maximum of 4 garbage files can be selected for deletion at a time. Simultaneously, the maximum data size (i.e., a fixed data size) to delete from a single file is set to 16 MiB, with deletion starting from the end of each file and proceeding backwards. Figure 2The code simultaneously selects four junk files—file_01, file_02, file_03, and file_04—for parallel deletion. The total file size of file_1 is 32 GiB, file_02 is 4 KiB, file_03 is 32 GiB, and file_04 is 4 MiB. The slashed portion indicates the amount of data to be deleted. Specifically, since file_01 and file_03 have a total file size greater than 16 MiB, the amount of data deleted from these two junk files is a fixed 16 MiB. Since file_02 and file_04 have a total file size less than 16 MiB, all file data from these two junk files can be deleted during this process. It's important to note that because a preset number and a fixed data size are set, the preset data size can be the product of the preset number and the fixed data size.

[0050] Furthermore, it's important to note that existing technologies that sequentially delete junk files are based on the data organization of a database cluster. This means that all data blocks of the same junk file may belong to the same placement group. If processed sequentially, a large number of deletion operations may fall into the same placement group. Operations within the same placement group may be sequential or involve resource contention, thus affecting the efficiency of file deletion. This application employs concurrent processing to delete data from multiple junk files simultaneously. The placement group to which the data blocks of each junk file belong is completely random, thereby fully utilizing the processing power of the database cluster, avoiding one or more placement groups becoming bottlenecks, and ultimately improving the efficiency of file deletion.

[0051] Step S103: After each deletion operation is completed, the total amount of deleted data is counted, and when the total amount of deleted data is greater than or equal to a preset threshold, the metadata information is updated.

[0052] In step S103, the metadata information includes at least the latest target cursor information and the remaining target data volume of each file to be deleted. The target cursor information is used to represent the first file to be deleted when the next deletion operation is performed. Specifically, in the prior art, after a distributed file system encounters an anomaly and restarts, because the cursor information is not updated, each file needs to be re-detected, resulting in some already deleted files being retrieved again, causing low file deletion efficiency. To solve this problem, this application triggers a metadata information update mechanism after the total amount of deleted data is greater than or equal to a preset threshold. The metadata information is the metadata information of the garbage collection space, which specifically includes: the latest target cursor information, i.e., which garbage file to start deleting from next; the latest data volume information of each garbage file; and the garbage file records of the failed deletion files being reinserted into the garbage collection space.

[0053] Optionally, the aforementioned preset threshold can be customized, and this embodiment does not impose any special limitations on it. It should be noted that in the prior art, especially when performing garbage collection on files with large amounts of data, if the distributed file system reads 100 garbage file records and performs garbage collection on these 100 garbage files sequentially, but the main MDS process of the distributed file system unexpectedly exits due to software errors or other reasons during the operation, the main MDS will switch over. The new main MDS will still read 100 records from the previous garbage file cursor. At this time, a large amount of already deleted file data (database key-value pairs) will be re-searched and deleted, which not only increases unnecessary garbage collection time but also leads to a serious decrease in search efficiency.

[0054] To address the problems of the prior art, this application modifies the data deletion method for each file, deleting data from the end of each file to be deleted backwards. Furthermore, when the total deleted data exceeds or equals a preset threshold, the cursor information and the remaining data in each file to be deleted are updated. This ensures that the distributed file system records the next file to be deleted as promptly as possible. Even after the distributed file system restarts, it can continue deleting the remaining files and data based on the latest target cursor information, without re-searching for already deleted files and data, thus further improving file deletion efficiency.

[0055] Based on the content of steps S101 to S103 above, it can be seen that in this application, a parallel deletion method is adopted for files to be deleted in the database cluster. First, a file processing request is received, wherein the file processing request includes at least cursor information, which is used to identify the first file to be deleted in the database cluster. Then, according to a preset deletion order, partial file data of at least one file to be deleted in the database cluster is deleted in parallel according to the cursor information. After each deletion operation is completed, the total amount of deleted data is counted, and when the total amount of deleted data is greater than or equal to a preset threshold, the metadata information is updated. The amount of partial file data is less than or equal to a preset amount of data. When deleting each file to be deleted, the file data is deleted from the end of each file backwards. The metadata information includes at least the latest target cursor information and the remaining target amount of data for each file to be deleted. The target cursor information is used to identify the first file to be deleted when the next deletion operation is performed.

[0056] As can be seen from the above, this application changes the method of deleting files one by one to a parallel deletion method. In each deletion operation, at least one file to be deleted is deleted at the same time, thereby solving the problem of low file deletion efficiency caused by the inability to delete multiple files at the same time in the prior art. On the other hand, this application also changes the data deletion method for each file. It deletes file data from the end of each file to be deleted backwards, and when the total amount of deleted data is greater than or equal to a preset threshold, it updates the cursor information and the amount of data remaining in each file to be deleted. In other words, this application records which file is currently to be deleted in a timely manner. Even after the distributed file system restarts, the distributed file system can continue to delete the remaining files to be deleted based on the latest target cursor information, without having to search for the already deleted files again, thereby further improving the efficiency of file deletion.

[0057] Therefore, the technical solution of this application achieves the goal of avoiding the need to re-search for deleted files, thereby improving the efficiency of file deletion and solving the technical problem of low efficiency in deleting junk files in the prior art.

[0058] In one optional embodiment, the distributed file system determines a preset number of files to be deleted from the database cluster according to a preset deletion order, and then deletes the preset number of files to be deleted in a parallel deletion manner. The deletion process includes at least one deletion operation, and the maximum amount of file data deleted in each deletion operation is a preset amount of data.

[0059] Optionally, assuming a garbage collection reads 1000 garbage file records (000-999) from the garbage collection space in one operation, with a preset number of 10, and each garbage file can delete a fixed amount of data of up to 16 MiB in a single deletion operation, then according to the preset deletion order, the distributed file system selects 10 garbage files to delete their data simultaneously each time, and the maximum amount of data deleted in each deletion operation (i.e., the preset amount of data) is 16 MiB * 10 = 160 MiB.

[0060] In one optional embodiment, each time a file to be deleted is deleted, the distributed file system will detect whether there is any abnormality in the deletion process. If there is an abnormality in the deletion process, the distributed file system will determine that the file to be deleted is an abnormal file and move the garbage file record corresponding to the abnormal file to the first file queue.

[0061] Among them, the files corresponding to the garbage file records in the first file queue are all files that failed to be deleted.

[0062] Optional, Figure 3 A flowchart illustrating an optional junk file deletion process according to an embodiment of this application is shown, such as... Figure 3 As shown, the distributed file system first traverses the garbage file records in the garbage collection space and selects garbage files for processing according to the set maximum number of concurrent files to be processed (i.e., the preset number, assumed to be 4) and the preset deletion order. The garbage file records corresponding to the selected garbage files are first added to the "active file index queue," where the index is the file sequence number. For example... Figure 3 As shown, assume the selected junk files are file 1, file 2, file 3, and file 4.

[0063] Optionally, after adding the four junk file records corresponding to files 1, 2, 3, and 4 to the "active file index queue," the distributed file system sends a data deletion request to the database cluster. Upon receiving the request, the database cluster simultaneously deletes the data from the four junk files. If the deletion of any junk file fails, the distributed file system identifies it as an abnormal file and moves its junk file record to the first file queue. For example... Figure 3 As shown, an error occurred during the deletion of file 2. Therefore, the distributed file system moved file 2 to the "index queue of failed files", which is the first file queue.

[0064] In one optional embodiment, each time a file to be deleted is performed, the distributed file system also checks whether all the file data of the file to be deleted has been deleted. If the file data of the file to be deleted has not been completely deleted, the garbage file record corresponding to the file to be deleted is moved to the second file queue; if the file data of the file to be deleted has been completely deleted, the garbage file record corresponding to the file to be deleted is moved to the third file queue. The file corresponding to the garbage file record in the second file queue is the file to be deleted in the next deletion operation.

[0065] like Figure 3 As shown, file 4 has deleted all its data in this deletion operation, meaning that the latest file data size of file 4 is 0. Based on this, the distributed file system moves the garbage file record corresponding to file 4 to the "index queue of file size=0", which is the third file queue mentioned above.

[0066] Additionally, some junk files have a large amount of data, and the data cannot be completely deleted in this deletion operation (i.e., the file data size is greater than a fixed amount). For example, Figure 3 Files 1 and 3 in the list are junk files whose data was not completely deleted during this deletion operation. Therefore, the junk file records for files 1 and 3 remain in the "active file index queue." Simultaneously, the distributed file system will continue to select new junk file records to enter the "active file index queue" according to a preset deletion order. For example... Figure 3 The garbage file records for file 5 and file 6 are shown in the table. The "active file index queue" is the second file queue.

[0067] In one optional embodiment, the distributed file system further determines the target file based on the garbage file records in the second file queue. The target file is the file with the smallest file sequence number among all files to be deleted in the second file queue, and the file sequence number indicates the deletion order of the files to be deleted. Then, the distributed file system updates the cursor information based on the target file to obtain target cursor information, where the target cursor information indicates that the target file is the first file to be deleted when the next deletion operation is performed. Simultaneously, the distributed file system updates the data volume of the file corresponding to each garbage file record in the second file queue to obtain the target data volume, and updates the data volume of the file corresponding to each garbage file record in the third file queue to 0. Finally, the distributed file system updates the metadata information based on the target data volume and the target cursor information.

[0068] Optional, such as Figure 3 As shown, if the amount of deleted file data reaches a preset threshold, the distributed file system will update the metadata information of the garbage collection space. The update includes at least the following: updating the data size of the file corresponding to each garbage file record in the "active file index queue" to obtain the latest remaining target data size for each file; updating the data size of the garbage file corresponding to each garbage file record in the "size=0 file index queue" to 0; and rewriting the garbage file records corresponding to abnormal files in the "failed file index queue" into the garbage collection space. The purpose of rewriting the garbage file records corresponding to abnormal files into the garbage collection space is to allow files that failed to be deleted to be re-deleted after a certain period of time.

[0069] Additionally, when the total amount of deleted data is greater than or equal to a preset threshold, the distributed file system will also check the target file corresponding to the second file queue. The target file is the file with the smallest file sequence number among all files in the second file queue. Then, the distributed file system updates the cursor information based on the target file to obtain the target cursor information. For example, in... Figure 3 In the first instance, when the total amount of data deleted reaches the preset threshold, the target file corresponding to the "active file index queue" is file 1. Since the old cursor information also indicates that the first file to be deleted is file 1, the distributed file system will not update the cursor information at this time. However, when the total amount of data deleted reaches the preset threshold for the second time, the target file corresponding to the "active file index queue" is file 5. At this time, the distributed file system will update the cursor information, and the latest target cursor information indicates that file 5 is the first file to be deleted in the next deletion operation.

[0070] In one optional embodiment, after updating the metadata information, the distributed file system also zeroes out the total amount of deleted data and recalculates the total amount of deleted data after the next deletion operation is completed. Simultaneously, the distributed file system also clears the garbage file records in the first file queue and the third file queue.

[0071] Optional, such as Figure 3 As shown, after each update to the metadata information of the garbage collection space, the distributed file system clears the "index queue of files with size=0" and the "index queue of failed files". Then, the distributed file system resets the total amount of deleted data to zero and continues to send data deletion requests to the database cluster. Once the total amount of deleted data reaches the preset threshold again, the distributed file system updates the metadata information of the garbage collection space a second time. Figure 3In the process, when the total amount of deleted data reaches the preset threshold again, the garbage file records in the "index queue of files with size=0" correspond to files 1 and 3, respectively, while the garbage files in the "index queue of active files" are files 5, 6, 7, and 8. It's important to note that failed deletions can also be considered as having been processed. Therefore, files 1, 2, 3, and 4 have now been completely processed. Based on this, in this metadata update, in addition to updating the file data volume, the cursor information also needs to be updated. The latest target cursor information indicates that file 5 is the first file to be deleted in the next deletion operation.

[0072] In addition, when updating the cursor information based on the target file and obtaining the target cursor information, the distributed file system will determine the garbage file records to be deleted based on the target cursor information and the cursor information; and delete the garbage file records to be deleted.

[0073] Optionally, after all the data in a garbage file is deleted, the corresponding garbage file record can also be deleted. Specifically, to ensure the continuity of garbage file records, the distributed file system determines the garbage file record to be deleted based on the old cursor information and the new cursor information (i.e., the target cursor information). For example, ... Figure 3 As shown, the old cursor information points to the garbage file 1, and the new cursor information points to the garbage file 4. The distributed file system can determine that the garbage file records of file 1, file 2, and file 3 are garbage file records to be deleted. The distributed file system will delete these three garbage file records in the garbage collection space.

[0074] Finally, as Figure 3 As shown, after completing the second metadata update, the distributed file system will continue processing the garbage files corresponding to the garbage file records in the "active file index queue" until all 100 garbage files have been processed. Finally, after all 100 garbage files have been processed, the metadata information needs to be updated once more. This final update only needs to update the garbage files that failed to be deleted and the cursor information.

[0075] In an optional embodiment, the distributed file system in this application may also introduce a circuit breaker mechanism. Specifically, if the asynchronous update is slow to complete, the latest submitted asynchronous update needs to be changed to a synchronous update after the circuit breaker criterion is reached. This avoids a large gap between the actual amount of data deleted and the metadata records in the garbage collection space, thereby ultimately preventing a large number of requests to try to search and delete the deleted data blocks (database key-value pairs) after the garbage collection is interrupted and resumed, which would lead to a decrease in the performance of the garbage collection operation.

[0076] For example, the circuit breaker mechanism described above can be defined as follows: if there are three or more metadata update requests (including the latest metadata update request about to be committed) simultaneously in the asynchronous update queue, where each metadata update request is asynchronous (meaning two asynchronous update requests have not yet been completed), then, assuming the trigger criterion for the circuit breaker mechanism is three asynchronous update requests, the third metadata update request will become a synchronous request. That is, only after the metadata update is completed can the distributed file system continue to send data deletion requests to the database cluster.

[0077] In an optional embodiment, the distributed file system is further provided with a self-checking plugin, which is used to detect whether the distributed file system has encountered an anomaly and has been restarted. After the distributed file system restarts successfully, it will delete the remaining files to be deleted in the database cluster according to the target cursor information.

[0078] Optionally, unlike existing technologies that require re-searching for deleted junk files and file data after each abnormal restart of the distributed file system, the distributed file system of this application determines the remaining files to be deleted and the file data to be deleted based on the latest metadata information after an abnormal restart, thereby avoiding the repeated search process and improving the efficiency of deleting junk files.

[0079] In an optional embodiment, the distributed file system can further receive a file processing termination request, wherein the file processing termination request is used to request the termination of the currently executing file processing request within a preset duration. The distributed file system first determines a maximum waiting time based on the preset duration, then detects the first file to be deleted that has completed the deletion operation within the maximum waiting time, and obtains the latest data volume of the first file to be deleted, thereby updating the metadata information based on the latest data volume, while simultaneously prohibiting the execution of new file processing requests. The maximum waiting time is less than the preset duration.

[0080] Optionally, when a database cluster composed of MDSs detects a change in topology, it needs to immediately compare and synchronize metadata information between MDSs. Therefore, external and internal requests must be completed or abandoned within a controllable timeframe. Garbage collection, as an internal request, also needs to meet the above requirements.

[0081] To ensure that ongoing file processing requests are terminated within a preset timeframe, this application, upon receiving a file processing termination request, first prohibits the distributed file system from executing new file processing requests (including prohibiting the acceptance of automatically generated new file processing requests and prohibiting the processing of user-created new file processing requests). Then, for file processing requests already in progress (i.e., garbage collection requests), the distributed file system determines a maximum waiting time based on the preset timeframe; for example, the maximum waiting time can be half of the preset timeframe. Within the maximum waiting timeframe, if all file processing requests have returned (i.e., all files to be deleted corresponding to the file processing requests have been deleted), the distributed file system updates the metadata information of the garbage collection space using the latest data volume of each file to be deleted. Simultaneously, the distributed file system also needs to determine whether to update the cursor information based on the target files corresponding to the "active file index queue." If not all file processing requests have returned, the metadata information of the files to be deleted that have not yet returned will be updated directly using the latest data volume before the current deletion operation. Again, the distributed file system needs to determine whether to update the cursor information based on the target files corresponding to the "active file index queue." Finally, for file processing requests that return after exceeding the maximum waiting time, the metadata information of the garbage collection space will no longer be updated. However, the distributed file system will retain the return record and use the return record as the basis for subsequent metadata information updates.

[0082] In one optional embodiment, metadata updates are performed between MDS clusters. Delays in these updates may occur due to network, disk failures, or software issues. Therefore, the termination time for the metadata update operation needs to be determined based on the maximum waiting time. Specifically, after determining the maximum waiting time based on a preset duration, the distributed file system checks whether the metadata update operation has been completed within the maximum waiting time. If the update is completed within the maximum waiting time, the distributed file system generates a first response message, indicating that the file processing request has been completed. Optionally, if the metadata update is completed within the maximum waiting time, the distributed file system immediately generates the first response message and reports the completion of the garbage collection operation to the main MDS. If the metadata update is not completed within the maximum waiting time, the distributed file system releases the requested resources in the metadata update queue. Subsequently, even if the metadata update is completed on all MDSs, since there is no corresponding request record in the metadata update queue, a data deletion request will not be triggered to the database cluster (asynchronous requests will not trigger this; they will only trigger when asynchronous requests degenerate into synchronous requests). Finally, the distributed file system directly reports to the master mds that the garbage collection operation is complete.

[0083] As described above, in certain scenarios, the primary MDS needs to actively terminate the garbage collection process. For example, when the MDS topology changes, each MDS needs to terminate all external (client) and internal (e.g., garbage collection, automatic deletion of expired files in the recycle bin) requests before synchronizing information between MDSs. This requires the garbage collection process to be terminated quickly. This application provides users with the ability to actively terminate garbage collection by determining the maximum waiting time based on a preset duration and promptly terminating ongoing file processing requests within that maximum waiting time, thus improving the user experience.

[0084] Example 2

[0085] According to another aspect of the embodiments of this application, a document processing apparatus is also provided, wherein, Figure 4 This is a schematic diagram of an optional document processing apparatus according to an embodiment of this application. Figure 4 As shown, the device includes: a receiving module 401, a deletion module 402, and an update module 403.

[0086] The receiving module 401 is used to receive a file processing request, wherein the file processing request includes at least cursor information, which is used to identify the first file to be deleted in the database cluster; the deletion module 402 is used to delete a portion of the file data of at least one file to be deleted in the database cluster according to a preset deletion order and using a parallel deletion method based on the cursor information, wherein the amount of data in the portion of the file data is less than or equal to a preset amount of data, and when deleting each file to be deleted, the file data is deleted from the end of each file to be deleted; the updating module 403 is used to count the total amount of data deleted after each deletion operation is completed, and update the metadata information when the total amount of data deleted is greater than or equal to a preset threshold, wherein the metadata information includes at least the latest target cursor information and the remaining target amount of data for each file to be deleted, and the target cursor information is used to identify the first file to be deleted when the next deletion operation is performed.

[0087] It should be noted that the receiving module 401, the deletion module 402 and the updating module 403 mentioned above correspond to steps S101 to S103 in the above embodiments. The three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0088] Optionally, the deletion module further includes a first determining unit and a first deleting unit. The first determining unit is used to determine a preset number of files to be deleted from the database cluster according to a preset deletion order; the first deleting unit is used to delete the preset number of files to be deleted using a parallel deletion method, wherein the deletion process includes at least one deletion operation, and the maximum amount of file data deleted in each deletion operation is a preset data amount.

[0089] Optionally, the first deletion unit further includes: a first deletion submodule, a first determination submodule, and a first move submodule. The first deletion submodule is used to detect whether there is any abnormality in the file to be deleted during each deletion operation; the first determination submodule is used to determine that the file to be deleted is an abnormal file when an abnormality is detected during the deletion process; and the first move submodule is used to move the garbage file record corresponding to the abnormal file to a first file queue.

[0090] Optionally, the first deletion unit further includes: a first detection submodule, a second moving submodule, and a third moving submodule. The first detection submodule is used to detect whether all file data of the file to be deleted has been deleted each time a deletion operation is performed. The second moving submodule is used to move the corresponding junk file record of the file to be deleted to a second file queue if the file data of the file to be deleted has not been completely deleted. The third moving submodule is used to move the corresponding junk file record of the file to be deleted to a third file queue if the file data of the file to be deleted has been completely deleted.

[0091] Optionally, the first deletion unit further includes: a second determining submodule, a second deleting submodule, and a third deleting submodule. The second determining submodule is used to determine the amount of file data that has not yet been deleted in the file to be deleted as the amount of data to be deleted each time a file to be deleted is performed. The second deleting submodule is used to delete the fixed amount of file data from the end of the file to be deleted, starting from the end and working backwards, when the amount of data to be deleted is greater than or equal to a fixed amount of data, where the fixed amount of data is the maximum amount of data that can be deleted from a file to be deleted in each deletion operation. The third deleting submodule is used to delete the entire file to be deleted when the amount of data to be deleted is less than the fixed amount of data.

[0092] Optionally, the above-mentioned update module further includes: a second determining unit, a first updating unit, a second updating unit, a third updating unit, and a fourth updating unit. The second determining unit is used to determine the target file based on the junk file records in the second file queue, wherein the target file is the file with the smallest file sequence number among all files to be deleted in the second file queue, and the file sequence number is used to represent the deletion order of the files to be deleted; the first updating unit is used to update the cursor information based on the target file to obtain target cursor information, wherein the target cursor information indicates that the target file is the first file to be deleted when the next deletion operation is performed; the second updating unit is used to update the data volume of the file corresponding to each junk file record in the second file queue to obtain the target data volume; the third updating unit is used to update the data volume of the file corresponding to each junk file record in the third file queue to 0; and the fourth updating unit is used to update the metadata information based on the target data volume and the target cursor information.

[0093] Optionally, the file processing device further includes a determining module and a first deleting module. The determining module is used to determine the junk file records to be deleted based on the target cursor information and the cursor information when updating the cursor information according to the target file to obtain target cursor information; the first deleting module is used to delete the junk file records to be deleted.

[0094] Optionally, the file processing device further includes: a zeroing module and a clearing module, wherein the zeroing module is used to zero out the total amount of deleted data and recalculate the total amount of deleted data after the next deletion operation is completed; the clearing module is used to clear the junk file records in the first file queue and the junk file records in the third file queue.

[0095] Optionally, the file processing device further includes a first detection module and a second deletion module. The first detection module is used to detect whether the distributed file system used to manage the database cluster has malfunctioned and restarted; the second deletion module is used to delete the remaining files to be deleted in the database cluster based on the target cursor information after the distributed file system has successfully restarted.

[0096] Optionally, the file processing device further includes: a first receiving module, a first determining module, a second detecting module, an acquiring module, a first updating module, and a prohibition module. The first receiving module is used to receive a file processing termination request, wherein the file processing termination request requests termination of the currently executing file processing request within a preset time period; the first determining module is used to determine a maximum waiting time based on the preset time period, wherein the maximum waiting time is less than the preset time period; the second detecting module is used to detect a first file to be deleted that has completed the deletion operation within the maximum waiting time period; the acquiring module is used to acquire the latest data volume of the first file to be deleted; the first updating module is used to update metadata information based on the latest data volume; and the prohibition module is used to prohibit the execution of new file processing requests.

[0097] Optionally, the file processing device further includes a third detection module and a generation module. The third detection module is used to detect whether the metadata information update operation has been completed within the maximum waiting time. The generation module is used to generate a first response message if the metadata information update operation has been completed within the maximum waiting time, wherein the first response message indicates that the file processing request has been completed.

[0098] Example 3

[0099] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program is configured to execute the file processing method in Embodiment 1 above when it is run.

[0100] Example 4

[0101] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are configured to run the programs, wherein the programs are configured to execute the file processing method in Embodiment 1 above when running.

[0102] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0103] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0105] The units described as separate components may or may not be physically separate. Similarly, the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0108] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A file processing method, characterized in that, include: Receive a file processing request, wherein the file processing request contains at least cursor information, the cursor information being used to identify the first file to be deleted in the database cluster; According to the preset deletion order, partial file data of at least one file to be deleted in the database cluster is deleted in parallel according to the cursor information. The amount of partial file data is less than or equal to the preset amount of data. When deleting each file to be deleted, the file data is deleted from the end of each file to be deleted. After each deletion operation is completed, the total amount of deleted data is counted, and when the total amount of deleted data is greater than or equal to a preset threshold, the metadata information is updated. The metadata information includes at least the latest target cursor information and the remaining target data amount of each file to be deleted. The target cursor information is used to represent the first file to be deleted when the next deletion operation is performed. The total amount of deleted data is the total amount of deleted data of multiple files deleted in parallel. Specifically, according to a preset deletion order, and based on cursor information, a parallel deletion method is used to delete partial file data of at least one file to be deleted in the database cluster, including: According to the preset deletion order, a preset number of files to be deleted are determined from the database cluster; If not all the file data of the file to be deleted has been deleted, the junk file record corresponding to the file to be deleted will be moved to the second file queue. If all the file data of the file to be deleted has been deleted, the junk file record corresponding to the file to be deleted is moved to the third file queue; When the total amount of deleted data is greater than or equal to a preset threshold, update the metadata information, including: The target file is determined based on the garbage file records in the second file queue, wherein the target file is the file with the smallest file number among all the files to be deleted corresponding to the second file queue, and the file number is used to represent the deletion order of the files to be deleted; The cursor information is updated according to the target file to obtain the target cursor information, wherein the target cursor information indicates that the target file is the first file to be deleted when the next deletion operation is performed; Update the data volume of the file corresponding to each garbage file record in the second file queue to obtain the target data volume; Update the data volume of the file corresponding to each garbage file record in the third file queue to 0; update the metadata information according to the target data volume and the target cursor information.

2. The method according to claim 1, characterized in that, According to a preset deletion order, and based on cursor information, partial file data of at least one file to be deleted in the database cluster is deleted using a parallel deletion method, including: The preset number of files to be deleted are deleted using a parallel deletion method. The deletion process includes at least one deletion operation, and the maximum amount of file data deleted in each deletion operation is the preset amount of data.

3. The method according to claim 2, characterized in that, The preset number of files to be deleted are deleted using a parallel deletion method, including: Each time a deletion operation is performed on the file to be deleted, it is detected whether there are any abnormalities in the file during the deletion process; If an anomaly occurs during the deletion process of the file to be deleted, the file to be deleted is determined to be an abnormal file; Move the garbage file record corresponding to the abnormal file to the first file queue.

4. The method according to claim 3, characterized in that, The preset number of files to be deleted are deleted using a parallel deletion method, including: Each time a deletion operation is performed on the file to be deleted, it is checked whether all the file data of the file to be deleted has been deleted.

5. The method according to claim 2, characterized in that, The preset number of files to be deleted are deleted using a parallel deletion method, including: Each time a file to be deleted is attempted to be deleted, the amount of file data that has not yet been deleted is determined as the amount of data to be deleted. When the amount of data to be deleted is greater than or equal to a fixed amount of data, the fixed amount of file data is deleted from the end of the file to be deleted, starting from the end of the file to be deleted. The fixed amount of data is the maximum amount of data that can be deleted from a file to be deleted in each deletion operation. When the amount of data to be deleted is less than the fixed amount of data, all files to be deleted will be deleted.

6. The method according to claim 1, characterized in that, The method further includes: When updating the cursor information based on the target file to obtain the target cursor information, the junk file records to be deleted are determined based on the target cursor information and the cursor information. Delete the junk file records to be deleted.

7. The method according to claim 3, characterized in that, After updating the metadata information, the method further includes: The total amount of deleted data is reset to zero, and the total amount of deleted data is recalculated after the next deletion operation is completed. Clear the junk file records in the first file queue and the third file queue.

8. The method according to claim 1, characterized in that, After updating the metadata information, the method further includes: The system checks whether the distributed file system used to manage the database cluster has malfunctioned and has been restarted. After the distributed file system restarts successfully, the remaining files to be deleted in the database cluster are deleted according to the target cursor information.

9. The method according to claim 1, characterized in that, The method further includes: Receive a file processing termination request, wherein the file processing termination request is used to request the termination of the currently executing file processing request within a preset time period; The maximum waiting time is determined according to a preset duration, wherein the maximum waiting time is less than the preset duration; Detect the first file to be deleted that has completed the deletion operation within the maximum waiting time; Get the latest data volume of the first file to be deleted; Update the metadata information according to the latest data volume; Prevent new file processing requests from being executed.

10. The method according to claim 9, characterized in that, After determining the maximum waiting time based on a preset duration, the method further includes: Detect whether the update operation of the metadata information has been completed within the maximum waiting time; If the update operation of the metadata information is completed within the maximum waiting time, a first response message is generated, wherein the first response message indicates that the file processing request has been completed.

11. A document processing device, characterized in that, include: A receiving module is used to receive a file processing request, wherein the file processing request contains at least cursor information, and the cursor information is used to identify the first file to be deleted in the database cluster; The deletion module is used to delete partial file data of at least one file to be deleted in the database cluster according to a preset deletion order and in parallel deletion mode based on cursor information. The amount of partial file data is less than or equal to a preset amount of data. When deleting each file to be deleted, the file data is deleted from the end of each file to the beginning. The update module is used to count the total amount of deleted data after each deletion operation is completed, and update the metadata information when the total amount of deleted data is greater than or equal to a preset threshold. The metadata information includes at least the latest target cursor information and the remaining target data amount of each file to be deleted. The target cursor information is used to represent the first file to be deleted when the next deletion operation is performed. The total amount of deleted data is the total amount of deleted data of multiple files deleted in parallel. The deletion module further includes: a first determining unit, used to determine a preset number of files to be deleted from the database cluster according to a preset deletion order; The device further includes: a second moving submodule, used to move the junk file record corresponding to the file to be deleted to a second file queue when the file data of the file to be deleted has not been completely deleted; and a third moving submodule, used to move the junk file record corresponding to the file to be deleted to a third file queue when the file data of the file to be deleted has been completely deleted. The update module further includes: a second determining unit, used to determine the target file based on the junk file records in the second file queue, wherein the target file is the file with the smallest file sequence number among all files to be deleted in the second file queue, and the file sequence number is used to represent the deletion order of the files to be deleted; a first updating unit, used to update the cursor information based on the target file to obtain target cursor information, wherein the target cursor information indicates that the target file is the first file to be deleted when the next deletion operation is performed; a second updating unit, used to update the data volume of the file corresponding to each junk file record in the second file queue to obtain the target data volume; a third updating unit, used to update the data volume of the file corresponding to each junk file record in the third file queue to 0; and a fourth updating unit, used to update the metadata information based on the target data volume and the target cursor information.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the file processing method according to any one of claims 1 to 10 when it is run.

13. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the file processing method according to any one of claims 1 to 10.

Images