A small file merging space recovery method based on distributed logs

Abstract

This invention proposes a small file merging space reclamation method based on distributed logs. When a small file is overwritten or deleted, the master node determines the affected aggregation object and range using index information and generates a space reclamation log to record the space to be reclaimed. This log is then synchronized to slave nodes and filtered and organized by a background thread. Space reclamation operations are performed at the aggregation object granularity, including performing data zeroing operations to release the range, and updating progress information after reclamation is complete. To address node failures, this invention also includes a log anomaly recovery process, ensuring that the storage process with the most complete operation log is selected as the authoritative storage process, and recovery is based on its log to guarantee data consistency. This method, through distributed logs and background thread processing, improves the space reclamation efficiency after small file merging, optimizes storage resource utilization, reduces costs, and enhances system stability and reliability.

Description

Technical Field

[0001] The invention relates to distributed storage technology, and in particular to a small file merging space recovery method based on distributed logs. Background Art

[0002] In the era of information explosion, the data generated every day not only has a huge storage capacity, but also has the remarkable characteristics of various data types, large data size changes, and fast flow, which often generates a large number of small files in the tens of millions, hundreds of millions, even billions, and tens of billions. The current file systems, including local file systems, distributed file systems, and object storage systems, are all designed for large files, such as XFS / EXT4, GlusterFS, GFS, and HDFS. In terms of metadata management, data layout, cache management and other implementation strategies, they focus on large files, while the performance and storage efficiency of massive small file applications will be greatly reduced.

[0003] In the scenario of storing a large number of small files, in a common Linux file system, reading a file involves three disk IOs: first, reading the directory metadata into memory, then loading the file's inode node into memory, and finally reading the actual file content. Due to the large number of small files, it is impossible to cache all the directory and file inode information into memory, so the number of disk IOs is difficult to achieve the ideal state of only one disk IO per image read.

[0004] For LOSF storage, IOPS / OPS is a key performance indicator. The main reasons for the reduction of performance and storage efficiency include metadata management, data layout and IO management, cache management, network overhead, etc. From the theoretical analysis of LOSF optimization time, optimization should be carried out from metadata management, cache mechanism, small file merging, etc. Among the existing solutions, small file merging storage is currently the most successful strategy for optimizing LOSF problems, and some distributed storage systems have adopted it. It uses multiple logical files to share the same physical file, merges multiple small files into a large file, and realizes efficient small file storage.

[0005] In distributed storage, in order to improve read and write performance, it is usually supported to set a storage resource pool as the cache layer (distributed cache) of another storage resource pool. The logical pool used as the cache layer is called the cache pool, and the logical pool that actually stores data is called the data pool.

[0006] The technical goal of distributed caching is to place hot data with high user access frequency in high-performance, small-capacity storage media (such as Cachepool created based on NVME SSD), and place a large amount of cold data in large-capacity storage media (such as Datapool created based on HDD). Distributed caching improves hot data access performance while reducing storage costs.

[0007] Distributed cache allows cold data to be freely and safely migrated to lower-level storage media (data pool), thus saving storage costs; hot data can be automatically migrated from the lower level (data pool) to the higher level storage layer (cachetier), thus improving the performance of accessing hot data. Common implementation solutions include: Figure 1 shown.

[0008] The online merging solution for small files based on distributed cache is to pre-allocate cached large objects. When receiving a file write request from the client, the small file is written to the cache object of the cache pool in an appended manner. When the cache object size meets a certain threshold, subsequent requests are switched to another cache object, and the full cache object is flushed back to the backend pool, such as Figure 2 shown.

[0009] The main steps of online merging are as follows:

[0010] 1. Each storage process has a fixed number of index objects, which are used to record the mapping index information between client files and cache objects, and the metadata information of client objects;

[0011] 2. After receiving the write operation of the client object, the main process converts it into a write operation on the current cache object. While writing the data, it stores the mapping relationship between the client object and the cache object, as well as the metadata information of the client object in the corresponding index object;

[0012] 3. The master process synchronizes the data writing operation and the operation log corresponding to this operation to each slave process; the operation log contains information such as the cache object and client object of this operation;

[0013] 4. When the cache object meets the flushing conditions, the data of the cache object is flushed back to the backend pool and the index information corresponding to the cache object is updated.

[0014] CN202010366322.3 discloses a space recovery method, device, storage medium and processor. The method includes:

[0015] (1) Based on the recycling scan, the statistical counting objects of the predetermined index pool are traversed to determine the space usage data of the aggregated objects in the statistical counting objects;

[0016] (2) Determine the reclaimable space of the data pool associated with the predetermined index pool based on the space usage data;

[0017] (3) Reclaim the reclaimable space in the data pool according to the preset space reclamation rules.

[0018] When reclaiming space, this method needs to rely on the space usage data information recorded in the statistical counting object. In the IO process, in addition to writing data to the aggregation object, it is also necessary to simultaneously update the space usage data information recorded in the statistical counting object. Multiple objects need to be operated in one IO, which brings complex link processes and performance overhead to the atomicity guarantee of IO. The introduction of statistical counting objects will also bring the overhead of frequent access and update mutual exclusion of such objects.

[0019] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0020] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a small file merging space recovery method based on distributed logs.

[0021] To achieve the above object, the present invention adopts the following technical solutions:

[0022] A small file merging space recovery method based on distributed logs, comprising:

[0023] S1. Small file overwrite processing: The master node receives a small file overwrite request, uses the index information of the small file and the aggregate object to determine the overwritten aggregate object and interval range; generates a space recovery log to record the space information to be recovered;

[0024] S2. Small file deletion processing: The master node receives a small file deletion request, uses index information to determine the aggregation object and interval range where the deleted small file is located; generates a space recovery log to record the space information to be recovered;

[0025] S3. Space recovery background thread processing: The master node synchronizes the space recovery log to the slave node, and each node saves the log information locally; the space recovery thread background screens the space recovery log, sorts it according to the granularity of the aggregate object, and sends it to the thread pool; the thread pool performs data clearing operations to release the recyclable interval of the aggregate object; after completing the space recovery, delete the corresponding space recovery log key-value pair and update the progress information of the space recovery;

[0026] S4. Abnormal recovery processing of space recovery log: When the storage process fails and recovers, the main storage process triggers the operation log recovery processing and selects the storage process with the most complete operation log as the authoritative storage process; the main storage process takes the operation log of the authoritative storage process as the basis, synchronizes and applies the operation log to the local according to the version number, and if the operation log of the version number corresponds to the space recovery log, the synchronous update of the space recovery log is triggered. After the main storage process and the authoritative storage process reach an agreement, the main storage process pushes the operation log and space recovery log to each slave storage process, thereby completing the abnormal recovery of the operation log and space recovery log.

[0027] Furthermore, step S1 specifically includes:

[0028] S11. After the main storage process receives the small file write operation request, it obtains the index metadata information of the small file from the database, and obtains the mapping relationship between the written interval and the corresponding aggregation object through the index information of the small file;

[0029] S12. Determine whether this write operation is an overwrite. If so, compare the index with the interval to be written this time, calculate the interval of the small file to be overwritten in the aggregate object, and generate a space recovery log to record the overwritten interval information; associate the space recovery log with the version number of this operation log, and mark it in the operation log to indicate whether there is a corresponding space recovery log for the current operation log version;

[0030] S13. The master storage process packages the operation log and space recovery log of this overwrite operation into the metadata of this overwrite operation, then writes the packaged data to the local data disk, writes the metadata to the local database, and synchronizes the data to the slave storage process; after both the master and slave storage processes have completed the data and metadata write to the disk, the master storage process returns a confirmation to the client that the overwrite operation is complete;

[0031] S14. Run the space recovery thread in the background of the main storage process, regularly scan and obtain the space recovery log from the database, then sort it according to the aggregate object granularity, construct the space recovery operation of the aggregate object granularity, and send it to the space recovery thread pool queue;

[0032] S15. The space recovery thread pool processes the space recovery operation of the aggregate object, performs data clearing operation on the covered interval of the aggregate object to recover space, and deletes the aggregate object if all data of the aggregate object is cleared; the master storage process also sends the data clearing or deletion operation required on the aggregate object to each slave storage process for synchronous processing. After the master and slave storage processes have completed the processing, the space recovery is completed.

[0033] Furthermore, step S2 specifically includes:

[0034] S21. After the main storage process receives a small file deletion request, it obtains the index metadata information of the small file from the database, and obtains the interval where the small file has been written and the corresponding mapping relationship on the aggregate object through the index information of the small file;

[0035] S22. Encode the interval information of the small file mapped on the aggregate object into a space recovery log, which has the same version number as the operation log of the deletion operation, and mark it in the operation log to indicate whether there is a corresponding space recovery log for the current operation log version;

[0036] S23. The master storage process determines whether the space recovery involved in this deletion operation is for a single aggregate object or multiple aggregate objects, and performs the corresponding space recovery operation: if it is for a single aggregate object, directly construct a data clearing operation to recover the space; if it is for multiple aggregate objects, encapsulate the space information to be recovered into a space recovery log, and write it to the database together with the operation log and the metadata of the deletion transaction for persistence; at the same time, the master storage process synchronizes the space recovery operation involved to each slave storage process, and after both the master and slave storage processes have completed the metadata write to the disk, returns a confirmation of the completion of the deletion operation to the client;

[0037] S24. Run the space recovery thread in the background of the main storage process, regularly scan and obtain the key-value pairs of the space recovery log from the database, then sort them according to the aggregate object granularity, construct the space recovery operation of the aggregate object granularity, and send it to the space recovery thread pool queue;

[0038] S25. The space recovery thread pool processes the space recovery operation of the aggregate object, performs data clearing operation on the deleted interval of the aggregate object to recover space. If all the data of the aggregate object is cleared, the aggregate object is deleted; the master storage process also sends the data clearing or deletion operation required on the aggregate object to each slave storage process for synchronous processing. After the master and slave storage processes have completed the processing, the space recovery is completed.

[0039] Furthermore, step S3 specifically includes:

[0040] S31. The space recovery background screening thread periodically obtains a batch of key-value pairs of space recovery log prefixes from the database and parses them into space recovery logs;

[0041] S32. The parsed space recovery logs are integrated according to different aggregation objects to determine the aggregation objects and corresponding interval ranges involved in the batch of space recovery logs;

[0042] S33. According to the integration results, a space recovery operation is constructed according to the granularity of the aggregated object, and the operation is placed in the space recovery thread pool queue;

[0043] S34. Multiple threads in the space recovery thread pool take out the space recovery operations in the queue, parse the corresponding aggregate object and the interval range to be recovered, perform data clearing operation on the aggregate object, that is, mark the specified interval range as invalid, and return the interval to the disk space allocator to complete the space recovery of the interval range on the aggregate object; if the space recovery interval this time contains all valid data intervals of the aggregate object, delete the aggregate object and release its data space and metadata space;

[0044] S35. After completing the space recovery operation for the batch of space recovery logs, delete the corresponding space recovery log key-value pair in the database, and update the record of the maximum version number of the last space recovery operation to the maximum version number of the current batch of space recovery logs to be recovered, so that the next time the space recovery thread filters out the space recovery logs with version numbers greater than the maximum version number of this batch of space recovery logs from the database for processing.

[0045] Furthermore, step S4 specifically includes:

[0046] S41. When the storage process is started, the operation log and the space recovery log are loaded from the database, and a specified number of space recovery logs are saved in the memory cache for direct hits from the memory during subsequent recovery;

[0047] S42. During the storage process recovery phase, the main storage process compares the operation logs of each storage process at the granularity of the basic storage unit, selects the storage process with the most complete operation logs as the authoritative storage process, and then redoes the missing operations of each storage process based on the operation logs of the authoritative storage process to restore the consistency of each object during the period when the failed storage process was offline;

[0048] S43 confirms that the authoritative storage process of the space recovery log is consistent with the authoritative storage process of the operation log. In the operation log recovery phase, it is determined according to the mark in the operation log whether there is a corresponding space recovery log. If so, the recovery of the space recovery log is triggered;

[0049] S44. The main storage process pulls the space recovery log on the authoritative storage process, and trims and merges the space recovery log valid interval range on each storage process to complete the update of the local space recovery log;

[0050] S45. After the master storage process completes the update of the local operation log and the space recovery log, it pushes the update to each slave storage process, and each slave storage process applies the update to the local, thereby achieving data consistency between the master and slave storage processes.

[0051] Furthermore, step S44 specifically includes the following space recovery log recovery process between the main storage process and the authoritative storage process:

[0052] After the failure is recovered, the primary storage process determines whether there is an intersection between the space recovery log and the authoritative storage process;

[0053] If there is no intersection, the master storage process deletes the existing space recovery log, and the master storage process synchronizes the space recovery log from the authoritative storage process to achieve log consistency;

[0054] If there is an intersection, the main storage process deletes the space recovery logs between the version number of the space recovery operation last completed locally and the version number of the space recovery operation last completed by the authoritative storage process; the main storage process pulls the space recovery logs from the authoritative storage process from the latest local space recovery operation version number to the latest space recovery operation version number of the authoritative storage process; the main storage process merges the pulled space recovery logs with the local logs to complete the synchronization of the space recovery logs.

[0055] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the small file merging and space recovery method based on distributed logs.

[0056] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the distributed log-based small file merging space recovery method is implemented.

[0057] The present invention has the following beneficial effects:

[0058] The present invention provides a small file merging space recovery method based on distributed logs. When deletion or overwriting occurs in a small file scenario in distributed storage, the corresponding small file deleted or overwritten interval is quickly found in the aggregated large file to achieve efficient space recovery. In the small file merging scenario, the present invention records space recovery logs in the small file overwriting and deletion scenarios, batch acquires the space recovery logs in the space recovery thread background, and performs space recovery operations based on the aggregate object granularity, which efficiently solves the space recovery problem under small file merging.

[0059] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of a commonly used distributed cache implementation solution.

[0061] Figure 2 Schematic diagram of the online merging solution for small files based on distributed cache.

[0062] Figure 3This is the overall process of space recovery log recovery in an embodiment of the present invention.

[0063] Figure 4 The space recovery logs of the authoritative storage process and the main storage process in the embodiment of the present invention have no intersection.

[0064] Figure 5 The space recovery logs of the authoritative storage process and the main storage process of the embodiment of the present invention have an intersection.

[0065] Figure 6 The figure is an overall flow chart of a method for merging and recovering small files based on distributed logs according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0067] Definitions of Abbreviations and Key Terms:

[0068] Storage process: The management process of a single disk in distributed storage.

[0069] Index: Small file merge metadata index, used to record the mapping relationship between small files and aggregate objects.

[0070] Basic storage unit: represents a collection of a part of objects. A storage process has multiple basic storage units.

[0071] Operation log: An operation log recorded at the granularity of a basic storage unit, which records the IO operation records of all objects under the basic storage unit.

[0072] Space recovery log: records the space recovery information of all objects under the basic storage unit.

[0073] See also Figure 6 The embodiment of the present invention provides a small file merging space recovery method based on distributed logs, comprising:

[0074] (1) Small file overwrite processing: The master node receives a small file overwrite request and uses the index information of the small file and the aggregate object to determine the aggregate object and interval range to be overwritten; it generates a space recovery log to record the space information to be recovered.

[0075] In this scenario, it means that the small file has been written before, and the small file was merged and appended to a certain aggregate object. Then this time, the previously written interval of the small file is overwritten. At this time, it is necessary to accurately find the aggregate object where the overwritten interval is located (which may correspond to multiple aggregate objects and multiple intervals) and reclaim the space.

[0076] (2) Small file deletion processing: The master node receives a small file deletion request and uses index information to determine the aggregation object and interval range where the deleted small file is located; it generates a space recovery log to record the space information to be recovered.

[0077] In this scenario, it means that the small file has been written before and is deleted this time. At this time, it is also necessary to accurately find all the aggregation objects and corresponding interval ranges where the small file is located and reclaim the space.

[0078] (3) Space recovery background thread processing: The master node synchronizes the space recovery log to the slave node, and each node stores the log information locally; the space recovery thread background screens the space recovery log, organizes it according to the granularity of the aggregate object, and sends it to the thread pool; the thread pool performs the data clearing operation to release the recyclable range of the aggregate object; after the space recovery is completed, the corresponding space recovery log key-value pair is deleted and the progress information of the space recovery is updated.

[0079] The present invention uses a distributed log to record space recovery related information (referred to as space recovery log), and the master and slave nodes perform space recovery processing based on the space recovery logs of their own nodes.

[0080] When performing space recovery, it involves operating multiple aggregate objects in the transaction to release space. In order to ensure the atomicity of a single space recovery operation and minimize the impact of the space recovery operation on the foreground business io, the present invention uses a space recovery background thread to perform space recovery.

[0081] (4) Abnormal recovery of space recovery logs: When a storage process fails and recovers, the main storage process triggers the operation log recovery process and selects the storage process with the most complete operation logs as the authoritative storage process. The main storage process uses the operation logs of the authoritative storage process as the basis, synchronizes and applies the operation logs to the local computer in order according to the version number. If the operation log with the version number corresponds to a space recovery log, the synchronization update of the space recovery log is triggered. After the main storage process and the authoritative storage process reach an agreement, the main storage process pushes the operation logs and space recovery logs to each slave storage process, thereby completing the abnormal recovery of the operation logs and space recovery logs.

[0082] When the storage process of the master and slave nodes fails, the space recovery logs on the master and slave nodes will differ. When the storage process of the failed node is started, the space recovery logs need to be recovered abnormally to ensure that the space recovery information and space recovery progress of the abnormal node are restored to the same level as those of the normal node.

[0083] The present invention provides an efficient small file merging space recovery method, which is particularly suitable for processing the space recovery problem in a massive small file storage environment. In traditional storage systems, the existence of a large number of small files will lead to a significant decrease in performance and storage efficiency, because these systems are usually optimized for large files, and the implementation strategies of metadata management, data layout and cache mechanism of small files are not efficient. The present invention records space recovery logs when small files are overwritten and deleted, and uses background threads to batch process these logs, thereby realizing the rapid positioning and effective recovery of small file intervals that are deleted or overwritten in aggregated large files. The present invention utilizes a distributed logging mechanism to avoid the complexity and performance overhead of updating space usage data caused by relying on statistical counting objects, thereby improving the efficiency of space recovery. In a distributed storage environment, the method ensures the atomicity of space recovery operations and the consistency of data through the collaborative work of master and slave nodes. In this way, even in the event of a node failure, the most authoritative operation log can be restored by comparing the operation logs of each node, thereby ensuring that the system can quickly recover from an abnormal state and continue effective space recovery operations. In summary, the present invention not only improves the performance and storage efficiency of small file storage through an innovative space recovery mechanism, but also optimizes the use of storage resources, and realizes efficient space recovery operations through distributed logging and background thread processing technology.

[0084] Specific embodiments of the present invention are further described below.

[0085] Small file overwrite process

[0086] In this scenario, when a small file write io arrives, the overwritten interval of the small file and the corresponding aggregation object can be obtained through the index metadata information of the small file. The overwritten interval information is recorded in the space recovery log in the overwrite transaction. Later, the space recovery log, the operation log of this overwrite transaction, and the metadata key-value pair of this overwrite transaction are packaged together and written to the database for persistence.

[0087] The specific processing flow of the overwrite scenario is as follows:

[0088] (1) After the main storage process receives a write operation for a small file, it first obtains the index metadata information of the small file from the database. Through the index information of the small file, it can obtain the interval where the small file has been written and the corresponding mapping relationship on the aggregate object. Then, it compares the interval to be written by the small file this time to determine whether the write operation is an overwrite.

[0089] (2) If it is an overwrite, compare the index and the interval to be written this time, calculate the interval of the small file that is overwritten in the aggregate object, and encapsulate the overwritten interval into a space recovery log. The space recovery log has the same version number as the operation log of this operation. Then, record a mark in the operation log to indicate whether there is a space recovery log corresponding to the current version of the operation log. This is used to quickly index the space recovery log during subsequent abnormal recovery.

[0090] (3) The master storage process packages the operation log and space recovery log of this overwrite operation into the metadata of this overwrite operation, and then writes the packaged data to the local data disk and the metadata to the local database. It also sends it to each slave storage process to write to the disk. After the master and slave storage processes have completed the data and metadata writing to the disk, the master storage process will return the overwrite io completion to the upper-level client.

[0091] (4) A space recovery thread will run in the background inside the main storage process to handle space recovery transactions. This thread periodically scans and obtains a batch of space recovery logs from the database, then organizes them based on the granularity of aggregate objects, constructs a space recovery op at the granularity of aggregate objects, and sends it to the space recovery thread pool queue.

[0092] (5) The space recovery thread pool on the master storage process handles the space recovery op of the aggregate object, clears the data in the interval corresponding to the aggregate object to recover the space. If all the data on the aggregate object has been cleared, the aggregate object is directly deleted. The master storage process also sends the data clearing or deletion operations that need to be performed on the aggregate object to each slave storage process for synchronous processing. When the master and slave storage processes have completed the processing, the space recovery of the aggregate object is completed.

[0093] Small file deletion process

[0094] In this scenario, the index metadata information of the small file can be used to obtain the aggregate object and corresponding interval range that have been written to the small file before. The deleted interval information is recorded in the space recovery log in the deletion transaction. After that, the space recovery log, the operation log of this deletion transaction, and the metadata key-value pair of this deletion transaction are packaged together and written to the database for persistence.

[0095] The specific process of deleting a scene is as follows:

[0096] (1) After the main storage process receives a deletion op for a small file, it first obtains the index metadata information of the small file. Through the index information of the small file, it can obtain the interval where the small file has been written and the corresponding mapping relationship on the aggregate object, and then encode its interval on the aggregate object into a space recovery log.

[0097] (2) The space recovery log has the same version number as the operation log of this operation. Then a mark is recorded in the operation log to indicate whether there is a space recovery log corresponding to the current version of the operation log, so that the space recovery log can be quickly indexed during subsequent abnormal recovery.

[0098] (3) The master storage process determines whether this deletion operation involves one or more aggregate objects for space recovery. If it involves only one, it directly constructs a data clearing operation for the aggregate object to perform space recovery. If all data on the aggregate object has been cleared, the aggregate object is directly deleted. The master storage process also sends the data clearing or deletion operation that needs to be performed on the aggregate object to each slave storage process for synchronous processing; if it involves multiple aggregate objects, the master storage process encapsulates the space information to be recovered on each aggregate object into a space recovery log, and then packages the space recovery log and the operation log of this write transaction, as well as the metadata key-value pair of this deletion transaction, and writes them to the metadata database for persistence. At the same time, it also sends them to each slave storage process for disk storage. After the master and slave storage processes have completed the metadata disk storage, the master storage process returns a response to the upper-level client indicating that the overwrite IO is completed.

[0099] (4) A space recovery thread will run in the background inside the main storage process to handle space recovery transactions. This thread periodically scans a batch of key-value pairs of space recovery logs from the database, organizes them at the granularity of aggregate objects, constructs a space recovery op at the granularity of aggregate objects, and sends it to the space recovery thread pool queue.

[0100] (5) The space recovery thread pool on the master storage process handles the space recovery op of the aggregate object, clears the data in the interval corresponding to the aggregate object to recover the space. If all the data on the aggregate object has been cleared, the aggregate object is directly deleted. The master storage process also sends the data clearing or deletion operations that need to be performed on the aggregate object to each slave storage process for synchronous processing. When the master and slave storage processes have completed the processing, the space recovery of the aggregate object is completed.

[0101] Space Reclamation Background Thread Processing Flow

[0102] In the io process, the space released when overwriting and deleting small files is recorded in the space recovery log, and then processed by the space recovery background thread, which mainly includes the space recovery background screening thread and the corresponding space recovery background thread pool. The specific process is as follows:

[0103] (1) The space recovery thread periodically obtains a batch of key-value pairs of space recovery log prefixes from the database and parses them into space recovery logs.

[0104] (2) The parsed space recovery logs are integrated according to the space recovery logs of different aggregate objects. After the integration, the space recovery of the aggregate objects involved in this batch of space recovery logs and the corresponding interval range can be determined.

[0105] (3) According to the aggregate object granularity, construct the space recovery op operation and put it into the space recovery thread pool queue.

[0106] (4) Multiple threads in the space recovery thread pool take out the space recovery op from the queue according to the basic storage unit mapping relationship, parse the corresponding aggregate object and the range to be recovered, and perform data zeroing on the aggregate object, that is, invalidate the specified range mark, and return this range to the disk space allocator to complete the space recovery of this range on the aggregate object. If the space recovery range is all valid data ranges of the aggregate object, the space recovery directly deletes the aggregate object and releases its data space and metadata space.

[0107] (5) When the space recovery operation for this batch of space recovery logs is completed, the key-value pair records corresponding to this batch of space recovery logs in the database are deleted, and the maximum version number of the space recovery logs that have been recovered is updated to the maximum version number of the current batch of space recovery logs to be recovered. In this way, the next time the space recovery thread traverses and selects the space recovery logs with a version number greater than the maximum version number of the recovered space recovery logs from the database, and then puts them into the space recovery thread pool queue for the next round of space recovery processing.

[0108] Abnormal recovery process of space recovery log

[0109] The space recovery log is a distributed log. When an overwrite or deletion operation is performed on the master and slave storage processes, the corresponding space recovery log will be persisted to the database of this node. Therefore, in some failure scenarios, the space recovery logs between the master and slave storage processes may be different. Therefore, it is necessary to recover the space recovery log in the event of a storage process failure.

[0110] The overall process of space recovery log recovery

[0111] When a storage process fails and is pulled up again, it will trigger the basic storage unit carried by the storage process to enter the recovery process. This process mainly compares the operation logs on each storage process where each basic storage unit is located, and selects the storage process with the most complete operation log entries as the authoritative storage process for the recovery process of the basic storage unit. Then, the subsequent recovery process is carried out based on the operation logs and space recovery logs of the authoritative storage process.

[0112] like Figure 3 As shown in the figure, when the storage process is started, the specific processing flow is as follows:

[0113] (1) When the storage process starts, the operation log and space recovery log are loaded, and the specified number of space recovery logs are obtained from the database and saved in the memory cache for direct hits from the memory during subsequent recovery.

[0114] (2) During the recovery phase of the storage process, the main storage process compares the operation logs of each storage process according to the granularity of the basic storage unit, selects the storage process with the most complete authoritative log as the authoritative storage process, and then redoes some missing operations on each storage process according to the operation log of the authoritative storage process, so as to restore the operations on each object to a consistent state during the period when the failed storage process was offline.

[0115] (3) Given that the version number of the space recovery log and the operation log correspond to each other, and that aggregate objects are operated when space recovery is performed, corresponding operation logs are also generated at this time, so the authoritative storage process of the space recovery log is consistent with the authoritative storage process of the operation log. During the operation log recovery phase, based on the mark recorded in the operation log, it can be determined whether the version number has a corresponding space recovery log. If so, the recovery of the corresponding space recovery log is triggered.

[0116] (4) The effective interval range of the space recovery log on each storage process is [last_gc, gc_last_update], where last_gc indicates the version number of the storage process's last space recovery operation, and gc_last_update indicates the current latest space recovery log version number. The space recovery log recovery process is also based on the space recovery log on the authoritative storage process. The primary storage process pulls the space recovery log of the authoritative storage process and applies it locally to complete the pruning and merging of the local space recovery log.

[0117] (5) After the master storage process completes the update of the local operation log and space recovery log, it pushes the operation log and space recovery log to each slave storage process. Each slave storage process applies the operation log and space recovery log locally to make the data between the master and slave storage processes consistent.

[0118] Space recovery log recovery process between the master storage process and the authoritative storage process

[0119] After the primary storage process obtains the space recovery log of the authoritative storage process, it is restored based on the space recovery log of the authoritative storage process. Different scenarios are distinguished based on whether the space recovery log of the authoritative storage process and the primary storage process have intersections. There are mainly the following two scenarios.

[0120] (1) The space recovery logs of the authoritative storage process and the main storage process have no intersection

[0121] like Figure 4 As shown in the figure, after the main storage process fails briefly, the authoritative storage process reclaims a large amount of space through the space recovery log. When the authoritative storage process reclaims this part of the space, the corresponding data clearing / remove operation log will also be generated in the operation log. Then, the space recovery op that the main storage process missed on the aggregate object can be automatically synchronized after the aggregate object is restored. Therefore, the existing space recovery log of the main storage process is no longer needed. After deleting the existing space recovery log, the space recovery log of the authoritative storage process is directly synchronized.

[0122] (2) The space recovery logs of the authoritative storage process and the main storage process overlap

[0123] like Figure 5 As shown in the figure, after the main storage process fails, the authoritative storage process handles part of the space recovery by consuming the space recovery log. However, if the space recovery logs of both parties still overlap, the main storage process only needs to delete the space recovery logs between the local last_gc and the last_gc of the authoritative storage process, and then pull the space recovery logs of the authoritative storage process with version numbers between the local gc_last_update and the gc_last_update of the authoritative storage process to merge them, and the synchronization of the space recovery logs can be completed.

[0124] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0125] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0126] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0127] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0128] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0129] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0130] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0131] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0132] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0133] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0134] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0135] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0136] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A small file merging space recovery method based on distributed logs, characterized in that: include: S1. Small file overwrite processing: The master node receives a small file overwrite request and uses the index information of the small file and the aggregate object to determine the overwritten aggregate object and interval range; Generate a space recovery log to record the space information to be recovered; S2. Small file deletion processing: The master node receives a small file deletion request, uses index information to determine the aggregation object and interval range where the deleted small file is located; generates a space recovery log to record the space information to be recovered; S3. Space recovery background thread processing: The master node synchronizes the space recovery log to the slave node, and each node saves the log information locally; the space recovery thread background screens the space recovery log, sorts it according to the granularity of the aggregate object, and sends it to the thread pool; the thread pool performs data clearing operations to release the recyclable interval of the aggregate object; after completing the space recovery, delete the corresponding space recovery log key-value pair and update the progress information of the space recovery; S4. Abnormal recovery of space recovery log: When the storage process fails and recovers, the main storage process triggers the operation log recovery process and selects the storage process with the most complete operation log as the authoritative storage process; the main storage process takes the operation log of the authoritative storage process as the basis, synchronizes and applies the operation log to the local according to the version number, and if the operation log of the version number corresponds to the space recovery log, the synchronization update of the space recovery log is triggered. After the main storage process and the authoritative storage process reach an agreement, the main storage process pushes the operation log and space recovery log to each slave storage process, thereby completing the abnormal recovery of the operation log and space recovery log; Step S1 specifically includes: S11. After the main storage process receives the small file write operation request, it obtains the index metadata information of the small file from the database, and obtains the mapping relationship between the written interval and the corresponding aggregation object through the index information of the small file; S12. Determine whether this write operation is an overwrite operation. If so, compare the index with the interval to be written this time, calculate the interval of the small file to be overwritten in the aggregate object, and generate a space recovery log to record the overwritten interval information; Associating the space recovery log with the current operation log version number and marking it in the operation log to indicate whether there is a corresponding space recovery log for the current operation log version; S13. The master storage process packages the operation log and space recovery log of this overwrite operation into the metadata of this overwrite operation, then writes the packaged data to the local data disk, writes the metadata to the local database, and synchronizes the data to the slave storage process; after both the master and slave storage processes have completed the data and metadata write to the disk, the master storage process returns a confirmation to the client that the overwrite operation is complete; S14. Run the space recovery thread in the background of the main storage process, regularly scan and obtain the key-value pairs of the space recovery log from the database, then sort them according to the aggregate object granularity, construct the space recovery operation of the aggregate object granularity, and send it to the space recovery thread pool queue; S15. The space recovery thread pool processes the space recovery operation of the aggregate object, performs data clearing operation on the covered interval of the aggregate object to recover space, and deletes the aggregate object if all data of the aggregate object is cleared; the master storage process also sends the data clearing or deletion operation required on the aggregate object to each slave storage process for synchronous processing. After the master and slave storage processes have completed the processing, the space recovery is completed.

2. The method for merging small files and recovering space based on distributed logs according to claim 1, characterized in that: Step S2 specifically includes: S21. After the main storage process receives a small file deletion request, it obtains the index metadata information of the small file from the database, and obtains the interval where the small file has been written and the corresponding mapping relationship on the aggregate object through the index information of the small file; S22. Encode the interval information of the small file mapped on the aggregate object into a space recovery log, which has the same version number as the operation log of the deletion operation, and mark it in the operation log to indicate whether there is a corresponding space recovery log for the current operation log version; S23. The master storage process determines whether the space recovery involved in this deletion operation is for a single aggregate object or multiple aggregate objects, and performs the corresponding space recovery operation: if it is for a single aggregate object, directly construct a data clearing operation to recover the space; if it is for multiple aggregate objects, encapsulate the space information to be recovered into a space recovery log, and write it to the database together with the operation log and the metadata key-value pair of the deletion transaction for persistence; at the same time, the master storage process synchronizes the space recovery operation involved to each slave storage process, and after the master and slave storage processes have completed the metadata write to the disk, returns the confirmation of the completion of the deletion operation to the client; S24. Run the space recovery thread in the background of the main storage process, regularly scan and obtain the key-value pairs of the space recovery log from the database, then sort them according to the aggregate object granularity, construct the space recovery operation of the aggregate object granularity, and send it to the space recovery thread pool queue; S25. The space recovery thread pool processes the space recovery operation of the aggregate object, performs data clearing operation on the deleted interval of the aggregate object to recover space. If all the data of the aggregate object is cleared, the aggregate object is deleted; the master storage process also sends the data clearing or deletion operation required on the aggregate object to each slave storage process for synchronous processing. After the master and slave storage processes have completed the processing, the space recovery can be completed.

3. The small file merging space recovery method based on distributed logs according to claim 1 is characterized in that: Step S3 specifically includes: S31. The space recovery background screening thread periodically obtains a batch of key-value pairs of space recovery log prefixes from the database and parses them into space recovery logs; S32. The parsed space recovery logs are integrated according to different aggregation objects to determine the aggregation objects and corresponding interval ranges involved in the batch of space recovery logs; S33. According to the integration results, a space recovery operation is constructed according to the granularity of the aggregated object, and the operation is placed in the space recovery thread pool queue; S34. Multiple threads in the space recovery thread pool take out the space recovery operations in the queue, parse the corresponding aggregate object and the interval range to be recovered, perform data clearing operation on the aggregate object, that is, mark the specified interval range as invalid, and return the interval to the disk space allocator to complete the space recovery of the interval range on the aggregate object; if the space recovery interval this time contains all valid data intervals of the aggregate object, delete the aggregate object and release its data space and metadata space; S35. After completing the space recovery operation for the batch of space recovery logs, delete the corresponding space recovery log key-value pair in the database, and update the record of the maximum version number of the last space recovery operation to the maximum version number of the current batch of space recovery logs to be recovered, so that the next time the space recovery thread filters out the space recovery logs with version numbers greater than the maximum version number of this batch of space recovery logs from the database for processing.

4. The method for merging small files and recovering space based on distributed logs according to claim 1, characterized in that: Step S4 specifically includes: S41. When the storage process is started, the operation log and the space recovery log are loaded from the database, and a specified number of space recovery logs are saved in the memory cache for direct hits from the memory during subsequent recovery; S42. During the storage process recovery phase, the main storage process compares the operation logs of each storage process at the granularity of the basic storage unit, selects the storage process with the most complete operation logs as the authoritative storage process, and then redoes the missing operations of each storage process based on the operation logs of the authoritative storage process to restore the consistency of each object during the period when the failed storage process was offline; S43 confirms that the authoritative storage process of the space recovery log is consistent with the authoritative storage process of the operation log. In the operation log recovery phase, it is determined according to the mark in the operation log whether there is a corresponding space recovery log. If so, the recovery of the space recovery log is triggered; S44. The main storage process pulls the space recovery log on the authoritative storage process, and trims and merges the space recovery log valid interval range on each storage process to complete the update of the local space recovery log; S45. After the master storage process completes the update of the local operation log and the space recovery log, it pushes the update to each slave storage process, and each slave storage process applies the update to the local, so that data consistency can be achieved between the master and slave storage processes.

5. The method for merging small files and recovering space based on distributed logs as claimed in claim 4, characterized in that: Step S44 specifically includes the following space recovery log recovery process between the main storage process and the authoritative storage process: After the failure is recovered, the primary storage process determines whether there is an intersection between the space recovery log and the authoritative storage process; If there is no intersection, the main storage process deletes the existing space recovery log, and the main storage process synchronizes the space recovery log from the authoritative storage process to achieve log consistency; If there is an intersection, the main storage process deletes the space recovery logs between the version number of the last space recovery operation completed locally and the version number of the space recovery operation completed by the authoritative storage process; the main storage process pulls the space recovery logs from the latest local space recovery operation version number to the latest space recovery operation version number of the authoritative storage process on the authoritative storage process; the main storage process merges the pulled space recovery logs with the local logs to complete the synchronization of the space recovery logs.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the small file merging space recovery method based on distributed logs is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the small file merging space recovery method based on distributed logs is implemented.

Images