Aggregate file dynamic recovery method and device, electronic equipment and storage medium
By setting a reference counter for each aggregate file, monitoring its change frequency in real time, and performing differentiated storage and atomic operations, the problems of delayed aggregate file reclamation and resource waste in the prior art are solved, achieving efficient and accurate resource management.
Patent Information
- Application Number
- CN202511247160.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-02
- Publication Date
- 2025-12-19
AI Technical Summary
Existing aggregated file recycling methods suffer from latency and resource waste in high-concurrency scenarios, leading to system performance bottlenecks and resource fragmentation, making it difficult to achieve accurate and efficient resource recycling.
By setting a reference counter for each aggregate file, monitoring its change frequency in real time and storing it differently, and combining atomic operations to update the counter with a real-time monitoring mechanism, dynamic recycling of aggregate files is achieved.
It reduces performance overhead, optimizes resource utilization, improves migration efficiency and data consistency, shortens deletion latency, and reduces invalid space usage.
Smart Images

Figure CN121166623A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of distributed storage, and particularly relates to a method and device for dynamic recovery of aggregated files, electronic equipment and storage medium. BACKGROUND
[0002] In a distributed storage system, small file aggregation technology, as an important means to improve storage efficiency and reduce metadata management pressure, is widely used in log processing, image storage, time series data analysis and other scenarios. In related technologies, through the coordinated operation of data aggregation, reference management and recovery mechanism, a complete processing flow from file migration, counter maintenance to resource release is constructed. Specifically, this technical system covers the whole process from aggregated file creation, reference counter update to final recovery, including key links such as metadata synchronization, counter persistence, and recovery directory management. With the evolution of storage scale to EB level, the traditional recovery mechanism gradually exposes performance bottlenecks, and a more efficient and low-overhead recovery solution is needed to support large-scale and high-concurrency storage requirements. However, in the existing aggregated file recovery method, the recovery station mechanism is directly used, and relies on periodic scanning and double-checking process, which cannot effectively solve the problems of aggregated file deletion delay and resource waste. Specifically, the existing technology usually adopts a four-stage process of "moving to the recovery directory, periodic scanning, double-checking, and deleting", which has an average delay of several hours, and may occupy 20% of the system bandwidth at peak times, significantly affecting the storage performance. In addition, the recovery directory retains the to-be-deleted files for a long time, resulting in 30%-50% of invalid space occupation, further exacerbating the fragmentation of storage resources. Due to the lack of real-time tracking of the reference state of aggregated files, the traditional method cannot achieve accurate and efficient resource recovery in a high-concurrency scenario, limiting the scalability and stability of the system. SUMMARY
[0003] The present disclosure provides a method and device for dynamic recovery of aggregated files, electronic equipment and storage medium to at least solve the problem of low scalability and stability in a high-concurrency scenario in related technologies.
[0004] The present disclosure provides a method for dynamic recovery of aggregated files, comprising:
[0005] Obtaining the value of the reference counter corresponding to each aggregated file, and calculating the change frequency of the reference counter based on the value of the reference counter;
[0006] According to the change frequency, the aggregated files are differentially stored;
[0007] When the target file is migrated from the source aggregated file to the target aggregated file, updating the reference counters of the target file, the source aggregated file and the target aggregated file;
[0008] Monitoring the value of the reference counter, when detecting that the reference counter is zero, the corresponding aggregate file is added to the buffer queue;
[0009] When the number of aggregate files in the buffer queue reaches the preset threshold or the system is in an idle state, the aggregate files in the buffer queue are batch deleted.
[0010] Optionally, according to the change frequency, the aggregate files are differentially stored, including:
[0011] Based on the change frequency, all the aggregate files are divided into at least one data classification;
[0012] According to the storage strategy corresponding to the data classification, all the aggregate files are differentially stored.
[0013] Optionally, based on the change frequency, all the aggregate files are divided into at least one data classification, including:
[0014] Based on the global sorting result according to the change frequency, the aggregate files are divided into first data, second data and third data according to the first percentage, the second percentage and the third percentage.
[0015] Optionally, according to the storage strategy corresponding to the data classification, all the aggregate files are differentially stored, including:
[0016] The reference counter of the first data adopts the memory resident mode, and the persistence is realized through asynchronous batch writing;
[0017] The reference counter of the second data adopts the combination of memory cache and synchronous persistence, and the single batch input and output is executed by combining multiple operations;
[0018] The reference counter of the third data adopts the disk storage mode, and is loaded into the memory on demand when accessed.
[0019] Optionally, the reference counters of the target file, the source aggregate file and the target aggregate file are updated, including:
[0020] The metadata of the target file is updated to point the source aggregate file associated with the target file to the target aggregate file;
[0021] The reference counter of the source aggregate file is decremented, and the reference counter of the target aggregate file is incremented.
[0022] Optionally, the method further includes:
[0023] When a new aggregate file is created, the value of its reference counter is initialized as the number of small files migrated in.
[0024] Optionally, the reference counter is stored in a metadata header of the aggregate file, and the reference counter is bound to the corresponding aggregate file.
[0025] The present disclosure further provides an apparatus for dynamic recovery of aggregate files, comprising:
[0026] a computing unit configured to obtain a value of a reference counter corresponding to each aggregate file, and calculate a change frequency of the reference counter based on the value of the reference counter;
[0027] a storage unit configured to store the aggregate files differentially according to the change frequency;
[0028] an updating unit configured to update the reference counters of a target file, a source aggregate file and a target aggregate file when the target file is migrated from the source aggregate file to the target aggregate file;
[0029] a monitoring unit configured to monitor the value of the reference counter, and add the corresponding aggregate file to a buffer queue when the reference counter is detected to be zero;
[0030] a deleting unit configured to delete the aggregate files in the buffer queue in batches when the number of aggregate files in the buffer queue reaches a preset threshold or the system is in an idle state.
[0031] The present disclosure further provides an electronic device, comprising a memory configured to store a computer program, and a processor configured to implement the steps of the above-mentioned method for dynamic recovery of aggregate files when executing the computer program.
[0032] The present disclosure further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned method for dynamic recovery of aggregate files.
[0033] The present disclosure further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned method for dynamic recovery of aggregate files.
[0034] The method and device for dynamic recycling of aggregated files, the electronic device and the storage medium provided by the present disclosure can solve the problems of high I / O pressure and response delay caused by disk persistence when the counter is frequently updated in the prior art, and the problem of resource waste caused by cold data counter memory residence, thereby achieving the technical effects of reducing performance overhead and optimizing resource utilization. Since the target file is migrated from the source aggregated file to the target aggregated file, the target file metadata is updated by atomic operation, and the source aggregated file counter is decremented and the target aggregated file counter is incremented, the problem of large verification overhead and difficult consistency guarantee caused by complex verification mechanism when small files are migrated in the prior art can be solved, thereby achieving the technical effects of real-time and accurate tracking of reference relationship and improving migration efficiency and data consistency. Since the technical solution monitors the counter value in real time, when the counter is zero, the corresponding aggregated file is added to the buffer queue, and when the queue reaches the preset threshold or the system is idle, the aggregated files are deleted in batches, thereby solving the problems of recycling delay, storage resources cannot be released in time and additional I / O load caused by the four-stage process of relying on the recycle bin in the prior art, and achieving the technical effects of shortening the deletion delay, reducing the invalid space occupation and reducing the system resource consumption.
[0035] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS
[0036] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:
[0037] Figure 1 A flowchart of a method for dynamic recycling of aggregated files provided by an embodiment of the present disclosure;
[0038] Figure 2 A structural diagram of a device for dynamic recycling of aggregated files provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0039] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description in the following description omits the description of well-known functions and structures.
[0040] In order for those skilled in the art in this technical field to better understand the present disclosure, the present disclosure is further described in detail below in combination with the drawings and specific embodiments.
[0041] In combination with the specific application environment architecture or the specific hardware architecture on which the execution of the hardware event management method depends, the specific application environment architecture or the specific hardware architecture is described herein.
[0042] Embodiments of the present disclosure provide a method for dynamic recovery of aggregated files, Figure 1 A flowchart of a method for dynamic recovery of aggregated files provided by embodiments of the present disclosure is shown.
[0043] As Figure 1 shown, the method comprises the following steps:
[0044] In step 101, the value of the reference counter corresponding to each aggregated file is obtained, and the change frequency of the reference counter is calculated based on the value of the reference counter.
[0045] In the present disclosure, when managing a large number of small files (such as logs, pictures, and time series data) in a distributed file system, a small file aggregation technology is usually used to combine multiple small files into large aggregated files (usually 4MB-512MB) to reduce the metadata management pressure. To achieve efficient recovery of aggregated files, a corresponding reference counter needs to be configured for each aggregated file. The reference counter is a tool specifically used to record the number of times the aggregated file is referenced by the small files it contains. The initial value of the reference counter is set to the total number of small files contained in the aggregated file. This initial value is directly related to the number of small files contained when the aggregated file is created, and can accurately reflect the reference situation of the aggregated file in the initial state. During system operation, the reference counter is updated in real time with the migration of small files and other operations, so the current value of the reference counter corresponding to each aggregated file needs to be obtained continuously to grasp its dynamic reference state.
[0046] Based on the obtained value of the reference counter, the change frequency thereof is further calculated. The change frequency specifically refers to the number of times the reference counter increases or decreases in a unit of time. The calculation process relies on a sliding time window. The sliding time window is a time interval that can be automatically updated with system time. The default setting is 30 minutes. In each sliding time window, the system will count the total number of times the reference counter increases or decreases. By associating this number with the length of the sliding time window (i.e., 30 minutes), the change frequency of the reference counter in a unit of time can be obtained. This frequency can intuitively reflect the dynamic change degree of the reference state of the aggregated file, providing a key basis for subsequent classification management of the aggregated file.
[0047] By obtaining the reference counter value and calculating the change frequency, the real-time reference dynamics of each aggregated file can be accurately captured, providing a data basis for subsequent differentiated strategies according to the heat level, and helping to improve the perception accuracy of the system to the state of the aggregated file, laying a foundation for optimizing storage resource management.
[0048] In step 102, the aggregated files are differentiated according to the change frequency.
[0049] In the embodiments of the present disclosure, after obtaining the change frequency of the reference counter of each aggregated file, the system divides the aggregated files into heat levels based on the frequency, and then implements differentiated storage strategies. Specifically, the change frequency is reflected by the number of increment and decrement operations of the reference counter (ΔCount) within a unit time. The system periodically (e.g., every 60 minutes) sorts all aggregated files by ΔCount value in a global manner, and divides them into three levels of hot data, warm data and cold data according to the percentage of the sorting result. Usually, the top 10% is hot data, the middle 80% is warm data, and the last 10% is cold data. This division method can accurately reflect the dynamic reference activity of different aggregated files and provide a basis for targeted storage.
[0050] For hot data, since its reference counter changes frequently (ΔCount value is high), it needs to update the counter state frequently, so the storage strategy of "memory resident + asynchronous batch persistence" is adopted: the counter data is resident in memory to ensure fast response to update operations, and at the same time, multiple updates are batched and persisted to WAL (Write-Ahead Log) asynchronously, which not only reduces the performance loss caused by frequent disk I / O, but also guarantees the safety of data update.
[0051] For warm data, its reference counter change frequency is between hot data and cold data, so the hybrid strategy of "memory cache + synchronous persistence" is adopted: the counter is temporarily cached in memory, and when a certain number of update operations are accumulated, it is combined into a single batch I / O and written to disk synchronously, balancing real-time performance and resource consumption, and avoiding the inefficient problem of frequent small I / O.
[0052] The reference counter of cold data changes very little (ΔCount value is low) and is in a relatively stable state for a long time, so the counter is directly stored in the disk and uses the "on-demand lazy loading" mechanism: only when the counter needs to be queried or updated, it is loaded from the disk to the memory, and after the update is completed, it is written back to the disk. This way significantly reduces the occupation of memory resources and avoids invalid memory consumption.
[0053] Through the differential storage strategy based on the change frequency, the system can dynamically adjust the storage mode of the counter according to the actual reference activity of the aggregated file, maximize the optimization of the utilization of memory and disk resources while ensuring the data processing efficiency, and effectively solve the problem that in the traditional storage strategy, either the response delay is caused by frequent disk operations, or the resource waste is caused by excessive memory occupation.
[0054] In step 103, when the target file is migrated from the source aggregated file to the target aggregated file, the reference counters of the target file, the source aggregated file and the target aggregated file are updated.
[0055] In the embodiments of the present disclosure, in the distributed file system, with the dynamic change of business data, the small files (i.e. target files) in the aggregated file may be migrated from the original source aggregated file to the new target aggregated file due to the storage optimization requirement. During this migration process, the metadata of the target file itself and the reference counters of the source aggregated file and the target aggregated file need to be updated coordinately to ensure the accuracy and consistency of the reference relationship. Specifically, the update process follows the atomic migration protocol to avoid the counting error caused by the race condition in the multi-thread or multi-node environment. First, the system will lock the reference counters of the source aggregated file and the target aggregated file through the distributed lock mechanism, ensuring that other processes cannot modify these two counters before the migration operation is completed, guaranteeing the exclusivity and consistency of the operation. Then, for the source aggregated file, since the target file no longer exists as a small file contained therein, its reference counter needs to perform an atomic decrement operation (i.e. source aggregated file.count--) to reflect the decrease in the number of references; at the same time, the target aggregated file adds the reference of the target file, so its reference counter needs to perform an atomic increment operation (i.e. target aggregated file.count++) to record the increase in the number of references. At the same time of updating the counters, the system will synchronously modify the metadata of the target file, updating the associated aggregated file identifier of the target file from the source aggregated file to the target aggregated file, so that the metadata information is consistent with the actual reference relationship, ensuring that the subsequent access or management of the target file can accurately point to the aggregated file to which it currently belongs. Finally, after the counter update and metadata modification are completed, the lock on the counters of the source aggregated file and the target aggregated file is released, allowing other operations to access or modify them.
[0056] Among them, the atomic operation refers to an indivisible operation unit, which ensures that the increment and decrement process of the counter will not be interrupted by other threads or processes, avoiding the situation of missing increment, missing decrement or repeated increment and decrement; the distributed lock mechanism is used to ensure that the operations of different nodes on the same counter will not interfere with each other in the multi-node cluster environment, maintaining the consistency across nodes; the metadata, as the data recording the file association information, its timely update can ensure that the identification of the target file belonging to the aggregated file matches the state of the reference counter.
[0057] Through the series of cooperative updating operations, the reference counter states of the source aggregation file and the target aggregation file and the accuracy of the target file metadata can be accurately maintained during the target file migration process, the reference count error or the inconsistency between the metadata and the counter caused by the migration is avoided, and a reliable basis is provided for subsequent judgment on whether the aggregation file is recyclable based on the reference counter state.
[0058] In step 104, the value of the reference counter is monitored, and when it is detected that the reference counter is zero, the corresponding aggregation file is added to the buffer queue.
[0059] In the embodiments of the present disclosure, in the distributed file system, for the reference counter of the aggregation file, the system starts a real-time monitoring mechanism to continuously track the value change of the reference counter corresponding to each aggregation file. This monitoring is not a periodic scan, but an instant perception based on event driving - whenever the reference counter increases or decreases due to small file migration or other operations, the monitoring mechanism will synchronously capture its latest value, ensuring real-time grasp of the counter state. The value of the reference counter directly reflects the number of aggregation files referenced by the current small file. When the value is zero, it means that the aggregation file is no longer referenced by any small file, that is, all the small files contained in it have been migrated to other aggregation files, and itself becomes an invalid residual file, which has the condition of recycling and releasing storage space.
[0060] When the monitoring mechanism detects that the reference counter of a certain aggregation file is zero, the system will immediately mark the aggregation file as a to-be-deleted state and add it to the preset buffer queue. The buffer queue here is a temporary storage structure specially used to temporarily store all aggregation files that meet the deletion condition (i.e., the reference counter is zero), and its role is to avoid the system performance fluctuations that may be caused by the immediate deletion of aggregation files that meet the conditions - for example, a large number of files may be deleted at the same time in a short time, which may cause I / O peak and affect normal business operations. By first including the zeroed aggregation file in the buffer queue, off-peak processing of the deletion operation can be achieved, providing a basis for subsequent batch deletion strategies.
[0061] In this process, real-time monitoring ensures timely discovery of the recyclable state of the aggregation file, avoiding the delay caused by periodic scanning in the traditional recycle bin mechanism; and adding the zeroed file to the buffer queue lays a foundation for orderly and efficient deletion operations, ensuring accurate identification of to-be-deleted files and providing protection for smooth use of system resources. The beneficial effects are that, compared with the way of relying on timed scanning to find deletable files in the traditional scheme, real-time monitoring of the reference counter zero and adding to the buffer queue can significantly shorten the time from file invalidation to entering the to-be-deleted process, reduce the occupancy time of invalid space, and avoid the impact of immediate deletion on system performance through queue temporary storage.
[0062] Step 105, when the number of aggregated files in the buffer queue reaches the preset threshold or the system is in an idle state, batch deleting the aggregated files in the buffer queue.
[0063] In the embodiments of the present disclosure, the buffer queue is a temporary storage structure for temporarily storing the aggregated files whose reference counters have been zeroed, and its core role is to provide scheduling buffer for batch deletion operation. The system continuously monitors the number of aggregated files in the buffer queue, and when the number reaches the preset threshold (such as 100 files), the batch deletion mechanism is triggered - at this time, the system performs the deletion operation on all the aggregated files to be deleted in the queue at one time, and by combining multiple single file deletions into a single batch processing, the I / O interaction overhead caused by frequent single deletions is significantly reduced. At the same time, the system will monitor its running state in real time, and when it detects that the system is in an idle state (i.e. the overall load is low, the I / O resources are sufficient, and there is no high-priority business operation occupying the core resources), even if the number of files in the buffer queue does not reach the preset threshold, the batch deletion process will be actively started to ensure that the to-be-deleted files will not be stranded in the queue for a long time.
[0064] The "preset threshold" here is the optimal value (such as 100) set in advance according to the system performance and business scenarios, and the purpose of its setting is to balance between avoiding excessive accumulation of the queue to cause waste of storage space and ensuring that the deletion operation will not impact the normal operation of the system; the judgment of "system idle state" is based on CPU utilization, memory occupation, I / O throughput and other indicators, and by dynamically monitoring these indicators, it ensures that the deletion operation is performed at a time when resources are sufficient. During the batch deletion process, the system will release the storage space occupied by each aggregated file in the order of the file entering the queue, and update the metadata information synchronously, completely removing the records related to these aggregated files, and ensuring the metadata consistency of the storage system.
[0065] The beneficial effects of this mechanism are: through the combination of threshold triggering and idle triggering batch deletion strategy, it not only avoids the frequent I / O fluctuations caused by immediate deletion of single files, but also prevents the to-be-deleted files from occupying storage space for a long time, achieving timely release of storage space; at the same time, by performing the deletion operation during the system idle period, it reduces the interference to normal business and improves the utilization efficiency of overall system resources.
[0066] The method for dynamically recycling aggregated files provided by the present disclosure can solve the problems of high I / O pressure and response delay caused by disk persistence when the counter is frequently updated, and the problem of resource waste caused by cold data counter memory residence in the prior art, thereby achieving the technical effects of reducing performance overhead and optimizing resource utilization. Since the target file metadata pointer is updated and the source aggregated file counter is decremented and the target aggregated file counter is incremented through atomic operation when the target file is migrated from the source aggregated file to the target aggregated file, the problem of large verification overhead and difficult consistency guarantee caused by complex verification mechanism when small files are migrated in the prior art can be solved, thereby achieving the technical effects of real-time and accurate tracking of reference relationship and improving migration efficiency and data consistency. Since the technical solution monitors the reference counter value in real time, the corresponding aggregated file is added to the buffer queue when the counter is zero, and batch deletion is performed when the queue reaches the preset threshold or the system is idle, thereby solving the problems of recycling delay, inability to release storage resources in time, and additional I / O load caused by the four-stage process of relying on the recycle bin in the prior art, and achieving the technical effects of shortening deletion delay, reducing invalid space occupation, and reducing system resource consumption.
[0067] In the embodiments of the present disclosure, for the operation of "differentially storing the aggregated files according to the change frequency", the specific implementation manners are various. For the sake of clear illustration, some of the implementable manners are listed as follows: based on the change frequency, all the aggregated files are divided into at least one data category; and all the aggregated files are differentially stored according to the storage strategy corresponding to the data category.
[0068] Further, in the embodiments of the present disclosure, "based on the change frequency, all the aggregated files are divided into at least one data category", which includes but is not limited to: based on the global sorting result of the change frequency, the aggregated files are divided into first data, second data and third data according to the first percentage, the second percentage and the third percentage.
[0069] Further, in the embodiments of the present disclosure, "all the aggregated files are differentially stored according to the storage strategy corresponding to the data category", which includes but is not limited to: the reference counter of the first data is stored in the memory residence mode, and the persistence is realized through asynchronous batch writing; the reference counter of the second data is stored in the mode of combination of memory cache and synchronous persistence, and a single batch input and output is performed by combining multiple operations; and the reference counter of the third data is stored in the disk storage mode, and is loaded into the memory on demand when accessed.
[0070] Specifically, in the embodiments of the present disclosure, for the operation of "storing the aggregated files differently according to the change frequency", the core is to classify the data of the aggregated files based on the change frequency of the reference counter first, and then execute the corresponding storage strategy according to the classification result. The change frequency refers to the number of increase and decrease operations (ΔCount) of the reference counter per unit time, which is obtained by statistical analysis through a sliding time window (default 30 minutes, automatically updated with system time), and can directly reflect the activity level of the reference state of the aggregated files. When classifying the data based on the global sorting result of the change frequency, the system will periodically (such as every 60 minutes) sort all the ΔCount values of the aggregated files, and then divide the first data, the second data and the third data according to the first percentage, the second percentage and the third percentage. For example, in actual application, the first 10% of the aggregated files can be divided into the first data (with the highest ΔCount value, representing the most active reference state, which can be called hot data), the middle 80% can be divided into the second data (with moderate ΔCount value, the reference state is relatively stable, which can be called warm data), and the last 10% can be divided into the third data (with the lowest ΔCount value, the reference state changes rarely, which can be called cold data). This classification method can accurately distinguish the dynamic characteristics of different aggregated files, and provide a clear basis for differentiated storage.
[0071] When performing differentiated storage according to the storage strategy corresponding to the data classification, for the first data (hot data), since the reference counter of the first data changes frequently, it is necessary to quickly respond to the update operation, and therefore the memory resident mode is adopted for the reference counter of the first data, that is, the counter data is always stored in the memory, so as to ensure that the high-frequency increase and decrease operation can be completed in real time; at the same time, in order to guarantee the data security, the asynchronous batch writing mode is adopted to realize the persistence, that is, when the counter in the memory accumulates a certain number of updates, the data is written into the WAL (Write-Ahead Log, pre-write log) through a batch operation, so as to avoid the performance loss caused by frequent disk I / O. For the second data (warm data), the change frequency of the reference counter of the second data is between that of the first data and that of the third data, and therefore the combination of the memory cache and the synchronous persistence is adopted: the counter is temporarily stored in the memory cache, and when a plurality of update operations are accumulated, the operations are combined into a single batch I / O and are synchronously written into the disk. This way not only guarantees a certain response speed by using the memory cache, but also reduces the number of disk operations through batch I / O, and a balance is achieved between real-time performance and resource consumption. For the third data (cold data), since the reference counter of the third data is in a stable state for a long time and changes very little, the disk storage mode is adopted for the reference counter of the third data, that is, the counter data is directly stored in the disk, and only when the counter needs to be queried or updated, the counter is loaded into the memory as needed, and after the update is completed, the counter is written back to the disk. This "on-demand loading" mechanism significantly reduces the occupation of the memory resources and avoids invalid memory consumption, and is particularly suitable for aggregate files with long-term unchanged reference states.
[0072] Through the above differentiated storage strategy based on data classification, the system can dynamically adjust the storage mode of the reference counter according to the actual reference activity level of the aggregate file, which not only guarantees the efficient update of the hot data and the balanced processing of the warm data, but also avoids the waste of memory resources by the cold data, effectively solves the problems of high I / O pressure, response delay or resource waste caused by the unified storage mode in the traditional storage strategy, and significantly improves the overall efficiency of the aggregate file management in the distributed file system.
[0073] In the embodiments of the present disclosure, for the operation of "updating the reference counters of the target file, the source aggregate file and the target aggregate file", the specific implementation modes are various, and for the sake of clear description, some of the implementable modes are listed as follows: updating the metadata of the target file to point the source aggregate file associated with the target file to the target aggregate file; decreasing the reference counter of the source aggregate file and increasing the reference counter of the target aggregate file.
[0074] Specifically, in the distributed file system, when a target file (i.e., a small file that needs to be migrated) is migrated from a source aggregate file to a target aggregate file due to business needs such as storage optimization, defragmentation, and the like, the reference counters of the target file, the source aggregate file, and the target aggregate file need to be updated in strict operation logic to ensure the accuracy of the reference relationship and data consistency.
[0075] Specifically, first, the metadata of the target file needs to be updated. The metadata is data that records file association information, which includes identification information of the aggregate file to which the target file belongs. Before migration, the identification points to the source aggregate file; during migration, the identification needs to be modified to the identification of the target aggregate file, so that the metadata accurately reflects the aggregate file to which the target file currently belongs, and ensures that subsequent operations such as access and management of the target file can correctly point to the new storage location. At the same time, the reference counters of the source aggregate file and the target aggregate file need to be adjusted. The source aggregate file is referenced less because of the migration of the target file, so a decrement operation (i.e., source aggregate file.count--) needs to be performed on its reference counter; and the target aggregate file is referenced more because of the addition of the reference of the target file, so an increment operation (i.e., target aggregate file.count++) needs to be performed on its reference counter.
[0076] To avoid race conditions (such as counting errors caused by simultaneous migration) in a multi-threaded or multi-node environment, the decrement and increment operations of the above-mentioned counters are implemented by atomic operations. Atomic operations are indivisible operation units that ensure that the increment and decrement processes of the counters will not be interrupted by other threads or processes, thereby avoiding situations such as missing decrement, missing increment, or repeated increment and decrement. In addition, before the operation is performed, the system will lock the counters of the source aggregate file and the target aggregate file through a distributed lock mechanism to prevent multiple nodes from simultaneously modifying the same counter, thereby ensuring the consistency of operations in a cross-node environment; after the operation is completed, the lock is released to allow other operations to access the counter.
[0077] Through the above operations, the metadata of the target file can be ensured to be consistent with the actual aggregate file to which it belongs, and the reference counters of the source aggregate file and the target aggregate file can be accurately updated, providing a reliable basis for subsequent judgment of whether an aggregate file can be recycled based on the state of the reference counter, and effectively avoiding problems such as confusion of reference relationships or counting errors caused by migration operations.
[0078] In the embodiments of the present disclosure, the method for dynamically recycling aggregate files further includes other specific implementation steps in addition to the foregoing. To clearly present these components, the following detailed description of the related specific implementation is provided: when a new aggregate file is created, the value of its reference counter is initialized to the number of migrated small files.
[0079] Specifically, in the distributed file system, when small file aggregation optimization (such as defragmentation, storage strategy adjustment, etc.) is performed, a new aggregation file is created to accommodate small files that need to be migrated, and at this time, the reference counter needs to be initialized for the new aggregation file. The reference counter is a tool for recording the number of times that the aggregation file is referenced by the small files contained therein in real time. The initial value of the reference counter needs to strictly correspond to the number of small files first migrated into the new aggregation file, that is, when the new aggregation file is created, the initial value of the reference counter is set to the total number of small files first migrated into the file. The core purpose of this initialization operation is to ensure that the counter can accurately reflect the referenced state of the new aggregation file from the beginning of creation, and lay a data foundation for subsequent tracking of small file migration and migration operations. For example, if 100 small files are migrated from an old aggregation file to a newly created aggregation file in an aggregation operation, the initial value of the reference counter of the new aggregation file is set to 100 to correspond to the reference relationship of the 100 small files to it. Subsequently, if more small files are migrated into the new aggregation file, the reference counter will be dynamically incremented to accumulate the total number of small files currently contained, and always keep consistent with the actual reference situation.
[0080] The beneficial effect of this initialization method is that an accurate reference count benchmark is established from the creation of the new aggregation file, ensuring that all subsequent migration operations involving the file (such as incrementing the counter when small files are migrated in, and decrementing the counter when small files are migrated out) can be based on the correct initial value, avoiding misjudgment of the reference relationship due to initial count deviation, providing reliable starting data support for the entire aggregation file dynamic recycling mechanism, and guaranteeing the accuracy of the count and the effectiveness of the recycling judgment.
[0081] In the embodiments of the present disclosure, the reference counter is stored in the metadata header of the aggregation file, and the reference counter is bound to the corresponding aggregation file.
[0082] Specifically, in the distributed file system, the metadata of the aggregate file is data used to describe basic information of the aggregate file (such as creation time, size, index of contained small files, etc.), and the metadata header is a key area at the starting position in the metadata and is a part that is preferentially read by the system when accessing the aggregate file. In the embodiment of the present disclosure, the reference counter is stored in the metadata header of the aggregate file, and this design enables the system to quickly read the value of the reference counter when obtaining the basic metadata information of the aggregate file, without additional disk addressing or data block reading operations, thereby significantly reducing the I / O overhead of accessing the reference counter and improving the query and update efficiency of the counter state. Meanwhile, the reference counter and the corresponding aggregate file form a strict binding relationship, and this binding means that the life cycle of the reference counter is completely synchronized with the aggregate file: when the aggregate file is created, the metadata header of the aggregate file will be initialized with the reference counter (the initial value is the total number of contained small files); when the aggregate file exists in the system, the reference counter always exists as part of the metadata of the aggregate file, and reflects the number of times that the aggregate file is referenced by the small file in real time; and when the aggregate file is finally deleted (such as removed by the batch deletion process after the reference counter is zero), the reference counter in the metadata header of the aggregate file will also be completely removed together with the metadata, and the reference counter will not exist separately from the corresponding aggregate file. This binding relationship ensures the one-to-one correspondence between the reference counter and the aggregate file, avoids the confusion of the count caused by the separation of the counter and the file (such as mistakenly associating the counter of file A to file B), and provides a basic guarantee for accurately tracking the reference state of the aggregate file and realizing accurate recycling. The beneficial effects are that, by storing the reference counter in the metadata header and binding the reference counter with the aggregate file, the access efficiency of the counter is improved, and the strict correspondence between the count and the file is ensured, thereby reducing the system overhead and error risk caused by improper storage location or association confusion.
[0083] It should be noted that the embodiments of the present disclosure can include a plurality of steps, which are numbered for the convenience of description, but these numbers are not a limitation on the execution time slot and execution order between the steps; the steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.
[0084] Corresponding to the above-mentioned method for dynamically recycling the aggregate file, the present disclosure also proposes an apparatus for dynamically recycling the aggregate file. Since the apparatus embodiment of the present disclosure corresponds to the above-mentioned method embodiment, for the details not disclosed in the apparatus embodiment, reference can be made to the above-mentioned method embodiment, and the present disclosure will not be described in detail.
[0085] Figure 2 A structural schematic diagram of an apparatus for dynamically recycling the aggregate file provided by the embodiments of the present disclosure is shown in FIG. 1, which includes: Figure 2
[0086] The computing unit 21 is configured to obtain the value of the reference counter corresponding to each aggregated file, and calculate the change frequency of the reference counter based on the value of the reference counter.
[0087] The storage unit 22 is configured to store the aggregated files differentially according to the change frequency.
[0088] The updating unit 23 is configured to update the reference counter of the target file, the source aggregated file and the target aggregated file when the target file is migrated from the source aggregated file to the target aggregated file.
[0089] The monitoring unit 24 is configured to monitor the value of the reference counter, and add the corresponding aggregated file to the buffer queue when the reference counter is detected to be zero.
[0090] The deleting unit 25 is configured to delete the aggregated files in the buffer queue in batches when the number of aggregated files in the buffer queue reaches a preset threshold or the system is in an idle state.
[0091] It should be noted that the above description of the method embodiment is also applicable to the device of the present embodiment, and the principle is the same. In the present embodiment, the device is not limited.
[0092] The features of the device for dynamic recycling of aggregated files can be referred to the related description of the method for dynamic recycling of aggregated files, which will not be repeated here.
[0093] The embodiments of the present application also provide an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments for dynamic recycling of aggregated files.
[0094] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments for dynamic recycling of aggregated files when running.
[0095] In an exemplary embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.
[0096] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program realizes the steps in the method embodiments of the dynamic aggregation file recycling when executed by a processor.
[0097] The embodiment of the present application further provides another computer program product, which comprises a nonvolatile computer readable storage medium, and the nonvolatile computer readable storage medium stores a computer program, and the computer program realizes the steps in the method embodiments of the dynamic aggregation file recycling when executed by a processor.
[0098] Those skilled in the art will further appreciate that the steps of the example units and algorithms described in connection with the embodiments disclosed herein can be embodied in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
[0099] The above describes in detail the method and device for dynamic aggregation file recycling, the electronic device and the storage medium provided by the present application. The principle and implementation mode of the present application are described by applying specific examples in the present application. The above description of the embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principle of the present application. These improvements and modifications also fall within the protection scope of the claims of the present application.
Claims
1. A method for dynamically recycling aggregated files, characterized in that, include: Obtain the value of the reference counter corresponding to each aggregated file, and calculate the change frequency of the reference counter based on the value of the reference counter; Based on the frequency of change, the aggregated files are stored differently. When a target file is migrated from a source aggregate file to a target aggregate file, the reference counters of the target file, the source aggregate file, and the target aggregate file are updated; Monitor the reference counter value, and when the reference counter is detected to be zero, add the corresponding aggregate file to the buffer queue; When the number of aggregated files in the buffer queue reaches a preset threshold or the system is in an idle state, the aggregated files in the buffer queue are deleted in batches.
2. The method for dynamic recycling of aggregated files according to claim 1, characterized in that, The step of differentially storing the aggregated file based on the change frequency includes: Based on the frequency of change, all aggregated files are divided into at least one data category. According to the storage strategy corresponding to the data classification, all aggregated files are stored differently.
3. The method for dynamic recycling of aggregated files according to claim 2, characterized in that, Based on the frequency of change, all aggregated files are divided into at least one data category, including: Based on the global sorting results of the change frequency, the aggregated file is divided into first data, second data, and third data according to the first percentage, the second percentage, and the third percentage.
4. The method for dynamic recycling of aggregated files according to claim 3, characterized in that, The step of storing all aggregated files differently according to the storage strategy corresponding to the data classification includes: The reference counter for the first data is kept in memory and persisted through asynchronous batch writing; The reference counter for the second data uses a combination of memory caching and synchronous persistence to merge multiple operations into a single batch input / output. The reference counter for the third data is stored on disk and loaded into memory as needed during access.
5. The method for dynamic recycling of aggregated files according to claim 1, characterized in that, The step of updating the reference counters of the target file, the source aggregation file, and the target aggregation file includes: The metadata of the target file is updated to point the source aggregation file associated with the target file to the target aggregation file; Decrease the reference counter of the source aggregate file and increment the reference counter of the target aggregate file.
6. The method for dynamic recycling of aggregated files according to claim 1, characterized in that, The method further includes: When a new aggregate file is created, its reference counter is initialized to the number of small files that are moved in.
7. The method for dynamic recycling of aggregated files according to any one of claims 1-6, characterized in that, The reference counter is stored in the metadata header of the aggregate file, and the reference counter is bound to the corresponding aggregate file.
8. A device for dynamically recycling aggregated files, characterized in that, include: A calculation unit is used to obtain the value of the reference counter corresponding to each aggregated file, and calculate the change frequency of the reference counter based on the value of the reference counter; A storage unit is used to differentiate the storage of the aggregated files according to the change frequency; An update unit is used to update the reference counters of the target file, the source aggregation file, and the target aggregation file when the target file is migrated from the source aggregation file to the target aggregation file; The monitoring unit is used to monitor the value of the reference counter. When the reference counter is detected to be zero, the corresponding aggregate file is added to the buffer queue. The deletion unit is used to delete aggregated files in the buffer queue in batches when the number of aggregated files in the buffer queue reaches a preset threshold or when the system is in an idle state.
9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for dynamic recycling of aggregated files as described in any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method of dynamic recycling of aggregated files according to any one of claims 1-7.
