Distributed file tiered storage method and device, computer device and storage medium

Abstract

The application discloses a distributed file hierarchical storage method and device, computer equipment and a storage medium. The distributed file hierarchical storage method comprises the following steps: determining a snapshot start time corresponding to a current snapshot operation and a target data object, wherein the target data object comprises a dirty data object and a normal data object; determining a snapshot preparation time according to the snapshot start time and a preset time interval; performing a disk brushing operation on the dirty data object according to the snapshot preparation time, and pausing a hierarchical migration operation on the normal data object; performing a snapshot operation on the target data object according to the snapshot start time; and performing a hierarchical migration operation on the target data object after the snapshot operation is completed. The distributed file hierarchical storage method can realize the full life cycle cooperation of data hierarchical migration and timing snapshot.

Description

Technical Field

[0001] This application relates to the field of file storage technology, and in particular to a distributed file hierarchical storage method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Distributed tiered file storage is a key technology in current large-scale storage systems. It typically combines a front-end cache pool with a back-end storage pool, retaining frequently accessed (hot) data in the front-end cache pool to improve performance, while migrating infrequently accessed (cold) data to the back-end storage pool to reduce storage costs. In real-world production environments, tiered storage often needs to be used in conjunction with scheduled snapshot functionality to achieve data backup and rapid recovery. However, in existing technologies, there are timing conflicts between scheduled snapshot operations and tiered migration operations, leading to competition for system resources and potentially causing performance fluctuations. Summary of the Invention

[0003] This application provides a distributed file hierarchical storage method, apparatus, computer device, and storage medium to solve the technical problem of timing conflicts between timed snapshot operations and hierarchical migration operations, which compete for system resources and easily cause performance fluctuations.

[0004] To address the aforementioned technical problems, in a first aspect, this application provides a distributed file hierarchical storage method, comprising: determining the snapshot start time and target data object corresponding to the current snapshot operation, wherein the target data object includes dirty data objects and ordinary data objects; determining a snapshot preparation time based on the snapshot start time and a preset time interval; performing a flush operation on the dirty data object according to the snapshot preparation time, and pausing the hierarchical migration operation on the ordinary data object; performing a snapshot operation on the target data object according to the snapshot start time; and performing a hierarchical migration operation on the target data object after the snapshot operation is completed.

[0005] In the aforementioned distributed file hierarchical storage method, if the snapshot triggers at a time when data objects are migrating from the cache pool to the backend pool or vice versa, the snapshot may only capture partial data or incorrect location information, leading to file corruption or loss during snapshot recovery. By pausing all hierarchical migration operations for ordinary data in advance during snapshot preparation, the positions of all ordinary data objects remain static during snapshot execution, preventing cross-pool movement. Simultaneously, a forced flush of dirty data objects ensures they are persisted to the backend storage pool before snapshot generation, thus capturing a complete, consistent, and disk-persistent data view. By decoupling and co-scheduling the entire lifecycle of snapshot operations with hierarchical migration operations, the incompatibility between snapshots and hierarchical storage in existing technologies is fundamentally solved, enabling a 100% snapshot recovery success rate and completely eliminating snapshot inconsistencies caused by partial data migration. Furthermore, in Write-Back caching mode, newly written data is first stored in the frontend cache pool, awaiting asynchronous flushing. If a snapshot occurs during this interval, dirty data in the frontend cache pool has not yet been flushed to disk; if the cache pool fails, this data will be permanently lost in the snapshot. This method proactively triggers dirty data flushing to disk during the snapshot preparation time and waits for all dirty data to be confirmed to be written to disk before continuing snapshot creation. This effectively converts asynchronous flushing to a synchronous operation, ensuring that all data is safely stored in the backend pool at the snapshot time, reducing the risk of dirty data loss to zero and significantly improving data reliability. Furthermore, by introducing a snapshot preparation time, a clear snapshot window is defined on the timeline. Flushing to disk and pausing migration are completed before the window opens; only snapshots are executed within the window; and hierarchical migration resumes after the window closes. This phased scheduling strategy allows two previously interfering functions to run alternately in an orderly manner without disrupting each other's state, achieving temporal decoupling between hierarchical migration and snapshot operations, avoiding resource contention and state conflicts caused by their random interleaving. Moreover, this distributed file hierarchical storage method has good versatility and low invasiveness, does not depend on a specific hierarchical algorithm, and only controls the start and stop timing of hierarchical migration, making it adaptable to various heat assessment models. Simultaneously, this method is easily implemented as a plug-in in existing distributed storage systems without modifying the underlying kernel.

[0006] In some embodiments, the step of performing a tiered migration operation on the target data object after the snapshot operation is completed includes: determining the enhanced temperature density value of each data object in the front-end cache pool and the back-end storage pool, wherein the enhanced temperature density value is used to characterize the cache priority of each data object; when the utilization rate of the front-end cache pool is greater than a preset utilization threshold, identifying cold data in the front-end cache pool and migrating the cold data to the back-end storage pool, wherein the cold data is the data object in the front-end cache pool whose enhanced temperature density value is less than a first density threshold; identifying hot data in the back-end storage pool and migrating the hot data to the front-end cache pool, wherein the hot data is the data object in the back-end storage pool whose enhanced temperature density value is greater than a second density threshold, wherein the first density threshold is less than the second density threshold.

[0007] In this embodiment, by performing bidirectional hierarchical migration based on enhanced temperature density values ​​after the snapshot is completed and setting differentiated thresholds, it is possible to accurately distinguish between hot and cold data, avoid the snapshot reference data being mistakenly evicted, and improve cache hit rate and space utilization.

[0008] In some implementations, determining the enhanced temperature density value of each data object in the front-end cache pool and the back-end storage pool includes: obtaining the number of accesses to each data object within a preset sliding time window and the access time corresponding to each access; determining the access popularity of each data object based on the current time, the access time, and a preset time decay coefficient; determining the temperature density value of each data object based on the access popularity and the storage capacity occupied by the corresponding data object, wherein the temperature density value is equal to the ratio of the access popularity to the storage capacity occupied; obtaining the total number of valid snapshots corresponding to each data object and the creation time of each valid snapshot; determining the snapshot validity factor of each valid snapshot based on the current time, the creation time, and a preset snapshot validity decay coefficient; and determining the enhanced temperature density value of each data object based on the temperature density value, the snapshot validity factor, and a preset snapshot influence coefficient.

[0009] In this embodiment, the frequency of access is counted by sliding window, the heat is calculated by time decay, and the temperature density is obtained by dividing by the capacity. The effective snapshot reference count and the time factor are combined to make the data value assessment more comprehensive and ensure that the snapshot data receives a reasonable cache protection priority.

[0010] In some implementations, before determining the snapshot start time and target data object corresponding to the current snapshot operation, the process includes: obtaining the hierarchical scan interval time, which is used to characterize the time length between the start times of two adjacent hierarchical scan tasks; and updating the snapshot start time according to the hierarchical scan interval time and a preset snapshot window lock time.

[0011] In this embodiment, by obtaining the hierarchical scanning interval time and combining it with the snapshot window lock time, the snapshot start time is pre-adjusted, which fundamentally avoids the overlap of hierarchical scanning and snapshot operations, reduces resource contention, and improves the overall stability of the system.

[0012] In some embodiments, updating the snapshot start time based on the hierarchical scan interval includes: determining a reference time interval based on the hierarchical scan interval and a preset peak-shaving coefficient, wherein the reference time interval is equal to the product of the hierarchical scan interval and the preset peak-shaving coefficient; determining a snapshot interval based on the reference time interval and the snapshot window lock time; and updating the snapshot start time based on the snapshot interval.

[0013] In this embodiment, the snapshot interval is calculated by multiplying the hierarchical scanning interval time by the peak shifting coefficient and adding the window locking time, thereby achieving a deterministic offset of the snapshot cycle, ensuring that the snapshot window falls precisely between two scans, with low scheduling overhead and easy implementation.

[0014] In some implementations, after migrating the cold data to the backend storage pool, the process includes: obtaining a first storage location, which represents the storage location of the cold data in the backend storage pool; and updating the metadata corresponding to the cold data based on the first storage location.

[0015] In this implementation, the new location is obtained and the metadata is updated immediately after the cold data is migrated to the backend pool, ensuring the consistency between the metadata and the actual storage location, avoiding addressing errors during subsequent access or snapshot recovery, and ensuring data accessibility.

[0016] In some implementations, after migrating the hot data to the front-end cache pool, the process includes: obtaining a second storage location, the second storage location being used to characterize the storage location of the hot data in the front-end cache pool; and updating the metadata corresponding to the hot data according to the second storage location.

[0017] In this implementation, the location information in the metadata is updated synchronously after hot data is promoted to the cache pool, so that subsequent accesses can correctly hit the cache, give full play to the acceleration effect of SSD, and keep the tiered storage transparent to the client.

[0018] Secondly, this application provides a distributed file hierarchical storage device, comprising: The first determining module is used to determine the snapshot start time and the target data object corresponding to the current snapshot operation, wherein the target data object includes dirty data objects and ordinary data objects; the second determining module is used to determine the snapshot preparation time based on the snapshot start time and a preset time interval; the execution module is used to perform a disk flush operation on the dirty data object according to the snapshot preparation time and pause the hierarchical migration operation on the ordinary data object; perform a snapshot operation on the target data object according to the snapshot start time; and perform a hierarchical migration operation on the target data object after the snapshot operation is completed.

[0019] Thirdly, this application provides a computer device including a processor and a memory, wherein the memory is used to store a computer program, and the computer program, when executed by the processor, implements the above-described distributed file hierarchical storage method.

[0020] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described distributed file hierarchical storage method. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a distributed file hierarchical storage method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a distributed file hierarchical storage device shown in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device shown in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a distributed hierarchical file storage method provided in an embodiment of this application. Figure 1 As shown, the distributed file hierarchical storage method of this embodiment includes steps S11 to S15, which are detailed below: Step S11: Determine the snapshot start time and target data object corresponding to the current snapshot operation, wherein the target data object includes dirty data objects and normal data objects.

[0024] In this step, the timed snapshots in the distributed storage system are executed periodically, for example, automatically every hour. The current snapshot operation corresponds to the upcoming next snapshot, and its start time is a preset periodic time point, such as the hour. The target data object refers to all the data that this snapshot will cover, that is, all data objects in the file system or under a specified path. These data objects can be divided into two categories according to whether they have been persisted from the cache pool to the backend storage pool: dirty data objects and ordinary data objects. Dirty data objects are temporary data that only exists in the cache pool and has not yet been written back to the backend storage pool, while ordinary data objects are stable data that already has a complete copy in the backend storage pool.

[0025] Step S12: Determine the snapshot preparation time based on the snapshot start time and the preset time interval.

[0026] In this step, the preset time interval is a pre-determined window length, such as 30 seconds. The snapshot preparation time is equal to the snapshot start time minus this preset time interval. Setting the preparation time allows sufficient time to complete necessary preparatory work before the actual snapshot creation operation is executed, including pausing tiered migration operations that may interfere with snapshot consistency and forcing dirty data to be written to disk.

[0027] Step S13: Perform a disk flush operation on the dirty data object according to the snapshot preparation time, and pause the hierarchical migration operation on the ordinary data object.

[0028] In this step, when the snapshot preparation time arrives, two types of actions are initiated simultaneously: First, for dirty data objects, they are forcibly written from the front-end cache pool to the back-end storage pool (i.e., a disk flush operation) to ensure that all data is persisted before the snapshot begins, preventing data loss due to cache failures. Second, for ordinary data objects, all regular tiered migration operations are temporarily suspended, including pausing the promotion of hot data from the back-end storage pool to the front-end cache pool and pausing the eviction of cold data from the front-end cache pool to the back-end storage pool. Pausing tiered migration aims to keep the position of ordinary data objects static during snapshot execution, preventing the snapshot from capturing incomplete data that is moving across pools. The disk flush operation and the tiered migration operation together ensure the consistency and security of the snapshot's underlying data.

[0029] Step S14: Perform a snapshot operation on the target data object according to the snapshot start time.

[0030] In this step, the snapshot creation process is formally triggered when the system clock reaches the preset snapshot start time. At this point, since all dirty data has been flushed to disk and tiered migration has been frozen, the entire storage system is in a state of data quiescence and consistency. The snapshot operation, based on copy-on-write or redirected write mechanisms, records a snapshot view of the metadata of all target data objects at the current moment and persists the snapshot metadata, such as snapshot ID, creation time, and list of referenced objects, to the metadata service. This snapshot operation serves as the basis for subsequent data recovery, version rollback, or cloning operations.

[0031] Step S15: After the snapshot operation is completed, perform a hierarchical migration operation on the target data object.

[0032] In this step, once the snapshot metadata is persisted, the snapshot operation is officially completed. At this point, the temporary restrictions imposed in step S13 need to be lifted: First, the pause on the hierarchical migration of ordinary data objects is lifted, and the promotion of hot data from the backend storage pool to the frontend cache pool and the eviction of cold data from the frontend cache pool to the backend storage pool are resumed, so that the system can continue to dynamically adjust the data distribution between the cache pool and the backend pool according to the data's hotness or coldness.

[0033] In some embodiments, when performing step S15, firstly, the enhanced temperature density value of each data object in the front-end cache pool and the back-end storage pool is determined, the enhanced temperature density value being used to characterize the cache priority of each data object; when the utilization rate of the front-end cache pool is greater than a preset utilization threshold, cold data in the front-end cache pool is determined and migrated to the back-end storage pool, wherein the cold data is the data object in the front-end cache pool whose enhanced temperature density value is less than a first density threshold; hot data in the back-end storage pool is determined and migrated to the front-end cache pool, wherein the hot data is the data object in the back-end storage pool whose enhanced temperature density value is greater than a second density threshold, the first density threshold being less than the second density threshold.

[0034] In this embodiment, when the utilization rate of the front-end cache pool exceeds a preset utilization threshold, cold data in the front-end cache pool is first identified and then migrated to the back-end storage pool; hot data in the back-end storage pool is identified and migrated to the front-end cache pool. Since the first density threshold is less than the second density threshold, this asymmetric threshold design acts as a buffer: the enhanced temperature density value of a data object is only downgraded when it falls below the lower threshold and upgraded when it rises above the higher threshold. Data in the intermediate region remains in its current position, avoiding frequent migrations due to small fluctuations in temperature, significantly reducing system overhead. By promptly evicting cold data with low enhanced temperature density values ​​to the back-end pool, the freed-up front-end cache pool space can accommodate higher-value hot data; simultaneously, actively promoting hot data with high density values ​​from the back-end storage pool to the cache pool can significantly improve the hit rate of subsequent accesses. Compared to traditional temperature density algorithms, this method can significantly improve cache hit rate and support higher business loads with the same cache capacity. Since the enhanced temperature density value is dynamically updated over time, this hierarchical migration strategy can automatically adapt to changes in business access patterns and the evolution of snapshot lifecycles without manual parameter adjustment.

[0035] In some implementations, when determining the enhanced temperature density value of each data object in the front-end cache pool and the back-end storage pool, the following steps are taken: First, the number of accesses to each data object within a preset sliding time window and the access time corresponding to each access are obtained. Then, based on the current time, the access time, and a preset time decay coefficient, the access popularity of each data object is determined. Next, the temperature density value of each data object is determined based on the access popularity and the storage capacity occupied by the corresponding data object, wherein the temperature density value is equal to the ratio of the access popularity to the storage capacity occupied. Finally, the total number of valid snapshots corresponding to each data object and the creation time of each valid snapshot are obtained. Based on the current time, the creation time, and a preset snapshot time decay coefficient, the snapshot time factor of each valid snapshot is determined. Finally, the enhanced temperature density value of each data object is determined based on the temperature density value, the snapshot time factor, and a preset snapshot influence coefficient.

[0036] In this embodiment, the access frequency of each data object can be calculated using the following formula: ,in, Indicates visit popularity; (This refers to the data object to be evaluated, such as a file block or an object storage unit). This indicates the total number of access events within the statistical period; Representation Object exist The number of accesses at any given time, with each access counted as 1, and supports weighted configuration for read / write access; Indicates the time decay coefficient. The default value is 0.01 / h. Indicates the current system time.

[0037] in, Directly reflects the level of attention the data receives; frequently accessed data receives a higher base score; exponential decay term Ensure that recent accesses have a higher weight than older accesses to prevent historical access patterns from interfering with current tiering decisions. For example, access records from one month ago have a weight close to 0 after decay, preventing cold data from occupying the cache for extended periods. The access popularity of each data object can be quantified in a two-dimensional frequency-time format, more accurately capturing changes in data popularity; time decay coefficient... It supports dynamic configuration and can adapt to different business scenarios, such as database operations requiring rapid decay and archiving operations requiring slow decay. It can be seen that the access frequency of each data object can accurately distinguish between continuously hot data, temporarily hot data, and long-term cold data, providing a reliable basis for subsequent cache replacement assessment.

[0038] In this embodiment, the temperature density value of each data object can be calculated using the following formula: ,in, This indicates the storage capacity occupied by the corresponding data object, in GB or MB, and supports precise calculation based on the actual space occupied. This formula can balance popularity and space efficiency. Since cache resources are limited, sorting solely by popularity may result in large-volume, low-popularity data occupying a large amount of cache space, while small-volume, high-popularity data cannot enter the cache. For example: object A popularity =10, size =10GB, temperature density =1; Object B: Popularity =8, size =2GB, temperature density =4; Sorted by temperature density, object B has a higher cache value and should be retained first to avoid resource waste. The method of calculating temperature density values ​​can avoid the invalid occupation of cache resources by large volumes of low-temperature data and reduce the frequency of cache replacement.

[0039] In this embodiment, the snapshot timeliness factor can be calculated using the following formula: ,in, Indicates the snapshot timeliness factor; This represents the snapshot time-depreciation coefficient. Indicates the current system time. Indicates snapshot Creation time. It's important to note that snapshots generated at different times have different importance and access probabilities: newly generated snapshots (e.g., those generated within 1 hour) have a much higher probability of being restored than older snapshots (e.g., those from 30 days ago). Giving all snapshot-referenced objects the same protection would cause older snapshot-referenced data to occupy the cache for a long time, squeezing out space for active, hot data. The snapshot timeliness factor is calculated in the following formula: Exponential decay over time, the new snapshot Approaching 1, old snapshot Approaching 0; snapshot aging coefficient It can be dynamically configured to adapt to different business snapshot retention strategies, such as slow snapshot decay for core business and fast snapshot decay for non-core business. It can be seen that the calculation of the snapshot timeliness factor can break through the traditional logic of equal protection for snapshot reference data, achieving differentiated protection of snapshot timeliness and balancing snapshot reliability with cache resource utilization. The exponential decay model ensures high priority for new snapshot data while gradually reducing the protection weight of old snapshot data, avoiding long-term cache resource occupation. This significantly improves the cache retention rate of new snapshot data and significantly reduces the cache occupancy rate of old snapshot data. The proportion of valuable data in the cache pool is significantly increased, preventing cache resources from being wasted on low-value old snapshot data.

[0040] In this embodiment, the enhanced temperature density value can be calculated using the following formula: ,in, Indicates the enhanced temperature density value; This represents the snapshot impact coefficient, used to adjust the degree of influence of snapshot features on cache value; Indicates snapshot The effectiveness weight, =1 indicates that the snapshot is valid, has not expired, and has not been deleted; =0 indicates that the snapshot has expired or has been manually deleted; Represents a reference data object All snapshots can be obtained through MDS metadata queries.

[0041] The enhanced temperature density value calculation formula incorporates snapshot reference weight and timeliness on top of temperature density, forming a comprehensive cache value assessment system that combines business value with snapshot value. (Basic items) The core objective of tiered storage is to prioritize caching hot data and migrating cold data; correction items. Dynamic correction of snapshot characteristics: For objects referenced by valid snapshots, a correction term greater than 1 amplifies the enhanced temperature density value, increases cache priority, and prevents erroneous eviction; the newer the snapshot and the more references it has, the larger the correction term, the higher the enhanced temperature density value, and the higher the cache priority; after the snapshot expires... =0, the correction term equals 1, and the enhanced temperature density value regresses to... Objects are subject to normal hot / cold tier classification for cache replacement. Snapshot impact factor. To limit the excessive influence of snapshot features and prevent cold data from occupying the cache for extended periods due to snapshot references, ensuring that the performance advantages of tiered storage are not diminished, the method for determining temperature density values ​​is enhanced. A four-dimensional cache value assessment model is constructed, incorporating heat, space, snapshot references, and snapshot timeliness. This mathematically resolves the compatibility issue between tiered storage and snapshots, requiring no manual intervention. Cache priority can be dynamically adjusted based on data access patterns and snapshot lifecycles to adapt to dynamic business scenarios. The correction term ensures both the positive enhancement effect of snapshot features and avoids negative or zero enhanced temperature density values, ensuring algorithm stability.

[0042] In some embodiments, before performing step S11, the method further includes: obtaining the hierarchical scanning interval time, which is used to characterize the time length between the start times of two adjacent hierarchical scanning tasks; and updating the snapshot start time according to the hierarchical scanning interval time and a preset snapshot window lock time.

[0043] In this embodiment, by obtaining the inherent cycle of the hierarchical scan and adjusting the snapshot start time based on this, the snapshot window can be embedded in the gap between two scans, achieving staggered execution. By introducing a snapshot window locking time as a safety interval, it can be ensured that snapshot operations such as disk flushing, marking, and metadata persistence will not encroach on the time period of the next hierarchical scan, preserving a buffer between the two and further reducing the probability of overlap. This method does not change the original cycle of the hierarchical scan, but only adjusts the offset of the snapshot trigger time, which has minimal impact on the existing hierarchical scheduling logic and is easy to implement.

[0044] In some embodiments, when updating the snapshot start time based on the hierarchical scan interval, a reference time interval is first determined based on the hierarchical scan interval and a preset peak-shaving coefficient. The reference time interval is equal to the product of the hierarchical scan interval and the preset peak-shaving coefficient. The snapshot interval is then determined based on the reference time interval and the snapshot window lock time. The snapshot start time is then updated based on the snapshot interval.

[0045] In this embodiment, the snapshot interval can be calculated using the following formula: ,in, Indicates the snapshot interval. This indicates the preset peak shifting coefficient. Indicates the interval between graded scans. The snapshot window locking time is represented by this formula, which ensures that the snapshot start time always falls within an integer multiple offset of the hierarchical scan cycle. This allows the snapshot window to be precisely positioned in the gap between two scans, achieving theoretical zero overlap. Furthermore, the snapshot frequency can be flexibly controlled by adjusting the preset staggered peak coefficient. When the preset staggered peak coefficient = 1, the snapshot cycle is slightly longer than the scan cycle, resulting in a higher snapshot frequency. When the preset staggered peak coefficient is larger, the snapshot frequency is lower, suitable for scenarios with less stringent real-time requirements for snapshots, making this method adaptable to different business needs. This update method is a feedforward scheduling method, meaning that the subsequent snapshot time points are pre-calculated before snapshot execution, avoiding the overhead of frequent conflict detection and dynamic adjustment during runtime, resulting in high scheduling determinism.

[0046] In some embodiments, after migrating the cold data to the backend storage pool, a first storage location can be obtained, which is used to characterize the storage location of the cold data in the backend storage pool; then the metadata corresponding to the cold data is updated according to the first storage location.

[0047] In this embodiment, updating the metadata corresponding to the cold data based on the first storage location ensures consistency between the metadata and the actual storage location of the data. After data migration, if the location information in the metadata is not updated in a timely manner, subsequent accesses, such as snapshot recovery and client reading, will rely on the old address for addressing, leading to I / O errors. By immediately obtaining the new location and updating the metadata after migration, strong consistency is achieved. The snapshot metadata records the location mapping of each data object; only by ensuring that the metadata is updated synchronously after migration can the data be accurately located during snapshot recovery. This step, together with the snapshot operation in claim 1, forms a closed loop, jointly ensuring the reliability of the snapshot throughout its entire lifecycle. Furthermore, this update operation is fine-grained and incremental, targeting only the migrated cold data, and does not trigger a full metadata scan, resulting in low performance overhead.

[0048] In some embodiments, after migrating the hot data to the front-end cache pool, a second storage location can be obtained, which is used to characterize the storage location of the hot data in the front-end cache pool; then the metadata corresponding to the hot data is updated according to the second storage location.

[0049] In this embodiment, hot data is promoted to the cache pool to accelerate subsequent access. If the metadata is not updated, subsequent requests will still read from the backend pool, causing the promotion operation to fail. This embodiment ensures that the promoted hot data can be correctly addressed, fully leveraging the acceleration effect of SSD caching and guaranteeing timely metadata updates after hot data promotion. This embodiment supports tiered migration and client access through real-time metadata updates. Clients do not need to be aware of changes in data location; all accesses are routed to the correct storage pool through a unified namespace, achieving transparent tiering. By designing a complete metadata synchronization mechanism covering both cold data eviction and hot data promotion, it ensures that the metadata always remains up-to-date regardless of which direction the data migrates.

[0050] In the aforementioned distributed file hierarchical storage method, if the snapshot triggers at a time when data objects are migrating from the cache pool to the backend pool or vice versa, the snapshot may only capture partial data or incorrect location information, leading to file corruption or loss during snapshot recovery. By pausing all hierarchical migration operations for ordinary data in advance during snapshot preparation, the positions of all ordinary data objects remain static during snapshot execution, preventing cross-pool movement. Simultaneously, a forced flush of dirty data objects ensures they are persisted to the backend storage pool before snapshot generation, thus capturing a complete, consistent, and disk-persistent data view. By decoupling and co-scheduling the entire lifecycle of snapshot operations with hierarchical migration operations, the incompatibility between snapshots and hierarchical storage in existing technologies is fundamentally solved, enabling a 100% snapshot recovery success rate and completely eliminating snapshot inconsistencies caused by partial data migration. Furthermore, in Write-Back caching mode, newly written data is first stored in the frontend cache pool, awaiting asynchronous flushing. If a snapshot occurs during this interval, dirty data in the frontend cache pool has not yet been flushed to disk; if the cache pool fails, this data will be permanently lost in the snapshot. This method proactively triggers dirty data flushing to disk during the snapshot preparation time and waits for all dirty data to be confirmed to be written to disk before continuing snapshot creation. This effectively converts asynchronous flushing to a synchronous operation, ensuring that all data is safely stored in the backend pool at the snapshot time, reducing the risk of dirty data loss to zero and significantly improving data reliability. Furthermore, by introducing a snapshot preparation time, a clear snapshot window is defined on the timeline. Flushing to disk and pausing migration are completed before the window opens; only snapshots are executed within the window; and hierarchical migration resumes after the window closes. This phased scheduling strategy allows two previously interfering functions to run alternately in an orderly manner without disrupting each other's state, achieving temporal decoupling between hierarchical migration and snapshot operations, avoiding resource contention and state conflicts caused by their random interleaving. Moreover, this distributed file hierarchical storage method has good versatility and low invasiveness, does not depend on a specific hierarchical algorithm, and only controls the start and stop timing of hierarchical migration, making it adaptable to various heat assessment models. Simultaneously, this method is easily implemented as a plug-in in existing distributed storage systems without modifying the underlying kernel.

[0051] To implement the distributed file hierarchical storage method corresponding to the above method embodiments, and to achieve the corresponding functions and technical effects, see [link to documentation]. Figure 2 , Figure 2 This diagram illustrates a structural block diagram of a distributed file hierarchical storage device according to an embodiment of this application. For ease of explanation, only the parts relevant to this embodiment are shown. The distributed file hierarchical storage device 20 provided in this embodiment includes: The first determining module 21 is used to determine the snapshot start time and the target data object corresponding to the current snapshot operation, wherein the target data object includes dirty data objects and ordinary data objects; The second determining module 22 is used to determine the snapshot preparation time based on the snapshot start time and the preset time interval; The execution module 23 is configured to perform a disk flush operation on the dirty data object according to the snapshot preparation time and pause the hierarchical migration operation on the ordinary data object; perform a snapshot operation on the target data object according to the snapshot start time; and perform a hierarchical migration operation on the target data object after the snapshot operation is completed.

[0052] The distributed file hierarchical storage device described above can implement the distributed file hierarchical storage method of the above method embodiments. The options in the above method embodiments are also applicable to this embodiment, and will not be detailed here. The remaining content of this application embodiment can be referred to the content of the above method embodiments, and will not be repeated in this embodiment.

[0053] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 3 As shown, the computer device 30 of this embodiment includes: at least one processor 31 ( Figure 3 (Only one is shown in the diagram), memory 32, and computer program 33 stored in the memory 32 and executable on the at least one processor 31, wherein the processor 31 executes the computer program 33 to implement the steps in any of the above method embodiments.

[0054] The computer device 30 may be a smartphone, tablet, desktop computer, or cloud server, among other computing devices. The computer device 30 may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that... Figure 3 The computer device 30 is merely an example and does not constitute a limitation on the computer device 30. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0055] The processor 31 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0056] In some embodiments, the memory 32 may be an internal storage unit of the computer device 30, such as a hard disk or memory of the computer device 30. In other embodiments, the memory 32 may be an external storage device of the computer device 30, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 30. Furthermore, the memory 32 may include both internal and external storage units of the computer device 30. The memory 32 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 32 can also be used to temporarily store data that has been output or will be output.

[0057] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0058] This application provides a computer program product that, when run on a computer device, enables the computer device to execute the steps described in the various method embodiments above.

[0059] In the several embodiments provided in this application, it will be understood that each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0060] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0061] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A distributed hierarchical file storage method, characterized in that, include: Determine the snapshot start time and target data object corresponding to the current snapshot operation, wherein the target data object includes dirty data objects and normal data objects; The snapshot preparation time is determined based on the snapshot start time and the preset time interval; The dirty data object is flushed to disk according to the snapshot preparation time, and the hierarchical migration operation for the ordinary data object is paused. Perform a snapshot operation on the target data object according to the snapshot start time; After the snapshot operation is completed, a hierarchical migration operation is performed on the target data object.

2. The distributed file hierarchical storage method as described in claim 1, characterized in that, The step of performing a hierarchical migration operation on the target data object after the snapshot operation is completed includes: Determine the enhanced temperature density value of each data object in the front-end cache pool and the back-end storage pool. The enhanced temperature density value is used to characterize the cache priority of each data object. When the utilization rate of the front-end cache pool is greater than a preset utilization threshold, cold data in the front-end cache pool is identified and the cold data is migrated to the back-end storage pool. The cold data is data objects in the front-end cache pool whose enhanced temperature density value is less than a first density threshold. Identify the hot data in the backend storage pool and migrate the hot data to the frontend cache pool, wherein the hot data is a data object in the backend storage pool whose enhanced temperature density value is greater than a second density threshold, and wherein the first density threshold is less than the second density threshold.

3. The distributed file hierarchical storage method as described in claim 2, characterized in that, The process of determining the enhanced temperature density value of each data object in the front-end cache pool and the back-end storage pool includes: Obtain the number of times each data object is accessed within a preset sliding time window and the access time corresponding to each access. The access popularity of each data object is determined based on the current time, the access time, and the preset time decay coefficient. The temperature density value of each data object is determined based on the access popularity and the storage capacity occupied by the corresponding data object, wherein the temperature density value is equal to the ratio of the access popularity to the storage capacity occupied; Obtain the total number of valid snapshots for each data object and the creation time of each valid snapshot; Based on the current time, the creation time, and the preset snapshot expiration coefficient, the snapshot expiration factor for each valid snapshot is determined. The enhanced temperature density value of each data object is determined based on the temperature density value, the snapshot timeliness factor, and the preset snapshot influence coefficient.

4. The distributed file hierarchical storage method as described in claim 1, characterized in that, Before determining the snapshot start time and target data object corresponding to the current snapshot operation, the following steps are included: Obtain the hierarchical scanning interval time, which is used to characterize the time length between the start times of two adjacent hierarchical scanning tasks; The snapshot start time is updated based on the hierarchical scan interval and the preset snapshot window lock time.

5. The distributed file hierarchical storage method as described in claim 4, characterized in that, The step of updating the snapshot start time according to the hierarchical scanning interval includes: A reference time interval is determined based on the graded scanning interval time and a preset peak-shaving coefficient, wherein the reference time interval is equal to the product of the graded scanning interval time and the preset peak-shaving coefficient. The snapshot interval time is determined based on the reference time interval and the snapshot window lock time; The snapshot start time is updated based on the snapshot interval.

6. The distributed file hierarchical storage method as described in claim 2, characterized in that, After migrating the cold data to the backend storage pool, the process includes: Obtain a first storage location, which is used to characterize the storage location of the cold data in the backend storage pool; The metadata corresponding to the cold data is updated based on the first storage location.

7. The distributed file hierarchical storage method as described in claim 2, characterized in that, After migrating the hot data to the front-end cache pool, the process includes: Obtain a second storage location, which is used to characterize the storage location of the hot data in the front-end cache pool; The metadata corresponding to the hot data is updated according to the second storage location.

8. A distributed file hierarchical storage device, characterized in that, include: The first determining module is used to determine the snapshot start time and the target data object corresponding to the current snapshot operation, wherein the target data object includes dirty data objects and ordinary data objects; The second determining module is used to determine the snapshot preparation time based on the snapshot start time and the preset time interval; The execution module is configured to perform a flush operation on the dirty data object according to the snapshot preparation time, and pause the hierarchical migration operation on the ordinary data object; perform a snapshot operation on the target data object according to the snapshot start time; and perform a hierarchical migration operation on the target data object after the snapshot operation is completed.

9. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, which, when executed by the processor, implements the distributed file hierarchical storage method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the distributed file hierarchical storage method as described in any one of claims 1 to 7.

Images