A method and apparatus for hierarchical storage of file data

Abstract

This invention discloses a method and apparatus for hierarchical file data storage, comprising recursively traversing the size and type attributes of all files, and prioritizing the migration of files to the target layer of the file system based on the file attribute type. This hierarchical file data storage method calculates the migration score for each file by comprehensively considering file access frequency, the previous migration score, and an adjustment coefficient. The adjustment system adjusts the file based on its size and type attributes. This method of calculating migration scores has a wider range of applications, resulting in more reasonable and accurate file migration, while also improving the access performance of the hierarchical file system storage system. When migrating across file system layers, this method maintains a unified business IO access path for files, eliminating the need to create linked inodes to redirect to the migrated file path, thereby reducing resource consumption, minimizing IO amplification during file read / write operations, and reducing system complexity.

Description

Technical Field

[0001] This invention belongs to the field of file system data storage, and specifically relates to a method and apparatus for hierarchical storage of file data. Background Technology

[0002] In the field of data storage, tiered storage is a common data storage method. It stores data on storage media with different characteristics based on the frequency of business access, thereby improving the performance of the storage system while also considering cost. By using different storage media to store business data with varying access frequency, the overall cost-effectiveness of the storage system is improved.

[0003] Tiered storage is primarily used in block storage systems and file storage systems. In file storage systems, the industry mainly manages data tiering automatically, using built-in strategies to identify data access frequency and migrate data to different storage media. Cold data is placed on low-cost, low-speed media, while hot data is placed on high-cost, high-speed media to improve the overall cost-effectiveness of the storage system. While this approach is convenient, reducing the cost of data tiering and improving storage efficiency, current tiered storage methods for file storage systems require prior statistical and detailed analysis of data access frequency. This is not only logically complex but also prone to errors. Inappropriate migration strategies can easily lead to over-migration of data, consuming excessive controller resources and impacting business I / O. For example, if only a portion of a large file's data blocks are accessed, the entire file might be migrated to a higher-performance storage medium, wasting controller resources and causing fluctuations in business I / O during the migration process.

[0004] Patent CN111741107A discloses a layering method, apparatus, and electronic device based on a file storage system. This method determines file access frequency solely by querying file attributes to obtain file access time and uses this frequency as the basis for data layering and migration. All file migration strategies are treated the same, failing to consider different access patterns and file sizes (e.g., in database scenarios, small files are predominant, and migration of small files to the hot data layer is prioritized; in unstructured data scenarios such as images and videos, large files are the primary focus, and migration of large file objects is also prioritized). This fails to fully leverage the efficiency of layered migration for various file types. Furthermore, in existing solutions, after cross-layer file migration, symbolic links need to be created pointing to the original file system path. Business I / O continues to access the file system path before migration to maintain the access path of the migrated files, thus not affecting external business. However, creating additional symbolic links leads to excessive consumption of file system inodes, especially in scenarios involving the migration of a large number of small files. This increases the complexity of file access, fails to fully utilize the file system's space utilization efficiency, and also introduces additional read / write amplification. Summary of the Invention

[0005] The purpose of this invention is to address the problems raised in the background art by proposing a method and apparatus for hierarchical storage of file data.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The present invention proposes a hierarchical storage method for file data, which includes recursively traversing the size and type attributes of all files, and prioritizing the migration of files to the target layer of the file system based on the type of file attributes.

[0008] The system collects statistics on file read and write requests over a recent period and calculates the access popularity of each file based on these requests.

[0009] The migration score for each file is calculated by combining access popularity, the previous migration score, and the adjustment coefficient.

[0010] The files are sorted according to their migration scores.

[0011] Each file is migrated to its corresponding file system target layer based on its migration score.

[0012] Preferably, the target layer of the file system includes a hot data layer, a data layer, and a cold data layer.

[0013] Preferably, the read / write request information includes the number of read / write operations for each file and the range of read / write data lengths.

[0014] When calculating access popularity, the number of reads and writes for each file and the range of read and write data lengths are weighted.

[0015] Preferably, when calculating the range of read and write data lengths, the data offset address and length of each file's read and write operations are calculated, and overlapping data is removed.

[0016] Preferably, the adjustment system α adjusts according to the attributes of the files to be migrated set by the user:

[0017] When the file attribute type to be migrated is set to migrate to the hot data layer, α>0;

[0018] When the file attribute type to be migrated is set to migrate to the cold data layer, α < 0.

[0019] Preferably, migrating each file to its corresponding file system target layer according to its sorted migration score includes:

[0020] When the migration score of the file to be migrated is greater than the migration threshold of the hot data layer, the file will be migrated to the hot data layer.

[0021] When the migration score of the file to be migrated is less than the migration threshold of the data layer, the file will be migrated to the cold data layer.

[0022] When the migration score of the file to be migrated is between the migration threshold of the hot data layer and the migration threshold of the data layer, the file is migrated to the data layer.

[0023] Preferably, the file data hierarchical storage device includes an access module, a statistics module, a migration score calculation module, a sorting module, and a migration module, wherein:

[0024] The access module is used to access the file system.

[0025] The statistics module collects data on file read and write requests over a recent period and calculates the access popularity of each file based on this data.

[0026] The migration score calculation module calculates the migration score for each file based on access popularity, the previous migration score, and the adjustment coefficient.

[0027] The sorting module sorts the files based on their migration scores.

[0028] The migration module migrates each file to the hot data layer, data layer, and cold data layer file systems based on the migration scores.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] 1. This file data hierarchical storage method calculates the migration score of each file by comprehensively considering file access popularity, the previous migration score, and the adjustment coefficient. File access popularity is calculated by weighting the number of reads and writes and the range of read and write data lengths for each file. The adjustment system adjusts the file based on its size and type attributes. This method of calculating migration scores has a wider range of applications, makes file migration more reasonable and accurate, and also improves the access performance of the file system hierarchical storage system.

[0031] 2. When migrating files across file system layers using this hierarchical data storage method, the files still maintain a unified business I / O access path. There is no need to create linked inodes to redirect to the migrated file path, thereby reducing resource consumption and reducing I / O amplification during file read and write, thus reducing system complexity. Attached Figure Description

[0032] Figure 1 This is a flowchart of the hierarchical storage method for file data of the present invention;

[0033] Figure 2 This is a block diagram of the file system layer of the present invention;

[0034] Figure 3This is a flowchart for calculating the migration score of the document in this invention. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there may be an intervening component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the application.

[0037] In one embodiment, such as Figure 1-3 As shown, a method for hierarchical storage of file data includes:

[0038] Step S1: Recursively traverse the size and type attributes of all files, and migrate the files to the target layer of the file system based on the type of the file attributes.

[0039] It should be noted that the target layer of the file system includes a hot data layer, a data layer, and a cold data layer. The relationship between file attributes and the target layer of the file system is pre-defined: for example, files of type ".mp4 / .doc / .jpg" or larger than 10MB are preferentially migrated to the hot data layer, while files of type ".mp4 / .doc / .jpg" or smaller than 5MB are preferentially migrated to the cold data layer (this preferential migration does not guarantee the actual migration; migration will be based on the migration score below, but file attribute settings will affect the adjustment coefficient).

[0040] Step S2: Statistically analyze the read and write request information of files in the recent period, and calculate the access popularity of each file based on the read and write request information.

[0041] Specifically, a unified migration cycle should be set as needed.

[0042] Read / write request information includes the number of read / write operations for each file and the range of read / write data lengths;

[0043] When calculating access popularity, the number of reads and writes for each file and the range of read and write data lengths are weighted and calculated. When calculating the range of read and write data lengths, the data offset address and length of the read and write operations for each file are counted and overlapping data is removed. This allows us to obtain the total range of file data reads and writes within the statistical time period. Finally, we calculate the proportion of the read and write range to the total file size, and files with a high proportion are preferentially migrated to the hot data layer.

[0044] Step S3: Calculate the migration score for each file based on access popularity, the previous migration score, and the adjustment coefficient.

[0045] Specifically, the migration score for each file is a weighted average of access popularity, the previous migration score (if any), and an adjustment coefficient (the default value of the adjustment coefficient is 0, which changes according to the file's attributes). Figure 3 As shown, the access popularity corresponds to Figure 3 The migration score for the current cycle is scoren', the migration score for the previous cycle corresponds to score(n-1), and the migration score for each file corresponds to scoren'.

[0046] The adjustment system α adjusts based on the attributes of the files to be migrated as set by the user:

[0047] When the file attribute type to be migrated is set to migrate to the hot data layer, α>0, thereby increasing the file's migration score;

[0048] When the file attribute type to be migrated is set to migrate to the cold data layer, α < 0, which reduces the migration score of the file.

[0049] For example, if a user wants to migrate files of type ".mp4 / .doc / .jpg" or files larger than a certain value to the hot data layer, the underlying system automatically adjusts the α value of such files to be greater than 0, increasing the overall migration score of such files and prioritizing their migration to the hot data layer. Conversely, if a user wants to migrate files of type ".mp4 / .doc / .jpg" or files smaller than a certain value to the cold data layer, after setting the migration strategy, the underlying system automatically adjusts the adjustment system α value of such files to be less than 0, reducing the overall migration score of such files. This results in these files being ranked lower in the migration score order, below the migration threshold of the data layer, and thus being prioritized for migration to the cold data layer.

[0050] Step S4: Sort the files according to their migration scores.

[0051] Specifically, the files are sorted from highest to lowest migration score so that subsequent migration modules can migrate each file to the target data layer based on its score.

[0052] Step S5: Migrate each file to the corresponding file system target layer according to the sorted migration score.

[0053] Specifically, the hot data layer, data layer and cold data layer are arranged from top to bottom, and migration thresholds are set for the hot data layer and the data layer respectively.

[0054] When the migration score of the file to be migrated is greater than the migration threshold of the hot data layer, the file will be migrated to the hot data layer.

[0055] When the migration score of the file to be migrated is less than the migration threshold of the data layer, the file will be migrated to the cold data layer.

[0056] When the migration score of the file to be migrated is between the migration threshold of the hot data layer and the migration threshold of the data layer, the file is migrated to the data layer.

[0057] It also allows cross-layer migration. As long as the file migration score meets the migration threshold of the hot data layer and the data layer, the file can be directly migrated to the corresponding file system layer, for example, directly from the cold data layer to the hot data layer.

[0058] like Figure 2 As shown, the hard disk types corresponding to the hot data layer, data layer, and cold data layer of the file system are SSD, SAS, and NL-SAS, respectively.

[0059] In another embodiment, the file data hierarchical storage device includes an access module, a statistics module, a migration score calculation module, a sorting module, and a migration module, wherein:

[0060] The access module is used to access the file system;

[0061] Specifically, the access module shields the different file systems at each backend layer, providing a unified file system access interface. After registering custom file system interface callback functions with the operating system, when an application needs to access the file system via I / O, the kernel virtual file system calls the callback method registered by S101 to forward the I / O to the access module, which then receives the business I / O and performs subsequent processing. The access module does not store the file data; the actual file data is still recorded in the actual xfs / ext4 file systems at each backend layer.

[0062] The access module establishes the absolute access path for each file by combining the virtual file system mount directory and the actual mount path of the backend file system. All absolute path information is maintained by the access module, allowing external systems to access and read / write data to the various backend file systems through the access module. Furthermore, the initial file system layer for data writing can be configured according to user needs. For example, data can be written to the hot data layer by default, or it can be configured to be written to the data layer or cold data layer by default.

[0063] When a cross-layer migration occurs (e.g., a file named 'file' is migrated from the data layer's xfs file system to the hot data layer's ext4 file system), the actual backend storage path of 'file' will change (e.g., the xfs file path is / mnt / xfs / file, and after migration to the ext4 file system, the file path becomes / mnt / ext4 / file). However, after passing through the access module, the file's virtual mount directory path remains unchanged (e.g., from an external perspective, the path of 'file' remains / xfs / file before and after the migration, without change). Business I / O is unaware of the migration process. The specific process is as follows... Figure 2 As shown.

[0064] In the access module, after a file (`file`) is migrated across file systems, the access module is responsible for updating and maintaining the file's path information after the migration. When accessing a file, the access module automatically forwards the I / O to the migrated file system layer. This file data hierarchical storage method maintains a unified business I / O access path after migrating across file system layers, without needing to create linked inodes to redirect to the migrated file path. This reduces resource consumption, minimizes read / write I / O amplification, and lowers system complexity.

[0065] The statistics module collects statistics on file read and write requests over a recent period and calculates the access popularity of each file based on the read and write request information.

[0066] The migration score calculation module calculates the migration score for each file based on access popularity, the previous migration score, and the adjustment coefficient.

[0067] The sorting module sorts the files according to their migration scores;

[0068] The migration module migrates each file to the hot data layer, data layer, and cold data layer file systems based on the migration scores.

[0069] This file data tiered storage method calculates the migration score for each file by comprehensively considering file access popularity, the previous migration score, and an adjustment coefficient. File access popularity is weighted by the number of read / write operations and the range of read / write data lengths for each file. The adjustment system adjusts the score based on the file size and type attributes. This method of calculating migration scores has a wider range of applications, resulting in more reasonable and accurate file migrations, while also improving the access performance of the file system tiered storage system. When migrating across file system layers, this method maintains a unified business I / O access path for files, eliminating the need to create linked inodes to redirect to the migrated file path. This reduces resource consumption, minimizes I / O amplification during file read / write operations, and lowers system complexity.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. A method for hierarchical storage of file data, characterized in that: The file data hierarchical storage method includes: Recursively traverse the size and type attributes of all files, and migrate the files to the target layer of the file system based on the type of the file attributes. Collect statistics on file read and write requests over a recent period of time, and calculate the access popularity of each file based on the read and write request information; The migration score for each file is calculated by combining access popularity, previous migration score, and adjustment coefficient. Sort the files according to their migration scores; Each file is migrated to its corresponding file system target layer based on its migration score. The target layer of the file system includes a hot data layer, a data layer, and a cold data layer. The read / write request information includes the number of read / write operations for each file and the range of read / write data lengths; When calculating access popularity, the number of reads and writes for each file and the range of read and write data lengths are weighted and calculated. When calculating the read / write data length range, the data offset address and length of each file's read / write operation are calculated, and overlapping data is removed; The adjustment factor α is adjusted according to the attributes of the files to be migrated set by the user. When the file attribute type to be migrated is set to migrate to the hot data layer, adjust α>0; When the file attribute type to be migrated is set to migrate to the cold data layer, adjust α < 0.

2. The file data hierarchical storage method as described in claim 1, characterized in that: The process of migrating each file to its corresponding file system target layer based on its sorted migration score includes: When the migration score of the file to be migrated is greater than the migration threshold of the hot data layer, the file will be migrated to the hot data layer. When the migration score of the file to be migrated is less than the migration threshold of the data layer, the file will be migrated to the cold data layer. When the migration score of the file to be migrated is between the migration threshold of the hot data layer and the migration threshold of the data layer, the file is migrated to the data layer.

3. A file data hierarchical storage device, characterized in that: The hierarchical file data storage device includes an access module, a statistics module, a migration score calculation module, a sorting module, and a migration module, wherein: The access module is used to access the target layer of the file system; The statistics module collects statistics on file read and write request information over a recent period of time and calculates the access popularity of each file based on the read and write request information. The migration score calculation module calculates the migration score for each file based on access popularity, the previous migration score, and the adjustment coefficient. The sorting module sorts the files according to their migration scores; The migration module migrates each file to the hot data layer, data layer, and cold data layer of the target layer of the file system according to the sorted migration scores. The read / write request information includes the number of read / write operations for each file and the range of read / write data lengths; When calculating access popularity, the number of reads and writes for each file and the range of read and write data lengths are weighted and calculated. When calculating the read / write data length range, the data offset address and length of each file's read / write operation are calculated, and overlapping data is removed; The adjustment factor α is adjusted according to the attributes of the files to be migrated set by the user. When the file attribute type to be migrated is set to migrate to the hot data layer, adjust α>0; When the file attribute type to be migrated is set to migrate to the cold data layer, adjust α < 0.

Images