File size and time attribute-based popularity discrimination and data migration method

By combining the heat discrimination method of file size and time attributes, the problem of insufficient rigorous judgment of hot and cold data in the existing technology is solved, and the reasonable storage and management of files are realized, taking into account storage efficiency and cost.

CN119025049APending Publication Date: 2024-11-26TIANJIN UNIVERSITY OF TECHNOLOGY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411494650.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art only relies on the access frequency and access time of files in data storage to determine hot and cold data, which cannot accurately reflect the hot and cold level of data, and there is a dilemma when selecting to store large files or multiple small files when there is insufficient storage space.

Method used

Using a heat discrimination method based on file size and time attributes, the heat value is adjusted by obtaining the timestamp, file size and type of the file, the attenuation function is established and the weight value is set, and the heat value is adjusted according to the heat value. Store files in different backup methods.

Benefits of technology

It improves the rationality of hot and cold division of files, takes into account the reading and writing speed and cost issues, and ensures the reasonable storage and management of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025049A_ABST
    Figure CN119025049A_ABST
Patent Text Reader

Abstract

The invention provides a file size and time attribute-based popularity discrimination and data migration method, which comprises the following steps of: acquiring information of each file, including a timestamp, a size and a type, establishing an attenuation function, setting a weight value according to the file type, calculating a first popularity value, calculating an attenuation factor through a preset attenuation strategy, and performing data migration according to the attenuation factor. The file information comprises a first heat value, the first heat value is combined to obtain a second heat value, the second heat value is adjusted according to a threshold value of a disk storage space, the file information further comprises a black list and a white list, the adjusted heat value is subjected to secondary adjustment according to the black list and the white list to obtain a final heat value, and different backup modes are selected according to the final heat value. The method has the advantages that the cold temperature and the hot temperature of the files are divided by comprehensively considering the file size, the timestamp and the file type, the file division reasonability can be improved, then the files are stored in the corresponding positions according to file division, and therefore the read-write speed and cost problems can be considered.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data storage, and in particular relates to a method for heat determination and data migration based on file size and time attributes. Background Art

[0002] At present, the development of the big data era is inseparable from the storage of massive data, and data is crucial to all walks of life. Therefore, the storage of data has always been the focus of research. In the file system, the metadata of each file usually includes multiple timestamps, such as creation time, modification time and access time. These timestamps play an important role in system management, backup, security and performance optimization. At the same time, according to different data such as access time or access frequency, data can be divided into hot and cold data and stored separately. However, currently, most of them only consider the access frequency and access time of data as the basis for judging the heat of data, which cannot accurately reflect whether the data is hot or cold data, that is, the hot and cold division of data is not rigorous enough. The change tendency of different timestamps of each data is different. For example, the access time of a document file has a higher probability of change than the modification time and creation time, while the modification time and creation time of a database file have a higher probability of change than the access time. At the same time, when the storage space is insufficient, whether to choose a very hot large file or multiple hot small files to store on a high-speed disk is also a thorny problem.

[0003] In patent 202010296769.8, a data migration system and method for file hierarchical storage, the hotness or coldness of files is determined based on the number of file accesses and the most recent access time within a period, and in patent 202211259603.4, a method, device, equipment and storage medium for hot and cold data redundancy, data is divided into hot data and cold data based on the access frequency after processing and the time interval between the latest accesses after processing. This traditional method for distinguishing hot and cold data still has room for improvement. For example, the change tendency of different timestamps of each type of data mentioned above is different, so it is not comprehensive to only consider the number of file accesses and the most recent access time within a period, and the hot and cold division of data is not rigorous enough. For example, it is also not comprehensive to only take the access frequency and the most recent access time and ignore the specific access time of each time. It is also possible that a file that has been accessed ten times intermittently within a year is hotter than another file that has been accessed five times within a day. Summary of the invention

[0004] In view of this, the present invention aims to propose a method for heat determination and data migration based on file size and time attributes, in order to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] The method for heat identification and data migration based on file size and time attributes includes the following steps:

[0007] Obtaining file information of each file, wherein the file information includes at least a file timestamp, a file size, and a file type;

[0008] Establishing a decay function according to the acquired file timestamp, setting a weight value for the decay function according to the file type, and calculating a first heat value using the decay function containing the weight value;

[0009] An attenuation factor is calculated using a preset attenuation strategy, and a second heat value is calculated using the attenuation factor and the first heat value;

[0010] Setting a storage threshold according to the storage space size of the disk, and adjusting the second heat value according to a comparison result between the storage threshold and the file size;

[0011] The file information also includes a blacklist and a whitelist, and the adjusted second heat value is adjusted again according to the blacklist and a whitelist to obtain a final heat value;

[0012] According to the final heat value, each file is stored in different storage spaces through different backup methods.

[0013] Furthermore, the process of establishing the attenuation function includes:

[0014] The difference between the current time and the file timestamp is calculated to obtain the time difference, and a decay function is established using the time difference and a preset decay rate; wherein the file timestamp includes an access time, a creation time, and a modification time, and the time difference is the difference between the current time and any file timestamp.

[0015] Furthermore, the process of setting a weight value for the attenuation function according to the file type and calculating the first heat value using the attenuation function containing the weight value includes:

[0016] Classifying files into different types according to their access and modification conditions, and presetting weight values ​​according to the file types, wherein three weight values ​​are provided, and the three weight values ​​correspond to three different file timestamps respectively;

[0017] The decay function established according to the file timestamp is multiplied by the corresponding weight value and the sum is accumulated to obtain the first heat value.

[0018] Furthermore, when a file is accessed or modified, the corresponding access time or modification time is updated, and the changed file information is stored. The file is preset with a number of storage entries. If the number of changes in the file's access time exceeds the number of storage entries, the file information with the earliest access time is deleted and the current file information is stored.

[0019] Furthermore, the process of calculating the attenuation factor by using a preset attenuation strategy includes:

[0020] The timestamp corresponding to the last update of the file is found, and the time difference between the current time and the last update is calculated, and the attenuation factor is established according to the time difference and the preset time standard.

[0021] Furthermore, the process of calculating the second heat value using the attenuation factor and the first heat value includes:

[0022] The heat value of the last update of the file is calculated using a decay function containing a weight value, and the product of the decay factor and the heat value of the last update of the file is added to the first heat value to obtain a second heat value.

[0023] Furthermore, the process of adjusting the second heat value according to the comparison result between the storage threshold and the file size includes:

[0024] Infer the storage space usage of files based on their size;

[0025] If the storage space occupancy rate of the file is greater than or equal to the storage threshold, the heat value is adjusted according to the file size;

[0026] If the storage space occupancy rate of the file is lower than the storage threshold, the heat value is not changed.

[0027] Furthermore, the process of adjusting the heat value according to the file size includes:

[0028] Divide the file size into four different preset values, and establish a piecewise function according to the four preset values, wherein the piecewise function includes four file preset functions;

[0029] If the file size is smaller than the first preset value or larger than the fourth preset value, the second heat value is adjusted using the first file preset function;

[0030] If the file size is greater than the first preset value and less than the second preset value, the second heat value is adjusted using the second file preset function;

[0031] If the file size is greater than the second preset value and less than the third preset value, the second heat value is adjusted using the third file preset function;

[0032] If the file size is greater than the third preset value and less than the fourth preset value, the second heat value is adjusted using a fourth file preset function, wherein the fourth file preset function is a constant function.

[0033] Further, the process of performing a secondary adjustment on the adjusted second heat value according to the black and white list to obtain the final heat value includes:

[0034] Get the blacklist and whitelist of the file. If the file is in the whitelist, increase the heat value of the file; if the file is in the blacklist, decrease the heat value of the file.

[0035] Furthermore, the process of storing each file in different storage spaces in different backup modes according to the final heat value includes:

[0036] According to the final heat value, each file is divided into files of different heats, wherein the files of different heats include cold files, warm files, and hot files;

[0037] If the current file is the cold file, it is stored in a low-speed disk for backup through erasure coding;

[0038] If the current file is the warm file, it is stored in a medium-speed disk in a second copy for backup;

[0039] If the current file is the hot file, it is stored in a high-speed disk in three copies for backup.

[0040] Compared with the prior art, the method for heat identification and data migration based on file size and time attributes of the present invention has the following beneficial effects:

[0041] By comprehensively considering the three factors of file size, timestamp and file type to divide files into hot, warm and cold types, the rationality of file division can be improved. Then, the files can be stored in corresponding locations according to the file division, so as to take into account both reading and writing speed and cost issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 The figure is a flow chart of a method for heat determination and data migration based on file size and time attributes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0045] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0047] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0048] The method for heat identification and data migration based on file size and time attributes includes the following steps:

[0049] S1. Obtain file information of each file, wherein the file information at least includes a file timestamp, a file size, and a file type;

[0050] It should be noted that when a file is accessed or modified, the corresponding access time or modification time is updated, and the changed file information is stored. The file is preset with a number of storage entries. If the number of changes in the file's access time exceeds the number of storage entries, the file information with the earliest access time is deleted and the current file information is stored.

[0051] Preferably, a specific execution process of step S1 is as follows:

[0052] The user enters the absolute path of the directory he wants to monitor, and the client will monitor and record the file timestamps (including: access time atime, creation time ctime, modification time mtime), file size, and file type of all files in the corresponding directory in a file. If the time information of a file is monitored multiple times, there may be multiple records for the same file, and these multiple records can all participate in the calculation of the file heat value. The number of storage entries can be set by yourself.

[0053] S2. Establish a decay function according to the acquired file timestamp, set a weight value for the decay function according to the file type, and use the decay function containing the weight value to calculate and obtain a first heat value.

[0054] The process of establishing the attenuation function in step S2 includes:

[0055] S21. Calculate the difference between the current time and the file timestamp to obtain a time difference, and use the time difference and a preset decay rate to establish a decay function; wherein the file timestamp includes an access time, a creation time, and a modification time, and the time difference is the difference between the current time and any file timestamp.

[0056] Preferably, a specific execution process of step S21 is as follows:

[0057] The heat value is calculated based on the obtained file size, absolute path, atime, ctime, mtime values ​​and file type. In order to ensure that the calculation results will not be too different and easily tend to 0 due to simply using the inverse of the time difference to calculate the heat value, a decay function is established:

[0058] ;

[0059] Wherein, α is the decay rate, Δt is the time difference, in one embodiment, α=0.001, Δt=current_time-time;

[0060] Among them, α is the decay rate and α=0.001, which is used to adjust the rate of time decay. Δt is the time difference between the calculated current time and atime, ctime or mtime, which is calculated by using the above monitored time information. This decay function makes the value decrease rate gradually decrease as the time difference increases, and can calculate the value of the time difference more smoothly and reasonably reflect the information of the time difference.

[0061] In step S2, a weight value is set for the attenuation function according to the file type, and the process of calculating the first heat value using the attenuation function containing the weight value includes:

[0062] S22. Classify the files into different types according to the access and modification conditions of the files, and pre-set weight values ​​according to the file types, wherein three weight values ​​are provided, and the three weight values ​​correspond to three different file timestamps respectively.

[0063] Preferably, a specific execution process of step S22 is as follows:

[0064] The weight value is determined according to a preset value, specifically including:

[0065] Text files are often read and modified relatively frequently, especially during editing, while their status may not change as frequently as other file types. Therefore, the weights of document files doc, pdf, and txt are: a=0.6, b=0.2, c=0.2 respectively;

[0066] Images and media files may be frequently accessed and not modified. Therefore, the weights of images and media files jpg, png, mp3, mp4 are: a=0.5, b=0.3, c=0.2;

[0067] Programs and executable files may be executed regularly. Once written, executable files or scripts are not often modified. Changes in permissions and ownership may be more important. Therefore, the weights of programs and executable files exe, dll, sh are: a=0.4, b=0.2, c=0.4 respectively;

[0068] Compressed files are not frequently accessed or modified, so the weights of compressed files zip, rar, and targz are: a=0.35, b=0.25, c=0.4 respectively;

[0069] Configuration files are accessed relatively infrequently. The system or application may update the configuration files as needed. The permissions and properties of the configuration files may change due to security updates. Therefore, the weights of the configuration files and scripts cfg, conf, ini, bat, and ps1 are: a=0.1, b=0.5, and c=0.4 respectively.

[0070] Database files may be accessed regularly, and they may be frequently modified due to data updates. Changes in the status and attributes of database files may affect the entire application. Therefore, the weights of the database and data files db, sql, csv, json, and xml are: a=0.3, b=0.35, and c=0.35 respectively;

[0071] System files are basically not accessed or modified, so the weight values ​​of system files sys and dll are: a=0.1, b=0.1, c=0.8;

[0072] The web page files are relatively balanced in all aspects, but they are still accessed more. The weight values ​​of the web page files html, css, and js are: a=0.4, b=0.3, c=0.3;

[0073] Scientific and engineering files are mostly used to realize engineering. The weight values ​​of scientific and engineering files mat, cad, and psd are: a=0.25, b=0.5, c=0.25;

[0074] Temporary files are frequently accessed only when they are created, and are rarely changed or modified. Therefore, the weights of temporary files tmp and swp are: a=0.5, b=0.3, c=0.2 respectively;

[0075] The weight value of other files is: 0.3333.

[0076] S23. The attenuation function established according to the file timestamp is multiplied by the corresponding weight value and the sum is accumulated to obtain the first heat value.

[0077] Preferably, a specific execution process of step S23 is as follows:

[0078] In the calculation of file heat, the file type is often overlooked. Different types of files often have different emphasis on the change of timestamp information, which means that the weights of different atime, ctime, and mtime timestamp information should be different. The following is a preliminary calculation of the file heat value using the predefined time decay function and file type. The formula is as follows:

[0079] new_heat=a*decay(atime)+b*decay(ctime)+c*decay(mtime);

[0080] The weights of the timestamp information of each atime, ctime, and mtime are obtained according to the type of the obtained file, where a, b, and c are specific weight values.

[0081] S3. Calculate an attenuation factor using a preset attenuation strategy, and use the attenuation factor and the first heat value to calculate a second heat value.

[0082] The process of calculating the attenuation factor by using the preset attenuation strategy in step S3 includes:

[0083] S31. Find the timestamp corresponding to the last update of the file, calculate the time difference between the current time and the last update, and establish a decay factor according to the time difference and a preset time standard.

[0084] A specific execution process of step S31 is as follows:

[0085] After calculating the initial heat value of the file, in order to refine this heat value, it is necessary to calculate the time-related decay factor. If there are multiple records for the same file, the processing process follows the following steps:

[0086] Select and calculate the oldest record: select the oldest record from all records, calculate the heat value of this record based on its specific parameters, and then save this heat value together with its corresponding timestamp;

[0087] Process subsequent records one by one: Turn to the second oldest record. When calculating the heat of the second record, not only the decay factor of the record itself should be considered, but also the previously saved heat value should be referred to to calculate the heat value of the second record, and this value and the corresponding timestamp should also be saved;

[0088] This process is repeated: the popularity value of each record is calculated one by one in the chronological order of the records. Each calculation is based on the previously saved popularity value, ensuring that each popularity value can accurately reflect the popularity of the file over time.

[0089] Complete all calculations: process the latest record in the file and get the final heat value.

[0090] The entire process ensures that each step of the calculation is based on the actual time-decreasing impact and the heat result of the previous step, so that the final result more accurately reflects the changes in the file's attention over time.

[0091] The function of the attenuation factor is as follows:

[0092] ;

[0093] Among them, a1 and b1 are used to adjust the rate of time decay, time_diff is the difference between the time when the timestamp was last updated and the current time, time_current is a preset value used as a time standard, and the obtained decayfactor is a decay coefficient, which will decrease as time_diff increases, which means that as time goes by, the heat will gradually decay according to this formula.

[0094] The process of using the attenuation factor and the first heat value to calculate the second heat value in step S3 includes:

[0095] S32. Calculate the heat value of the last update of the file using a decay function containing a weight value, and add the product of the decay factor and the heat value of the last update of the file to the first heat value to obtain a second heat value.

[0096] The calculation formula of the second heat value proposed in step S32 is as follows:

[0097] new_heat_value=(decayfactor*currentheat+(1-a)*new_heat);

[0098] Among them, decayfactor is the decay factor calculated above and is used to calculate the coefficient of old records, currentheat is the heat value of the file stored last time, a is a preset value used to indicate the proportion of the initial heat value, and new_heat is the initial heat value, which is calculated by the above formula.

[0099] S4. Set a storage threshold according to the storage space size of the disk, and adjust the second heat value according to a comparison result between the storage threshold and the file size.

[0100] The process of adjusting the second heat value according to the comparison result between the storage threshold and the file size in step S4 includes:

[0101] S41, inferring the storage space occupancy rate of the file according to the file size;

[0102] S42: If the storage space occupancy rate of the file is greater than or equal to the storage threshold, adjust the heat value according to the file size;

[0103] S43: If the storage space occupancy rate of the file is lower than the storage threshold, the heat value is not changed.

[0104] Preferably, a specific execution process of step S4 is as follows:

[0105] When calculating the final heat value, the size of the disk's occupied space is first determined. If the size of the disk's occupied space is less than the preset value β, it is considered that the space is now sufficient, and the file size is not considered as a parameter for heat value calculation. If the size of the disk's occupied space is greater than or equal to the preset value β, it is considered that the space is now relatively tight, and the file size is used as a parameter in the heat value calculation. This can comprehensively measure which files are truly suitable for high-speed devices and which files should be placed in low-speed devices, and put files into high-speed devices as reasonably as possible. The specific formula for the above process is as follows:

[0106] when size<β;

[0107] new_heat_value=new_heat_value;

[0108] when size>=β;

[0109] new_heat_value=new_heat_value*f(size).

[0110] The process of adjusting the heat value according to the file size in step S42 includes:

[0111] Divide the file size into four different preset values, and establish a piecewise function according to the four preset values, wherein the piecewise function includes four file preset functions;

[0112] If the file size is smaller than the first preset value or larger than the fourth preset value, the second heat value is adjusted using the first file preset function;

[0113] If the file size is greater than the first preset value and less than the second preset value, the second heat value is adjusted using the second file preset function;

[0114] If the file size is greater than the second preset value and less than the third preset value, the second heat value is adjusted using the third file preset function;

[0115] If the file size is greater than the third preset value and less than the fourth preset value, the second heat value is adjusted using a fourth file preset function, wherein the fourth file preset function is a constant function.

[0116] Preferably, a specific execution process of step S42 is as follows:

[0117] In a specific execution process of step S4, new_heat_value is the second heat value, f(size) is a designed piecewise function for processing file size values ​​in different size ranges. If the file size is particularly small, its weight is considered to be relatively low, because there is no obvious difference in the reading and writing speeds of files that are too small when stored on different high-speed devices or low-speed devices. When the file size is particularly large, its weight is considered to be relatively low, because files that are too large will make the situation of insufficient disk space more severe, but when its heat value is very high, it still has the possibility of being stored on high-speed devices. When the file size is medium, its weight will be higher. Medium-sized files will not take up too much space, and can feel obvious differences in the reading and writing speeds of storage devices with different speeds. Through this piecewise function, the relationship between file size and file heat value can be reasonably measured, and the results of different devices suitable for storage can be calculated.

[0118] The specific file size function f (size) formula is as follows:

[0119] f(size)= ;

[0120] When the file size is smaller than a preset value α1 or larger than a preset value α4, the value of the file size function is a preset value q;

[0121] When the file size is greater than the preset value α1 and less than the preset value α2, the file size function is an activation function;

[0122] Where k and x0 are preset values, x0 is the midpoint value of the file, k is used to control the rate of change, x is the file size, and k and x0 are parameters for calculating the expected result based on the change in file size;

[0123] When k is large, the file size will be very sensitive near x0;

[0124] When k is relatively small, the result of the function is adjusted more smoothly;

[0125] When the file size is greater than the preset value α2 and less than the preset value α3, the file size function is the same as the above activation function, but the parameters k1 and x1 are different to adapt to the growing file size;

[0126] When the file size is greater than the preset value α3 and less than the preset value α4, the file size function is a preset value multiplied by the inverse of the file size, which can directly reflect the reaction of the file size;

[0127] Among them, γ is a preset value used to adjust the value range of the file size function.

[0128] S5. The file information further includes a blacklist and a whitelist. The adjusted second heat value is adjusted again according to the blacklist and a whitelist to obtain a final heat value.

[0129] The process of performing a secondary adjustment on the adjusted second heat value according to the black and white lists in step S5 to obtain the final heat value includes:

[0130] Get the blacklist and whitelist of the file. If the file is in the whitelist, increase the heat value of the file; if the file is in the blacklist, decrease the heat value of the file.

[0131] Preferably, a specific execution process of step S5 is as follows:

[0132] After adjusting the second heat value according to step S4, the files need to be sorted by heat value;

[0133] Before sorting, it is necessary to determine the impact of the blacklist and whitelist on the heat value. When a file is on the blacklist, a large heat value will be subtracted from the file to ensure that the file is ranked lower. When the file is on the whitelist, a large heat value will be added to the file to ensure that the file is ranked higher. After the heat values ​​of the files on the blacklist and whitelist are added and subtracted, the files are sorted according to the heat values.

[0134] S6. Store each file in different storage spaces through different backup methods according to the final heat value.

[0135] The process of storing each file in different storage spaces in different backup modes according to the final heat value in step S6 includes:

[0136] According to the final heat value, each file is divided into files of different heats, wherein the files of different heats include cold files, warm files, and hot files;

[0137] If the current file is the cold file, it is stored in a low-speed disk for backup through erasure coding;

[0138] If the current file is the warm file, it is stored in a medium-speed disk in a second copy for backup;

[0139] If the current file is the hot file, it is stored in a high-speed disk in three copies for backup.

[0140] Preferably, a specific execution process of step S6 is as follows:

[0141] According to the sorting result in step S5, the files are stored in different pools of the corresponding ceph, and the process includes:

[0142] When a file is at the first preset value in the sorting result, the file will be stored in the hot_pool, where the hot_pool is composed of M.2SSD, with extremely fast read and write speeds, and the storage method is three-copy storage, with higher fault tolerance and read and write speeds. It can make frequently used hot files run quickly;

[0143] When the file is at the second preset value after the sorting result, the file will be stored in the cold_pool, where the cold_pool is composed of HDDs, with slow reading and writing speeds, and the storage method is 4:2 erasure code storage, which has a good fault tolerance rate and can save costs well, and can well store infrequently used cold files;

[0144] When a file is between the first and second preset values ​​of the sorting result, the file will be stored in warm_pool, where warm_pool is composed of SSDs, has fast read and write speeds, and is stored in three copies. It has good fault tolerance and read and write speeds, and can provide good access speeds for occasionally accessed warm files.

[0145] In the above process, some terms related to computer hardware are mentioned, which are explained as follows:

[0146] SSD (Solid State Drive) is a storage device that uses flash memory technology to store data. It usually uses flash memory chips as storage media. These chips can store data through NAND or MLC technology.

[0147] The data reading method used by SSD solid state drives is random reading, rather than the sequential reading method of mechanical hard drives, which means that SSD solid state drives can quickly access stored data, making operations such as starting the operating system, opening files and applications faster.

[0148] M.2 solid state drive (M.2SSD) is a solid state drive that uses the M.2 interface. The M.2 interface is a new type of storage device interface that can provide faster transmission speeds and higher performance;

[0149] M.2 SSDs usually adopt single-sided or double-sided design. The single-sided design is smaller in size and suitable for compact computer systems or mobile devices, while the double-sided design can provide higher storage capacity. In addition, M.2 SSDs also support a variety of flash memory technologies, including TLC, MLC and SLC, as well as a variety of storage controllers and cache technologies, which can improve data read and write speeds and reliability.

[0150] A mechanical hard disk (Hard Disk Drive) is a non-volatile storage device based on a hard rotating disk used in computers. It stores and retrieves digital data on a flat magnetic surface. Information passes through a magnetic head very close to the magnetic surface, and is written to the disk by an electromagnetic current that changes its polarity. At the same time, information can be read in the opposite way. For example, when the read head passes over the recorded data, the magnetic field causes the electrical signal in the coil to change. Its reading and writing uses random access, so the data in the hard disk can be read in any order.

[0151] The internal structure of a mechanical hard disk includes one or several high-speed rotating magnetic disks and a magnetic head placed on the actuator cantilever. According to its implementation principle, it is not difficult to find that the working method of a modern mechanical hard disk is just as its name suggests, which is realized mechanically, that is, the electric motor drives the disk to rotate at high speed, and then uses the operation of the magnetic head to access data.

[0152] Working process:

[0153] Based on the above-mentioned method for determining the heat of file size and time attributes, the present invention provides the following embodiments as specific examples of the above-mentioned scheme:

[0154] Assume that there are ten files in the system, namely text1, text2, text3, ..., text10, and these ten files are accessed in sequence;

[0155] First, the timestamp information, file size and file type of the ten files are monitored and recorded in a file. Then, the final heat value is calculated using the above file heat value calculation method. Since the access interval of each file is relatively short, the heat value calculation results are not much different. The heat values ​​of text1, text2, text3, ... text10 are 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 8 respectively.

[0156] At this time, the disk space usage is automatically checked. If the disk usage rate exceeds 50%, it is considered that the space is tight, and the file size is increased as the basis for judging the file heat value. After the calculation of the above file size function, a new heat value result is obtained. For example, the file sizes of text1, text2, text3...text10 are 20KB, 2M, 3M, 4M, 5M, 6M, 7M, 8M, 100M, and 1G respectively. Then the final calculation result may be that the heat values ​​of text1, text2, text3...text10 are: 0.71, 3.6, 3.5, 3.4, 3.3, 3.2, 3.1, 3.0, 0.07, and 0.01 respectively;

[0157] Because the file sizes of text10 and text9 are too large, the heat values ​​are too low, and the file size of text1 is too small to be stored on a high-speed device. Then, the heat value results with the file size function added and the black and white lists are sorted;

[0158] If text1 is on the whitelist, the heat value of text1 will directly reach 100;

[0159] If text10 is on the blacklist, the heat value of text10 will be directly reduced to -100, and the files will be sorted from high to low as text1, text2, text3, text4, text5, text6, text7, text8, text9, text10;

[0160] Finally, according to the sorting results: text1, text2, and text3 will be placed in the hot pool, text4, text5, text6, text7, and text8 will be placed in the warm pool, and text8, text9, and text10 will be placed in the cold pool.

[0161] If the disk usage rate does not exceed 50%, the file size is not considered and the files are sorted according to the heat value and the blacklist and whitelist calculated above. The sorting result from high to low is text10, text9, text8, text7, text6, text5, text4, text3, text2, text1;

[0162] Among them, text1, text2, and text3 will be placed in the cold pool, text4, text5, text6, text7, and text8 will be placed in the warm pool, and text8, text9, and text10 will be placed in the hot pool.

[0163] Now, assuming that the disk usage rate does not exceed 50%, 10 files are accessed multiple times, where the number of accesses to text1, text2, text3, ..., text10 are 90, 80, ..., 10, 0 times respectively. The timestamp information, file size, and file type of the ten files are monitored again and recorded in the above files;

[0164] By using the above file heat value calculation method to calculate the final heat value, we can see that the heat value calculation results of text1, text2, text3...text10 are: 10, 9.8, 9.6, 9.5, 9.4, 9.2, 9.1, 9, 8.8, 8;

[0165] The reason why the heat value of file text10 decreases instead of increases is that in these multiple accesses, text10 is the only one that has not been accessed. As the difference between the timestamp and the current time increases, the heat value of text10 will naturally decrease. Sort by the calculation result of the updated heat value;

[0166] The final sorting results of the files from high to low are text1, text2, text3, text4, text5, text6, text7, text8, text9, text10;

[0167] At this time, text1, text2, and text3 originally in the cold pool will be migrated to the hot pool, and text8, text9, and text10 in the hot pool will be migrated to the cold pool. Files text4, text5, text6, text7, and text8 will not be migrated.

[0168] In the above process of migrating files to the cold, hot and warm pools according to the heat value, the migration process of the files is recorded through the migration log, including successful migration, failed migration or the file has been stored in the target pool.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. Heat identification and data migration method based on file size and time attributes, It is characterized in that The steps include: Obtaining file information of each file, wherein the file information includes at least a file timestamp, a file size, and a file type; Establishing a decay function according to the acquired file timestamp, setting a weight value for the decay function according to the file type, and calculating a first heat value using the decay function containing the weight value; An attenuation factor is calculated using a preset attenuation strategy, and a second heat value is calculated using the attenuation factor and the first heat value; Setting a storage threshold according to the storage space size of the disk, and adjusting the second heat value according to a comparison result between the storage threshold and the file size; The file information also includes a blacklist and a whitelist, and the adjusted second heat value is adjusted again according to the blacklist and a whitelist to obtain a final heat value; According to the final heat value, each file is stored in different storage spaces through different backup methods.

2. The method for heat identification and data migration based on file size and time attributes according to claim 1, It is characterized in that The process of establishing the attenuation function includes: The difference between the current time and the file timestamp is calculated to obtain the time difference, and a decay function is established using the time difference and a preset decay rate; wherein the file timestamp includes an access time, a creation time, and a modification time, and the time difference is the difference between the current time and any file timestamp.

3. The method for heat identification and data migration based on file size and time attributes according to claim 2, It is characterized in that The process of setting a weight value for the attenuation function according to the file type and calculating the first heat value using the attenuation function containing the weight value includes: Classifying files into different types according to their access and modification conditions, and presetting weight values ​​according to the file types, wherein three weight values ​​are provided, and the three weight values ​​correspond to three different file timestamps respectively; The decay function established according to the file timestamp is multiplied by the corresponding weight value and the sum is accumulated to obtain the first heat value.

4. The method for heat determination and data migration based on file size and time attributes according to claim 1, Features: When a file is accessed or modified, the corresponding access time or modification time is updated, and the changed file information is stored. The file is preset with a number of storage entries. If the number of changes in the file's access time exceeds the number of storage entries, the file information with the earliest access time is deleted and the current file information is stored.

5. The method for heat determination and data migration based on file size and time attributes according to claim 4, It is characterized in that The process of calculating the attenuation factor by using the preset attenuation strategy includes: The timestamp corresponding to the last update of the file is found, and the time difference between the current time and the last update is calculated, and the attenuation factor is established according to the time difference and the preset time standard.

6. The method for heat identification and data migration based on file size and time attributes according to claim 1, It is characterized in that The process of calculating the second heat value using the attenuation factor and the first heat value includes: The heat value of the last update of the file is calculated using a decay function containing a weight value, and the product of the decay factor and the heat value of the last update of the file is added to the first heat value to obtain a second heat value.

7. The method for heat determination and data migration based on file size and time attributes according to claim 1, It is characterized in that The process of adjusting the second heat value according to the comparison result between the storage threshold and the file size includes: Infer the storage space usage of files based on their size; If the storage space occupancy rate of the file is greater than or equal to the storage threshold, the heat value is adjusted according to the file size; If the storage space occupancy rate of the file is lower than the storage threshold, the heat value is not changed.

8. The method for heat determination and data migration based on file size and time attributes according to claim 7, It is characterized in that The process of adjusting the heat value according to the file size includes: Divide the file size into four different preset values, and establish a piecewise function according to the four preset values, wherein the piecewise function includes four file preset functions; If the file size is smaller than the first preset value or larger than the fourth preset value, the second heat value is adjusted using the first file preset function; If the file size is greater than the first preset value and less than the second preset value, the second heat value is adjusted using the second file preset function; If the file size is greater than the second preset value and less than the third preset value, the second heat value is adjusted using the third file preset function; If the file size is greater than the third preset value and less than the fourth preset value, the second heat value is adjusted using a fourth file preset function, wherein the fourth file preset function is a constant function.

9. The method for heat identification and data migration based on file size and time attributes according to claim 1, It is characterized in that The process of performing a secondary adjustment on the adjusted second heat value according to the black and white list to obtain the final heat value includes: Get the blacklist and whitelist of the file. If the file is in the whitelist, increase the heat value of the file; if the file is in the blacklist, decrease the heat value of the file.

10. The method for heat determination and data migration based on file size and time attributes according to claim 1, It is characterized in that The process of storing each file in different storage spaces in different backup modes according to the final heat value includes: According to the final heat value, each file is divided into files of different heats, wherein the files of different heats include cold files, warm files, and hot files; If the current file is the cold file, it is stored in a low-speed disk for backup through erasure coding; If the current file is the warm file, it is stored in a medium-speed disk in a second copy for backup; If the current file is the hot file, it is stored in a high-speed disk in three copies for backup.