A hierarchical storage method for predicting and optimizing data classification

Abstract

The present invention discloses a hierarchical storage method for predicting and optimizing data classification, which is applied to a hierarchical storage system for predicting and optimizing data classification. The hierarchical storage system for predicting and optimizing data classification includes a heat monitoring module, a single-cycle data arrangement module, a future-cycle activity prediction module, and a data migration module. This hierarchical storage method uses the heat monitoring module to collect the activity information of data blocks within multiple cycles, and the activity information includes multiple factors, so that the accuracy of the cold and heat evaluation of each data block is higher. The judgment module ensures the number of activity score files, thereby reducing the error in the cold and heat evaluation of each data. The future-cycle activity module predicts the future-cycle activity scores of each data, improving the accuracy of each data classification. Then, the data migration module is used to perform data migration to achieve the classified storage of each data according to cold and heat, thereby improving the utilization rate of the high-performance layer in the hierarchical storage system.

Description

Technical Field

[0001] The present invention belongs to the field of data storage, and particularly relates to a hierarchical storage method for optimizing data classification by prediction. Background Art

[0002] When an enterprise is developing rapidly, the amount of data that needs to be saved is increasing. However, not all the stored data has very high usage value. In fact, the storage system resources will be occupied by a large amount of data with low usage value, resulting in the impact on the performance of the storage system and the waste of high-cost storage space. For the problem of storing a large amount of low-access data brought about by the explosive growth of data, hierarchical storage is a good solution. Its idea is to mix SSD and HDD in the same storage device, divide the data according to the usage value of the data, and then store it in storage devices with different access speed levels. The reading and writing of frequently used data, that is, hot data, can be accessed more efficiently and quickly, and cold data can not occupy the space of high-performance disks.

[0003] In the prior art, data is generally divided into cold data or hot data according to the number of access times in the past period of time, without considering that the information validity of data in historical access gradually decreases. That is, for two pieces of data, if one piece of data was read and written a long time ago, and the other piece of data was accessed most recently, when dividing cold and hot data, if the number of read and write clicks of the two pieces of data is the same, both pieces of data will be uniformly divided into hot data, which is obviously inaccurate. Summary of the Invention

[0004] The purpose of the present invention is to propose a hierarchical storage method for optimizing data classification by prediction to solve the problems raised in the background art.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A hierarchical storage method for optimizing data classification by prediction proposed by the present invention is applied to a hierarchical storage system for optimizing data classification by prediction. The hierarchical storage system for optimizing data classification by prediction includes a heat monitoring module, a single-cycle data arrangement module, a future-cycle activity prediction module, and a data migration module. The hierarchical storage method for optimizing data classification by prediction includes:

[0007] The heat monitoring module collects the activity information of each data block in the hierarchical volume within a single monitoring cycle, statistically forms a data read-write record file, and sends the data read-write record file to the single-cycle data arrangement module.

[0008] The single-cycle data arrangement module receives the read-write record file sent by the heat monitoring module, calculates the activity score of each data block through the activity score formula, and statistically forms an activity score file. The activity score formula is as follows:

[0009] score sum = a0×(rs0 + ws0) + a1×(rs1 + ws1) +... + a n ×(rs n + ws n );

[0010] Wherein:

[0011] a i = e -(βΔt) ;

[0012] rs i = readHit × hitScore × readMultiplier;

[0013] ws i = writeHit × hitScore × writeMultiplier;

[0014] Wherein: score sum represents the activity score of a single data block, a i ×(rs i + ws i ) represents the activity information score calculated for each read and write access operation of the data block during a single monitoring period, a i represents the decay factor, β represents the decay factor constant, Δt represents the decay duration, rs i represents the score of the read data block, ws i represents the score of the write data block, readMultiplier and writeMultiplier represent the read operation weight and the write operation weight respectively, readHit and writeHit represent the number of reads and the number of writes respectively, and hitScore represents the score of the number of clicks.

[0015] The future cycle activity prediction module predicts the future cycle activity scores of each data block in the storage level according to the activity score file through a prediction algorithm, divides the storage levels to which each data block belongs according to the policy, and sends them to the data migration module.

[0016] The data migration module migrates each data block to the corresponding storage level according to the storage level division result of the data block sent by the future cycle activity prediction module.

[0017] Preferably, the activity information includes the read click count, write click count, operation interval time, and read and write operation weights of the data block.

[0018] Preferably, the hierarchical storage system for predicting and optimizing data classification further includes a judgment module. The judgment module receives the activity score file sent by the single-cycle data layout module and makes the following judgments based on the number of files:

[0019] When the number of activity score files is greater than or equal to a preset value, the judgment module sends the activity score file to the future-cycle activity prediction module, and the future-cycle activity prediction module predicts the future-cycle activity scores of each data block;

[0020] When the number of activity score files is less than the preset value, the judgment module does not send the activity score file, and the heat monitoring module continues to collect the activity information of each data block in the next cycle.

[0021] Preferably, the storage level includes a high-performance layer, a performance layer, and a capacity layer.

[0022] Preferably, the hierarchical storage system for predicting and optimizing data classification further includes an inspection configuration module. The inspection configuration module checks whether the system environment is normal, whether the heat monitoring module is normal, whether the hierarchical volume exists, and whether there is sufficient storage space at the storage level.

[0023] Preferably, the future-cycle activity prediction module divides the data blocks in each layer of the storage level into hot data, relatively hot data, and cold data according to the future-cycle activity scores.

[0024] Preferably, the prediction algorithm is the least squares method.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] This hierarchical storage method uses a heat monitoring module to collect the activity information of data blocks in multiple cycles, and the activity information includes multiple factors, so the accuracy of the cold and heat evaluation of each data block is higher. The judgment module ensures the number of activity score files, thereby reducing the error in the cold and heat evaluation of each data. The future-cycle activity module predicts the future-cycle activity scores of each data, improving the accuracy of each data classification. Then, through the data migration module, data migration is performed to realize the classification storage of each data block according to cold and heat, thereby improving the utilization rate of the high-performance layer in the hierarchical storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a module block diagram of the hierarchical storage method for predicting and optimizing data classification of the present invention;

[0028] Figure 2 It is a flowchart of the hierarchical storage method for predicting and optimizing data classification of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0030] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there may also be an intermediate component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application herein are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0031] In one embodiment, as Figure 1-2 shown, a hierarchical storage method for predicting and optimizing data classification is applied to a hierarchical storage system for predicting and optimizing data classification. The hierarchical storage system for predicting and optimizing data classification includes a heat monitoring module, a single-cycle data arrangement module, a future-cycle activity prediction module, and a data migration module. The hierarchical storage method for predicting and optimizing data classification includes:

[0032] Step S1: The hierarchical storage system for predicting and optimizing data classification includes an inspection configuration module. The inspection configuration module checks whether the system environment is normal, whether the heat monitoring module is normal, whether the hierarchical volume exists, and whether there is sufficient storage space at the storage level.

[0033] It should be noted that, as Figure 1 , this embodiment is implemented based on the LVM (Logical Volume Manager) architecture. This software architecture is a very mature and stable architecture for managing disk partitions under the Linux system. Its external manifestation is to abstract the disk into a volume and provide it to the user. By creating a file system on the volume, files can be normally created, saved, deleted, etc. on this volume. In this embodiment, the inspection configuration module first checks the hierarchical storage system environment to see if the configuration parameters are correct to ensure the normal operation of the hierarchical storage function. It also includes checking whether the monitoring function of the heat monitoring module is available, whether the hierarchical volume to be monitored exists, and whether there is sufficient storage space at the storage level.

[0034] Step S2: The heat monitoring module collects the activity information of each data block in the hierarchical volume within a single monitoring cycle, statistically forms a data read / write record file, and sends the data read / write record file to the single-cycle data arrangement module.

[0035] In this embodiment, a tiered volume is a type of logical volume (LV, Logical Volume), which refers to a logical volume composed of spaces occupying different levels of hard disks. The tiered volume stores data blocks in the storage level according to different storage media. The storage level includes a high-performance layer, a performance layer, and a capacity layer. The heat monitoring module collects the activity information of each data block in each layer within a cycle (that is, the heat monitoring module collects the data blocks in the high-performance layer, the performance layer, and the capacity layer respectively), statistically outputs a data read / write record file, and grasps the read / write access situation of the upper-layer application to the tiered volume in real time through the data read / write record file, so as to judge the cold and heat degrees of each data block in each layer. The heat monitoring module also sends the data read / write record file to the single-cycle data layout module, which is used to quantify the activity information of each data block in each layer into an activity score for comprehensive evaluation. It should be noted that the activity score is a weighted cumulative value, that is, a value that takes into account various meaningful measurable factors and is updated in real time. These factors include the read click count, write click count, operation interval time, and read / write operation weight of the data block. Quantify various factors affecting the cold and heat degrees of data blocks to form an activity score for comprehensive evaluation, improving the accuracy of cold and hot data classification.

[0036] Step S3: The single-cycle data layout module receives the read / write record file sent by the heat monitoring module, calculates the activity scores of each data block in the storage level through the activity score formula, and statistically forms an activity score file.

[0037] In this embodiment, the single-cycle data layout module converts the activity information of the data blocks monitored by the heat monitoring module within a cycle into the activity scores of each data block. That is, after each cycle ends, the activity scores of each data block are calculated through the activity score formula, statistically output an activity score file, and send the activity score file to the judgment module.

[0038] The activity score formula is as follows:

[0039] score sum =a0×(rs0+ws0)+a1×(rs1+ws1)+...+a n ×(rs n +ws n );

[0040] Where:

[0041] a i =e -(βΔt) ;

[0042] rs i =readHit×hitScore×readMultiplier;

[0043] wsi = writeHit × hitScore × writeMultiplier;

[0044] Where: score sum represents the activity score of a single data block; a i × (rs i + ws i ) represents the score of the activity information for each read or write click of the data block within a single monitoring period; a i represents the decay factor, indicating that at a certain moment during monitoring, a data block will record a historical activity information due to read and write operations, and the activity score score(i) corresponding to this activity information will gradually decrease as the monitoring cumulative time increases, or in other words, the effectiveness of the historical access information of the data block will gradually decrease; β represents the decay factor constant, which is the set data value in the configuration file, and the default value is 1 / (3 × 24 × 60 × 60) = 0.00000385802, representing the decay amount in 3 days (it can also be set to the decay amount in one day: 1 / (1 × 24 × 60 × 60)); Δt represents the decay duration and operation interval time, that is, the difference between the time when this read and write access operation is generated recorded in the activity score file and the current time, with the unit of seconds; rs i represents the score of reading the data block, ws i represents the score of writing the data block; readMultiplier and writeMultiplier respectively represent the read operation weight and write operation weight, that is, the amplification multiples of reading and writing. The weights of reading or writing can be set according to the actual situation. The higher the weight, the higher the priority. The data blocks with frequent reads or writes are preferentially stored in a higher storage level, so that when these data have access requirements, they can be accessed more efficiently. In different scenarios, more appropriate read and write amplification multiples can be set. For example, when hierarchical storage is applied to storing data on a movie video website or other application scenarios that prioritize read access, the read weight can be appropriately increased; when hierarchical storage is applied to storing data in archives, log systems, etc., which prioritize frequently written data, the write weight can be appropriately increased; readHit and writeHit respectively represent the number of reads and the number of writes, hitScore represents the score of the number of clicks, and the number of clicks represents how many bit operations occur in a read and write operation, that is, how many times of clicks. The value of hitScore can be set according to actual needs.

[0045] Step S4: The hierarchical storage system for predicting and optimizing data classification further includes a judgment module. The judgment module receives the activity score file sent by the single-cycle data arrangement module and makes the following judgments according to the number of files:

[0046] When the number of activity score files is greater than or equal to a preset value, the judgment module sends the activity score files to the future cycle activity prediction module, and the future cycle activity prediction module predicts the future cycle activity scores of each data block;

[0047] When the number of activity score files is less than the preset value, the judgment module does not send the activity score files, and the heat monitoring module continues to collect the activity information of each data block in the next cycle.

[0048] In this embodiment, a preset value is set for this hierarchical storage system, which is used to compare the preset value with the number of files received by the judgment module. The judgment module ensures the number of activity score files, thereby ensuring the accuracy of the cold and heat evaluation of each data block.

[0049] Step S5: The future cycle activity prediction module predicts the future cycle activity scores of each data block in the storage level according to the activity score files through a prediction algorithm, divides the storage levels to which each data block belongs according to the policy, and sends them to the data migration module.

[0050] In this embodiment, the prediction algorithm is the least squares method, but it is not limited to the least squares method, and other prediction algorithms can also be used. The future cycle activity prediction module uses the least squares method to generate the future cycle activity scores of each predicted data block, and then divides the data blocks in each layer into hot data, relatively hot data, and cold data according to the policy, that is, the data blocks in the high-performance layer are divided into hot data, relatively hot data, and cold data, and the data blocks in the performance layer are divided into hot data, relatively hot data, and cold data, and the data blocks in the capacity layer are divided into hot data, relatively hot data, and cold data. The policy in this embodiment is to divide the data blocks in each layer according to the percentage according to the level of the future cycle activity score. For example, the data blocks are arranged according to the future cycle activity score from high to low, the first 20% are hot data, the last 30% are cold data, and the middle 50% are relatively hot data. The division policy of the data blocks in each layer is not limited to the percentage method, and other policies can also be used for division.

[0051] Step S6: The data migration module migrates each data block to the corresponding storage level according to the division result of the storage level to which the data block belongs sent by the future cycle activity prediction module.

[0052] In this embodiment: When the data migration module performs migration, for the performance layer: hot data is migrated to a relatively higher layer (i.e., the high-performance layer), relatively hot data is retained in the current layer (i.e., the performance layer), and cold data is migrated to a relatively lower layer (i.e., the capacity layer); for the high-performance layer: since there is no higher layer, hot data is retained in the current layer (i.e., the high-performance layer), relatively hot data is retained in the current layer (i.e., the high-performance layer), and cold data is migrated to a relatively lower layer (i.e., the performance layer); for the capacity layer: since there is no lower layer, cold data is retained in the current layer (i.e., the capacity layer), relatively hot data is retained in the current layer (i.e., the capacity layer), and hot data is migrated to a relatively higher layer (i.e., the performance layer).

[0053] The storage-level division in the hierarchical volume is shown in the following table:

[0054]

[0055] In another embodiment, the simulation experiment uses a two-layer hierarchical storage system with a total of two levels.

[0056] First, record files of activity scores in different periods are backed up, and then n periods of data blocks are extracted from the set of data blocks in each period after preprocessing, where n is greater than or equal to the dimension of the weight vector W in the least squares method. Using the activity scores of the data blocks in n periods in the data window as known information, and taking the activity score of the data blocks in the (n + 1)-th period that has been recorded as the actual activity score, which is used as reference data. Then, using the activity score of the n-th period directly as the result recorded by the naive method of the hot and cold data classification method as the first group of data; and using the result of the activity score of the (n + 1)-th period obtained through the least squares operation as the second group of data. By analyzing the comparison between the hot data obtained by the two prediction methods and the hot data in the parameter data, these two prediction methods are evaluated.

[0057] To evaluate the similarity between the classification results of the two prediction methods and the reference group, two quantitative indicators are used here. The first quantitative indicator is the selection accuracy P of hot data blocks b , and its calculation formula is as follows:

[0058]

[0059] Among them, N real represents the number of actual hot data blocks measured in the (n + 1)-th period, while N same represents the number of elements in the intersection of the set of hot data blocks obtained by using a certain prediction method (naive method or least squares method) and the set of actual hot data blocks, that is, the number of hot data that coincides with the actual hot data for a certain prediction method. Therefore, the ratio P of N same to N real b ​That is the correct rate of hot data block selection for a certain prediction method.

[0060] The second quantization index is the total accuracy rate P of the number of hot data block accesses c , and the calculation formula is as follows:

[0061]

[0062] Among them, S real represents the total number of accesses corresponding to the logical number of hot data blocks in the (n + 1)-th cycle, while S predict represents the total number of accesses corresponding to the serial numbers of the hot data blocks selected by a certain prediction method. Therefore, the ratio P predict of S real to S c is the total accuracy rate of the number of hot data block accesses for a certain prediction method.

[0063] Through 6 simulation experiments of testing, each time replacing the backup data of multiple new cycles, and respectively selecting the cases of W dimensions 3, 5, 7, 8, 9, and 10, the accuracy rate P b of hot data block selection and the total accuracy rate P c . of the number of hot data block accesses are as shown in the following table:

[0064]

[0065]

[0066] In the above table, by comparing the hot data blocks in the (n + 1)-th cycle obtained by the naive method with the actual (n + 1)-th cycle of hot data blocks, it can be confirmed that hot data actually has temporal correlation. Even if the previous activity information is cleared, the files or data in the system that are frequently accessed by users or the system will still be considered as hot data multiple times. Therefore, it is completely necessary to predict the data or, on the other hand, to perform stratification. It should be noted that even if the data blocks corresponding to the data that have been migrated to the upper layer obtain a very high activity score again, they will not be used as data blocks to be migrated. In addition, when the vector dimension W = 8, the accuracy rate P b of the serial number selection of hot data blocks by the least squares method is 6.7% higher than that of the naive method, but P c is 0.3%, indicating that for the total access volume of classified hot data, the performance of the two methods is similar. When the vector dimension W = 9, the accuracy rate P b of the serial numbers of the hot data blocks classified by the least squares method is 91.1%, and the accuracy rate P c of the hot data access volume reaches 99.1% of the theoretical optimal value. At this time, the total access volume accuracy rate P cIt is 4.4% higher than the set selected by the naive method, closer to the theoretical optimal layering result of the hierarchical storage system, and improves the utilization rate of high-performance storage media in the hierarchical storage system. When the vector dimension W = 10, the accuracy is further improved, but the improvement is small, while using a higher vector dimension W will occupy more computing resources.

[0067] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0068] The above-described embodiments only express the embodiments of the present application that are described more specifically and in detail, but should not be construed as limiting the scope of the patent application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A hierarchical storage method for predicting and optimizing data classification, applied to a hierarchical storage system for predicting and optimizing data classification, characterized in that: The hierarchical storage system for predicting and optimizing data classification includes a heat monitoring module, a single-cycle data arrangement module, a future-cycle activity prediction module, and a data migration module. The hierarchical storage method for predicting and optimizing data classification includes: The heat monitoring module collects the activity information of each data block in the hierarchical volume within a single monitoring cycle, statistically forms a data read / write record file, and sends the data read / write record file to the single-cycle data arrangement module; The single-cycle data arrangement module receives the read / write record file sent by the heat monitoring module, calculates the activity score of each data block through the activity score formula, and statistically forms an activity score file. The activity score formula is as follows: score sum = a0×(rs0 + ws0) + a1×(rs1 + ws1) +... + a n ×(rs n + ws n ); Where: a i = e -(βΔt) ; rs i = readHit × hitScore × readMultiplier; ws i = writeHit × hitScore × writeMultiplier; Among them: score sum represents the activity score of a single data block, a i ×(rs i +ws i ) represents the activity information score calculated for each read and write access operation of the data block during a single monitoring period. a i represents the decay factor, β represents the decay factor constant, Δt represents the decay duration, rs i represents the score of the read data block, ws i represents the score of the written data block. readMultiplier and writeMultiplier represent the read operation weight and write operation weight respectively. readHit and writeHit represent the number of reads and writes respectively. hitScore represents the score of the number of clicks; The future-cycle activity prediction module predicts the future-cycle activity scores of each data block in the storage level according to the activity score file, divides the storage levels to which each data block belongs according to the policy, and sends them to the data migration module; The data migration module migrates each data block to the corresponding storage level according to the division result of the storage levels to which the data blocks belong sent by the future-cycle activity prediction module; Among them, the hierarchical storage system for predicting and optimizing data classification further includes a judgment module. The judgment module receives the activity score file sent by the single-cycle data arrangement module and makes the following judgment according to the number of files: When the number of activity score files is greater than or equal to the preset value, the judgment module sends the activity score file to the future-cycle activity prediction module, and the future-cycle activity prediction module predicts the future-cycle activity scores of each data block; When the number of activity score files is less than the preset value, the judgment module does not send the activity score file, and the heat monitoring module continues to collect the activity information of each data block in the next cycle.

2. The hierarchical storage method for predicting and optimizing data classification according to claim 1, wherein: The activity information includes the read click count, write click count, operation interval time, and read / write operation weight of the data block.

3. The hierarchical storage method for predicting and optimizing data classification according to claim 1, wherein: The storage levels include a high-performance layer, a performance layer, and a capacity layer.

4. The hierarchical storage method for predicting and optimizing data classification according to claim 1, characterized in that: The hierarchical storage system for predicting and optimizing data classification further includes an inspection configuration module. The inspection configuration module checks whether the system environment is normal, whether the heat monitoring module is normal, whether the hierarchical volume exists, and whether there is sufficient storage space in the storage levels.

5. The hierarchical storage method for predicting and optimizing data classification according to claim 1, characterized in that: The future-cycle activity prediction module divides the data blocks in each layer of the storage level into hot data, relatively hot data, and cold data according to the future-cycle activity scores.

6. The hierarchical storage method for predicting and optimizing data classification according to claim 1, characterized in that: The prediction algorithm is the least squares method.

Images