Method for intelligently prefetching data of storage system

Abstract

The invention relates to the field of storage data processing, and discloses a method for intelligently prefetching data of a storage system, which comprises the following steps of: constructing a data prefetching algorithm pool and a prefetching algorithm selection table; according to the IO request, the data prefetching algorithm manager searches a prefetching algorithm at the current moment in a data prefetching algorithm pool, if the prefetching algorithm at the current moment exists, the prefetching algorithm with the closest time is selected or data of adjacent LBAs before and after the current IO is prefetched, and if the prefetching algorithm at the current moment does not exist, the data prefetching algorithm with the similar time is selected; the data is prefetched according to the selected data prefetching algorithm, the IO hit rate is obtained, the data prefetching algorithm manager updates the IO prefetching hit rate of the prefetching algorithm according to the minimum time scale in different time scales, the process data and the selection result are updated to the prefetching algorithm selection table, and intelligent data prefetching of the storage system is completed. Dynamic adaptation is carried out through the data prefetching algorithm selector, and all scene requirements can be met.

Description

Technical Field

[0001] The invention relates to the field of storage data processing, and in particular to a method for intelligent pre-fetching of storage system data. Background Art

[0002] In the process of accessing stored data, a key factor that determines the data reading performance is the effectiveness of pre-fetching the stored data into the memory. This is because the access latency of stored data on the hard disk and the access latency of the memory are orders of magnitude different. The hard disk access latency is generally in milliseconds, while the memory access latency is generally in nanoseconds. How to pre-read data into the memory before the application accesses it can greatly improve the running speed of the application.

[0003] Current traditional storage systems generally simply target a read IO. In addition to reading the current read data into the memory, they also read some data around the IO address into the memory. This data may be required for the next IO access and can be directly returned when the memory hits, improving the storage access performance.

[0004] The first read IO latency will be very high, but the next read IO latency will be very low. However, the current data pre-fetching algorithm has a great limitation, that is, it is difficult to adjust in time according to the changes in business conditions. For example, the following two IO access scenarios require different pre-fetching strategies.

[0005] In the IO continuous LBA access scenario, you only need to pre-fetch the LBA3 data into the memory when IO1 accesses LBA2, so that IO2 can directly hit the data in the memory when accessing.

[0006] In the scenario where IO is accessed in LBA sequential intervals, the above prefetching algorithm will not work. The LBA3 data prefetched when IO1 accesses is not what IO2 needs.

[0007] In more complex situations, such as the access scenario where IO is an LBA jump, the data to be pre-fetched is not only intermittent but also non-sequential, so the current data pre-fetching algorithm is more difficult to adapt.

[0008] In specific application access scenarios, the IO access model for data is different in different time periods, so there will be a problem that a fixed data prefetching algorithm cannot meet the needs of all scenarios. Summary of the invention

[0009] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for intelligent pre-fetching of data in a storage system, comprising the following steps:

[0010] Step 1: construct a data pre-fetching algorithm pool and a pre-fetching algorithm selection table;

[0011] Step 2: According to the IO request, the data prefetching algorithm manager searches the data prefetching algorithm pool for the current prefetching algorithm. If the current prefetching algorithm exists, the process proceeds to step 3. If not, the data prefetching algorithm with a similar time is selected and the process proceeds to step 4.

[0012] Step 3: Select the prefetch algorithm with the closest time or prefetch the data of adjacent LBAs before and after the current IO;

[0013] Step 4: Prefetch data according to the selected data prefetch algorithm and obtain the IO hit rate. The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, updates the process data and selection results to the prefetch algorithm selection table, and completes the intelligent prefetching of storage system data.

[0014] Furthermore, the construction of the data pre-fetching algorithm pool and the pre-fetching algorithm list includes:

[0015] The data prefetch algorithm pool includes prefetch algorithms corresponding to different time scales and IO prefetch hit rates; the prefetch algorithm list includes different time scales and IO prefetch hit rates corresponding to the prefetch algorithms corresponding to the different time scales.

[0016] Further, the data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, and updates the process data and the selection result to the prefetch algorithm selection table, including:

[0017] The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale, and determines whether the IO prefetch hit rate of the prefetch algorithm reaches the preset threshold. If so, the IO prefetch hit rate of the prefetch algorithm is updated. Otherwise, the IO prefetch hit rate of the prefetch algorithm of the next time scale is obtained according to different time scales from small to large. If it is reached, the prefetch algorithm is changed to select the prefetch algorithm of the next time scale. When the prefetch algorithm of the maximum time scale is adjusted, if the IO prefetch hit rate of the prefetch algorithm still cannot reach the preset threshold, the prefetch algorithm with the largest IO prefetch hit rate among the prefetch algorithms corresponding to different time scales is selected and updated to the prefetch algorithm selection table.

[0018] Furthermore, the data pre-fetching algorithm training includes the following process:

[0019] By collecting the LBA of IO and training it through the LSTM network, the LSTM network contains: c is the memory unit state, h is the hidden state; x is the input data, that is, the LBA information of the IO in the current period of time; y is the output data, that is, the LBA information of the data that needs to be pre-fetched; collect the LBA data of IO in the corresponding time period, and output the corresponding pre-fetch algorithm model through training; through the pre-fetch algorithm model, in the same time period, input the LBA information of IO in the current period of time, and predict the LBA information of the next IO.

[0020] The beneficial effects of the present invention are:

[0021] 1. Continuously collect IO LBA data and train the data pre-fetching algorithm according to the time dimension through the LSTM network. The dimension can be based on hour, day, week, or month;

[0022] 2. Dynamically adapt the data prefetching algorithm selector, continuously monitor the data hit rate of the currently selected prefetching algorithm, and select the optimal prefetching algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a method for intelligent pre-fetching of data in a storage system;

[0024] Figure 2 A schematic diagram of a data storage system for a method for intelligent pre-fetching of application data;

[0025] Figure 3 A schematic diagram of a flow chart for selecting an algorithm for a data prefetching algorithm manager;

[0026] Figure 4 A schematic diagram of a process for evaluating and selecting a prefetch algorithm for a data prefetch algorithm manager;

[0027] Figure 5 Expanded diagram of the LSTM model. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0029] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0030] like Figure 1 As shown, a method for intelligent pre-fetching of data in a storage system comprises the following steps:

[0031] Step 1: construct a data pre-fetching algorithm pool and a pre-fetching algorithm selection table;

[0032] Step 2: According to the IO request, the data prefetching algorithm manager searches the data prefetching algorithm pool for the current prefetching algorithm. If the current prefetching algorithm exists, the process proceeds to step 3. If not, the data prefetching algorithm with a similar time is selected and the process proceeds to step 4.

[0033] Step 3: Select the prefetch algorithm with the closest time or prefetch the data of adjacent LBAs before and after the current IO;

[0034] Step 4: Prefetch data according to the selected data prefetch algorithm and obtain the IO hit rate. The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, updates the process data and selection results to the prefetch algorithm selection table, and completes the intelligent prefetching of storage system data.

[0035] The construction of the data pre-fetching algorithm pool and the pre-fetching algorithm list includes:

[0036] The data prefetch algorithm pool includes prefetch algorithms corresponding to different time scales and IO prefetch hit rates; the prefetch algorithm list includes different time scales and IO prefetch hit rates corresponding to the prefetch algorithms corresponding to the different time scales.

[0037] The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, and updates the process data and the selection result to the prefetch algorithm selection table, including:

[0038] The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale, and determines whether the IO prefetch hit rate of the prefetch algorithm reaches the preset threshold. If so, the IO prefetch hit rate of the prefetch algorithm is updated. Otherwise, the IO prefetch hit rate of the prefetch algorithm of the next time scale is obtained according to different time scales from small to large. If it is reached, the prefetch algorithm is changed to select the prefetch algorithm of the next time scale. When the prefetch algorithm of the maximum time scale is adjusted, if the IO prefetch hit rate of the prefetch algorithm still cannot reach the preset threshold, the prefetch algorithm with the largest IO prefetch hit rate among the prefetch algorithms corresponding to different time scales is selected and updated to the prefetch algorithm selection table.

[0039] Data prefetching algorithm training includes the following process:

[0040] By collecting the LBA of IO and training it through the LSTM network, the LSTM network contains: c is the memory unit state, h is the hidden state; x is the input data, that is, the LBA information of the IO in the current period of time; y is the output data, that is, the LBA information of the data that needs to be pre-fetched; collect the LBA data of IO in the corresponding time period, and output the corresponding pre-fetch algorithm model through training; through the pre-fetch algorithm model, in the same time period, input the LBA information of IO in the current period of time, and predict the LBA information of the next IO.

[0041] Specifically, LSTM (Long Short Term Memory networks) are recurrent networks that are widely used in time series prediction. This solution continuously collects IO LBA data, trains the data prefetching algorithm according to the time dimension through the LSTM network, and dynamically adapts the data prefetching algorithm selector to continuously monitor the data hit rate of the currently selected prefetching algorithm and select the optimal prefetching algorithm, which can effectively solve the problem that the fixed data prefetching algorithm cannot meet the needs of all scenarios. Figure 2 A schematic diagram of a data storage system using a method for intelligent pre-fetching of application data is shown.

[0042] The overall process of data prefetching is as follows: Figure 3 As shown:

[0043] When an IO request arrives, the data prefetch algorithm manager selects the corresponding prefetch algorithm based on time correlation, records the hit rate of the prefetch algorithm during execution, and automatically adjusts the selected prefetch algorithm.

[0044] The data prefetch algorithm manager will select the optimal prefetch algorithm based on time correlation and hit rate statistics. However, if there is no trained data prefetch algorithm at the initial operation of the storage system, the default adjacent prefetch algorithm will be returned, that is, the data of the adjacent LBAs before and after the current IO will be prefetched. At the same time, if there is no corresponding data prefetch algorithm in the current time range, the data prefetch algorithm with a similar time will be selected.

[0045] The data prefetch algorithm manager continuously updates the IO hit rate of the prefetch algorithm in hours to determine which prefetch algorithm has the best effect in the current time period, and updates the process data and selection results to the prefetch algorithm selection table. Figure 4 shown.

[0046] 2. Prefetch algorithm training process:

[0047] The training process of the data prefetching algorithm is to continuously collect the LBA of IO and train it through the LSTM network, and then output the corresponding prefetching algorithm model.

[0048] The LSTM model contains the following concepts: c is the memory cell state, h is the hidden state, x is the input data, i.e. the LBA information of the corresponding io in the current period, and y is the output data, i.e. the LBA information of the data to be pre-fetched. The training process is to collect the LBA data of the io in the corresponding time period. However, continuous training allows the LBA information of the io in the current period to be input in the same time period, and the LBA information of the next io can be relatively accurately predicted to ensure the hit rate of data pre-fetching. Figure 5 The expanded diagram of the LSTM model is shown.

[0049] Parameter description and corresponding calculation formula description:

[0050]

[0051]

[0052] Example of algorithm model training process:

[0053] 1. Collect the IO LBA information sequence [LBA1, LBA2, LBA3…, LBNn] within a period of time (hour, day, week, month);

[0054] 2. Assuming that the length of the LBA information input to the model for a period of time is input_size=j, and the length of the LBA information predicted to be output for the next period of time is output_size=k, then the training process is to take the values ​​of the collected IO LBA information sequence from small to large, such as taking a sequence of length k [LBA1, LBA2…, LBAj] and inputting it into the model, and outputting a sequence of length j [LBAj+1', LBAj+2'…, LBAj+k'];

[0055] 3. Compare [LBAj+1', LBAj+2'…, LBAj+k'] with the collected real sequence [LBAj+1, LBAj+2…, LBAj+k], calculate the loss value and perform backpropagation to update the parameters in the algorithm model.

[0056] This solution uses time series prediction technology to dynamically adjust the data pre-fetching algorithm and effectively ensure that the data pre-fetching algorithm meets the current business needs.

[0057] During the operation of current IT systems, the access model of APPs to storage generally has time correlation. For example, the financial system regularly performs batch analysis on users' business data. The weekly, monthly, and annual batch analysis operations have different storage IO access models.

Claims

1. A method for intelligent pre-fetching of data in a storage system, characterized in that: The steps include: Step 1: construct a data pre-fetching algorithm pool and a pre-fetching algorithm selection table; Step 2: According to the IO request, the data prefetching algorithm manager searches the data prefetching algorithm pool for the current prefetching algorithm. If the current prefetching algorithm exists, the process proceeds to step 3. If not, the data prefetching algorithm with a similar time is selected and the process proceeds to step 4. Step 3: Select the prefetch algorithm with the closest time or prefetch the data of adjacent LBAs before and after the current IO; Step 4: Prefetch data according to the selected data prefetch algorithm and obtain the IO hit rate. The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, updates the process data and selection results to the prefetch algorithm selection table, and completes the intelligent prefetching of storage system data.

2. A method for intelligent pre-fetching of storage system data according to claim 1, characterized in that: The construction of the data pre-fetching algorithm pool and the pre-fetching algorithm list includes: The data prefetch algorithm pool includes prefetch algorithms corresponding to different time scales and IO prefetch hit rates; the prefetch algorithm list includes different time scales and IO prefetch hit rates corresponding to the prefetch algorithms corresponding to the different time scales.

3. A method for intelligent pre-fetching of storage system data according to claim 2, characterized in that: The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, and updates the process data and the selection result to the prefetch algorithm selection table, including: The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale, and determines whether the IO prefetch hit rate of the prefetch algorithm reaches the preset threshold. If so, the IO prefetch hit rate of the prefetch algorithm is updated. Otherwise, the IO prefetch hit rate of the prefetch algorithm of the next time scale is obtained according to different time scales from small to large. If it is reached, the prefetch algorithm is changed to select the prefetch algorithm of the next time scale. When the prefetch algorithm of the maximum time scale is adjusted, if the IO prefetch hit rate of the prefetch algorithm still cannot reach the preset threshold, the prefetch algorithm with the largest IO prefetch hit rate among the prefetch algorithms corresponding to different time scales is selected and updated to the prefetch algorithm selection table.

4. A method for intelligent pre-fetching of storage system data according to claim 3, characterized in that: Data prefetching algorithm training includes the following process: By collecting the LBA of IO and training it through the LSTM network, the LSTM network contains: c is the memory unit state, h is the hidden state; x is the input data, that is, the LBA information of the IO in the current period of time; y is the output data, that is, the LBA information of the data that needs to be pre-fetched; collect the LBA data of IO in the corresponding time period, and output the corresponding pre-fetch algorithm model through training; through the pre-fetch algorithm model, in the same time period, input the LBA information of IO in the current period of time, and predict the LBA information of the next IO.

Images