A method for intelligent pre-fetching of storage system data

Abstract

The present invention relates to the field of storage data processing and discloses a method for intelligent prefetching of storage system data, comprising: constructing a data prefetching algorithm pool and a prefetching algorithm selection table; according to an IO request, a data prefetching algorithm manager searches the data prefetching algorithm pool for the current prefetching algorithm; if the current prefetching algorithm exists, the prefetching algorithm with the closest time is selected or data of adjacent LBAs before and after the current IO is prefetched; if the prefetching algorithm does not exist, a data prefetching algorithm with a similar time is selected, data is prefetched according to the selected data prefetching algorithm, and an IO hit rate is obtained; the data prefetching algorithm manager updates the IO prefetching hit rate of the prefetching algorithm according to the smallest time scale among different time scales, updates the process data and the selection result to the prefetching algorithm selection table, and completes intelligent prefetching of storage system data. The present invention performs dynamic adaptation through the data prefetching algorithm selector and can meet the needs of all scenarios.

Description

Technical Field

[0001] The present invention relates to the field of storage data processing, and in particular to a method for intelligent pre-fetching of storage system data. Background Art

[0002] During the storage data access process, a key factor determining data read performance is the effectiveness of pre-fetching stored data into memory. This is because the access latency of stored data on the hard disk and the memory is orders of magnitude different. Hard disk access latency is generally in milliseconds, while memory access latency is generally in nanoseconds. Pre-fetching data into memory before the application accesses it can greatly improve the application's running speed.

[0003] Current traditional storage systems generally simply store a read IO address in memory. In addition to reading the current read data into the memory, they also read some data around the IO address into the memory. This data may be required for the next IO access and can be directly returned in the memory, improving storage access performance.

[0004] The initial read I / O latency will be very high, but subsequent read I / O latency will be very low. However, current data prefetching algorithms have significant limitations. They are difficult to adjust in a timely manner as business changes occur. For example, the following two I / O access scenarios require different prefetching strategies.

[0005] In the LBA continuous access scenario, you only need to pre-fetch the LBA3 data into the memory when IO1 accesses LBA2, so that IO2 can directly hit the data in the memory when accessing.

[0006] IO is accessed in LBA sequential intervals. If the above prefetch algorithm is used, it will not work. The LBA3 data prefetched during IO1 access is not what IO2 needs.

[0007] In more complex situations, such as IO access scenarios with LBA jumps, the data to be prefetched may be not only intermittent but also non-sequential, and the current data prefetching algorithm is more difficult to adapt to.

[0008] In specific application access scenarios, the IO access model for data is different in different time periods, so there will be a problem that a fixed data prefetching algorithm cannot meet the needs of all scenarios. Summary of the Invention

[0009] The present invention aims to overcome the deficiencies of the prior art and provides a method for intelligently prefetching data in a storage system, comprising the following steps:

[0010] Step 1: Build a data prefetching algorithm pool and a prefetching algorithm selection table;

[0011] Step 2: Based on the IO request, the data prefetch algorithm manager searches the data prefetch algorithm pool for the current prefetch algorithm. If the current prefetch algorithm exists, the process proceeds to step 3. If not, the data prefetch algorithm with a similar time is selected and the process proceeds to step 4.

[0012] Step 3: Select the prefetch algorithm with the closest time or prefetch the data of the adjacent LBAs before and after the current IO;

[0013] Step 4: Prefetch data according to the selected data prefetch algorithm and obtain the IO hit rate. The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, updates the process data and selection results to the prefetch algorithm selection table, and completes the intelligent prefetching of storage system data.

[0014] Furthermore, the construction of the data prefetching algorithm pool and the prefetching algorithm list includes:

[0015] The data prefetch algorithm pool includes prefetch algorithms corresponding to different time scales and IO prefetch hit rates; the prefetch algorithm list includes different time scales and IO prefetch hit rates corresponding to the prefetch algorithms corresponding to different time scales.

[0016] Furthermore, the data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale among different time scales, and updates the process data and selection results to the prefetch algorithm selection table, including:

[0017] The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale, and determines whether the IO prefetch hit rate of the prefetch algorithm reaches the preset threshold. If so, the IO prefetch hit rate of the prefetch algorithm is updated. Otherwise, the IO prefetch hit rate of the prefetch algorithm of the next time scale is obtained according to different time scales from small to large. If it is reached, the prefetch algorithm is changed to select the prefetch algorithm of the next time scale. When the prefetch algorithm of the maximum time scale is adjusted, the IO prefetch hit rate of the prefetch algorithm still cannot reach the preset threshold, then the prefetch algorithm with the largest IO prefetch hit rate among the prefetch algorithms corresponding to different time scales is selected instead, and updated to the prefetch algorithm selection table.

[0018] Furthermore, data prefetching algorithm training includes the following process:

[0019] By collecting the LBA of IO and training it through the LSTM network, the LSTM network contains: c is the memory unit state, h is the hidden state; x is the input data, that is, the LBA information of the IO in the current period; y is the output data, that is, the LBA information of the data to be pre-fetched; the LBA data of IO is collected in the corresponding time period, and the corresponding pre-fetching algorithm model is output through training; through the pre-fetching algorithm model, the LBA information of IO in the current period is input in the same time period to predict the LBA information of the next IO.

[0020] The beneficial effects of the present invention are:

[0021] 1. Continuously collect I / O LBA data and use the LSTM network to train the data prefetching algorithm based on the time dimension. The dimension can be hourly, daily, weekly, or monthly.

[0022] 2. Dynamically adapt the data prefetching algorithm selector, continuously monitor the data hit rate of the currently selected prefetching algorithm, and select the optimal prefetching algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a method for intelligent pre-fetching of data in a storage system;

[0024] Figure 2 A schematic diagram of a data storage system for a method for intelligent prefetching of application data;

[0025] Figure 3 Schematic diagram of algorithm selection process for data prefetching algorithm manager;

[0026] Figure 4 A schematic diagram of the process of evaluating and selecting prefetch algorithms for a data prefetch algorithm manager;

[0027] Figure 5 Expanded diagram of the LSTM model. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0029] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0030] like Figure 1 As shown, a method for intelligent pre-fetching of data in a storage system includes the following steps:

[0031] Step 1: Build a data prefetching algorithm pool and a prefetching algorithm selection table;

[0032] Step 2: Based on the IO request, the data prefetch algorithm manager searches the data prefetch algorithm pool for the current prefetch algorithm. If the current prefetch algorithm exists, the process proceeds to step 3. If not, the data prefetch algorithm with a similar time is selected and the process proceeds to step 4.

[0033] Step 3: Select the prefetch algorithm with the closest time or prefetch the data of the adjacent LBAs before and after the current IO;

[0034] Step 4: Prefetch data according to the selected data prefetch algorithm and obtain the IO hit rate. The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the smallest time scale among different time scales, updates the process data and selection results to the prefetch algorithm selection table, and completes the intelligent prefetching of storage system data.

[0035] The construction of the data prefetching algorithm pool and prefetching algorithm list includes:

[0036] The data prefetch algorithm pool includes prefetch algorithms corresponding to different time scales and IO prefetch hit rates; the prefetch algorithm list includes different time scales and IO prefetch hit rates corresponding to the prefetch algorithms corresponding to different time scales.

[0037] The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale among different time scales, and updates the process data and selection results to the prefetch algorithm selection table, including:

[0038] The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale, and determines whether the IO prefetch hit rate of the prefetch algorithm reaches the preset threshold. If so, the IO prefetch hit rate of the prefetch algorithm is updated. Otherwise, the IO prefetch hit rate of the prefetch algorithm of the next time scale is obtained according to different time scales from small to large. If it is reached, the prefetch algorithm is changed to select the prefetch algorithm of the next time scale. When the prefetch algorithm of the maximum time scale is adjusted, the IO prefetch hit rate of the prefetch algorithm still cannot reach the preset threshold, then the prefetch algorithm with the largest IO prefetch hit rate among the prefetch algorithms corresponding to different time scales is selected instead, and updated to the prefetch algorithm selection table.

[0039] Data prefetching algorithm training includes the following process:

[0040] By collecting the LBA of IO and training it through the LSTM network, the LSTM network contains: c is the memory unit state, h is the hidden state; x is the input data, that is, the LBA information of the IO in the current period; y is the output data, that is, the LBA information of the data to be pre-fetched; the LBA data of IO is collected in the corresponding time period, and the corresponding pre-fetching algorithm model is output through training; through the pre-fetching algorithm model, the LBA information of IO in the current period is input in the same time period to predict the LBA information of the next IO.

[0041] Specifically, LSTM (Long Short Term Memory networks) are recurrent neural networks that are widely used in time series prediction. This solution continuously collects IO LBA data, trains the data prefetching algorithm according to the time dimension through the LSTM network, and dynamically adapts the data prefetching algorithm selector to continuously monitor the data hit rate of the currently selected prefetching algorithm and select the optimal prefetching algorithm. This can effectively solve the problem that a fixed data prefetching algorithm cannot meet the needs of all scenarios. Figure 2 The diagram shows a data storage system that uses a method for intelligent data pre-fetching.

[0042] The overall process of data prefetching, such as Figure 3 As shown:

[0043] When an IO request arrives, the data prefetch algorithm manager selects the corresponding prefetch algorithm based on time correlation, records the hit rate of the prefetch algorithm during execution, and automatically adjusts the selected prefetch algorithm.

[0044] The data prefetch algorithm manager selects the optimal prefetch algorithm based on time correlation and hit rate statistics. However, if a trained data prefetch algorithm does not exist at the initial operation of the storage system, the default adjacent prefetch algorithm is returned, that is, the data of the adjacent LBAs before and after the current IO is prefetched. At the same time, if there is no corresponding data prefetch algorithm for the current time range, the data prefetch algorithm with a close time is selected.

[0045] The data prefetch algorithm manager continuously updates the IO hit rate of the prefetch algorithm in hours to determine which prefetch algorithm has the best effect in the current time period, and updates the process data and selection results to the prefetch algorithm selection table. Figure 4 shown.

[0046] 2. Prefetch algorithm training process:

[0047] The training process of the data prefetching algorithm is to continuously collect IO LBA and train it through the LSTM network, and then output the corresponding prefetching algorithm model.

[0048] The LSTM model contains the following concepts: c is the memory cell state, h is the hidden state, x is the input data, i.e. the LBA information of the io in the current period, and y is the output data, i.e. the LBA information of the data to be pre-fetched. The training process is to collect the LBA data of the io in the corresponding time period. However, continuous training is carried out so that in the same time period, by inputting the LBA information of the io in the current period, the LBA information of the next io can be relatively accurately predicted, thereby ensuring the hit rate of data pre-fetching. Figure 5 The expanded diagram of the LSTM model is shown.

[0049] Parameter description and corresponding calculation formula description:

[0050]

[0051]

[0052] Example of algorithm model training process:

[0053] 1. Collect the IO LBA information sequence [LBA1, LBA2, LBA3…, LBNn] within a period of time (hour, day, week, month);

[0054] 2. Assume that the length of the LBA information input to the model for a period of time is input_size=j, and the length of the LBA information predicted to be output for the next period of time is output_size=k. Then the training process is to take the collected IO LBA information sequence from small to large. For example, take the sequence of length k [LBA1, LBA2…, LBAj] and input it into the model, and output the sequence of length j [LBAj+1', LBAj+2'…, LBAj+k'].

[0055] 3. Compare [LBAj+1', LBAj+2'…, LBAj+k'] with the collected real sequence [LBAj+1, LBAj+2…, LBAj+k], calculate the loss value and perform backpropagation to update the parameters in the algorithm model.

[0056] This solution uses time series prediction technology to dynamically adjust the data prefetching algorithm, effectively ensuring that the data prefetching algorithm meets current business needs.

[0057] During the operation of current IT systems, the access model of apps to storage generally has time correlation. For example, financial systems regularly perform batch analysis on user business data. The storage IO access model varies depending on the weekly, monthly, and annual batch analysis.

Claims

1. A method for intelligent pre-fetching of data in a storage system, characterized in that: The steps include: Step 1: Build a data prefetching algorithm pool and a prefetching algorithm selection table; Step 2: Based on the IO request, the data prefetch algorithm manager searches the data prefetch algorithm pool for the current prefetch algorithm. If the current prefetch algorithm exists, the process proceeds to step 3. If not, the data prefetch algorithm with a similar time is selected and the process proceeds to step 4. Step 3: Select the prefetch algorithm with the closest time or prefetch the data of the adjacent LBAs before and after the current IO; Step 4: Prefetch data according to the selected data prefetch algorithm and obtain the IO hit rate. The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm based on the minimum time scale among different time scales, updates the process data and selection results to the prefetch algorithm selection table, and completes the intelligent prefetching of storage system data. The construction of the data prefetching algorithm pool and prefetching algorithm list includes: The data prefetch algorithm pool includes prefetch algorithms corresponding to different time scales and IO prefetch hit rates; the prefetch algorithm list includes different time scales and IO prefetch hit rates corresponding to the prefetch algorithms corresponding to different time scales.

2. The method for intelligent pre-fetching of storage system data according to claim 1, characterized in that: The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale among different time scales, and updates the process data and selection results to the prefetch algorithm selection table, including: The data prefetch algorithm manager updates the IO prefetch hit rate of the prefetch algorithm according to the minimum time scale, and determines whether the IO prefetch hit rate of the prefetch algorithm reaches the preset threshold. If so, the IO prefetch hit rate of the prefetch algorithm is updated. Otherwise, the IO prefetch hit rate of the prefetch algorithm of the next time scale is obtained according to different time scales from small to large. If it is reached, the prefetch algorithm is changed to select the prefetch algorithm of the next time scale. When the prefetch algorithm of the maximum time scale is adjusted, the IO prefetch hit rate of the prefetch algorithm still cannot reach the preset threshold, then the prefetch algorithm with the largest IO prefetch hit rate among the prefetch algorithms corresponding to different time scales is selected instead, and updated to the prefetch algorithm selection table.

3. The method for intelligent pre-fetching of storage system data according to claim 2, characterized in that: Data prefetching algorithm training includes the following process: By collecting the LBA of IO and training it through the LSTM network, the LSTM network contains: c is the memory unit state, h is the hidden state; x is the input data, that is, the LBA information of the IO in the current period; y is the output data, that is, the LBA information of the data to be pre-fetched; the LBA data of IO is collected in the corresponding time period, and the corresponding pre-fetching algorithm model is output through training; through the pre-fetching algorithm model, the LBA information of IO in the current period is input in the same time period to predict the LBA information of the next IO.

Images