Data prefetching method based on multi-head attention mechanism and RNN-LSTM network

Abstract

The present application relates to the technical field of data prefetching, and particularly provides a data prefetching method based on a multi-head attention mechanism and an RNN-LSTM network. The method comprises the following steps: obtaining an initial data set, and performing proportional division to obtain a training data set, a verification data set and a test data set; preprocessing the initial data set to obtain a preprocessed data set; constructing a recurrent neural network fusion model through a multi-head attention mechanism and an RNN-LSTM recurrent neural network; and training, verifying and testing the recurrent neural network fusion model through the preprocessed training data set, the verification data set and the test data set, so as to realize data prefetching. Through the fusion technology, the method utilizes the powerful basis of RNN-LSTM in processing long time sequences and a specific multi-head attention mechanism, improves the accuracy and efficiency of data prefetching, and reduces the delay of data reading.

Description

Technical Field

[0001] This invention relates to the field of data prefetching technology, and in particular to a data prefetching method based on a multi-head attention mechanism and an RNN-LSTM network. Background Technology

[0002] Computer systems typically consist of multiple levels of memory operating at different performance levels, primarily including registers, cache, DRAM, flash memory, and hard disk drives (HDDs). The purpose of storage tiering in computer systems is to reduce overall access time and storage costs. Within this storage tiering system, data prefetching technology emerged to further reduce access latency and improve system performance. Data prefetching is a technique that predicts data that may be accessed in the future and loads it in advance from a lower-speed storage tier to a higher-speed storage tier. Its core idea is to place data in a storage tier closer to the processor before it is actually requested, thereby reducing latency overhead caused by storage access. Effective data prefetching techniques can significantly reduce cache miss rates and main memory access latency, and optimize the utilization of system storage resources. Data prefetching is mainly implemented through hardware mechanisms or software instructions, suitable for automatic triggering and program-based scenarios, respectively. With the development of computer technology and hardware, processor performance has continuously improved, and the speed gap between the processor and storage has gradually widened, leading to the "memory wall" problem. The processor may be idle for extended periods while waiting for data to load, severely impacting system performance. Data prefetching reduces cache miss rates by loading data in advance, thus effectively mitigating the storage wall problem.

[0003] In data-intensive applications such as high-performance computing (HPC), artificial intelligence, and big data processing, data access latency is a key bottleneck for performance optimization. Data prefetching technology accelerates task processing and significantly improves system throughput and response speed by shortening the data transfer time from storage to processor. However, traditional hardware data prefetchers (such as stream prefetching and stride prefetching) rely on simple access patterns with linear and fixed steps. In real-world applications, data access patterns are often more complex and diverse, leading to reduced accuracy of traditional prefetchers and potentially increasing invalid prefetching. Inefficient prefetching strategies may load unnecessary data, replace useful data, reduce cache hit rate, and even impact performance. Summary of the Invention

[0004] In view of this, the present invention provides a data prefetching method based on a multi-head attention mechanism and an RNN-LSTM network to improve the accuracy and efficiency of data prefetching and reduce the latency of data reading.

[0005] In a first aspect, the present invention provides a data prefetching method based on a multi-head attention mechanism and an RNN-LSTM network, the method comprising:

[0006] Step 1: Obtain the initial dataset and divide it proportionally to obtain the training dataset, validation dataset, and test dataset; preprocess the initial dataset to obtain the preprocessed dataset.

[0007] Step 2: Construct a recurrent neural network fusion model using a multi-head attention mechanism and an RNN-LSTM recurrent neural network;

[0008] Step 3: Using the preprocessed training dataset, validation dataset, and test dataset, the recurrent neural network fusion model is trained, validated, and tested respectively to achieve data prefetching.

[0009] Optionally, obtaining the initial dataset in step 1 includes:

[0010] First, the cache of the Master node's flash memory chip needs to be read to generate timestamps. The timestamps start from the unique address time record of the first file read from the cache and are generated every 5 minutes. Second, statistics are performed every 5 minutes to record all unique pages in memory, the memory offset between all unique pages, and the network usage rate for reading each unique page. Finally, the data is integrated to generate a dataset in the form of timestamp timescamp, offset target, bandwidth usage dynamic_feat, and unique page mapping ID id.

[0011] The ratio division includes: collecting 10,000 data points, removing data with large offsets, leaving 8,000 valid data points; and using custom code to set the entire dataset into a 6:2:2 ratio for the training dataset, validation dataset, and test dataset.

[0012] The preprocessing of the initial dataset to obtain the preprocessed dataset includes:

[0013] Data normalization: Normalization helps speed up model training, improves model convergence speed, and reduces the impact of model initialization on the training process. The normalization operation used is implemented by the MinMaxScaler function in Sklearn. MinMaxScaler scales all values ​​to a specified range, and its formula is:

[0014]

[0015] Where X is the feature value of the original data; X min It is the minimum value of the original data feature value X in the data; X maxIt is the maximum value of the original data feature value X in the data; X scaled It is the final result of normalization;

[0016] Normalize the bandwidth utilization (dynamic_feat), unique page mapping ID, and offset (target) columns in the dataset.

[0017] Data prediction pattern processing: Define a data input and prediction pattern, which is implemented using a custom function create_fixed_sequnences. The specific steps are as follows: first, input sequence_length as the input time series step size for the model; then, the model returns a prediction result, i.e., a prediction time step size, and returns the corresponding timestamp. Here, the sequence_length is set to 64, which means that 64 time series steps are input, allowing the model to learn and thus make predictions for a time series of one step size.

[0018] Optionally, the recurrent neural network fusion model in step 2 includes:

[0019] a. Input Layer: The input layer has dimensions (sequence_length, feature_dim); where sequence_length represents the number of time steps; feature_dim represents the feature dimension of each time step; the input layer inputs the raw time series data into the model for feature extraction;

[0020] b. Self-Attention Mechanism Layer: In the attention mechanism, each input element is assigned a weight value based on its relationship with other elements. These weight values ​​are calculated using a dot product. The formula for calculating the attention mechanism is:

[0021]

[0022] Where Q is the query vector, representing the target to be focused on; K is the key vector, representing the features in all inputs; V is the value vector, i.e., the actual information associated with the key vector; d k It represents the dimension of the key vector, used for scaling.

[0023] c. Residual Blocks: Residual blocks address the model degradation problem that occurs as the number of network layers increases. They are introduced because shallower networks perform better than deeper networks when network degradation occurs. Therefore, residual blocks directly pass the computation results from lower-level networks to higher-level networks, allowing deeper networks to outperform shallower networks. The formula for residual blocks is:

[0024] X l+1 =X l+A(X l +W l );

[0025] Among them, X l As the input of the l-th layer, A(X) l +W l ) represents the output of the masked multi-head self-attention mechanism module in layer l;

[0026] d. Normalization Layer: Layer Normalization is used to process the data; layer normalization normalizes all neurons in a layer; assuming the i-th input of the neuron in the l-th layer is z i (l) The expressions for its mean and variance are as follows:

[0027]

[0028] Where, n (l) This represents the number of neurons in the l-th layer.

[0029] Layer normalization is defined as:

[0030]

[0031] Where γ represents the scaling parameter vector and β represents the translation parameter vector, the dimensions of both parameter vectors γ and β are the same as z. (l) They have the same dimension;

[0032] e. RNN-LSTM recurrent neural network layer:

[0033] The RNN-LSTM recurrent neural network layer consists of two layers of recurrent neural network (RNN), three layers of long short-term memory network (LSTM), and three fully connected layers.

[0034] (1) The basic unit of an RNN is a Recurrent Unit, which receives an input and a hidden state from the previous time step, and outputs the hidden state of the current time step. The basic RNN structure consists of an input layer, a hidden layer, and an output layer; by unfolding the basic RNN structure in the time dimension, we obtain the entire network structure framework of the RNN;

[0035] The RNN network receives input x at time t. t After that, the value of the hidden layer is s t The formula is as follows:

[0036] s t =f(U×x) t +W×s t-1 );

[0037] U is the weight matrix of the input x, and W is the value s of the previous hidden layer. t-1 As the current input weight matrix, f is the activation function, and s t The value depends on x t and s t-1 ;

[0038] The output value is o t The calculation formula is as follows:

[0039] o t =g(V×s) t );

[0040] V is the weight matrix of the output layer, and g is the activation function;

[0041] The hidden layer has two inputs: the first is the weight matrix U from the input layer to the hidden layer, and the second is the input x. t The product of the two; the second is the value of the previous hidden layer s. t-1 The product of W and s, i.e., s calculated at the previous time step. t-1 Cache is required, along with the input x t Calculate and output the final 'o'. t ;

[0042] A two-layer RNN neural network is used. The first layer of the RNN has 128 neurons, and the second layer of the RNN has 64 neurons. The activation function of both the first and second layers is the ReLU function. The two RNN layers are used to receive and process the output of the multi-head attention mechanism layer, learn the relationship between multi-step time series features, and output them to the Long Short-Term Memory (LSTM) network layer.

[0043] (2) The core of LSTM is a unit where each neuron contains a forget gate, an input gate, and an output gate; each gate is a special structure used to control the flow of information, passing information from previous moments to the current moment, thereby capturing the temporal dependencies in the data; the LSTM neural network includes:

[0044] Forget Gate:

[0045] The forget gate determines what information to discard from a cell's state; it is calculated using the following formula:

[0046] f t =σ(W f ×[h t-1 ,x t ]+b f );

[0047] Among them, f t This represents the output of the forget gate at time t, where σ is the sigmoid function and W is the output of the forget gate. f and bf The weights and biases of the forget gate, h t-1 It is the previous hidden state, x t This is the current input;

[0048] Input Gate:

[0049] The input gate is used to update the cell state and consists of two parts: a sigmoid layer and a tanh layer.

[0050] i t =σ(W i ×[h t-1 ,x t ]+b i );

[0051]

[0052] Among them, i t It is the input of the input gate. It is a candidate value vector;

[0053] Output Gate:

[0054] The output gate is used to determine the next hidden state, which contains information about the previous input and is used for prediction.

[0055] o t =σ(W o ×[h t-1 ,x t ]+b o );

[0056] h t =o t *tanh(C t );

[0057] Among them, o t It is the output of the output gate, h t This is the current hidden state, C t This is the current cell state;

[0058] The LSTM layer consists of three layers: the first LSTM layer has 64 neurons, the second LSTM layer has 32 neurons, and the third LSTM layer has 32 neurons. The activation function for the first, second, and third layers is the Tanh function.

[0059] (3) Fully connected layer:

[0060] In the fully connected layer Dense, all input nodes and output nodes are fully connected; three fully connected layers are used as the last fully connected layer of the RNN-LSTM recurrent neural network.

[0061] The first fully connected layer has 32 neurons, the second fully connected layer has 16 neurons, and the last fully connected layer has 8 neurons. The activation function for all three fully connected layers is the ReLU function.

[0062] f. Feedforward layer:

[0063] The feedforward layer is used to provide more learning capabilities. The feedforward network is a two-layer network. The activation function of the first layer is the ReLU function. For a vector x at each position in the input sequence, its expression is:

[0064] FFN(x)=max(0,xW1+b1)W2+b2.

[0065] g. Output layer:

[0066] The number of neurons in the output layer is consistent with the feature dimension, and no activation function is set; the predicted value is generated directly.

[0067] Optionally, step 3 includes:

[0068] (31) Initialize the model: Integrate the RMM-LSTM recurrent neural network framework and add a multi-head attention mechanism to enhance the parallel learning ability of the model and its response to key features;

[0069] (32) Optimizer: Choose the Adam optimizer, which combines momentum and adaptive learning rate features to help optimize large-scale datasets quickly and stably;

[0070] (33) Training Epochs: Monitor model performance by checking the loss on the validation dataset. Stop training when the loss on the validation dataset stops decreasing or increases for several consecutive epochs. Parameter selection: 200 training epochs.

[0071] (34) Batch size: This parameter determines the amount of data used for each gradient update, affecting training stability and efficiency. Choose an appropriate batch size based on hardware performance and task requirements to achieve a balance between training stability and computational efficiency. Set it to Batch_Size = 64.

[0072] (35) Evaluation indicators:

[0073] ①Accuracy

[0074] Accuracy is used to measure whether the predicted value is within the error threshold. Its principle is as follows: it provides intuitive feedback on the overall performance of the model on the training dataset, validation dataset and test dataset.

[0075]

[0076] Among them, y i It is the true value of the i-th sample, i.e., the target value, which is the actual observed value of the time series. ε is the predicted value of the i-th sample, i.e., the predicted output of the model; ε is the error tolerance or threshold, which sets an error range, indicating that when the difference between the predicted value and the true value is less than the threshold, the prediction is considered correct; the error tolerance is 5% or 10%; n is the total number of samples.

[0077] ②Loss rate

[0078] A custom binary cross-entropy loss function was selected as the basic loss function for the recurrent neural network fusion model to evaluate its predictive performance.

[0079] The formula for the binary cross-entropy loss function is:

[0080]

[0081] Where N is the total number of samples; y i p is the true label of the i-th sample, with a value of 0 or 1; i It is the probability that the i-th sample predicted by the model belongs to category 1, and its value ranges between (0, 1).

[0082] The BCE loss function adjusts the model's parameters during training to ensure that the predicted probability p is... i Closer to the real label y i It encourages the model to output probability values ​​close to the true class, thereby improving classification accuracy.

[0083] To prevent the model from overfitting the training data, L2 regularization is introduced on the basis of the BCE loss function. L2 regularization, also known as weight decay, is based on the idea of ​​penalizing the model's parameters to force the model to maintain smaller weight values, thereby preventing the model from overfitting the training data.

[0084] The L2 regularization formula is:

[0085]

[0086] Where λ is a hyperparameter of regularization strength, controlling the weight of the regularization term; the larger λ is, the stronger the regularization effect. j These are the model parameters, i.e., weights; n is the total number of model parameters; the L2 regularization term is the sum of squares of all weights, and the larger the weight value, the greater the penalty.

[0087] By combining L2 regularization and the binary cross-entropy loss function (BCE), a new loss function is obtained, the formula of which is:

[0088]

[0089] The binary cross-entropy loss function (BCE) calculates the model's prediction error, while the L2 regularization term helps the model avoid overfitting by penalizing larger weights.

[0090] ③R 2 value

[0091] R 2 The value measures the proportion of the variance of the target variable explained by the model, and its range is [0,1]. The principle is illustrated by the following formula; R 2 It reflects the goodness of fit of the model; the closer the value is to 1, the stronger the model's ability to explain the target variable.

[0092]

[0093] Among them, y i It is the true value of the i-th sample, i.e., the target value; It is the predicted value of the i-th sample, which is the predicted output of the model; It is the average of the true values, that is It is the residual sum of squares (RSS), which is the sum of the squares of the differences between the actual and predicted values. It is the total sum of squares (TSS), which is the sum of the squares of the differences between the true values ​​and the true mean.

[0094] (36) Training strategy:

[0095] To improve the multi-step prediction performance of the nonlinear autoregressive neural network NARX-NN, a prediction feedback retraining strategy is adopted. The prediction feedback retraining strategy is designed to address the inconsistency between the inputs in the training phase and the multi-step prediction phase of the NARX-NN model, thereby reducing the discrepancy and improving the multi-step prediction performance of the model.

[0096] The basic idea of ​​FR is to reconstruct training samples using the error-laden single-step prediction results and retrain the model to enhance its robustness to prediction errors. The specific steps are as follows:

[0097] ① First, use the initial dataset and complete the initial training using the conventional training strategy, that is, use the backpropagation algorithm to train the network. At this time, the input target variables are all actual observation values.

[0098] ② Use the trained network to make a single-step prediction to obtain the prediction result; single-step prediction ensures that the error of the prediction result is within a reasonable range. If multi-step prediction results are used, the reconstructed sample in step ③ below will deviate from the dynamic characteristics of the system to be modeled due to the excessive prediction error.

[0099] ③ Replace the measured target value in the model input with its corresponding single-step prediction result to obtain a new training sample. Multiple replacements are required to reconstruct the dataset. The collective operation is as follows: given the real data of the first 63 time steps, predict the data of the next time step, replace the real data in the prediction time step with the predicted data, and complete one reconstruction of the dataset. The data of 64 time steps is one cycle. Repeat the operation until the data in all cycles of the entire dataset has been replaced with the real value and the predicted value.

[0100] ④ Retrain the network using the reconstructed samples; after training, check whether the validation set loss is greater than the recorded minimum validation set loss for several consecutive training rounds; if so, it indicates that the model's performance on the validation set no longer improves, end the training, and obtain a multi-step prediction model; otherwise, if the validation set loss decreases in the current round, update the minimum validation set loss, proceed to steps ② and ③, and repeat single-step prediction and network training.

[0101] In a second aspect, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute the data prefetching method based on a multi-head attention mechanism and an RNN-LSTM network in the first aspect or any possible implementation thereof.

[0102] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the data prefetching method based on a multi-head attention mechanism and an RNN-LSTM network in the first aspect or any possible implementation of the first aspect.

[0103] The technical solution provided by this invention includes a method that involves acquiring an initial dataset and dividing it proportionally to obtain a training dataset, a validation dataset, and a test dataset; preprocessing the initial dataset to obtain a preprocessed dataset; constructing a recurrent neural network fusion model using a multi-head attention mechanism and an RNN-LSTM recurrent neural network; and training, validating, and testing the recurrent neural network fusion model using the preprocessed training dataset, validation dataset, and test dataset, respectively, to achieve data prefetching. This method, through fusion technology, leverages the powerful foundation of RNN-LSTM in processing long-term sequences and a specific multi-head attention mechanism to improve the accuracy and efficiency of data prefetching and reduce data reading latency. Attached Figure Description

[0104] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0105] Figure 1 A flowchart of a data prefetching method provided in an embodiment of the present invention;

[0106] Figure 2 A flowchart for obtaining the initial dataset provided in an embodiment of the present invention;

[0107] Figure 3 A flowchart of data prediction pattern processing provided in an embodiment of the present invention;

[0108] Figure 4 A schematic diagram of a recurrent neural network fusion model provided in an embodiment of the present invention;

[0109] Figure 5 This is a schematic diagram of the multi-head attention mechanism layer structure provided in an embodiment of the present invention;

[0110] Figure 6 A schematic diagram of an RNN-LSTM recurrent neural network provided in an embodiment of the present invention;

[0111] Figure 7 A schematic diagram of the basic RNN structure provided for embodiments of the present invention;

[0112] Figure 8 A schematic diagram of the LSTM neural network structure provided in an embodiment of the present invention;

[0113] Figure 9 A flowchart of the training strategy provided in the embodiments of the present invention;

[0114] Figure 10 A flowchart illustrating another training strategy provided in an embodiment of the present invention;

[0115] Figure 11 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0116] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0117] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0118] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0119] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0120] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0121] Figure 1 A flowchart of the data prefetching method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the method includes:

[0122] Step 1: Obtain the initial dataset and divide it proportionally to obtain the training dataset, validation dataset, and test dataset; preprocess the initial dataset to obtain the preprocessed dataset.

[0123] The dataset uses cached data from the flash memory chip of the Master node of the Shandong Computing Network Unified Storage Platform. Each data sequence contains a unique address, a unique page it resides on, unique offsets between the preceding and following files, the time the file was read, and the network usage rate during file reading. However, some data is not closely related to data prefetching and may even affect the accuracy of data prefetching. Therefore, the unique page, offset, and network usage rate are included in the dataset, and the timestamps are re-edited.

[0124] In embodiments of the present invention, such as Figure 2 As shown, obtaining the initial dataset in step 1 includes:

[0125] First, the cache of the Master node's flash memory chip needs to be read to generate timestamps. The timestamps start from the unique address time record of the first file read from the cache and are generated every 5 minutes. Second, statistics are performed every 5 minutes to record all unique pages in the current memory, the memory offset between all unique pages, and the network usage rate for reading each unique page. Finally, the data is integrated to generate a dataset in the form of timestamps (timescamp), offsets (target), bandwidth usage rate (dynamic_feat), and unique page mapping IDs (id), as shown in Table 1.

[0126] Table 1 Dataset Format

[0127] Timestamp (timescamp) Broadband utilization (dynamic_feat) Unique page mapping ID (id) Offset (target) 2024-01-0100:00:00 0.01 0 0 2024-01-01 00:05:00 0.17 8 512 2024-01-01 00:10:00 0.13 0 512 2024-01-01 00:15:00 0.34 15 960 .

[0128] The ratio division includes: collecting 10,000 data points, removing data with large offsets, leaving 8,000 valid data points; and using custom code to set the entire dataset into a 6:2:2 ratio for the training dataset, validation dataset, and test dataset; the dataset is a Data.csv file, and its content format is shown in Table 2.

[0129] Table 2 Dataset Data.csv file

[0130] Timestamp (timescamp) Broadband utilization (dynamic_feat) Unique page mapping ID (id) Offset (target) 2024-01-0100:00:00 0.01 0 0 2024-01-0100:05:00 0.17 8 512 2024-01-01(00:10:00 0.13 0 512 2024-01-01000:15:00 0.34 15 960 2024-01-01 00:20:00 0.17 4 704 .

[0131] Only a single CSV file is needed as the dataset to train, validate, and test the model, reducing data preprocessing time. Furthermore, this data format accurately reflects the uniqueness of the cached page, page offset, and changes in network occupancy.

[0132] The preprocessing of the initial dataset to obtain the preprocessed dataset includes:

[0133] Data normalization: Normalization helps speed up model training, improves model convergence speed, and reduces the impact of model initialization on the training process. The normalization operation used is implemented by the MinMaxScaler function in Sklearn. MinMaxScaler scales all values ​​to a specified range, usually [0,1]. Its formula is:

[0134]

[0135] Where X is the feature value of the original data; X min It is the minimum value of the original data feature value X in the data; X maxIt is the maximum value of the original data feature value X in the data; X scaled It is the final result of normalization;

[0136] Normalize the bandwidth utilization (dynamic_feat), unique page mapping ID (id), and offset (target) columns in the dataset. This standardized data makes it easier and faster for neural network models to learn because it ensures that the network input features are of the same magnitude, which helps improve the efficiency and stability of gradient descent during the optimization process.

[0137] Data prediction pattern processing: such as Figure 3 As shown, a data input and prediction pattern is defined. This pattern is implemented using a custom function `create_fixed_sequnences(data, time_data, sequence_length)`. The specific steps are as follows: first, input `sequence_length` as the input time series step size for the model; then, the model returns a prediction result, i.e., a prediction time step size, and returns the corresponding timestamp. Here, `sequence_length` is set to 64, representing an input of 64 time series steps, allowing the model to learn and thus make predictions for a time series of one step size.

[0138] Step 2: Construct a recurrent neural network fusion model using a multi-head attention mechanism and an RNN-LSTM recurrent neural network.

[0139] In this embodiment of the invention, the recurrent neural network fusion model employs a decoder structure with a self-attention mechanism. The self-attention mechanism is used because it can capture the global and local connections in one step and solves the problem of parallel computation in recurrent neural network or LSTM-based models. The decoder structure originates from the encoder-decoder model. Predicting the next offset is not a generative task but a classification task; therefore, the model does not require the entire encoder-decoder model. The decoder structure is chosen because the offset sequence has a temporal order, and each time step only interacts with the previous sequence. Therefore, the model uses a masked decoder structure to mask future information.

[0140] In embodiments of the present invention, such as Figure 4 As shown, the recurrent neural network fusion model in step 2 includes:

[0141] a. Input Layer: The input layer has dimensions (sequence_length, feature_dim); where sequence_length represents the number of time steps; feature_dim represents the feature dimension of each time step; the input layer inputs the raw time series data into the model for feature extraction;

[0142] b. Self-Attention Mechanism Layer: In the attention mechanism, each input element is assigned a weight value based on its relationship with other elements (e.g., the relationship between words). These weight values ​​are calculated using a dot product. The formula for calculating the attention mechanism is:

[0143]

[0144] Where Q is the query vector, representing the target to be focused on; K is the key vector, representing the features in all inputs; V is the value vector, i.e., the actual information associated with the key vector; d k is the dimension of the key vector, used for scaling; the essence of the above calculation process is to measure the relevance between the query and the key through the dot product, and then to sum the corresponding value vector V according to these relevance weights.

[0145] Multi-head attention mechanisms enhance a model's ability to capture diverse information by dividing the input query, key, and value into multiple heads (i.e., multiple subspaces) and computing them in parallel. Its core idea is to parallelize the computation of multiple attention mechanisms, each focusing on a different aspect of the input data, ultimately aggregating this information. For example... Figure 5 As shown, the multi-head attention mechanism layer structure includes:

[0146] I. Input: The preprocessed dataset is input into the model. In this invention, the actual data to be predicted consists of three columns (dynamic_feat, id, target). Therefore, the input shape in the model is as follows:

[0147] Input shape=(batch_size, sequence_length, 3);

[0148] Here, `batch_size` is the number of samples included in each training iteration, set to 128; `sequence_length` is the length of the time series, which, as explained in the data processing module above, is set to 64; and the final parameter is the dimension of the data features at each time step, corresponding to the three columns of data that this invention needs to predict. Therefore, the input model consists of a (128, 64, 3) matrix.

[0149] II. The multi-head attention layer internally performs linear transformations of the data in the matrix. This process is performed separately within each head, such as... Figure 5 As shown, a total of 5 attention mechanism layers were used. The linear transformation process yields three vector matrices: query Qi, key Ki, and value Vi. The linear transformation formula is as follows:

[0150] Query i =X*W i Q ;

[0151] Key i =X*W i K ;

[0152] Value i =X*W i V ;

[0153] Where X represents the sequence in the input dataset during training; W i Q W is the query transformation matrix for the i-th head; i K W is the key transformation matrix of the i-th head; i V These are the value transformation matrices for the i-th head, and the weight matrices are the learned parameters, which are typically optimized during training.

[0154] Each head independently calculates its corresponding query, key, and value matrix, and its own attention. A mask matrix is ​​then added to this matrix, preventing the neural network model from seeing information beyond the current time step. In other words, for a given offset sequence, at time step t, the model's output should only depend on the outputs before that offset, without seeing subsequent information. This ensures the generated causal relationships and increases the model's flexibility in processing sequence data. The mask formula is shown below:

[0155] Given a sequence length T, the future information mask M is the previous triangular matrix:

[0156]

[0157] Therefore, the output of each head i The attention weights and corresponding value matrix V are calculated based on this head. i The result of weighted summation.

[0158]

[0159] head i =AttentionWeight*Vi ;

[0160] III. Head the results from each attention layer. i Place the elements in the merge layer and concatenate them for output. The concatenation formula is as follows:

[0161] Concat(head1,head2,...,head5)=[head1,head2,...,head5];

[0162] The output shape after splicing is:

[0163]

[0164] `batch_size` specifies the number of samples in a batch, and its parameter is set to 64. `seq_len` is the length of the time series, and its parameter is also set to 64. head For each attention layer, there is a dimension with a parameter of 64, and h is the number of attention heads, which is 5.

[0165] IV. In order to map the concatenated output back to the model dimension d model It requires a linear transformation, through a weight matrix W. o Mapping to the model dimension, the formula is as follows:

[0166] Output = Concat × W o ;

[0167] W o It is a linear weight matrix used to integrate the concatenated results of multi-head attention, with a shape of (h*d). head d model Essentially, this is a linear transformation. Through this linear transformation, the high-dimensional tensor is integrated into the total model dimension d. model This further compresses the information, allowing the output of the attention head to be processed by subsequent model structures (RNN-LSTM).

[0168] c. Residual Blocks: Residual blocks address the model degradation problem that occurs as the number of network layers increases. They are introduced because shallower networks perform better than deeper networks when network degradation occurs. Therefore, residual blocks directly pass the computation results from lower-level networks to higher-level networks, allowing deeper networks to outperform shallower networks. The formula for residual blocks is:

[0169] X l+1 =X l +A(X l +W l );

[0170] Among them, X l As the input of the l-th layer, A(X)l +W l ) represents the output of the masked multi-head self-attention mechanism module in layer l;

[0171] d. Normalization Layer: Layer Normalization (LN) is used to process the data; layer normalization normalizes all neurons in a layer; assuming the i-th input of the neuron in the l-th layer is z i (l) The expressions for its mean and variance are as follows:

[0172]

[0173] Where, n (l) This represents the number of neurons in the l-th layer.

[0174] Layer normalization is defined as:

[0175]

[0176] Where γ represents the scaling parameter vector and β represents the translation parameter vector, the dimensions of both parameter vectors γ and β are the same as z. (l) They have the same dimension;

[0177] e. RNN-LSTM recurrent neural network layer:

[0178] In embodiments of the present invention, such as Figure 6 As shown, the RNN-LSTM recurrent neural network layer consists of two layers of recurrent neural network (RNN), three layers of long short-term memory network (LSTM), and three fully connected layers.

[0179] (1) The basic unit of a recurrent neural network (RNN) is a recurrent unit, which receives an input and a hidden state from the previous time step, and outputs the hidden state of the current time step. For example... Figure 7 As shown, the basic RNN structure consists of an input layer, a hidden layer, and an output layer. Figure 7 In this diagram, x is the input vector, o is the output vector, and s represents the value of the hidden layer; U is the weight matrix from the input layer to the hidden layer, and V is the weight matrix from the hidden layer to the output layer. The value s of the hidden layer in a recurrent neural network depends not only on the current input x but also on the previous hidden layer value s. The weight matrix W represents the weights of the previous hidden layer values ​​used as the current input. Expanding the basic RNN structure along the time dimension yields the overall network structure framework of the RNN.

[0180] The RNN network receives input x at time t. t After that, the value of the hidden layer is s t The formula is as follows:

[0181] s t =f(U×x) t +W×s t-1 );

[0182] U is the weight matrix of the input x, and W is the value s of the previous hidden layer. t-1 As the current input weight matrix, f is the activation function, and s t The value depends on x t and s t-1 ;

[0183] The output value is o t The calculation formula is as follows:

[0184] o t =g(V×s) t );

[0185] V is the weight matrix of the output layer, and g is the activation function;

[0186] The hidden layer has two inputs: the first is the weight matrix U from the input layer to the hidden layer, and the second is the input x. t The product of the two; the second is the value of the previous hidden layer s. t-1 The product of W and s, i.e., s calculated at the previous time step. t-1 Cache is required, along with the input x t Calculate and output the final 'o'. t ;

[0187] A two-layer RNN neural network is used. The first layer of the RNN has 128 neurons, and the second layer of the RNN has 64 neurons. The activation function of both the first and second layers is the ReLU function. The two RNN layers are used to receive and process the output of the multi-head attention mechanism layer, learn the relationship between multi-step time series features, and output them to the Long Short-Term Memory (LSTM) network layer.

[0188] In data prefetching tasks, the prefetching operation often involves long-term dependencies, meaning that information from multiple time steps ago is needed for accurate predictions. RNN models are particularly prone to vanishing and exploding gradients when processing long-term series data. Vanishing gradients make it difficult for the model to effectively learn long-distance dependencies, thus reducing the accuracy of data prefetching.

[0189] Therefore, this invention introduces an LSTM neural network on the basis of RNN. Long Short-Term Memory (LSTM) is a special recurrent neural network structure that is particularly suitable for processing sequence data and time series tasks.

[0190] (2) The core of LSTM is a unit in which each neuron contains a forget gate, an input gate, and an output gate; each gate is a special structure used to control the flow of information, passing information from previous moments to the current moment, thereby capturing the temporal dependencies in the data; such as Figure 8 As shown, the LSTM neural network includes:

[0191] Forget Gate:

[0192] The forget gate determines what information to discard from a cell's state; it is calculated using the following formula:

[0193] f t =σ(W f ×[h t-1 ,x t ]+b f );

[0194] Among them, f t This represents the output of the forget gate at time t, where σ is the sigmoid function and W is the output of the forget gate. f and b f The weights and biases of the forget gate, h t-1 It is the previous hidden state, x t This is the current input;

[0195] Input Gate:

[0196] The input gate is used to update the cell state. It consists of two parts: a sigmoid layer and a tanh layer. The sigmoid layer determines which values ​​will be updated. The tanh layer creates a new candidate value vector, which will be added to the state.

[0197] i t =σ(W i ×[h t-1 ,x t ]+b i );

[0198]

[0199] Among them, i t It is the input of the input gate. It is a candidate value vector;

[0200] Output Gate:

[0201] The output gate is used to determine the next hidden state, which contains information about the previous input and is used for prediction.

[0202] o t =σ(W o×[h t-1 ,x t ]+b o );

[0203] h t =o t *tanh(C t );

[0204] Among them, o t It is the output of the output gate, h t This is the current hidden state, C t This is the current cell state;

[0205] The LSTM layer consists of three layers: the first LSTM layer has 64 neurons, the second LSTM layer has 32 neurons, and the third LSTM layer has 32 neurons. The activation function for the first, second, and third layers is the Tanh function.

[0206] The working mechanism of LSTM involves several advanced mathematical concepts:

[0207] The Sigmoid function (σ) is an activation function widely used in neural networks. It compresses any input in the range (-inf, inf) to a value in the interval (0, 1), making it very suitable for gating structures.

[0208]

[0209] Hyperbolic tangent (tanh): This is another activation function that maps any value to between -1 and 1. In LSTM, the tanh function helps regulate the flow of information and maintain gradient stability.

[0210]

[0211] Element-wise multiplication: In LSTM, the element-wise multiplication operation (denoted by *) is used in gated structures. This element-wise multiplication operation ensures that information can flow only when the gate is open.

[0212] Regarding data prefetching, since some data have greater context offsets, and recurrent neural networks and context offset prediction are relatively complex concepts and techniques, combining them may lead to more complex models and algorithms, increasing the difficulty of implementation and understanding. In particular, when dealing with long-term dependencies, it may make training more difficult, requiring more computational resources and more complex optimization strategies, while also increasing the overhead cost of the model.

[0213] LSTM solves two major problems when building neural network prefetch models:

[0214] 1. Long-term dependency problem: During the data offset prediction process, LSTM can capture long-term dependencies through memory units and gating mechanisms, making it more suitable for processing long sequence data than traditional neural networks.

[0215] 2. Combating vanishing and exploding gradients: By using gating mechanisms to control the flow of information, LSTM can effectively mitigate the effects of vanishing and exploding gradients.

[0216] (3) Fully connected layer:

[0217] In the fully connected layer Dense, all input nodes and output nodes are fully connected; three fully connected layers are used as the last fully connected layer of the RNN-LSTM recurrent neural network.

[0218] The first fully connected layer has 32 neurons, the second fully connected layer has 16 neurons, and the last fully connected layer has 8 neurons. The activation function for all three fully connected layers is the ReLU function.

[0219] f. Feedforward layer:

[0220] The feedforward layer is used to provide more learning capabilities. The feedforward network is a two-layer network. The activation function of the first layer is the ReLU function. For a vector x at each position in the input sequence, its expression is:

[0221] FFN(x)=max(0,xW1+b1)W2+b2.

[0222] The introduction of feedforward networks can effectively relearn the data output by the RNN-LSTM layer, and through final normalization, can output multi-step prediction sequences more accurately.

[0223] g. Output layer:

[0224] The number of neurons in the output layer is consistent with the feature dimension, and no activation function is set; the predicted value is generated directly.

[0225] Step 3: Using the preprocessed training dataset, validation dataset, and test dataset, the recurrent neural network fusion model is trained, validated, and tested respectively to achieve data prefetching.

[0226] In this embodiment of the invention, step 3 includes:

[0227] (31) Initialize the model: Integrate the RMM-LSTM recurrent neural network framework and add a multi-head attention mechanism to enhance the parallel learning ability of the model and its response to key features;

[0228] (32) Optimizer: Choose the Adam optimizer, which combines momentum and adaptive learning rate features to help optimize large-scale datasets quickly and stably;

[0229] (33) Training Epochs: Monitor model performance by checking the loss on the validation dataset. Stop training when the loss on the validation dataset stops decreasing or increases for several consecutive epochs. Parameter selection: 200 training epochs.

[0230] (34) Batch size: This parameter determines the amount of data used for each gradient update, affecting training stability and efficiency. Choose an appropriate batch size based on hardware performance and task requirements to achieve a balance between training stability and computational efficiency. Set it to Batch_Size = 64.

[0231] (35) Evaluation indicators:

[0232] ①Accuracy

[0233] Accuracy is used to measure whether the predicted value is within the error threshold. Its principle is as follows: it provides intuitive feedback on the overall performance of the model on the training dataset, validation dataset and test dataset.

[0234]

[0235] Among them, y i It is the true value of the i-th sample, i.e., the target value, which is the actual observed value of the time series. ε is the predicted value of the i-th sample, i.e., the predicted output of the model; ε is the error tolerance or threshold, which sets an error range, indicating that when the difference between the predicted value and the true value is less than the threshold, the prediction is considered correct; the error tolerance is 5% or 10%; n is the total number of samples.

[0236] ②Loss rate

[0237] A custom binary cross-entropy loss function was selected as the basic loss function for the recurrent neural network fusion model to evaluate its predictive performance.

[0238] The formula for the binary cross-entropy loss function is:

[0239]

[0240] Where N is the total number of samples; y i p is the true label of the i-th sample, with a value of 0 or 1; i It is the probability that the i-th sample predicted by the model belongs to category 1, and its value ranges between (0, 1).

[0241] The BCE loss function adjusts the model's parameters during training to ensure that the predicted probability p is... i Closer to the real label y i It encourages the model to output probability values ​​close to the true class, thereby improving classification accuracy.

[0242] To prevent the model from overfitting the training data, L2 regularization is introduced on the basis of the BCE loss function. L2 regularization, also known as weight decay, is based on the idea of ​​penalizing the model's parameters to force the model to maintain smaller weight values, thereby preventing the model from overfitting the training data.

[0243] The L2 regularization formula is:

[0244]

[0245] Where λ is a hyperparameter of regularization strength, controlling the weight of the regularization term; the larger λ is, the stronger the regularization effect. j These are the model parameters, i.e., weights; n is the total number of model parameters; the L2 regularization term is the sum of squares of all weights, and the larger the weight value, the greater the penalty.

[0246] By combining L2 regularization and the binary cross-entropy loss function (BCE), a new loss function is obtained, the formula of which is:

[0247]

[0248] The binary cross-entropy loss function (BCE) calculates the model's prediction error, while the L2 regularization term helps the model avoid overfitting by penalizing larger weights.

[0249] ③R 2 value

[0250] R 2 The value measures the proportion of the variance of the target variable explained by the model, and its range is [0,1]. The principle is illustrated by the following formula; R 2 It reflects the goodness of fit of the model; the closer the value is to 1, the stronger the model's ability to explain the target variable.

[0251]

[0252] Among them, y i It is the true value of the i-th sample, i.e., the target value; It is the predicted value of the i-th sample, which is the predicted output of the model; It is the average of the true values, that is It is the residual sum of squares (RSS), which is the sum of the squares of the differences between the actual and predicted values. It is the total sum of squares (TSS), which is the sum of the squares of the differences between the true values ​​and the true mean.

[0253] (36) Training strategy:

[0254] To improve the multi-step prediction performance of the nonlinear autoregressive neural network NARX-NN, a prediction feedback retraining strategy is adopted. The prediction feedback retraining strategy is designed to address the inconsistency between the inputs in the training phase and the multi-step prediction phase of the NARX-NN model, thereby reducing the discrepancy and improving the multi-step prediction performance of the model.

[0255] The basic idea of ​​FR is to reconstruct training samples using the error-laden single-step prediction results and retrain the model to enhance its robustness to prediction errors. The specific steps are as follows:

[0256] ① First, use the initial dataset and complete the initial training using the conventional training strategy, that is, use the backpropagation algorithm to train the network. At this time, the input target variables are all actual observation values.

[0257] ② Use the trained network to make a single-step prediction to obtain the prediction result; single-step prediction ensures that the error of the prediction result is within a reasonable range. If multi-step prediction results are used, the reconstructed sample in step ③ below will deviate from the dynamic characteristics of the system to be modeled due to the excessive prediction error.

[0258] ③ Replace the measured target value in the model input with its corresponding single-step prediction result to obtain a new training sample. Multiple replacements are required to reconstruct the dataset. The collective operation is as follows: given the real data of the first 63 time steps, predict the data of the next time step, replace the real data in the prediction time step with the predicted data, and complete one reconstruction of the dataset. The data of 64 time steps is one cycle. Repeat the operation until the data in all cycles of the entire dataset has been replaced with the real value and the predicted value.

[0259] ④ Retrain the network using the reconstructed samples; after training, check whether the validation set loss is greater than the recorded minimum validation set loss for several consecutive training rounds; if so, it indicates that the model's performance on the validation set no longer improves, end the training, and obtain a multi-step prediction model; otherwise, if the validation set loss decreases in the current round, update the minimum validation set loss, proceed to steps ② and ③, and repeat single-step prediction and network training.

[0260] In data-intensive applications such as high-performance computing (HPC), artificial intelligence, and big data processing, data access latency is a key bottleneck for performance optimization. Traditional data prefetching strategies often suffer from insufficient prefetching accuracy and excessive memory consumption, resulting in slower-than-expected data access speeds. To address these issues, the technical solution provided in this invention not only effectively utilizes the RNN-LSTM model's ability to capture dependencies in long-term series and its strong modeling capabilities for long-term trends and periodic patterns in time-series data, but also leverages the LSTM's gating mechanism to effectively solve the gradient vanishing problem. Furthermore, by integrating with a multi-head attention mechanism, it further captures the connections between features such as temporal correlation, user behavior, and application characteristics. Finally, a designed feedback retraining (FR) strategy further enhances the model's multi-step prediction performance, thereby improving the accuracy and efficiency of model prefetching. This fusion technique aims to leverage the powerful foundation of RNN-LSTM in processing long-term series data and the specific multi-head attention mechanism to improve the accuracy and efficiency of data prefetching and reduce data reading latency.

[0261] The technical solution provided by this invention includes a method that involves acquiring an initial dataset and dividing it proportionally to obtain a training dataset, a validation dataset, and a test dataset; preprocessing the initial dataset to obtain a preprocessed dataset; constructing a recurrent neural network fusion model using a multi-head attention mechanism and an RNN-LSTM recurrent neural network; and training, validating, and testing the recurrent neural network fusion model using the preprocessed training dataset, validation dataset, and test dataset, respectively, to achieve data prefetching. This method, through fusion technology, leverages the powerful foundation of RNN-LSTM in processing long-term sequences and a specific multi-head attention mechanism to improve the accuracy and efficiency of data prefetching and reduce data reading latency.

[0262] The various steps in the embodiments of the present invention can be performed by electronic devices. These electronic devices include, but are not limited to, mobile phones, tablet computers, portable PCs, and desktop computers.

[0263] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the electronic device containing the computer-readable storage medium to execute the above-described embodiment of the data prefetching method based on multi-head attention mechanism and RNN-LSTM network.

[0264] Figure 11 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 11As shown, the electronic device 21 includes a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the data prefetching method based on the multi-head attention mechanism and RNN-LSTM network in the embodiment. To avoid repetition, it will not be described in detail here.

[0265] Electronic device 21 includes, but is not limited to, processor 211 and memory 212. Those skilled in the art will understand that... Figure 11 This is merely an example of electronic device 21 and does not constitute a limitation on electronic device 21. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0266] The processor 211 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0267] The memory 212 can be an internal storage unit of the electronic device 21, such as a hard disk or RAM of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 21. Furthermore, the memory 212 can include both internal and external storage units of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0268] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0269] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data prefetching method based on a multi-head attention mechanism and an RNN-LSTM network, characterized in that, The method comprises: Step 1, obtaining an initial data set, and performing proportion division to obtain a training data set, a verification data set and a test data set; preprocessing the initial data set to obtain a preprocessed data set; Step 2, constructing a recurrent neural network fusion model through a multi-head attention mechanism and an RNN-LSTM recurrent neural network; Step 3, training, verifying and testing the recurrent neural network fusion model through the preprocessed training data set, the verification data set and the test data set respectively to realize data prefetching; The obtaining of the initial data set in step 1 comprises: Firstly, the cache area of the flash chip of the Master node is read to generate a timestamp, and the timestamp is generated once every 5 minutes from the time record of the unique address of the first file read from the cache area; secondly, statistics are performed once every 5 minutes to record all unique pages in the current memory; the memory offset between all unique pages is recorded; the network occupancy rate of reading each unique page is recorded; finally, data integration is performed to generate a data set in the form of a timestamp timescamp, an offset target, a broadband occupancy rate dynamic_feat, a unique page mapping ID; The proportion division comprises: collecting 10,000 data, removing data with large offsets, and leaving 8,000 effective data; and using a custom code to set the training data set, the verification data set and the test data set according to a proportion of 6:2:2; The preprocessing of the initial data set to obtain the preprocessed data set comprises: Data normalization: the normalization operation adopted is realized by the MinMaxScaler function in Sklearn, and MinMaxScaler scales all values to a specified range, and the formula is: where X is the characteristic value of the original data; X min is the minimum value of the characteristic value X of the original data in the data; X max is the maximum value of the characteristic value X of the original data in the data; X scaled is the normalized final result; The broadband occupancy rate dynamic_feat, the unique page mapping ID and the offset target column in the data set are subjected to data normalization operation; Data prediction mode processing: a data input and prediction mode is defined, and the mode is realized by using a custom function create_fixed_sequnences, and the specific operation steps are: first input sequence_length as the input time sequence step length of the model, and then the model returns a prediction result, that is, a prediction time step is obtained, and the corresponding timestamp is returned; wherein, the sequence_length adopted is set to 64, representing that 64 step time sequences are input to enable the model to learn and thus predict a step time sequence; The recurrent neural network fusion model in step 2 comprises: a, input layer: the dimension of the input layer is (sequence_length, feature_dim); wherein, sequence_length represents the time step number; feature_dim represents the feature dimension of each time step; the input layer inputs the original time sequence data into the model for feature extraction; b. Self-attention mechanism layer: In the attention mechanism, each element of the input is assigned a weight value according to its relationship with other elements, which is calculated by dot product. The formula for calculating the attention mechanism is: where Q is the query vector query, representing the target of current interest; K is the key vector key, representing the features in all inputs; V is the value vector value, i.e., the actual information associated with the key vector; d k is the dimension of the key vector, used for scaling; c. Residual block: The residual block directly passes the calculation result of the low-level network to the high-level network. The formula for the residual block is: X l+1 = X l + A(X l + W l ); wherein X l is the input of the l-th layer, A(X l + W l ) is the output of the multi-head self-attention mechanism module with mask of the l-th layer; d、Normalization layer: select layer normalization Layer Normalization to process data; layer normalization is to normalize all neurons in a layer; suppose the i-th input of the neurons in the l-th layer is z i (l) The expressions of the mean and variance are respectively wherein n (l) is the number of neurons in the lth layer; Layer normalization is defined as: where γ denotes a parameter vector of scaling, β denotes a parameter vector of translation, the dimension of the parameter vectors γ and β are the same as the dimension of z (l) . e. RNN-LSTM recurrent neural network layer: The RNN-LSTM recurrent neural network layer is composed of two layers of recurrent neural network (RNN) and three layers of long short-term memory network (LSTM) and three layers of fully connected layer. (1) The basic unit of RNN is a recurrent unit that receives an input and a hidden state from the previous time step and outputs the hidden state at the current time step. The basic RNN structure consists of an input layer, a hidden layer, and an output layer. Expanding the basic RNN structure in the time dimension, we get the overall network structure framework of RNN. The RNN network receives an input x at time t t After that, the value of the hidden layer is s t The formula is as follows: s t = f(U x x + W x s) t t-1 );​ U is the weight matrix for input x, W is the previous hidden layer value s t-1 as the current input weight matrix, f is the activation function, s t value depends on x t and s t-1 ; The output value is o t The calculation formula is as follows: o t = g(V x s t ); V is the weight matrix of the output layer, and g is the activation function. The hidden layer has two inputs: the first is the weight matrix U from the input layer to the hidden layer, and the second is the input x. t The product of the two; the second is the value of the previous hidden layer s. t-1 The product of W and s, i.e., s calculated at the previous time step. t-1 Cache is required, along with the input x t Calculate and output the final 'o'. t ; Two layers of RNN neural network are used, with 128 neurons in the first layer and 64 neurons in the second layer. The activation function of the first and second layers is ReLU function. The two layers of RNN are used to receive and process the output results of the multi-head attention mechanism layer, learn the relationship between multi-step time series features, and output them to the long short-term memory network (LSTM) layer. (2) The core of LSTM is a unit that contains a forget gate, an input gate, and an output gate. Each gate is a special structure that controls the flow of information, passing information from the previous time step to the current time step to capture the time dependence in the data. The LSTM neural network includes: Forget Gate: The role of the forget gate is to decide what information to discard from the cell state. It is calculated by the following formula: f t = σ(W f x [h t-1 , x t + b f ); where f t represents the forget gate output at time t, σ is the sigmoid function, W f and b f are the weights and bias of the forget gate, h t-1 is the previous hidden state, x t is the current input; Input Gate: The input gate is used to update the cell state, which consists of a sigmoid layer and a tanh layer. i t = σ(W i x [h t-1 , x t + b i ); wherein i t is an input of the input gate, is a vector of candidate values; Output Gate: The output gate is used to determine the next hidden state, which contains information about the previous input and is used for prediction. o t = σ(W o x [h t-1 , x t + b o ); h t = o t tanh(C t ); wherein o t is the output of the output gate, h t is the current hidden state, C t is the current cell state; The LSTM layer uses three layers, with 64 neurons in the first layer, 32 neurons in the second layer, and 32 neurons in the third layer. The activation function of the first, second, and third layers is the Tanh function. (3) Fully connected layer: In the fully connected layer (Dense), all input nodes and output nodes are fully connected. Three layers of fully connected layers are used as the last fully connected layer of the RNN-LSTM recurrent neural network layer. The first layer of fully connected layer has 32 neurons, the second layer of fully connected layer has 16 neurons, and the last layer of fully connected layer has 8 neurons. The activation function of the three layers of fully connected layer is the ReLU function. f. Feedforward layer: The feedforward layer is used to provide more learning ability, and the feedforward network is a two-layer network, the activation function of the first layer is ReLU function, and the expression is as follows for the vector x at each position in the input sequence: FFN(x)=max(0,xW1+b1)W2+b2; g, output layer: The number of neurons of the output layer is consistent with the feature dimension, and no activation function is set, and the prediction value is directly generated.

2. The method of claim 1, wherein, The step 3 comprises: (31) initialization model: integrate the RMM-LSTM recurrent neural network framework and add a multi-head attention mechanism to enhance the parallel learning ability and response to key features of the model; (32) optimizer: select the Adam optimizer which combines the momentum and adaptive learning rate characteristics to help quickly and stably optimize large-scale data sets; (33) training rounds Epochs: monitor the model performance through the validation data set loss, and stop training when the validation data set loss no longer decreases or continuously increases for several rounds; the parameter is selected as 200 training rounds; (34) batch size Batch_Size: this parameter determines the amount of data used for gradient update each time, which affects the training stability and efficiency; according to the hardware performance and task demand, a suitable batch size is selected to balance the training stability and computing efficiency; it is set as Batch_Size=64; (35) evaluation index: ① accuracy The accuracy is used to measure whether the prediction value is within the error threshold range, and its principle is shown as follows, which provides intuitive feedback for the comprehensive performance of the model on the training data set, the validation data set and the test data set; wherein y i is the true value, i.e. the target value, of the ith sample, and x is the predicted value, i.e. the model's prediction output, of the ith sample; ε is the error tolerance or threshold value, which sets an error range, indicating that when the difference between the predicted value and the true value is less than the threshold value, the prediction is considered correct; the error tolerance is 5% or 10%; n is the total number of samples. ② loss rate The binary cross-entropy loss function is selected as the basic loss function of the recurrent neural network fusion model to judge the prediction performance of the recurrent neural network fusion model; The formula of the binary cross-entropy loss function is as follows: where N is the total number of samples; y i is the true label of the i-th sample, taking values 0 or 1; p i is the probability that the i-th sample belongs to class 1, predicted by the model, taking values in the range (0, 1). The BCE loss function adjusts the model's parameters during the training process to make the predicted probability p i closer to the true label y i ; it encourages the model to output probability values close to the true class, thereby improving the accuracy of classification; In order to prevent the model from overfitting the training data, L2 regularization is introduced on the basis of the BCE loss function; L2 regularization, also known as weight decay, its core idea is to punish the model parameters, forcing the model to keep small weight values, so as to prevent the model from overfitting the training data; The formula of L2 regularization is as follows: where λ is a hyperparameter of the regularization strength, controlling the weight of the regularization term, the larger λ is, the stronger the regularization effect is; w j is the parameter of the model, i.e., the weight; n is the total number of model parameters; the L2 regularization term is the sum of squares of all weights, the larger the weight value is, the greater the penalty is; The L2 regularization and the binary cross-entropy loss function BCE are combined to obtain a new loss function, and the formula is as follows: The binary cross-entropy loss function BCE calculates the prediction error of the model, and the L2 regularization term punishes the larger weights to help the model avoid overfitting; 2 values​ R 2 The value measures the proportion of the variance of the target variable, and the value range is [0, 1], and the principle is shown in the following formula; R 2 Reflects the goodness of fit of the model, and the value is closer to 1, indicating that the model has stronger explanatory ability for the target variable; wherein y i is the true value, i.e. the target value, of the i-th sample; is the predicted value, i.e. the model’s predicted output, of the i-th sample; is the mean of the true values, i.e. is the residual sum of squares, RSS, i.e. the sum of the squares of the differences between the true values and the predicted values; is the total sum of squares, TSS, i.e. the sum of the squares of the differences between the true values and the mean of the true values; (36) training strategy: In order to improve the multi-step prediction performance of the nonlinear autoregressive neural network NARX-NN, a prediction value feedback retraining FR strategy is adopted; the prediction value feedback retraining strategy is aimed at the inconsistency of the input in the training stage and the multi-step prediction stage of the NARX-NN model, so as to reduce the difference and improve the multi-step prediction performance of the model; The basic idea of FR is to use the single-step prediction result with error to reconstruct the training sample and retrain the model to enhance the robustness of the model to prediction error, and the specific steps are as follows: ①First, use the initial data set, using the conventional training strategy to complete the initial training, that is, using the back propagation algorithm to train the network, at this time the input target variable is the actual observation value; ②Using the trained network for single-step prediction, the prediction result is obtained; single-step prediction ensures that the prediction error is within a reasonable range, if multi-step prediction results are used, the reconstruction sample in step ③ below will deviate from the dynamic characteristics of the system to be modeled due to the large prediction error; ③Replace the measured target value in the model input with its corresponding single-step prediction result to obtain a new training sample, which needs to be replaced multiple times to reconstruct the data set; the collective operation is: given the true data of the first 63 time steps, the data of the next time step is predicted, and the predicted data is replaced with the real data in the prediction time step to complete one reconstruction data set operation, 64 time steps of data is a period, repeat the operation until all the data in the entire data set is replaced with the true value and the prediction value; ④Use the reconstructed sample to train the network again; after training, check whether the validation set loss is greater than the recorded minimum validation set loss for several consecutive training rounds; if so, it indicates that the performance of the model on the validation set is no longer improved, the training is ended, and a multi-step prediction model is obtained; otherwise, if the validation set loss is reduced in the current round, update the minimum validation set loss, enter steps ② and ③, and repeat single-step prediction and network training.

3. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a stored program, wherein when the program runs, the device where the computer readable storage medium is located executes the data prefetching method based on the multi-head attention mechanism and the RNN-LSTM network in any one of claims 1-2.

4. An electronic device, comprising: Comprise: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs include instructions, when the instructions are executed by the device, the device executes the data prefetching method based on the multi-head attention mechanism and the RNN-LSTM network in any one of claims 1-2.

Images