A time series data reconstruction method based on a multi-head attention mechanism

Abstract

The application relates to a time series data reconstruction method based on a multi-head attention mechanism, which adopts a bidirectional sequence-to-sequence model based on a multi-head self-attention mechanism to reconstruct data, the bidirectional sequence-to-sequence model based on the multi-head self-attention mechanism comprising: an encoder part, which takes sequence data on both sides of a missing part of time series data as input of the encoder; the data on both sides respectively enter two multi-head self-attention layers to extract key features; then the two multi-head self-attention layers are respectively input into two bidirectional gated recurrent unit neural networks; a decoder part, which combines output data of the two encoders with a sequence of decoder embedding data; a multi-head attention mechanism is used to combine the key features learned from the two encoders to predict input time series data, and output reconstructed time series data. The method and the model effectively utilize time patterns in the time series, sufficiently learn the correlation between variables, and have strong generalization ability.

Description

Technical Field

[0001] This invention relates to the field of underwater wireless sensor network data processing technology, and in particular to a time series data reconstruction method based on a multi-head attention mechanism. Background Technology

[0002] Time-series data sensed by Underwater Wireless Sensor Networks (UWSNs) plays a crucial role in prediction and decision-making in applications such as marine environmental monitoring, resource exploration, intelligent navigation, disaster early warning, and military operations. For example, UWSN-based ship route planning is of great significance for the safety and low energy consumption of ship navigation. Figure 1 As shown, UWSNs typically consist of many underwater sensor nodes, each capable of sensing different information in the environment. They can relay and forward data. The sensor nodes collaborate with each other through distributed algorithms to efficiently complete tasks such as environmental perception, information collection, and data packet transmission.

[0003] However, UWSNs may suffer from issues such as node damage, unstable wireless channels, inter-channel interference, or the need for regular equipment maintenance. These problems can lead to the loss of significant amounts of sensor time-series data over a certain period. UWSN communication is also susceptible to interference from various underwater environments. For example, displacement of reflection points caused by wind, waves, and currents can result in intermittent transmission connections. Scattering and Doppler spread caused by variations in transmission path length can lead to multipath effects and inter-symbol interference. Of course, environmental noise and noise in specific scenarios can also negatively impact signals, affecting communication accuracy and potentially causing signal delays or loss. Changes in temperature, salinity, and pressure can affect signal transmission in water. These physical property variations affect the availability, reliability, and accuracy of water as a transmission medium. All of these factors can potentially interfere with UWSN data transmission and may even lead to data loss. These factors are complex and variable in different water bodies. Therefore, data recovery methods applied to UWSNs need to have a certain degree of generalization capability.

[0004] Common data recovery methods include smoothing, regression, and interpolation. These methods are effective for single, static datasets but unsuitable for complex datasets. They are also inaccurate when reconstructing continuously missing data. In some cases, continuous observations from sensor nodes bring time-series data. Time-series data adds an extra challenge to identifying temporal patterns. Furthermore, most environmental data collected by UWSNs is multidimensional, and parameters exhibit correlations. This makes the ability to learn temporal patterns and inter-series dependencies a key evaluation criterion for time-series models. Therefore, it is crucial to develop a model with strong generalization ability that can effectively identify temporal patterns and inter-series dependencies.

[0005] Currently, there are two main approaches to reconstructing consecutively missing values ​​in time series: specialized models and data-driven general-purpose models. Specialized models can address data missingness issues in specific domains. These models can learn time patterns and dependencies between sequences well. However, these models typically lack strong generalization ability. Significant adjustments are usually required when relevant parameters change or the environment evolves. Data-driven general-purpose models have received considerable attention. Machine learning and deep learning are frequently used for the prediction and reconstruction of consecutively missing data. These methods can extract deeper hidden features through data mining. These models typically have strong generalization ability. However, different model architectures vary in their ability to learn time patterns and dependencies between sequences. Since the environment in UWSNs is often more complex, higher standards are needed for the model's generalization ability and its ability to identify time patterns and dependencies between sequences. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a time series data reconstruction method based on a multi-head attention mechanism.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A time series data reconstruction method based on a multi-head attention mechanism is disclosed. This method employs a pre-trained bidirectional sequence-to-sequence model based on a multi-head self-attention mechanism for data reconstruction. The bidirectional sequence-to-sequence model includes an encoder and a decoder.

[0009] In the encoder part, the sequence data on both sides of the missing part of the time series data are used as the encoder input; the data on both sides are respectively fed into two multi-head self-attention layers for key feature extraction; and then respectively fed into two bidirectional gated recurrent unit neural networks to capture forward and backward time patterns.

[0010] In the decoder section, the output data of the dual encoders is combined with the sequence of data embedded in the decoder; a multi-head attention mechanism is used to combine key features learned from the two encoders to predict the input time series data; and the reconstructed time series data is output.

[0011] Furthermore, the specific steps for extracting key features in the decoder's head self-attention layer are as follows:

[0012] To obtain K, V, Q by performing a linear transformation on the embedded data of the same encoder input data;

[0013] Calculate the inner product of matrices Q and K, and divide the result by .

[0014] The weights of V are obtained using an activation function;

[0015] The result of the self-attention mechanism is obtained by performing an inner product between the weight matrix of V and V.

[0016] Multiple self-attention calculations are concatenated together and linearly mapped to the same dimension as the input matrix.

[0017] Furthermore, the specific steps for the decoder to predict the input time series data include:

[0018] By utilizing the temporal patterns captured from the valid data on both sides of the missing data, the hidden state output by the decoder is used as the input set of the decoder;

[0019] Based on the multi-head attention mechanism of the decoder, the input information is reweighted based on the attention score to obtain the multi-head attention vector a. t ;

[0020] At each time index t, the decoder bases its work on the state at time index t-1, the predicted value obtained at t-1, and the multi-head attention vector a at the current time index t. t This is used to update the state of the recurrent gated recursive unit layer, and each gated recursive unit contains a linear layer, which generates the final prediction y. t :

[0021] y t =Linear(W[s t ;a t ]+b)

[0022] s t =GRU(y t-1 ,s t-1 ,a t )

[0023] Among them, s tIt is the hidden state of the decoder at time index t, a t It is the decoder multi-head attention context vector at time index t, [s t ;a t ] is the concatenation of the decoder's hidden state and context vector, and b is the linearization parameter.

[0024] Furthermore, the multi-head attention vector a t The specific calculation steps are as follows:

[0025] The decoder embedded data is processed by a multi-head self-attention algorithm. The result of the algorithm is linearly transformed to obtain Q, which is then used to calculate the attention weight a1 by a multi-head attention mechanism with the linearly transformed K and V of the concatenated feature map.

[0026] In time index t, the multi-head attention vector a t Described as a weighted sum of the hidden states passed by the dual encoders:

[0027]

[0028] Each hidden state h i weight α ti The calculation is as follows:

[0029] α ti =softmax(attention(s) t-1 ,h i ))

[0030] The attention layer is a multi-head self-attention layer that uses Softmax as the activation function.

[0031] Furthermore, the encoder section includes a residual block after the multi-head self-attention mechanism for soft thresholding.

[0032] Furthermore, the training steps for the bidirectional sequence-to-sequence model based on the multi-head self-attention mechanism are as follows:

[0033] A sliding window algorithm is used to generate a large number of training samples from the collected time series data;

[0034] In the encoder part, the sequence data on both sides of the missing part are used as the input of the network model. After the multi-head self-attention mechanism and residual block calculation, the output is used as the input of the bidirectional gated recurrent unit neural network.

[0035] When the output data of the dual encoders are concatenated and enter the decoder, the output data of the dual encoders is combined with the sequence of the embedded data of the decoder. A multi-head attention mechanism is used to train the input time series data by combining key features learned from the two encoders, and the reconstructed time series data is output.

[0036] The output data is evaluated, and the network parameters are updated in reverse.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1) The MSSM provided by this invention effectively utilizes the time patterns in time series, fully learns the correlation between variables, and can effectively solve the problem of continuous missing time series data in harsh communication environments of UWSNs, and has strong generalization ability.

[0039] 2) Building upon the sequence-to-sequence model with dual encoders, a multi-head self-attention mechanism, a masked multi-head self-attention mechanism, and residual blocks are introduced. The combination of multi-head self-attention and bidirectional gated recurrent unit neural networks can capture key features in multidimensional long-term sequence data. Soft thresholding is achieved using residual blocks, thereby reducing interference from various noise sources. This novel model structure makes MSSM well-suited for handling the problem of consecutively missing data in UWSNs. Attached Figure Description

[0040] Figure 1 Schematic diagrams of UWSNs in different dimensions;

[0041] Figure 2 This is a schematic diagram of the structure of the MSSM of the present invention;

[0042] Figure 3 This is a schematic diagram illustrating a task example of the present invention;

[0043] Figure 4 This is a schematic diagram of the encoder portion of the present invention;

[0044] Figure 5 This is a schematic diagram of the bidirectional gated recursive unit of the present invention;

[0045] Figure 6 This is a schematic diagram of the decoder part of the present invention;

[0046] Figure 7 Comparison of model metrics for different datasets

[0047] Figure 8 The image shows a comparison of data prediction and reconstruction, where (a) is the CARINA dataset; (b) is the Meteorological dataset; (c) is the Volume dataset; and (d) is the Intel dataset. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0049] Example 1

[0050] 1. Network Model:

[0051] This invention addresses the problem of continuous missing time-series data in harsh communication environments of UWSNs by proposing a time-series data reconstruction method based on a multi-head attention mechanism and a novel bidirectional sequence interpolation model, namely the bidirectional sequence-to-sequence model (MSSM) based on a multi-head self-attention mechanism. First, a sliding window algorithm is used to generate a large number of training samples from the collected time-series data, reducing time complexity. Second, in the encoder part, the sequence data on both sides of the missing portion are used as input to the network model. Third, after computation by the multi-head self-attention mechanism and residual blocks, the outputs are used as inputs to the bidirectional gated recurrent unit neural network. Then, when the output data of the dual encoders is concatenated and enters the decoder part, it is combined with the sequence of data embedded in the decoder. Finally, the output data is evaluated, and the network parameters are updated in reverse. Figure 2 This is a network structure diagram of MSSM.

[0052] To facilitate the description of MSSM, its internal structure will be shown in detail. First, the input multivariate time series data will be defined as follows:

[0053]

[0054] Where, {x1,…,x t} represents the column vector corresponding to t time steps. This represents the horizontal quantity corresponding to k sequences. This represents the vector of these k sequences at time step t. This represents a vector containing all the variable values ​​of the i-th sequence.

[0055] For the case where UWSNs data loss occurs when all data within a certain time period is missing, assume that the consecutive missing time period for data in the i-th sequence is m, starting from time index l+1. Let D... left and D right The remaining data to the left and right of the missing sequence can be represented as:

[0056]

[0057]

[0058]

[0059] Where l and r represent the size of the available time series on both sides of the missing data, and k represents the number of variables measured at each time step. This invention uses known time series data of all variables on both sides of the missing sequence to predict and reconstruct consecutive missing values ​​of one or more variables within a certain time period. The rationale for this design is that it can fully infer the complex inter-series dependencies between the observed variables and capture the temporal patterns of the time series data. Furthermore, we can define the predicted sequence as... Figure 3 This is a schematic diagram illustrating a task example of the present invention. For the prediction and reconstruction of continuously missing time series data in UWSNs, the present invention uses a sliding window strategy for data training. The target size of each training sample is set to m, and the model has two input sequences of sizes l and r, respectively, for the forward and reverse directions.

[0060] 2. Encoder section:

[0061] To fully utilize the time series data on both sides of the missing value for complete data prediction and reconstruction, this invention designs two encoders to process the input information from both sides of the missing sequence, a multi-head attention layer for extracting key features, and a residual layer for soft thresholding, thereby improving the robustness of the model, accelerating convergence, and improving the model's computational performance.

[0062] 2.1 Dual Encoder Multi-Head Self-Attention Mechanism

[0063] Multi-head attention is a key component of the Transformer model. Compared to self-attention, multi-head attention prevents the model from excessively focusing its attention on its own position when encoding information about the current location. For example... Figure 4 As shown, the data from both sides enter two multi-head self-attention layers to extract their key features, thereby accelerating the convergence speed. A residual layer is then applied for soft thresholding to denoise the UWSNs' sensory data.

[0064] In practical applications, given the same set of inputs, we want the model to learn different behaviors based on the same attention mechanism, and then combine these different behaviors as knowledge, such as capturing dependencies of various ranges within a sequence, including short-term and long-term dependencies. Therefore, multi-head attention mechanisms, which represent different subspaces of the input data, can effectively describe inter-sequence dependencies in time series data recovery. Unlike using a single attention pooling, a multi-head attention mechanism can independently learn n different linear projections. These n transformed queries, keys, and values ​​are then attention-pooled in parallel. Finally, the outputs of these n attention poolings are concatenated and transformed through another learnable linear projection to produce the final output. This design is called a multi-head attention mechanism, where the number of output groups is called the number of heads.

[0065] Taking the known data to the left of the missing data as an example, the following is the specific formula derivation:

[0066]

[0067] Here, K, V, and Q can be understood as linear transformations of the same embedded data. Next, the inner product of matrices Q and K is calculated; the result represents the attention intensity between sequences, and is then divided by... To prevent gradient vanishing, the SoftMax activation function is used to obtain the weights of V. Finally, the inner product of the weight matrix of V and V is performed to obtain the result of the self-attention mechanism. The inner product is used for more efficient computation. The self-attention formula is as follows:

[0068]

[0069] The multi-head self-attention mechanism essentially concatenates the results of multiple self-attention calculations and linearly maps them to the same dimension as the input matrix. The calculation formulas are shown in equations (7) and (8) below:

[0070]

[0071] MultiHead(Q,K,V)=Concat(head1,…,head n (8)

[0072] 2.2 Bidirectional Gated Recursive Unit

[0073] In the proposed model, the latter half of the encoder consists of bidirectional gated recursive units, such as... Figure 5As shown, this model is used to process time series inputs on both sides of the missing sequence. GRU is a recurrent neural network with a structure similar to LSTM. Compared to LSTM, GRU replaces the forget gate and input gate with a single update gate, mixes cell states and hidden states, and makes some other modifications. The final model structure is more simplified, with fewer parameters and fewer gates, accelerating convergence and thus improving prediction accuracy.

[0074] Furthermore, it can effectively handle the long-term memory and gradient backpropagation problems of RNNs. Information can be selectively extracted; for example, only data with obvious features can be retained, while unnecessary noise or data with few features can be eliminated. This model is suitable for processing and predicting time series data. Simultaneously, the bidirectional gated recursive unit solves the problem that gated recursive units can only obtain temporal information from time series data in one direction. This model is applied in both forward and backward directions, and then the results obtained in the two opposite directions are concatenated as the final output. The bidirectional gated recursive unit can capture forward and backward temporal patterns, thus enabling better data prediction and reconstruction.

[0075] The following is the formula for the gated recursive unit, z t This represents the update gate of the GRU, r t denoted as reset gate, tanh as tanh activation function, σ as sigmoid function, and ⊙ as element-wise multiplication.

[0076] r t =σ(W r x t +U r h t-1 +b r (9)

[0077] z t =σ(W z x t +U z h t-1 +b z (10)

[0078]

[0079]

[0080] 3. Decoder section:

[0081] In the decoder section, a multi-head attention mechanism is used to train the input time-series data by combining key features learned from both encoders. As per the previous section, each encoder generates a series of hidden states H. The last hidden state of each encoder is used as the initial hidden state of the decoder. Therefore, to utilize the temporal patterns captured from the valid data on both sides of the missing data, the hidden states H output by the decoder are used as one of the input sets of the decoder.

[0082] Decoder in =H=[H left H right (13)

[0083] Among them, H lef and H right These represent the hidden states output by encoder 1 and encoder 2, respectively.

[0084] Figure 6 The decoder's detailed structure includes a masked multi-head self-attention layer, a residual layer, a recursive attention layer, and a recurrent gated recursive unit layer. In the decoder, the prediction and reconstruction of missing values ​​occur sequentially. At each time index t, the decoder bases its work on the state at time index t-1, the predicted value obtained at t-1, and the attention vector a at the current time index t. t This is used to update the state of the gated recursive GRU unit. Each gated recursive GRU layer contains a linear layer that generates the prediction y. t The calculation process is as follows:

[0085] y t =Linear(W[s t ;a t ]+b) (14)

[0086] s t =GRU(y t-1 ,s t-1 ,a t (15)

[0087] Among them, s t It is the hidden state of the decoder at time index t, a t It is the decoder multi-head attention context vector at time index t, and [s t ;a t The [] symbol is the concatenation of the decoder's hidden state and context vector. The linear layer produces the final prediction y. t The decoder's embedded data, after undergoing a linear transformation of the result of the masked multi-head self-attention calculation, yields Q. This Q is then used in conjunction with K and V, the linearly transformed concatenated feature maps, to calculate the attention weight a1 using a multi-head attention mechanism.

[0088] In time index t, the multi-head attention vector a t This can be described as a weighted sum of the hidden states passed by the dual encoders:

[0089]

[0090] Each hidden state h i weight α ti Calculation as follows

[0091] α ti =softmax(attention(s) t-1 ,h i (17)

[0092] The attention layer is a multi-head self-attention layer. Softmax is used as the activation function to ensure weight normalization.

[0093] Based on the multi-head attention mechanism of the decoder, the model can reweight the input information based on the attention score, as shown in (14) and (15). Since UWSNs have continuous missing data at different time indices, the model can find key information from the input sequence when predicting missing values ​​at different time indices, and effectively capture the temporal patterns of time series data and the dependencies between variables.

[0094] Specific implementation verification:

[0095] The performance of the proposed model was evaluated using partial data from the CARINA Atlantic and Southern Ocean Water Quality Dataset, the Meteorological Marine Meteorological Dataset, the Metro_Interstate_Traffic_Volume Traffic Dataset, and Intel Lab Data.

[0096] Table 1 Parameter Settings

[0097]

[0098]

[0099] The experimental section compares MSSM with the following algorithms: Dual-SSIM, KNN, EM, MICE, and LOCF.

[0100] Dual-SSIM is a missing value imputation model that includes two encoders with bidirectional gated recurrent units and a cross-attention mechanism. These handle temporal information before and after the missing value gap, effectively combining them with the decoder. KNN is a method that finds the k nearest neighbors of an observation with a missing value and then predicts the missing value based on the non-missing values ​​in the sequence. EM stands for Expectation-Maximization scheme, a probabilistic imputation method that computes the maximum likelihood estimate of an incomplete dataset. MICE is a prediction model that imputes missing data through multiple iterations. In each iteration, the specified variable is estimated using other variables in the dataset, and this process continues until convergence. LOCF is a method that uses the last data of an object that exits the dataset before exiting to predict subsequent data.

[0101] The following four classic machine learning evaluation metrics are used to assess the accuracy of the proposed model in missing value prediction and reconstruction. These metrics include Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Root Mean Square Logarithmic Error (RMSLE), and Dynamic Time Warping (DTW). The formulas for the four metrics are shown below:

[0102]

[0103]

[0104]

[0105]

[0106] Different metrics have their advantages and disadvantages, and the appropriate metrics for evaluating prediction accuracy may vary in practice. MAE and RMSE are two commonly used variable precision metrics, which are scale-dependent indicators that represent the difference between the actual sequence and the predicted value. RMSLE can be understood as RMSE on a logarithmic scale; the target value is usually non-negative, and it tends to predict values ​​smaller than the target value. RMSLE penalizes underprediction more than overprediction, therefore, it is suitable for scenarios where greater underprediction loss is required. DTW, on the other hand, is a time series similarity metric that measures the similarity between two time series of different lengths based on the nearest proximity principle; it is a non-linear regularization technique.

[0107] In real-world UWSNs, random data loss can occur. Encoder and decoder designs based on multi-head attention mechanisms enhance the ability to generate imputation results for variable-length missing data. Furthermore, the amount of available data around the missing value can be configured and adjusted according to the user's experimental needs. Therefore, MSSM aims to fully learn and utilize the inter-sequence dependencies and temporal patterns of time series data to handle data gaps of arbitrary sizes in time series.

[0108] To ensure a fair comparison with other imputation methods, a fixed gap size was first selected, and three datasets of different lengths were used for model evaluation. The algorithms used in the experiments were all scalable to support gap sizes of varying sizes. A gap size of 6 was chosen. Furthermore, to effectively utilize information near the gap, 10 data points from each side of the gap were used as model input. The model input size can also be adjusted based on the time pattern of the target variable. MSSM has a wide range of applicability and can predict and reconstruct missing values ​​for larger gaps.

[0109] Table 2. Evaluation Indicators for Evaluation Algorithms

[0110]

[0111] Table 2 shows the normalized imputation performance of six data prediction and reconstruction algorithms on four datasets of different sizes. The MSSM model achieved the best performance in MAE, RMSLE, RMSE, and DTW scores across all imputation tasks. For example, when filling CARINA's Phos data, MSSM's MAE, RMSLE, RMSE, and DTW scores were 0.0871, 0.0815, 0.1073, and 0.1927, respectively; when filling Meteorological's dpt data, MSSM's MAE, RMSLE, RMSE, and DTW scores were 0.0724, 0.0621, 0.0939, and 0.1907, respectively; when filling Volume's Volume Flow Rate RMS data, MSSM's MAE, RMSLE, RMSE, and DTW scores were 0.0349, 0.0233, 0.0415, and 0.1012, respectively; and when filling Intel's NO3 data, MSSM's MAE, RMSLE, RMSE, and DTW scores were 0.0481, 0.0419, 0.0547, and 0.1157, respectively. Furthermore, the performance variance was low across different test cases on the dataset, indicating that the proposed MSSM is robust and maintains good generalization performance across various datasets. Meanwhile, the Dual-SSIM model also demonstrated good experimental performance, therefore a detailed experimental comparison between MSSM and Dual-SSIM was conducted.

[0112] from Figure 7 As can be seen, MSSM improves the statistical score of MAE by at least 8.7% compared to Dual-SSIM on the CARINA dataset. Furthermore, RMSLE, RMSE, and DTW are also improved by at least 8.1%, 6.2%, and 7.6%, respectively. MSSM also achieves better performance on the Volume and Intel datasets. MSSM accurately predicts and reconstructs water quality, traffic flow, and chip index data, demonstrating its strong generalization ability. While the performance of the training portion of the dataset fluctuates with the increase in the number of missing data points, MSSM's performance remains robust. Therefore, it can be demonstrated that MSSM is well-suited for applications in environments with irregular data gaps, such as UWSN. Figure 7 The metrics comparison between MSSM and Dual-SSIM is shown across four types of datasets.

[0113] MSSM and Dual-SSIM achieved superior scores across various metrics, including CARINA, Meteorological, Volume, and Intel. Therefore, the MSSM and Dual-SSIM models were trained using these datasets. Figure 8 The comparison between the data reconstructed by MSSM and Dual-SSIM and the original data is shown.

[0114] Figure 8 The results show that the MSSM model more closely resembles the trends and inherent structural patterns of continuously missing time-series data. Experimental results indicate that the model performs well in most cases.

[0115] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A time-series data reconstruction method based on a multi-head attention mechanism, characterized in that, The data reconstruction method uses a pre-trained bidirectional sequence-to-sequence model based on a multi-head self-attention mechanism to reconstruct time-series data sensed by an underwater wireless sensor network. This bidirectional sequence-to-sequence model includes an encoder and a decoder. In the encoder part, the sequence data on both sides of the missing part of the time series data are used as the input of the encoder; the data on both sides are respectively fed into two multi-head self-attention layers for key feature extraction; Then, the data is input into two bidirectional gated recurrent unit neural networks to capture forward and backward time patterns. In the decoder section, the output data of the dual encoders is combined with the sequence of data embedded in the decoder; a masked multi-head attention mechanism is used to combine key features learned from the two encoders to predict the input time series data; the reconstructed time series data is output; the encoder section has a residual block after the multi-head self-attention mechanism for soft thresholding.

2. The time series data reconstruction method based on multi-head attention mechanism according to claim 1, characterized in that, The specific steps for extracting key features using the multi-head self-attention layer of the decoder are as follows: To obtain the embedded data of the same encoder input data by performing a linear transformation , ; Calculate matrix and The inner product; Activation function is used to obtain The weights; pass The weight matrix and Perform the inner product to obtain the result of the self-attention mechanism; Multiple self-attention calculations are concatenated together and linearly mapped to the same dimension as the input matrix.

3. The time series data reconstruction method based on multi-head attention mechanism according to claim 1, characterized in that, The specific steps of the decoder in predicting the input time series data include: By utilizing the temporal patterns captured from the valid data on both sides of the missing data, the hidden state output by the decoder is used as the input set of the decoder; Based on the multi-head attention mechanism of the decoder, the input information is reweighted based on the attention scores to obtain the multi-head attention vector. ; Index at each time The decoder is based on a time index. The state, in The predicted value and the current time index Multi-head attention vector This is used to update the state of the recurrent gated recursive unit layer, and each gated recursive unit contains a linear layer, which generates the final prediction. : in, It is a time index The hidden state of the time decoder It is a time index Multi-head attention context vector of the time decoder It is the concatenation of the decoder's hidden state and the context vector. These are the linearization parameters.

4. The time series data reconstruction method based on multi-head attention mechanism according to claim 3, characterized in that, Multi-head attention vector The specific calculation steps are as follows: The decoder embedding data is obtained by linear transformation of the result of the masked multi-head self-attention calculation. After linear transformation with concatenated feature maps Attention weights are calculated using a multi-head attention mechanism. ; In time index t, the multi-head attention vector Described as a weighted sum of the hidden states passed by the dual encoders: Each hidden state weight The calculation is as follows: The attention layer is a multi-head self-attention layer that uses Softmax as the activation function.

5. The time series data reconstruction method based on multi-head attention mechanism according to claim 1, characterized in that, The training steps for the bidirectional sequence-to-sequence model based on the multi-head self-attention mechanism are as follows: A sliding window algorithm is used to generate a large number of training samples from the collected time series data; In the encoder part, the sequence data on both sides of the missing part are used as the input of the network model. After the multi-head self-attention mechanism and residual block calculation, the output is used as the input of the bidirectional gated recurrent unit neural network. When the output data of the dual encoders are concatenated and enter the decoder, the output data of the dual encoders is combined with the sequence of the embedded data of the decoder. A multi-head attention mechanism is used to train the input time series data by combining key features learned from the two encoders, and the reconstructed time series data is output. The output data is evaluated, and the network parameters are updated in reverse.

Images