BiLSTM auto-encoder model and method for data anomaly detection and correction

Abstract

The invention relates to the technical field of ocean information, and discloses a BiLSTM auto-encoder model and method for data anomaly detection and correction, and the model comprises an input layer which receives input data; the BiLSTM encoder is used for encoding the input data to extract key features; the BiLSTM decoder receives the features extracted by the BiLSTM encoder, decodes the features and reconstructs input data; a BiLSTM predictor which receives the features extracted by the BiLSTM encoder and predicts to generate future data points; and the output layer is used for outputting the reconstructed data decoded by the BiLSTM decoder and the future data points predicted by the BiLSTM predictor. The abnormal data can be effectively recognized, the recognized abnormal data are reasonably corrected, and the reliability and accuracy of the data are improved.

Description

Technical Field

[0001] The present invention belongs to the field of marine information technology, and in particular relates to a BiLSTM autoencoder model and method for data anomaly detection and correction. Background Art

[0002] Buoys are important tools in the field of marine science and meteorology. They are used to collect ocean and atmospheric data to support research and applications in the fields of weather forecasting, meteorological research, marine ecology and resource management. However, these buoy data are often affected by various factors such as sensor errors, missing data, outliers and environmental interference, which seriously affects the accuracy and credibility of marine scientific research. Therefore, standardized data processing processes and quality control methods are needed to ensure the high quality of ocean buoy observation data.

[0003] At present, the quality control of ocean observation data is technically divided into automated quality control and expert quality control. Automated quality control automatically detects and corrects errors through algorithms and software tools; expert quality control relies on the rich theoretical knowledge and practical experience of oceanographic experts to further visually inspect the data and identify complex problems that are difficult to capture by automation. However, both methods have the following problems: (1) Although traditional automated quality control technologies, such as range checks, correlation checks, and statistical characteristic tests, can effectively identify obvious abnormal data, they often have difficulty capturing abnormal changes within a reasonable value range; in addition, they are limited to identifying anomalies in time series data without further correcting the data; (2) Expert quality control is costly.

[0004] In view of this, how to design a technology that can reduce the cost of relying on expert quality control and improve the efficiency and accuracy of data processing is the technical problem to be solved by the present invention. Summary of the invention

[0005] The present invention provides a BiLSTM autoencoder model and method for data anomaly detection and correction, which can effectively identify abnormal data and reasonably correct the identified abnormal data to improve the reliability and accuracy of the data.

[0006] In order to achieve the above technical objectives, the present invention adopts the following technical solutions: In one aspect, the present invention provides a BiLSTM autoencoder model for data anomaly detection and correction, comprising: An input layer, which receives input data; A BiLSTM encoder encodes the input data to extract key features; A BiLSTM decoder receives the features extracted by the BiLSTM encoder for decoding and reconstructs the input data; A BiLSTM predictor that receives the features extracted by the BiLSTM encoder and predicts and generates future data points; The output layer outputs the reconstructed data decoded by the BiLSTM decoder and the future data points predicted by the BiLSTM predictor.

[0007] As described above, the BiLSTM autoencoder model for data anomaly detection and correction, the formula for the BiLSTM encoder to encode the input data and extract key features is as follows: ; in, represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t, wherein the hidden state captures the characteristics of the input data; represents the cell state of the BiLSTM encoding layer of the BiLSTM encoder at time step t; represents the input data; represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1; represents the cell state of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1.

[0008] As described above, for the BiLSTM autoencoder model for data anomaly detection and correction, the formula for the BiLSTM decoder to decode the features extracted by the BiLSTM encoder and reconstruct the input data is as follows: ; in, Represents reconstructed data; represents the weight of the linear reconstruction layer; represents the hidden state of the BiLSTM decoder at time step t; Represents the bias parameters of the linear reconstruction layer.

[0009] As described above, the BiLSTM autoencoder model for data anomaly detection and correction, the BiLSTM predictor also includes an attention layer, which is used to assign weights to the features extracted by the BiLSTM encoder, and its formula is as follows: ; in, represents a score matrix used to quantify the importance of different time steps in a time series; represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t; represents the learnable attention weights.

[0010] As described above, the BiLSTM autoencoder model for data anomaly detection and correction, the formula for generating future data points by the BiLSTM predictor is as follows: ; in, represents predicted future data points; Represents the weight of the linear prediction layer; represents the hidden state of the BiLSTM prediction layer of the BiLSTM predictor at time step t; Represents the bias parameters of the linear prediction layer.

[0011] In another aspect, the present invention provides a method for data anomaly detection and correction based on a BiLSTM autoencoder model, comprising the following steps: S1, preprocessing the data to form raw data; S2, inputting the original data into the BiLSTM autoencoder model as described above, wherein the BiLSTM autoencoder model processes the original data and outputs reconstructed data and predicted future data points; S3, calculating a reconstruction error, identifying abnormal data in the original data according to the reconstruction error and marking the abnormal data; S4, using future data points to replace and correct the abnormal data.

[0012] In the method for data anomaly detection and correction as described above, the step of preprocessing the data includes: S11, duplicates in merged data; S12, handling missing data; S13, using the Pearson correlation coefficient method to draw a heat map to select the data after step S12, select the data whose absolute value of the Pearson correlation coefficient is greater than or equal to the set value, and remove irrelevant and redundant data; S14: Standardize the data after step S13 to form original data.

[0013] In the method for data anomaly detection and correction as described above, the step of calculating the reconstruction error includes: The reconstructed data is denormalized, the original data and the reconstructed data are compared, and a reconstruction error is calculated.

[0014] In the method for data anomaly detection and correction as described above, the steps of identifying and marking abnormal data in the original data according to the reconstruction error include: The quality symbol of the data point whose reconstruction error is greater than the set threshold is marked as abnormal; The quality symbol with reconstruction error less than or equal to the set threshold is marked as normal.

[0015] In the method for data anomaly detection and correction as described above, the calculation formula of the threshold is as follows: ; in, Indicates setting threshold; represents the mean of the reconstruction error; Threshold value indicating the strictness of the threshold; represents the standard deviation of the reconstruction error.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are: (1) A BiLSTM autoencoder model for data anomaly detection and correction is constructed. The input data is reconstructed through the encoder and decoder. During the reconstruction process, normal data will generate a small reconstruction error, while abnormal data will cause a significant reconstruction error. Therefore, the reconstruction error can be used to identify outliers in the data. This model also predicts the future data points of the input data through the encoder and predictor. The abnormal data can be replaced with future data points to achieve reasonable correction of the abnormal data. (2) A data anomaly detection and correction method based on the BiLSTM autoencoder model is provided, which can effectively improve the reliability and accuracy of the data, solve the limitations of traditional quality control technology in dealing with weak anomalies, and provide an effective data processing method for scientific research. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 A schematic diagram of the structure of an embodiment of a BiLSTM autoencoder model for data anomaly detection and correction according to the present invention; Figure 2 for Figure 1 Detailed schematic diagram of Figure 3 A schematic diagram of the internal structure of a single LSTM neural unit in an embodiment of a BiLSTM autoencoder model for data anomaly detection and correction of the present invention; Figure 4 A BiLSTM neural network layer structure diagram in an embodiment of a BiLSTM autoencoder model for data anomaly detection and correction of the present invention; Figure 5 The figure is a flow chart of the method for data anomaly detection and correction of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that in the description of the present invention, the terms "upper" and "lower" and other terms indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings, which are only for the convenience of description, and do not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0021] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0022] The disclosure below provides many different embodiments or examples to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplicity and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the present invention provides various specific examples of processes and materials, but those of ordinary skill in the art can be aware of the application of other processes and / or the use of other materials.

[0023] like Figures 1 to 4As shown, the present application provides a BiLSTM autoencoder model for data anomaly detection and correction. The core neural unit of the BiLSTM autoencoder model is BiLSTM (bidirectional long short-term memory network). BiLSTM consists of a forward LSTM (long short-term memory artificial neural network) and a reverse LSTM. LSTM is an improved recurrent neural network (RNN). The single cell layer structure of LSTM includes three structures: forget gate, input gate and output gate. Figure 3 shown; and Represent the cell states of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1 and time step t, respectively. and represent the hidden states of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1 and time step t, respectively. is the input data at the current moment.

[0024] Forget Gate An activation function sigmoid is used to calculate how much past information is retained in the memory cell; the input gate An activation function sigmoid is used to calculate how much new information enters the memory unit and the updated value of the cell state By inputting data and the hidden state at the previous moment Perform matrix weighted summation and then pass the tanh activation function to get a new value; output gate Determines how much information to read from the memory cell by and input data Perform weighted summation and then pass the sigmoid activation function to get the output The BiLSTM autoencoder connects two LSTM networks, one training sample data from time step t to t+1 in forward time order, and the other training sample data from time step t+1 to t in reverse time order. The BiLSTM neural network layer structure is as follows: Figure 4 shown.

[0025] By combining two directional LSTM networks, BiLSTM utilizes both past and future information when processing time series data, thereby improving the performance and accuracy of the model.

[0026] The BiLSTM autoencoder model consists of an input layer, a BiLSTM encoder, a BiLSTM decoder, a BiLSTM predictor, and an output layer. Each part is described in detail below.

[0027] The input layer receives data. The input layer is a linear layer 'encoder_input' (linear). The linear layer performs a high-dimensional transformation on the input data, expanding the feature dimension of the input data from 'm' to 'm*16'. After the linear layer, the shape of the input data changes from (b, s, m) to (b, s, m*16). The formula for performing linear transformation in the linear layer is as follows: ; in, Represents an output vector of shape (b, s, m*16); represents the weight of the linear layer; Represents the original input data, that is, the input data with shape (b, s, m); Represents the bias parameter of the linear layer.

[0028] The output of the linear layer then passes through the ReLU activation function to introduce nonlinear characteristics. The ReLU activation function sets all negative values ​​to zero. The formula is as follows: ; in, express The output of the vector after the ReLU activation function, that is, the input data of the BiLSTM encoder after the output layer performs high-dimensional transformation on the input data.

[0029] The BiLSTM encoder can encode the input data to extract key features. In this application, the BiLSTM encoder includes three layers of BiLSTM encoding layers ('encoder1', 'encoder2', 'encoder3') arranged in sequence, and the dropout regularization technology is used between the BiLSTM encoding layers to randomly discard some network nodes to prevent overfitting and improve the generalization ability of the model.

[0030] These three BiLSTM encoding layers can gradually reduce the feature dimension of the input data until the final encoding dimension is e ('encoding_dim'), where 'encoding_dim' is a custom parameter representing the feature dimension after encoding; the data shapes of these three BiLSTM encoding layers change in sequence: (b, s, m*16) -> (b, s, e*4) -> (b, s, e*2) -> (b, s, e), and the three BiLSTM encoding layers can convert the input data into a set of hidden states that capture the key features of the input data.

[0031] The formula for the BiLSTM encoding layer to encode the input data and extract key features is as follows: ; in, represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t. The hidden state captures the characteristics of the input data; represents the cell state of the BiLSTM encoding layer of the BiLSTM encoder at time step t; Represents input data; represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1; represents the cell state of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1.

[0032] The BiLSTM decoder receives the features extracted by the BiLSTM encoder and decodes and reconstructs the input data. The BiLSTM decoder in this application includes three layers of BiLSTM decoding layers ('decode1', 'decode2', 'decode3') arranged in sequence. The BiLSTM decoding layer is used to gradually increase the dimension of the encoded data and finally restore it to 'm*16'. The data shape of the three layers of BiLSTM decoding layers changes (b, s, e)->(b, s, e*2)->(b, s, e*4)->(b, s, m *16).

[0033] The formula for the BiLSTM decoding layer to decode the features extracted by the encoder is as follows: ; in, represents the hidden state of the BiLSTM decoding layer at time step t; represents the cell state of the BiLSTM decoding layer at time step t; represents the hidden state of the BiLSTM encoding layer at time step t; represents the hidden state of the BiLSTM decoding layer at time step t-1; Represents the cell state of the BiLSTM decoding layer at time step t-1.

[0034] At this time, the data dimension is 'm*16', and the dimension of the original data received by the input layer is 'm', so the decoded data enters a linear reconstruction layer 'Reconstructed_output' (linear) to reduce the dimension, such as Figure 2 As shown, reconstructed data is generated, and the shape of the reconstructed data changes to (b, s, m), and its formula is as follows: ; in, Represents the reconstructed data output by the linear reconstruction layer; represents the weight of the linear reconstruction layer; represents the hidden state of the BiLSTM decoding layer at time step t; Represents the bias parameters of the linear reconstruction layer.

[0035] The BiLSTM predictor receives the features extracted by the BiLSTM encoder and predicts and generates future data points. The BiLSTM predictor in this application introduces an attention mechanism, that is, the BiLSTM predictor includes an attention layer. The attention mechanism is used to weight the output of the BiLSTM encoding layer and assign weights to the features extracted by the BiLSTM encoder, so that the model can focus on the most important part of the extracted features. The formula is as follows: ; in, represents a score matrix used to quantify the importance of different time steps in a time series; represents the hidden state of the last BiLSTM encoding layer at time step t; represents the learnable attention weights.

[0036] The output of the attention layer is the hidden state after the input data is encoded, and its shape is (b, e), where b is the batch size and e is the encoding dimension. In order to input the output of the attention layer into the subsequent BiLSTM layer of the BiLSTM predictor, the output of the attention layer needs to be dimensionally expanded. Through dimension expansion, the data shape of the attention layer output is expanded from (b, e) to (b, s, e), where s is the time step.

[0037] The attention layer calculates the importance score of each time step of the encoded output and produces a weighted context vector that contains the key new aggregation of the features of the entire input data.

[0038] Softmax normalization formula: ; Indicates the attention weight obtained by normalizing the score by softmax; Represents a score matrix used to quantify the importance of different time steps in a time series.

[0039] The context vector is obtained by weighted summing the input vector using the attention weights: ; in Represents the weighted context vector of all time steps; represents the attention weight at time step t; represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t.

[0040] This part is used to calculate the score of the attention mechanism, which generates a weighted context vector by calculating the importance score of each time step of the encoding output. By using the attention mechanism, the model can better capture the important features in the time series, thereby improving the prediction accuracy. Figure 2 Attention mechanism layers shown.

[0041] The obtained context vector is then used to predict future data. The context vector is processed through the BiLSTM layer in the BiLSTM predictor to integrate the information of the data points arranged in chronological order. The formula is as follows: ; in, represents the hidden state of the BiLSTM prediction layer of the BiLSTM predictor at time step t; represents the cell state of the BiLSTM prediction layer of the BiLSTM predictor at time step t; represents the context vector; represents the hidden state of the BiLSTM prediction layer of the BiLSTM predictor at time step t-1; represents the cell state of the BiLSTM prediction layer of the BiLSTM predictor at time step t-1.

[0042] Finally, the predicted future data points are output through a linear prediction layer (Prddicted_output), whose formula is as follows: ; in, represents predicted future data points; Represents the weight of the linear prediction layer; represents the hidden state of the BiLSTM prediction layer of the BiLSTM predictor at time step t; Represents the bias parameters of the linear prediction layer.

[0043] The output layer outputs the reconstructed data decoded by the BiLSTM decoder and the future data points predicted by the BiLSTM predictor.

[0044] After the BiLSTM autoencoder model is built, it needs to be trained, optimized and tested.

[0045] The steps for training and optimizing the BiLSTM autoencoder model are as follows: First of all, before training and optimizing the BiLSTM autoencoder model, you need to set hyperparameters such as learning rate (lr), batch size (batch_size), number of iterations (epochs), prediction step size (future_step), and regularization (Dropout) number.

[0046] Secondly, initialize the parameters: m is the number of features of the input data, encoding_dim is the encoding dimension. Use mean square error (MSELoss) as the loss function, Adam as the optimizer, and StepLR as the learning rate scheduler.

[0047] Then, the model is trained in a loop: During the training process, the model performs multiple iterations (epochs), and each iteration includes the following steps. Batch processing, the data is divided into multiple batches, each batch size is b, and each batch is processed in a loop; model output, the model input shape is x (b, s, m) feature data, the model obtains the reconstructed output sequence and the predicted output .

[0048] The next steps include loss calculation, adjusting learning rate, and evaluating the results.

[0049] Then, the loss is calculated: the loss calculation includes the calculation of reconstruction loss and prediction loss, and the total loss is the sum of the two. The reconstruction loss is the mean square error between the model output and its corresponding target. The prediction loss is the mean square error between the predicted output and the predicted target. The total loss calculation formula is as follows: ; in, represents the reconstruction loss; represents the predicted loss; represents the total loss; Represents the original input data; represents the predicted value; Represents the reconstructed data output by the linear reconstruction layer; Represents a predicted future data point.

[0050] Next, perform gradient update: In each iteration, after calculating the loss, update the model parameters through the back propagation algorithm. Use the Adam optimizer to handle the optimization problem of data and parameters. The update formula is: ; in, represents the parameters of the model at time step t; Indicates the step size of each parameter update; represents the bias correction for the first moment estimate of the gradient; represents the bias correction for the second moment estimate of the gradient; Represents a small constant to prevent the denominator from being zero.

[0051] Next, adjust the learning rate: At the end of each training cycle (epoch), adjust the learning rate according to the preset strategy. Use the StepLR method to adjust the learning rate lr. Every certain training cycle, the learning rate is multiplied by a factor. StepLR is a learning rate scheduler, which is used in PyTorch (open source Python machine learning library) to control the adjustment of the learning rate after each training cycle (epoch). The formula is: ; in, represents the initial learning rate; Represents the factor by which the learning rate is multiplied each time; Indicates the current training cycle number; Indicates the cycle of adjusting the learning rate; represents the adjusted learning rate.

[0052] Finally, we get the evaluation results: various losses recorded during the training process are returned to facilitate performance evaluation and further analysis. By observing the trend of the loss, we can determine whether the model has converged and whether the hyperparameters need to be adjusted.

[0053] Save the state dictionary of the trained and optimized model to a file. The state dictionary contains all the parameters of the model. Save the model as a file in the form of model.pth.

[0054] The steps to test the BiLSTM autoencoder model are as follows: The model is called to reconstruct the data in the test set, determine and mark abnormal data points. Before reconstructing each data window, the quality control column and iteration window need to be initialized. The quality control column is initialized by adding a new quality column 'quality_control' to the test set, and the quality of the initial data points is normal; the iteration window is to traverse the test set by sliding the window and standardize the data in the window. The window size is specified by the parameter window_size, and the window moving step is specified by the parameter step_size.

[0055] The identified outliers are then replaced with future data points predicted by the model.

[0056] The test set after anomaly detection and correction is saved to a file for subsequent scientific analysis and application. Anomaly identification and correction of data ensures the quality and consistency of the data and reduces the impact of abnormal data on subsequent analysis.

[0057] like Figure 5 As shown, the present application also provides a data anomaly detection and correction method based on a BiLSTM autoencoder model, comprising the following steps: S1, preprocess the data to form raw data.

[0058] The data in this application is ocean buoy observation data, which includes multiple elements such as timestamp, seawater flow rate, flow direction, temperature, dissolved oxygen, salinity, air pressure, humidity and conductivity. Since the elements of ocean buoy observation data are commonly used existing knowledge in this field, they are not listed here one by one. The preprocessing includes the following steps: S11, merge duplicates in data: Use statistical methods that retain the mean value to merge duplicates of buoy observation data to ensure that the buoy observation data at each observation time is unique.

[0059] S12, processing missing data: use weighted moving average to fill missing values ​​to ensure the integrity of buoy observation data.

[0060] S13, use the Pearson correlation coefficient method to draw a heat map to select the data after step S12, select the data whose absolute value of the Pearson correlation coefficient is greater than or equal to the set value, and remove irrelevant and redundant data. Specifically: in this application, when the absolute value of the Pearson correlation coefficient of two buoy observation elements is greater than or equal to 0.7, it is determined that the two have a strong correlation; when the absolute value of the Pearson correlation coefficient of the two buoy observation elements is between 0.3 and 0.7, it is determined that the two are weakly correlated; when the absolute value of the Pearson correlation coefficient of the two buoy observation elements is less than 0.3, it is determined that the two are irrelevant. Through the Pearson correlation coefficient, select the elements with higher correlation to form a data set with the shape of X (n, m), where n is the total number of data points in the data set, and m is the characteristic dimension of the data, and irrelevant and redundant data are removed at the same time.

[0061] S14: Perform Z-standardization on the data set to form the original data: For each feature in the data set, calculate its mean and standard deviation. The formula for calculating the mean is as follows: ; in, represents the mean; Represents the total number of data points in the data set; Represents a single data point for the i-th feature.

[0062] ; in, represents standard deviation; represents the mean; Represents the total number of data points in the data set; Represents a single data point for the i-th feature.

[0063] After calculating the mean and standard deviation of each feature, use the standardization formula to convert each data point into a standardized value, that is, a value that can be processed based on the BiLSTM autoencoder model. The standardization formula is as follows: ; in, represents a standardized value; Represents the eigenvalues ​​in the original data; Represents standard deviation.

[0064] Before training and optimizing the BiLSTM autoencoder model, the training set and test set are preprocessed using the same method as above. The training set contains the first 80% of the time series (X_train(n_train,m)), and the test set contains the remaining 20% ​​(X_test(n_test,m)).

[0065] The standardized training set (X_train(n_train,m)) is segmented into small batches to meet the computational requirements of the BiLSTM encoding layer for small batch gradient descent during training. The small batches of data segmented in this process are used for both data reconstruction and data prediction. First, the training set data X_train(n_train,m) is segmented by sliding windows to divide the data into several data sequences of the shape (n_train-s+1,s,m). Then, these sequences are further segmented into small batches, each of which has a shape of x (b, s, m) to ensure that a small batch of data of a reasonable size is used in each training, so as to achieve effective gradient descent. Among them, n_train is the total number of training samples, m is the feature dimension of the data, b is the size of each batch (that is, the number of sequences contained in each batch, batch_size), and s is the time step (that is, the size of the sliding window, window_size).

[0066] S2, inputs the preprocessed raw data into the BiLSTM autoencoder model, reconstructs the raw data based on the BiLSTM autoencoder model and predicts future data points, and then outputs the reconstructed raw data and predicted future data points.

[0067] S3, calculate the reconstruction error, identify the abnormal data in the original data according to the reconstruction error and mark it; The reconstruction error is calculated by comparing the original data with the reconstructed data. The calculation formula is as follows: ; in, represents the reconstruction error; Represents the i-th sample of the reconstructed data; Represents the i-th sample of the original data.

[0068] For each data point, if its reconstruction error exceeds the set threshold, the quality symbol of the data point is marked as abnormal. If the reconstruction error is less than or equal to the set threshold, the quality symbol of the data point is marked as normal. The corresponding marking operations are as follows: ; in, Indicates the quality symbol column, which is used to indicate whether the data point is abnormal; represents the reconstruction error of the i-th data point; Indicates setting a threshold.

[0069] The threshold calculation formula is as follows: ; in, Indicates setting threshold; represents the mean of the reconstruction error; Threshold value indicating the strictness of the threshold; represents the standard deviation of the reconstruction error.

[0070] S4, use future data points to replace and correct abnormal data.

[0071] The original data after anomaly detection is traversed through the sliding window method. In each window, if there are data points marked as abnormal, the original data in the window are processed and standardized. After the model predicts the original data in the window to obtain future data points, the original data in the window is de-standardized and the corrected values, i.e., future data points, are used to replace the abnormal data points in the window.

[0072] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present invention.

Claims

1. A BiLSTM autoencoder model for data anomaly detection and correction, characterized in that: include: An input layer, which receives input data; A BiLSTM encoder encodes the input data to extract key features; A BiLSTM decoder receives the features extracted by the BiLSTM encoder for decoding and reconstructs the input data; A BiLSTM predictor that receives the features extracted by the BiLSTM encoder and predicts and generates future data points; The output layer outputs the reconstructed data decoded by the BiLSTM decoder and the future data points predicted by the BiLSTM predictor.

2. The BiLSTM autoencoder model for data anomaly detection and correction according to claim 1, characterized in that: The formula for the BiLSTM encoder to encode the input data and extract key features is as follows: ； in, represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t, wherein the hidden state captures the characteristics of the input data; represents the cell state of the BiLSTM encoding layer of the BiLSTM encoder at time step t; represents the input data; represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1; represents the cell state of the BiLSTM encoding layer of the BiLSTM encoder at time step t-1.

3. The BiLSTM autoencoder model for data anomaly detection and correction according to claim 2, characterized in that: The formula for the BiLSTM decoder to decode the features extracted by the BiLSTM encoder and reconstruct the input data is as follows: ； in, Represents reconstructed data; represents the weight of the linear reconstruction layer; represents the hidden state of the BiLSTM decoder at time step t; Represents the bias parameters of the linear reconstruction layer.

4. The BiLSTM autoencoder model for data anomaly detection and correction according to claim 2, characterized in that: The BiLSTM predictor also includes an attention layer, which is used to assign weights to the features extracted by the BiLSTM encoder, and its formula is as follows: ； in, represents a score matrix used to quantify the importance of different time steps in a time series; represents the hidden state of the BiLSTM encoding layer of the BiLSTM encoder at time step t; represents the learnable attention weights.

5. The BiLSTM autoencoder model for data anomaly detection and correction according to claim 4, characterized in that: The formula for generating future data points by the BiLSTM predictor is as follows: ； in, represents predicted future data points; Represents the weight of the linear prediction layer; represents the hidden state of the BiLSTM prediction layer of the BiLSTM predictor at time step t; Represents the bias parameters of the linear prediction layer.

6. A method for data anomaly detection and correction, characterized in that: The following steps are involved: S1, preprocessing the data to form raw data; S2, inputting the original data into the BiLSTM autoencoder model according to any one of claims 1 to 5, wherein the BiLSTM autoencoder model processes the original data and outputs reconstructed data and predicted future data points; S3, calculating a reconstruction error, identifying abnormal data in the original data according to the reconstruction error and marking the abnormal data; S4, using future data points to replace and correct the abnormal data.

7. The method for data anomaly detection and correction according to claim 6, characterized in that: The step of preprocessing the data comprises: S11, duplicates in merged data; S12, handling missing data; S13, using the Pearson correlation coefficient method to draw a heat map to select the data after step S12, select the data whose absolute value of the Pearson correlation coefficient is greater than or equal to the set value, and remove irrelevant and redundant data; S14: Standardize the data after step S13 to form original data.

8. The method for data anomaly detection and correction according to claim 6, characterized in that: The steps to calculate the reconstruction error include: The reconstructed data is denormalized, the original data and the reconstructed data are compared, and a reconstruction error is calculated.

9. The method for data anomaly detection and correction according to claim 8, characterized in that: The step of identifying abnormal data in the original data according to the reconstruction error and marking the abnormal data comprises: The quality symbol of the data point whose reconstruction error is greater than the set threshold is marked as abnormal; The quality symbol with reconstruction error less than or equal to the set threshold is marked as normal.

10. The method for data anomaly detection and correction according to claim 9, characterized in that: The calculation formula of the threshold is as follows: ； in, Indicates setting threshold; represents the mean of the reconstruction error; Threshold value indicating the strictness of the threshold; represents the standard deviation of the reconstruction error.

Images