A method for processing environmental time series data based on Transformer model

Abstract

The invention discloses a method for processing environmental time series data based on a Transformer model, which belongs to the field of data processing and includes steps S1: collecting historical environmental time series data from environmental monitoring sites; S2: model selection and training; S3: forming a model group with models trained with different window lengths T; using the models in the model group to predict environmental time series data respectively, and taking the average value of the prediction results. The processing method can complete the missing data of environmental time series data and identify abnormal values ​​in environmental time series data. Compared with traditional methods, it no longer relies on a specific embedding network and has higher accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental monitoring data processing, and in particular to an environmental time series data processing method based on a Transformer model. Background Art

[0002] Environmental monitoring data quality control plays a fundamental role in the statistical analysis and information mining of environmental monitoring data. The quality problems of monitoring data mainly include data anomalies and data missing. Traditional methods mainly include data anomaly identification based on threshold and outlier monitoring and average value supplement quality control methods. These methods do not take into account the local fluctuations of environmental monitoring data itself, which may result in insufficient accuracy in anomaly identification and data completion.

[0003] In recent years, time series anomaly detection methods based on deep learning have been developed, and the main application architectures include LSTM, CNN, Transformer, etc. Methods based on deep learning usually identify abnormal data by training the model, and make numerical predictions for abnormal points or missing points to replace abnormal values ​​or supplement missing values. Among them, the Transformer architecture has the advantages of strong time series correlation capture capability and high accuracy. The Chinese application with publication number CN115758273A proposes a time series data detection method, device, equipment and medium. In the prior art, the data needs to be normalized preprocessed and embedded as the input of the Transformer model. Among them, the data embedding process depends on a specific embedding network, and the time series change information of the original data is extracted, which may cause information loss, resulting in insufficient data accuracy. Summary of the invention

[0004] The present invention discloses a method for processing environmental time series data based on a Transformer model, which does not rely on an embedding network and solves the problem of insufficient accuracy in anomaly recognition and data completion.

[0005] The present invention provides a method for processing environmental time series data based on a Transformer model, comprising the following steps:

[0006] S1: Collect the historical environmental monitoring data time series of environmental monitoring sites;

[0007] Assume that the time series of historical environmental monitoring data is: S={S1, S2, …, S N}; where S1, S2, ..., S N represents the environmental data at the corresponding moment; let the time interval in the historical environmental monitoring data time series S be δt;

[0008] S2: Model selection and training;

[0009] The specific steps include:

[0010] S2.1: Select the Transformer model structure;

[0011] S2.2: Select the window length T and convert the continuous data points are deleted to form the input sequence S'; the deleted P consecutive data points are constructed as the original target sequence y;

[0012] in, Indicates rounding up; T is the window length;

[0013] Because there are N environmental data in S, and the position of the first data point in the deleted consecutive P data points is uncertain, a total of N-P+1 different input sequences S' can be constructed, and a total of N-P+1 corresponding original target sequences y can be generated; the input sequence S' and the corresponding original target sequence y form a data pair S'-y; a total of N-P+1 S'-y data pairs can be constructed;

[0014] In the case of different window lengths T, there are N-P+1 S'-y data pairs; however, different values ​​of the window length T, different numbers of deleted data points P, and different numbers of S'-y data pairs N-P+1;

[0015] S2.3: For a certain window length T, the input sequence S' in a certain S'-y data pair under the window length T is input into the Transformer model; the Transformer model predicts the missing data in the input sequence S' in the S'-y data pair, and the prediction result is recorded as the predicted target sequence ;

[0016] S2.4: For the S'-y data pair in S2.3, establish the model training optimization target and iteratively optimize the Transformer model parameters to form the final Transformer model;

[0017] The final Transformer models composed of different window lengths T are combined into a model group in the form of a set, and the model group is recorded as M0;

[0018] S3: Use model group M0 to treat the sequence S0={S 01 ,S 02 ,……,S 0k ,……,S 0N} to make predictions; k∈[1,N], k is a positive integer;

[0019] The specific steps include:

[0020] S3.1: Delete the first original data S in S0 01 ;

[0021] S3.2: Use the three models in model group M0 to analyze the first missing raw data S in S0. 01 Make a prediction and take the average of the three prediction results of model group M0 to replace the first missing original data S in S0 01 ; Record the first missing original data S of the three models 01 The average value of the prediction results is S 01，平 ; Generate prediction sequence S 预，1 ={S 01，平 , S 02 , ..., S 0k , ..., S 0N};

[0022] S3.3: Use the same method as steps S3.1 and S3.2 to continue deleting S 预，k The k+1th original data S 0(k+1) ;

[0023] S3.4: Use the three models in model group M0 to analyze S 预，k The missing k+1th original data S 0(k+1) Make a prediction and take the average of the three prediction results of model group M0 instead of S 预，k The missing k+1th original data S 0(k+1) , remember the three models for the missing k+1th original data S 0(k+1) The average value of the prediction results is S 0(k+1)，平 ; Generate prediction sequence S 预，k+1 ={S 01，平 , ..., S 0(k+1)，平 , S 0(k+2) , ..., S 0N};

[0024] S3.5: Repeat S3.3-S3.4, from K to N-1, until S 预，N-1 Medium S 0N is deleted, generating the prediction sequence S 预，N , then, S 预，N ={S 01,平 , S 02，平 , ..., S 0N，平}; sequence S 预，N is the prediction result of sequence S0.

[0025] Further, in step S2.4, the predicted target sequence of the input sequence S' in the S'-y data pair is The mean square error e of the original target sequence y in the S'-y data pair is used as the model training optimization target, and the formula is:

[0026]

[0027] Among them, y i Represents the i-th data in the original target sequence y; Represents the predicted target sequence The i-th data in ; P is the number of deleted data points; the Transformer model parameters are iteratively updated through gradient back propagation, and it is gradually iterated until e no longer continues to decrease, then the Transformer model converges, forming the final Transformer model under the window length T;

[0028] Replace the window length T in S2.2, repeat S2.2-S2.4, and form the final Transformer model under other window lengths T;

[0029] The final Transformer model is denoted as Mx, where x represents the time length of the window length T.

[0030] Furthermore, the window length T is 1 hour, 6 hours or 12 hours, and the time interval δt is 1 hour.

[0031] Furthermore, the Transformer model structure selects the Encoder-Decoder architecture.

[0032] Beneficial effects:

[0033] This paper proposes a Transformer-based data quality control method that does not require data embedding and feature extraction. By taking long time series as input and using the Transformer's attention mechanism to learn the continuous change rules of the original time series data, the integrated learning of multiple Transformer models is realized by constructing different time window models, further improving accuracy and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the method;

[0035] Figure 2 This is a comparison chart of the prediction results of the real environment time series by this method and the actual true value;

[0036] Figure 3 This is the result of this method completing the environmental time series of missing data;

[0037] Figure 4 This is a comparison chart of COD data from water quality monitoring at Chehan Bridge in Gaoyou City. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1: See Figure 1 , a method for processing environmental time series data based on a Transformer model, comprising the following steps:

[0040] S1: Collect the time series of historical environmental monitoring data from environmental monitoring sites, and assume that the time interval in the time series of historical environmental monitoring data is δt.

[0041] For example, in this embodiment, hourly ozone concentration data of a certain environmental monitoring station is collected for three years, and the time series of the historical environmental monitoring data is: S={S1, S2, ..., S N}; where S1, S2, ..., S N Indicates the original data of ozone concentration at the corresponding moment. In the historical environmental monitoring data time series, the time interval δt between two adjacent original data is 1 hour, that is, the time interval δt in the historical environmental monitoring data time series is 1 hour.

[0042] S2: Model selection and training.

[0043] The specific steps include:

[0044] S2.1: Choose an appropriate Transformer model structure.

[0045] In this embodiment, the Encoder-Decoder architecture is selected.

[0046] S2.2: Select the window length T and convert the continuous data points are deleted to form the input sequence S'; the deleted P consecutive data points are constructed as the original target sequence y.

[0047] in, Indicates rounding up. T is the set window length. In this embodiment, the window length T can be selected as 1 hour, 6 hours or 12 hours. When T=1 hour, P=1; when T=6 hours, P=6; when T=12 hours, P=12.

[0048] Example:

[0049] Assume that P data points are deleted from S starting from the first original data, then S'={S P+1 , S P+2 , ..., S N}; y = {S1, S2, ..., S P};

[0050] Let’s re-examine the example:

[0051] Assume that P data points are deleted from S starting from the second original data, then S'={S1, S P+2 , ..., S N}; y = {S2, ..., S P+1};

[0052] Let’s repeat the example again:

[0053] Assume that P data points are deleted from S starting from the third original data point, then S'={S1, S2, S P+3 , ..., S N}; y = {S3, ..., S P+2};

[0054] And so on…

[0055] For a certain window length T, because there are N original data in S and the position of the first data point among the deleted consecutive P data points is uncertain, as in the above example, at most N-P+1 different sequences S' can be constructed, and there are N-P+1 corresponding original target sequences y. Sequence S' and the corresponding original target sequence y are combined into data pairs, recorded as S'-y data pairs; a total of N-P+1 S'-y data pairs can be constructed.

[0056] In the case of other window lengths T, a total of N-P+1 S'-y data pairs can still be constructed. However, because the window length T is different, the number of deleted data points P is different, so the number of S'-y data pairs N-P+1 finally formed is also different.

[0057] S2.3: For a certain window length T, the input sequence S' in a certain S'-y data pair under the window length T is input into the Transformer model. The Transformer model predicts the missing data in the input sequence S' in the S'-y data pair, and the prediction result is recorded as the predicted target sequence ;

[0058] S2.4: For the S'-y data pair in S2.3, establish the model training optimization goal and iteratively optimize the Transformer model parameters.

[0059] The predicted target sequence of the input sequence S' in the S'-y data pair The mean square error e of the original target sequence y in the S'-y data pair is used as the model training optimization target, and the formula is:

[0060]

[0061] Among them, y i Represents the i-th data in the original target sequence y; Represents the predicted target sequence The i-th data in ; P is the number of deleted data points.

[0062] Through gradient back propagation, the Transformer multi-head attention parameters, feedforward neural network, position encoding, layer normalization, output layer and other model parameters are iterated and gradually iterated until e no longer decreases. The Transformer model converges and forms the final Transformer model under the window length T.

[0063] Replace the window length T in S2.2, repeat S2.2-S2.4, and form the final Transformer model under other window lengths T; the final Transformer model is denoted as Mx, where x represents the time length of the window length T;

[0064] In this embodiment, because the time window lengths T are 1 hour, 6 hours, and 12 hours respectively, three final Transformer models M1, M6, and M7 will be generated. 12 , the three models are given in the form of a model group set, denoted as model group M0={M1, M6, M 12}.

[0065] S3: Use model group M o The sequence to be processed S0={S 01 ,S 02 ,……,S 0k ,……,S 0N} to make predictions; k∈[1,N], k is a positive integer;

[0066] The specific steps include:

[0067] S3.1: Delete the first original data S in S0 01 ;

[0068] S3.2: Use the three models in model group M0 to analyze the first missing raw data S in S0. 01 Make a prediction and take the average of the three prediction results of model group M0 to replace the first missing original data S in S0 01 ; Record the first missing original data S of the three models 01The average value of the prediction results is S 01，平 ; Generate prediction sequence S 预，1 ={S 01，平 , S 02 , ..., S 0k , ..., S 0N};

[0069] S3.3: Use the same method as steps S3.1 and S3.2 to continue deleting S 预，k The k+1th original data S 0(k+1) ;

[0070] S3.4: Use the three models in model group M0 to analyze S 预，k The missing k+1th original data S 0(k+1) Make a prediction and take the average of the three prediction results of model group M0 instead of S 预，k The missing k+1th original data S 0(k+1) , remember the three models for the missing k+1th original data S 0(k+1) The average value of the prediction results is S 0(k+1)，平 ; Generate prediction sequence S 预，k+1 ={S 01，平 , ..., S 0(k+1)，平 , S 0(k+2) , ..., S 0N};

[0071] S3.5: Repeat S3.3-S3.4, from K to N-1, until S 预，N-1 S 0N is deleted, generating the prediction sequence S 预，N , then, S 预，N ={S 01,平 , S 02，平 , ..., S 0N，平}; sequence S 预，N is the prediction result of sequence S0.

[0072] If the i-th original data S in the sequence S0 to be processed 0i If it is missing originally, there is no need to delete the i-th original data S during the processing. 0i , directly through each model in the model group M0 for the i-th original data S 0i To make a prediction, take the average of the prediction results of each model in the model group M0 to replace the missing i-th original data S in the sequence S0 to be processed 0i , which solves the problem of missing original data of sequence S0 to be processed.

[0073] Suppose a certain original data S in the sequence S0 to be processed 0dThe average prediction result of each model in model group M0 is S 0d，平 , if the average value of the predicted results is S 0d，平 With the original data S 0d If the deviation exceeds 200%, the original data S 0d is abnormal data.

[0074] Example 2: The final Transformer models M1, M6, M7 with different time windows are respectively 12 Anomaly identification and analysis of a piece of actual monitoring data. Figure 2 The black solid line represents the original data, and the blue, purple, and yellow lines represent M1, M6, and M 12 The orange solid line represents the average value predicted by the three models. The original data in the box indicated by the green dotted line does not deviate from the predicted average value by more than the threshold, and is judged as normal data; the original data in the box indicated by the red dotted line deviates from the predicted average value by more than the threshold, and is judged as abnormal data, which means data error.

[0075] Figure 3 In the figure, the black line is the missing sequence predicted by the model group M0, which is very consistent with the monitoring truth value; note that the monitoring truth value of the blue solid line completely overlaps with the black prediction sequence, so it cannot be seen in color. Delete a section in the middle of the monitoring truth value and let the average value predicted by the model group M0 complete it. The completion result, that is, the predicted mean of the orange dotted line, is also almost consistent with the originally deleted monitoring truth value (the deleted monitoring truth value is represented by the blue dotted line), indicating that the model is highly reliable. Figure 3 In the figure, the blue dotted line indicates the deleted monitoring truth value. Figure 2 and Figure 3 The horizontal axis represents time, unit: hour.

[0076] See Figure 4 In addition, taking the COD data of Chehan Bridge water quality monitoring in Gaoyou City as an example, 8548 points of manually reviewed continuous monitoring data were selected as the test set, and 93 points were randomly selected as abnormal points for deletion. The relative deviation of the traditional method mean substitution method was 50.2%, the relative deviation of the linear fitting method was 14.6%, and the relative deviation of the prediction method of this scheme was 9.1%, which also shows that the accuracy of the model of this scheme is high.

[0077] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A method for processing environmental time series data based on a Transformer model, characterized in that: The following steps are involved: S1: Collect the time series of historical environmental monitoring data from environmental monitoring sites; Assume that the time series of historical environmental monitoring data is: S={S1, S2, …, S N }; where S1, S2, ..., S N represents the environmental data at the corresponding moment; let the time interval in the historical environmental monitoring data time series S be δt; S2: Model selection and training; The specific steps include: S2.1: Select the Transformer model structure; S2.2: Select the window length T and convert the continuous data points are deleted to form the input sequence S'; the deleted P consecutive data points are constructed as the original target sequence y; in, Indicates rounding up; T is the window length; Because there are N environmental data in S, and the position of the first data point in the deleted consecutive P data points is uncertain, a total of N-P+1 different input sequences S' can be constructed, and a total of N-P+1 corresponding original target sequences y can be generated; the input sequence S' and the corresponding original target sequence y form a data pair S'-y; a total of N-P+1 S'-y data pairs can be constructed; In the case of different window lengths T, there are N-P+1 S'-y data pairs; however, different values ​​of the window length T, different numbers of deleted data points P, and different numbers of S'-y data pairs N-P+1; S2.3: For a certain window length T, the input sequence S' in a certain S'-y data pair under the window length T is input into the Transformer model; the Transformer model predicts the missing data in the input sequence S' in the S'-y data pair, and the prediction result is recorded as the predicted target sequence ; S2.4: For the S'-y data pair in S2.3, establish the model training optimization target and iteratively optimize the Transformer model parameters to form the final Transformer model; The final Transformer models composed of different window lengths T are combined into a model group in the form of a set, and the model group is recorded as M0; S3: Use model group M o The sequence to be processed S0={S 01 ,S 02 ,……,S 0k ,……,S 0N } to make predictions; k∈[1,N], k is a positive integer; The specific steps include: S3.1: Delete the first original data S in S0 01 ; S3.2: Use the three models in model group M0 to analyze the first missing raw data S in S0. 01 Make a prediction and take the average of the three prediction results of model group M0 to replace the first missing original data S in S0 01 ; Record the first missing original data S of the three models 01 The average value of the prediction results is S 01，平 ; Generate prediction sequence S 预，1 ={S 01，平 , S 02 , ..., S 0k , ..., S 0N }; S3.3: Use the same method as steps S3.1 and S3.2 to continue deleting S 预，k The k+1th original data S 0(k+1) ; S3.4: Use the three models in model group M0 to analyze S 预，k The missing k+1th original data S 0(k+1) Make a prediction and take the average of the three prediction results of model group M0 instead of S 预，k The missing k+1th original data S 0(k+1) , remember the three models for the missing k+1th original data S 0(k+1) The average value of the prediction results is S 0(k+1)，平 ; Generate prediction sequence S 预，k+1 ={S 01，平 , ..., S 0(k+1)，平 , S 0(k+2) , ..., S 0N }; S3.5: Repeat S3.3-S3.4, from K to N-1, until S 预，N-1 S 0N is deleted, generating the prediction sequence S 预，N , then, S 预，N ={S 01,平 , S 02，平 , ..., S 0N，平 }; sequence S 预，N is the prediction result of sequence S0.

2. According to the method for processing environmental time series data based on the Transformer model in claim 1, it is characterized in that: In step S2.4, the predicted target sequence of the input sequence S' in the S'-y data pair is The mean square error e of the original target sequence y in the S'-y data pair is used as the model training optimization target, and the formula is: ； Among them, y i Represents the i-th data in the original target sequence y; Represents the predicted target sequence The i-th data in ; P is the number of deleted data points; the Transformer model parameters are iteratively updated through gradient back propagation, and it is gradually iterated until e no longer continues to decrease, then the Transformer model converges, forming the final Transformer model under the window length T; Replace the window length T in S2.2, repeat S2.2-S2.4, and form the final Transformer model under other window lengths T; The final Transformer model is denoted as Mx, where x represents the time length of the window length T.

3. According to the method for processing environmental time series data based on the Transformer model in claim 1, it is characterized in that: The window length T is 1 hour, 6 hours or 12 hours, and the time interval δt is 1 hour.

4. According to the method for processing environmental time series data based on the Transformer model in claim 1, it is characterized in that: The Transformer model structure selects the Encoder-Decoder architecture.

Images