Tailing pond abnormal data monitoring method and system based on improved Transform

Abstract

The invention provides a tailing pond abnormal data monitoring method based on an improved Transform, and belongs to the technical field of tailing pond safety monitoring. Adding a time convolutional network before the self-attention mechanism, performing fine modeling on different time sequence associations by adopting a branch structure to obtain an optimized attention mechanism, and constructing a detection model based on the optimized attention mechanism and the time convolutional network; and training the constructed detection model based on the reconstruction loss function and in combination with a maximum and minimum value strategy, calculating an abnormal score based on the trained detection model, and identifying an abnormal point in the abnormal score. By introducing a time convolutional network component to replace an absolute position coding mechanism in a traditional Transform model, complex long-term dependency in time sequence data is successfully captured, effective fusion of global time sequence information is realized, and the accuracy of anomaly detection is improved. And secondly, a self-attention mechanism module is optimized, the recognition capability of the model on an abnormal mode is enhanced, and the detection performance is further improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of tailing pond safety monitoring, and particularly relates to a tailing pond abnormal data monitoring method and system based on an improved Transformer. BACKGROUND

[0002] As an important facility in the production process of mines, the safety of tailing ponds is directly related to the surrounding ecological environment, the safety of life and property of downstream residents, and the sustainable development of mining enterprises. However, in the field of tailing pond safety monitoring technology, it is currently facing multiple severe technical challenges, which are like a series of difficult-to-cross gullies, hindering the further development and improvement of tailing pond safety monitoring technology. The field of tailing pond safety monitoring technology is facing multiple technical challenges.

[0003] Firstly, frequent abnormal data occurs in the multi-sensor monitoring system, which seriously affects the accuracy of safety assessment. The existence of these abnormal data makes it difficult for safety assessment personnel to accurately judge the actual safety status of the tailing pond. If these abnormal data are directly used for safety assessment, it may lead to misjudgment or missed judgment of the potential danger of the tailing pond, thus bringing great hidden dangers to the safe operation of the tailing pond.

[0004] Secondly, the tailing pond monitoring data has high dimensionality and complexity, and traditional models perform poorly in high-dimensional unsupervised scenarios, which are prone to misjudgment of normal samples. Traditional models perform poorly in high-dimensional unsupervised scenarios and are difficult to effectively process these complex data. In the unsupervised learning scenario, since there is no pre-labeled training sample, the model needs to automatically mine the internal structure and regularity of the data from the vast amount of high-dimensional data. However, traditional models are often based on some simplified assumptions, such as linear separability of data, independence between features, etc. But in the tailing pond monitoring data, these assumptions are difficult to hold. For example, there is a complex nonlinear relationship between the displacement change of the tailing pond and the factors such as seepage pressure and water level, and there may be mutual influence and coupling between various factors, which makes it difficult for traditional models to accurately describe the distribution characteristics of data.

[0005] In addition, there are complex long-term dependencies in the time series data of tailing ponds, which are difficult for traditional models to effectively capture, limiting the accuracy and robustness of anomaly detection. Some traditional models based on recurrent neural networks (RNN) have certain advantages in processing sequence data, but also have problems such as gradient vanishing or gradient explosion, making it difficult to effectively learn long-term dependencies. When the sequence length is long, the gradient may gradually vanish or explode in the backpropagation process of the RNN model, making the model unable to accurately update the parameters, and thus unable to capture the long-distance dependencies in the data.

[0006] In view of these challenges, an improved Transformer-based tailings pond safety monitoring abnormal data analysis method emerges as the times require. SUMMARY

[0007] The present application aims to overcome the above-mentioned deficiencies, and provide an improved Transformer-based tailings pond abnormal data monitoring method and system.

[0008] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions: In the first aspect, the present application provides an improved Transformer-based tailings pond abnormal data monitoring method, comprising the following steps: Obtain the historical time series data of the tailings pond; Preprocess the obtained historical time series data; Add a time convolution network before the self-attention mechanism, use a branch structure to finely model the correlation of different time series, obtain an optimized attention mechanism, and construct a detection model based on the optimized attention mechanism and the time convolution network; Train the constructed detection model based on the reconstruction loss function combined with the maximum and minimum value strategy to obtain a trained detection model; Calculate the abnormal score based on the trained detection model to identify the abnormal points.

[0009] In the preprocessing step of the obtained tailings pond historical time series data, it includes jump point removal, missing value filling and normalization processing, wherein the jump point removal uses a wavelet method based on three standard deviation criteria, the missing value filling uses a linear interpolation method for filling, and the data normalization uses standard deviation normalization to scale the processed data to the range of standard normal distribution.

[0010] In the step of adding a time convolution network before the self-attention mechanism, using a branch structure to finely model the correlation of different time series, obtaining an optimized attention mechanism, and constructing a detection model based on the optimized attention mechanism and the time convolution network, The time convolution network stacks multiple dilated causal convolution layers before the self-attention mechanism, extracts long-term dependencies and global information in the time series through the time convolution network, and the architecture of the time convolution network is represented as follows:

[0011] Among them, represents the vector mapping relationship of the dilated causal convolution layer, represents the input multi-dimensional time series data, represents an activation function.

[0012] An autoregressive module is introduced in the time convolution network to capture the linear features in the time series data, and the output autoregressive component is represented as follows:

[0013] wherein, represents a weight value, represents noise when the model is reconstructed, represents the i-th autoregressive component output at the moment.

[0014] The branch structure is used to finely model different time series associations to obtain an optimized attention mechanism, and the specific method is as follows: The time series association between each time point and the entire sequence is obtained through the self-attention mechanism, and the association weight distribution between each time point in the sequence is calculated; The distribution calculated by the self-association weight in the sequence is called sequence association; the case where the abnormal point and the adjacent time point form a strong association is called prior association; The branch structure design is used to model the prior association and the sequence association respectively, and the specific formula is as follows:

[0015]

[0016]

[0017] wherein, represents the multi-dimensional time series data of the i-th layer, and there are L layers in total; respectively represent the query vector Q, the key vector K and the value vector V in the self-attention mechanism and the learnable scale parameter; respectively represent the parameter matrix of the i-th layer of ; represents the parameter matrix of the i-th layer of ; represents the prior association model, represents the i-th feature of the input sequence data, is the mean; is the sequence association model, represents the feature dimension of the model.

[0018] The association difference is used to represent the quantification of the distance between the prior association and the sequence association of each time point, which is formalized as the KL divergence between the prior association and the sequence association, and the specific formula is as follows:

[0019] wherein, is the corresponding and ​The KL divergence is calculated between two discrete distributions for each row of is the pointwise difference in correlation of X relative to the multi-layer prior correlation P and the serial correlation S.

[0020] In the step of training the constructed detection model based on the reconstruction loss function and combining the maximum and minimum strategies to obtain the trained detection model, the formula of the reconstruction loss function is expressed as follows:

[0021] in, The original sequence The reconstruction sequence of and represent the F norm and K norm respectively, It is a balancing item.

[0022] Based on the maximum and minimum strategy, in the minimization stage, the prior association is optimized to make it approximate to the sequence association learned from the original time series; in the maximization stage, the sequence association is optimized to increase the difference between the associations. This process makes the sequence association pay more attention to non-adjacent points, thereby increasing the difficulty of outlier reconstruction. The specific formula is as follows:

[0023]

[0024] in, Indicates that the sequence association does not participate in the minimization model training phase, Indicates that the prior association does not participate in the maximization model training phase.

[0025] In the step of calculating the anomaly score based on the trained detection model and identifying the outliers, the anomaly score calculation formula is as follows:

[0026] where ⊙ represents element-wise multiplication.

[0027] In a second aspect, the present invention provides a tailings pond abnormal data monitoring system based on an improved Transformer, comprising: Data acquisition module, used to obtain historical time series data of tailings ponds; Data preprocessing module, used to preprocess the acquired historical time series data; The model building module is used to add a temporal convolutional network before the self-attention mechanism, use a branching structure to finely model the associations of different time series, obtain an optimized attention mechanism, and build a detection model based on the optimized attention mechanism and the temporal convolutional network; The model training optimization module is used to train the constructed detection model based on the reconstruction loss function and the maximum and minimum strategies to obtain a trained detection model; The outlier identification module is used to calculate the anomaly score based on the trained detection model and identify the outliers.

[0028] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method for monitoring abnormal data in tailings ponds based on an improved Transformer, comprising the following steps: obtaining historical time series data of the tailings pond; preprocessing the obtained historical time series data; adding a temporal convolutional network before the self-attention mechanism, using a branching structure to finely model the associations between different time series to obtain an optimized attention mechanism, and constructing a detection model based on the optimized attention mechanism and the temporal convolutional network; training the constructed detection model based on a reconstruction loss function combined with a maximum-minimum strategy to obtain a trained detection model; and calculating anomaly scores based on the trained detection model to identify anomalies. By introducing a temporal convolutional network component to replace the absolute position encoding mechanism in the traditional Transformer model, the method successfully captures the complex long-term dependencies in the time series data and effectively integrates global time series information, thereby improving the accuracy of anomaly detection. Secondly, the self-attention mechanism module is optimized, a branching structure design is adopted, and a maximum-minimum strategy is combined to enhance the model's ability to recognize abnormal patterns. This allows reconstruction errors and association differences to form a synergistic effect during the detection process, further improving detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flow chart of the method of the present invention; Figure 2 This is a diagram of the model architecture of the present invention; Figure 3 This is a diagram of the TCN component architecture in the present invention; Figure 4 This is the architecture diagram of the optimized self-attention mechanism in the present invention. DETAILED DESCRIPTION

[0030] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0031] Example 1 like Figure 1 As shown in FIG, a tailings pond abnormal data monitoring method based on an improved Transformer includes the following steps: S1: Obtain historical time series data of tailings pond; S2: preprocessing the acquired historical time series data; S3: adding a time convolution network before the self-attention mechanism, using a branch structure to finely model different time series correlations, obtaining an optimized attention mechanism, and constructing a detection model based on the optimized attention mechanism and the time convolution network; S4: training the constructed detection model based on a reconstruction loss function and combining a max-min value strategy to obtain a trained detection model; S5: calculating an anomaly score based on the trained detection model to identify abnormal points.

[0032] Specifically, in S1, the tailings pond historical time series data includes immersion line monitoring data (JRX), dry beach length and beach top high monitoring data (GTD), and external displacement rate monitoring data (WWS).

[0033] Specifically, in S2, the acquired historical time series data of the tailings pond is subjected to jump point removal, missing value filling, and normalization processing to ensure data quality and lay a foundation for subsequent analysis. The jump point removal uses a wavelet method based on the 3σ (three standard deviations) criterion to identify and remove abnormal jump points in the historical time series data. The missing data filling uses linear interpolation to fill in the missing values by calculating the slope of adjacent data points to ensure data continuity and integrity. The data normalization uses a standard deviation normalization method to scale the processed data to a standard normal distribution range, as shown in equation (1): (1) wherein: is the normalized data, is the data to be normalized, and are the minimum and maximum values of the training data, is a very small constant vector to prevent division by zero. The data is normalized to be within the range of [0, 1].

[0034] Specifically, in S3, a time convolution network is added before the attention mechanism, a branch structure is used to finely model different time series correlations, an optimized attention mechanism is obtained, and a detection model is constructed based on the optimized attention mechanism and the time convolution network. The specific method is as follows: When using the Transformer model for anomaly detection in tailings dam time series data, the traditional absolute position encoding mechanism has limitations in expressing sequence information. This mechanism tends to incorrectly map similar data patterns to different time windows, ignoring the complexity and dynamic nature of tailings dam multi-sensor time series data. This mismatch not only weakens the model's ability to capture temporal dependencies but also limits its in-depth understanding of the data's inherent structure and changing patterns, thus affecting the accuracy and reliability of anomaly detection.

[0035] like Figure 2 As shown, the detection model (TCN-Transformer) builds on the traditional Transformer architecture by introducing a temporal convolutional network (TCN) component to replace absolute position encoding. This model leverages the advantages of convolutional operations to extract features across time steps, effectively capturing both long-term and local time series patterns in the data. Furthermore, it optimizes the self-attention mechanism and employs a branching structure to fine-tune modeling of different time series, resulting in a well-constructed detection model that can more accurately identify potential abnormal patterns.

[0036] 1) The TCN component is introduced before the self-attention mechanism. The TCN component stacks multiple dilated causal convolutional layers to fully capture the long-term dependencies in time series data. TCN extracts long-term dependencies and global information in time series. The architecture of the TCN component is shown as follows:

[0037] in, Represents the vector mapping relationship of the expanded causal convolutional layer, Represents the input multidimensional time series data, Represents the activation function.

[0038] Since the reconstruction error of the model is generally large, the error intervals of normal data and abnormal data overlap, resulting in missed detection. Therefore, an autoregressive module is introduced into the TCN component to capture the linear features in time series data. The output autoregressive component is expressed as follows:

[0039] in, Represents the weight value, represents the noise during model reconstruction, Indicates The i-th autoregressive component output at time t.

[0040] Further, such as Figure 3 As shown, the TCN component retains the The convolutional layer. The convolutional layer is more conducive to retaining shallow features and further enriching the information features extracted by this component. Represents the output of the convolutional layer, then the final output of the TCN component can be used Calculate, where Represents the output of the autoregressive module.

[0041] 2) If Figure 4 As shown in the figure, the detection model optimizes the self-attention mechanism. This mechanism captures the temporal correlation between each time point and the entire sequence, specifically by calculating the distribution of correlation weights between each time point in the sequence. This correlation weight distribution provides a rich description of the entire temporal context, revealing dynamic patterns such as periodicity or trends. This distribution calculated from the self-correlation weights in the sequence is called serial correlation. However, since outliers are relatively rare compared to normal points, it is difficult for them to establish a clear correlation with the entire sequence. However, they may form strong correlations with adjacent time points, which is called prior correlation. This prior correlation is calculated using a learnable Gaussian kernel. A branching structure design is used to model both prior correlation and serial correlation separately, thereby more accurately capturing the characteristics of outliers.

[0042] With the unimodal characteristics of the Gaussian kernel, the detection model can focus more on the correlation between adjacent time points. In order to enhance the adaptability of the model to different time series patterns, we introduce a learnable scale parameter for the Gaussian kernel. , enabling it to flexibly handle abnormal segments of different lengths. The sequence association branch learns associations from the original sequence and can adaptively find the most effective association structure to achieve more accurate anomaly detection. The formula is as follows:

[0043]

[0044]

[0045] in, Indicates the Layers of multidimensional time series data, a total of L layers; Represent the query vector Q, key vector K, value vector V and learnable scale parameters in the self-attention mechanism respectively; Respectively represent layer The parameter matrix of represents the Gaussian distribution probability density function, that is, the prior association model, represents the i-th feature of the input sequence data, is the mean; is the attention weight calculation formula of the detection model, that is, the sequence association model, Represents the feature dimension of the model.

[0046] The association of abnormal locations in tailings dam time series data should be concentrated on adjacent time points, which are more likely to contain similar abnormal patterns due to continuity. The dominant normal time points are associated with information of the entire series, not limited to adjacent areas. Based on this, the association difference is used to quantify the distance between the prior association and the serial association of each time point. It is formalized as the KL divergence between the prior association and the serial association. The larger the value of the KL divergence, the greater the information loss or error introduced when approximating one distribution with another. This makes the KL divergence very intuitive in measuring the error or information loss of the model. The expression is as follows:

[0047] in, corresponds to and The KL divergence is calculated between two discrete distributions for each row of is the pointwise difference in correlation of X relative to the multi-layer prior correlation P and the serial correlation S.

[0048] Specifically, in S4, the detection model is optimized based on the reconstruction loss function. The reconstruction loss guides the sequence association to find the most informative association, and combined with the minimax strategy, it significantly enhances the distinguishability of association differences, allowing the reconstruction error and association differences to cooperate with each other in the detection process, thereby improving the accuracy of anomaly detection.

[0049] The formula of the reconstruction loss function is as follows:

[0050] in, The original sequence The reconstruction sequence of and represent the F norm and K norm respectively, It is a balancing item.

[0051] If you want to minimize the model loss, you must maximize the correlation difference. However, directly maximizing the correlation difference will cause the scale parameter of the Gaussian kernel to decrease dramatically, and the graph will become narrower and more concentrated, which makes the prior sequence meaningless. The present invention uses a minimax strategy. In the minimization stage, the prior correlation is optimized to make it approximate to the sequence correlation learned from the original time series. This step can adapt the prior correlation to different time series patterns while avoiding the scale parameter of the prior correlation being too small. In the maximization stage, the sequence correlation is optimized to increase the difference between the correlations. This process makes the sequence correlation pay more attention to non-adjacent points, thereby increasing the difficulty of reconstructing outliers.

[0052] The specific formula is as follows:

[0053]

[0054] in, Indicates that the sequence association does not participate in the minimization model training phase, Indicates that the prior association does not participate in the maximization model training phase. In order to switch the maximum and minimum learning directions, the Direction implementation: Since the prior association and sequence association need to be optimized separately, the gradient of the minimization stage is passed to the prior association, and the gradient of the maximization stage is passed to the sequence association.

[0055] Specifically, in S5, based on the trained detection model, the anomaly score of each time point is calculated, and the anomaly point is determined according to the threshold. The anomaly score calculation formula is as follows:

[0056] where ⊙ represents element-wise multiplication.

[0057] If the anomaly score is greater than the threshold, it is determined that there is abnormal data at that time point.

[0058] Example 2 A tailings pond abnormal data monitoring system based on an improved Transformer, comprising: Data acquisition module, used to obtain historical time series data of tailings ponds; Data preprocessing module, used to preprocess the acquired historical time series data; The model building module is used to add a temporal convolutional network before the self-attention mechanism, use a branching structure to finely model the associations of different time series, obtain an optimized attention mechanism, and build a detection model based on the optimized attention mechanism and the temporal convolutional network; The model training optimization module is used to train the constructed detection model based on the reconstruction loss function and the maximum and minimum strategies to obtain a trained detection model; The outlier identification module is used to calculate the anomaly score based on the trained detection model and identify the outliers.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A tailings pond abnormal data monitoring method based on improved Transformer, characterized in that: The steps include: Obtain historical time series data of tailings ponds; Preprocess the acquired historical time series data; A temporal convolutional network is added before the self-attention mechanism, and a branching structure is used to finely model the associations between different time series to obtain an optimized attention mechanism. A detection model is then constructed based on the optimized attention mechanism and the temporal convolutional network. The constructed detection model is trained based on the reconstruction loss function and combined with the maximum and minimum strategies to obtain a trained detection model; Calculate the anomaly score based on the trained detection model and identify the anomalies.

2. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 1 is characterized in that: The steps of preprocessing the acquired historical time series data of the tailings pond include jump point removal, missing value filling and normalization processing, wherein jump point removal adopts a wavelet method based on the triple standard deviation criterion, missing value filling adopts linear interpolation method, and data normalization adopts standard deviation normalization to scale the processed data to the standard normal distribution range.

3. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 1 is characterized in that: In the step of adding a temporal convolutional network before the self-attention mechanism, using a branching structure to finely model the associations of different time series to obtain an optimized attention mechanism, and constructing a detection model based on the optimized attention mechanism and the temporal convolutional network, A temporal convolutional network is added before the self-attention mechanism. The temporal convolutional network stacks multiple dilated causal convolutional layers to extract long-term dependencies and global information in time series. The architecture of the temporal convolutional network is shown as follows: in, Represents the vector mapping relationship of the expanded causal convolutional layer, Represents the input multidimensional time series data, Represents the activation function.

4. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 3 is characterized in that: An autoregressive module is introduced into the temporal convolutional network to capture the linear features in time series data. The output autoregressive component is expressed as follows: in, Represents the weight value, represents the noise during model reconstruction, Indicates The i-th autoregressive component output at time t.

5. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 3 is characterized in that: The branch structure is used to finely model the associations of different time series to obtain an optimized attention mechanism. The specific method is as follows: The self-attention mechanism is used to obtain the temporal association between each time point and the entire sequence, and the association weight distribution between each time point in the sequence is calculated; The distribution calculated by the autocorrelation weight in the sequence is called serial correlation; the situation where a strong correlation is formed between an outlier and adjacent time points is called prior correlation; A branch structure design is used to model the prior association and sequence association respectively. The specific formula is as follows: in, Indicates the Multi-dimensional time series data of layers, with a total of L layers; Represent the query vector Q, key vector K, value vector V and learnable scale parameters in the self-attention mechanism respectively; Respectively represent layer The parameter matrix of represents the prior association model, represents the i-th feature of the input sequence data, is the mean; is a sequential association model, Represents the feature dimension of the model.

6. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 5 is characterized in that: The correlation difference is used to quantify the distance between the prior correlation and the serial correlation at each time point, which is formalized as the KL divergence between the prior correlation and the serial correlation. The specific formula is as follows: in, corresponds to and The KL divergence is calculated between two discrete distributions for each row of is the pointwise difference in correlation of X relative to the multi-layer prior correlation P and the serial correlation S.

7. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 6 is characterized in that: In the step of training the constructed detection model based on the reconstruction loss function and combining the maximum and minimum strategies to obtain the trained detection model, the formula of the reconstruction loss function is expressed as follows: in, The original sequence The reconstruction sequence of and represent the F norm and K norm respectively, It is a balancing item.

8. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 7 is characterized in that: Based on the maximum and minimum strategy, in the minimization stage, the prior association is optimized to make it approximate to the sequence association learned from the original time series; in the maximization stage, the sequence association is optimized to increase the difference between the associations. This process makes the sequence association pay more attention to non-adjacent points, thereby increasing the difficulty of outlier reconstruction. The specific formula is as follows: in, Indicates that the sequence association does not participate in the minimization model training phase, Indicates that the prior association does not participate in the maximization model training phase.

9. The tailings pond abnormal data monitoring method based on improved Transformer according to claim 8 is characterized in that: In the step of calculating the anomaly score based on the trained detection model and identifying the outliers, the anomaly score calculation formula is as follows: where ⊙ represents element-wise multiplication.

10. The tailings pond abnormal data monitoring system based on improved Transformer according to any one of claims 1 to 9, characterized in that: include: Data acquisition module, used to obtain historical time series data of tailings ponds; Data preprocessing module, used to preprocess the acquired historical time series data; The model building module is used to add a temporal convolutional network before the self-attention mechanism, use a branching structure to finely model the associations of different time series, obtain an optimized attention mechanism, and build a detection model based on the optimized attention mechanism and the temporal convolutional network; The model training optimization module is used to train the constructed detection model based on the reconstruction loss function and the maximum and minimum strategies to obtain a trained detection model; The outlier identification module is used to calculate the anomaly score based on the trained detection model and identify the outliers.

Images