Multi-source heterogeneous anomaly detection method based on time correlation

Abstract

The invention discloses a multi-source heterogeneous anomaly detection method based on time correlation, and belongs to the technical field of water diversion engineering, and the method comprises the steps: S1, obtaining multi-source sensor time sequence data in multi-source heterogeneous data, carrying out the preprocessing of the multi-source sensor time sequence data, and dividing a training set and a test set; s2, constructing a double-branch depth feature extraction network model, training by adopting the training set, and testing through the test set to obtain a trained double-branch depth feature extraction network model; the double-branch depth feature extraction network model comprises a double-branch unit, an attention feature fusion unit, a classifier unit and an output layer unit which are connected in sequence; and S3, inputting to-be-detected data into the trained double-branch depth feature extraction network model, and finally outputting an anomaly diagnosis result. A double-branch depth feature extraction network is constructed, adaptive fusion is realized through an attention mechanism, and the problem of poor modal adaptability of heterogeneous data is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of water diversion engineering, and in particular relates to a multi-source heterogeneous anomaly detection method based on time correlation. BACKGROUND

[0002] With the continuous expansion of large-scale water diversion projects in China, the monitoring data generated during the construction and operation process presents the characteristics of multi-source heterogeneity, strong non-stationarity, high dimensionality, and sparse abnormal samples. Traditional anomaly detection methods mainly rely on threshold judgment, statistical models or single modal machine learning algorithms, which are difficult to effectively deal with the problem of multi-dimensional, cross-modal anomaly feature expression under complex working conditions. The main problems include: (1) Traditional methods mainly rely on single data modal for anomaly detection, which is difficult to cover the complex characteristics of multi-source heterogeneous data in engineering, and is prone to miss key abnormal features, resulting in insufficient detection accuracy. (2) Long-period gradual changes in non-stationary time series data are masked by short-term fluctuations, and existing methods are difficult to effectively model long-term dependence, and have limited ability to identify multi-time node collaborative anomalies.

[0003] (3) In terms of multi-modal fusion, traditional methods mainly use early fusion or late fusion strategies, lack the ability to model dynamic correlation between heterogeneous features, and are prone to introduce redundant noise, resulting in low fusion efficiency.

[0004] (4) Most current intelligent diagnosis models are "black box" structures, lack explainability mechanisms, cannot visually display decision-making basis, are difficult to locate key time nodes and spatial regions corresponding to anomalies, and cannot provide explainable technical support for engineering trouble shooting.

[0005] Therefore, there is an urgent need for a multi-modal intelligent diagnosis framework that can accurately identify multi-dimensional anomalies in complex engineering environments and provide explainable decision support to improve the intelligent management level of the whole life cycle of water diversion projects. SUMMARY

[0006] The purpose of the present application is to overcome the deficiencies of the prior art in the fusion of multi-source sensor time series data and image data, such as insufficient capture of time series dynamic features, poor adaptability of heterogeneous data modal differences, and poor explanation of anomaly positioning, and to provide a multi-source heterogeneous anomaly detection method based on time correlation.

[0007] To solve the above technical problems, the present application provides the following technical solutions: A multi-source heterogeneous anomaly detection method based on time correlation, comprising the following steps: S1: Obtain multi-source sensor time series data in multi-source heterogeneous data, and pre-process the multi-source sensor time series data to divide training set and test set; S2: a double-branch deep feature extraction network model is constructed, the training set is used for training, and the test set is used for testing, so as to obtain the trained double-branch deep feature extraction network model; The double-branch deep feature extraction network model comprises a double-branch unit, an attention feature fusion unit, a classifier unit and an output layer unit connected in sequence; S3: the to-be-tested data is input into the trained double-branch deep feature extraction network model, and finally an abnormal diagnosis result is output.

[0008] By adopting the technical scheme, a double-branch deep feature extraction network is constructed, and self-adaptive fusion is realized through an attention mechanism, so that the problem of poor adaptability of heterogeneous data modalities is solved.

[0009] As a preferred scheme of the application, the preprocessing of the multi-source sensor time series data in step S1 comprises: S11: the multi-source sensor time series data is sampled through a sliding window to obtain normal and abnormal data sets, and the feature dimension is 2000*6; S12: the normal and abnormal data sets are adaptively decomposed by combining ICEEMDAN, a new data set is obtained by introducing a residual-driven integration strategy and dynamic noise amplitude adjustment, and the training set and the test set are divided from the new data set.

[0010] As a preferred scheme of the application, the introduction of the residual-driven integration strategy in step S12 comprises: First step: adding I=100 independent Gaussian white noises to the multi-source sensor time series data to construct a signal set, the signal set is: ; wherein s(t) is each one-dimensional subsequence signal, is a Gaussian white noise with a mean of 0 and a standard deviation of 1, is a noise coefficient, is the integration number; Second step: EMD decomposition is performed on each signal set, the first-order intrinsic mode function is extracted, and the integrated average is calculated as the final first-order intrinsic mode function, the calculation formula is: ; Wherein, is the value of the first-order intrinsic mode function at time point t, is the integration number, is the value of the signal after adding the Gaussian white noise for the i-th time at time point t; Third step: update the residual error For the extraction of the k(th) (k≥2) order intrinsic mode function, a residual-driven strategy is adopted, noise with a decaying amplitude is superimposed on the current residual error, and the extraction formula is: ; in, For the current residual signal The value of the signal with added noise at time point t. Let be the value of the (k-1)th order residual signal at time point t. Here, k is the noise amplitude adjustment factor, and k is the order of the currently extracted IMF. Let be the value of the i-th independent Gaussian white noise at time point t; Step 4: Obtain the k-th order intrinsic mode function through the EMD decomposition and ensemble averaging, and update the residuals. The iteration stops when the maximum number of decomposition levels is reached, and a set of intrinsic mode functions is finally obtained. .

[0011] As a preferred embodiment of the present invention, the dynamic noise amplitude adjustment in step S12 includes: First, the energy-dominant criterion is used to screen effective components, and the energy of each intrinsic mode function is calculated using the following formula: ; in, Let be the energy of the k-th eigenmode function. Let N be the value of the k-th order intrinsic mode function at time point t, N be the length of the time series, k be the order of the currently extracted IMF, and K be the maximum number of decomposition layers. Then, the eigenmode functions are sorted from largest to smallest energy, and the first three eigenmode functions are selected to form an eigenma matrix. For each eigenmode function component in the selected eigenma matrix... Perform independent Z-score standardization. The Z-score standardization formula is: ;in, For the standardized first The nth sample, the nth The value of the k-th eigenmode function in the eigenmatrix at time t. For the first The nth sample, the nth The original values ​​of the k-th eigenmode function in the eigenmatrix at time t. For the first The nth sample, the nth The mean of the k-th order eigenmode function in the eigenmatrix For the first The nth sample, the nth The standard deviation of the k-th eigenmode function in the eigenmatrix, where k is the selected IMF order; , , where L is the length of the time series, thus obtaining a new dataset and dividing it into training and test sets.

[0012] As a preferred embodiment of the present invention, the dual-branch unit in step S2 includes a temporal feature extraction branch and an image feature extraction branch; The temporal feature extraction branch includes a two-layer Transformer encoder structure, each layer containing four attention heads, with the hidden layer dimension set to 128 and the model dimension to 64. The image feature extraction branch adopts a three-parallel-branch MSCNN structure, with convolutional kernel sizes set to 3×3, 5×5 and 7×7 respectively, and each branch has 32 output channels. The pooling layer adopts 2×2 max pooling.

[0013] As a preferred embodiment of the present invention, the attention feature fusion unit adopts an attention mechanism to achieve adaptive weighted fusion of temporal and image features, and the number of neurons in the fully connected layer is set to 16.

[0014] In a preferred embodiment of the present invention, the temporal feature extraction branch processes the temporal data in the training set and the test set, and uses a Transformer encoder to capture long-range dependencies. Let the input feature matrix be... ; First, it is mapped to a high-dimensional latent space using linear projection. The formula for the output dimension is: ; in, Let be the projection weight matrix. For bias terms, For output dimensions; Then, learnable positional encoding is introduced to preserve time step information: ; Where P represents the trainable location embedding parameters. Add the dimension after position encoding; It adopts a stacked Transformer encoder structure with a total of L=2 layers. Each layer contains a multi-head self-attention mechanism and a feedforward neural network, and is equipped with residual connections and layer normalization. The calculation formula for the multi-head attention mechanism is as follows: ; in, , For the first The input feature matrix of the layer, For the projected weight matrix of the query, Let be the projection weight matrix of the key. The projection weight matrix is ​​the value of the value. For the output of the h-th attention head, Let be the dimension of the key vector. These are query, key, and value matrices, respectively. H=4; Next, multi-head output splicing and linear transformation are performed, followed by residual connection and layer normalization to obtain... ; The calculation formula for the feedforward neural network is as follows: ; in, This is the weight matrix of the first layer of the feedforward neural network. This is the weight matrix of the second layer of the feedforward neural network. This is the bias vector of the first layer of the feedforward neural network; This is the bias vector for the second layer of the feedforward neural network; Final output: ; in, For the first The features of the layer after processing by the multi-head self-attention mechanism, FFN(.) is the feedforward neural network function, and LayerNorm(.) is the layer normalization function; The enhanced temporal representation is obtained after two layers of encoders, and the temporal dimension is aggregated by global average pooling. Finally, the dimensions are compressed using a linear transformation. The compression formula is as follows: ; in, The weight matrix is ​​a linear transformation matrix. For linear transformation bias terms, The feature vector after global average pooling. This is the compressed feature vector.

[0015] As a preferred embodiment of the present invention, the image feature extraction branch converts the one-dimensional signal into a two-dimensional Gram angle field, and then uses a convolutional neural network to extract its geometric structural features. First, regarding a one-dimensional signal segment... First, perform min-max standardization to map it to the interval [−1, 1]. The mapping formula is: ; in, This represents the value of the original one-dimensional signal segment at time point t. It is the minimum value in the original signal segment. The maximum value in the original signal segment. This represents the value of a standardized one-dimensional signal segment at time point t. The polar angle sequence is calculated using the following formula: ; in, The value of the standardized one-dimensional signal segment at time point t. The extreme angle at time point t; Combined with time angle Construct a summation-type Gram angle field, as shown in the following formula: ; in To calculate the value of the summation-type Gram angle field at position (i,j), the final image tensor I is generated.

[0016] As a preferred embodiment of the present invention, the three-parallel-branch MSCNN structure is configured with 3×3, 5×5, and 7×7 convolutional kernels respectively to capture multi-granularity spatial features. The input heatmap is used, and each branch consists of two convolutional layers, followed by batch normalization and ReLU activation functions. Finally, max pooling is used to downsample and compress the spatial dimension. The calculation formula is as follows: ; in, (.) represents the ReLU activation function, and BN(.) represents the batch normalization operation. This indicates a convolution operation using a k×k kernel in the i-th layer. For max pooling operation, , , Feature maps output for different branches; Next, adaptive average pooling is used to compress the feature maps of each branch into feature vectors, and the three feature vectors are concatenated to obtain... The obtained time series features and image features The data is concatenated, and the attention intermediate representation is calculated, along with the normalized weights: ; in, For attention intermediate representation, To normalize the weights, , These are the weight matrix and bias term for generating the intermediate representation of the attention function, respectively. , These are the weight matrix and bias term, representing the normalized weights generated by the attention intermediates. Obtain fusion features The fused features are input into a two-layer fully connected network, and the anomaly probability is output. The calculation formula is as follows: ; Among them, the output of the hidden layer h is... (.) represents the Sigmoid function. The output represents the probability of an anomaly occurring. , These are the weight matrix and bias terms of the first fully connected layer network, respectively. , These are the weight matrix and bias term of the second fully connected network, respectively. End-to-end training is performed using the binary cross-entropy loss function, and the training formula is as follows: ; Where N is the sample size. For loss training.

[0017] As a preferred embodiment of the present invention, the calculation steps of the heatmap include: calculating the prediction score of target category c. For feature maps The gradient of the feature map is calculated, and global average pooling is performed on the gradient at all spatial locations (u, v) of each feature map to obtain the importance weight of that feature map. , Where Z is the total number of pixels in the feature map. The predicted score for target category c. Let be the value of the k-th feature map at spatial location (u,v); All weights With the corresponding feature map We perform a weighted summation and retain features that contribute positively to classification using the ReLU activation function. The calculation formula is as follows: ; in, For the k-th feature map, The importance weights of the kth feature map Activate the mapping for the class; Finally, bilinear interpolation is used to... Upsampled to the original GAF image size, and superimposed on the original image to generate a heatmap. The color intensity reflects the degree of influence of the structural changes in the phase space of the corresponding time domain interval on the anomaly detection.

[0018] Compared with the prior art, the advantages of the present invention are as follows: (1) A novel architecture combining improved adaptive noise complete set empirical mode decomposition (ICEEMDAN) with a dual-branch deep network is proposed. By processing multi-scale temporal features and Gram angle field image features through temporal and image dual branches respectively, the problem of feature extraction and fusion of heterogeneous modal data is effectively solved.

[0019] (2) An improved ICEEMDAN method is used, employing a residual-driven integration strategy and a dynamic noise amplitude adjustment mechanism, which significantly improves the decomposition stability and feature extraction effect of intrinsic mode function (IMF).

[0020] (3) A collaborative working mechanism between multi-scale CNN and Transformer encoder was designed. The multi-scale CNN branch adopts three parallel convolutional structures: 3×3, 5×5, and 7×7, which effectively captures multi-scale information of image features. The Transformer branch captures temporal dependencies through a multi-head self-attention mechanism, realizing the deep extraction of temporal dynamic features. In addition, an adaptive feature fusion mechanism based on attention weights is used, which can dynamically adjust the contribution of temporal features and image features in the final decision, and realize the optimal fusion of the two modal features. (4) A multi-level interpretability analysis framework including Grad-CAM feature visualization, attention weight distribution analysis, and feature space visualization is established to make the model decision process transparent and traceable, and enhance the credibility of anomaly detection results. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings: Figure 1 This is a flowchart of a time-correlated multi-source heterogeneous anomaly detection method according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a dual-branch deep feature extraction network model for a time-correlated multi-source heterogeneous anomaly detection method as described in Embodiment 1 of the present invention. Figure 3 The waveform diagram shows the training loss and accuracy of a time-correlated multi-source heterogeneous anomaly detection method according to Embodiment 2 of the present invention. Figure 4 This is a schematic diagram of the confusion matrix of a time-correlated multi-source heterogeneous anomaly detection method according to Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of the multi-scale IMF feature space distribution of a time-correlated multi-source heterogeneous anomaly detection method according to Embodiment 2 of the present invention. Figure 6 This is a schematic diagram of T-SNE feature visualization of the fusion features of a time-correlated multi-source heterogeneous anomaly detection method according to Embodiment 2 of the present invention. Figure 7 This is a Grad-CAM heatmap of anomaly samples from a time-correlated multi-source heterogeneous anomaly detection method as described in Embodiment 2 of the present invention. Figure 8 This is a Grad-CAM heatmap of a normal sample from a time-correlated multi-source heterogeneous anomaly detection method as described in Embodiment 2 of the present invention. Figure 9 This is a schematic diagram of the attention weight distribution of a time-correlated multi-source heterogeneous anomaly detection method according to Embodiment 2 of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0023] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between components or an indirect connection via other components.

[0025] In recent years, with the deep integration of emerging technologies such as artificial intelligence, the Internet of Things, edge computing, and cloud computing, water conservancy engineering construction is accelerating its transformation towards "smart construction." Multi-source heterogeneous data, including personnel behavior recognition, equipment status monitoring, and environmental parameter perception, are widely collected at construction sites, forming a comprehensive perception network covering all elements from "people to machines to materials to methods to environment." Against this backdrop, building an intelligent diagnostic system based on multi-modal data fusion has become a core support for realizing intelligent management throughout the entire lifecycle of water diversion projects.

[0026] Currently, anomaly detection in water diversion projects mainly relies on mechanistic models or traditional data-driven methods. Mechanism-based methods require constructing hydraulic, structural, or progress evolution equations. However, due to variable geological conditions, complex processes, and dynamic adjustments in operating conditions, it is difficult to establish accurate and unified mathematical models, especially in scenarios involving the fusion of multi-source heterogeneous data, which limits their practical application. This type of data not only includes traditional structural and hydraulic monitoring signals such as stress, displacement, flow, and vibration, but also involves visual information collected by cameras, construction progress and environmental parameters uploaded in real time by edge nodes, and spatial geometric data from BIM models, exhibiting complex characteristics of high dimensionality, dynamics, non-stationarity, and spatiotemporal coupling. Therefore, data-driven intelligent diagnostic methods are gradually becoming the mainstream approach.

[0027] However, existing data-driven methods still face numerous technical bottlenecks. Single signal processing techniques (such as empirical mode decomposition, wavelet transform, and variational mode decomposition) struggle to reliably extract time-frequency features from non-stationary monitoring data, are susceptible to noise interference, and lack robustness. Traditional dimensionality reduction methods (such as principal component analysis and kernel principal component analysis) are prone to losing key anomaly information when processing high-dimensional, strongly coupled features. Machine learning models such as Support Vector Machine (SVM) and Random Forest (RF) rely on manual feature engineering, making it difficult to capture nonlinear correlations between multiple dimensions and exhibiting weak generalization ability for rare anomaly samples. Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term Memory Network (BiLSTM) are insufficient for modeling long-period dependencies, while Convolutional Neural Network (CNN) struggles to balance local mutations with global temporal evolution. More critically, existing deep learning models generally lack interpretability mechanisms, failing to pinpoint the time and spatial region of anomaly occurrences, making it difficult for engineers to trust and verify diagnostic results.

[0028] In summary, this invention proposes a multimodal intelligent diagnostic framework that integrates Improved Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (ICEEMDAN), Gramian Angular Field (GAF), Multi-Scale Convolutional Neural Network (MSCNN), and Transformer architecture. It also introduces a self-attention heterogeneous fusion mechanism and a gradient-weighted class activation mapping (Grad-CAM) visualization mechanism. The aim is to achieve accurate identification and interpretable decision support for multidimensional anomalies in complex engineering environments, thereby improving the intelligent management level of the entire lifecycle of water diversion projects.

[0029] Example 1

[0030] A time-correlation-based method for detecting multi-source heterogeneous anomalies, such as Figure 1 As shown, it includes the following steps: S1: Obtain multi-source sensor time-series data from multi-source heterogeneous data, and preprocess the multi-source sensor time-series data to divide it into training set and test set; S2: Construct a dual-branch deep feature extraction network model, such as Figure 2 As shown, the training set is used for training, and the test set is used for testing to obtain a trained dual-branch deep feature extraction network model. The dual-branch deep feature extraction network model includes a dual-branch unit, an attention feature fusion unit, a classifier unit, and an output layer unit connected in sequence. A dual-branch deep feature extraction network model was constructed to solve the problem of poor modal adaptability for heterogeneous data.

[0031] S3: Input the data to be tested into the trained dual-branch deep feature extraction network model, and finally output the anomaly diagnosis result.

[0032] Example 2

[0033] This embodiment is a preferred embodiment of the time-correlated multi-source heterogeneous anomaly detection method described in Embodiment 1; Preferably, step S1, which involves preprocessing the multi-source sensor time-series data to divide it into a training set and a test set, includes: S11: The time-series data from the multi-source sensors are sampled through a sliding window to obtain normal and abnormal datasets with a feature dimension of 2000×6, thereby solving the problem of insufficient capture of time-series dynamic features; S12: Combine ICEEMDAN (Improved Complete Ensemble Empirical Mode Decomposition with Adaptive Noise) to adaptively decompose the normal and abnormal datasets. By introducing a residual-driven ensemble strategy and dynamic noise amplitude adjustment, more stable and efficient intrinsic mode function (IMF) extraction is achieved, a new dataset is obtained, and a training set and a test set are separated from it, thereby solving the problem of insufficient capture of temporal dynamic features.

[0034] Preferably, the integration strategy described in step S12, which involves introducing residual-driven integration, includes: Step 1: Add I=100 independent Gaussian white noise to the multi-source sensor time-series data to construct a signal set, the signal set being: Where s(t) is each one-dimensional subsequence signal, It is Gaussian white noise with a mean of 0 and a standard deviation of 1. For noise system, For the number of integrations; Step 2: Perform EMD decomposition on each signal set, extract its first-order intrinsic mode functions (IMFs), and calculate the ensemble average as the final first-order IMF. The calculation formula is as follows: ; in, The value of the first-order intrinsic mode function at time point t. For the number of integrations, Let be the value of the signal at time t after the i-th addition of Gaussian white noise; Step 3: Update the residuals For the extraction of the k-th (k≥2) eigenmode function, a residual-driven strategy is adopted, which superimposes noise with attenuated amplitude onto the current residual. The extraction formula is as follows: ; in, For the current residual signal The value of the signal with added noise at time point t. Let be the value of the (k-1)th order residual signal at time point t. Here, k is the noise amplitude adjustment factor, and k is the order of the currently extracted IMF. Let be the value of the i-th independent Gaussian white noise at time point t; Step 4: Obtain the k-th order intrinsic mode function through the EMD decomposition and ensemble averaging, and update the residuals. The iteration stops when the maximum number of decomposition levels is reached, and a set of intrinsic mode functions is finally obtained. .

[0035] Further, the dynamic noise amplitude adjustment in step S12 includes: first, using the energy-dominant criterion to screen effective components, and calculating the energy of each intrinsic mode function, using the following formula: ; in, Let be the energy of the k-th eigenmode function. Let N be the value of the k-th order intrinsic mode function at time point t, N be the length of the time series, k be the order of the currently extracted IMF, and K be the maximum number of decomposition layers. Then, the eigenmode functions are sorted from largest to smallest energy, and the first three eigenmode functions are selected to form an eigenma matrix. For each eigenmode function component in the selected eigenma matrix... Perform independent Z-score standardization. The Z-score standardization formula is: ; in, For the standardized first The nth sample, the th The value of the k-th eigenmode function in the eigenmatrix at time t. For the first The nth sample, the th The original values ​​of the k-th eigenmode function in the eigenmatrix at time t. For the first The nth sample, the th The mean of the k-th order eigenmode function in the eigenmatrix For the first The nth sample, the th The standard deviation of the k-th eigenmode function in the eigenmatrix, where k is the selected IMF order; , , where L is the length of the time series, thus obtaining a new dataset and dividing it into training and test sets.

[0036] Preferably, the dual-branch unit in step S2 includes a temporal feature extraction branch and an image feature extraction branch; The temporal feature extraction branch includes a two-layer Transformer encoder structure, each layer containing four attention heads, with the hidden layer dimension set to 128 and the model dimension to 64. The image feature extraction branch adopts a three-parallel-branch MSCNN structure, with convolutional kernel sizes set to 3×3, 5×5 and 7×7 respectively, and each branch has 32 output channels. The pooling layer adopts 2×2 max pooling.

[0037] Furthermore, the attention feature fusion unit employs an attention mechanism to achieve adaptive weighted fusion of temporal and image features, and the number of neurons in the fully connected layer is set to 16.

[0038] Furthermore, the temporal feature extraction branch processes the temporal data in the training set and the test set, using a Transformer encoder to capture long-range dependencies. Let the input feature matrix be... First, it is mapped to a high-dimensional latent space using linear projection. The formula for the output dimension is: ;in, Let be the projection weight matrix. For bias terms, For output dimensions; Then, learnable positional encoding is introduced to preserve time step information: ; Where P represents the trainable location embedding parameters. Add the dimension after position encoding; It adopts a stacked Transformer encoder structure with a total of L=2 layers. Each layer contains a multi-head self-attention mechanism and a feedforward neural network, and is equipped with residual connections and layer normalization. The calculation formula for the multi-head attention mechanism is as follows: ; in, , For the first The input feature matrix of the layer, For the projected weight matrix of the query, Let be the projection weight matrix of the key. The projection weight matrix is ​​the value of the value. For the output of the h-th attention head, Let be the dimension of the key vector. These are query, key, and value matrices, respectively. H =4; Next, multi-head output splicing and linear transformation are performed, followed by residual connection and layer normalization to obtain... ; The calculation formula for the feedforward neural network is as follows: ; in, This is the weight matrix of the first layer of the feedforward neural network. This is the weight matrix of the second layer of the feedforward neural network. This is the bias vector of the first layer of the feedforward neural network; This is the bias vector for the second layer of the feedforward neural network; Final output: ; in, For the first The features of the layer after processing by the multi-head self-attention mechanism, FFN(.) is the feedforward neural network function, and LayerNorm(.) is the layer normalization function; The enhanced temporal representation is obtained after two layers of encoders, and the temporal dimension is aggregated by global average pooling. Finally, the dimensions are compressed using a linear transformation. The compression formula is as follows: ; in, The weight matrix is ​​a linear transformation matrix. For linear transformation bias terms, The feature vector after global average pooling. This is the compressed feature vector.

[0039] Furthermore, the image feature extraction branch converts the one-dimensional signal into a two-dimensional gram angle field, and then uses a convolutional neural network to extract its geometric structural features; First, regarding a one-dimensional signal segment... First, perform min-max standardization to map it to the interval [−1, 1]. The mapping formula is: ; in, This represents the value of the original one-dimensional signal segment at time point t. It is the minimum value in the original signal segment. The maximum value in the original signal segment. This represents the value of a standardized one-dimensional signal segment at time point t. The polar angle sequence is calculated using the following formula: ; in, The value of the standardized one-dimensional signal segment at time point t. The extreme angle at time point t; Combined with time angle Construct a summation-type Gram angle field, as shown in the following formula: ; in, To calculate the value of the summation-type Gram angle field at position (i,j), the final image tensor I is generated.

[0040] The three-parallel-branch MSCNN structure is configured with 3×3, 5×5, and 7×7 convolutional kernels respectively to capture multi-granularity spatial features. The input heatmap is used, and each branch consists of two convolutional layers, followed by batch normalization and ReLU activation. Finally, max pooling is used to downsample and compress the spatial dimension. The calculation formula is as follows: ; in, (.) represents the ReLU activation function, and BN(.) represents the batch normalization operation. This indicates a convolution operation using a k×k kernel in the i-th layer. For max pooling operation, , , Feature maps output for different branches; Next, adaptive average pooling is used to compress the feature maps of each branch into feature vectors, and the three feature vectors are concatenated to obtain... The obtained time series features and image features The data is concatenated, and the attention intermediate representation is calculated, along with the normalized weights: ; in, For attention intermediate representation, To normalize the weights, , These are the weight matrix and bias term for generating the intermediate representation of the attention function, respectively. , These are the weight matrix and bias term, representing the normalized weights generated by the attention intermediates. Obtain fusion features The fused features are input into a two-layer fully connected network, and the anomaly probability is output. The calculation formula is as follows: ; Among them, the output of the hidden layer h is... (.) represents the Sigmoid function. The output represents the probability of an anomaly occurring. , These are the weight matrix and bias terms of the first fully connected layer network, respectively. , These are the weight matrix and bias term of the second fully connected network, respectively. End-to-end training is performed using the binary cross-entropy loss function, and the training formula is as follows: ; Where N is the sample size. For loss training.

[0041] Preferably, in response to the problem of insufficient explanation for anomaly localization, this invention introduces gradient-weighted class activation mapping for interpretability analysis. For the MSCNN-Transformer model in this invention, it is applied to the last convolutional layer of the multi-scale CNN image encoder to generate heatmaps of anomaly types. The calculation steps of the heatmap include: calculating the prediction score of target category c. For feature maps The gradient of the feature map is calculated, and global average pooling is performed on the gradient at all spatial locations (u, v) of each feature map to obtain the importance weight of that feature map. , Where Z is the total number of pixels in the feature map. The predicted score for target category c. Let be the value of the k-th feature map at spatial location (u,v); All weights With the corresponding feature map We perform a weighted summation and retain features that contribute positively to classification using the ReLU activation function. The calculation formula is as follows: ; in, For the k-th feature map, The importance weights of the kth feature map Activate the mapping for the class; Finally, bilinear interpolation is used to... Upsampled to the original GAF image size, and superimposed on the original image to generate a heatmap. The color intensity reflects the degree of influence of the structural changes in the phase space of the corresponding time domain interval on the anomaly detection.

[0042] In the proposed dual-branch anomaly detection model, the temporal feature extraction branch employs a two-layer Transformer encoder structure, with each layer containing four attention heads. The hidden layer dimension is set to 128, and the model dimension is 64. The image feature extraction branch uses a three-parallel-branch MSCNN structure, with convolutional kernel sizes of 3×3, 5×5, and 7×7, respectively. Each branch has 32 output channels, and the pooling layer uses 2×2 max pooling. The feature fusion layer achieves adaptive weighted fusion of temporal and image features through an attention mechanism, and the number of neurons in the fully connected layer is set to 16. During model training, the initial learning rate is set to 1e-4, the batch size is 32, the total number of iterations is 80, the AdamW optimization algorithm is used, and the weight decay coefficient is set to 1e-5. The training environment is configured for GPU acceleration, and the gradient clipping threshold is set to 1.0. The preprocessed training set is input into the network for training, and the results are as follows. Figure 3 As shown, the classification accuracy curve and loss curve obtained by the method proposed in this invention tend to converge in the later stage of training iteration, indicating that the model is well trained.

[0043] The test set is input into the trained two-branch anomaly detection model to perform classification prediction of unknown samples, resulting in... Figure 4 The confusion matrix shown in Table 1 demonstrates the effectiveness of the proposed method in detecting abnormal samples in water diversion projects. The classification accuracy, precision, and F1-score of the model were calculated using the method proposed in this invention. The accuracy of both normal and faulty samples was above 98%, as shown in Table 1.

[0044] Table 1 Performance Indicators for Fault Diagnosis by Category To further verify the diagnostic performance of the proposed dual-branch multimodal fusion method, its F1 score and accuracy were compared with those of Transformer temporal single-modal, MSCNN image single-modal, simple feature concatenation fusion, and traditional single-branch CNN models. The experiment was repeated 10 times with the same training set, test set, and hyperparameter configuration, and the average results are shown in Table 2. It can be seen that the proposed method achieves the best performance in both F1 score and accuracy, with an F1 score of 98.67% and an accuracy of 98.67%. Compared to the second-best simple feature concatenation fusion model, the F1 score is improved by 3.2%, and the accuracy by 2.8%.

[0045] Table 2 Model Performance Comparison like Figure 5 As shown, the feature space distribution of the multi-scale IMF component input obtained based on ICEEMDAN decomposition is illustrated. The figure shows that before deep network feature extraction, the filtered and standardized multi-scale features exhibit a certain clustering trend in the low-dimensional space, but some overlapping regions still exist between samples of different categories. Figure 6 As shown in the figure, the deep fusion feature distribution is obtained after processing by the proposed dual-branch multimodal fusion model. It is evident from the figure that after nonlinear transformation and attention fusion, the feature representations of similar samples are more compact and cohesive, and normal and abnormal samples form clearly separated clusters. The decision boundaries between different clusters are also clearer. Furthermore, the feature distribution exhibits a good manifold structure, indicating that the feature representations learned by this model possess excellent discriminative and generalization abilities, providing a high-quality feature representation foundation for subsequent anomaly detection.

[0046] Figure 7 and Figure 8 The images show a comparison of Grad-CAM heatmaps for abnormal and normal samples. Figure 7The heatmaps of the anomalous samples clearly show significant energy concentration in the central region of the image and near the time diagonal. The color distribution indicates high activation intensity, suggesting that the model generated a strong activation response in these regions. From... Figure 8 The heatmap of the normal samples shows that the activation regions are relatively evenly distributed, the color distribution is relatively smooth, and there are no obvious local bright areas, indicating that the model pays relatively balanced attention to various regions of the image when processing normal samples. This comparison clearly demonstrates that the model can accurately locate key regions related to anomalous features in the Gram field image, making the model's decision-making process transparent and traceable, and providing intuitive visualization evidence for anomaly diagnosis.

[0047] In addition, to verify the effectiveness of the adaptive attention fusion mechanism proposed in this invention, a statistical distribution analysis was performed on the fusion weights of all samples in the test set, and the results are as follows: Figure 9 As shown.

[0048] As can be seen from the figure, the distributions of temporal feature weights and image feature weights are highly concentrated and overlapped, with their peak values ​​all concentrated around 0.5. This indicates that the fusion mechanism has a certain degree of stability and reliability. In addition, it shows that the attention mechanism can effectively balance the importance of two different modal information, fully explore the complementary advantages of multimodal fusion, avoid modal bias, and also indirectly confirm the rationality and effectiveness of the dual-branch structure and attention fusion strategy adopted in this invention.

[0049] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0050] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for detecting multi-source heterogeneous anomalies based on time correlation, characterized in that, Includes the following steps: S1: Obtain multi-source sensor time-series data from multi-source heterogeneous data, and preprocess the multi-source sensor time-series data to divide it into training set and test set; S2: Construct a dual-branch deep feature extraction network model, train it using the training set, and test it using the test set to obtain the trained dual-branch deep feature extraction network model; The dual-branch deep feature extraction network model includes a dual-branch unit, an attention feature fusion unit, a classifier unit, and an output layer unit connected in sequence. S3: Input the data to be tested into the trained dual-branch deep feature extraction network model, and finally output the anomaly diagnosis result.

2. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 1, characterized in that, Step S1, which involves preprocessing the multi-source sensor time-series data to divide it into training and testing sets, includes: S11: The time-series data from the multi-source sensors are sampled through a sliding window to obtain normal and abnormal datasets with a feature dimension of 2000×6. S12: Adaptively decompose the normal and abnormal datasets using ICEEMDAN, and obtain a new dataset by introducing a residual-driven ensemble strategy and dynamic noise amplitude adjustment, and then divide it into training and test sets.

3. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 2, characterized in that, The integration strategy described in step S12, which involves introducing residual-driven approaches, includes: Step 1: Add I=100 independent Gaussian white noise to the multi-source sensor time-series data to construct a signal set, the signal set being: ; Where s(t) represents each one-dimensional subsequence signal, It is Gaussian white noise with a mean of 0 and a standard deviation of 1. For noise system, For the number of integrations; Step 2: Perform EMD decomposition on each signal set, extract its first-order intrinsic mode functions (IMFs), and calculate the ensemble average as the final first-order IMF. The calculation formula is as follows: ; in, The value of the first-order intrinsic mode function at time point t. For the number of integrations, Let be the value of the signal at time t after the i-th addition of Gaussian white noise; Step 3: Update the residuals For the extraction of the k-th (k≥2) eigenmode function, a residual-driven strategy is adopted, which superimposes noise with attenuated amplitude onto the current residual. The extraction formula is as follows: ； in, For the current residual signal The value of the signal with added noise at time point t. Let be the value of the (k-1)th order residual signal at time point t. Here, k is the noise amplitude adjustment factor, and k is the order of the currently extracted IMF. Let be the value of the i-th independent Gaussian white noise at time point t; Step 4: Obtain the k-th order intrinsic mode function through the EMD decomposition and ensemble averaging, and update the residuals. The iteration stops when the maximum number of decomposition levels is reached, and a set of intrinsic mode functions is finally obtained. .

4. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 2, characterized in that, The dynamic noise amplitude adjustment mentioned in step S12 includes: First, the energy-dominant criterion is used to screen effective components, and the energy of each intrinsic mode function is calculated using the following formula: ; in, Let be the energy of the k-th eigenmode function. Let N be the value of the k-th order intrinsic mode function at time point t, N be the length of the time series, k be the order of the currently extracted IMF, and K be the maximum number of decomposition layers. Then, the eigenmode functions are sorted from largest to smallest energy, and the first three eigenmode functions are selected to form an eigenma matrix. For each eigenmode function component in the selected eigenma matrix... Perform independent Z-score standardization. The Z-score standardization formula is: ； in, For the standardized first The nth sample, the th The value of the k-th eigenmode function in the eigenmatrix at time t. For the first The nth sample, the nth The original values ​​of the k-th eigenmode function in the eigenmatrix at time t. For the first The nth sample, the th The mean of the k-th order eigenmode function in the eigenmatrix For the first The nth sample, the th The standard deviation of the k-th eigenmode function in the eigenmatrix, where k is the selected IMF order; , , where L is the length of the time series, thus obtaining a new dataset and dividing it into training and test sets.

5. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 1, characterized in that, The dual-branch unit mentioned in step S2 includes a temporal feature extraction branch and an image feature extraction branch; The temporal feature extraction branch includes a two-layer Transformer encoder structure, each layer containing four attention heads, with the hidden layer dimension set to 128 and the model dimension to 64. The image feature extraction branch adopts a three-parallel-branch MSCNN structure, with convolutional kernel sizes set to 3×3, 5×5 and 7×7 respectively, and each branch has 32 output channels. The pooling layer adopts 2×2 max pooling.

6. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 1, characterized in that, The attention feature fusion unit uses an attention mechanism to achieve adaptive weighted fusion of temporal and image features, and the number of neurons in the fully connected layer is set to 16.

7. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 5, characterized in that, The temporal feature extraction branch processes the temporal data in the training set and the test set, and uses a Transformer encoder to capture long-range dependencies. Let the input feature matrix be... ; First, it is mapped to a high-dimensional hidden space using linear projection. The formula for the output dimension is: ;in, The projection weight matrix is... For bias terms, For output dimensions; Then, learnable positional encoding is introduced to preserve time step information: ; Where P represents the trainable location embedding parameters. Add the dimension after position encoding; It adopts a stacked Transformer encoder structure with a total of L=2 layers. Each layer contains a multi-head self-attention mechanism and a feedforward neural network, and is equipped with residual connections and layer normalization. The calculation formula for the multi-head attention mechanism is as follows: ; in, , For the first The input feature matrix of the layer, For the projected weight matrix of the query, Let be the projection weight matrix of the key. The projection weight matrix is ​​the value of the value. For the output of the h-th attention head, Let be the dimension of the key vector. These are query, key, and value matrices, respectively. H =4; Next, multi-head output splicing and linear transformation are performed, followed by residual connection and layer normalization to obtain... ; The calculation formula for the feedforward neural network is as follows: ; in, This is the weight matrix of the first layer of the feedforward neural network. This is the weight matrix of the second layer of the feedforward neural network. This is the bias vector of the first layer of the feedforward neural network; This is the bias vector for the second layer of the feedforward neural network; Final output: ; in, For the first The features of the layer after processing by the multi-head self-attention mechanism, FFN(.) is the feedforward neural network function, and LayerNorm(.) is the layer normalization function; The enhanced temporal representation is obtained after two layers of encoders, and the temporal dimension is aggregated by global average pooling. Finally, the dimensions are compressed using a linear transformation. The compression formula is as follows: ; in, The weight matrix is ​​a linear transformation matrix. For linear transformation bias terms, The feature vector after global average pooling. This is the compressed feature vector.

8. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 5, characterized in that, The image feature extraction branch converts a one-dimensional signal into a two-dimensional Gram angle field, and then uses a convolutional neural network to extract its geometric structural features. First, regarding a one-dimensional signal segment... First, perform min-max standardization to map it to the interval [−1, 1]. The mapping formula is: ; in, This represents the value of the original one-dimensional signal segment at time point t. The minimum value in the original signal segment. The maximum value in the original signal segment. This represents the value of a standardized one-dimensional signal segment at time point t. The polar angle sequence is calculated using the following formula: ; in, The value of the standardized one-dimensional signal segment at time point t. The extreme angle at time point t; Combined with time angle Construct a summation-type Gram angle field, as shown in the following formula: ; in, To calculate the value of the summation-type Gram angle field at position (i,j), the final image tensor I is generated.

9. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 5, characterized in that, The three-parallel-branch MSCNN structure is configured with 3×3, 5×5, and 7×7 convolutional kernels respectively to capture multi-granularity spatial features. The input heatmap is used, and each branch consists of two convolutional layers, followed by batch normalization and ReLU activation. Finally, max pooling is used to downsample and compress the spatial dimension. The calculation formula is as follows: ； in, (.) represents the ReLU activation function, and BN(.) represents the batch normalization operation. This indicates a convolution operation using a k×k kernel in the i-th layer. For max pooling operation, , , Feature maps output for different branches; Next, adaptive average pooling is used to compress the feature maps of each branch into feature vectors, and the three feature vectors are concatenated to obtain... The obtained time series features and image features The data is concatenated, and the attention intermediate representation is calculated, along with the normalized weights generated. ; in, For attention intermediate representation, To normalize the weights, , These are the weight matrix and bias term used to generate the intermediate representation of the attention function, respectively. , These are the weight matrix and bias term, representing the normalized weights generated by the attention intermediates. Obtain fusion features The fused features are input into a two-layer fully connected network, and the anomaly probability is output. The calculation formula is as follows: ； Among them, the output of the hidden layer h is... (.) represents the Sigmoid function. The output represents the probability of an anomaly occurring. , These are the weight matrix and bias terms of the first fully connected layer network, respectively. , These are the weight matrix and bias term of the second fully connected network, respectively. End-to-end training is performed using the binary cross-entropy loss function, and the training formula is as follows: ; Where N is the sample size. For loss training.

10. The method for detecting multi-source heterogeneous anomalies based on time correlation according to claim 9, characterized in that, The calculation steps for the heat map include: Calculate the predicted score for target category c For feature maps The gradient of the feature map is calculated, and global average pooling is performed on the gradient at all spatial locations (u, v) of each feature map to obtain the importance weight of that feature map. , Where Z is the total number of pixels in the feature map. The predicted score for target category c. Let be the value of the k-th feature map at spatial location (u,v); All weights With the corresponding feature map We perform a weighted summation and retain features that contribute positively to classification using the ReLU activation function. The calculation formula is as follows: ; in, For the k-th feature map, The importance weights of the k-th feature map Activate the mapping for the class; Finally, bilinear interpolation was used to... Upsampled to the original GAF image size, and superimposed on the original image to generate a heatmap. The color intensity reflects the degree of influence of the structural changes in the phase space of the corresponding time domain interval on the anomaly detection.

Images